HyperHuman: Hyper-Realistic Human Generation with Latent Structural Diffusion
Authors: Xian Liu
Jian Ren
Aliaksandr Siarohin
Ivan Skorokhodov
Yanyu Li
Dahua Lin
Xihui Liu
Ziwei Liu
Sergey Tulyakov
Venue: ICLR 2024 poster (2024)
Keywords: Human Image Generation
Latent Structural Diffusion
Primary area: generative models
TL;DR: We propose a unified framework, HyperHuman with latent structural diffusion, that generates in-the-wild human images of high realism and diverse layouts.

Abstract: Despite significant advances in large-scale text-to-image models, achieving hyper-realistic human image generation remains a desirable yet unsolved task. Existing models like Stable Diffusion and DALL·E 2 tend to generate human images with incoherent parts or unnatural poses. To tackle these challenges, our key insight is that human image is inherently structural over multiple granularities, from the coarse-level body skeleton to fine-grained spatial geometry. Therefore, capturing such correlations between the explicit appearance and latent structure in one model is essential to generate coherent and natural human images. To this end, we propose a unified framework, HyperHuman, that generates in-the-wild human images of high realism and diverse layouts. Specifically, 1) we first build a large-scale human-centric dataset, named HumanVerse, which consists of 340M images with comprehensive annotations like human pose, depth, and surface normal. 2) Next, we propose a Latent Structural Diffusion Model that simultaneously denoises the depth and surface normal along with the synthesized RGB image. Our model enforces the joint learning of image appearance, spatial relationship, and geometry in a unified network, where each branch in the model complements to each other with both structural awareness and textural richness. 3) Finally, to further boost the visual quality, we propose a Structure-Guided Refiner to compose the predicted conditions for more detailed generation of higher resolution. Extensive experiments demonstrate that our framework yields the state-of-the-art performance, generating hyper-realistic human images under diverse scenarios.

Published as a conference paper at ICLR 2024
HYPERHUMAN: HYPER-REALISTIL HUMAN GENERA- TION WITH LATENT STRUCTURAL DIFFUSION
Xian Liu l
Jian Ren'
Aliaksandr Siarohin'
Ivan Skorokhodoy
Yanyu Li'
Dahua Lin
Xihui Liu
Ziwei Liu
Sergey Tulyakov'
'Snap Inc.
2CUHK HKK
NNU
Project Page:
ABSTRACT
Despite significant advances in large-scale text-to-image models, achieving hyper- realistic human image generation remains desirable yet unsolved task. Existing models like Stable Diffusion and DALL-E 2 tend to generate human images with incoherent parts unnatural poses. To tackle these challenges, our key insight is that human image is inherently structural over multiple granularities, from the coarse-level body skeleton to the fine- fine-grained spatial geometry. Therefore, captur- ing such correlations between the explicit appearance and latent structure in one model is essential to generate coherent and natural human images. To this end, we propose a unified framework, Hyper Human, that generates in-the-wild human images of high realism and diverse layouts. Specifically, 1) we first build a large- scale human-centric dataset, named Human Verse, which consists of 340M images with comprehensive annotations like human pose, depth, and normal. 2) Next, we propose Latent Structural Diffusion Model that simultaneously de- noises the depth and surface-normal along with the synthesized RGB image. Our model enforces the joint learning of image appearance, spatial relationship, and geometry unified network, where each branch in the model complements each other with both structural awareness and textural richness. 3) Finally, further boost the visual quality, we propose a Setucctrr-Guiied Refiner to com- pose the predicted conditions for more detailed generation of higher resolution. Extensive experiments demonstrate that our framework yields the performance, generating hyper-realistic human images under diverse scenarios.
INTRODUCTION
Generating hyper-realistic human images from conditions, e-g-, text and pose, is of great impor- tance to various applications, such as image animation (Liu et al., 2019) and virtual try-on (Wang al., 2018). To this end, many efforts explore the task of controllable human image generation. Early methods either resort to variational auto-encoders (VAEs) in a reconstruction manner (Ren al., 2020), or improve the realism by generative adversarial networks (GANs) (Siarohin et al., 2019). Though some of them create high-quality images (Zhang et al., 2022; Jiang et al., 2022), the unstable training and limited model capacity confine them to small datasets of low diversity. Recent emergence of diffusion models (DMs) (Ho et al., 2020) has set a new paradigm for realistic synthesis and become the predominant architecture in Generative AI (Dhariwal & Nichol, 2021). Nevertheless, the exemplar text-to-image (T2I) models like Stable Diffusion (Rombach et al., 2022) and DALL-E 2 (Ramesh et al., 2022) still struggle to create human images with coherent anatomy, e.g., arms and legs, and natural poses. The main reason lies in that human is articulated with non- rigid deformations, requiring structural information that can hardly be depicted by text prompts.
To enable structural control for image generation, recent works like ControlNet (Zhang & Agrawala, 2023) and T2I-Adapter (Mou et al., 2023) introduce a learnable branch to modulate the pre-trained DMs, e.g., Stable Diffusion, in plug-and-play manner. However, these approaches suffer from the
Work done during an internship at Snap Inc. Email: alvinliu euu.c. Corresponning author; jren snapchat.com.

---

Published as a conference paper at ICLR 2024
alttlng with brawe toing ahevlnen.
The NONKE, rriling Seceoa willcanlling AttrOOMMEN
Bii in Tenen, pacct Pitnding nddr agg
noman paa with Fictele Arndiaco uasS? octiher rubrunac
An calrriy tewwa Gonu be Cile
in AFanETiaza Lo ratburrc
bilture ofa mas wils sall, ball.
Yhing mase stondingg Sitte nipgot ausuttiiin behind.
Lattl get pacing for tue sarr with stall Car dhiped titatiiioo
pub on tng sidw wite wrts and snawy hills tahind.
showllng laies
Minttoost Michor a baiie US SPARNIN morrtt FUN with eleci. orringi
wnw.niiiii of www.aaoorraaaaaaaaa. Sharral
parandrr doon ANN Edding
Lrams Sort MALL WEE oitiie atz,, sinnyy with ww
Oo Size Danolidng
Carrs Full
EO CastrolNue
T23- Adiggte
(1 SOXL IT Beffeer
Figure 1: Example Results and Visual Comparison. Top: The proposed HyperHuman simulta- neously generates the coarse RGB, depth, normal, and high- regh-reeolution images conditioned on text and skeleton. Both photo-realistic images and stylistic renderings can be created. Bottom: We com- pare with recent T2I models, showing better realism, quality, diversity, and controllability. Note that in each 2 x 2 grid (left), the upper-left is input skeleton, while the others are jointly denoised normal, depth, and coarse RGB of 512 512. With full model, we synthesize images up to 1024 x 1024 (right). Please refer Sec. A.15, A.16 for more comparison and results. Best viewed zoom in.
feature discrepancy between the main and auxiliary branches, leading to inconsistency between the control signals (e.g., pose maps) and the generated images. To address the issue, HumanSD (Ju al., 2023b) proposes to directly input body skeleton into the diffusion U-Net by channel-wist concatena- tion. However, it is confined to generating artistic style images of limited diversity. Besides, human images synthesized only with pose control, while other structural information like depth maps and surface-normal maps are not considered. In a nutshell, previous studies either take a singular control signal input condition, treat different control signals separately independent guid- ance, instead of modeling the multi-level correlations between human appearance and different types of structural information. Realistic human generation with coherent structure remains unsolved.
2

---

Published as a conference paper at ICLR 2024
In this paper, we propose a unified framework HyperHuman to generate wn-the-wild human images of high realism and diverse layouts. The key insight is that human image is inherently structural over maltiple gramularities, from the coarse-level body skeleton fine- grained spatial geometry. Therefore, capturing such correlations between the explicit appearance and latent structure in one model is essential to generate coherent and natural human images. Specifically, we first establish a large-scale human-centric dataset called Human that contains 340M in-the-wild human im- ages of high quality and diversity. It has comprehensive annotations, such as the coarse-level body skeletons, the fine- ime-grained depth and surface-normal maps, and the high-level image captions and attributes. Based on this, two modules are designed for hyper-realistic controllable human image generation. In Latent Structural Diffusion Model, we augment the pre-trained diffusion backbone to simultaneously denoise the RGB, depth, and normal. Appropriate network layers are chosen to be replicated as structural expert branches, so that the model can both handle input/output of different domains, and guarantee the spatial alignment among the denoised textures and structures. Thanks to such dedicated design, the image appearance, spatial relationship, and geometry are jointly modeled within a unified network, where each branch is complementary to each other with both structural awareness and textural richness. To generate monotonous depth and surface-normal that have similar values in local regions, we utilize an improved noise schedule to eliminate low-frequency informa- tion leakage. The same timestep is sampled for each branch to achieve better learning and feature fusion. With the spatially-aligned structure maps, in Structure- Guidedd Refiner, we compose the predicted conditions for detailed generation of high resolution. Moreover, we design a robust con- ditioning scheme to mitigate the effect of eπror accumulation in our two-stage generation pipeline.
To summarize, our main contributions are threc-fold: 1) We propose novel HyperHuman frame- work for -ild controllable human image generation of high realism. A large-scale human- centric dataset Human Verse is curated with comprehensive annotations like human pose, depth, and surface normal. As one of the earliest attempts in human generation foundation model, we hope to benefit future research. 2) We propose the Latent Structural Diffusiio Model to jointly capture the image appearance, spatial relationship, and geometry in unified framework. The Structure- Guided Refiner is further devised to compose the predicted conditions for generation of better visual qual- ity and higher resolution. 3) Extensive experiments demonstrate that our HyperHuman yields the state-of-theear performance, generating hyper-realistic human images under diverse scenarios.
2 RELATED WORK
Text-to-Image Image Difftusion Models. Text-to-image (T2I) generation, the endeavor to synthesize high- fidelity images from natural language descriptions, has made remarkable strides in recent years. Dis- tinguished by the superior scalability and stable training, diffusion- T2I models have eclipsed conventional GANs in terms of performance (Dhariwal & Nichol, 2021), becoming the predominant choice in generation (Nichol et al., 2021; Saharia et al., 2022; Balaji et al., 2022; Li et al., 2023). By formulating the generation as an iterative denoising process (Ho et al., 2020), exemplar works like Stable Diffusion (Rombach et al., 2022) and D 2 (Ramesh et al., 2022) demonstrate un- precedented quality. Despite this, they mostly fail to create high-fidelity humans. One main reason is that existing models lack inherent structural awareness for human, making them even struggle to generate human of reasonable anatomy, e.g., correct number of arms and legs. To this end, our proposed approach explicitly models human structures within the latent space of diffusion model.
Controllable Human Image Generation. Traditional approaches for controllable human genera- tion can be categorized into GAN-based (Zhu et al., 2017; Siarohin et al., 2019) and VAE-based (Ren et al., 2020; Yang et al., 2021), where the reference image and conditions are taken as input. To fa- cilitate user-friendly applications, recent studies explore text prompts as generation guidance (Roy et al., 2022; Jiang et al., 2022), yet are confined to simple pose or style descriptions. The most relevant works that enable open- vocabulary cose-guided controllable human synthesis are Control- Net (Zhang& & Agrawala, 2023), T21-Adapter (Mou et al., 2023), and HumanSID (Ju et al., 2023b). However, they either suffer from inadequate pose control, or are confined to artistic styles of limited diversity. Besides, most previous studies merely take pose as input, while ignoring the multi-level correlations between human appearance and different types of structural information. In this work, we propose to incorporate structural awareness from coarse-level skeleton to fine fine-grained depth and surface-norman by joint denoising with expert branch, thus simultancously capturing both the ex- plicit appearance and latent structure in unified framework for realistic human image synthesis.
3

---

Published as a conference paper at ICLR 2024
Seructrr-Caadd Gastel Refieerr (he.
Ne.dy,
du
Pre SDXL
Irage Normal
Juint Danihe walh Branch
Jan Derxie Buach
Prsc Stelehes
Kerea Poger Soaeis
Skpp Coeaorion
Desh Expat Bladd
Insag Capticu Tine Step
Prodicted Image Depith d. Normal
HUl Exper Brundh
Shard Blacss
Figure 2: Overview of HyperHuman Framework. In Latent Structural Diffussio Model (purple), the image x, depth d, and surface-normal n are jointly denoised conditioning on caption and pose skeleton p. For the notation simplicity, we denote pixel-/latent-space targets with the same variable. In Structure- Grrdeed Refiner (blue), we compose the predicted conditions for higher-resolutin gen- eration. Note that the grey images refer to randomly dropout conditions for more robust training.
Datasets for Human Image Generation. Large datasets are crucial for image generation. Existing human-centric collections are mainly confronted with following drawbacks: Low-resolution of poor quality. For example, Market-1501 (Zheng et al., 2015) contains noisy pedestrian images of resolution 128 x 64, and VITON (Han et al., 2018) has human-clothing pairs of 256 x 192, which inadequate for training high-definition models. 2) Limited diversity of certain domain. For example, SHHQ (Fu et al., 2022) is mostly composed of full-body humans with clean background, and DeepFashion (Liu et al., 2016) focuses fashion images of little pose variations. 3) Insufficient dataset scale, where LIP (Gong et al., 2017) and Human-Art (Ju et al., 2023a) only contain 50K samples. Furthermore, none of the existing datasets contain rich annotations, which typically label singular aspect of images. In this work, we take a step further by curating in-the-wild Human Verse dataset with comprehensive annotations like human pose, depth map, and nnrface-normal map.
OUR APPROACH
We present HyperHuman that generates in-the-will human images of high realism and diverse layouts. The overall framework is illustrated Fig. 2. To make the content self-contained and narration clearer, we first introduce some pre-requisites of diffusion models and the problem setting in Sec. 3.1. Then, we present the Latent Structural Diffusion Model which simultaneously denoises the depth, surface-normal along with the RGB image. The explicit appearance and latent structure are thus jointly learned in a unified model (Sec. 3.2). Finally, we elaborate the StructureGGuide Refiner to compose the predicted conditions for detailed generation of higher resolution in Sec. 3.3.
3.1 PRELIMINARIES AND PROBLEM SETTING
Diffusion Probabilistic Models define forward diffusion process to gradually convert the sample from a real data distribution Paata (x) into a noisy version, and learn the reverse generation process in an iterative denoising manner (Sohl-Dickstein et al., 2015; Song al., 2020b). During the sampling stage, the model can transform Gaussian noise of normal distribution to real samples step-by-step The denoising network (-).) estimates the additive Gaussian noise, which is typically structured as UNet (Ronneberger et al., 2015) to minimize the ensemble of mean-squared error (Ho et al., 2020);
min E Ex,c.e. [w (a, + o) E
where Pdata are the sample-condition pairs from the training distribution; ~VO, is the ground-truth noise; t ~ 2[11,7 T is the time-step and is the training step number; Otz, Tr, and we the terms that control the noise schedule and sample quality decided by the diffusion sampler.
Latent Diffusion Model & Stable Diffusion. The widely-used latent diffusson model (LDM), with its improved version Stable Diffusion (Rombach et al., 2022), performs the denoising process in separate latent space to reduce the computational cost. Specifically, a pre-trained VAE (Esser et al., 2021) first encodes the image latent embedding E(x) for DM training. At the inference

---

Published as a conference paper at ICLR 2024
stage, reconstruct the generated image through the decoder x D(z). Such design enables the SD to scale up to broader datasets and larger model size, advancing from the SD Ixx 2.x series to SDXL of heavier backbone on higher resolution (Podell et al., 2023). In this work, we extend SD 2.0 to Latent Structural Diffusion Model for efficient capturing of explicit appearance and latent structure, while the Structure- Structure-Guided Refiner is built SDXI. 1.0 for more pleasing visual quality.
Problem Setting for Controllable Human Generation. Given a collection of N human images with their captions c, we annotate the depth d, surface-normal n, and pose skeleton p for each sample (details elaborated in Sec. 4). The training dataset can be denoted as {x,, Ci, di, n2 P: J= In the first-stage Latent Structural Diffusion Model G1, estimate the RGB image x, depth d, and surface-normal conditioned on the caption с and skeleton p. In the second-stage Structure- Guided Refiner G2, the predicted structures of d and n further serve as guidance for the generation of higher-resolution results x highh-es The training setting for our pipeline can be formulated as:
8 š, á (c, P). Lhiigh = G2 p. n.
During inference, only the text prompt and body skeleton needed to synthesize well-aligned RGB image, depth, and surface- -oommall. Note that the users are free to substitute their own depth and surface-normal conditions to G2 if applicable, enabling more flexible and controllable generation.
3.2 LATENT STRUCTURAL DIFFUSION MODEL
To incorporate the body skeletons for pose control, the simplest way is by feature residual (Mou et al.. 2023) or imput concatenation (Ju et., al., 2023b). However, three problems remain: 1) The sparse keypoints only depict the coarse human structure, while the fine -me-graindd geometry and foreground- background relationship are ignored. Besides, the naive DM training is merely supervised by RGB signals, which fails to capture the inherent structural information. 2) The image RGB and structure representations are spatially aligned but substantially different in latent space. How to jointly model them remains challenging. 3) In contrast to the colorful RGB images, the structure maps are mostly monotonous with similar values in local regions, which are hard to learn by DMs (Lin et al., 2023).
Unified Model for Simultancous Denoising. Our solution to the first problem is to simultaneously denoise the depth and surface-normal along with the synthesized RGB image. We choose them as additional learning targets due to two reasons: 1) Depth and normal can be easily annotated for large-scale dataset, which are also used in recent controllable T2I generation (Zhang & Agrawala, 2023). 2) As two cose structural guidance, they complement the spatial relationship and geometry information, where the depth (Deng et al., 2022), normal (Wang et al., 2022), or both (Yu et al., 2022b) are proven beneficial in recent 3D studies. To this end, naive method is to train three separate networks to denoise the RGB, depth, and normal individually. But the spatial alignment between them is hard to preserve. Therefore, we propose to capture the joint distribution in a unified model by simultaneous denoising, which can be trained with simplified objective (Ho et al., 2020):
cepred_ = F.x.d.ne..p.... || c. - 1x + [(ēo(d).; :., - call2 + [ - 2 (3)
denoise image x
denoise depth
denoise normal
where Ex. Ed, and en ~ (O, I) are three independently sampled Gaussian noise (shortened as e in expectation for conciseness) for the RGB, depth, and normal, respectively; Xt_ Or,xx + Ex.
dia = ar d oto and D,, = a., n + o,,, En are the noised feature maps of three learning targets; tx fa. and fn ~ L[[1, TJ are the sampled time-steps that control the scale of added Gaussian noise.
Structural Expert Branches with Shared Backbone. The diffusion UNet contains down-sample, middle, and up- up-sample blocks, which are interleaved with convolution and aooosasrosttt lay- ers. In particular, the DownBlocks compress input noisy latent to the hidden states of lower resolu- tion, while the UpBlocks conversely upscale intermediate features to the predicted noise. Therefore, the most intuitive manner is to replicate the first several DownBlocks and the last several UpBlocks for each expert branch, which are the most neighboring layers to the input and output. In this way, each expert branch gradually maps input noisy latent of different domains (i.e., Xtg, dig- and n,,)) to similar distribution for feature fusion. Then, after a series of shared modules, the same feature is distributed to each expert branch to output noises (i.e., Cx2 Ca. and c,) for apatially-aligna results.
Furthermore, we find that the number of shared modules can trade-off between the spatial alignment and distribution learning: On the one hand, more shared layers guarantee the more similar features
5

---

Published as a conference paper at ICLR 2024
of final output, leading to the paired texture and structure corresponding to the same image. On the other hand, the RGB, depth, and normal can be treated as different views of the same image, where predicting them from the same feature resembles Empirically, we find the optimal design to replicate the conv in, first DownBlock, last UpBlock, and conv_out for each expert branch, where each branch's skip-connections are maintained separately (as depicted in Fig. 2). This yields both the spatial alignment and joint capture of image texture and structure. Note that such design is not limited three targets, but can generalize to arbitrary number of paired distributions by simply involving more branches with little computation overhead.
image-to-image translation task in essence.
Noise Schedule for Joint Learning. A problem arises when we inspect the distribution of depth and surface-normal: After annotated by off- she-the-shllf estimators, they are regularized to certain data range with similar values local regions, e.g., [0, 1] for depth and unit vector for surface-normal. Such monotonous images may leak low-frequency signals like the mean of each channel during training. Besides, their latent distributions are divergent from that of RGB space, making them hard to exploit common noise schedules (Lin et al., 2023) and diffusion prior. Motivated by this, we first normalize the depth and normal latent features to the similar distribution of RGB latent, so that the pre-trained denoising knowledge can be adaptively used. The zero terminal SNR (a. 0, or 1) is further enforced to eliminate structure map's 's low- fow-frequeccy information. Another question is how to sample time-step for each branch. An alternative is perturb the data of different modalities with different levels (Bao et al., 2023), which samples different for each target as in Eq. 3. However, as we aim to jointly model RGB, depth, and normal, such strategy only gives 10 probability to sample each perturbation situation (given total steps T = 1000), which is too sparse to obtain good results. In contrast, we propose to densely sample with the same time-step & for all the targets, so that the sampling sparsity and learning difficulty will not increase even when we learn more modalities. With the same noise level for each structural expert branch, intermediate features follow the similar distribution when they fuse in the shared backbone, which could better complement to each others. Finally, we utilize the v-prediction (Salimans & Ho, 2022) learning target as network objective:
C B...d.n...pv.e [||vo(x; - v 12 + ([volde c, p) v² 11 + | c) - v.l (4)
where V.- O1Ex dyx, vil atEd o,d, and v.- agEn den are the v-prediction learning targets at time-step for the RGB, depth, and normal, respectively. Overall, the unified simultanc- ous denoising network vo with the structural expert branches, accompanied by the improved noise schedule and time-step sampling strategy give the first-stage Latent Structural Diffusion Model G1.
3.3 STRUCTURE- REFINER
Compose Structures for Controllable Generation. With the unified latent structural diffusion model, spatially-alignel conditions of depth and surface-normal can be predicted. We then learn a refiner network to render high-quality image higg-res by composing multi-conditions of caption c, pose skeleton p. the predicted depth d, and the predicted surface-normal n. In contrast to Zhang & Agrawala (2023) and Mou et al. (2023) that can only handle singular condition per run, propose to unify multiple control signals at the training phase. Specifically, we first project each condition from input image size (e.g.. 1024 1024) feature space vector that matches the size of SDXL (e.g.. 128 x 128). Each condition is encoded via a light-weight embedder of four stacked convolutional layers with 4 X kernels, 2 x strides, and ReLU activation. Next, the embeddings from each branch are summed up coordinate-wise and further feed into the trainable copy of SDXL Encoder Blocks. Since involving more conditions only incurs negligible computational overhead of tiny encoder network, our method can be trivially extended to new structural conditions. Although a recent work also incorporates multiple conditions in one model (Huang et al., 2023), they have to re- train the whole backbone, making the training cost unaffordable when scaling up to high resolution.
Random Dropout for Robust Conditioning. Since the predicted depth and surface-normal condi- tions from G1 may contain artifacts, a potential issue for such two-stage pipeline is the error accu- mulation, which typically leads to the train-test performance gap. To solve this problem, we propose to dropout structural maps for robust conditioning. In particular, we randomly mask out any of the control signals, such as replace text prompt with empty string, or substitute the structural maps with veroe images. In this way, the model will not solely rely thus balancing the impact of each condition robustly. To sum up, the couuttur--coposingg refiner network with robust conditioning scheme constitute the second-stage Structure- Guided Refiner G2-
single guidance for synthesis,

---

Published as a conference paper at ICLR 2024
4 HUMANVERSE DATASET
Large-scale datasets with high quality samples, rich annotations, and diverse distribution are crucial for image generation tasks (Schuhmann et al., 2022; Podell et al., 2023), especially in the human domain (Liu et al., 2016 6; Fu et al., 2022). To facilitate controllable human generation of high-fidelity, we establish comprehensive human dataset with extensive annotations named Human Verse. Please kindly refer to Appendix A.17 for more details about the dataset and annotation resources we use.
Dataset Preprocessing. We curate from two principled datasets: LAION-2B-or (Schuhmann al., 2022) and COYO-700M (Byeon et al., 2022). To isolate human images, we employ YOLOS (Fang et al., 2021) for human detection. Specifically, only those images containing to 3 human bounding boxes are retained, where people should be visible with an area ratio exceeding 15%. We further rule out samples of poor aesthetics (< 4.5) or low resolution (< 200 x 200). This yields a high-quality subset by eliminating blurry and over-small humans. Unlike existing models that mostly train on full-body humans of simple context (Zhang & Agrawala, 2023), our dataset encompasses a wider spectrum, including various backgrounds and partial human regions such clothing and limbs.
2D Human Poses. 2D human poses (skeleton of joints), which serve as one of the most flexible and easiest obtainable coarse-level condition signals, are widely used in controllable human generation studies (Ju et al., 2023b; Zhu et al., 2023; Yu et al., 2023; Liu et al., 2023; 2022a;b;c). To achieve accurate keypoint annotations, we resort to MMPose (Contributors, 2020) as inference interface and choose ViTPose-H (Xu et al., 2022) as backbone that performs best over several pose estimation benchmarks. In particular, the per-instance bounding box, keypoint coordinates and confidence are labeled, including whole-body skeleton, body skeleton, hand, and facial landmarks.
Depth and Murface-Normal Maps are fine-grained structures that reflect the spatial geometry of images (Wu et al., 2022), which are commonly used in conditional generation (Mou et al., 2023). We apply Omnidata (Eftekhar et al., 2021) for monocular depth and normal. The MiDaS (Ranftl et al., 2022) is further annotated following recent depth-to-image pipelines (Rombach et al., 2022).
Outpaint for Accurate Annotations. Diffusion models have shown promising results on image in- painting and outpainting, where the appearance and structure of unseen regions can be hallucinated based on the visible parts. Motivated by this, we propose to outpaint each image for a more holistic view given that most off-the-shelf structure estimators are trained on the "complete" image views. Although the outpainted region may be imperfect with artifacts, it can complement a more compre- hensive human structure. To this end, we utilize the powerful SD- 3D-Iniannt to outpaint the surrounding areas of the original canvas. These images are further processed by off-the-shel estimators, where only use the labeling within the original image region for more accurate annotations.
Overall Statistics. In summary, COYO subset contains 90, 948, 474 (91M) images and LAION-2B subset contains 248, 396, 109 (248M) images, which is 18.12% and 20.77% of fullset, respectively. The whole annotation process takes 640 16/32G NVIDIA V100 GPUs for two weeks in parallel.
5 EXPERIMENTS
Experimental Settings. For the comprehensive evaluation, we divide our comparisons into two settings: 1) Quantitative analysis. All the methods are tested on the same benchmark, using the same prompt with DDIM Scheduler (Song et al., 2020a) for 50 denoising steps to generate the same resolution images of 512 512. 2) Qualitative analysis. We generate high-resolution 1024 x 1024 results for each model with the officially provided best configurations, such as the prompt engineer- ing, noise scheduler, and classifier-free guidance (CFG) scale. Note that we use the RGB output of the first-stage Latent Struciural Diffusioo Model for numerical comparison, while the improved results from the second-stage Structure- Guided Refiner are merely utilized for visual comparison.
Datasets. We follow common practices in T2I generation (Yu et al., 2022a) and filter out a human subset from MS-COCO 2014 validation (Lin et al., 2014) for zero-shot evaluation. In particular, off- the-shelf human detector and pose estimator are used to obtain 8, 236 images with visarly humans for evaluation. All the ground truth images are resized and center- croppe to 512 x 512. To guarantee fair comparisons, we train first-stage Latent Structural Diffusion on Human Verse, which is a subset of public LAION-2B and COYO, to report quantitative metrics. In addition, an internal dataset is adopted to train second-stage Sruideur-Guudedd Refiner only for visually pleasing results.
7

---

Published as a conference paper at ICLR 2024
Table 1: Zero-Shot Evaluation on MS-COCO 2014 Validation Human. We compare our model with recent SOTA general T2I models (Rombach et al., 2022; Podell et al., 2023; DeepFloyd, 2023) and controllable methods (Zhang Agrawala, 2023; Mou et al., 2023; Ju et al., 2023b). Note that first generate
SDDL generates artistic style in 512, and IIF only creates fixed-size images, 1024 x 1024 results, then resize back to 512 x 512 for these two methods. We bold the best and underline the second results for clarity. Our improvements over the second method are shown in red.
Image Quality
Alignment
Pose Accuracy
Methods
FID +
KIDx +
FIDELP 4
CLIP f
APT
AR f
APokan t
ARatem 1
SD 1.5
24.26
8.69
12.93
31.72
SD 2.0
22.98
9.45
11.41
32.13
SD 2.1
24.63
9.52
15.01
32.11
SDXL
29.08
12.16
19.00
32.90
DeepFloyd-IF
29.72
15.27
17.01
32.11
ControlNet
27.16
10.29
15,59
31.60
20,46
30.23
25.92
38.67
T21-Adapter
23.54
7,98
11.95
32.16
27.54
36.62
34.86
46.53
HumanSD
52.49
33.96
21.11
29.48
26.71
36.85
32.84
45.87
HyperHuman
17.18 25.2
4.11
48.5%L
7.82
31.8%L
32.17
30.38
37.84
38.84
48.70
Comparison Methods. We compare with two categories of open-source SOTA works: 1) General T21 models, including SD (Rombach et al., 2022) (SD 1.x & 2.x). SDXL (Podell et al., 2023), and IF (DeepFloyd, 2023). 2) Controllable methods with pose condition. Notably, ControlNet (Zhang & Agrawala, 2023) and T2I-Adapter (Mou et al., 2023) can handle multiple structural signals like canny, depth, and normal, where we take their ixeleton-conditionee variant for comparison. Hu- manSD (Ju et al., 2023b) is the most recent work that specializes in pose-guided human generation.
Implementation Details. We resize and random-crop the RGB, depth, and normal the target resolution of each stage. To enforce the model with size and location awareness, the original image height/width and crop coordinates are embedded 2023). Our code is developed based on diffusers (von Platen et al., 2022). 1) For the Latent Struc- tural Diffusion, we fine-tune the whole UNet from the pretrained SD-2.0-base to v-prediction (Sal- imans & Ho, 2022) in 512 512 resolution. The DDIMScheduler with improved noise schedule is used for both training and sampling. We train on 128 80G NVIDIA A 1100 GPUs in a batch size of 2, 048 for one week. 2) For the Structure Guided Refiner, backbone and fine-tune to e-prediction for high-resolution synthesis of 1024 x 1024. We train on 256 80G NVIDIA A100 GPUs in batch size of 2, 048 for one week. The whole two-stage inference process takes 12 seconds on a single 40G NVIDIA A 110 GPU. The overall framework is optimized with Adam W (Kingma& & Ba, 2015) in le 5 learning rate, and 0.01 weight decay.
similar way to time embedding (Podell et al.,
choose SDXL-1.0-bas 1.0-base as the frozen
5.1 MAIN RESULTS
Evaluation Metrics. We adopt commonly-usec metrics to make comprehensive comparisons from three perspectives: 1) Image Quality. FID, KID, and FID used to reflect quality and diversity. 2) Text-Image Alignment, where the CLIP similarity between text and image embeddings is reported.
3) Pose Accuracy. We use the state-of-the-art pose estimator to extract poses from synthetic images and compare with the input (GT) pose conditions. The Average Precision (AP) and Average Recall (AR) are adopted to evaluate the pose alignment. Note that due to the noisy pose estimation of in- thel COCO, we also use AP and AR to only evaluate on the three most salient persons.
Quantitative Analysis. We report zero-shot evaluation results in Tab. For all methods, we use the default CFG scale of 7.5, which well balances the quality and diversity with appealing results. Thanks to the structural awareness from expert branches, our proposed HyperHuman outperforms previous works by clear margin, achieving the best results on image quality and pose accuracy metrics and ranks second on CLIP score. Note that SDXL (Podell et al., 2023) uses two text encoders with 3x larger UNet of more cross-attention layers, leading to superior text-image alignment. In spite of this, we still obtain an on- CLIP score and surpass all the other baselines that have similar text encoder parameters. We also show the FID-CLID and FID CLIP curves over multiple CFG scales in Fig. 3, where our model balances well between image quality and text-alignment, especially for the commonly-used CFG scales (bottom right). Please Sec. A. for more quantitative results.
8

---

Published as a conference paper at ICLR 2024
Table 2:
Ablation Results. We explore design choices for simultaneous denoising largets, num- ber of expert branch layers, and noise schedules. The image quality and alignment are evaluated.
215
Plad
A.3
Ablation Settings
FID
FiDeu
C |
C7
Denciss ROB
21
10:27
13
Denokse RGB Depth
19.89
9.30
544.2
030
1.0
0218
6.35
1.00
3225
038
1300
E200
0.355
02/24
C22
Dencise -Nomal
Dowefllocks Uptllocss
508.3
Figure 3: Evaluation Curves COCO-Va Val Human. We show FID-CLIP (leff) and FIDeun-CLLL ELIP (right) curves with CFG scale ranging from 4.0 to 20.0 for all methods.
DevaBlocks Updlleds
17.94
677,4
Defanii SNR with pred
17.70
8:41
367.0
1902
Differen: Timessepe
29.36
18.29
854.8
176.1
HyperHumma (Dars)
17.18
7.52
5921
1214
Table 3: User Preference Comparisons. We report the ratio of users prefer our model to baselines.
Methods SD 2.1
SDXL.
IF
ControlNet
T21-Adapter
HumanSI)
HyperHuman 89.24%
60.45%
82.45%
92.33%
98.06%
99.08%
Qualitative Analysis. Fig. shows results (top) and comparisons with baselines (bottom). We can generate both photo-realistic images and stylistic rendering, showing better realism, quality, diversity, and controllability. A comprehensive user study further conducted as shown in Tab. 3, where the users prefer HyperHuman to the general and controllable T2I models. Please refer to Appendix A.4, A.15, and A,16 for more user study details, comparisons, and qualitative results.
5.2 ABLATION STUDY
In this section, we present the key ablation studies. Except for the image quality metrics, we also use the depth/normal prediction error as a proxy for spatial alignment between the synthesized RGB and structural maps. Specifically, we extract the depth and surface-normal by off-the-shelf estimator as pseudo ground truth. The L and CT denote the Ly-error of depth and normal, respectively.
Simultaneous Denoise with Expert Branch. We explore whether latent structural diffusion model helps, and how many layers to replicate in the structural expert branches: 1) Denoise RGB, that only learns to denoise an image. 2) Denoise RGB + Depth, which also predicts depth. 3) Denoise RGB Normal, which also predicts surface-normal map. 4) Half DownBlock & UpBlock. We replicate half of the first DownBlock and the last UpBlock, which contains one down/up-sample ResBlock and one AttnBlock. 5) Two DownBlocks & UpBlocks, where we copy the first two DownBlocks and the last two UpBlocks. The results are shown in Tab. 2 (top), which prove that the joint learning of image appearance, spatial relationship, and geometry is beneficial. We also find that while fewer replicate layers give more spatially aligned results, the per-branch parameters are insufficient to capture distributions of each modality. In contrast, excessive replicate layers lead to less feature fusion across different targets, which fails to complement to each other branches.
Noise Schedules. The ablation is conducted on two settings: 1) Default SNR with e-pred, where use the original noise sampler schedules with e-prediction. 2) Different Timesteps t. We sample different noise levels (fx, ta. and fn) for each modality. We can see from Tab. (bottom) that zero- terminal SNR is important for learning of monotonous structural maps. Besides, different timesteps harm the performance with more sparse perturbation sampling and harder information sharing.
DISCUSSION
Conclusion. In this paper, we propose a novel framework HyperHuman to generate in-the-wn the-wild hu- man images of high quality. To enforce the joint learning of image appearance, spatial relationship, and geometry in unified network, we propose Latent Structural Diffusion Model that simultane- ously denoises the depth and normal along with RGB. Then we devise Seructure-Guidee Refiner to compose the predicted conditions for detailed generation. Extensive experiments demonstrate that our framework yields superior performance, generating realistic humans under diverse scenarios.
Limitation and Future Work. As an early attempt in human generation foundation model, our approach creates controllable human of high realism. However, due to the limited performance of existing pose/ddeth/normal estimators for in thld humans, find it sometimes fails generate subtle details like finger and eyes. Besides, the current pipeline still requires body skeleton as input, where deep priors like LLMS can be explored to achieve text-to-pose generation in future work.

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENT
This study supported by the Ministry of Education, Singapore, under its MOE AcRF Tier (MOE- T2EP20221- 0012) and NTU NAP.
REFERENCES
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01322 2022.
Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion at scale. arXiv preprint (rXXv:23333.06555 2023. 6
Mikolaj Binkowski, Danica Sutherland, Michael Arbel, and Arthur Gretton, Demystifying mmd gans. arXiv preprint arXiv: /801,004011,1101 2018. 15
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. 7oyo-700m: Image-text pair dataset. https::/gitthnn httpchh...oommcccoooommmc.oeeeee wwt.//coo/aaakaaaooae kttpkkkkaabbi///k/ coyo-dataset, 2022. 7, 34
MMPose Contributors. Openmmlab pose estimation toolbox and benchmark.
https://githu.
conlopen-nmilabnmmpon 2020. 7,34
DeepFloyd. Deepfloyd if. deep-floyd/IE 8,
Github Repository, 2023.
URL https:://gtthiti com/
Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12882-12891, 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8799 8794,, 2021. 3,
Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. Omnidata: A scalable pipeline for making multi-task mid-level vision datasets from scans. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10786-10796, 2021. 7, 34
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVE conference on computer vision and pattern recogni- fion, pp.
2021. 4
Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, and Wenyu Liu. You only look at one sequence: Rethinking transformer in vision through object detection. CoRR, abs/2106.00666, 2021. URL https::/t//// httpss://arxiiii wwt./oo///////aa//ooo 00666. 7,34
Jannlin Fu, Shikai Li, Yuming Jiang, Kwan- Yee Lin, Chen Qian, Chen Change Loy, Wayne Wu, and Ziwei Liu. Stylegan-human: A data-centric odyssey of human generation. In European Conference on Computer Vision, pp. 1-19. Springer, 2022. 4,
Ke Gong, Xiaodan Liang, Dongyu Zhang, Xiaohui Shen, and Liang Lin. Look into person: Self- supervised stusttue-sensitive learning and new benchmark for human parsing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 932-940, 2017. 4
Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Tiong. Boyang Li, Dacheng Tao, and Steven HOL. From images to textual prompts: Zero-shot VQA with frozen large language models, 2023. URL 34
https: opw.reviewwwwweeeeeeeeeeeeeeewee net forum?
Xintong Han, Zuxuan Wu, Zhe Wu, Ruichi Yu, and Larry S Davis. Viton: An image- based virtual try-on network. In Proceedings of the IEEE conference on computer vision and pattern recogni- fion, pp. 7543-7552, 2018.
10

---

Published as a conference paper at ICLR 2024
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017. 15
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 2020. 1, 3, 4,
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799, PMLR, 2019. 20
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.099 09685, 2021. 20
Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and controllable image synthesis with composable conditions. arXiv preprint arXiv:3302.09778 2023. 6
Yuming Jiang, Shuai Yang, Haonan Qiu, Wayne Wu, Chen Change Loy, and Ziwei Liu. Text2human: Text-driven controllable human image generation. ACM Transactions on Graphics (TOG), -1(1) 2022. 3,
Xuan Ju, Ailing Zeng, Jianan Wang, Qiang Xu, and Lei Zhang. Human-art: A versatile human- centric dataset bridging natural and artificial scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 618-629, 2023a. 4, 19
Xuan Ju, Ailing Zeng, Chenchen Zhao, Jianan Wang, Lei Zhang, and Qiang Xu. Humansd: native skeleton-guide diffusion model for human image generation. arXiv preprint arXiv:2304.04222 2023b. 2, 3, 5, 7, 8, 15, 16, 22, 24
Bo-Kyeong Kim, Hyoung-Kyy Kyu Song, Thibault Castells, and Shinkook Choi. On architectural com- pression of text-to-image diffusion models. arXiv preprint arXiv:2305. /5798, 2023. 20
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL http: waaxxiv.orgee abs 14 1412. 6980.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. arXiv preprint NrXiv:2305.01566 2023. 15
Yanyu Li, Huan Wang, Qing Jin, Ju Hu, Pavlo Chemerys, Yun Fu, Yanzhi Wang, Sergey Tulyakov, and Jian Ren. Snapfusion: Text-to-image diffusion model on mobile devices within two seconds. arXiv preprint arXiv:2306.0090 2023. 3
Shanchuan Lin, Bingchen Liu, Jiashi Li, and Xiao Yang. Common diffusion noise schedules and sample steps are flawed. arXiv preprint arXiv:2305.0888 2023. 5, 6, 21
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014. 3,
Wen Liu, Zhixin Piao, Jie Min, Wenhan Luo, Lin Ma, and Shenghua Gao. Liquid warping gan: A unified framework for human motion imitation, appearance transfer and novel view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5904-5913, 2019.
Xian Liu, Qianyi Wu, Hang Zhou, Yuanqi Du, Wayne Wu, Dahua Lin, and Ziwei Liu. Audio-driven co-speech gesture video generation. Advances in Neural Information Processing Systems, 35: 21 1.386-21399, 2022a.
11

---

Published as a conference paper at ICLR 2024
Xian Liu, Qianyi Wu, Hang Zhou, Yinghao Xu, Rui Qian, Xinyi Lin, Xiaowei Zhou, Wayne Wu, Bo Dai, and Bolei Zhou. Learning hierarchical cross modal association for co-speech gesture generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 10462-10472, 2022b. 7
Xian Liu, Yinghao Xu, Qianyi Wu, Hang Zhou, Wayne Wu, and Bolei Zhou. Semantic-ave implicit neural audio-driven video portrait generation. In European Conference on Computer Vision, pp. 106-125. Springer, 2022c. 7
Xian Liu, Xiaohang Zhan, Jiaxiang Tang, Ying Shan, Gang Zeng, Dahua Lin, Xihui Liu, and Ziwei Liu. Humangaussian: Text-driven 3d human generation with gaussian splatting. arXiv preprint arXiv:2311.1066. 2023." 7
Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1096-1104, 2016. 4,
Chong Mou, Xintao Wang, Lianghin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. 12i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXiv:2302.08455: 2023. 1, 3, 6, 7, 8, 16, 22, 24
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112 2021. 3. 15
Gaurav Parmar, Richard Zhang, and Jun-Yan Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In CVPR, 2022. 34
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. XrXiv preprint arXiv:2307.01992 2023. 5, 7, 8, 18, 24, 34
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.
PMLR, 2021. 15, 18, 34
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 22Ki:22204006125, 1(2):3, 2022. 1, 3
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 44(3), 2022. 7, 34
Yurui Ren, Xiaoming Yu, Junming Chen, Thomas H Li, and Ge Li. Deep image spatial transfor- mation for person image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7690-7699, 2020. 1,3
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Patlern Recognition (CVPR), pp. 10684-10695, June 2022. 1,3, 4, 7, 8, 15, 18, 24, 34
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Computer-Assiste Intervention- MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceed- ings, Part III 18, pp. 234-241. Springer, 2015. 4
Prasun Roy, Subhankar Ghosh, Saumik Bhattacharya, Umapada Pal, and Michael Blumenstein. Tips: Text-induced pose synthesis. In European Conference on Computer Vision, pp. 161-178. Springer, 2022.
12

---

Published as a conference paper at ICLR 2024
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- sion Processing Systems, 35:36479-36494 2022. 3, 15
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. arXiv preprint Sttiv:2202.005111 2022. 6, 8
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-2522 2022. 7,34
Aliaksandr Siarohin, Stéphane Lathuilière, Enver Sangineto, and Nicu Sebe. Appearance and pose- conditioned human image generation using deformable gans. IEEE transactions on pattern anal- ysis and machine intelligence, (144):156-1111 117-1171, 2019. 3,3
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei (eds.), Pro- ceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 2256 2265, Lille, France, 07-09 Jul 2015. PMLR. URL html. 4
wwm. wal..press//i /v37 /swhh-
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2020b.
Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https:
//github. www.coo////oo//o wwt.ccow.o.ccccccc
ht//dfffuueeree fuuggingfac///.fuusssrs. 2022. 34
Bochao Wang, Huabin Zheng, Xiaodan Liang, Yimin Chen, Liang Lin, and Meng Yang. Toward characteristic- preseevvng mmage-based virtual try-on network. In Proceedings of the European conference on computer vision (ECCV), pp. 589-604, 2018.
Jiepeng Wang, Peng Wang, Xiaoxiao Long, Christian Theobalt, Taku Komura, Lingjie Liu, and Wenping Wang. Neuris: Neural reconstruction of indoor scenes using normal priors. In European Conference on Computer Vision, pp. 139-155. Springer, 2022. 5
Qianyi Wu, Xian Liu, Yuedong Chen, Kejie Li, Chuanxia Zheng, Jianfei Cai, and Jianmin Zheng- Object-compositionaal neural implicit surfaces. In European Conference on Computer Vision, pp. 197-213. Springer, 2022.1 7
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences text-to- image synthesis. arXiv preprint arXiv:2306. 09341, 2023. 15
Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao. Vitpose: Simple vision transformer base- lines for human pose estimation. Advances in Neural Information Processing Systems, 35:38571- 38584, 2022. 7, 34
Lingbo Yang. Pan Wang. Chang Liu, Zhanning Gao, Peiran Ren, Xinfeng Zhang, Shanshe Wang, Siwei Ma, Xiansheng Hua, and Wen Gao. Towards fine-grained human pose transfer with detail replenishing network. IEEE Transactions on Image Processing, 30:2422-2435, 2021.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint arXiv:2206 10789, 2(3):5, 2022a.
13

---

Published as a conference paper at ICLR 2024
Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Ex- ploring monocular geometric cues for neural implicit surface reconstruction. Advances in neural information processing systems, 35:25018 2--20288-250,, 2022b. 5
Zhengming Yu, Wei Cheng, Xian Liu, Wayne Wu, and Kwan-Ye Lin. Monohuman: Animatable human neural field from monocular video. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. (6943-16953) 2023.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. arXiv preprint arXi::2302005543 2023, 3, 5, 6, 7, 8, 16, 22, 24
Pengze Zhang, Lingxiao Yang, Hian-Huang Lai, and Xiaohua Xie. Exploring dual-task correla- tion for pose guided person image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 7713-7722, 2022.
Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Tian. Scalable person re-identification: A benchmark. In Proceedings of the IEEE international conference on computer vision, pp. 11 1116-10 1124, 2015. 4
Lingting Zhu, Xian Liu, Xuanyu Liu, Rui Qian, Ziwei Liu, and Lequan Yu. Taming diffusion models for audio-driven co-speech gesture generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10544-10553, 2023.
Shizhan Zhu, Raquel Urtasun, Sanja Fidler, Dahua Lin, and Chen Change Loy. Be your own prada: Fashion synthesis with structural coherence. In Proceedings of the IEEE international conference on computer vision, pp. 1680-1688, 2017. 3
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
In this supplemental document, provide more details of the following contents: 1) Additional quantitative results (Sec. A.1). 2) More implementation details like network architecture, hyper- parameters, and training setups, (Sec. A.2). 3) More ablation study results (See. A.3). 4) More user study details (Sec. A.4). 5) The impact of random seed to our model to show the robustness of our method (Sec. A. 6) Boarder impact and the ethical consideration of this work (Sec. A.6). 7) Model's 's robustness on the unseen and challenging pose (Sec. A.7). 8) Potential optimization for the annotation and training pipeline (Sec. A.8). textb19) Model's performance on unconditional generation without input poses (Sec. A.9). 10) Model's performance on the jittered poses and image animation results (Sec. A.I textbf11)
More first-stage generation results (Sec. A.11). 12) The detailed intuition of updated noise schedule (Sec. A.12). 13) More details pose processing and encoding (Sec. A.13). 14) Reconstruction performance RGB VAE on other modality-specific in- puts (Sec. 14). 15) More visual comparison results with recent T2I models (Sec. A.15). 16) More qualitative results of our model (Sec. A.16). 17) The asset licenses we use in this work (Sec. A.17).
A.1 ADDITIONAL QUANTITATIVE RESULTS
FID-CLIP Curves. Due to the page limit, only show tiny-size FID-CLIP and FIDcur-CLIIP CLIP curves in the main paper and omit the curves of HumanSD (Ju et al., 2023b) due to its too large FID and FIDeuP results for reasonable axis scale. Here, we show clearer version of FID-CLIP and
curves in Fig. 4. As broadly proven in recent text-to-imape studies (Rombach al., 2022; Nichol et al., 2021; Saharia al., 2022), the classifier free guidance (CFG) plays an important role in trading-off image quality and diversity, where the CFG scales around 7.0 - 8.0 (corresponding to the bottom-right part of the curve) are the commonly-used choices in practice. We can from Fig. 4 that our model achieve competitive CLIP Score while maintaining superior image quality results, showing the efficacy of our proposed HyperHuman framework.
Human Preference- Related Metrics. As shown in recent text-to-image generation evaluation stud- ies, conventional image quality metrics like FID (Heusel et al., 2017), KID (Binkkowski et al., 2018) and text-image alignment CLIP Score (Radford et al., 2021) diverge lot from the human pref- erence (Kirstain et al., 2023). To this end, we adopt two very recent human preference-relatee metrics: 1) PickScore (Kirstain al., 2023), which is trained on the side-by-side comparisons of two T21 models. 2) HPS (Human Preference Score) V2 (Wu et al., 2023), which takes the user like/dislike statistics for scoring model training. The evaluation results are reported in Tab. which show that our framework performs better than the baselines. Although the improvement seems to be marginal, we find current human preference-relanee related metrics be highly biased: The scoring models are mostly trained on the synthetic data with highest resolution of 1024 x 1024, which makes them favor unrealistic images of 1024 resolution, as they rarely see real images of higher resolution in score model training. In spite of this, we still achieve superior quantitative and qualitative results on these two metries and a comprehensive user study, outperforming all the baseline methods.
55
22.5
50
SD v1.5
20.0
SD v1.5
v2.0
v2.0
40
SD) v2.1
FIDcur
SDXL
SDXL
35
15.0
DespFloyd-IF
Deep
30
ControlNet
12.5
ControlNet
T21-Adapter
Adl-teapter
25
HummnSD
HunansD
20
Ours
Ours
7.5
0.29
0.30
0.31
0.32
0.33
0.29
0.30
0.31
0.32
0.33
score
CLIP score
Figure 4: Clear Evaluation Curves on MS-COC02014 Validation Human. We show FID-CLIP (left) and FID CLIP (right) curves with CFG scale ranging from 4.0 to 20.0 for all methods.
15

---

Published as a conference paper at ICLR 2024
Table 4: Quantitative Results on Human Preference-Relatee Metrics. We report on two recent metrics PickScore and HPS V2. The first row denotes the ratio of preferring ours to others, where larger than 50% means the superior one. The second row the human preference score, where the higher the better. It can be seen that our proposed HyperHuman achieves the best performance.
Methods
Ours
SD 2.1
SDXL
IF
ControlNet
Adapter
HumanSD
PickScore
66.87%
52.11%
63.37%
74.47%
83.25%
87.18%
HPS V2
0.2905
0.2772
0.2832
0.2849
0.2783
0.2732
0.2656
Pose Accuracy Results on Different CFG Scales. We additionally report the pose accuracy results over different CFG scales. Specifically, we evaluate the conditional human generation methods of ControlNet (Zhang & Agrawala, 2023), T21-Adapter (Mou et al., 2023), HumanSS (Ju et al., 20236), and ours four metrics Average Precision (AP), Average Recall (AR), clean AP (APetem)) and clean AR aeR) as mentioned Sec. 5.1. We report on CFG scales ranging from 4.0 to 13.0 in Tab. 5, where our method constantly better in terms of pose accuracy and controllability.
A.2 MORE IMPLEMENTATION DETAILS
We report implementation details like training Syper-parameters, and model architecture in Tab. 6.
A.3 MORE ABLATION STUDY RESULTS
We implement additional ablation study experiments the second stage Structure- Guided Refiner. Note that due to the training resource limit and the resolution discrepancy between MS- -OCC real images (512 512) and high-quality renderings (1024 x 1024), we conduct several toy ablation experiments in the lightweight 512 x 512 variant of our model: 1) w/o random dropout, where the all the input conditions are not dropout or masked out during the conditional training stage. 2) Only Text, where not any structural prediction is input to the model and we only use the text prompt as condition. 3) Condition on P. where we only use human pose skeleton p as input condition to the refiner network. 4) Condition on d that uses depth map d as input condition. 5) Condition on n that uses surface-norma, n as input condition. And their combinations of 6) Condition on p. d: 7) Condition p. n; 8) Condition necessity of using such multi-level hierarchical structural guidance for fine-grained generation. The results are reported in Tab. 7. We can see that the random dropout conditioning scheme is crucial for more robust training with better image quality, especially in the two-stage generation pipeline. Besides, the structural map/guidance contains geometry and spatial relationship information, which are beneficial to image generation of higher quality. Another interesting phenomenon is that only conditioned on surface-normal n is better than conditioned on both the pose skeleton p and depth map d, which aligns with our intuition that nurmace-normal conveys rich structural information that mostly cover coarse-level skeleton and depth map, except for the keypoint location and foreground- background relationship. Overall, we can conclude from ablation results that: 1) Each condition (i.e., pose skeleton, depth map, and surface-normal) is important for higher-quality and more aligned generation, which validates the necessity of our first-stage Latent Structural Diffusion Model to jointly capture them. 2) The random dropout scheme for robust conditioning can essentially bridge the train-test error accumulation in two-stage pipeline, leading to better image results.
d, to verify the impact of each condition and the
A.4 MORE USER STUDY DETAILS
The study involves 25 participants and annotates for a total of 8236 images in the zero-shot MS- COCO 2014 validation human subset. They take 2-3 days to complete all the user study task, with a final review to examine the validity of human preference. Specifically, we conduct bide-b comparisons between our generated results and each baseline model's results. The asking question is "Considering both the image aesthetics and text-image alignment, which image is better? Prompt: <prompt>." ? The labelers are unaware of which image corresponds to which baseline, Le. the place of two compared images are shuffled to achieve fair comparison without bias.
16

---

Published as a conference paper at ICLR 2024
Table 5: Additional Pose Accuracy Results for Different CFG Scales. We evaluate four pose alignment metrics AP, AR, AP APelam and AR for the CFG scales ranging from 4.0 to 13.0.
CFG 4.0
CFG 5.0
Methods
AP
AR
APal
ARatem t
AP
f
AP APaemn f
ARetean f
ControlNet
20.37
29.54
25.98
37.96
20.42
29.94
26.09
38.31
T2I-Adapter
28.18
36.71
35.68
46.77
27.90
36.76
35.31
46.78
HumanSD
26.05
35.89
32.27
44.90
26.51
36.44
32.84
45.48
HyperHuman
30.45
37.87
38.88
48.75
30.57
37.96
39.01
48.84
CFG 6.0
CFG 7.0
Methods
AP 1
AR f
APalean f
ARatem f
AP
AR f
APolen f
ARetemn 1
ControlNet
20.54
30.16
26.09
38.64
20.44
30.29
26.01
38.79
T21-Adapter
27.90
36.77
35.37
46.80
27.66
36.62
35.00
46.55
HumanSD
26.79
36.79
33.10
45.91
26.73
36.84
32.94
45.80
HyperHuman
30.44
37.92
38.91
48.77
30.49
37.90
38.82
48.72
CFG 8.0
CFG 9.0
Methods
AP
AR f
APalan 1
AR t
AP
AR 1
APalun f
ARelean 1
ControlNet
20.54
30.28
26.06
38.74
20.35
30.11
25.80
38.43
T21-Adapter
27.46
36.50
34.80
46.39
27.10
36.32
34.14
46.04
HumanSD
26.76
36.86
32.96
45.88
26.67
36.91
32.74
45.93
HyperHuman
30.23
37.80
38.72
48.59
29.93
37.67
38.30
48.45
CFG 10.0
CFG 11.0
Methods
AP
AR 1
AP
AR
AP
AR
AP
A.Ratan t
ControlNet
20.10
30.08
25.50
38.29
19.81
29.93
25.23
38.23
T21-Adapter
26.89
36.19
33.83
45.83
26.65
36.10
33.51
45.67
HumanSD
26.67
36.86
32.80
46.00
26.53
36.74
32.63
45.85
HyperHuman
29.75
37.60
38.20
48.38
29.58
37.31
37.88
48.07
CFG 12.0
CFG 13.0
Methods
AP
AR f
AR
AP
f
AP deen f
ARetem f
ControlNet
19.57
29.84
25,02
38.15
19.52
29.74
24.93
38.08
T21-Adapter
26.49
35.95
33.39
45.52
26.41
35.90
33.22
45.44
HumanSD
26.46
36.71
32.53
45.82
26.26
36.65
32.39
45.70
HyperHuman
29.40
37.18
37.75
47.90
29.29
37.11
37.64
47.87
17

---

Published as a conference paper at ICLR 2024
Table 6: Training per-parameters and Network Architecture in HyperHluman
Structural Diffassion
Structure Guided Refiner
Activation Function
SiLU
SiLU Time
Additional Finhed Type
of Heads in Additional Emhed
64
64
Additional Time Embed Dimension
256
Attcution Head Dinension
G. 10, 20, 20) 3200, 640, 1280, 1280]
5. 10, 20) 320, 6400 1280)
Block Out Channels
Cross Atten-nntnioonn Dimension
1024
2048
Down Block Types
("CrossAu"" "x3, x3.ResBlock"">
["ResBlock" "xxxxxx____ "io.CCossAnnn
Input Channel
4
#o Input Head
3
Condition Embedder Channels
[16] D6, 256
Transformer Layers Block
1, 1. 2,
(1, 10
Layers per
[2. 2,
Inpul Class Embedding Dimemsion
2816
Sampler Training Step 1
1000
Learning Rate
le -
1e
Weight Decay
0.01
Warmup Steps
0
0
AdamW Betas
(0.9, 0.909)
(0.9. 0.909)
Batch Size
2048
2048
Condition Dropout
15%
50%
Text Encoder
OpenCLIP VITH (Radford al. 2021)
CLIP VT-L & OpenCLIP ViT-bigG (Radford 2021)
Pretrnined Model
SD:2,0-but. (Rombach al., 2022)
3DXL--100-bas (Podell et al., 2023)
Table 7: Additional Ablation Results for Structure-Guided Refiner. Due to the resource limit
and resolution discrepancy, we experiment on 512 x 512 resolution to illustrate our design's 's efficacy.
Ablation Settings
FID
KID x1k 4
FID +
CLIP f
w/o random dropout
25.69
11.84
13.48
31,83
Only Text
23.99
10.42
13.22
32.23
Condition p
20.97
7.51
12.86
31.95
Condition on d
14.97
3.75
9.88
31.74
Condition n
12.67
2.61
7.09
31.59
Condition on p. Condition p. Condition on d, n
14.98
3.78
9.47
31.74
12.65
2.66
6.93
31.63
12.42
2.59
6.89
31.57
Ours w/ Refiner
12.38
2.55
6.76
32.23
We note that all the labelers well-trained for such text-to-image generation comparison tasks, who have passed the examination on a test set and have experience this kind of comparisons for over 50 times. Below, we include the user study rating details for method vs. baseline models. Each labeler can click on four options: a) The left image is better, in this case the corresponding model will get +1 grade. b) The right image is better. e) NSFW, which means the prompt/image contain NSFW contents, in this case both models will get grade. d) Hard Case, where the labelers find it hard to tell which one's image quality better, in this case both models will get +0.55 grade. The detailed comparison statistics are shown in Table 8, where we report the grades of HyperHu- man vs. baseline methods. It can be clearly seen that our proposed framework is superior than all the existing models, with better image quality, realism, aesthetics, and text-image alignment.
Table 8: Detailed Comparison Statistics in User Study. We conduct a comprehensive user study on zero-shot MS-COCO 2014 validation human subset with well-trained participants.
Methods
SD 2.1
SDXL
IF
HyperHuman
7350 vs. 886 4978.5 vs. 3257.5 6787.5 vs. 1444.5
Methods
ControlNet
T2I-Adapter
HumanSI)
HyperHuman
7604 vs. 632
8076 vs. 160
8160 vs. 76
18

---

Published as a conference paper at ICLR 2024
A.5 IMPACT OF RANDOM SEED AND MODEL ROBUSTNESS
To further validate our model's robustness to the impact of random seed, we inference with the same input conditions (i.e., text prompt and pose skeleton) and use different random seeds for generation. The results are shown in Fig. 5, which suggest that our proposed framework is robust to generate high-quality and text-aligned human images over multiple arbitrary random seeds.
A bearded, bald man wears a multiccloree tie.
A smiling man in a skiing outfit holds hís siıss and poles
A handsome man holds glass of white wine and Looks at the camera.
Figure 5: Impact of Random Seed and Model Robustness. We use the same input text prompt and pose skeleton with different random seeds to generate multiple results. The results suggest that our proposed framework is robust to generate high-quality and text-aligned human images.
A.6 BOARDER IMPACT AND ETHICAL CONSIDERATION
Generating realistic humans conditioned on text benefits a wide range of applications. It enriches creative domains such as art, design, and entertainment by enabling the creation of highly realis- tic and emotionally resonant visuals. Besides, it streamlines design processes, reducing time and resources needed for tasks like content production. However, could misused for malicious purposes like deepfake forgery generation. We believe that the proper use of this technique will enhance the machine learning research and digital entertainment. We also advocate all the generated images should be labeled as "synthetic" to avoid negative social impacts.
A.7 MODEL ROBUSTNESS UNSEEN AND CHALLENGING POSE
In this section, show the robustness of HyperHuman to generalize to unseen challenging poses. Specifically, we choose an ccrobatic-related image from the Human-Art dataset (Ju et al., 2023a), which is highly challenging and rare pose unseen from the common human-centrr images.
19

---

Published as a conference paper at ICLR 2024
The results are shown in Fig. 6. In the visualized results, (a) is the ground-truth image from the Human- Art dataset; (b) is the associated pose skeleton, which challenging and unseen; (c), (d), (e), and (f) are four generated images from proposed framework. It can be seen that HyperHuman robust to unseen poses, even for the rare acrohatic case.
Acrabstios of doing känulsttna wooslew floor.
Ground Truth Image
(b) Challenging Puse
(c) Our Resalts 1
(d) Our Results 22
(+) Our Results
( Our Results
Figure 6: Model Robustness on Unseen and Challenging Pose. We show multiple high-quality generation results on the unseen acrobatic pose, which shows the robustness our method.
A.8 POTENTIAL OPTIMIZATION FOR ANNOTATION AND TRAINING
From the perspective of optimizing training: 1) We can change our models into smaller diffusion backbone to save the training and memory cost, e.g., Small and Tiny (Kim et al., 2023), which achieve on-par performance with Stable Diffusion, but lighter and faster in training and in- ference. 2) We can leverage some efficient parameter finetuning techniques like LoRA (Hu et al., 2021) and Adapter (Houlsby al., 2019) to finetune the shared backbone with fewer parameters. 3) We can adopt some common engineering tricks to reduce memory consumption, e-g., gradient checkpointing, gradient accumulation with smaller batch size, deepspeed model parallelism, lower floating point precision like fp16, efficient xformers, etc.
From the perspective of optimizing annotation: 1) Our efficient architecture design (only add lightweight branches) can actually produce reasonable results with smaller dataset scale and fewer training iterations, capturing the joint distribution RGB, depth, and surface-normal. Before the large-scale training, we first verify method effectiveness on a small-scale 1M subset, which is less than 3% of the Human Verse fullset scale. In spite of this, we can still obtain good results with only 8 40GB A 1100 within one day, generating spatially aligned results for each modality. A generation sample shown in Fig. where (a) is the conditioning pose skeleton; (b), (c), and (d) are the si- multaneously denoised depth map, surface-normal map, and RGB images. Note that since this is an early-stage experiment, the pose conditioning and visualization are little bit different from the final version have used. In spite of this, manage achieve simultaneous denoising of multiple modalities with a much smaller dataset scale. 2) The annotation overhead mostly comes from the diffusion-based image outpainting process (Sec. 4), while the cost for depth and normal estimation is
20

---

Published as a conference paper at ICLR 2024
relatively low. Though facilitating more accurate pose annotations, is not mandatory step. More- over, the final evaluation process, we use the raw human pose without the help of outpainting, but can still achieve superior performance.
beautiful women walleingg ow, the street.
Oo Plec Stabiten
Do Deeelied Depet
(3) Denwiled Surtace-Neerral
(4) Denateed RIG
An Early-Stage Generation Sample on Small-Scale Dataset. We show a generation sample on small-scale LM subset, which less than 3% of the Human Verse fullset scale. Note that since this is an early-stage experiment, the pose conditioning and visualization are little bit different from the final version we have used.
Figure 7:
A.9 MODEL PERFORMANCE WITHOUT INPUT POSE
In this section, we show the unconditional generation results our model, where no pose input is taken. The generated images are shown in Fig. 8. All the text prompts are from the zero-shot MS- COCO 2014 Human Validation dataset, which is unseen during the model training process. Thanks to our framework design of robust conditioning scheme, the model is trained predict reasonable denoising results, even when the conditions are dropout or masked. Therefore, we manage to create realistic human images with superior performance even without the pose skeleton input.
A.10 MODEL PERFORMANCE ON JITTERED POSE AND IMAGE ANIMATION
We show additional results on the jittered human poses in Fig. 9. Specifically, we first condition on the original pose skeleton (a) and obtain the generated image (b) based on text prompt "A woman standing near lake with snow capped mountain behind"'. Then gradually add Gaussian noise to all the joints, from the sigma scale of 2.5 to 12.5. It can be seen that HyperHuman could produce pleasant results under Gaussian noises to all joints, creating highly pose-aligned images.
To further verify we can animate a certain image by gradually changing the input pose, we fix the random seed, the initial starting noise Xy, and text prompt. The sequential generation results shown in Fig. 10. Note that we fix the text prompt of "A woman standing near a lake with a snow capped mountain behind".. The input skeleton are shifted towards the right side, each step by 10 pixels. Even though we maintain other conditions fixed, we can still see background and appearance changes. We regard this promising research problem and will explore in future work.
A. MORE TIRST-SSTGE GENERATION RESULTS
We show more first-stage Latent Structural Diffission Model generation results in Fig. 11, where the spatially aligned RGB images, depth maps, and surface-noree maps are simultaneously denoised and generated. Though not high-quality the final output from the second-stage pipeline, it can still generate plausible humans with coherent structures.
A.12 DETAILED INTUITION OF UPDATED NOISE SCHEDULE
First, it is hard to finetune the Stable Diffusion to generate pure-color images. As shown in the paper (Lin et al., 2023), "Solid black background". The main reason is that common diffusion noise schedules are flawed, which corrupts image incompletely when sampling / T at the training phase: XT Xo orre,
overfit to single solid-black image with the text prompt
21

---

Published as a conference paper at ICLR 2024
MuBee sitting Rewn with Drowts.
Two noLdinng Gurfboarols while swiiting at cameera.
Briy brown, jackett standing mear signs kelding oeliphowe kés
soddly bear
WOMAAA poses evecads sanshwaal lockoh et SA outtllorr
elderty wamaa, the she sits ia frowt of chesse piazz lnk N.
plotucre of with tis sinal wildl heir.
Figure 8: Unconditional Generation Results without Input Pose. All the text prompts are from the zero-shot MS-COCO 2014 Human Validation dataset.
but or + 0,0r + 1. Due to this reason, a small amount of signal is still included, which leaks the lowest frequency information such as the overall mean of each channel. In contrast, the inference stage, the sampling starts from a pure Gaussian noise, which has zero mean. Such train-test gap hinders SD from generating pure-color images.
Second, similar to pure color images, the depth and surface-normal maps are visualized based on certain scheme, where its color and patterns are highly constrained. For example, the depth map is grey-scale image without colorful textures, and current estimators tend to infer similar depth values for each local patch. Therefore, the low frequency information per-channel mean and standard deviation could be misused by network as shortcut for denoising, which harms the joint learning of multiple modalities (RGB, depth, and surface-normal). Motivated by this, we propose to enforce zero-terminal SNR (x7 0.0 x0 + 1.0- ee that is, ar 0, Or 1) to fully eliminate wwwffreuenccy information the training stage, so that we manage generate both RGB images and structural maps of high quality at the inference stage.
A.13 MORE DETAILS ON POSE PROCESSING AND ENCODING
The encoder used for pose the pretrained VAE encoder of Stable Diffusion, which the same as the encoder used for RGB, depth, and surface-normal maps. Before pose encoding, we visualize the body keypoints on a black canvas to form a skeleton map, similar to previous controllable meth- ods (Zhang & Agrawala, 2023; Mou et al., 2023: Ju et al.. 2023b) with pose condition. Specifically, we use exactly the same pose drawing method as HumanSD (Ju et al., 2023b) and T2I-Adapter (Mou al., 2023) ensure fairness.
22

---

Published as a conference paper at ICLR 2024
A womdnw Sttnding netr a Lake wite Snow cuppss mnsuniill sthind.
Original
Nalo Pate, 4graa125
vignor25
(p Noio Sgrea-775
vigree1.5
(0 Nolay Tols, Ggra-1iL
1) Generation, igrom118..
(5) Nokey Pas,, Pgraeii
Goneratie, 4gmam122
Figure 9: Generation Results under the Jittered Poses. We use the text prompt "A woman stand- ing near lake with snow capped mountain behind" and gradually add Gaussian noise to all the joints, from the sigma scale of 2.5 to 12.5.
A.14 VAE RECONSTRUCTION PERFORMANCE ON MODALITY-SPECIET INPUT
We use improved auto-encoder of the pretrained Stable Diffusion "sd-vac-ft-mse" as VAE to encode inputs from all the modalities, including RGB, depth, surface-normal, and body skeleton maps. To further validate that RGB VAE can be directly used for other structural maps, exten- sively evaluate the reconstruction metrics of all the involved structural maps on 100k samples. The results are reported in Tab. which show that the pre-trained RGB VAE robust enough handle different modality images, including the structural maps we use in this work. Besides, we addition- ally show some visualized reconstruction samples in Fig. 12, where in each group. the first row the input structural maps, and the second row is the reconstructed structural maps from the pretrained RGB VAE. Therefore, both the quantitative metrics and visual results show that the pretrained RGB VAE is robust enough to faithfully reconstruct structural maps.
Table 9: RGB VAE Reconstruction Performance. We evaluate the reconstruction performance of the pretrained RGB VAE on the depth and surface-normal maps.
Modality
rFID |
PSNR f
SSIM f
PSIM 1
Body Skeleton
0.49
39.24
0.96
0.188
MiDaS Depth
0.19
47.08
0.99
0.004
Surface-Normal
0.24
40.11
0.97
0.010
'nttps://hughe nace...aaac..ooooooo co/ stts//aaaiiiiie Eyyl /add ctt. ft-mse
23

---

Published as a conference paper at ICLR 2024
woman standingg wear Laker with snow cappee Hapurstali beniasd.
Figure 10: Animation Results. We gradually shift skeleton to right side, each step by 10 pixels.
A.15 MORE COMPARISON RESULTS
We additionally compare our proposed HyperHuman with recent open-source general text-to- image models and controllable human generation baselines, including ControlNet (Zhang & Agrawala, 2023), 121-Adapter (Mou et al., 2023), HumanSI) (Ju et al., 2023b), SD v2. (Rombach al., 2022), DeepFloyd-IF (DeepFloydd, 2023), SDXL 1.0 w/ refiner (Podell et al., 2023). Besides, also compare with the concurrently released T2I-Adapter+SDXL We use the officially-relcased models to generate high-resolution images of 1024 x 1024 for all methods. The results are shown in Fig. 13, 14, 15, and 16, which demonstrates that can generate humans of high realism.
A.16 ADDITIONAL QUALITATIVE RESULTS
We further inference on the challenging zero-shot MS-COCO 2014 validation human subset prompts and show additional qualitative results in Fig. 17, 18, and 19. All the images are in high resolution of 1024 x 1024. It can be seen that our proposed HyperHuman framework manages to synthesize realistic human images of various layouts under diverse scenarios, e.g. different age groups baby, child, young people, middle-aged people, and old persons; different contexts of canteen, in-the-wild roads, snowy mountains, and streetview, etc. Please kindly zoom in for the best viewing.
Phttps://hughe htt.aaa.../.aaaaeeeeeeeeeeeeeeee wttpaaeeeecaaatteteeeee/cc/aaa adapter adaptee
24

---

Published as a conference paper at ICLR 2024
strinding baay
ayes
sitting af rendly game.
dressed
wemsa wardag a watding onwe te.ding The sotill younng g girla hrld hanuls 2s they Litto beslroous.
yovinger nav. and an cloer mare delling in the forest:
A goung wide, the midole of cuttiin his hair wield poning.
M
with up shirt, individuall weakille cay expression.
Smilling skiling owtfit keles
young
Figure 11: First-Stage Results. We show the jointly denoised RGB, depth, and normal maps.
25

---

Published as a conference paper at ICLR 2024
(a) GT Pose
Recon. Pose
(c) GT Depth
(d) Recon. Depth
(c) GT Normal
(D Recon, Normal
Figure 12: RGB VAE Reconstruction Results. We show the visualized reconstruction results on depth and surface-normal maps.
26

---

Published as a conference paper at ICLR 2024
A man. kitzbbardingg on the ocean. on top of a wave.
HyperHuman (Ours)
ControlNet
121-Adapter
HumanSD
SD)
(1) DeepFlovd-1F
(g) SDXL
(h) 12F-Adapter
Group of swowboardders in tran.portation vehiole wear ski Area.
(a) HyperHuman (Ours)
(b) ControlNet
(c) 121-Adapter
(d) HumanND
(+) +2,1
(i) DeepPloyd-1F
(e) SDXL
(h) THI-Adapter SDXL
wuR. on vactoraycle that road that has grass fields both sides and stop sigv
Hlyperlluman (Ours)
ControlNet
(c) 121-Adapter
HumanSD
S6P
(0)8D v2.1
(1 DeepFloyd-IF
(g) SDXL.
(h) T2I-AdaptetSSDI.
Figure 13: Additional Comparison Results.
27

---

Published as a conference paper at ICLR 2024
A girl standing by storefrout while talkinng ow ker phone.
Hyper (Ours)
ControlNet
(c) 121-Adapter
(c) Human
(e) SD v2.1
(0 DeepPlyd-11 Small silver cell phone being held up any person's hand.
(g) SDXL
(h) 121-Adapter ISXLL
(a) HyperHuman (Ours)
(b) ControlNet
(c) 121-Adapter
(d) HumamSD
(c)
(0 DeepElyd--IF
(e) SDXL
(h) T24-Adapter/SDDX
A W.Rn holding various kites wext to oabinet.
(4) HyperIluman (Our))
121-Adapter
SD v2.1
(1 DeepHloyd-1F
(0) SDXL.
(b) T2I-Adatte+SSDX.
Figure 14: Additional Comparison Results.
28

---

Published as a conference paper at ICLR 2024
Mastering the art of skaatbbardinng is profoundlly benefloial.
HyperHuman
ControlNet
121-Adapter
HumanSD
(c) v2.1
(1) DecpFloyd-1F
(2) SDXL
(0) T21-Adapter-SSDXI
womaau equestrian outfit riding white horse.
(a) HyperHuman (Ours)
(b) ControlNet
(c) 121-Adapter
(d) HumanSDD
DeepEloyd-IF
T21-Adapter SDXL
A group of men who are standing behind a banner that has various fags on it.
cobrr or
(a) Ilyperkluman (Ours)
(b) ControlNet
(c) 121-Adapter
(d) HumanSD
(+)
(1 DeepFloyd-1F
(p) SDXL
(I) T2---Adppoo-SDNN..
Figure 15: Additional Comparison Results.
29

---

Published as a conference paper at ICLR 2024
A person is typing on a lap top and there is a person up on soreen.
(4) Hyper (Ours)
ControlNet
(c) 121-Adapter
( DeepFloyd-1F A black and white photo of girl's face with teddy bears behiogl her.
v2.1
SDAL
(hi 12I-Adapter SDXL
HyperlHuman (Our)
(b) ControlNet
(6) T21-Adapter
(d) HumanSD
(+) SD v2.1
( DeepFlayd-IF
(g) SDXL
(h) T2D-A
Several People walking in city street, one WOmAn wears head scarf, others have umbrella
(2) HyperHumae (Ours)
(c) 121-Adapter
v2.1
DeepFloyd-1F
SDXL
(h) T21-Adapter SDXL SDXL
Figure 16: Additional Comparison Results,
30

---

Published as a conference paper at ICLR 2024
A baby girl with beautiful blue eyes standing next to brown teddy bear.
A little girl with wavy hair and smile holding teddy bear.
A man and woman seated table in a restaurant.
A cow laying on the grass behind a man holding a oup of coffee.
A young kid stands before a birthday oake decorated with captain America.
A man who is sitting in a bus Looking away from the window.
Figure 17: Additional Qualitative Results on Zero-Shot MS-COCO Validation.
31

---

Published as a conference paper at ICLR 2024
A man in red shirt is holding skate board up over hás head.
Two men who are sitting next to each other with a Large pizza in front of them.
Two children earry an enormous stuffed teddy bear.
The upper helf of a man posing for & photograph wearing suit with blue tie and matching pooket cerver.
An older man is wearing funny hat in hís dining room.
Young wan on top of a snowboard wearing jacket.
Figure 18: Additional Qualitative Results on Zero-Shot MS-COCO Validation.
32

---

Published as a conference paper at ICLR 2024
Man sitting on brick covered ground, appearing dirty and tired.
A man wearing a purple neck tie and glasses while sitting in a car.
A man standing on grassy area next to trees.
A girl with blue hair és taking a self portrait.
A wan wearing a helmet is sitting hés blue motoreyole.
A person dressed up taking a pioture at street with hís fist up.
Figure 19: Additional Qualitative Results on Zero-Shot MS-COCO Validation.
33

---

Published as a conference paper at ICLR 2024
A.17 LICENSES
Image Datasets:
LAION-SB' (Schuhmann et al., 2022): Creative Common CC-BY 4.0 license.
COYO-700M' (Byeon al., 2022): Creative Common CC-BY 4.0 license.
MS-COCO' (Lin al., 2014): Creative Commons Attribution 4.0 License.
Pretrained Models and Off-the-Shat Shelf Annotation Tools:
diffusers (von Platen et al., 2022): Apache 2.0 License.
CLIP (Radford al., 2021): MIT License.
Stable Diffusion (Rombach al., 2022): CreativeML Open RAIL++-M License.
YOLOS-Tiny' (Fang et al., 2021): Apache 2.0 License.
BLIP2 (Guo et al., 2023): MIT License.
MMPose (Contributors, 2020): Apache 2.0 License.
ViTPose 12 (Xu et al., 2022): Apache 2.0 License.
Omnidata 13 (Eftekhar et al., 2021): OMNIDATA STARTER DATASET License.
MiDaS (Ranftl al., 2022): MIT License.
clean-fil S (Parmar et al., 2022): MIT License.
SDy2-inpainting" 16 (Rombach et al., 2022): CreativeML Open RAIL++M License.
SDXL-base-vI.0" (Podell et al., 2023): CreativeML Open RAIL++MM License.
Improved Aesthetic Predictor Apache 2.0 License.
waboonnxxxxb-e1l 1.0
34


Reviews:
[{"review_id": "dQlu7mqOiN", "reviewer": "Reviewer_zFqJ", "summary": "The paper introduces HyperHuman, a novel framework designed to address the challenge of generating hyper-realistic human images from text and pose inputs. It builds upon the foundations of diffusion models, improving upon the limitations of existing models like Stable Diffusion and DALL·E 2 that often fail to generate human images with coherent parts and natural poses. The key innovation is the integration of structural information across different granularities within a unified model to generate realistic images. The authors also curated a new dataset, HumanVerse, featuring a vast number of images with comprehensive annotations.", "questions": "Please see the Weaknesses.", "limitations": "", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The Structure-Guided Refiner and improved noise schedule for eliminating low-frequency information leakage showcase thoughtful technical innovation.\n\n2. The Latent Structural Diffusion Model's ability to simultaneously process and align RGB images, depth, and surface normals could result in more accurate and detailed images.\n\n3. The paper achieves state-of-the-art performance in generating diverse and high-quality human images, which, if validated, marks a significant advancement.\n\n4. The HumanVerse dataset's extensive size and detailed annotations can greatly benefit the generative model by providing a diverse range of training examples. Is valuable for the community.", "weaknesses": "I don't have major concerns, but it would be beneficial to discuss the following questions:\n\n1. Given the model's architecture involves shared modules for different data modalities, could you elaborate on how the framework ensures the distinctiveness of modality-specific features? In particular, is there any mechanism within the model to prevent potential feature homogenization and maintain the integrity of the unique distributions associated with RGB, depth, and normal maps?\n\n2. It would be insightful to understand how the model generalizes to unseen poses and whether there are specific pose complexities that present challenges.\n\n3. The computational requirements for annotation and training are significant, given the use of multiple GPUs. Are there potential optimizations or simplifications that could maintain performance while reducing resource demands?\n\n4. The model can generate photo-realistic and stylistic images. How does the model balance realism with artistic style variations, and could there be a more detailed explanation of how this balance is achieved?\n\n5. Considering the rapid pace of advancement in this field, what are the authors' plans for updating or maintaining the model to keep up with emerging techniques and standards?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YzhguaNfQT", "reviewer": "Reviewer_HmdV", "summary": "The paper constructs a large-scale dataset HumanVersea containing 340M images for better human image generation with stable diffusion. The proposed approach also denoised depth and surface normal in addition to the RGB space and shows further improvements. The paper compares with multiple baselines to demonstrate the superiority of the proposed method.", "questions": "Pease see my questions above. I think in general the paper shows promising results but I am not sure if this comes with additional cost of the diversity.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed approach is simple yet effective. It’s interesting to see how predicting depth and normal improves the human image quality.\n\n2. The paper addresses an important problem where existing methods show various limitations. The paper compares multiple baseline methods and shows promising results. The constructed dataset is a notable contribution.\n\n3. The paper is in general easy to read.", "weaknesses": "1. The curated dataset is filtered with the criteria: only those images containing 1 to 3 human bounding boxes are retained; people should be visible with an area ratio exceeding 15%; plus rule out samples of poor aesthetics (< 4.5) or low resolution (< 200 × 200). I wonder if these rules reduce the diversity of the images that the model could produce, e.g., group/family photo or small faces? This is also observed in some visualizations in the supplementary materials where it does seem like the model improves the visual quality at the cost of diversity. I wonder if the authors could comment more on this? Besides, the CLIP alignment score is also a bit lower than SDXL, is it related to this diversity issue?\n\n2. Since providing accurate poses in real applications is hard, it would be very useful if the model also produces pleasant unconditional results. So I am also curious about the unconditional results. To be specific, how about images synthesized with just the text input without the poses? \n\n3. Related to the previous question, how robust the model is to the input pose? If the input skeleton is jittered for all joints, would the model still produce high-quality images? I am also curious if the sampled noise is fixed and only the input pose is animated, is it possible to animate the image?\n\n4. In Table 2, how about RGB + normal? Since depth and normal are closely related, is it necessary to include both?\n\n5. Regarding the results in Table 1, I am not sure why SDXL is much worse than SD? Could the authors further clarify on this? In addition, it also looks a bit strange to have DeepFloyd-IF 1024 images downsampled. Is it possible to directly compare the 1024 resolution with the proposed method?\n\n6. Since the numerical results are mainly without the second-stage refiner plus the second-stage dataset is internal, it would be helpful to show visual results with only the first stage.\n\n7. I am not sure if this intuition is well explained. Maybe the authors could further clarify “Such monotonous images may leak low-frequency signals like the mean of each channel during training; The zero terminal SNR (αT = 0, σT = 1) is further enforced to eliminate structure map’s low-frequency information.” in subsection “Noise Schedule for Joint Learning”.\n\n8. Will the dataset be released for both stages?\n\n9. Does the input pose also contain the face landmarks or hand poses? Or just the body skeleton? If it's just skeleton, would adding hand poses helps with the generated details?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MhjQwyZ3iu", "reviewer": "Reviewer_1Ma5", "summary": "The paper proposes a unified framework called HyperHuman for generating high-quality human images with diverse poses, appearances, and layouts. The authors curate a new dataset called HumanVerse with rich annotations (human pose, depth, and surface normal) to train the model. The framework consists of two modules: the Latent Structural Diffusion Model and the Structure-Guided Refiner. The former module denoises the depth and surface normal along with the RGB image conditioned on the caption and skeleton, while the latter module generates higher-resolution images based on the predicted depth, surface normal, and provided skeleton. The paper's contributions include introducing a new dataset with rich annotations, proposing a novel framework for human image generation which yields superior performance, and generating realistic humans under diverse scenarios.", "questions": "1. According to Figure 2, the pose skeleton is encoded before being concatenated with noise. What is the encoder used for the pose, and how does the author process the pose input before encoding it?\n2. What about inference requirements (GPU memory, time)?\n3. How do we ensure the input pose and caption are aligned with the predicted depth, map, and surface normal? Is there any evaluation on this?\n4. In the qualitative result, the comparison among other SOTA methods does not include the pose skeleton figure, I suggest adding it for better visualization.\n5. The description of Figure 2 lacks clarity, particularly when the authors refer to the colours \"purple\" and \"blue.\" However, the figure itself makes it confusing to recognize which part they mention. I suggest the authors should redraw it.  \n6. I suggest adding the ablation study between joint denoising the targets and individually denoising the targets to show the effectiveness of joint denoising. This is the main technical contribution beside the new dataset. \n7. In Table 7: Additional Ablation Results for Structure-Guided Refiner, no ablation study on conditioning on predicted low-resolution image x (from the first module).", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes a large-scale curated human-centric dataset with comprehensive annotations (human pose, depth map, surface normal) that may benefit a lot in future research in the field of human image generation.\n2. The author introduces a new approach for incorporating the body skeletons, depth and surface normals by jointly denoising depth, surface normal and RGB images and a robust conditioning scheme with text prompt and human pose. The paper also conducts ablation study to prove the effectiveness of this.\n3. The proposed method is extensively evaluated on the human-centric dataset MS-COCO 2014 Validation Human to many other SOTA approaches and outperforms on almost quantitative metrics.\n4. The paper also extensively experiments and does an ablation study on the effectiveness of the Structure-Guided Refiner with and without many conditions, noise scheduler and expert branch denoising mechanism.", "weaknesses": "1. The label of the HumanVerse dataset has been created based on other pre-trained models, such as depth and pose estimation models. However, the quality of the dataset does not guarantee accurate labelling (especially in crowded scene or extreme lighting condition), and its reusability for other tasks remains uncertain.\n2. The approach has a noteworthy computational cost. However, when evaluating the results in comparison to the second approach, it becomes evident that the gains in Pose Accuracy are not substantial. (Tab. 1)\n3. The paper's objective is to address incoherent parts and unnatural poses. However, it falls short in terms of providing quantitative metrics to evaluate the effectiveness of the proposed method in addressing these issues.\n4. This paper lacks a fair comparison on two fronts. Firstly, it compares itself to other non-pose conditional guided methods (such as SD and DeepFloy-IF). Secondly, the method discussed in the paper was trained on the HumanVerse dataset, which exclusively contains the rich human class, while other methods are trained on a broader set of classes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BY4xGHXp94", "reviewer": "Reviewer_jWKR", "summary": "This work proposes a framework aiming to better preserve and enhance human generation using T2I models. It has multiple contributions, it first collects a large-scale human-century dataset with structure information like depth/skeleton and surface-normal. It also proposes a latent structure module to jointly model images and other structure output, further, it proposes a structurally guided Refiner to better compose the predicted conditions and improve image quality. Experiments show HyperHuman can achieve SoTA performance in diverse scenarios.", "questions": "– It is not clear to me why to use independent noise for modeling the different modalities, have you tried modeling using a single noise?\n– Depth, normal etc have different distributions to RGB images, how do you ensure the VAE can capture its semantic in the latent space?\n– It is not clear to me how you use the output of the first stage model which is in lower resolution, and input into the Refiner model", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "– This paper is well written and has clear motivation – trying to improve the control consistency in human generation, which is missing from the current control pipeline work like controlnet and T2I-adapter\n– The proposed expert branches are interesting and well-study, i.e. how to balance feature sharing and model capacity for each modality,", "weaknesses": "– A study on modality-specific reconstruction using RGB VAE is missing, I think the author needs to study if RGB VAE is suitable to encode structure information or if further finetuning is needed.\n-- An important baseline that finetunes on HumanVerse images without structure outputs using the SoTA network architecture is missing. It is not clear to me how much improvement the structure prediction brings to the performance. i.e. do we really need to predict the structure information or we can just fine-tune the general-purpose T2I model to the human images dataset?\n-- The curated HumanVerse dataset is claimed to be one of the contributions, but the author did not mention if they will open-source the dataset. I hope the authors can clarify this as it can hurt the contribution of this paper if the dataset is kept internal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)