Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models
Authors: Fei Shen
Hu Ye
Jun Zhang
Cong Wang
Xiao Han
Yang Wei
Venue: ICLR 2024 poster (2024)
Keywords: Diffusion Model;  Pose-Guided Image Synthesis
Primary area: generative models

Abstract: Recent work has showcased the significant potential of diffusion models in pose-guided person image synthesis.
However, owing to the inconsistency in pose between the source and target images, synthesizing an image with a distinct pose, relying exclusively on the source image and target pose information, remains a formidable challenge.
This paper presents Progressive Conditional Diffusion Models (PCDMs) that incrementally bridge the gap between person images under the target and source poses through three stages.
Specifically, in the first stage, we design a simple prior conditional diffusion model that predicts the global features of the target image by mining the global alignment relationship between pose coordinates and image appearance.
Then, the second stage establishes a dense correspondence between the source and target images using the global features from the previous stage, and an inpainting conditional diffusion model is proposed to further align and enhance the contextual features, generating a coarse-grained person image.
In the third stage, we propose a refining conditional diffusion model to utilize the coarsely generated image from the previous stage as a condition, achieving texture restoration and enhancing fine-detail consistency.
The three-stage PCDMs work progressively to generate the final high-quality and high-fidelity synthesized image.
Both qualitative and quantitative results demonstrate the consistency and photorealism of our proposed PCDMs under challenging scenarios.
The code and model will be available at https://github.com/tencent-ailab/PCDMs.

Published as a conference paper at ICLR 2024
ADVANCING POSE-GUIDED IMAGE SYNTHESIS WITH PROGRESSIVE CONDITIONAL DIFFUSION MODELS
Fei Shen; Hu Ye', Jun Zhang!, Cong Wang, Xiao Han, Wei Yang
Tencent Al Lab (ffei shen, huye, juneizhang, xvencewang, haroldhan, wil willyang)@tencent..oo stencent.co
ABSTRACT
Recent work has showcased the significant potential of diffusion models in pose- guided person image synthesis. However, owing to the inconsistency in pose be- tween the source and target images, synthesizing an image with a distinct pose, relying exclusively on the source image and target pose information, remains a formidable challenge. This paper presents Progressive Conditional Diffusion Models (PCDMs) that incrementally bridge the gap between person images under the target and source poses through three stages. Specifically, in the first stage, we design a simple prior conditional diffusion model that predicts the global features of the target image by mining the global alignment relationship between pose co- ordinates and image appearance. Then, the second stage establishes dense corre- spondence between the source and target images using the global features from the previous stage, and an inpainting conditional diffusion model is proposed to fur- ther align and enhance the contextual features, generating a coarse-grainet person image. In the third stage, we propose a refining conditional diffusion model uti- lize the coarsely generated image from the previous stage as condition, achieving texture restoration and enhancing fine-detail consistency. The stage PCDMs work progressively to generate the final high-quality and high-fidelity synthesized image. Both qualitative and quantitative results demonstrate the consistency and photorealism of our proposed PCDMs under challenging scenarios. The code and model will be available at https://gituub.com//eaaa aaiaaaaaa//
INTRODUCTION
Given image of specific person under a particular pose, pose-guided image synthesis (Zhang et al., 2022; Ren et al., 2022; Bhunia et al., 2023) aims to generate images of the person with the same appearance and meanwhile under the given target pose, which more importantly, are expected to be as photorealistic as possible. It holds broad and robust application potential in e-commerce and content generation. Meanwhile, the generated images can be used improve the performance of downstream tasks, such as person re-identification (Ye et al., 2021; Shen et al., 2023b). However, since pose disparities between the source and target images, generating an image with a different pose solely based on the source image and target pose information remains significant challenge.
Previous work usually focuses on the generative adversarial network (GAN) (Creswell et al., 2018), variational autoencoder (VAE) (Kingma et al., 2019), and flow-based model (Li et al., 2019). GAN- base methods (Zhu et al., 2019; Tang et al., 2020) insert multiple repeating modules to mine the sparse correspondence between source and target pose image features. The outputs produced by these approaches often exhibit distorted textures, unrealistic body shapes, and localized blurriness, particularly when generating images of occluded body parts. In addition, owing the nature of the adversarial min-max objective, GAN-based methods are susceptible to unstable training dynamics, limiting the diversity of the generated samples. Although VAE-based approaches (Siarohin et al., 2018; Esser et al., 2018) are relatively stable, they suffer from blurring of details and misalignment of target pose due to their reliance on surrogate loss for optimization. Flow-based methods (Li et al., 2019; Ren et al., 2021) have emerged to deal with this problem, which guide the source
"Equal contribution.
1 "ooresponniig author

---

Published as a conference paper at ICLR 2024
image features to distort to reasonable target pose by predicting the correspondence between the source and target pose. However, when the source and target poses undergo large deformations or occlusions, this can easily lead to apparent artifacts in the generated images. Likewise, some methods (I.v et al., 2021; Zhang et al., 2021) utilize human parsing maps to learn the correspondence between image semantics and poses to ensure that the generated images are consistent with the target pose. Although these methods can generate images that meet pose consistency requirements, they
still struggle maintain consistent style and capture realistic texture details.
Recently, diffusion models (Bhunia et al., 2023; Zhang & Zhou, 2023) have made significant strides in the field of person image synthe- sis. They utilize the source image and target pose as conditions and generate the target im- age through a multi-step denoising process in- stead of completing it in a single step. So, these approaches help better retain the input informa- tion. However, shown in Figure (a), due to the pose inconsistency between the source and target images, this essentially constitutes an unaligned image-to-image generation task at the conditional level. Moreover, the lack of dense correspondence between the source and target images regarding image, pose, and appearance often results in less realistic results.
Pose
Image
Image
Generate
Targel
[Source]
Target
Unaligned
Existing Methods
Eslimate
Figure 1: Existing methods typically utilize unaligned
imago-to-image generation at the conditional level.
This paper presents Progressive Conditional Diffusion Models (PCDMs) to tackle the aforemen- tioned issues through three stages, as shown in Figure 1 (b). Initially, we propose a prior condi- tional diffusion model to predict global features given target pose. This prediction is significantly simpler than directly generating target images, as we allow the prior model to concentrate solely on one task and thereby do not worry about realistically detailed texture generation. Given a source image and pose coordinates as conditions, the prior conditional diffusion model employs trans- former network to predict global features under the target pose. In the second stage, we use the global features from the previous stage to establish the dense correspondence between the source and target images, and propose an inpainting conditional diffusion model to further align and en- hance contextual features, generating coarse-grain synthetic image. Finally, we develop a refining conditional diffusion model. This model utilizes the coarse-grain grain image generated in the previous stage and applies iae--mmaatt--maaee techniques to restore texture and enhance detail consistency. The three stages of the PCDMs operate progressively to generate visually more appealing outcomes, particularly when handling intricate poses.
We summarize the contributions of this paper as follows: (a) we devise simple prior conditional diffusion model that explicitly generates and complements the embedding of the target image by mining the global alignment relationship between the source image appearance and the target pose coordinates. (b) we propose a novel inpainting conditional diffusion model to explore the dense correspondence between source and target images. (c) we introduce a new refining conditional diffusion model by further using post hoc image-to-image techniques to enhance the quality and fidelity of synthesized images. (d) We conduct comprehensive experiments on two public datasets to showcase the competitive performance of our method, Additionally, we implement user study and downstream task to evaluate the qualitative attributes of the images generated by our method.
RELATED WORK
Person Image Synthesis. The task of person image synthesis has achieved great development during these years, especially with the unprecedented success of deep learning. The earlier methods (Ma et al., 2017; Men al., 2020) treat the synthesis task as conditional image generation, using conditional generative adversarial networks (CGANs) (Mirza & Osindero, 2014) to generate the target image with the source appearance image and target pose as conditions. However, due to the inconsistency between the source and target poses, the effectiveness of directly connecting the source image with the target pose is limited. To overcome this challenge, V'unet (Esser et al., 2018) adopts joint application of VAE and U-Net to decouple the appearance and pose of the character image. Furthermore, Def-GAN (Siarohin et al., 2018) proposes a deformable GAN that decomposes the overall deformation through set of local affine transformations to address the misalignment issues caused by different poses. On the other hand, some works (Liu et al., 2019; Ren et al., 2020) utilize flow-based deformation transform source information, improving pose alignment. For example,
2

---

Published as a conference paper at ICLR 2024
First Stage
Second Storp
Third Staaee
Prior Conditionall
Inpainting Conditionnl Diffussion Model
Refining RedininnConnditioonnn Diffissin Model
Diffusson Mode
Edaa
Esiadde
Prugresaive
Progresiiv
Plrn Cvantrade
Searc
The three-stage pipeline of Progressive Conditional Diffusion Models (PCDMs) progressively oper- ates to generate the final high-quality and high-fidelity synthesized image. Our approach progressively predicts the global features, dense correspondences, and texture restoration of target image, enabling image synthesis.
Figure 2:
GFLA (Ren al., 2020) obtains global flow fields and occlusion masks to warp local patches of the source image to match the desired pose. Similarly, ClothFlow (Han et al., 2019) is model human image generation. Subsequently, methods such as PISE (Zhang et al., 2021), SPGnet (Lv et al., 2021), and CASD (Zhou et al., 2022) leverage parsing maps to generate the final image. CoCosNet (Zhou et al., 2021) extracts dense correspondence between cross-domain images through attention- based operations. However, person image synthesis is essentially a transformation from non non-allgned images to images, and the absence of global appearance features of the target image lead to less realistic results. Besides, there are some multi-stage methods in other generative domains. For example, Grigorev et al. (Grigorev al., 2019) proposed framework based CNNs that first performs pose warping, followed by texture repair. Unselfie (LastName, 2014) introduces a pipeline that first identifies the target's 's neutral pose, repairs body texture, and then perfects and synthesizes the character in the background. While these methods can fit the target pose well, they lose sense of realism when combined with the background or the human body.
designed based on this concept for segmentation-guin
Diffusion Models. Diffusion models (Ho et al., 2020; Song et al., 2020) have recently emerged as a prominent generative method, renowned for synthesizing high-quality images. Following the success in unconditional generation tasks, diffusion models have expanded to conditional generation tasks, demonstrating competitive and superior performance compared to GANs and VAEs. Unlike other generative methods, diffusion models employ a multi-step denoising process instead of gener- ating the target image in single step. which helps to better preserve input information. Moreover, this denoising process can enhance texture details, often producing sharper images than GANs and VAEs. Recent studies (Bhunia et al., 2023; Zhang& & Zhou, 2023) have already explored person im- age synthesis based on diffusion models. MGD (Baldrati et al., 2023) guides the generation process by constraining latent diffusion model with the model's pose, the garment sketch, and textual de- scription of the garment itself. PIDM (Bhunia et al., 2023) introduces a texture diffusion module and disentangled classifier-free free guidance to ensure that the conditional input and the generated output are consistent regarding pose and appearance information. Given the robust generation capabilities of the diffusion model, we devise framework with progressive conditional diffusion models, which consist of three pivotal stages: prior, inpainting, and refining.
3 METHOD
An overview of our Progressive Conditional Diffusion Models (PCDMs) is described in Figure 2, which contains prior conditional diffusion model, inpainting conditional diffusion model, and a refining conditional diffusion model. Our method aims to leverage three-stage diffusion models to incrementally bridge the gap between person images under the target and source poses. The prior conditional diffusion model predicts the global features of the target image by mining the global alignment relationship between pose coordinates and image appearance (Section 3.2). Subsequently, the inpainting conditional diffusion model utilizes the global features from the previous stage to fur- ther enhance contextual features, generating a coarse coarse-grained synthetic image (Section 3.3). Further- more, the refining conditional diffusion model leverages the coarse searse-grained image generated in the prior stage, aiming to accomplish texture refinement and enhance detail consistency (Section 3.4).
3.1 PRELIMINARIES
Diffusion Model. Diffusion models are type of generative models trained to reverse the diffusion process. The diffusion process gradually adds Gaussian noise to the data using fixed Markov chain, while denoising model is trained to generate samples from Gaussian noise. Given an input data
3

---

Published as a conference paper at ICLR 2024
sample zo and an additional condition c, the training objective of diffusion model usually adopts mean square error loss Limple as follows,
Esimpe - c,,,
where and eg represent the actual noise injected the corresponding diffusion timestep and the noise estimated by the diffusion model 6, respectively; 2; = a,To + dyE is the noisy data at t step, and at, de are fixed functions of in the diffusion process. To reduce the computational re- sources, latent diffusion models (LDMss) (Rombach et al., 2022) operate the diffusion and denoising processes on the latent space encoded by pretrained auto-encoder model.
Classifier-Free Guidance. In the context of conditional diffusion models, classifier-free guid- ance (Ho Salimans, 2022) is a technique commonly used to balance image fidelity and sample diversity. During the training phase, conditional and unconditional diffusion models are jointly trained by randomly dropping c. In the sampling phase, the noise predicted by the conditional model Eo (z,, t) and the unconditional model eg (x, t) according to Eq. 2, as follows,
Éø (at, c, t) = (x) c, t) + (1 W)Ea( (1t, t),
(2)
where to is the guidance scale used to control the strength of condition c.
3.2 PRIOR CONDITIONAL DIFFUSION MODEL
Tagget
Saurco
Tiegt
In the first stage, we propose simple prior conditional diffusion model, designed to predict the global embedding of the target image. Here, we choose the image embed- ding extracted from CLIP (Radford al., 2021) image en- coder as the global embedding of the target image. CLIP is trained via contrastive learning on large-scale image-text paired dataset. Hence, the image embedding can capture rich image content and style information, which can be used to guide subsequent target image synthesis.
Tes my"
mz.
Image Encoder
Parc: Nedwark
Threse
Extra Embedding
As depicted in Figure 3, the prior conditional diffusion model is transformer network, conditioned on the pose of the source image, the pose of the target image, and the source image. We first adopt OpenPose (Cao et al., 2017) to acquire the pose coordinates for the pose of source and target images. A compact trainable pose network, com- posed of linear layers, is used to project the pose coor- dinates into the pose embedding. For the source image, we also use a CLIP image encoder to extract the image embedding and add a linear layer to project the image em- bedding. Additionally, we add an extra embedding to pre- dict the unnoised global embedding above embeddings plus timestep embedding and noisy im- age embedding
Transformer
Targel Global Embedding Predict
Head
Figure Illustration of the prior condi- tional diffusson model. The prior condi- tional diffusion model uses pose coordi- nates and global alignment relationship of
the image to predict the global features of the target image.
target image. The
target image are concatenated into sequence of embeddings as the input of the transformer network.
Following unCLIP (Ramesh al., 2022), the prior diffusion model is trained predict the unnoised image embedding directly rather than the noise added to the image embedding. Given the source and target pose features p, and pt, and the source image global feature x., the training loss p of prior diffusion model Øy is defined as follows,
prior Epo. p...e...........laa....... 11 Ig [z,, [, T., P.. Pt, t),
(3)
Once the model learns the conditional distribution, the inference is performed according to Eq. 4, as follows,
Bo F., Ps. Pt.tt.t) (E,, Ps: t), + (1 w))x
4

---

Published as a conference paper at ICLR 2024
Luzdan
Pric Mali
Edimake
Enleddisp
Fine Tampt
Estrad
Esiinde
Trrge
Searca
Embeddiing
lsag Luaoder
NLF
Prim: Mede
Trandious
Bedfect Boott
I Concatemation
Image E Emcoder
Figure 4: Overview of the inpainting conditional dif- fusion model. The inpainting conditional diffusion model utilizes the global features obtained from the previous stage to establish dense correspondences.
Figure Illustrative of the refining conditional diffu- sion model. The refining conditional diffusion model leverages the coarse-grained image generated in the previous stage to rectify textures and ensure consis- tency.
3.3 INPAINTING CONDITIONAL DIFFUSION MODEL
With the global features of the target image obtained in the first stage, we propose an inpainting conditional diffusion model establish dense correspondences between the source and target, and transform the unaligned image-to-iman we concatenate the source and target images, the source and target poses, and the source and mask images along the width dimension. To prevent confusion between black and white in the source and target images, we add single-channel marker symbol (omitted the figure) with the same width and height as the input. We use and to represent masked and unmasked pixels, respectively. We then concatenate the global features of the target obtained from the prior conditional diffusion model (prior model) and the local features of the source image. This ensures that the input conditions of model include the entirety the source and target and are aligned at three levels: image, pose, and feature, which is overlooked in existing work.
generation task into an aligned one. As shown in Figure 4,
Specifically, we use a pose encoder with four convolution layers similar to ControlNet (Zhang & Agrawala, 2023) to extract the pose features from the pose skeleton image. Unlike the prior model that uses pose coordinates, we expect this model to maintain image modality alignment throughout the learning phase, especially spatial information. For the source image, encoder and trainable MLP to extract the fine-grained features of the source image, Inspired by (Chen et al., 2023), opt for DINOv2 (Oquab al., 2023) the image encoder because can extract fine details. To better utilize the global features of the target image obtained from the previous stage, we also add the timestep embedding, which is embedded in the ResNet blocks of the entire network. The loss function L inppinting of inpainting conditional diffusion model according Eq. 5, as follows,
frozen image
L, inpainting fut al/e c0 (Iit, fst: Pst. sumn 11
(5)
where fst. Pst, and dem respectively represent the feature embeddings obtained by concatenating the source and target global features, the feature embeddings of source and target poses, and the feature embeddings of source and mask images.
In the inference stage, we also use classifier-fre free guidance according to Eq. 6, as follows,
Ég(Ze, far, Pat, ism; t) weg(xx, far, ism t) + -w) Pat.
(6)
3.4 REFINING CONDITIONAL DIFFUSION MODEL
Following the second stage, we obtain a preliminary generated coarse-grained target image. To further enhance the image quality and detail texture, as shown in Figure we propose refining conditional diffusion model. This model uses the coarse -aasied image generated in the previous stage as a condition to improve the quality and fidelity of the synthesized image. We first concatenate the moarse-graines target image with the noisy image along the channel, which can be easily achieved by modifying the first convolutional layer of the diffusion model based on the UNet architecture. Then, the DINOv2 image encoder and learnable MLP layer to extract features for the source image. Finally, we infuse texture features into the network through a cross attention mechanism to guide the model in texture repair and enhance detail consistency.
5

---

Published as a conference paper at ICLR 2024
Table 1: Quantitative comparison of the proposed PCDMs with several state-of-the-art models.
Datasct
Methods
SSIM (f)
LPIPS (U)
FID (
Def- (Sturohin al., 2018)
0,6786
0.2330
18.457
PAIN (Zhu et al., 2019)
0.6709
0.2562
20.751
ADGAN (Men et al, 2020)
0,6721
0.2283
14,458
PISE (Zhang ot sl., 2021)
0.6629
0.2059
13.610
DcepFashion (Liu all, 2016) (256 176)
GFLA (Ron al, 2020)
2020)
0,7074
0.2341
10.573
(Zhang et al., 2022)
0.7112
0.1931
11.387
CASD (Zhou ot al, 2022)
0.7248
0.1936
11.373
(Ren et al., 2022)
0.7182
0. 0.1752
8.6838
PIDM (Bhunia et al., 2023y
0.7312
0.1678
6.3671
PCDMs w/b Refining
0.7357
0.1426
7.7815
PCDMs (Ours)
0.7444
0.1365
7.4734
DeopFashioo (Liu et al., 2016)
CocosNet2 (Zhou et ol. 2021)
0.7236
0.2265
13.325
(512 352)
NTED (Ren et al., 2022)
0,7376
0.1980
7:7821
PIDM (Bhunia ot al., 2023)
0,7419
0.1768
5.8365
PCDMs w/o Refining
0,7532
0.1588
7.8422
PCDMs (Ours)
0.7601
7.5519
Def-GAN (Siarohin ot al., 2018)
0,2683
0.2994
255. 364
Market-1501 (Zheng et al., 2015) (128 64)
PTN (2hu et al., 2019)
0,2821
0.3196
22.657
GFLA (Ron ct 2020)
0,2883
0:2811
19.75
(Zhang et al., 2022)
0.2854
0.2711
PIDM (Bhunia el al., 2023)
0.30.5
0.2415
14.451
PCDMs w/b Refining
0.3107
PCDMs (Ours)
0.3169
0.22.3
13.897
Assume that the coarse target features ét and source image features a are given, the loss function of refining conditional diffusion model is defined as follows,
L refiningg = m.,.
||| - Eg (2re dot; Vs, t)11?
In the inference phase, we use the following Eq. 8,
Cøl det, fa: - web (2r: dot; Ssi () + (1 w))e(((x)) 1).
(8)
4 EXPERIMENTS
Datasets. We carry out experiments on DeepFashion (Liu et al., 2016), which consists of 52,712 high-resolution images of fashion models, and M5rket-1501 (Zheng et al., 2015) including 32,668 low-resolution images with diverse backgrounds, viewpoints, and lighting conditions. We extract the skeletons using OpenPose (Cao et al., 2017) and follow the dataset splits provided by (Bhunia et al., 2023). Note that the person ID of the training and testing sets do not overlap for both datasets.
Metrics. We conduct comprehensive evaluation of the model, considering both objective and subjective metrics. Objective indicators include structural similarity index measure (SSIM) (Wang al., 2004), learned perceptual image patch similarity (LPIPS) (Zhang et al., 2018), and fréchet inception distance (FID) (Heusel et al., 2017). In contrast, subjective assessments prioritize user- oriented metrics, including the percentage of real images misclassified generated images (R2G) (Ma et al., 2017), the percentage of generated images misclassified as real images (G2R) (Ma et al., 2017), and the percentage of images deemed superior among all models (Jab) (Siarohin al., 2018).
Implementations. We perform our experiments on 8 NVIDIA V100 GPUs. Our configurations can be summarized as follows: (1) the transformer of the prior model has 20 transformer blocks with a width of 2,048. For the inpainting model and refining model, we use the pretrained stable diffusion V2.1 and modify the first convolution layer to adapt additional conditions.
(2) We employ the AdamW optimizer with a fixed learning rate of le-4 in all stages. (3) Following (Ren et al., 2022; Bhunia et al., 2023), we train our models using images of sizes 256 176 and 512 x 352 for DeepFashion dataset. For the Market- 1501 dataset, we utilize images of size 128 x 64. Please refer to B of the Appendix for more detail.
4.1 QUANTITATIVE AND QUALITATIVE RESULTS
We quantitatively compare our proposed PCDMs with several sfate-of-the--a art methods, including Def-GAN (Siarohin et al., 2018), PATN (Zhu et al., 2019), ADGAN (Men et al., 2020), PISE (Zhang et al., 2021), GFLA (Ren et al., 2020). DPTN (Zhang et al., 2022), CASD (Zhou et al., 2022), CocosNet2 (Zhou et al., 2021), NTED (Ren et al., 2022) and PIDM (Bhunia al., 2023).
Quantitative Results. From Table 1, PCDMs excels in two out of three metrics on the DeepFashion compared to other models, regardless of whether it is based GAN, VAE, baseb model, or
httg:///nggniiiiecccaaaaiiiiiiit 2-1-base

---

Published as a conference paper at ICLR 2024
Target Pose
Source Image
GT
ADGAN
PISE
GFLA
OPEN
CASD
NTED
PIDM
Ours
U3
Figure Qualitative comparisons with several state-of-the-ar models the DeepFashion dataset.
diffusion model. For example, when using the flow framework (i.e., CASD), PCDMs outperforms the SOTA flow-based method, CASD, even without explicitly decoupling pose and style. Although PCDMs worse FID score than PIDM (which also employs the diffusion model), we surpass it on the other two metrics, and subsequent experiments further demonstrate our method's superiority.
The comparison results on the Market- 1501 are summarized in Table 1. Notably, PCDMs outshines all SOTA methods, achieving the best SSIM, LPIPS, and FID in all experiments. Specifically, com- pared to methods that consider fine-grained texture features, PCDMs outperforms NTED signifi- cantly in terms of both LPIPS and SSIM. While NTED explores the texture feature of the target in the source image and target pose, PCDMs extract target features by mining the global alignment re- lationship between pose coordinates and image appearance. Furthermore, regarding SSIM. LPIPS, and FID, PCDMs performs better than PIDM, which validated that PCDMs can accurately transfer the texture from the source image the target pose while maintaining high consistency and realism.
7

---

Published as a conference paper at ICLR 2024
Qualitative Results. As shown in Figure 6, comprehensively compare PCDMs and other state-of-tth-aar methods on the DeepPashion dataset. Several observations can be drawn from the results: (1) despite the minuscule size of the necklace in the source image (as seen in the first and second rows), only our proposed PCDMs and the PIDM, which also utilizes a diffusion model, can focus on it. But PCDMs can generate higher-quality images compared the PIDM. (2) In scenarios involving extreme poses and large area occlusions (as seen in the third and fourth rows), only our method can generate images that align reasonably with the target. This can be attributed our method's ability to capture and enhance the global features of the target image. (3) In situations with complex textures and numbers (as seen in the fifth to seventh rows), our method significantly surpasses others in preserving appearance textures, primarily due to our method's capability to refine textures and maintain consistency. (4) The last two rows present source images with invisible logo and target images with visible logo. The results indicate that PCDMs do not overfit, and our results demonstrate better visual consistency than other SOTA methods. To sum up, our method consistently produces more realistic and lifelike person images, demonstrating the advantage of our PCDMs multi-stage progressive generation approach. See C.2 for more examples.
User Study. The above quantitative and qual- itative comparisons underscore the substantial superiority of our proposed PCDMs in gener- ating results. However, tasks of pose-guided person image synthesis are typically human perception-orientted. Consequently, we con- ducted user study involving 30 volunteers with computer vision backgrounds.
@
8
COR aJab
Permtaa:
213
20.3
227
26.3
211
301
AIIGAN
PISE:
GFIA
UPIN
CASI) NIED
PIDM
Ours
Different SOTA Methoks
This study included comparisons with funda- mental facts (namely, R2G and G2R) and com- parisons with other methods (i.e., J2b). A higher score on these three metrics indicates superior performance. As shown in Figure 7, PCDMs exhibit commendable performance across all three baseline metrics on DecpFashion. For instance, the proportion of PCDMs images perceived as real in all instances is 56.2% (G2R), nearly 20.4% higher than the next best model. Our Jab score stands at 44.1%, suggesting a preference for our method among the participants. Please refer to C.6 of the Appendix for more detail.
Figure 7: User study results on DeepFashion in terms of R2G, G2R and Jab metric. Higher values in these three metrics indicate better performance.
4.2 ABLATION STUDY
To demonstrate the efficacy of each stage introduced within this paper, we have devised the following variations, all of which fall under the PCDMs framework but incorporate distinct configurations. B1 represents using the inpainting conditional diffusion model only. B2 denotes the simulta- neous use of the prior conditional diffusion model and the inpainting conditional diffusion model without using the fining conditional diffusion model. B3 stands for the simul- taneous use of the inpainting conditional diffusion model and the refining conditional diffusion model, excluding the prior conditional diffusion model.
TecgeFoo Somrolarp
GT
BL
D2
PCOM
Figure 8 shows the impact of each stage on the DeepFash- ion. BI can generate person images that roughly conform to the target pose. However, there are severe distortions in the generated image, such as limb loss and confusion between appearance and limbs. This indicates that generating images from unaligned inputs is highly challenging task without global features. In contrast, in B2, once the global features of the target image obtained from the prior conditional diffusion model are added, the generated person images basically match the target pose in structure. Although B2's capability to generate images with consistent appearance is somewhat limited, it has already achieved a striking resemblance to the actual image. This shows that the refining condi- tional diffusion model can establish a dense correspondence between the source and target images, enhancing the contextual features. In addition, from visual perception perspective, B3 is superior to BI and B2 regarding detail texture, while is slightly inferior to B2 regarding pose coordination. Finally, when the PCDMs of the three stages in the last column, it is visually superior to B1,
Figure 8: Qualitative ablation results. See C.3 for more examples.
8

---

Published as a conference paper at ICLR 2024
B2, and B3. This indicates that when dealing with complex poses, the PCDMs of the three stages can gradually produce visually more satisfying results.
To more comprchensively validate the effectiveness of the proposed refining conditional diffusion model, we apply to other state-of-the--rt (SOTA) methods. As shown in Figure 9, the first and fourth rows denote the source im- age and ground truth (target image). The second and third rows present the results before and after refinement via re- fining the conditional diffusion model. We can observe that the refining conditional diffusion model significantly im- proves the results of all state-of-the-a methods. For in- stance, when dealing with ADGAN and PIDM, our method helps to fill in minor details, such as missing belts and hats, thereby enhancing the completeness of the generated im- ages. For methods like GFLA, DPTN, CASD, and NTED, model can finely process textures, maintain shape and texture consistency, and generate more explicit and realistic images. These results indicate that the refining conditional diffusion model has universality across different state-of-the-art methods, offering potential improvements for various person image generation tasks.
Figure 9: Effect of refining conditional diffusion model on other SOTA methods.
Table 2: Comparison with SOTA on person re-identification. Stan- dard denotes not using synthesized pedestrian images.
4.3 APPLICATION
In Table 2, we further evaluate the applicability of images generated by
Methods
Percentage of real images
100%
20%
40%
60%
80%
PCDMs in a downstream task, i.e., person re-identification (Ye et al., 2021; Shen et al., 2023a). We con- duct these re-identification experi- ments on the Market-1501 dataset, adhering to the PIDM protocol. Ini- tially, we randomly select subsets of 20%, 40%, 60%, and 80% from the total real training set of the Market-1501 dataset, ensuring each identity is represented by at least one image. This selection pro- cess yields a new dataset. Following this, we employ BoT (Luo et al., 2019) as the base network and conduct baseline training with each subset of the data. We then generate synthetic images from the new dataset, randomly selecting those that share the same identity and pose. This synthesized data is then merged with the original dataset train BoT. The Rankl results are presented in Table 2. The results indicate that PCDMs significantly boost the re- re-ientificaaton performance compared to the baseline. Furthermore, when compared with offte-ofthe-- art methods such as PTN, GFLA, DPTN, and PIDM, PCDMs consistently demonstrate superior performance in re-identification tasks.
Standard
33.4
56.6
64.9
69.2
76.7
PTN (7hu et 2019)
55.6
57.3
67.1
72.5
76.8
GFLA (Ren et al., 2020)
57.3
59.7
67.6
73.2
76.8
DPTN (Zhang et al,, 2021)
58.1
62.6
69.0
74.2
77.1
PIDM (Bhunia et al., 2023)
61.3
64.8
71.6
75.3
78.4
PCDMs (Ours)
63.8
67.1
73.3
76.4
80.3
CONCLUSION
This paper has demonstrated the significant potential of Progressive Conditional Diffusion Models (PCDMs) in addressing the challenges of pose-guided person image synthesis through three-stage process. In the first stage, a simple prior conditional diffusion model is designed to predict the global features of the target image by mining the global alignment relationship between pose coor- dinates and image appearance. The second stage establishes a dense correspondence between the source and target images using the global features from the previous stage, and an inpainting condi- tional diffusion model is proposed to further align and enhance the contextual features, generating a coarse- coarse-grained person image. In the final stage, a refining conditional diffusion model proposed to utilize the coarsely generated image from the previous stage as a condition, achieving texture restoration and enhancing fine-detail consistency. The three stages of the PCDMs work progres- sively to generate the final high-quality and high-fidelin synthesized image. Both qualitative and quantitative results have demonstrated the consistency and photorealism of our proposed PCDMs under challenging scenarios.
Future Work. Our method significantly improves person image synthesis quality, while the use of two additional prior and refining models leads to increased computational resource consumption and longer inference time. Future work should explore efficient methods that provide equivalent or superior quality while reducing computational overhead and inference time.
9

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
This study introduces a novel multi-stage person image synthesis technique capable of generating new person images based on different poses and original images. However, there risk that ma- licious actors could misuse this manipulation of real photos to create false content and disseminate misinformation. This is a well-known issue faced by virtually all person image synthesis methods. Nevertheless, research has made significant progress in identifying and preventing malicious tam- pering. Our work will provide valuable support for research in this field and external audits, helping to balance its value against the risks posed by unrestricted open access. This will ensure that this technology can be used safely and beneficially.
REFERENCES
Alberto Baldrati, Davide Morelli, Giuseppe Cartella, Marcella Cornia, Marco Bertini, and Rita Cuc- chiara. Multimodal garment designer: Human-centric latent diffusion models for fashion image editing, arXiv preprint arXiv:2304 02051,, 2023.
Ankan Kumar Bhunia, Salman Khan, Hisham Cholakkal, Rao Muhammad Anwer, Jorma Laakso- nen, Mubarak Shah, and Fahad Shahbaz Khan. Person image synthesis via denoising diffusion model. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- 2023.
fion, pp.
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7291-7299, 2017.
Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero- shot object-level image customization. arXiv preprint arXiv:2307.0948) 2023.
Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. Generative adversarial networks: An overview. IEEE signal processing magazine, 35 (1):53-65, 2018
Patrick Esser, Ekaterina Sutter, and Bjöm Ommer. A variational u-net for conditional appearance and shape generation. In CVPR, 2018
Artur Grigorev, Artem Sevastopolsky, Alexander Vakhitov, and Victor Lempitsky. Coordinate- texture inpainting for pose-guided human image generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12135-12144, 2019.
Xintong Han, Xiaojun Hu, Weilin Huang, and Matthew R Scott. Clothflow: A flow-based model for clothed person generation. Proceedings of the IEEF/CVF international conference on computer vision, pp. 10471-10480, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans.
Classifier-free diffusion guidance.
arXiv preprint
arXiv:2207.12598, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
Diederik Kingma, Max Welling, et al. An introduction variational autoencoders. Foundations and Trends Machine Learning, 12(4)3307-392, 2019.
FirstName LastName. Frobnication tutorial, 2014. Supplied supplemental material .r.
Yining Li, Chen Huang, and Chen Change Loy. Dense intrinsic appearance flow for human pose transfer. In CVPR, 2019.
10

---

Published as a conference paper at ICLR 2024
Wen Liu, Zhixin Piao, Jie Min, Wenhan Luo, Lin Ma, and Shenghua Gao. Liquid warping gan: A unified framework for human motion imitation, appearance transfer and novel view synthesis. In ICCV, 2019.
Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1096-1104, 2016.
Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei Jiang. Bag of tricks and strong baseline for deep person re- re-identification. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition workshops, pp. 0-0, 2019.
Zhengyao Lv, Xiaoming Li, Xin Li, Fu Li, Tianwei Lin, Dongliang Hc, and Wangmeng Zuo. Learn- ing semantic person image generation by region- region-adaptive normalization. In CVPR, 2021.
Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, and Luc Van Gool. Pose guided person image generation. NeurlPS, 2017.
Yifang Men, Yiming Mao, Yuning Jiang, Wei- Ying Ma, and Zhouhui Lian. Controllable person image synthesis with atribuue-deccomoosed gan. In CVPR, 2020.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint
arXiv:141 1784, 2014.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8177 PMLR, 2021.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint 2rX11:2304.07/93, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.005. 200.2225..0,,,, 06125, 1(2):3, 2022.
Yurui Ren, Xiaoming Yu, Junming Chen, Thomas H Li, and Ge Li. Deep image spatial transforma- tion for person image generation. In CVPR, 2020.
Yurui Ren, Yubo Wu, Thomas H Li, Shan Liu, and Ge Li. Combining attention with flow for person image synthesis. In Proceedings of the 29th ACM International Conference on Multimedia, pp- 3737-3745, 2021.
Yurui Ren, Xiaoqing Fan, Ge Li, Shan Liu, and Thomas H Li. Neural texture extraction and distri- bution for controllable person image synthesis. In CVPR, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and puttern recognition, pp. 10684-10695, 2022.
Fei Shen, Xiaoyu Du, Liyan Zhang, and Jinhui Tang. Triplet contrastive learning for unsupervised vehicle re-identification. arXiv preprint arXiv:2301.0944 2023a.
Fei Shen, Yi Xie. Jianqing Zhu, Xiaobin Zhu, and Huanqiang Zeng. Git: Graph interactive trans- former for vehicle re-identification. IEEE Transactions on Image Processing. 2023b.
Aliaksandr Siarohin, Enver Sangineto, Stéphane Lathuiliere, and Nicu Sebe. Deformable gans for pase-based human image generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3408-3416, 2018.

---

Published as a conference paper at ICLR 2024
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13356 2020.
Hao Tang, Song Bai, Li Zhang, Philip HS Torr, and Nicu Sebe. Xinggan for person image genera- tion. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV 16. pp. 717-734. Springer, 2020.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions image processing, 13(4):600- 612, 2004.
Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven CH Hoi. Deep learning for person re-identification: A survey and outlook. IEEE transactions pattern analysis and machine intelligence, 44(6)2282-22893, 2021.
Jinsong Zhang, Kun Li, Yu-Kun Lai, and Jingyu Yang. Pise: Person image synthesis and editing with decoupled gan. In CVPR, 2021.
Lymin Zhang and Maneesh Agrawala. Adding conditional control text-to-imaae diffusion models. arXiv preprint arXiv:2302.05553; 2023.
Pengze Zhang, Lingxiao Yang, Jian HuanHuung Lai, and Xiaohua Xie. Exploring dual-task correlation for pose guided person image generation. In CVPR, 2022.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Songchuan Zhang and Yiwei Zhou. Posediffusion: pose-guided human image generator via cas- caded diffusion models. Available at SSRN 4552596, 2023.
Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re-identification: A benchmark. In ICCV, 2015.
Xingran Zhou, Bo Zhang, Ting Zhang, Pan Zhang, Jianmin Bao, Dong Chen, Zhongfei Zhang, and Fang Wen. Cocosnet v2: Full-resolution correspondence learning for image translation. In CVPR, 2021.
Xinyue Zhou, Mingyu Yin, Xinyuan Chen, Li Sun, Changxin Gao, and Qingli Li. Cross attention based style distribution for controllable person image synthesis. In European Conference Computer Vision, pp. 161-178. Springer, 2022.
Zhen Zhu, Tengteng Huang, Baoguang Shi, Miao Yu, Bofei Wang, and Xiang Bai. Progressive pose attention transfer for person image generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2347-2356, 2019.
12

---

Published as a conference paper at ICLR 2024
SUPPLEMENTARY MATERIAL
This supplementary material offers more detailed exploration of the experiments and methodolo- gies proposed in the main paper. Section A provides series of symbols and definitions for enhanced comprehension. Section delves deeper into the implementation specifics of our experiments. Sec- tion C presents additional experimental outcomes, including broader range of qualitative compar- ison examples with state- ofte--ofthee of-the-art art methods, an expanded set of ablation study cases, supplemented results from random sampling, and a detailed explanation of our user studies.
A SOME NOTATIONS AND DEFINITIONS
Table 3: Some notations and definitions.
Notation
Definition
E0
Real image Additional condition
Timestep
Diffusion model
Gaussian noise
Guidance scale
Noisy data at step
as
Feature of source image
Pa
Feature of source pose coordinate
Pt
Feature of target pose coordinate
fat
Feature of source and target image
sm
Feature of source and mask image
Pst
Feature of source and target pose coordinate
Feature of coarse target image
B IMPLEMENT DETAILS
Table 4: Hyperparametees for the PCDMs. All models trained on 8 V100 GPUs.
Hyperparameters
Frist Stage
Second Stage
Third Stage
Diffusion Steps
1000
1000
1000
Noise Schedule
cosine
linear
linear
Optimizer
Adam
Adam
Adam
Weight Decay
0.01
0.01
0.01
Batch Size
256
128
128
Iterations
100k
200k
100k
Learning Rate
le
le 4
le
Our experiments are conducted on 8 NVIDIA V100 GPUs. We follow the standard training strate- gies and hyperparameters of diffusion models. We utilize the AdamW optimizer with consistent learning rate 1e-4 across all stages. The probability of random dropout for condition c is set 10%. For the prior conditional diffusion model, we employ OpenCLIP ViT-H/14 as the CLIP image encoder. This model's 's transformer consists of 20 transformer blocks, each with a width of 2,048. The model is trained for 100k iterations with a batch size of 256, using a cosine noising sched- ule (Nichol & Dhariwal, 2021) with 1000 timesteps. For the inpainting and refining models, we use DINOv2-G/14 as the image encoder. We leverage the pretrained stable diffusion V2.1 4, modifying the first convolution layer to accommodate additional conditions. These models are trained for 200k and 100k iterations, respectively, each with batch size of 128, and linear noise schedule with 1000 timesteps is applied. In the inference stage, we use the DDIM sampler with 20 steps and set the guidance scale w to 2.0 for PCDMs on stages.
"http://githuu..cm//// http://iiwiiioooo/o/oooononnnnnn clip
https://hugginggfccc.. http://gwww....i.iiiiaaaaaaaaaaa 1--baae
13

---

Published as a conference paper at ICLR 2024
C ADDITIONAL RESULTS
C.I MORE QUALITATIVE RESULTS WITH REFINING CONDITIONAL DIFFUSION MODEL..
We show the qualitative results PIDM combined with refining conditional diffusion model in Figure 10.
C.2 MORE QUALITATIVE COMPARISONS FOR PCDMS
We provide additional examples for comparison with the state-of-the-ar (SOTA) methods Fig-
11.
C.3 MORE QUALITATIVE ABLATION RESULTS ON THE DEEPFASHION DATASET
We show more qualitative ablation results from our proposed PCDMs in Figure 12. B1 signifies the exclusive use of the inpainting conditional diffusion model. B2 represents the concurrent ap- plication of both the prior and inpainting conditional diffusion models, without incorporating the refining conditional diffusion model. B3 denotes the combined usage of the inpainting and reñning conditional diffusion models, while excluding the prior conditional diffusion model.
C.4 RANDOM SAMPLES
We show random samples from our proposed PCDMs Figure 13 and Figure 14. The generated results highly stable and realistic.
C.5 IMPACT PRETRAINED
We show random samples from our proposed PCDMs in Figure 15. The results show that even without pretrained, the outcomes are still remarkably realistic. This indicates that the improvement in text reconstruction is primarily due to the rationality and superiority of our framework.
C.6 USER STUDY
In Section 4.1, we conduct user study on the pose-guided person image synthesis task. We invite 30 volunteers with a computer science background to participate side-by-side view comparison of PCDMs and thee-oftthe-rrt work and to identify the authenticity of images. These samples are randomly drawn from our toolkit test set, originating from the same original image and pose. We conducted repeated tests with 50 samples for each model. The user study evaluates which model produces more realistic results through human perception. Example questions are shown in Figure 16 and Figure 17.
C.7 VISUALIZE THE DIFFUSION PROCESS
To facilitate a better understanding of the diffusion process, we have added visualizations for the second and third stages of the diffusion process, shown in Figure 18 and Figure 19.
14

---

Published as a conference paper at ICLR 2024
Sourca
GT
PIDM
Target Pose
Source
PIDM
Ours
//
Refining
Refining
Target Pose
Source Image
GT
PIDM
Ours
Target Pose
Source Image
GT
PIDM
Ours
A Itur Redining
A Iter Redining
Target Poop
Source Image
GT
PIDM
Ours
Target Puse
Source Image
GT
PIDM
Ours
Relining
Relining
Target Pose
Source Image
GT
PIDM
Ours
Target
Image
PIDM
After Refining
After Relining
Target Pase
Sourcn Image
GT
PIDM
Ours
Target Pose
Source Image
GT
PIDM
Ours
A Refining
A Refining
Figure More examples show refining conditional diffusion model on PIDM. is worth noting that our denotes results obtained only using the prior conditional diffusion model and inpainting conditional diffusion mode.
15

---

Published as a conference paper at ICLR 2024
Target Pose
Source Image
GT
ADGAN
PISE
GPLA
DPIN
CASD
NTED
PIDM
Ours
R
absrvve bugs Yor VOUL
Figure 11: More qualitative comparisons between PCDMs and SOTA methods on the DeepFashion dataset.
16

---

Published as a conference paper at ICLR 2024
Target Pose
Source Image
GT
B1
B2
B3
PCDMs
Figure 12: More qualitative ablation results on the DecpFashion datasct.
17

---

Published as a conference paper at ICLR 2024
Target Pose
Source Image
GT
Sample
Sample 2
Sample 3
Sample
Figure 13:: Random samples from DeepFashion dataset by PCDMs.
18

---

Published as a conference paper at ICLR 2024
Target Pose
Source Image
GT
Sample
Sample
Sample
Sample
Figure 14: Random samples from Market-150 1501 dataset by PCDMs.
19

---

Published as a conference paper at ICLR 2024
WO
w
Target Pose
Source Image
GT
Pretrained
Pretrained
yes
yes
yes
yes
83
89
Figure 15: Compared with who pretrained weghts on PCDMs.
Pleuse chocss the musl lifelike image each group.
18.
B
F
OA
A
B
E
F
Oe G
Oh
H
Figure 16: An example question used in our user study for pose-guided person image synthesis.
20

---

Published as a conference paper at ICLR 2024
Ploase determine the following image is real.
6.
True
['alse
Figure 17: An example question used in user study for pose-guided person image synthesis.
21

---

Published as a conference paper at ICLR 2024
SteStep
6- 10 Step
11-15 Step
20
Figure 18: Visualize diffusion process on the second stage.
22

---

Published as a conference paper at ICLR 2024
1-5Step
6 10 Step
11- 15 Step
16- 16-205B Step
Figure 19: Visualize diffusion process on the third stage.
23


Reviews:
[{"review_id": "wpexlslDpJ", "reviewer": "Reviewer_nRkN", "summary": "This work proposes a three-stage diffusion pipeline to perform human pose transfer in a progressive way. The first stage predicts a global alignment feature to address the pose-level alignment and the second stage further aligns and enhances the contextual features to generate a coarse result. Finally, the third further refine the details according to the source image. The experiments demonstrate the SOTA performance of the proposed method.", "questions": "- A quantitive ablation study is needed.\n- The qualitative refining comparisons with PIDM + refining is necessary.\n- A visual analysis of the progressive alignment procedure at each stage is needed.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method is conducted in a coarse-to-fine paradigm. The technical novelty of each component is not high. While the whole system combining three diffusion stages can achieve new SOTA performance on the human pose transfer task.\n\n- Comprehensive experiments, ablations and user studies have been conducted to show the effectiveness of the proposed method.\n\n- The paper is well-written and easy to follow.", "weaknesses": "- The qualitative ablation in Figure 8 is good to show the effectiveness of each component of the proposed method. However, the quantitive ablation is also very important. Besides, I would expect adding results of PCDMs w/o refining in Table 1.\n\n- The refining results reported on different methods are on different samples. To better understand the refining benefits for different methods, it would be good to show the results of before-refining and after-refining on the same samples both quantitatively and qualitatively. For example, authors can add visual samples in Figure 10, and add quantitative results of PIDM + refining. I wonder if PIDM + refining can also achieve close performance.\n\n- According to the ablation study, the features predicted by the first stage is important for pose-level alignment. I would expect more analysis of the progressive alignment procedure. For example, the authors may make some visual analysis based on the features, such as pixel correspondence or warp results similar to CoCosNet.\n\n- In the metric section, it would be good to add references for R2G, G2R, and Jab to make the paper more self-contained.\n\n- I would suggest adding some samples of source image with invisible logo and target image with visible logo to the main paper. Otherwise, one may be concerned about the overfitting issue.\n\n- Since the proposed method has three diffusion stages, I wonder if its time cost will be much higher. Some analysis of the inference time could be discussed, which is important for practical applications.\n\n- The pipeline contains stages and such a coarse-to-fine paradigm has been adopted in other works in different ways. For example, [a,b] also adopt a human prior + inpaint + refinement strategies. More discussions on the whole pipeline and related works could make the paper more comprehensive.\n\n[a] Coordinate-based Texture Inpainting for Pose-Guided Human Image Generation, CVPR’19\n\n[b] Unselfie: Translating Selfies to Neutral-pose Portraits in the Wild, ECCV’20", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0BZPhbBQry", "reviewer": "Reviewer_hURi", "summary": "This paper proposes Progressive Conditional Diffusion Models (PCDMs) that tackles pose-guided person image synthesis with three stages: 1. predicts the global feature of target image; 2. establish dense correspondence and inpaint; 3. enhance and refine details. The authors show that the proposed method outperforms prior work.", "questions": "In Figure 6 visualization, it seems the proposed method is much better at the reconstruction of text patterns. Is this because the proposed method utilize a pretrained stable diffusion v2? How much performance gain over prior works could be due to having a pretrained stable diffusion v2?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The proposed method is clear and well-motivated. Studying the stage decomposition for solving a specific task with diffusion models could be a worthy contribution.\n2. The evaluation is comprehensive, and the proposed method show advantages over prior works in general, with user study and clear visualization.\n3. The effect of different components are studied with ablations, which further demonstrated the effectiveness of the proposed techniques and provides a better understanding of the proposed method.", "weaknesses": "In Table 1, the FID of the proposed method is sometimes worse than the prior work PIDM. Since both models are diffusion models, what are the potential main reasons that PIDM has a better FID, which is inconsistent to the user study / visualization that shows the proposed method is better?\n\n[minor]\n\"Although PCDMs have a lower FID score than PIDM\" -> \"worse FID score\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7mVHfoAjsJ", "reviewer": "Reviewer_m9vj", "summary": "This paper proposes Progressive Conditional Diffusion Models (PCDMs) for pose-guided person image synthesis. Unlike existing diffusion-guided methods that directly generate an image from source image and target pose, PCDMs progressively predict the global feature, inpaint a coarse target image, and further refine it with a conditional diffusion model. Qualitative and quantitative results show that the proposed method can effectively generate images that are structurally aligned with the target pose while preserving faithful and detailed texture from the source image. It achieves the state-of-the-art performance on the DeepFashion and Market-1501 datasets in terms of both objective metrics and user study.", "questions": "[Questions]\n\n* What is the overall training and inference time and how is it compared to the prior methods?\n* Why use different image encoders (CLIP and DINOv2) in different stages?\n* Are the MLP networks in the inpainting stage (Figure 4) and refining stage (Figure 5) the same?\n* Do you use the same guidance scale w in all training stages?\n\n[Minor suggestions]\n\n* Figures 1 and 2 seem to have a lot of redundancy. Maybe consider consolidating them into one?\n* Typo in user study paragraph 2: “J2b” -> “Jab”\n* Typo in ablation study paragraph 2: “Although the personal images generated by B2 can retain the appearance of the source image is limited”", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1: Sensible model design\nThe idea of aligning source and targets progressively at the image, pose, and feature levels is sensible and shown to be effective. The design of the prior conditional diffusion model and inpainting conditional diffusion model are quite interesting and technically sound.\n\nS2: Convincing qualitative and quantitative results\nThe visual results show a consistent improvement from prior methods in terms of pose/structural alignment with the target pose as well as texture details and faithfulness to the source image. Quantitatively, PCDMs produce slightly higher FID than a prior method (PIDM) but outperforms existing methods in other metrics. The user study also show a clear preference for PCDM results over other methods.\n\nS3: Good writing\nThe paper is well-written and easy to follow overall.", "weaknesses": "W1: Model complexity and computation overhead\nThe proposed framework is quite complicated and involves training multiple diffusion models whose inputs/outputs depend on one another. Since it is done in a multi-stage fashion, I’m wondering if the model performance is sensitive to certain training strategies or hyper-parameter tuning. Perhaps a more detailed quantitative ablation study can better justify the model design. Also, it would be good to show the computation overhead of each stage as well as the training/inference time.\n\nW2: Incomplete ablation study\nSection 4.2 shows qualitative results of the ablation study. However, quantitative comparisons of these variations are missing. Since each progressive training stage requires additional computation overhead but the visual differences between B2, B3, and full model are quite subtle, it is essential to show more evidence to justify the effectiveness of individual components.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yBqvgTPmZM", "reviewer": "Reviewer_dqE5", "summary": "The paper proposes a new pipeline based on image diffusion to tackle pose-guideed human image generation problem. The problem is decomposed into three stages, each stage requiring a separate network. At least two stages use a diffusion process, which means they involve multiple iterations, with a neural network inference step at each iteration. The first stage extracts features from the source image and source and target pose encodings. Next stage generates a coarse image, and the final stage involves refinement of the image.\n\nThe results outperform state-of-the-art in most metrics with a significant margin. The code and models will be available upon acceptance.", "questions": "What is novel in the submission? I do not assume there is no novelty at all, but I'd like to see a clear formulation of the novelty.\n\nHow does the whole process of image generation look like? How many iterations are used?\n\nx_t should go under the expectation (E) symbol in (3), because x_t is x_0 plus random noise, am I right?\n\n\"Additionally, we add an extra embedding to predict\nthe target global embedding.\" - what is the target global embedding exactly? Is it a feature vector representation of the target image?\n\nFig. 3 shows some noisy image labelled as ‘Target’ fed into the Image Encoder - what is that exactly?\n\nIn 3.3., “ To prevent confusion between black and white in the source and\ntarget images,” - unclear\n\n\"the timestep embedding\" - means the timestep of the diffusion model, right?\n\nIn (5), x_0 is under the expectation symbol, but not in the formula - why?\n\nIn Fig. 4, what do the noisy Source and Target images actually represent?\n\nIn general, to disuambiguate the text, I would suggest to reserve a term ‘target’ only for the dataset target image, and call a generated image aimed to resemble the target as ‘target estimate’.\n\nIn the experiments section there are no images for ‘Market-1501’ results. The results for DeepFashion show un-natural faces, and all faces look the same. This makes me think that the performance of the model for person re-identification will be limited, however the experiments show good performance on Market-1501. Would be nice to see some illustrations.\n\nHow long does it take to generate a new image using the proposed pipeline?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper outperforms state-of-the-art on a challenging DeepFashion dataset, as well as on the Market-1501 dataset, both in algorithmic metrics and according to a user study. User study in particular shows a big gap between the results of the proposed method and the competitive methods.", "weaknesses": "The paper does not clearly define its novelty. The contributions are formulated very vaguely. For example, \"we propose a novel inpainting conditional diffusion model to explore the dense correspondence between source and target images\" - what does \"explore\" really mean in this sentence? \"we introduce a new refining conditional diffusion model by further using post hoc image-to-image techniques to enhance the quality and fidelity of synthesized images.\" - what is new in this model? Is it a typical diffusion model for image refinement, or is there something special? All in all, it is hard to say whether the theoretical/technical novelty is significant enough.\n\nNext, the method is described quite inaccurately. Only red arrows in Fig. 2 indicate that there is some iterative process of image generation. The paper should describe in details the whole process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)