IMAGPose: A Unified Conditional Framework for Pose-Guided Person Generation
Authors: Fei Shen
Jinhui Tang
Venue: NeurIPS 2024 poster (2024)
Keywords: Pose-Guided Person Image Generation
Diffusion Model
Consistency Generation
Primary area: diffusion_based_models

Abstract: Diffusion models represent a promising avenue for image generation, having demonstrated competitive performance in pose-guided person image generation. 
However, existing methods are limited to generating target images from a source image and a target pose, overlooking two critical user scenarios: generating multiple target images with different poses simultaneously and generating target images from multi-view source images.
To overcome these limitations, we propose IMAGPose, a unified conditional framework for pose-guided image generation, which incorporates three pivotal modules: a feature-level conditioning (FLC) module, an image-level conditioning (ILC) module, and a cross-view attention (CVA) module. 
Firstly, the FLC module combines the low-level texture feature from the VAE encoder with the high-level semantic feature from the image encoder, addressing the issue of missing detail information due to the absence of a dedicated person image feature extractor. 
Then, the ILC module achieves an alignment of images and poses to adapt to flexible and diverse user scenarios by injecting a variable number of source image conditions and introducing a masking strategy.
Finally, the CVA module introduces decomposing global and local cross-attention, ensuring local fidelity and global consistency of the person image when multiple source image prompts. 
The three modules of IMAGPose work together to unify the task of person image generation under various user scenarios.
Extensive experiment results demonstrate the consistency and photorealism of our proposed IMAGPose under challenging user scenarios. 
The code and model will be available at https://github.com/muzishen/IMAGPose.

IMAGPose: A Unified Conditional Framework for Pose-Guided Person Generation
Fei Shen, Jinhui Tang" Nanjing University of Science and Technology ffeishen, jinhuit ang @njust edu. en
Abstract
Diffusion models represent promising avenue for image generation, having demonstrated competitive performance in pose-guided person image generation. However, existing methods are limited generating target images from source image and target pose, overlooking two critical user scenarios: generating multi- ple target images with different poses simultaneously and generating target images from multi-view source images. To overcome these limitations, we propose IMAG- Pose, unified conditional framework for pose-guided image generation, which incorporates three pivotal modules: feature-level conditioning (FLC) module, an image-level conditioning (ILC) module, and a cross- view attention (CVA) module. Firstly, the FLC module combines the low-level texture feature from the VAE encoder with the high-level semantic feature from the image encoder, addressing the issue of missing detail information due to the absence of a dedicated person images scenarios by injecting variable
image feature extractor. Then, the ILC module achieves an alignment and poses to adapt to flexible and diverse number of source image conditions and introducing a masking strategy. Finally, the CVA module introduces decomposing global and local cross-attention, ensuring local fidelity and global consistency of the person image when multiple source image prompts. The three modules of IMAGPose work together to unify the task of person image generation under various user scenarios. Extensive experiment results demonstrate the consistency and photorealism of our proposed IMAG- Pose under challenging user scenarios. The code and model will be available at https//ggitbb..oommmmmms//////c
Introduction
Pose-guided person generation [51, 29] aims transform source person image into target person image under specific pose while maintaining appearance consistency. It has many applications, including virtual reality, film production, and e-commerce. Besides, generated images can used to enhance the performance of downstream tasks, such as person re-identification [32, 54, 48, 34].
Early methods [4, 57, 42] usually are developed based on generative adversarial networks (GANs). However, GAN-based methods easily suffer from the instability of the min-max training objective and the difficulty of generating high-quality images in a single forward pass. Recently, methods based on diffusion models [1, 22, 11, 36] are becoming increasingly popular in the community. As promising alternative GAN for image generation, diffusion models utilize the source im- age and target pose as conditions. They generate the target image through a multi-step denois- single step. So. diffusion models help better retain the
ing process instead of completing input information. For example, PIDM [1] proposes texture diffusion module that inserts fea- tures of the source image, extracted by frozen encoder, into different stages of the diffusion
CCoreepoodingg author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

model's UNet. Similarly, PoCoLD [[1] further constrains the correspondence between the per- son and the pose by introducing additional 3D Densepose [19] annotations and interacting with the appearance features of the source image extracted by the frozen encoder. Additionally, the PCDMs [36] introduce a three-stage diffusion model that is used for predicting the global fea- tures of the target image, generating coarse-grained person image, and refining texture details. Despite the above methods based on diffusion models' impressive results compared to GAN-based approaches, existing methods still exhibit 2 shortcomings. (1) For methods based on diffusion models, they neglect un- derlying texture detail information. As these methods employ frozen image encoder, e.g., CLIP, to extract ap- pearance features from the source image, which is trained on general image dataset, it is not a specialized feature extractor for person images and can only extract high- level semantic features.
(2) For existing methods, shown in Figure 1, they overlook two critical user see- narios: generating multiple target images with differ- ent poses simultaneously and generating target images from multi-view source images. In practice, users often need to generate multiple consistent target images based on different poses simultaneously, such as in portrait photography and clothing displays. In addition, users often upload multiple source images to provide rich details for generating target image. However, the fixed architecture of existing methods limits the ability of users to make arbitrary modifications and extensions.
Figure I: Existing methods can only support generating target image from one source image and one target pose.
We propose a unified conditional framework named IMAGPose to address the above issues. For the first issue, we devise that the feature-level conditioning (FLC) module extracts low-level texture features from the VAE encoder and high-level semantic features from the image encoder. Because the frozen VAE can almost reconstruct images losslessly, it can easily extract low-level information without training. By leveraging the VAE's encoding capability, we fuse the source image's 's low-level and high-level information, addressing the absence of dedicated person image feature extractor. To address the second issue, we propose the image-level conditioning (ILC) module randomly injecting various numbers of source image conditions and introducing a masking strategy adapt different user scenarios. Specifically, we combine all target images into joint target, and then the masking strategy randomly masks the target image into a joint mask according to user scenarios. Meanwhile, the ILC extracts the pose from the source image and combines it with the target pose to form a joint pose. Therefore, IMAGPose can inject image-level source image conditions from the input end, achieving one-to-one correspondence between the person's 's image and pose. Further, we present the cross-view attention (CVA) module, introducing decomposable global and local cross-attention to ensure the person images' local fidelity and global consistency when any user scenarios. We summarize the contributions of this paper as follows,
We are the first to explore gose-guided image generation tasks under different conditions and propose a unified conditional framework for synthesizing high-fidelity person images.
We present the FLC module, which addresses the issue of missing texture features due to the absence of a dedicated person image feature extractor.
We devise the ILC module to inject variable number of source image conditions and introduce a masking strategy, achieving an aligmment of images and poses.
We develop a CVA module to decompose global and local moos-aattentoon, ensuring the joint image's 's local fidelity and global consistency when multiple-view source image prompts.
Extensive experiments on two challenging datasets show that proposed IMAGPose surpasses existing models. We also conduct user study and applications for downstream tasks to comprehensively evaluate the advantages of IMAGPose.
2
Related Work
Based on GAN Methods. Early approaches [24, 25, 37, 57,37, 37, 49, 49, view the synthesis task as
conditional image generation, utilizing conditional generative adversarial networks (CGANs) [26] to generate target images based on source appearance images and target poses. For example, 2G2 [24]

---

proposed two-stage method that refines the generated images in an adversarial manner. To decouple pose and appearance information, VUNet [8] suggests learning pose-independent features tackle the complex structure problem of pose space transformation. Then, ADGAN [25] introduced character attribute decomposition module to explicitly parse the human body and blend and insert conditions. VariGANS [53] combine variational inference with GANs to generate multi-view images from single image, refining results from coarse to fine [5. 7, 6]. However, GAN-based methods [24, 25, 37] often suffer from issues such as blurred texture details and instability of the min-max training objective. To address the issue of blurred texture details, Grigorev et al. [9] proposed a CNN-based framework that first performs pose warping, followed by texture repair. Unselfie [18] introduced a pipeline that first identifies the target's s neutral pose, then repairs appearance textures, and finally perfects and synthesizes the character in the background. However, these multi-stage methods require the introduction of multiple additional models and costs, impacting the efficiency of generation.
Based on Diffusion Models. Diffusion models have found extensive applications in virtual dress- ing [33], story generation [35], and portrait animation [45]. To address the instability of GAN's min-max training, methods based on diffusion models [13, 44, 40, 38, 31, 28] are gradually becoming more popular in the community. The core idea is to start from a simple noise vector and gradually transform it into high-quality image through multiple denoising iterations, making it suitable for high-fidelity and context- aware generation. Besides unconditional generation [14, 39, 41]. various methods [50, 47, 27] have been introduced to incorporate user- providedd control signals into the generation process, thereby achieving more controllable image generation. For example, PIDM [1] proposed texture diffusion module that uses the source image features extracted by the frozen encoder as conditions to be inserted into different stages of UNet. CFLG [22] proposed a mixed- granularity attention module to inject multi-scale fine-grate fine-grained character appearance features into the generation process. PoCoLD [11] further constrains the correspondence between person images and poses by introducing additional 3D Densepose [19] annotations of poses and source image appear- ance features extracted by the frozen encoder. Subsequently, PCDMs [36] proposed a three-stage diffusion model that predicts global features under the target pose, aligns images and poses, and then refines texture details. Besides, APS [16] progressively couples target pose and appearance for effective human image synthesis. PoSynDA [20] simulates 3D pose distributions to address the lack of 2D-3D correspondences, enhancing data diversity. MotionEditor [43] uses a dual-branch structure that decouples key-value queries, preserving the background and character appearance for efficient content editing. However, the frozen encoders introduced in these diffusion model-based methods are trained on generic image datasets and can only extract high-level semantic features of person, easily overlooking low-level texture details. Furthermore, in real scenarios, these models do not support generating multiple target images at once and target generation under multi-view image prompts.
3 Method
Figure :illustrates IMAGPose, a unified conditional framework that encompasses 3 core modules: feature-level conditioning (FLC) module, image-level conditioning (ILC) module, and cross-view attention (CVA) module. This framework aims to generate high-fidelity and high-quality target images in any user scenario. The FL.C module combines low-level texture features from the VAE encoder with high-level semantic features from the image encoder, which addresses the issue of losing texture details due to lacking dedicated person image feature extractor. (Section 3.2). The ILC module introduces a masking strategy for the target image and incorporates a variable number of source image conditions. This module aligns the image and pose, making it suitable for different user scenarios. (Section 3.3). The CVA module proposes the decomposition of global and local cross-attention. This ensures the entire person images' local fidelity and global cSnsistency,(Section 3.4).
3.1 Preliminaries
Diffusion models belong to a class of generative models. And these models are trained to reverse a diffusion process, which systematically introduces Gaussian noise to the data through a fixed Markov chain over a series of timesteps, denoted as f. Concurrently, a denoising model is trained to generate samples starting from this Gaussian noise. The training objective of a diffusion model, denoted ea and parameterized by 0, typically employs a mean square error loss LDM- For each timestep 4, this loss is defined as follows,
LDM Exo.e..olle alle Ea (2,,cct))1 t)12
(I)
3

---

(b) Image-leve Levll Conditionng
Spotial theation
Cross theation
Crows Proo-View View Ateetionn
Enceder
Target
Juint Pow
Targe Pouss
Demoising
VAE Ercods
Fins
Jolnn Tarpet
Blewer Munk
Predit
Madked Lotent
Juint
Embeddings
Wine dd
VAE Encoder
Image Encoder
Layer
Concatenution
Source Image
Feature- Condilioning
Saurce
Figure The IMAGPose unified conditional framework designed generate high-fidelity and high-quality target person images under various conditions. IMAGPose aims address the issue of detail texture loss, achieve an alignment of person images and poses, and ensure the person images' local fidelity and global consistency.
here, Zo represents the original data, supplemented by condition The timestep of the diffusion process represented by t E [0, The noisy data at the t step, denoted as Tt, is defined as QgZo + d.c. In this context, ay and are predefined functions of that dictate the diffusion process.
Once the model tg trained, images can synthesized from random noise through an iterative process. During the sampling stage, the predicted noise is calculated based on the predictions of both the conditional model eg (zie; c, 1) and the unconditional model eo (xx, L). using classífier-free
guidance [15] according to Eq. 2.
(x c, t) = wep(Ie; c, t) + (1- w)eo(ar,
where is the guidance scale used adjust the condition
3.2 Feature-Level Conditioning Module
Existing methods typically depend frozen encoders to extract appearance features from source images. However, these frozen encoders are trained on general image datasets, not explicitly designed for person image feature extraction. So, can only capture high-level semantic features of the source image. To address this, as depicted in Figure 2(a), given that the frozen VAE can almost reconstruct images losslessly without training, we propose the FLC module, which simultaneously extracts low-level texture information from a VAE encoder [17] and high-level semantic information from an image encoder. Specifically, the FLC module consists of frozen VAE encoder, an image encoder, and a trainable tokenizer layer. Assume that the source image E Rexhxw where h, and w represent the channel, height, and width of the image, respectively. First, we extract the latent features Fi Rx4x//8w/88 of the source image from the frozen VAE encoder while simultaneously extracting the high-level semantic features Fx E Igxd of the person image from the frozen image encoder, where n and denote the token length and each token dimension, respectively. Then, we convert latent features FÚ into texture features Fr. E m through a learnable tokenizer layer consisting of a convolution and a flattened operation, where m stand for length of Fr, token. We ensure that the dimension of each token from the texture features Fr. matches the dimension of each token from semantic features Fи through the kernel size of the 2D convolution. Finally, according to Eq. concatenate Fi and FH along the token length to obtain the person's 's appearance features.
Fc concat( FL, FH),
where Fe E [md+ Therefore, FL.C leverages the frozen VAE encoder's ability to preserve detailed information, addressing the issue of not having dedicated person image feature extractor.

---

3.3 Image-Level Conditioning Module
As previously mentioned, existing methods only support the generation setting of one source image and one target pose, which limits user expectations in other use scenarios. To address this, we propose the ILCC module to inject random variable number of source image conditions and introduce masking strategy to adapt to different user scenarios. As shown in Figure (a)), when generating three target images based three target poses, the ILC first masks the three target images to be generated, then combines them with source image to form joint mask, while extracting the target pose from the source image and combining it with the three given target poses to form a joint pose; As shown in Figure 3(b), when generating a target image based on three source images, the II.C first masks one target image and then combines it with three source images to form a joint mask while extracting the poses from the three source images and combining them with one given target pose to form a joint pose. This achieves one-to-one correspondence between the person's 's image and pose.
Deseidig UNet
Menttle Terpe
la) Gensullng mahipk tagget leaqes frees mnkiipee tarqet poscs.
Denoiding UNet
froe multi-lee
Figure 3: The masking strategy flexibly unify different scenarios.
frozen VAE to extract latent space features from the joint target and add noise to obtain the noise latent. Similarly, we use a frozen VAE to get the masked latent from the joint mask. The ILC module concatenates the noise latent and masked latent along the channel dimension. Then, we introduce a binary mask, the width and height of which are the same as the noise latent, with 0 and representing the masked and unmasked parts, respectively. This ensures that the model correctly distinguishes the areas of the target image to be generated. Therefore, the input channels of our model are with the channel numbers of noise latent and masked latent both being and the binary mask being 1. Besides, noted that existing methods ignore the image-level conditions of the source image and only inject the feature-level conditions of the source image through the cross- acoss-attention module. In contrast, as shown in Figure 2(b). we inject the image-level conditions of the source image combined with the masking strategy, achieving richer inherent context information and spatial structure alignment between the person 's image and pose.
Specifically,
3.4 Cross- View Attention Module
To ensure that each target image in the generated joint objec- tives possesses realistic visual content and maintains consis- tency in the person subject images, inspired by [2, 10], we propose a cross-view attention (CVA) module decompose global and local cross-attention. As shown in Figure 2, the
Figure 4: Illustration of the CVA module.
CVA module is embedded after each cross-attention module in the denoising UNet. The schematic diagram of the CVA module is shown in Figure 4, which mainly consists of the split, projection layer, self-attention layer, and joint. We first take the output of the feature by cross-attention as the input features of CVA, split the global input features into four smaller local person features, and then add new temporal dimension to learn the attention between each local character image. Subsequently, the four person features are reshaped back into one global output feature, ignoring the temporal dimension. This approach allows IMAGPose capture both global and local attention, providing a more nuanced understanding of the relationships between different parts of the persons' image.
For the pose condition, introduced pose encoder identical ControlNet for injection after the first convolutional layer. Therefore, the loss function LMSE of IMAGPose according to Eq. 4, as follows. Here, Fc, F, and Fp denote the feature of the FLC module, the feature of the ILC module and pose feature, respectively.
LMSE = Fe (0 Fc, FJ, Fp, 11?. In the inference stage, classifier-free guidance according to Eq. 5.
(4)
EØ( Fc, Fr, = wep(se, Fc, Fr, t) + (1 - w)E Fp,
(5)
5

---

Table 1: Quantitative comparison of the proposed IMAGPose with several statc-of models.
Dataset
Methods
SSIM (1)
LPIPS (
FID ())
Def-GAN [37]
0.6786
0.2330
18.457
PATN [57]
0.6709
0.2562
20.751
ADGAN [25]
0.6721
0.2283
14,458
PISE [49]
0.6629
0.2059
13,610
GFLA [30]
0.7074
0.2341
10.573
DPTN [51]
0.7112
0.1931
11.387
DeepFashion [21] (256 x 176)
NTED [29]
0.7182
0.1752
8,6838
CASD [56]
0.7248
0.1936
11.373
PoCoLD [
0.7310
0.1642
8.0667
PIDM [T]
0,7312
0.1678
6.3671
CFLD [22]
0.7378
0.1519
6.8040
PCDMs [36]
0.7444
0.1365
7.4734
IMAGPose (Ours)
0.7561
0.1284
5.8738
CocosNet2 [55]
0.7236
0.2265
13.325
NTED [29]
0.7376
0.1980
7,7821
PIDM [1]
0.7419
0.1768
5,8365
DeepFashion [21] (512 x 352)
PoCoLD [11]
0.7430
0.1920
8,4163
CFLD [22]
0.7478
0.1819
7.1490
PCDMs [36]
0.7601
0.1475
7,5519
IMAGPose (Ours)
0.7718
0.1396
5.6298
Def-GAN [37]
0.2683
0.2994
25,364
PTN [57]
0.2821
0.3196
22.657
GFLA [30]
0,2883
0.2817
19.751
Market- 1501 [54]
DPTN [51]
0.2854
0,2711
18,995
(128 x 64)
PIDM [1]
0.3054
0.2415
14,451
PCDMs [36]
0.3169
0.2238
13.897
IMAGPose (Ours)
0.3282
0.2104
12.659
4 Experiments
Datasets. We conducted experiments on the DeepFashion dataset [21]. which consists of 52,712 high-resolution images of fashion models, and the Market-1501 dataset [54], which includes 32,668 low-resolution images with diverse backgrounds, viewpoints, and lighting conditions. We extracted skeletons using OpenPose [3] and followed the dataset splits provided by [1]. It's important to note that the person IDs of the training and testing sets do not overlap for both datasets.
Metrics. We conducted comprehensive evaluation of the model, including both objective and
subjective metrics. Objective metrics include the structural similarity index measure (SSIM) [46].
learned perceptual image patch similarity (LPIPS) [52], and fréchet inception distance (FID) [12].
On the other hand, subjective assessments focus on user-oriented metrics, such as the percentage of real images misclassified as generated (R2G) [24], the percentage of generated images misclassified as real (G2R) [24], and the percentage of images considered superior among all models (Jab) [37].
Implementations. We conduct experiments on 8 NVIDIA V100 GPUs. Our configuration can be summarized as follows: (a) We use the pre-trained Stable Diffusion V1.5 and modified the first convolutional layer accommodate additional conditions. Unless otherwise specified, Dinov2-G/14 as the image encoder. In the tokenizer layer, both the kernel size and stride of the 2D convolution are 16, and the dimensions of the input and output channels are and 768, respectively. (b) Following [1, 36], we train our model on the DeepFashion dataset with sizes of 256 x 176 and 512 x 352. For the Market-1500 dataset, we used images of size 128 64. (c) In the masking strategy. we defaulted to randomly occluding 1-4 images. (d) The model is trained for 300k steps using the AdamW optimizer with learning rate 5e -5, Each batch size is 4, and linear noise schedule of 1000 time steps is applied. (e) In the inference stage, we used DDIM sampler with 20 steps, and set w to 2.0 in the guidance scale.
4.1 Main Results
IMAGPose supports generating one multiple target images under different conditions based on user scenarios. For fair comparison and adaptation to the IMAGPose framework, we use three

---

Searce
ADGAN
DPTN
CASD
NTED
PIDM
PACALD
CTLD
PCDNs
0
K
4
4
Figure 5: Qualitative comparisons with several Hfte-offthe-ae models the DeepFashion dataset.
black images for padding in the masking strategy and replicate the target pose times the joint pose during the inference stage. Then IMAGPose generates same target images, and we randomly select one generated image for qualitative and quantitative evaluation. Unless otherwise specified, the above settings the default configurations.
Quantitative Results. From Table 1, we quantitatively compared proposed IMAGPose with several
late-of-theearr methods including Def-GAN [37], PATN [57], ADGAN [25]. PISE [49]. GFLA [30], DPTN [51], NTED [29], CocosNet2 [55], CASD [56], PoCoLD [11], PIDM [1]. PCDMs [36] and CFLD [22]. Compared to other generative methods, based on diffusion models [11, 1, 36, 22] perform significantly better, especially proposed IMAGPose, which outperforms all methods. For example, IMAGPose significantly leads the GAN-based method ADGAN the SSIM metric since the GAN framework's unstable training fails to generate high-quality images. Compared to CFLD, diffusion model framework, IMAGPssc demonstrates absolute performance
which is also based advantages at both 256 x 176 and 512 X 352 scales. Because existing architectures overlook texture details and only inject appearance features using generic image encoder.
Table also shows the quantitative results on the Market- 1501. Like the DeepFashion, IMAGPose outperforms all SOTA methods in SSIM, LPIPS, and FID. Compared to NTED, which uses fine- grained texture features, ours shows more significant advantages, thanks VAE's inherent ability to reconstruct low-level texture. Despite PCDMs' additional refinement model, IMAGPose provides richer context information by injecting source image features at both image and feature levels.
Qualitative Results. Figure 5 visually compares our IMAGPose with other SOTA methods on the DeepFashion dataset. The results lead to several conclusions: (1) As shown in the first to third rows, diffusion-basee methods capture minute clothing textures, but IMAGPose generates higher-quality details than other diffusion models. (2) cases of large-scale pose transformations (as shown in the fourth and fifth rows), CFLD and PCDMs exhibit clear overfitting to the pose (i.e., objects do should appear on the hand ), only our method can generate images aligned with the target and consistent in detail texture. (3) As shown in the sixth to seventh rows, when generating complex textures and invisible areas, IMAGPose reasonably infers detailed textures, providing better visual consistency even if the results are inconsistent with the target. In summary, the proposed method consistently produces more realistic and lifelike person images, demonstrating the advantages of the image-level and feature-level level conditions proposed by IMAGPose. Please see C. for more examples.
7

---

User Study. The above quantitative and qualita- tive comparisons reflect the significant advantages of IMAGPose in generating results. Further, we con- ducted a user study with 50 volunteers. This study includes comparisons with basic facts (i.e., R2G and G2R) and other methods (i.e., J2b). The higher the score for these three indicators, the better the perfor- mance. As shown Figure 6, IMAGPose exhibits commendable performance all three indicators on the DeepFashion dataset. For instance, volunteers judged our generated images 58.5% (G2R) real images, nearly 18.4% higher than the second-best model. The proposed IMAGPose scored 42.3% on the Jab index, indicating method is popular. Please see C.3 of the Appendix for more detail.
BKar cGa mjز
200
DPIN
CASD
NIED POOLD PIM CFLD PCINS Ous Differett SOTA Mithod
Figure User study results on DeepFashion in terms of R2G, G2R and Jab metrie. Higher values in these three metrics indicate better performance.
Uniformity. IMAGPose only needs be trained once support multiple user scenarios. Here, IMAGPose simulates two user scenarios: generating a target image given source image and target pose, and generating a target image given multiple source images and target pose. As shown in Figure 7, both TI and T2 provide one source image and one target pose, similar to the previous SOTA. The difference is that T1 replicates the target pose in the joint pose, while T2 replicates the source image in the joint image. T3 randomly uses multiple different source images.
0.727
0.77
0770
0.79
614
633
0.756
111.8
0.126
57.8H
0.33 300
200g
100k
0.1172
2016
5,006
Thinday Steps
Tindadop Skape
Figure 7: Quantitative comparison of I'MMAPoss under different scenarios the DeepFashion dataset.
We found that under these 3 different settings, compared to Table 1, T1, T2, and T3 demon- strate strong competitiveness on all metrics, indicating that IMAGPose can adapt to the above scenarios only once training. For example, T2 surpasses all previous SOTA methods and beats T1, suggesting the ILC module can effectively capture the appearance features of person by replicating the same source image multiple times during the inference stage. Then, when us- ing multiple different source images to generate target image, T3 achieves 0.7727, 0.1172, and 5.3286 on SSIM, LPIPS, and FID, respectively, outperformers T2 and other SOTA meth- ods, indicating that IMAGPose has uniformity when provided multiple different source images. Furthermore, IMAGPose can also adapt to the other sce- nario: generating multiple target images from source image and multiple target poses, i.c., IMAGPose* in Figure 8. Compared with the original IMAGPose, IMAG- Pose slightly reduces the SSIM performance but still surpasses other diffusion model-based methods, which proves that even when inferring multiple different target images at once, IMAGPose* can still maintain high qual- ity and fidelity. Regarding speed, due to the architectural Figure 8: Results of speed and performance. advantage of the masking strategy in IL.C, it is nearly 8 times faster than PIDM, and nearly 3 times faster than PoCoLD and CFL.D. Noted that both the default settings of IMAGPose and IMAGPose* are highly competitive regarding generation quality and speed. Therefore, once our method trained, users can choose the appropriate inference settings according to their needs in different scenarios. The visual results of IMAGPose in different user scenarios are shown in Figure 9, which is consistent with our above analysis. Overall, IMAGPose can unify different tasks in real-world scenarios. For more results and discussions, please refer to C.2 of the Appendix.
SSIS
87m
CELD
0.735
PUIBS
01722
Speed6o
4.2 Ablations
Effectiveness of the Module. Table 2 demonstrates the impact of different modules on the results of IMAGPose. BO denotes IMAGPose removes the CVA module, the texture features from the VAE in the FLC module, and the image-level conditions in the ILC module, which results in a model with only input channels. B1 represents, based on BO, the addition of the CVA module. B2 denotes, based on BO, adding the VAE's texture detail features, resulting complete FLC module. B3 indicates,
8

---

GT
T1 (efanitt
T2
13
IMAG
Figure 9: Visual comparison of our model's uniformity across different user scenarios.
based on BI, that adding image-level conditions forms a complete ILC module with 9 input channels. B4 represents IMAGPose using both the FLC and ILC modules, while ignoring the CVA module. As shown in Table 2, IMAGPose outperforms other settings
on three metrics and gradually improves performance as
Table 2: Ablation study on DeepFashion.
more modules are included. Specifically, B surpasses BO on SSIM, LPIPS, and FID, indicating that the CVA module can enhance local fidelity and global consistency. B2 and B3 outperform B1 by 0.0125 and 0.0172 on SSIM, suggest- ing that FLC and ILC can effectively provide exemplary texture detail guidance and ensure alignment of images and poses. Moreover, noted that the combination of the three modules brings additional performance improvements. For example, method outperforms B2, B3, and B4 on all metries, respectively.
Mothods
SSIM (f)
LPIPS (3)
FID (I)
B0
0.7235
0,1704
8,2153
BI 0.7281
0.1646
7,8209
B2
0.7406
0,1471
7,3284
B3
0,7453
0,1390
6.3655
B4
0.7528
0.1326
6.0242
Ours
0.7561
0.1284
5.8738
Model Variant. Table summarizes the variants based on IMAGPose. We keep other settings unchanged and only modify the base model and image encoder. The overall performance differences between various versions of the base diffusion model and image encoder are insignificant, indicating that the base diffusion model and image encoder have a minimal impact on the results of the generated im- ages. Specifically, the results suggest that when 'Base.' is SD V1.5, Dinov2-G/1s slightly outperforms on all metrics. Furthermore, when using the same image encoder, SD V2.1 is superior to SD V1.5 in most cases.
Table 3: Performance of variant version.
Image Ele.
SSIM
LPIIS
FID
CLIP-B/Ia 0.7516
0.1361
6.1342
CLIP-bigGi
07548
0.1351
5,9645
SDV
CLIP-H64
0.7522
0.1296
6,0231
Dinov2
02541
0.1343
5,9256
Dinov2 L/14
0.35
0,1321
5,9432
Din0v2-G/14
0.7561
0,1284
5,8738
CLIP-B/I4
CLiP-bigtit
0.2535
0.1251
6.222
SD V2.1
Dinev2: B/14
CLI--H/
0.7553
0.7542
0.1301
5,9120
Dinov2-L/14
0.7579
0.1276
6.1315
Dinev2-G/14
0,2572
0.1294
5,8543
4.3 Application
Table 4; Comparison with SOTA on per- re-identification. Standard denotes not using generated person images.
Table 4 evaluates the applicability of images generated by IMAGPose in downstream tasks, i.e., person re-identification. Initially, we randomly selected 20%, 40%. 60%, and 80% sub- sets from the training set of the M5rket-1501 dataset, ensuring at least one image per identity, thereby creating new dataset. We adopt BoT [23] the base network and perform baseline training on each data subset for a fair comparison. Then, we incorporate images generated by IMAGPose. These generated images are synthesized from randomly selected images with the same identity and pose from the new dataset. The Rank results are shown in Table 4. The experimental results indicate that the IMAGPose algorithm significantly improves re-identification performance compared the Standard. Moreover, IMAGPose consistently demonstrates superior performance in re-identification tasks compared SOTA methods.
Methods
Percentage of real images
20%
PIN [57]
72.5
GFLA [10]
57.3
59.7
67.6
73.2
DPTN [5L]
58.1
62.6
69.0
74.2
PIDM [1]
61.3
64.8
71.6
75.3
PCDMs [36]
63.8
67.1
73.3
76.4
IMAGPose
66.4
69.2
75.1
77.6
5 Conclusion
This paper presents IMAGPose, unified conditional framework for pose-guided image generation. This framework addresses the limitations of existing methods by enabling the generation of multiple target images with different poses simultancously and from multi-view source images. IMAGPose

---

incorporates three pivotal modules: Feature- Conditioning (FLC) module, an Image- Level Conditioning (ILC) module, and a Cross- View Attention (CVA) module. The FLC module effectively combines low-level texture features with high-level semantic features, addressing the issue of missing detail information. The ILC module aligns images and poses to adapt to diverse user scenarios, while the CVA module ensures local fidelity and global consistency when multiple source image prompts. The proposed framework has demonstrated competitive performance in both qualitative and quantitative evaluations, showing consistency and photorealism under challenging user scenarios. This work represents a significant step forward in the field of image generation, opening up new possibilities for user interaction and customization. Limitations. (1) IMAGPose supports generating images in different user scenarios and significantly improves the synthesis quality. However, due to computational limitations, IMAGPose currently supports generating up four images at time. Future work will continue to explore how to generate more target images in one forward process. (2) The performance of the model is subpar for generating cartoon characters and non- non-photoreallstic styles because our training data consists of photorealistic human images for fair comparison. In future work, we plan include broader range of data and design style transformation modules to overcome this bias.
Acknowledgments and Disclosure of Funding
This work is supported by the National Natural Science Foundation of China (Grant No. 61925204, and 62332010). The authors would thank all Reviewers and ACS of OpenReview for the thorough review and insightful suggestions.
References
[1] Ankan Kumar Bhunia, Salman Khan, Hisham Cholakkal, Rao Muhammad Anwer, Jorma Laaksonen, Mubarak Shah, and Fahad Shahbaz Khan. Person image synthesis via denoising diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5968 -976. 2023.
[2] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563- 22575, 2023.
[3] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7291-7299, 2017.
[4] Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. Generative adversarial networks: An overview. IEEE signal processing magazine, 35(1):53 65, 2018.
[5] Jisheng Dang, Huicheng Zheng, Jimming Lai, Yan, and Yulan Guo. Efficient and robust video object segmentation through isogenous memory sampling and frame relation mining. IEEE Trunsactions on Image Processing. 32:3924 3938, 2023.
[6] Jisheng Dang, Huicheng Zheng, Xiaohao Xu, Longguang Wang, and Yulan Guo. Beyond appearance: Multi-frame spatio-temporal context memory networks for efficient and robust video object segmentation. IEEE Transactions on Image Processing, 33:4853 2024.
[7] Jisheng Dang. Huicheng Zheng, Xiaohao Xu, Longguang Wang. Qingyong Hu, and Yulan Guo. Adaptive sparse memory networks for efficient and robust video object segmentation. IEFE Transactions on Neural Networks and Learning Systems, 2024.
[8] Patrick Esser, Ekaterina Sutter, and Bjöm Ommer. A variational u-net for conditional appearance and shape generation. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 8857-8866, 2018
[9] Artur Grigorev, Artem Sevastopolsky, Alexander Vakhitov, and Victor Lempitsky. Coordinate-based texture inpainting for pose-guided human image generation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 12135-12144, 2019.
[10] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animated- iff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint 00Xi:2307.044255.. 2023.
10

---

[11] Xiao Han. Xiatian Zhu, Jiankang Deng. Yi-Zhe Song, and Tao Xiang. Controllable person image synthesis with pose-constrained latent diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22768-22777, 2023.
[12] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances neural information processing systems, 30, 2017.
[13] Jonathan Ho, Ajay Jain, Pieter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[14] Jonathan Ho, Ajay Jain, Pieter Abbeel, and UC Berkeley. Denoising diffusion probabilistic models.
[15] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12550 2022.
[16] Siyu Huang, Haoyi Xiong, Zhi-Qi Cheng, Qingzhong Wang, Xingran Zhou, Bihan Wen, Jun Huan, and De- jing Dou. Generating person images with appearance-awart pose stylizer. arXiv preprint arXiv:2007.09077, 2020.
[17] Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6111 2013.
[18] FirstName LastName. Frobnication tutorial, 2014. Supplied supplemental material t .d.f..
[19] Yining Li, Chen Huang, and Chen Change Loy. Dense intrinsic appearance flow for human pose transfer. In CVPR, 2019.
[20] Hanbing Liu, Jun-Yan He, Zhi-Qu Qi Cheng. Wangmeng Xiang, Qize Yang. Wenhao Chai, Gaoang Wang. Bao, Bin Luo, Yifeng Geng, et Posynda: Multi-hy Multi-hypothesis pose synthesis domain adaptation for robust 3d human pose estimation. In Proceedings of the 31st ACM International Conference on Multimedia, pages 5542-5551, 2023.
[21] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1096-1104, 104, 2016.
[22] Yanzuo Lu, Manlin Zhang, Andy Ma, Xiaohua Xie, and Jian-Huang Lai. Coarse-to-fine latent diffusion for pose-guided person image synthesis. arXiv preprint arXiv:2402. 18078, 2024.
[23] Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei Jiang. Bag of tricks and a strong baseline for deep person re-identification. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition workshops, pages 0-0, 2019.
[24] Liqian Ma, Xu Jia, Qianru Sun, Bert Schiele, Tinne Tuytelaars, and Lue Van Gool. Pose guided person image generation. In NewrlPS, 2017.
[25] Yifang Men, Yiming Mao, Yuning Jiang, Wei- Ma, and Zhouhui Lian. Controllable person image synthesis with atcibuute-ccoomposed gan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5084-5093, 2020.
[26] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411..11 2014.
[27] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4296-4304, 2024.
[28] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the JEFF/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[29] Yurui Ren, Xiaoqing Fan, Ge Li, Shan Liu, and Thomas H Li. Neural texture extraction and distribution for controllable person image synthesis. In CVPR, 2022.
[30] Yurui Ren. Xiaoming Yu, Junming Chen, Thomas H Li, and Ge Li. Deep image spatial transformation for person image generation. In CVPR, 2020.
[31] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2022.

---

[32] Fei Shen, Xiaoyu Du, Liyan Zhang, and Jinhui Tang. Triplet contrastive learning for unsupervised vehicle re-identification. arXiv preprint arXiv:2301 09498, 2023.
[33] Fei Shen, Xin Jiang, Xin He, Hu Ye, Cong Wang. Xiaoyu Du, Zechao Li, and Jinghui Tang. Imagdressing- vI: Customizable virtual dressing, arXiv preprint arXiv:2407. 1205,, 2024.
[34] Fei Shen, Xiangbo Shu, Xiaoyu Du, and Jinhui Tang. Pedestrian specifiic bipartite-aware similarity learning for text-based person retrieval. In Proceedings of the 31th ACM International Conference on Multimedia, 2023.
[35] Fei Shen, Hu Ye, Sibo Liu, Jun Zhang, Cong Wang, Xiao Han, and Wei Yang. Boosting consistency in story visualization with contecont conditional diffusion models. arXiv preprint arXiv:2407.02443 2024.
[36] Fei Shen, Hu Ye. Jun Zhang, Cong Wang. Xiao Han, and Wei Yang. Advancing pose-guided image synthesis with progressive conditional diffusion models. arXiv preprint arXiv:2310.06:1 2023.
[37] Aliaksandr Siarohin, Enver Sangineto, Stéphane Lathuiliere, and Nieu Sebe. Deformable gans for pose- based human image generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3408 3416, 2018.
[38] Jascha DicksDiicst,in,, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015,
[39] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv: Learn- ing.arXiv: Learning, Oct 2020.
[40] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13455 2020.
[41] Yang Song, Jascha Sohl- Dickstein, DiederikP. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poolc. Score-based generative modeling through stochastic differential equations. International Conference on Learning Reprexentationss International Conference on Learning Representations, May 2021.
[42] Hao Tang, Song Bai, Li Zhang, Philip HS Torr, and Nicu Scbe. Xinggan for person image generation. In Computer VisionFECCV 2020: 16th European Conference, Glaxgow, UK, August 23-28, 2020, Proceedings, Part XXV 16, pages 717-734. Springer, 2020.
[43] Shuyuan Tu, Qi Dai, Zhi-Qu Qi Cheng, Han Hu, Xintong Han, Zuxuan Wu, and Yu-Gang Jiang. Motioneditor: Editing video motion via content-aware diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7882-7891, 2024.
[44] Cong Wang, Kuan Tian. Yonghang Guan, Jun Zhang. Zhiwei Jiang, Fei Shen, Xiao Han, Qing Gu, and Wei Yang. Ensembling diffusion models via adaptive feature aggregation. arXiv preprint arXiv:2405. 17082, 2024.
[45] Cong Wang, Kuan Tian, Jun Zhang, Yonghang Guan, Feng Luo, Fci Shen, Zhiwei Jiang, Qing Gu, Xiao Han, and Wei Yang. V-express: Conditional dropout for progressive training of portrait video generation. arXiv preprint arXiv:2406. 02511, 2024.
[46] Zhou Wang, Alan Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):6006613 2004.
[47] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.0662.1 2023.
[48] Mang Ye, Jianhing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven CH Hoi. Deep learning for person re-identification: A survey and outlook. IEFE transactions on pattern analysis and machine intelligence, 44(6):2872-2897) 2021.
[49] Jinsong Zhang, Kun Li, Yu-Kun Lai, and Jingyu Yang. Pise: Person image synthesis and editing with decoupled gan. In CVPR, 2021.
[50] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to text-to-image -o-imaaee diffusion models. arXiv preprint arX2i:2302.05544 05543, 2023.
12

---

[51] Pengze Zhang, Lingxiao Yang, Jian-Huang Lai, and Xiaohua Xie. Exploring dual-task correlation for pose guided person image generation. In CVPR, 2022.
[52] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2865595,200 2018.
[53] Bo Zhao, Xiao Wu, Zhi-Qi Cheng, Hao Liu, Zequn Jie, and Jiashi Feng. Multi-view image generation from single-view. In Proceedings of the 26th ACM international conference on Multimedia, pages 383-391, 2018.
[54] Liang Zheng, Liyue Shen, Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re-identification: A benchmark. In ICCV, 2015.
[55] Xingran Zhou, Zhang, Ting Zhang, Pan Zhang, Jianmin Bao, Dong Chen, Zhongfei Zhang, and Fang Wen. Cocosnet v2: Full-resolution correspondence learning for image translation. In CVPR, 2021.
[56] Xinyue Zhou, Mingyu Yin, Xinyuan Chen, Li Sun, Changxin Gao, and Qingli Cross attention based style distribution for controllable person image synthesis. In European Conference on Computer Vision, pages 161-178. Springer, 2022.
[57] Zhen Zhu, Tengteng Huang. Baoguang Shi, Miao Yu, Bofei Wang, and Xiang Bai. Progressive pose attention transfer for person image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 2347-2356, 2019.
13

---

Supplementary Material
This supplementary material offers detailed exploration of the experiments and methodologies proposed in the main paper. Section A provides a series of symbols and definitions for enhanced comprehension. Section delves deeper into the implementation specifies of our experiments. Section C presents additional experimental outcomes, including a broader range of qualitative comparison examples with state-of-the-art methods, more results on different user scenarios, and detailed explanation of our user studies. Section D discusses potential societal harms and impacts.
A
Some Notations and Definitions
The notations and definitions used this paper are shown in Table 5.
Table 5: Some notations and definitions.
Notation To
Definition
Target image
6
Gaussian noise
Additional condition
t
Timestep
0
Diffusion model
w
Guidance scale
Tt
Noisy data É step
Fh
High-Level semantic feature source image
FL
Low-Level texture feature of source image Feature of FLC module
Fc
Fi
Feature of ILC module
Fp
Feature of joint point
B Implement Details
Table 6: Hyperparameters IMAGPose.
Hyperparameters
IMAGPose
Diffusion Steps
1000
Noise Schedule
Linear
Optimizer
AdamW
Weight Decay
0.01
Batch Size
4
Iterations
300k
Learning Rate
5e-
Base Model
Stable Diffusion V1.5 Dinov2-G/14
Image Encoder
As shown Table 6, our experiments are conducted on 8 NVIDIA V100 GPUs. We use the pre- trained Stable Diffusion V1.5 and modify the first convolutional layer to accommodate additional conditions. Dinov2-G/14 is used as the image encoder. In the tokenizer layer, we set both the kernel size and stride of the 2D convolution 16, and the dimensions of the input and output channels set to 4 and 768, respectively. We introduce pose encoder identical to ControlNet, injected after the first convolutional layer, handle the pose condition. Our model is trained on the DeepFashion dataset, following previous studies, with image sizes of 256x 76 and 512x352. For the Market-1501 dataset, we use images of size 128x64. As part of our masking strategy, we default to randomly occluding 1-4 images. The model is trained for 300k steps using the AdamW optimizer with learning rate 5e-5. We use a batch size of and apply linear noise schedule of 1000 time steps. During the inference stage, we DDIM sampler with 20 steps and set the guidance scale to 2.0.
14

---

C
Additional Results
C.1
More Qualitative Comparisons for IMAGPose
We provide additional examples for comparison with the state-of-the-art (SOTA) methods in Figure 10.
Target Pose
Source Image
GT
PoCoLD
CFLD
DC
Ours
Figure More qualitative comparisons between IMAGPose SOTA methods on the DeepFashion dataset.
15

---

Denoising UNet
Denoising UNet
Targel Pose
Target Poses
Noise Lalent
Noise atent
Image Encoder
Image Encoder
Source Image
Source Image
(a) Common Framework: Generating target image from single ce image single target pose.
M1 Generating multiple target images with different poses simultaneously
Figure 1I: (a) The schematic diagram of the common frameworks based existing diffusion models only support generating target image from a single source image and single target pose. (b) During the development of IMAGPose, we devised a proprietary model to address the scenarios of generating multiple target images with different poses simultaneously.
Table 7: Comparison results between different architectures on DeepFashion.
Methods
SSIM (f)
LPIPS (U)
FID (J)
PIDM [1]
0.7312
0.1678
6.3671
M1
0,7145
0.1743
6.9406
Ours
0.7561
0.1284
5,8738
C.2 More Discussion Different User Scenarios
As discussed in the main text, previous pose pose-guided image generation methods only support scenarios where a target image is generated from single source image and target pose, as shown in Figure 11 (a). However, in practical user scenarios, target images with different poses, as shown in Figure 1. To address these scenarios, we develop a proprietary models M1 to generate multiple target images with different poses simultaneously during the development of IMAGPose, as shown in Figure 1 (bb... Similar to [2, 10]. M1 concatenates different noise latents along the batch dimension. Then, we chose the PIDM [1] based on diffusion model as the representative for generating target image from single source image and target pose. MI model are also included in our additional comparisons, as shown Table 7.
also common and important generate multiple
The results in Table indicate that while we can address the generation of target images from multiple target poses by training dedicated model MI, this approach increases the training burden and introduces complexity for users. Users must continually switch between models to meet their specific needs when generating target images. Moreover, from performance metric perspective, the dedicated model M1 does not offer any advantages, as it only injects semantic information from the source image using frozen image encoder. Most importantly, none of the current frameworks support the task of generating target images through multiple image prompts. In contrast, our proposed ILC module introduces a masking strategy to unify target image generation under different user scenarios. As shown in Figure 2 (b), we also incorporate the masking strategy to inject image-level conditions from the source image, and enhance the texture features of the VAE to achieve richer context information and alignment between the image and pose.
As previously mentioned, once IMAGPose is trained, it can accommodate different user scenarios. For generating target image from a single source image and target pose, IMAGPose only needs to replicate the target pose multiple times the joint pose during inference. For generating multiple target images from single source image and multiple target poses at once, IMAGPose only needs to combine different poses the joint pose during inference. For generating a target image from multiple source images and single target pose, IMAGPose only needs to combine different source images the joint image during inference. Therefore, IMAGPose has the ability to unify pose-guided image generation under different user scenarios.
16

---

C.3 User Study
To validate the disparity between the generated images and the real ones, and evaluate the superiority of our method compared to existing technologies, we conducted a user study involving 50 volunteers. (1) For the R2G and G2R metrics, we selected 40 test samples random and used each model to generate a corresponding set of images. The volunteers were tasked with distinguishing between the 40 generated images and 40 real images. (2) For the Jab metric, we randomly selected 40 pairs of source images and target poses, and each model was tested to generate a corresponding set of images. The users were then asked to select the highest quality and most faithful image from the side-by-side views. The user study allowed us to evaluate which model produced the most realistic results based on human perception. Example questions are illustrated 13.
D Ethics Statement
In this paper, we propose IMAGPose, unified conditional framework designed cater to various user scenarios for generating different human images. However, it is well-known that virtually all methods of human image synthesis, including ours, can be misused by malicious actors create false content and disseminate misinformation. This is a significant concern that we fully acknowledge. Nevertheless, substantial progress has been made in the field of detecting and preventing such malicious tampering. Our work aims to provide valuable support for research and external audits in this area, helping to strike a balance between the value of our technology and the risks associated with unrestricted access. This ensures that our technology can be used safely and beneficially.
17

---

(Defull
IMAGP
Figure 12: More visual comparison of our model's uniformity across different user scenarios.
18

---

23. Please indicate whether the following image is real or fake.
Real
Fake
Figure 13; An example question used in our user study for pose-guided person image synthesis.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly articulate that existing works have over- looked the scenarios of pose-guided image generation. To address this issue, IMAGPose is proposed. Extensive experiments have validated the effectiveness of IMAGPose.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: the conclusion section, we have included a discussion of the limitations and potential directions for future work.
3. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [ess]
Justification: We have detailed the hyperparameters and experimental settings in the main text and the appendix of the paper.
4. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification: Our code and models will be open-sourced upon acceptance.
5. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper specifies all the training and test details, including data splits, hyperparameters, how they were chosen, and the type of optimizer used, necessary to understand the results.
Justification:
6. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The results reported in the paper are reproducible.
7. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provides sufficient information on the computer resources
8. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses both potential positive societal impacts and negative societal impacts of the work performed.
9. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: The paper describes safeguards for responsibly releasing high-risk data or models, such as pretrained language models or image generators.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Proper credit is given to the creators original owners of assets used in the paper, and the license and terms use are explicitly mentioned and respected.
11. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Upon acceptance, documentation for the code, data, and logs will be provided alongside the assets.
12. Crowdsourcing and Research with Human Subjects
20

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: The details of the user study, including the full text instructions given to participants and any relevant screenshots, are provided in the appendix.
13. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [Yes]
The user study adheres to relevant national laws and regulations regarding participant safety and ethics. Risks incurred by study participants, as well any necessary disclosures, are described.
Justification:
21


Reviews:
[{"review_id": "rIuMfnAsnQ", "reviewer": "Reviewer_1dhK", "summary": "The paper introduces IMAGPose, a unified conditional framework designed to overcome the limitations of existing diffusion models in pose-guided person image generation. Traditional models primarily focus on generating a target image from a single source image and a target pose. IMAGPose extends this capability by addressing two additional scenarios: generating multiple target images with different poses simultaneously and generating target images from multi-view source images. The proposed framework incorporates three key modules: \n\n- Feature-Level Conditioning (FLC): Combines low-level texture features from a VAE encoder with high-level semantic features from an image encoder.\n- Image-Level Conditioning (ILC): Aligns images and poses through a masking strategy, supporting variable numbers of source images and poses.\n- Cross-View Attention (CVA): Ensures local fidelity and global consistency by decomposing global and local cross-attention.\n\nExtensive experiments demonstrate the framework's ability to produce consistent and photorealistic images under various challenging user scenarios.", "questions": "- Can the authors provide more details on the computational requirements for training and inference using IMAGPose? Specifically, what are the resource constraints, and how do they impact the practical usability of the framework?\n\n- How does the framework perform on datasets outside of DeepFashion and Market-1501? Are there plans to test IMAGPose on more diverse datasets to evaluate its generalization capabilities?\n\n- Can the authors elaborate on the potential impact of using fixed encoders (VAE and image encoders) on the quality and flexibility of the generated images? Are there scenarios where fine-tuning these encoders could be beneficial?\n\n- The paper mentions that IMAGPose currently supports generating up to four images simultaneously. Does this indicate computational resource constraints? How might these constraints affect large-scale image generation tasks, particularly for high-resolution or complex scenes?\n\n- IMAGPose introduces three core modules (FLC, ILC, CVA), each adding to the model's complexity. How does this complexity impact the training and inference process, and does it increase implementation and debugging difficulty?\n\n - The paper uses frozen VAE and image encoders for feature extraction. While this reduces training time, are these encoders fully suitable for the specific task demands? Could this approach limit the quality and diversity of generated images?\n\n- Although IMAGPose aims to unify different user scenarios, are there specific or special requirements that still need additional adjustments or extensions? Could this general approach introduce limitations in some applications?", "limitations": "- The paper could discuss potential biases in the generated images and how the framework handles diverse demographic attributes.\n\n- A discussion on the limitations related to real-time applications and potential solutions to mitigate computational overhead would be beneficial.\n\n- While the societal impact of misuse is mentioned, a more detailed discussion on the ethical implications and safeguards for responsible usage could strengthen the paper.\n\n- The following related work is recommended for citation & discussion:\n\nZhao, B., Wu, X., Cheng, Z.-Q., Liu, H., Jie, Z., & Feng, J. (2018). Multi-view image generation from a single-view. In Proceedings of the 26th ACM International Conference on Multimedia (pp. 383-391).\n\nHuang, S., Xiong, H., Cheng, Z.-Q., Wang, Q., Zhou, X., Wen, B., Huan, J., & Dou, D. (2020). Generating person images with appearance-aware pose stylizer. In Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI 2020).\n\nLiu, H., He, J.-Y., Cheng, Z.-Q., Xiang, W., Yang, Q., Chai, W., Wang, G., Bao, X., Luo, B., & Geng, Y. (2023). Posynda: Multi-hypothesis pose synthesis domain adaptation for robust 3D human pose estimation. In Proceedings of the 31st ACM International Conference on Multimedia (pp. 5542-5551).\n\nTu, S., Dai, Q., Cheng, Z.-Q., Hu, H., Han, X., Wu, Z., & Jiang, Y.-G. (2024). MotionEditor: Editing video motion via content-aware diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7882-7891).", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper introduces a unified framework that extends the capabilities of existing diffusion models, addressing important user scenarios that were previously overlooked.\n\n- The experimental results are robust, demonstrating the effectiveness of the proposed framework across multiple datasets.", "weaknesses": "- The framework's complexity, with multiple conditioning modules and attention mechanisms, may pose challenges for real-time applications and require significant computational resources.\n\n- While the framework shows promising results on specific datasets, it would be beneficial to see its performance on a broader range of datasets and in more diverse scenarios.\n\n- The use of frozen VAE and image encoders may limit the adaptability of the model to specific tasks, potentially impacting the quality of the generated images.\n\n- The paper could benefit from more detailed ablation studies to better understand the contribution of each module within the framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear  Reviewer 1dhK:\n\nThank you very much for your support and constructive suggestions.We are glad to see the positive assessment of our paper and appreciate the detailed feedback.\n\n**Q1&Q4&Q7: (1) Computational Requirements and Resource Constraints？(2) Impact on Practical Usability，such as high-resolution or complex scenes. (3) Special Requirements and Potential Limitations**\n\n**Response:** Thank you for your valuable feedback on our paper. (1) Based on your suggestion, we have provided detailed information regarding the computational requirements and memory overhead for training and inference.\n\n| Training Memory (G) | Training Time (H) | Testing Memory (G)  | Inference Time (s) |\n|----------|----------|----------|----------|\n| 28.3 x 8 GPUs| 45.3 | 14.8 x 1 GPU | 1.236|\n\nAdditionally, the resource constraints refer to our use of V100 GPUs. Users may encounter memory constraints if they wish to generate more than four images at once. However, we can address this by employing an autoregressive approach, allowing users to create more than four images sequentially.\n(2) For practical usability, IMAGPose is flexible in generating images for complex scenes. However, GPU memory limitations indeed affect high-resolution image generation since we use SD as the base model. This is a common issue for all SD-based models.\nFurthermore, recent advancements such as FouriScale and DiffuseHigh have effectively addressed this problem, allowing images generated by SD-based models to be easily upscaled to high resolutions. Additionally, IMAGPose can utilize other excellent base models, such as PixArt-α, to avoid memory and resolution constraints. However, we used SD as the base model for fair comparison purposes since most SOTA methods are based on SD.\n(3) This question is fascinating and essential. As you mentioned, there are specific scenarios (such as when generating images based on a single source image and a single target pose) where IMAGPose requires special settings. Specifically, IMAGPose needs to replicate the source image or the target pose three times to meet the model's input requirements. However, this hardly incurs any additional memory consumption or speed overhead. In conclusion, IMAGPose is not limited by different application scenarios.\n\n**Q2:  Results on datasets outside of DeepFashion and Market-1501**\n\n**Response:** Please refer to the **shared response** regarding the \"Results on Out-of-Domain Datasets.\" We have included additional visualization results on more diverse datasets.\n\n**Q3&Q6: Do frozen encoders and VAE limit the quality and diversity of generated images, and could fine-tuning these encoders be beneficial in certain scenarios?**\n\n**Response:** We genuinely appreciate this question and are keen to discuss it further. Using frozen encoders and VAE does have its trade-offs. While it significantly reduces training time and computational resources, it can potentially limit the quality and diversity of the generated images. Frozen encoders, trained on general datasets, might not capture task-specific nuances as effectively as fine-tuned encoders.\nFine-tuning these encoders could indeed be beneficial in certain scenarios. For instance, fine-tuning could enhance the model's ability to capture specific texture details and improve overall image fidelity in applications requiring high precision and detail, such as fashion or face. This approach would allow the model to better adapt to the unique characteristics of the target domain, thereby improving the quality and diversity of the generated images.\nHowever, it is essential to note that fine-tuning comes with increased computational costs and requires more extensive datasets. In our case, we propose the FLC module, which combines features from both CLIP and VAE, to mitigate some of these limitations while maintaining efficiency. Therefore, IMAGPose adopts this balanced and acceptable design.\n\n**Q5:  How does this complexity impact the training and inference process, and does it increase implementation and debugging difficulty?**\n\n**Response:** In fact, we employ a straightforward and effective two-stage training strategy. In the first stage, we train the diffusion model incorporating the FLC and ILC modules. In the second stage, we only train the CVA module. IMAGPose operates as an end-to-end model during inference, minimizing potential debugging difficulties. More importantly, upon acceptance of the paper, all training and testing codes and checkpoints will be made available. \n\n**L1&L2&L3: (1) Discuss potential biases and how to handle it? (2) Real-time application limitations and how to mitigate computational? (3) Safeguards for Responsible Usage**\n\n**Response:** Thank you for your thoughtful reminder. Based on your suggestion, we have added the following discussions. （1） The performance of the model is subpar for generating cartoon characters and non-photorealistic styles because our training data consists of photorealistic human images for fair comparison. In future work, we plan to include a broader range of data and design style transformation modules to overcome this bias. （2）To address the limitations of real-time applications, we propose several potential solutions. Model Optimization (such as model pruning, quantization, and distillation) and asynchronous processing (Implementing asynchronous processing pipelines where image generation is precomputed or processed in parallel). （3）To mitigate ethical risks, we have added several safeguards for responsible usage, including transparency and disclosure, usage policies, collaboration with regulators, and detection tools.\n\n\n**L4: The following related work is recommended for citation & discussion:**\n\n**Response:** Thank you for recommending the relevant work. Following your suggestions, we have referenced and discussed the literature you listed. Due to space limitations, we will provide specific references and discussions during the stage of our discussion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T8ptbQl0OU", "reviewer": "Reviewer_QBtt", "summary": "This paper thoroughly analyzes and considers the application scenarios of pose-guided person image synthesis from the perspective of real-world significance.  Author introduces previously unconsidered but intriguing scenarios and proposes the IMAGPose framework to unify different tasks.  Comprehensive experiments, ablation studies, and user research conducte to demonstrate the effectiveness of the proposed method.", "questions": "- The design of FLC is interesting, as it uses almost lossless features from VAE as conditions. However, for pose-guided image generation, would it be better to add a text caption condition? For example, T2I-Adapter incorporates both text and image features through a decoupled cross-attention mechanism. How does this differ from author concatenation approach?\n\n- In ILC,  mask strategy is a critical component for unifying different conditional generations. But I am slightly confused about the purpose of this binary mask. What are its benefits/aim?\n\n- I noticed that if multiple poses are given at once and these poses are very continuous, would this be similar to generating a video based on continuous poses, such as a dance sequence? In such a user scenario, what are the comparisons and advantages of IMAGPose?\n\n- Is there a trade-off between speed and efficiency? For example, if a user only wants to generate a target image based on a single pose, how should they proceed, and how should they make their choice?\n\n- In Figure 8, what does IMAGPose* refer to? \n\n- What is the guidance scale for CFG?\n\n- In Figure 9, does T1 (Default) refer to IMAGPose?\n\n---\nAfter the rebuttal,  I raised my score from **6 to 8** because this work provides a new perspective on traditional tasks (a novel user angle) and unifies the framework. The authors' efforts are commendable, and I believe this work deserves acceptance.", "limitations": "The authors addressed the limitations and the work does not have negative societal impact.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The proposed task scenarios are highly insightful, with a clear and well-motivated approach.\n\n- The study's contribution of a unified conditional diffusion model to address different task scenarios is of significant value.\n\n- The evaluation is comprehensive, and the proposed method generally demonstrates superior performance compared to existing works, supported by user studies and clear visualizations.\n\n- The impact of different components is analyzed through ablation studies, further proving the effectiveness of the proposed technique and providing a better understanding of the method.", "weaknesses": "The IMAGPose framework heavily relies on the detection results from OpenPose. I am curious about how the performance of IMAGPose would be affected if OpenPose produces poor results. Do the authors have any solutions to mitigate this issue?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer QBtt:\n\nThank you for your review and insightful comments. We address your questions as follows.\n\n**Q1: The IMAGPose framework heavily relies on the detection results from\nOpenPose. I am curious about how the performance of IMAGPose would be\naffected if OpenPose produces poor results. Do the authors have any\nsolutions to mitigate this issue?**\n\n**Response:** Thank you for your thoughtful suggestions. I am concerned that there may be a misunderstanding regarding our reliance on OpenPose detection results. As a crucial condition for guided image generation, Pose directly influences the outcomes, as you have described. We use the same OpenPose as all existing SOTA methods for a fair comparison. In other words, errors in OpenPose results affect our outcomes and other SOTA methods similarly. To mitigate inaccuracies in generated results caused by imprecise pose estimation, we plan to incorporate 3D prior information, such as depth maps and normals, in the future.\n\n**Q2: The design of FLC is interesting, as it uses almost lossless features\nfrom VAE as conditions. However, for pose-guided image generation, would\nit be better to add a text caption condition? For example, T2I-Adapter\nincorporates both text and image features through a decoupled\ncross-attention mechanism. How does this differ from author\nconcatenation approach?**\n\n**Response:** Thank you for your recognition and praise. We believe it would be beneficial to add some textual captions to the existing task; however, it should be noted that this would constitute an entirely new task. In pose-guided person generation, the only permissible conditions are pose and reference images of the person to ensure a fair comparison. Moreover, the T2I-Adapter introduces conditions through a parallel decoupling text and images. In contrast, IMAGPose employs a serial approach and optimizes global and local image processing to ensure local fidelity and global consistency of the personal images.\n\n**Q3: In ILC,  mask strategy is a critical component for unifying different\nconditional generations. But I am slightly confused about the purpose of\nthis binary mask. What are its benefits/aim?**\n\n**Response:** We appreciate the reviewer pointing out this issue. In fact, we use binary masks to mark and distinguish the areas to be generated. Black pixels in the source image can easily mislead the model into identifying these areas needing generation. To address this, we employ a binary marking approach using all-zero pixels, which helps to clearly differentiate the areas to be generated, reducing model confusion and ensuring accurate area generation.\n\n**Q4: I noticed that if multiple poses are given at once and these poses are\nvery continuous, would this be similar to generating a video based on\ncontinuous poses, such as a dance sequence? In such a user scenario,\nwhat are the comparisons and advantages of IMAGPose?**\n\n**Response:** Please refer to the **shared response** \"Differences with technologies like Animate Anyone.\" We have added and discussed these differences.\n\n**Q5: Is there a trade-off between speed and efficiency? For example, if a\nuser only wants to generate a target image based on a single pose, how\nshould they proceed, and how should they make their choice?**\n\n**Response:** Thank you for your insightful question. IMAGPose achieves a good balance between speed and efficiency. We have added results of speed and SSIM, showing that IMAGPose processes nearly 8 times faster than PIDM and about 3 times faster than PoCoLD and CFLD, while demonstrating superior quantitative results regarding generation quality. Moreover, suppose users must quickly generate target images based on a single pose. In that case, we recommend using the IMAGPose* setting, which involves multiple replications of the target pose. If users prioritize higher generation quality, we suggest using the default IMAGPose setting, which entails multiple replications of the source image.\n\nMethods| Speed (s) | SSIM|\n|----------|----------|----------|\n|PIDM| 9.377| 0.7312| \n|PoCoLD| 4.762| 0.7310 | \n|CFLD|3.764| 0.7378 |\n|**Ours**| **1.236**|**0.7561**|\n\n**Q6-Q8: In Figure 8, what does IMAGPose\\* refer to? What is the guidance scale for CFG? In Figure 9, does T1 (Default) refer to IMAGPose?**\n\n**Response:** We apologize for any confusion caused. In fact, once we have trained the IMAGPose model, we can use different testing configurations. For instance, when we have one source image and one target pose, we can replicate the source image three times, resulting in three identical source images and one target pose, the default IMAGPose setting. Alternatively, we can replicate the target pose three times, resulting in one source image and three identical target poses, which we refer to as IMAGPose*. Besides, the default value for CFG is set to 2.0. In Figure 9, T1 (default) refers to the default setting of IMAGPose, which involves replicating three source images.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tJT8aea54U", "reviewer": "Reviewer_rV96", "summary": "This paper considers three different pose-guided image generation scenarios from a scene perspective and attempts to cover all scenarios using a unified framework. In my opinion, it is very insightful and inspiring. The proposed IMAGPose framework unifies all scenarios through several ingenious components, namely FLC, ILC, and CVA. The qualitative and quantitative evaluations, as well as the user study, demonstrate its strong competitiveness.", "questions": "(1) The authors did not specify what type of pose encoder is used. Could you provide more details?\n\n(2) It is commendable that the authors have evaluated the speed for different SOTA models. However, I am curious about the training duration.\n\n[minor] Appendix Figure 17 needs some descriptive text. Although it is understandable based on the main manuscript, a good appendix should also have comprehensive captions.", "limitations": "The authors have already discussed limitations and societal impact in the conclusion and appendix.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "(1) Existing pose-guided image generation has only considered scenarios involving a single source image and a single target pose. However, this paper insightfully introduces two additional potential scenarios: providing multiple target poses at once and multiple source images at once. These assumptions are both reasonable and necessary.\n\n(2) The proposed IMAGPose framework addresses the needs of different scenarios through simple yet ingenious designs. For example, it directly uses VAE features to capture underlying texture details, learns image conditions implicitly by taking the image as input, and employs the CVA module's innovative setting of merging, splitting, and then merging again, which is particularly enlightening.\n\n(3) The figures and tables are well-organized and clearly presented. The experiments are comprehensive, the results are convincing, and the appendix provides detailed supplementary information.", "weaknesses": "(1) The Reference UNet [1]  also uses features from both the VAE and image encoder. It would be beneficial if the authors could discuss the differences and advantages of IMAGPose in comparison to this approach.\n\n(2) In line 152, does the term \"image encoder\" refer to CLIP? Why not use the image encoder features from DinoV2? \n\n[minor] In line 184, \"IFC\" should be corrected to \"ILC\".\n\n[1] Li Hu et al. Animate anyone: Consistent and controllable image-to-video synthesis for character animation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer rV96:\n\nWe thank the reviewer for their detailed feedback and encouraging comments.\n\n**Q1: Comparison with Reference UNet [1]**\n\n**Response:** Please refer to the **shared response** on \"Differences with technologies like Animate Anyone.\" We have added and discussed these differences.\n\n**Q2:In line 152, does the term \"image encoder\" refer to CLIP? Why not use the image encoder features from DinoV2?**\n\n**Response:** In fact, the default image encoder is set to Dinov2-G/14.  We have added the results of IMAGPose using various image encoders, and it is evident that Dinov2-G/14 performs better across all metrics.\n\n|Image Encoder| SSIM (↑)| LPIPS (↓) | FID (↓)|\n|----------|----------|----------|----------|\n| CLIP-B/14| 0.7516 | 0.1364 | 6.1342|\n| CLIP-bigG/14| 0.7548| 0.1331 | 5.9645|\n| CLIP-H/14| 0.7552| 0.1296 | 6.0231|\n| Dinov2-B/14| 0.7541| 0.1343 |5.9286|\n| Dinov2-L/14| 0.7556 | 0.1323 | 5.9432|\n| Dinov2-G/14|0.7561 | 0.1284 | 5.8738|\n\n\n**Q3: The authors did not specify what type of pose encoder is used. Could you provide more details?**\n\n**Response:** We apologize for any confusion caused. In fact, the pose encoder is implemented with several lightweight convolutional layers, similar to ControlNet. It is injected after the first convolutional layer of the Denoise UNet.\n\n**Q4:  Training Time.**\n\n**Response:** About 45.3 (H) on 8 V100 GPUs.\n\n**[m1]: In line 184, \"IFC\" should be corrected to \"ILC\".**\n\n**Response:** We apologize for the error in the manuscript here and\ngreatly appreciate the reviewer's patient reading. Furthermore, we have made every effort to check the revision thoroughly.\n\n**[m2]: Appendix Figure 17 needs some descriptive text. Although it is understandable based on the main manuscript, a good appendix should also have comprehensive captions.**\n\n**Response:** Thank you for your valuable feedback. We agree that providing comprehensive captions in the appendix can enhance readability and independent understanding. Following your suggestion, we will add detailed captions to the appendix in the revised manuscript. Thank you once again for your guidance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0HaWzJLvT8", "reviewer": "Reviewer_Cm6N", "summary": "This paper proposes a diffusion-based pose-guided image generation method. Specifically, given an image and a sequence of poses, this paper aims to generate images that follow the input poses and maintain the appearance of the input image. To capture the texture details of the input image, they propose to combine features from VAE and a pre-trained image encoder. They design an Image-level conditioning module, which combines four images into a joint representation.  The images that need to be predicted are masked. By doing this, their method can generate 1-3 images in one forward step, or use 1-3 condition images in the same time.", "questions": "1. Since this paper forms four images into a joint feature map, the self-attention in SD can conduct feature transformation among each images. Why do you split the joint feature map and conduct temporal attention among them (Sec 3.4)?\n2. This paper evaluates their method solely on person images with clean background. How does this method perform on images with complex, meaningful backgrounds.", "limitations": "Yes, they have provided discussions about limitations and negative social impact.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. They propose to combine VAE and image encoder features as the condition, which helps the model to generate more details images.\n2. Their Image-Level Conditioning design enables their models to condition on the dynamic number of input images.", "weaknesses": "1. Limited novelty. 1） The main contribution of this paper is forming four image latents into a 4-grid join latent, which is very similar to [1]. 2） The design of Sec 3.2 combines the VAE feature with the feature from Dino, which is similar to Animate Anyone[2], which uses CLIP image features as the condition.\n2. Lack of thorough comparison methods. Given an RGB person image $\\alpha$ and a pose image $\\beta$, this paper aims to generate a person that follows the pose image  $\\beta$ with the appearance of the $\\alpha$, which can also be called pose transfer. Thus, this paper should also compare their results with pose-guided human pose transfer methods: Animate Anyone [2]. DreamPose [3], Follow your pose [4] …. Besides, this paper injects pose guidance in ControlNet way, they should also compare their performance with ControlNet.\n3. Incorrect statements. In line 44-50, they claim that former diffusion-based generation using frozen image encoder pre-trained on general image dataset (not specific for person image), so these methods cannot extract texture detail information. However, Animate Anyone [2] trains a reference appearance net on large-scale human datasets, which can effectively encode detailed texture features.\n\n[1] Kara, Ozgur, et al. \"Rave: Randomized noise shuffling for fast and consistent video editing with diffusion models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n[2] Hu, Li, et al. \"Animate anyone: Consistent and controllable image-to-video synthesis for character animation.\" arXiv preprint arXiv:2311.17117 (2023).\n[3] Karras, Johanna, et al. \"Dreampose: Fashion image-to-video synthesis via stable diffusion.\" 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023.\n[4] Ma, Yue, et al. \"Follow your pose: Pose-guided text-to-video generation using pose-free videos.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 5. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Cm6N\n\nWe thank the reviewer for the positive feedback and valuable comments.\n\n**Q1: Limited novelty. 1） The main contribution of this paper is forming four image latents into a 4-grid join latent, which is very similar to [1]. 2） The design of Sec 3.2 combines the VAE feature with the feature from Dino, which is similar to Animate Anyone[2], which uses CLIP image features as the condition.**\n\n**Response:** (1) We sincerely disagree with your viewpoint. First, our method is not merely a simple 4-grid join latent. Our primary innovation involves first merging all target images into a unified target, followed by a carefully designed masking strategy that randomly masks these images into a joint mask, an approach not mentioned in reference [1]. Furthermore, our proposed ILC module aims to adapt to various user scenarios by randomly injecting different source image conditions and incorporating source image-level conditions, which are entirely different from the video editing tasks, motivations, and methods described in reference [1]. More importantly, IMAGPose introduces two new task settings and attempts to unify them within a single framework, achieving competitive results in each setting.\n(2) Please refer to the **shared response** on \"Differences with technologies like Animate Anyone.\" We have added and discussed these differences.\n\n**Q2: Lack of thorough comparison methods. Given an RGB person image  and a pose image , this paper aims to generate a person that follows the pose image   with the appearance of the ,\nwhich can also be called pose transfer. Thus, this paper should also\ncompare their results with pose-guided human pose transfer methods:\nAnimate Anyone [2]. DreamPose [3], Follow your pose [4] …. Besides, this\npaper injects pose guidance in ControlNet way, they should also compare\ntheir performance with ControlNet.**\n\n**Response:** Thank you for your kind reminder. Following your suggestion,  as shown in **Figure 2 of the submitted PDF file**, we have added comparisons with Animate Anyone [2], DreamPose [3],  and ControlNet [5] on deepfashion dataset. It is important to note that we cannot directly compare our method with Follow Your Pose [4], as the input conditions for Follow Your Pose [4] are text and pose, and it does not support guided by image references. Quantitative and qualitative results demonstrate that IMAGPose achieves highly competitive performance. This is because references [2,3,5] focus on continuous pose-guided generation; our approach is specifically designed for scenarios involving non-continuous poses. Moreover, we wish to highlight that IMAGPose explores and unifies image generation across different user scenarios, including Scenario (1), generating one target image from one source image and one target pose; Scenario (2), generating one target image from multiple source images and one target pose; and Scenario (3), generating multiple target images from one source image and multiple target poses. However, references [2,3,5] are optimized and designed for a single scenario task, lacking the capability for unification and multitasking. For example, references [2-3] only support Scenario (3), and ControlNet [5] only supports Scenario (1). Neither accommodates the user's need to generate from multiple source images.\n\n|Methods| SSIM (↑)| LPIPS (↓) | FID (↓)|\n|----------|----------|----------|----------|\n| ControlNet| 0.6725| 0.2443 |15.8762|\n|DreamPose | 0.7161| 0.1694 | 8.2510|\n| Animate Anyone| 0.7386 | 0.1319 | 6.8024|\n|**Ours**|**0.7561** | **0.1284** | **5.8738**|\n\n**Q3: Incorrect statements. In line 44-50, they claim that former diffusion-based generation using frozen image encoder pre-trained on general image dataset (not specific for person image), so these methods cannot extract texture detail information. However, Animate Anyone [2] trains a reference appearance net on large-scale human datasets, which can effectively encode detailed texture features.**\n\n**Response:** We wholeheartedly agree with your perspective. As you mentioned, enhancing the model's ability to encode texture features [2] involves integrating an additional reference network and a large dataset. In other words, without these additions, it becomes relatively challenging. Following your suggestion, we have added a constraint requiring no additional modules and datasets. Thank you once again for your meticulous guidance.\n\n**Q4: Since this paper forms four images into a joint feature map, the self-attention in SD can conduct feature transformation among each images. Why do you split the joint feature map and conduct temporal attention among them (Sec 3.4)?**\n\n**Response:** Thank you for your insightful question. We split the joint feature map and apply temporal attention for the following reasons: (1) Splitting allows for a more detailed exploration of the temporal context of each image, helping to capture subtle changes. (2) This approach enhances the model’s ability to track and integrate dynamics across varying poses and scenes, which is crucial for accurately modeling scenarios involving extensive pose transitions or scene changes. (3) Joint features can mask or dilute individual image characteristics. Splitting the feature map allows for more targeted feature transformations, maintaining the integrity of each individual image.\n\n**Q5: This paper evaluates their method solely on person images with clean\nbackground. How does this method perform on images with complex,\nmeaningful backgrounds.**\n\n**Response:** Please refer to the **shared response** regarding the \"Results on Out-of-Domain Datasets.\" We have included additional visualization results on more diverse datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank the reviewers for their helpful feedback and insightful comments.

We are glad that the reviewers find our paper “ *highly insightful* ” (**QBtt**), “ *clear and well-motivated* ” (**QBtt**), and “ *simple yet ingenious* ” (**rV96**),  “ *enlightening* ” (**rV96**) and “ *well-organized and clearly* ” (**rV96**), ”*addressing important user scenarios that were previously overlooked* ”(**1dhK**).  Also, our experiments are considered “ *reasonable and necessary.* ” (**rV96**), “ *robust,* ” (**1dhK**), and “ *generate more details images* ” (**Cm6N**).

From the view of pose-guided person generation, reviewer **1dhK** remarks that our work “ *addressing important user scenarios that were previously overlooked* ” and “ *idemonstrating the effectiveness of the proposed framework across multiple datasets.* ”.  Reviewer **QBtt** mentioned that our work  “*address different task scenarios is of significant value.* ”  Then, reviewer **rV96** agrees that “ *t is very insightful and inspiring.* ”.  Finally,  Reviewer **QBtt** mentioned that “ *the proposed task scenarios are highly insightful, with a clear and well-motivated approach.“ *

We further thank the reviewers for their constructive feedback. We have uploaded a PDF file which includes figures  to address the reviewers’ feedback.
**Next, we discuss two common comments raised by the reviewers.**

**Q1: Results on Out-of-Domain Datasets.** (suggested by reviewer **1dhK** and reviewer **rV96**)

**Response:** We greatly appreciate the reviewer's insightful comments. We have demonstrated the visualization results of IMAGPose on out-of-domain data. We randomly selected some poses and characters from out-of-domain sources, with detailed results provided in PDF format. The outcomes show that our IMAGPose continues to deliver excellent results, producing high-quality and high-fidelity images.

**Q2: Differences with technologies like Animate Anyone.** (suggested by reviewer **QBtt**, reviewer **rV96**, and reviewer **Cm6N**)

**Response:** We sincerely appreciate this question and look forward to discussing it further. Although Animate Anyone also injects features from an image encoder and VAE, our proposed IMAGPose differs in several key aspects: **(1) Implementation method.** Animate Anyone copies weights from the main UNet and maintains the same network structure, sharing latent spaces with the main UNet to facilitate feature interaction. In contrast, IMAGPose uses the same UNet to process both source and target images, resulting in a more harmonious and unified approach. **(2) Training parameter.** Animate Anyone introduces an additional Reference UNet, nearly doubling the parameter volume compared to IMAGPose, significantly increasing training complexity. **(3) Task objectives.** Animate Anyone supports only single reference image-guided image generation, while our IMAGPose unifies three common task types through designed FLC, ILC, and CVA, and supports image generation from multiple source images.

For other questions raised by the reviewers, please see our response (with text and other new experiments) to individual questions below each review. We will incorporate all our responses and additional results in the final version of the manuscript.

**Finally, we deeply appreciate the reviewer's detailed comments and thank them for helping us improve our work. We value the reviewer's insights and are very open to further discussions during the rebuttal period or afterward to explore this direction more. If the reviewer has any additional questions, please let us know. We are committed to being responsive until the end of this rebuttal period.**

Meta-review: This paper presents a method for unifying three types of conditions in pose-guided person image generation using diffusion models (1 source image + 1 target pose,  multiple source images + 1 target pose, and 1 source image + multiple target poses).  On the one hand, all the of reviewers praised the fairly complete work and experimental results.  One the other hand, the reviewer Cm6N had concerns of the overall technical novelty. The reviewer 1dhK initially recommended borderline accept with many questions raised in the initial comments, but did not participate in the reviewer-author discussion. The reviewer 1dhK lowered the score with detailed comments during the final stage of reviewer-AC discussion, while both reviewers rV96 and QBtt thought that most of  1dhK's comments have been addressed and suggested that it might be unfair to authors.  The meta-reviewer appreciates all the reviewers' comments and discussions.  This meta-review concurs with the overall positive scores and recommend **acceptance**.

The authors are encouraged to carefully revise the paper based on the discussions.  The meta-reviewer further suggests the authors to address several questions as follows: 

- For application scenarios with a single source image and a single pose, the authors elaborated in the rebuttal to Reviewer 1dhK that "*Specifically, IMAGPose needs to replicate the source image or the target pose three times to meet the model's input requirements. However, this hardly incurs any additional memory consumption or speed overhead. In conclusion, IMAGPose is not limited by different application scenarios.*"   

  - In comparisons with the prior art, which one of the four synthesized images is used? For computationally fair comparisons, do we need to run other baselines multiple times?

  - During training, how many of the training batches are needed to do this replicated samples as to close the training and inference distribution shifts?

  - Since there are convolutional operations in the UNet denoiser, will this "mosaic" have artifacts along the boundary, even with the binary mask used?

- It seems that details of the human pilot study are missing in terms of the selection of the 50 volunteers. The authors mentioned that they have followed IRB protocol, but without details provided, which should be addressed in revision.
Decision: Accept (poster)