Title: IMAGPose: A Unified Conditional Framework for Pose-Guided Person Generation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Pose-Guided Person Image Generation

Diffusion Model

Consistency Generation
Abstract: Diffusion models represent a promising avenue for image generation, having demonstrated competitive performance in pose-guided person image generation. 
However, existing methods are limited to generating target images from a source image and a target pose, overlooking two critical user scenarios: generating multiple target images with different poses simultaneously and generating target images from multi-view source images.
To overcome these limitations, we propose IMAGPose, a unified conditional framework for pose-guided image generation, which incorporates three pivotal modules: a feature-level conditioning (FLC) module, an image-level conditioning (ILC) module, and a cross-view attention (CVA) module. 
Firstly, the FLC module combines the low-level texture feature from the VAE encoder with the high-level semantic feature from the image encoder, addressing the issue of missing detail information due to the absence of a dedicated person image feature extractor. 
Then, the ILC module achieves an alignment of images and poses to adapt to flexible and diverse user scenarios by injecting a variable number of source image conditions and introducing a masking strategy.
Finally, the CVA module introduces decomposing global and local cross-attention, ensuring local fidelity and global consistency of the person image when multiple source image prompts. 
The three modules of IMAGPose work together to unify the task of person image generation under various user scenarios.
Extensive experiment results demonstrate the consistency and photorealism of our proposed IMAGPose under challenging user scenarios. 
The code and model will be available at https://github.com/muzishen/IMAGPose.

IMAGPose: A Unified Conditional Framework for Pose-Guided Person Generation

Abstract

Diffusion models represent a promising avenue for image generation, having demonstrated competitive performance in pose-guided person image generation. However, existing methods are limited to generating target images from a source image and a target pose, overlooking two critical user scenarios: generating multiple target images with different poses simultaneously and generating target images from multi-view source images. To overcome these limitations, we propose IMAGPose, a unified conditional framework for pose-guided image generation, which incorporates three pivotal modules: a feature-level conditioning (FLC) module, an image-level conditioning (ILC) module, and a cross-view attention (CVA) module. Firstly, the FLC module combines the low-level texture feature from the VAE encoder with the high-level semantic feature from the image encoder, addressing the issue of missing detail information due to the absence of a dedicated person image feature extractor. Then, the ILC module achieves an alignment of images and poses to adapt to flexible and diverse user scenarios by injecting a variable number of source image conditions and introducing a masking strategy. Finally, the CVA module introduces decomposing global and local cross-attention, ensuring local fidelity and global consistency of the person image when multiple source image prompts. The three modules of IMAGPose work together to unify the task of person image generation under various user scenarios. Extensive experiment results demonstrate the consistency and photorealism of our proposed IMAGPose under challenging user scenarios. The code and model will be available at https://github.com/muzishen/IMAGPose.

1 Introduction

Pose-guided person generation [51, 29] aims to transform a source person image into a target person image under a specific pose while maintaining appearance consistency. It has many applications, including virtual reality, film production, and e-commerce. Besides, the generated images can be used to enhance the performance of downstream tasks, such as person re-identification [32, 54, 48, 34].

Early methods [4, 57, 42] usually are developed based on generative adversarial networks (GANs). However, GAN-based methods easily suffer from the instability of the min-max training objective and the difficulty of generating high-quality images in a single forward pass. Recently, methods based on diffusion models [1, 22, 11, 36] are becoming increasingly popular in the community. As a promising alternative to GAN for image generation, diffusion models utilize the source image and target pose as conditions. They generate the target image through a multi-step denoising process instead of completing it in a single step. So, diffusion models help better retain the input information. For example, PIDM [1] proposes a texture diffusion module that inserts features of the source image, extracted by a frozen encoder, into different stages of the diffusion

Despite the above methods based on diffusion models' impressive results compared to GAN-based approaches, existing methods still exhibit 2 shortcomings. (1) For methods based on diffusion models, they neglect underlying texture detail information. As these methods employ a frozen image encoder, e.g., CLIP, to extract appearance features from the source image, which is trained on a general image dataset, it is not a specialized feature extractor for person images and can only extract highlevel semantic features. (2) For existing methods, as shown in Figure 1, they overlook two critical user scenarios: generating multiple target images with different poses simultaneously and generating target images from multi-view source images. In practice, users often

+

Generate

A Source Image

A Target Pose

A Target Image

+

Generate

A Source Image

+

Generate

Multi-View Source Images

A Target Pose

A Target Image

Existing

Methods

Ours

need to generate multiple consistent target images based on different poses simultaneously, such as in portrait photography and clothing displays. In addition, users often upload multiple source images to provide rich details for generating a target image. However, the fixed architecture of existing methods limits the ability of users to make arbitrary modifications and extensions.

We are the first to explore pose-guided image generation tasks under different conditions and propose a unified conditional framework for synthesizing high-fidelity person images.

We present the FLC module, which addresses the issue of missing texture features due to the absence of a dedicated person image feature extractor.

We devise the ILC module to inject a variable number of source image conditions and introduce a masking strategy, achieving an alignment of images and poses.

We develop a CVA module to decompose global and local cross-attention, ensuring the joint image's local fidelity and global consistency when multiple-view source image prompts.

Extensive experiments on two challenging datasets show that proposed IMAGPose surpasses existing models. We also conduct a user study and applications for downstream tasks to comprehensively evaluate the advantages of IMAGPose.

2 Related Work

Based on GAN Methods. Early approaches [24, 25, 37, 57, 37, 49, 30] view the synthesis task as conditional image generation, utilizing conditional generative adversarial networks (CGANs) [26] to generate target images based on source appearance images and target poses. For example, PG 2 [24]

proposed a two-stage method that refines the generated images in an adversarial manner. To decouple pose and appearance information, VUNet [8] suggests learning pose-independent features to tackle the complex structure problem of pose space transformation. Then, ADGAN [25] introduced a character attribute decomposition module to explicitly parse the human body and blend and insert conditions. VariGANs [53] combine variational inference with GANs to generate multi-view images from a single image, refining results from coarse to fine [5, 7, 6]. However, GAN-based methods [24, 25, 37] often suffer from issues such as blurred texture details and instability of the min-max training objective. To address the issue of blurred texture details, Grigorev et al. [9] proposed a CNN-based framework that first performs pose warping, followed by texture repair. Unselfie [18] introduced a pipeline that first identifies the target's neutral pose, then repairs appearance textures, and finally perfects and synthesizes the character in the background. However, these multi-stage methods require the introduction of multiple additional models and costs, impacting the efficiency of generation.

Based on Diffusion Models. Diffusion models have found extensive applications in virtual dressing [33], story generation [35], and portrait animation [45]. To address the instability of GAN's min-max training, methods based on diffusion models [13, 44, 40, 38, 31, 28] are gradually becoming more popular in the community. The core idea is to start from a simple noise vector and gradually transform it into a high-quality image through multiple denoising iterations, making it suitable for high-fidelity and context-aware generation. Besides unconditional generation [14, 39, 41], various methods [50, 47, 27] have been introduced to incorporate user-provided control signals into the generation process, thereby achieving more controllable image generation. For example, PIDM [1] proposed a texture diffusion module that uses the source image features extracted by the frozen encoder as conditions to be inserted into different stages of UNet. CFLG [22] proposed a mixedgranularity attention module to inject multi-scale fine-grained character appearance features into the generation process. PoCoLD [11] further constrains the correspondence between person images and poses by introducing additional 3D Densepose [19] annotations of poses and source image appearance features extracted by the frozen encoder. Subsequently, PCDMs [36] proposed a three-stage diffusion model that predicts global features under the target pose, aligns images and poses, and then refines texture details. Besides, APS [16] progressively couples target pose and appearance for effective human image synthesis. PoSynDA [20] simulates 3D pose distributions to address the lack of 2D-3D correspondences, enhancing data diversity. MotionEditor [43] uses a dual-branch structure that decouples key-value queries, preserving the background and character appearance for efficient content editing. However, the frozen encoders introduced in these diffusion model-based methods are trained on generic image datasets and can only extract high-level semantic features of person, easily overlooking low-level texture details. Furthermore, in real scenarios, these models do not support generating multiple target images at once and target generation under multi-view image prompts.

3 Method

Figure 2 illustrates IMAGPose, a unified conditional framework that encompasses 3 core modules: feature-level conditioning (FLC) module, image-level conditioning (ILC) module, and cross-view attention (CV A) module. This framework aims to generate high-fidelity and high-quality target images in any user scenario. The FLC module combines low-level texture features from the VAE encoder with high-level semantic features from the image encoder, which addresses the issue of losing texture details due to lacking a dedicated person image feature extractor. (Section 3.2). The ILC module introduces a masking strategy for the target image and incorporates a variable number of source image conditions. This module aligns the image and pose, making it suitable for different user scenarios. (Section 3.3). The CVA module proposes the decomposition of global and local cross-attention. This ensures the entire person images' local fidelity and global consistency.(Section 3.4).

3.1 Preliminaries

Diffusion models belong to a class of generative models. And these models are trained to reverse a diffusion process, which systematically introduces Gaussian noise to the data through a fixed Markov chain over a series of timesteps, denoted as t . Concurrently, a denoising model is trained to generate samples starting from this Gaussian noise. The training objective of a diffusion model, denoted as ϵ θ and parameterized by θ , typically employs a mean square error loss L DM . For each timestep t , this loss is defined as follows,

·  ·  ·

·  ·  ·

Embeddings

Tokenizer

Layer

C

First   Conv

Masked  Latent

Binary Mask

VAE   Encoder

Noise Latent

+ Noise

Joint Mask

L

MSE

L

MSE

Target Images

Joint Target

Denoising UNet

Source Image

Source Image

Image Encoder

VAE Encoder

(a)

Feature-Level Conditioning

（

b

）

Image-Level Conditioning

Element-Wise Add

Concatenation

C

Element-Wise Add

Concatenation

C

Predict Noise

Cross-View Attention

Cross Attention

Spatial Attention

Target Poses

4

1

2

3

4

1

2

3

4

1

2

3

4

1

2

3

公

here, x 0 represents the original data, supplemented by a condition c . The timestep t of the diffusion process is represented by t ∈ [0 , T ] . The noisy data at the t step, denoted as x t , is defined as α t x 0 + σ t ϵ . In this context, α t and σ t are predefined functions of t that dictate the diffusion process.

Once the model ϵ θ is trained, images can be synthesized from random noise through an iterative process. During the sampling stage, the predicted noise is calculated based on the predictions of both the conditional model ϵ θ ( x t , c, t ) and the unconditional model ϵ θ ( x t , t ) , using classifier-free guidance [15] according to Eq. 2.

where w is the guidance scale used to adjust the condition c .

3.2 Feature-Level Conditioning Module

Existing methods typically depend on frozen encoders to extract appearance features from source images. However, these frozen encoders are trained on general image datasets, not explicitly designed for person image feature extraction. So, it can only capture high-level semantic features of the source image. To address this, as depicted in Figure 2(a), given that the frozen V AE can almost reconstruct images losslessly without training, we propose the FLC module, which simultaneously extracts low-level texture information from a V AE encoder [17] and high-level semantic information from an image encoder. Specifically, the FLC module consists of a frozen VAE encoder, an image encoder, and a trainable tokenizer layer. Assume that the source image x ∈ R c × h × w , where c , h , and w represent the channel, height, and width of the image, respectively. First, we extract the latent features F l ∈ R 4 × h/ 8 × w/ 8 of the source image from the frozen V AE encoder while simultaneously extracting the high-level semantic features F H ∈ R n × d of the person image from the frozen image encoder, where n and d denote the token length and each token dimension, respectively. Then, we convert latent features F l into texture features F L ∈ R m × d through a learnable tokenizer layer consisting of a 2D convolution and a flattened operation, where m stand for the length of F L token. We ensure that the dimension of each token from the texture features F L matches the dimension of each token from the semantic features F H through the kernel size of the 2D convolution. Finally, according to Eq. 3, we concatenate F L and F H along the token length to obtain the person's appearance features.

where F C ∈ R ( m + n ) × d . Therefore, FLC leverages the frozen VAE encoder's ability to preserve detailed information, addressing the issue of not having a dedicated person image feature extractor.

3.3 Image-Level Conditioning Module

As previously mentioned, existing methods only support the generation setting of one source image and one target pose, which limits user expectations in other use scenarios. To address this, we propose the ILC module to inject a random variable number of source image conditions and introduce a masking strategy to adapt to different user scenarios. As shown in Figure 3(a), when generating three target images based on three target poses, the ILC first masks the three target images to be generated, then combines them with one source image to form a joint mask, while extracting the target pose from the source image and combining it with the three given target poses to form a joint pose; As shown in Figure 3(b), when generating a target image based on three source images, the ILC first masks one target image and then

Denoising UNet

？

Masking Strategy

Crop

A Target Image

Denoising UNet

Masking Strategy

？

？

？

(a) Generating multiple target images from multiple target poses.

Crop

Multiple Target Images

combines it with three source images to form a joint mask while extracting the poses from the three source images and combining them with one given target pose to form a joint pose. This achieves a one-to-one correspondence between the person's image and pose.

Specifically, we use a frozen V AE to extract latent space features from the joint target and add noise to obtain the noise latent. Similarly, we use a frozen VAE to get the masked latent from the joint mask. The ILC module concatenates the noise latent and masked latent along the channel dimension. Then, we introduce a binary mask, the width and height of which are the same as the noise latent, with 0 and 1 representing the masked and unmasked parts, respectively. This ensures that the model correctly distinguishes the areas of the target image to be generated. Therefore, the input channels of our model are 9, with the channel numbers of noise latent and masked latent both being 4, and the binary mask being 1. Besides, noted that existing methods ignore the image-level conditions of the source image and only inject the feature-level conditions of the source image through the cross-attention module. In contrast, as shown in Figure 2(b), we inject the image-level conditions of the source image combined with the masking strategy, achieving richer inherent context information and spatial structure alignment between the person's image and pose.

3.4 Cross-View Attention Module

To ensure that each target image in the generated joint objectives possesses realistic visual content and maintains consistency in the person subject images, inspired by [2, 10], we propose a cross-view attention (CVA) module to decompose global and local cross-attention. As shown in Figure 2, the

Project  In

Self - Attention

Project  Out

Joint

Split

Input Feature

Output Feature

CVA module is embedded after each cross-attention module in the denoising UNet. The schematic diagram of the CVA module is shown in Figure 4, which mainly consists of the split, projection layer, self-attention layer, and joint. We first take the output of the feature by cross-attention as the input features of CVA, split the global input features into four smaller local person features, and then add a new temporal dimension to learn the attention between each local character image. Subsequently, the four person features are reshaped back into one global output feature, ignoring the temporal dimension. This approach allows IMAGPose to capture both global and local attention, providing a more nuanced understanding of the relationships between different parts of the persons' image.

For the pose condition, we introduced a pose encoder identical to ControlNet 2 for injection after the first convolutional layer. Therefore, the loss function L MSE of IMAGPose according to Eq. 4, as follows. Here, F C , F I and F P denote the feature of the FLC module, the feature of the ILC module and pose feature, respectively.

In the inference stage, we use classifier-free guidance according to Eq. 5.

4 Experiments

Datasets. We conducted experiments on the DeepFashion dataset [21], which consists of 52,712 high-resolution images of fashion models, and the Market-1501 dataset [54], which includes 32,668 low-resolution images with diverse backgrounds, viewpoints, and lighting conditions. We extracted the skeletons using OpenPose [3] and followed the dataset splits provided by [1]. It's important to note that the person IDs of the training and testing sets do not overlap for both datasets.

Metrics. We conducted a comprehensive evaluation of the model, including both objective and subjective metrics. Objective metrics include the structural similarity index measure (SSIM) [46], learned perceptual image patch similarity (LPIPS) [52], and fréchet inception distance (FID) [12]. On the other hand, subjective assessments focus on user-oriented metrics, such as the percentage of real images misclassified as generated (R2G) [24], the percentage of generated images misclassified as real (G2R) [24], and the percentage of images considered superior among all models (Jab) [37].

Implementations. We conduct experiments on 8 NVIDIA V100 GPUs. Our configuration can be summarized as follows: (a) We use the pre-trained Stable Diffusion V1.5 3 and modified the first convolutional layer to accommodate additional conditions. Unless otherwise specified, we use Dinov2-G/14 4 as the image encoder. In the tokenizer layer, both the kernel size and stride of the 2D convolution are 16, and the dimensions of the input and output channels are 4 and 768, respectively. (b) Following [1, 36], we train our model on the DeepFashion dataset with sizes of 256 × 176 and 512 × 352 . For the Market-1501 dataset, we used images of size 128 × 64 . (c) In the masking strategy, we defaulted to randomly occluding 1-4 images. (d) The model is trained for 300k steps using the AdamW optimizer with a learning rate of 5 e -5 . Each batch size is 4, and a linear noise schedule of 1000 time steps is applied. (e) In the inference stage, we used a DDIM sampler with 20 steps, and set w to 2.0 in the guidance scale.

4.1 Main Results

IMAGPose supports generating one or multiple target images under different conditions based on user scenarios. For a fair comparison and adaptation to the IMAGPose framework, we use three

Target Pose

GT

Source Image

Ours

DPTN

CASD

NTED

PIDM

PoCoLD

GFLA

PISE

CFLD

PCDMs

black images for padding in the masking strategy and replicate the target pose 3 times in the joint pose during the inference stage. Then IMAGPose generates 3 same target images, and we randomly select one generated image for qualitative and quantitative evaluation. Unless otherwise specified, the above settings are the default configurations.

Quantitative Results. From Table 1, we quantitatively compared proposed IMAGPose with several state-of-the-art methods including Def-GAN [37], PATN [57], ADGAN [25], PISE [49], GFLA [30], DPTN [51], NTED [29], CocosNet2 [55], CASD [56], PoCoLD [11], PIDM [1], PCDMs [36] and CFLD [22]. Compared to other generative methods, based on diffusion models [11, 1, 36, 22] perform significantly better, especially proposed IMAGPose, which outperforms all methods. For example, IMAGPose significantly leads the GAN-based method ADGAN on the SSIM metric since the GAN framework's unstable training fails to generate high-quality images. Compared to CFLD, which is also based on a diffusion model framework, IMAGPose demonstrates absolute performance advantages at both 256 × 176 and 512 × 352 scales. Because existing architectures overlook texture details and only inject appearance features using a generic image encoder.

Table 1 also shows the quantitative results on the Market-1501. Like the DeepFashion, IMAGPose outperforms all SOTA methods in SSIM, LPIPS, and FID. Compared to NTED, which uses finegrained texture features, ours shows more significant advantages, thanks to V AE's inherent ability to reconstruct low-level texture. Despite PCDMs' additional refinement model, IMAGPose provides richer context information by injecting source image features at both image and feature levels.

Qualitative Results. Figure 5 visually compares our IMAGPose with other SOTA methods on the DeepFashion dataset. The results lead to several conclusions: (1) As shown in the first to third rows, diffusion-based methods capture minute clothing textures, but IMAGPose generates higher-quality details than other diffusion models. (2) In cases of large-scale pose transformations (as shown in the fourth and fifth rows), CFLD and PCDMs exhibit clear overfitting to the pose (i.e., objects do not should appear on the hand ), only our method can generate images aligned with the target and consistent in detail texture. (3) As shown in the sixth to seventh rows, when generating complex textures and invisible areas, IMAGPose reasonably infers detailed textures, providing better visual consistency even if the results are inconsistent with the target. In summary, the proposed method consistently produces more realistic and lifelike person images, demonstrating the advantages of the image-level and feature-level conditions proposed by IMAGPose. Please see C.1 for more examples.

User Study. The above quantitative and qualitative comparisons reflect the significant advantages of IMAGPose in generating results. Further, we conducted a user study with 50 volunteers. This study includes comparisons with basic facts (i.e., R2G and G2R) and other methods (i.e., J2b). The higher the score for these three indicators, the better the performance. As shown in Figure 6, IMAGPose exhibits commendable performance on all three indicators on the DeepFashion dataset. For instance, volunteers judged our generated images as 58.5% (G2R) real

22.5

23.8

24.4

26.9

30.1

32.4

35.2

45.3

25.3

26.7

28.2

30.3

34.5

37.6

40.1

58.5

3.2

5.4

6.9

6.6

7.5

13.3

14.8

42.3

0

10

20

30

40

50

60

DPTN

CASD

NTED PoCoLD PIDM

CFLD

PCDMs

Ours

Percentage (%)

Different SOTA Methods

R2G

G2R

Jab

images, nearly 18.4% higher than the second-best model. The proposed IMAGPose scored 42.3% on the Jab index, indicating our method is more popular. Please see C.3 of the Appendix for more detail.

Uniformity. IMAGPose only needs to be trained once to support multiple user scenarios. Here, IMAGPose simulates two user scenarios: generating a target image given a source image and a target pose, and generating a target image given multiple source images and a target pose. As shown in Figure 7, both T1 and T2 provide one source image and one target pose, similar to the previous SOTA. The difference is that T1 replicates the target pose in the joint pose, while T2 replicates the source image in the joint image. T3 randomly uses multiple different source images.

0.73

0.74

0.75

0.76

0.77

0.78

100k

150k

200k

250k

300k

SSIM

Training Steps

T1

T2

T3

5.1

5.5

5.9

6.3

6.7

100k

150k

200k

250k

300k

FID

Training Steps

T1

T2

T3

0.7561

0.7594

0.7727

0.1172

0.1255

0.1284

5.3286

5.7204

5.8738

0.11

0.12

0.13

0.14

0.15

0.16

100k

150k

200k

250k

300k

LPIPS

Training  Steps

T1

T2

T3

We found that under these 3 different settings, compared to Table 1, T1, T2, and T3 demonstrate strong competitiveness on all metrics, indicating that IMAGPose can adapt to the above scenarios only once training. For example, T2 surpasses all previous SOTA methods and beats T1, suggesting the ILC module can effectively capture the appearance features of a person by replicating the same source image multiple times during the inference stage. Then, when using multiple different source images to generate a target image, T3 achieves 0.7727, 0.1172, and 5.3286 on SSIM, LPIPS, and FID, respectively, outperformers T2 and other SOTA methods, indicating that IMAGPose has uniformity when provided multiple different source images.

Furthermore, IMAGPose can also adapt to the other scenario: generating multiple target images from a source image and multiple target poses, i.e., IMAGPose* in Figure 8. Compared with the original IMAGPose, IMAGPose* slightly reduces the SSIM performance but still surpasses other diffusion model-based methods, which proves that even when inferring multiple different target images at once, IMAGPose* can still maintain high quality and fidelity. Regarding speed, due to the architectural

0.725

0.735

0.745

0.755

0.765

0

2

4

6

8

10

SSIM

Speed (s)

IMAGPose

PIDM

PoCoLD

PCDMs

IMAGPose*

CFLD

advantage of the masking strategy in ILC, it is nearly 8 times faster than PIDM, and nearly 3 times faster than PoCoLD and CFLD. Noted that both the default settings of IMAGPose and IMAGPose* are highly competitive regarding generation quality and speed. Therefore, once our method is trained, users can choose the appropriate inference settings according to their needs in different scenarios. The visual results of IMAGPose in different user scenarios are shown in Figure 9, which is consistent with our above analysis. Overall, IMAGPose can unify different tasks in real-world scenarios. For more results and discussions, please refer to C.2 of the Appendix.

4.2 Ablations

Effectiveness of the Module. Table 2 demonstrates the impact of different modules on the results of IMAGPose. B0 denotes IMAGPose removes the CVA module, the texture features from the VAE in the FLC module, and the image-level conditions in the ILC module, which results in a model with only 5 input channels. B1 represents, based on B0, the addition of the CV A module. B2 denotes, based on B0, adding the VAE's texture detail features, resulting in a complete FLC module. B3 indicates,

T1 (Default)

GT

T2

T3

IMAGPose*

based on B1, that adding image-level conditions forms a complete ILC module with 9 input channels. B4 represents IMAGPose using both the FLC and ILC modules, while ignoring the CVA module.

As shown in Table 2, IMAGPose outperforms other settings on three metrics and gradually improves performance as more modules are included. Specifically, B1 surpasses B0 on SSIM, LPIPS, and FID, indicating that the CVA module can enhance local fidelity and global consistency. B2 and B3 outperform B1 by 0.0125 and 0.0172 on SSIM, suggesting that FLC and ILC can effectively provide exemplary texture detail guidance and ensure alignment of images and

poses. Moreover, noted that the combination of the three modules brings additional performance improvements. For example, our method outperforms B2, B3, and B4 on all metrics, respectively.

Model Variant. Table 3 summarizes the variants based on IMAGPose. We keep other settings unchanged and only modify the base model and image encoder. The overall performance differences between various versions of the base diffusion model and image encoder are insignificant, indicating that the base diffusion model and image encoder have a minimal impact on the results of the generated images. Specifically, the results suggest that when 'Base.' is SD V1.5, Dinov2-G/14 slightly outperforms on all metrics. Furthermore, when using the same image encoder, SD V2.1 is superior to SD V1.5 in most cases.

4.3 Application

Table 4 evaluates the applicability of images generated by IMAGPose in downstream tasks, i.e., person re-identification. Initially, we randomly selected 20%, 40%, 60%, and 80% subsets from the training set of the Market-1501 dataset, ensuring at least one image per identity, thereby creating a new dataset. We adopt BoT [23] as the base network and perform baseline training on each data subset for a fair comparison. Then, we incorporate images generated by IMAGPose. These generated images are synthesized from randomly selected images with the same identity and pose from the new dataset. The Rank1 results are shown in Table 4. The experimental results indicate

that the IMAGPose algorithm significantly improves re-identification performance compared to the Standard . Moreover, IMAGPose consistently demonstrates superior performance in re-identification tasks compared to SOTA methods.

5 Conclusion

This paper presents IMAGPose, a unified conditional framework for pose-guided image generation. This framework addresses the limitations of existing methods by enabling the generation of multiple target images with different poses simultaneously and from multi-view source images. IMAGPose

incorporates three pivotal modules: a Feature-Level Conditioning (FLC) module, an Image-Level Conditioning (ILC) module, and a Cross-View Attention (CVA) module. The FLC module effectively combines low-level texture features with high-level semantic features, addressing the issue of missing detail information. The ILC module aligns images and poses to adapt to diverse user scenarios, while the CVA module ensures local fidelity and global consistency when multiple source image prompts. The proposed framework has demonstrated competitive performance in both qualitative and quantitative evaluations, showing consistency and photorealism under challenging user scenarios. This work represents a significant step forward in the field of image generation, opening up new possibilities for user interaction and customization. Limitations. (1) IMAGPose supports generating images in different user scenarios and significantly improves the synthesis quality. However, due to computational limitations, IMAGPose currently supports generating up to four images at a time. Future work will continue to explore how to generate more target images in one forward process. (2) The performance of the model is subpar for generating cartoon characters and non-photorealistic styles because our training data consists of photorealistic human images for fair comparison. In future work, we plan to include a broader range of data and design style transformation modules to overcome this bias.

Acknowledgments and Disclosure of Funding

This work is supported by the National Natural Science Foundation of China (Grant No. 61925204, and 62332010). The authors would thank all Reviewers and ACs of OpenReview for the thorough review and insightful suggestions.

References

Ankan Kumar Bhunia, Salman Khan, Hisham Cholakkal, Rao Muhammad Anwer, Jorma Laaksonen, Mubarak Shah, and Fahad Shahbaz Khan. Person image synthesis via denoising diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5968-5976, 2023.

Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2256322575, 2023.

Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7291-7299, 2017.

Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath. Generative adversarial networks: An overview. IEEE signal processing magazine , 35(1):53-65, 2018.

Jisheng Dang, Huicheng Zheng, Jinming Lai, Xu Yan, and Yulan Guo. Efficient and robust video object segmentation through isogenous memory sampling and frame relation mining. IEEE Transactions on Image Processing , 32:3924-3938, 2023.

Jisheng Dang, Huicheng Zheng, Xiaohao Xu, Longguang Wang, and Yulan Guo. Beyond appearance: Multi-frame spatio-temporal context memory networks for efficient and robust video object segmentation. IEEE Transactions on Image Processing , 33:4853-4866, 2024.

Jisheng Dang, Huicheng Zheng, Xiaohao Xu, Longguang Wang, Qingyong Hu, and Yulan Guo. Adaptive sparse memory networks for efficient and robust video object segmentation. IEEE Transactions on Neural Networks and Learning Systems , 2024.

Patrick Esser, Ekaterina Sutter, and Björn Ommer. A variational u-net for conditional appearance and shape generation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 8857-8866, 2018.

Artur Grigorev, Artem Sevastopolsky, Alexander Vakhitov, and Victor Lempitsky. Coordinate-based texture inpainting for pose-guided human image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12135-12144, 2019.

Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725 , 2023.

Xiao Han, Xiatian Zhu, Jiankang Deng, Yi-Zhe Song, and Tao Xiang. Controllable person image synthesis with pose-constrained latent diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 22768-22777, 2023.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS , 2020.

Jonathan Ho, Ajay Jain, Pieter Abbeel, and UC Berkeley. Denoising diffusion probabilistic models.

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 , 2022.

Siyu Huang, Haoyi Xiong, Zhi-Qi Cheng, Qingzhong Wang, Xingran Zhou, Bihan Wen, Jun Huan, and Dejing Dou. Generating person images with appearance-aware pose stylizer. arXiv preprint arXiv:2007.09077 , 2020.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

FirstName LastName. Frobnication tutorial, 2014. Supplied as supplemental material tr.pdf .

Yining Li, Chen Huang, and Chen Change Loy. Dense intrinsic appearance flow for human pose transfer. In CVPR , 2019.

Hanbing Liu, Jun-Yan He, Zhi-Qi Cheng, Wangmeng Xiang, Qize Yang, Wenhao Chai, Gaoang Wang, Xu Bao, Bin Luo, Yifeng Geng, et al. Posynda: Multi-hypothesis pose synthesis domain adaptation for robust 3d human pose estimation. In Proceedings of the 31st ACM International Conference on Multimedia , pages 5542-5551, 2023.

Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1096-1104, 2016.

Yanzuo Lu, Manlin Zhang, Andy J Ma, Xiaohua Xie, and Jian-Huang Lai. Coarse-to-fine latent diffusion for pose-guided person image synthesis. arXiv preprint arXiv:2402.18078 , 2024.

Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei Jiang. Bag of tricks and a strong baseline for deep person re-identification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops , pages 0-0, 2019.

Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, and Luc Van Gool. Pose guided person image generation. In NeurIPS , 2017.

Yifang Men, Yiming Mao, Yuning Jiang, Wei-Ying Ma, and Zhouhui Lian. Controllable person image synthesis with attribute-decomposed gan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5084-5093, 2020.

Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 , 2014.

Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2iadapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 4296-4304, 2024.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4195-4205, 2023.

Yurui Ren, Xiaoqing Fan, Ge Li, Shan Liu, and Thomas H Li. Neural texture extraction and distribution for controllable person image synthesis. In CVPR , 2022.

Yurui Ren, Xiaoming Yu, Junming Chen, Thomas H Li, and Ge Li. Deep image spatial transformation for person image generation. In CVPR , 2020.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

Fei Shen, Xiaoyu Du, Liyan Zhang, and Jinhui Tang. Triplet contrastive learning for unsupervised vehicle re-identification. arXiv preprint arXiv:2301.09498 , 2023.

Fei Shen, Xin Jiang, Xin He, Hu Ye, Cong Wang, Xiaoyu Du, Zechao Li, and Jinghui Tang. Imagdressingv1: Customizable virtual dressing. arXiv preprint arXiv:2407.12705 , 2024.

Fei Shen, Xiangbo Shu, Xiaoyu Du, and Jinhui Tang. Pedestrian-specific bipartite-aware similarity learning for text-based person retrieval. In Proceedings of the 31th ACM International Conference on Multimedia , 2023.

Fei Shen, Hu Ye, Sibo Liu, Jun Zhang, Cong Wang, Xiao Han, and Wei Yang. Boosting consistency in story visualization with rich-contextual conditional diffusion models. arXiv preprint arXiv:2407.02482 , 2024.

Fei Shen, Hu Ye, Jun Zhang, Cong Wang, Xiao Han, and Wei Yang. Advancing pose-guided image synthesis with progressive conditional diffusion models. arXiv preprint arXiv:2310.06313 , 2023.

Aliaksandr Siarohin, Enver Sangineto, Stéphane Lathuiliere, and Nicu Sebe. Deformable gans for posebased human image generation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3408-3416, 2018.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning , pages 2256-2265. PMLR, 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv: Learning,arXiv: Learning , Oct 2020.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Yang Song, Jascha Sohl-Dickstein, DiederikP. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. International Conference on Learning Representations,International Conference on Learning Representations , May 2021.

Hao Tang, Song Bai, Li Zhang, Philip HS Torr, and Nicu Sebe. Xinggan for person image generation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV 16 , pages 717-734. Springer, 2020.

Shuyuan Tu, Qi Dai, Zhi-Qi Cheng, Han Hu, Xintong Han, Zuxuan Wu, and Yu-Gang Jiang. Motioneditor: Editing video motion via content-aware diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7882-7891, 2024.

Cong Wang, Kuan Tian, Yonghang Guan, Jun Zhang, Zhiwei Jiang, Fei Shen, Xiao Han, Qing Gu, and Wei Yang. Ensembling diffusion models via adaptive feature aggregation. arXiv preprint arXiv:2405.17082 , 2024.

Cong Wang, Kuan Tian, Jun Zhang, Yonghang Guan, Feng Luo, Fei Shen, Zhiwei Jiang, Qing Gu, Xiao Han, and Wei Yang. V-express: Conditional dropout for progressive training of portrait video generation. arXiv preprint arXiv:2406.02511 , 2024.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4):600-612, 2004.

Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721 , 2023.

Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven CH Hoi. Deep learning for person re-identification: A survey and outlook. IEEE transactions on pattern analysis and machine intelligence , 44(6):2872-2893, 2021.

Jinsong Zhang, Kun Li, Yu-Kun Lai, and Jingyu Yang. Pise: Person image synthesis and editing with decoupled gan. In CVPR , 2021.

Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543 , 2023.

Pengze Zhang, Lingxiao Yang, Jian-Huang Lai, and Xiaohua Xie. Exploring dual-task correlation for pose guided person image generation. In CVPR , 2022.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

Bo Zhao, Xiao Wu, Zhi-Qi Cheng, Hao Liu, Zequn Jie, and Jiashi Feng. Multi-view image generation from a single-view. In Proceedings of the 26th ACM international conference on Multimedia , pages 383-391, 2018.

Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian. Scalable person re-identification: A benchmark. In ICCV , 2015.

Xingran Zhou, Bo Zhang, Ting Zhang, Pan Zhang, Jianmin Bao, Dong Chen, Zhongfei Zhang, and Fang Wen. Cocosnet v2: Full-resolution correspondence learning for image translation. In CVPR , 2021.

Xinyue Zhou, Mingyu Yin, Xinyuan Chen, Li Sun, Changxin Gao, and Qingli Li. Cross attention based style distribution for controllable person image synthesis. In European Conference on Computer Vision , pages 161-178. Springer, 2022.

Zhen Zhu, Tengteng Huang, Baoguang Shi, Miao Yu, Bofei Wang, and Xiang Bai. Progressive pose attention transfer for person image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2347-2356, 2019.

Supplementary Material

This supplementary material offers a more detailed exploration of the experiments and methodologies proposed in the main paper. Section A provides a series of symbols and definitions for enhanced comprehension. Section B delves deeper into the implementation specifics of our experiments. Section C presents additional experimental outcomes, including a broader range of qualitative comparison examples with state-of-the-art methods, more results on different user scenarios, and a detailed explanation of our user studies. Section D discusses potential societal harms and impacts.

A Some Notations and Definitions

The notations and definitions used in this paper are shown in Table 5.

B Implement Details

As shown in Table 6, our experiments are conducted on 8 NVIDIA V100 GPUs. We use the pretrained Stable Diffusion V1.5 and modify the first convolutional layer to accommodate additional conditions. Dinov2-G/14 is used as the image encoder. In the tokenizer layer, we set both the kernel size and stride of the 2D convolution to 16, and the dimensions of the input and output channels are set to 4 and 768, respectively. We introduce a pose encoder identical to ControlNet, injected after the first convolutional layer, to handle the pose condition. Our model is trained on the DeepFashion dataset, following previous studies, with image sizes of 256x176 and 512x352. For the Market-1501 dataset, we use images of size 128x64. As part of our masking strategy, we default to randomly occluding 1-4 images. The model is trained for 300k steps using the AdamW optimizer with a learning rate 5e-5. We use a batch size of 4 and apply a linear noise schedule of 1000 time steps. During the inference stage, we use a DDIM sampler with 20 steps and set the guidance scale to 2.0.

C Additional Results

C.1 More Qualitative Comparisons for IMAGPose

Weprovide additional examples for comparison with the state-of-the-art (SOTA) methods in Figure 10.

Target Pose

GT

Source Image

Ours

PoCoLD

CFLD

PCDMs

Image

Encoder

Image

Encoder

Image

Encoder

Image

Encoder

Denoising

UNet

Denoising

UNet

Noise Latent

Target Pose

Source Image

Denoising

UNet

Denoising

UNet

Image

Encoder

Image

Encoder

Image

Encoder

Image

Encoder

Source Image

(b) M1 : Generating multiple target images

with different poses simultaneously.

from a single source image and a single target pose.

Noise Latent

Target Poses

C.2 More Discussion on Different User Scenarios

As discussed in the main text, previous pose-guided image generation methods only support scenarios where a target image is generated from a single source image and a target pose, as shown in Figure 11 (a). However, in practical user scenarios, it is also common and important to generate multiple target images with different poses, as shown in Figure 1. To address these scenarios, we develop a proprietary models M1 to generate multiple target images with different poses simultaneously during the development of IMAGPose, as shown in Figure 11(b). Similar to [2, 10], M1 concatenates different noise latents along the batch dimension. Then, we chose the PIDM [1] based on the diffusion model as the representative for generating a target image from a single source image and a target pose. M1 model are also included in our additional comparisons, as shown in Table 7.

The results in Table 7 indicate that while we can address the generation of target images from multiple target poses by training a dedicated model M1, this approach increases the training burden and introduces complexity for users. Users must continually switch between models to meet their specific needs when generating target images. Moreover, from a performance metric perspective, the dedicated model M1 does not offer any advantages, as it only injects semantic information from the source image using a frozen image encoder. Most importantly, none of the current frameworks support the task of generating target images through multiple image prompts. In contrast, our proposed ILC module introduces a masking strategy to unify target image generation under different user scenarios. As shown in Figure 2 (b), we also incorporate the masking strategy to inject image-level conditions from the source image, and enhance the texture features of the VAE to achieve richer context information and alignment between the image and pose.

As previously mentioned, once IMAGPose is trained, it can accommodate different user scenarios. For generating a target image from a single source image and a target pose, IMAGPose only needs to replicate the target pose multiple times in the joint pose during inference. For generating multiple target images from a single source image and multiple target poses at once, IMAGPose only needs to combine different poses in the joint pose during inference. For generating a target image from multiple source images and a single target pose, IMAGPose only needs to combine different source images in the joint image during inference. Therefore, IMAGPose has the ability to unify pose-guided image generation under different user scenarios.

C.3 User Study

To validate the disparity between the generated images and the real ones, and to evaluate the superiority of our method compared to existing technologies, we conducted a user study involving 50 volunteers. (1) For the R2G and G2R metrics, we selected 40 test samples at random and used each model to generate a corresponding set of images. The volunteers were tasked with distinguishing between the 40 generated images and 40 real images. (2) For the Jab metric, we randomly selected 40 pairs of source images and target poses, and each model was tested to generate a corresponding set of images. The users were then asked to select the highest quality and most faithful image from the side-by-side views. The user study allowed us to evaluate which model produced the most realistic results based on human perception. Example questions are illustrated in 13.

D Ethics Statement

In this paper, we propose IMAGPose, a unified conditional framework designed to cater to various user scenarios for generating different human images. However, it is well-known that virtually all methods of human image synthesis, including ours, can be misused by malicious actors to create false content and disseminate misinformation. This is a significant concern that we fully acknowledge. Nevertheless, substantial progress has been made in the field of detecting and preventing such malicious tampering. Our work aims to provide valuable support for research and external audits in this area, helping to strike a balance between the value of our technology and the risks associated with unrestricted access. This ensures that our technology can be used safely and beneficially.

T1 (Default)

IMAGPose*

T3

PCDMs[复制]

https://tp.wjx.top/vm/m2FRkvj.aspx

23. Please indicate whether the following image is real or fake.

Real

Fake

NeurIPS Paper Checklist

1. Claims

问卷星 提供技术支持 举报 Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction clearly articulate that existing works have overlooked the user scenarios of pose-guided image generation. To address this issue, IMAGPose is proposed. Extensive experiments have validated the effectiveness of IMAGPose.

2. Limitations

第2页 共2页

2024/5/22 19:00

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: In the conclusion section, we have included a discussion of the limitations and potential directions for future work.

3. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have detailed the hyperparameters and experimental settings in the main text and the appendix of the paper.

4. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

下一页

Answer: [Yes]

Justification: Our code and models will be open-sourced upon acceptance.

5. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper specifies all the training and test details, including data splits, hyperparameters, how they were chosen, and the type of optimizer used, necessary to understand the results.

6. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The results reported in the paper are reproducible.

7. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper provides sufficient information on the computer resources

8. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper discusses both potential positive societal impacts and negative societal impacts of the work performed.

9. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: The paper describes safeguards for responsibly releasing high-risk data or models, such as pretrained language models or image generators.

10. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Proper credit is given to the creators or original owners of assets used in the paper, and the license and terms of use are explicitly mentioned and respected.

11. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Upon acceptance, documentation for the code, data, and logs will be provided alongside the assets.

12. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes]

Justification: The details of the user study, including the full text of instructions given to participants and any relevant screenshots, are provided in the appendix.

13. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [Yes]

Justification: The user study adheres to relevant national laws and regulations regarding participant safety and ethics. Risks incurred by study participants, as well as any necessary disclosures, are described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a diffusion-based pose-guided image generation method. Specifically, given an image and a sequence of poses, this paper aims to generate images that follow the input poses and maintain the appearance of the input image. To capture the texture details of the input image, they propose to combine features from VAE and a pre-trained image encoder. They design an Image-level conditioning module, which combines four images into a joint representation.  The images that need to be predicted are masked. By doing this, their method can generate 1-3 images in one forward step, or use 1-3 condition images in the same time.\n\nSTRENGTHS:\n1. They propose to combine VAE and image encoder features as the condition, which helps the model to generate more details images.\n2. Their Image-Level Conditioning design enables their models to condition on the dynamic number of input images.\n\nWEAKNESSES:\n1. Limited novelty. 1） The main contribution of this paper is forming four image latents into a 4-grid join latent, which is very similar to [1]. 2） The design of Sec 3.2 combines the VAE feature with the feature from Dino, which is similar to Animate Anyone[2], which uses CLIP image features as the condition.\n2. Lack of thorough comparison methods. Given an RGB person image $\\alpha$ and a pose image $\\beta$, this paper aims to generate a person that follows the pose image  $\\beta$ with the appearance of the $\\alpha$, which can also be called pose transfer. Thus, this paper should also compare their results with pose-guided human pose transfer methods: Animate Anyone [2]. DreamPose [3], Follow your pose [4] …. Besides, this paper injects pose guidance in ControlNet way, they should also compare their performance with ControlNet.\n3. Incorrect statements. In line 44-50, they claim that former diffusion-based generation using frozen image encoder pre-trained on general image dataset (not specific for person image), so these methods cannot extract texture detail information. However, Animate Anyone [2] trains a reference appearance net on large-scale human datasets, which can effectively encode detailed texture features.\n\n[1] Kara, Ozgur, et al. \"Rave: Randomized noise shuffling for fast and consistent video editing with diffusion models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n[2] Hu, Li, et al. \"Animate anyone: Consistent and controllable image-to-video synthesis for character animation.\" arXiv preprint arXiv:2311.17117 (2023).\n[3] Karras, Johanna, et al. \"Dreampose: Fashion image-to-video synthesis via stable diffusion.\" 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023.\n[4] Ma, Yue, et al. \"Follow your pose: Pose-guided text-to-video generation using pose-free videos.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 5. 2024.\n\nQUESTIONS:\n1. Since this paper forms four images into a joint feature map, the self-attention in SD can conduct feature transformation among each images. Why do you split the joint feature map and conduct temporal attention among them (Sec 3.4)?\n2. This paper evaluates their method solely on person images with clean background. How does this method perform on images with complex, meaningful backgrounds.\n\nLIMITATIONS:\nYes, they have provided discussions about limitations and negative social impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers three different pose-guided image generation scenarios from a scene perspective and attempts to cover all scenarios using a unified framework. In my opinion, it is very insightful and inspiring. The proposed IMAGPose framework unifies all scenarios through several ingenious components, namely FLC, ILC, and CVA. The qualitative and quantitative evaluations, as well as the user study, demonstrate its strong competitiveness.\n\nSTRENGTHS:\n(1) Existing pose-guided image generation has only considered scenarios involving a single source image and a single target pose. However, this paper insightfully introduces two additional potential scenarios: providing multiple target poses at once and multiple source images at once. These assumptions are both reasonable and necessary.\n\n(2) The proposed IMAGPose framework addresses the needs of different scenarios through simple yet ingenious designs. For example, it directly uses VAE features to capture underlying texture details, learns image conditions implicitly by taking the image as input, and employs the CVA module's innovative setting of merging, splitting, and then merging again, which is particularly enlightening.\n\n(3) The figures and tables are well-organized and clearly presented. The experiments are comprehensive, the results are convincing, and the appendix provides detailed supplementary information.\n\nWEAKNESSES:\n(1) The Reference UNet [1]  also uses features from both the VAE and image encoder. It would be beneficial if the authors could discuss the differences and advantages of IMAGPose in comparison to this approach.\n\n(2) In line 152, does the term \"image encoder\" refer to CLIP? Why not use the image encoder features from DinoV2? \n\n[minor] In line 184, \"IFC\" should be corrected to \"ILC\".\n\n[1] Li Hu et al. Animate anyone: Consistent and controllable image-to-video synthesis for character animation.\n\nQUESTIONS:\n(1) The authors did not specify what type of pose encoder is used. Could you provide more details?\n\n(2) It is commendable that the authors have evaluated the speed for different SOTA models. However, I am curious about the training duration.\n\n[minor] Appendix Figure 17 needs some descriptive text. Although it is understandable based on the main manuscript, a good appendix should also have comprehensive captions.\n\nLIMITATIONS:\nThe authors have already discussed limitations and societal impact in the conclusion and appendix.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper thoroughly analyzes and considers the application scenarios of pose-guided person image synthesis from the perspective of real-world significance.  Author introduces previously unconsidered but intriguing scenarios and proposes the IMAGPose framework to unify different tasks.  Comprehensive experiments, ablation studies, and user research conducte to demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n- The proposed task scenarios are highly insightful, with a clear and well-motivated approach.\n\n- The study's contribution of a unified conditional diffusion model to address different task scenarios is of significant value.\n\n- The evaluation is comprehensive, and the proposed method generally demonstrates superior performance compared to existing works, supported by user studies and clear visualizations.\n\n- The impact of different components is analyzed through ablation studies, further proving the effectiveness of the proposed technique and providing a better understanding of the method.\n\nWEAKNESSES:\nThe IMAGPose framework heavily relies on the detection results from OpenPose. I am curious about how the performance of IMAGPose would be affected if OpenPose produces poor results. Do the authors have any solutions to mitigate this issue?\n\nQUESTIONS:\n- The design of FLC is interesting, as it uses almost lossless features from VAE as conditions. However, for pose-guided image generation, would it be better to add a text caption condition? For example, T2I-Adapter incorporates both text and image features through a decoupled cross-attention mechanism. How does this differ from author concatenation approach?\n\n- In ILC,  mask strategy is a critical component for unifying different conditional generations. But I am slightly confused about the purpose of this binary mask. What are its benefits/aim?\n\n- I noticed that if multiple poses are given at once and these poses are very continuous, would this be similar to generating a video based on continuous poses, such as a dance sequence? In such a user scenario, what are the comparisons and advantages of IMAGPose?\n\n- Is there a trade-off between speed and efficiency? For example, if a user only wants to generate a target image based on a single pose, how should they proceed, and how should they make their choice?\n\n- In Figure 8, what does IMAGPose* refer to? \n\n- What is the guidance scale for CFG?\n\n- In Figure 9, does T1 (Default) refer to IMAGPose?\n\n---\nAfter the rebuttal,  I raised my score from **6 to 8** because this work provides a new perspective on traditional tasks (a novel user angle) and unifies the framework. The authors' efforts are commendable, and I believe this work deserves acceptance.\n\nLIMITATIONS:\nThe authors addressed the limitations and the work does not have negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces IMAGPose, a unified conditional framework designed to overcome the limitations of existing diffusion models in pose-guided person image generation. Traditional models primarily focus on generating a target image from a single source image and a target pose. IMAGPose extends this capability by addressing two additional scenarios: generating multiple target images with different poses simultaneously and generating target images from multi-view source images. The proposed framework incorporates three key modules: \n\n- Feature-Level Conditioning (FLC): Combines low-level texture features from a VAE encoder with high-level semantic features from an image encoder.\n- Image-Level Conditioning (ILC): Aligns images and poses through a masking strategy, supporting variable numbers of source images and poses.\n- Cross-View Attention (CVA): Ensures local fidelity and global consistency by decomposing global and local cross-attention.\n\nExtensive experiments demonstrate the framework's ability to produce consistent and photorealistic images under various challenging user scenarios.\n\nSTRENGTHS:\n- The paper introduces a unified framework that extends the capabilities of existing diffusion models, addressing important user scenarios that were previously overlooked.\n\n- The experimental results are robust, demonstrating the effectiveness of the proposed framework across multiple datasets.\n\nWEAKNESSES:\n- The framework's complexity, with multiple conditioning modules and attention mechanisms, may pose challenges for real-time applications and require significant computational resources.\n\n- While the framework shows promising results on specific datasets, it would be beneficial to see its performance on a broader range of datasets and in more diverse scenarios.\n\n- The use of frozen VAE and image encoders may limit the adaptability of the model to specific tasks, potentially impacting the quality of the generated images.\n\n- The paper could benefit from more detailed ablation studies to better understand the contribution of each module within the framework.\n\nQUESTIONS:\n- Can the authors provide more details on the computational requirements for training and inference using IMAGPose? Specifically, what are the resource constraints, and how do they impact the practical usability of the framework?\n\n- How does the framework perform on datasets outside of DeepFashion and Market-1501? Are there plans to test IMAGPose on more diverse datasets to evaluate its generalization capabilities?\n\n- Can the authors elaborate on the potential impact of using fixed encoders (VAE and image encoders) on the quality and flexibility of the generated images? Are there scenarios where fine-tuning these encoders could be beneficial?\n\n- The paper mentions that IMAGPose currently supports generating up to four images simultaneously. Does this indicate computational resource constraints? How might these constraints affect large-scale image generation tasks, particularly for high-resolution or complex scenes?\n\n- IMAGPose introduces three core modules (FLC, ILC, CVA), each adding to the model's complexity. How does this complexity impact the training and inference process, and does it increase implementation and debugging difficulty?\n\n - The paper uses frozen VAE and image encoders for feature extraction. While this reduces training time, are these encoders fully suitable for the specific task demands? Could this approach limit the quality and diversity of generated images?\n\n- Although IMAGPose aims to unify different user scenarios, are there specific or special requirements that still need additional adjustments or extensions? Could this general approach introduce limitations in some applications?\n\nLIMITATIONS:\n- The paper could discuss potential biases in the generated images and how the framework handles diverse demographic attributes.\n\n- A discussion on the limitations related to real-time applications and potential solutions to mitigate computational overhead would be beneficial.\n\n- While the societal impact of misuse is mentioned, a more detailed discussion on the ethical implications and safeguards for responsible usage could strengthen the paper.\n\n- The following related work is recommended for citation & discussion:\n\nZhao, B., Wu, X., Cheng, Z.-Q., Liu, H., Jie, Z., & Feng, J. (2018). Multi-view image generation from a single-view. In Proceedings of the 26th ACM International Conference on Multimedia (pp. 383-391).\n\nHuang, S., Xiong, H., Cheng, Z.-Q., Wang, Q., Zhou, X., Wen, B., Huan, J., & Dou, D. (2020). Generating person images with appearance-aware pose stylizer. In Proceedings of the 29th International Joint Conference on Artificial Intelligence (IJCAI 2020).\n\nLiu, H., He, J.-Y., Cheng, Z.-Q., Xiang, W., Yang, Q., Chai, W., Wang, G., Bao, X., Luo, B., & Geng, Y. (2023). Posynda: Multi-hypothesis pose synthesis domain adaptation for robust 3D human pose estimation. In Proceedings of the 31st ACM International Conference on Multimedia (pp. 5542-5551).\n\nTu, S., Dai, Q., Cheng, Z.-Q., Hu, H., Han, X., Wu, Z., & Jiang, Y.-G. (2024). MotionEditor: Editing video motion via content-aware diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 7882-7891).", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}