Stable-Pose: Leveraging Transformers for Pose-Guided Text-to-Image Generation

Authors: jiajun  wang
morteza  ghahremani
yitong  li
björn  ommer
christian  wachinger
frozen / trainable
zero  convolution
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 19:33:07

Abstract: Controllable text-to-image (T2I) diffusion models have shown impressive performance in generating high-quality visual content through the incorporation of various conditions. Current methods, however, exhibit limited performance when guided by skeleton human poses, especially in complex pose conditions such as side or rear perspectives of human figures. To address this issue, we present Stable-Pose, a novel adapter model that introduces a coarse-to-fine attention masking strategy into a vision Transformer (ViT) to gain accurate pose guidance for T2I models. Stable-Pose is designed to adeptly handle pose conditions within pre-trained Stable Diffusion, providing a refined and efficient way of aligning pose representation during image synthesis. We leverage the query-key self-attention mechanism of ViTs to explore the interconnections among different anatomical parts in human pose skeletons. Masked pose images are used to smoothly refine the attention maps based on target pose-related features in a hierarchical manner, transitioning from coarse to fine levels. 
Additionally, our loss function is formulated to allocate increased emphasis to the pose region, thereby augmenting the model's precision in capturing intricate pose details.    We assessed the performance of Stable-Pose across five public datasets under a wide range of indoor and outdoor human pose scenarios. Stable-Pose achieved an AP score of 57.1 in the LAION-Human dataset, marking around 13\% improvement over the established technique ControlNet.  The project link and code is available at https://github.com/ai-med/StablePose.

Main paper: 
 introduction: 
 Pose-guided text-to-image (T2I) generation holds immense potential for swiftly producing photorealistic images that exhibit contextual relevance and accurate posing through the integration of text prompts and pose instructions. The kinematic or skeleton pose provides a set of key points (joints) that represent the skeletal framework of the human body (shown in Figure A.1) . Despite the sparsity, skeleton-pose data offers sufficient details of human poses with high flexibility and computational efficiency for T2I generation in various applications such as animation, robotics, sports training, and e-commerce, making it user-friendly and ideal for real-time applications [21] . Juxtaposed with other forms of pose information like volumetric pose with dense content, skeleton pose is capable of conveying heightened articulation information, facilitating intuitive interpretation and flexible manipulation of human poses [28; 20] .Traditional pose-guided human image generation methods require a source image during training for dictating the style of the generated images [22; 23; 42; 38; 47] . Such methods, while offering Figure 1 : Stable-Pose leverages the patch-wise attention of ViTs to address the complex pose conditioning problem in T2I generation, showing superior performance compared to current techniques. control over the appearance, limit the flexibility and diversity of the output and depend heavily on the need for paired source-target data during training. In contrast, recent advancements in controllable T2I diffusion models have shown the potential to eliminate the need for source images and allowed for higher creative freedom by relying on text prompts and external conditions [44; 46; 13; 25; 18] . Enabling more versatile visual content creation, these methods often face challenges in precisely aligning conditional images with sparse representations such as skeleton pose data, especially when dealing with complex pose scenarios like those depicting the back or side views of human figures (Figure 1 ). Moreover, existing methods may also fail to maintain accurate body proportions, resulting in an unnatural appearance of the body.To address the insufficient binding of sparse pose data in T2I generation models, a potential strategy is to capture long-range patch-wise relationships among various anatomical parts of human poses. In this paper, we introduce Stable-Pose that integrates vision Transformers (ViT) into pre-trained T2I diffusion models like Stable Diffusion (SD) [33] , with the goal of improving pose control by capturing patch-wise relationships from the specified pose. In Stable-Pose, the learnable attentions adhere to an innovative coarse-to-fine masking approach, ensuring that pose conditioning is directed toward the relevant pose areas while preserving the diversity of the overall image. To further enhance this effect, a pose-mask guided loss is introduced to optimize the fidelity of the generated images in adherence to the given pose instructions. We evaluated Stable-Pose across five distinct datasets, covering indoor and outdoor image/video datasets. Compared to the state-of-the-art methods, Stable-Pose achieved the highest accuracy and robustness in pose adherence and generation fidelity, making it a promising solution for enhancing pose control in T2I generation. We further performed comprehensive ablation studies to demonstrate the effectiveness of our design. In summary, our contributions are:• Addressing the challenge of generating photo-realistic human images in pose-guided T2I by integrating a novel ViT, achieving highly accurate synthesis in pose adherence and image fidelity, even under challenging conditions.• Introducing a hierarchical integration of pose masks for coarse-to-fine guidance, with a novel pose-masked self-attention mechanism and pose-mask guided loss function. Stable-Pose is designed as a lightweight adapter that can be easily integrated into any pre-trained T2I diffusion models to effectively enhance pose control.• Stable-Pose effectively learns to preserve intricate human shape structures and accurate body proportions, achieving exceptional performance across five publicly available datasets, encompassing both image and video data. [33] to improve the generation of pose-guided human images.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: Our claims in the abstract and introduction accurately match the theoretical and experimental results of our methodology.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Pose-Guided Human Image Generation: Traditional pose-guided human image generation takes a source image and pose as input, aiming to generate photo-realistic human images in specific poses while preserving the appearance from the source image. Prior works [22; 23; 42; 38; 9] primarily utilized generative adversarial network (GAN) or variational autoencoder (VAE), treating the synthesis task as conditional image generation. Zhu et al. [47] integrated attention mechanism for appearance optimization in a Pose Attention Transfer Network (PATN). Zhang et al. [45] implemented a Dual-task Pose Transformer Network (DPTN), using a Transformer module to incorporate features from two tasks: an auxiliary source image reconstruction task and the main pose-guided target image generation task, thereby capturing the dual-task correlation. With the recent emergence of diffusion models [12] , Bhunia et al. [4] proposed a texture diffusion module to transfer texture patterns from the source image to the denoising process. Moreover, classifier-free guidance [11] is applied to provide disentangled guidance for style and pose. Shen et al. [37] proposed a three-stage synthesis pipeline which progressively performs global feature extraction, target prediction, and refinement with three diffusion models. While the source image offers control over appearance, a limitation arises from the necessity of paired source-target data during training. Text, however, obviates this need and offers higher flexibility and diversity for the synthesis. Thus, incorporating text conditions for pose-skeleton-guided human image generation shows significant promise [19; 16] .Controllable Diffusion Models: Large-scale T2I diffusion models [33; 31; 32; 35; 26] excel at creating diverse and high-quality images, yet they often lack precise control with solely text-based prompts. Recent studies aim to enhance the control of T2I models using various conditions such as canny edge, sketch, and human pose [13; 44; 25; 18; 46; 16; 24; 29] . These approaches can be broadly classified into two groups: training the entire T2I model or developing plug-in adapters for pre-trained T2I models. As in the first group, Composer [13] trains a diffusion model from scratch with a decomposition-composition paradigm, enabling multi-control capability. HumanSD [16] fine-tunes the entire SD model using a heatmap-guided loss tailored for pose control. In contrast, T2I-Adapter [25] and GLIGEN [18] train lightweight adapters whose outputs are incorporated into the frozen SD. Similarly, ControlNet [44] employs a trainable copy of the SD encoder to encode conditions for the frozen SD. Uni-ControlNet [46] introduces a uni-adapter for multiple conditions injection to the trainable branch in a multi-scale manner. ControlNet++ [24] proposes to improve controllable generation by explicitly optimizing the cycle consistency between generated images and conditional controls. Our method aligns with the latter category, as it is characterized by reduced training time, cost-effectiveness, and generalizability.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: We discussed the limitations of our method in Sec. 5: Discussion and Conclusion. We also discussed the computational efficiency in Sec. A.2 and Table A .1.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
proposed method: 
 Stable-Pose has a trainable ViT unit that is integrated into the pre-trained T2I diffusion models to direct diffusion models toward the conditioned pose. In latent diffusion models (LDMs) [33] , a pre-trained encoder E and decoder D are employed to transform an input RGB image x ∈ R H×W ×3 with height H and width W into a latent space with reduced spatial dimensions and vice versa. The diffusion process is then efficiently conducted in the down-scaled latent space. During the training, the forward process in diffusion models adds noise to the encoded RGB image z 0 = E(x) to generate its noisy sample z t ∈ R C×h×w with height h, width w, and channel C viaEQUATIONwhere ᾱt is a pre-determined hyperparameter that controls the noise level at step t. The reverse process of diffusion models, so-called denoising, learns the statistics of the Gaussian distribution at each time step. The reverse process is formulated as:EQUATIONAs shown in Figure 2 , the denoising network ϵ θ adopts a UNet backbone, which is equipped with Stable-Pose for augmenting the latent encoding given the input conditional pose image. Let ϵ θ (z t , t, p), t ∈ {1, , where Stable-Pose υ θ conditions the latent encoding z t on the input pose skeleton φ ∈ R h×w×3 , and τ θ is a text encoder that maps the text prompt p to an intermediate sequence. The proposed framework is detailed in Figure 3 . Stable-Pose aims at improving the frozen decoder in the UNet of SD to condition the input latent encoding z t on the conditional pose image φ:z ′ t = z t + υ θ (z t , φ).(3) In Stable-Pose, the pose image φ and the given latent encoding z t are processed by two main blocks named Pose-Masked Self-Attention (PMSA) F θ and pose encoder β θ in such a way thatEQUATIONwhere the pose encoder β θ provides high-level features for the input pose while PMSA F θ explores the patch-wise relationship across input z t using a self-attention mechanism and the binary-masked version of the pose image. PMSA employs a coarse-to-fine framework that provides additional guidance to the latent encoding, directing it towards attending to the conditioned pose. We detail each block in the subsequent sections. The updated latent encoding z ′ t is subsequently fed through the encoder of SD, followed by a series of zero convolutional blocks, a structural resemblance to the architecture employed in ControlNet [44] for ensuring a robust encoding of conditional images.Pose encoder β θ : R H×W ×3 → R f ×h×w is a trainable encoder that maps input pose skeleton image into a feature with height h, width w, and channel f . To this end, we employ a combination of six convolutional layers with SiLU activation layers [8] , downsampling the input pose image by a factor of 8. A zero-convolutional layer is added in the end. As the input pose image contains sparse information, this straightforward pose encoder is sufficient for accurate encoding of the skeleton pose.PMSA F θ : R fin×h×w → R f ×h×w seeks the potential relationships between patches within the latent encoding z t . The interconnections of various parts of the human body suggest the presence of L ← m k .shape [1] 6:attn_mask ← initialize as tensor of size (N, L, L) with entries set to -inf 7: return attn_mask 13: end procedure cohesive relationships among them. To capture this, we leverage the self-attention mechanism. We divide z t into L non-overlapping patches of size p × p, i.e., z t = {zfor i ∈ [0, N -1](1) t , . . . , z (L) t |z (l) t ∈ R p 2 ×fin }. PMSA projects the patch embeddings z t into Query Q = z t W Q , Key K = z t W K , and Value V = z t W V via three learnable weight matrices W Q ∈ R fin×fq , W K ∈ R fin×f k ,and W V ∈ R fin×fv , respectively. Then it computes the attention scores between all patches viaEQUATIONIn this equation, m k denotes a binary mask derived from the input pose image, which is expanded by AttnMask(•) for being used in the self-attention computation (see Figure 3 ). m k is obtained by converting the pose image into a binary mask and downsampling it into the same size of the latent vector z t . The resultant mask is then dilated by a Gaussian kernel of length k. The dilated-binary mask m k is then partitioned into L non-overlapping patches of size p × p. The patches containing pose are labelled as 1 while others are marked as 0. We form a resulting L × L attention mask based on these L patches through the function AttnMask(•). As illustrated in Algorithm 1, for patch entries that correspond to pose regions, both the respective row and column in the mask are set to 0. For all other regions not associated with pose, we assign an extremely small integer value. The attention mask helps to enhance the focus of PMSA on the pose-specific regions.We implement a sequence of N blocks of ViTs, each associated with a unique pose mask, arranged in a coarse-to-fine progression on the latent encoding. This approach gradually steers the latent encoding towards conforming to the specified pose condition. If k = {k 1 , . . . , k N } denotes a set of Gaussian kernels where k 1 > • • • > k N , then the coarse-to-fine self-attention is obtained viaEQUATIONEach encoding A n undergoes further processing by a Feed Forward unit, with the resulting A N integrated into the feature from the pose encoder β θ , as shown in Figure 3 . The Feed Forward block consists of two linear transformations [3] combined with dropout layers, followed by ReLU non-linear activation functions [2] .Pose-mask guided loss criterion: The training of SD models requires high costs in hardware and datasets. Therefore, plug-in adapters for the frozen SD models, such as Stable-Pose, can enhance training efficiency by eliminating the need to compute gradients or maintain optimizer states for SD parameters. Instead, the optimization process focuses on improving Stable-Pose parameters. The loss in Stable-Pose aligns with the coarse-to-fine approach and is defined as follows:EQUATIONHere, α ≥ 1 represents a predetermined pose-mask guidance hyperparameter that emphasizes the significance of masked region contents. Datasets. We assessed the performance of the proposed Stable-Pose as well as competing methods on five large-scale human-centric datasets including Human-Art [15] , LAION-Human [16] , UBC Fashion [43] , Dance Track [40] , and DAVIS [27] dataset. Details of the datasets and processing steps can be found in Sec. A.1.Implementation Details. Similar to previous work [44; 25; 46] , we fine-tuned our model on SD with version 1.5. We utilized Adam [17] optimizer with a learning rate of 1 × 10 -5 . For our proposed PMSA ViT module, we adopted a depth of 2 and a patch size of 2, where coarse-to-fine pose masks were generated using two Gaussian filters, each with a sigma value of 3 but with differing kernel sizes of 23 and 13, respectively. We will explore the effects of these hyperparameters with more details in Sec. 4.2. In the pose-mask guided loss function, we set an α of 5 as the guidance factor. We also followed [44] to randomly replace text prompts as empty strings at a probability of 0.5, which aims to strengthen the control of the pose input. During inference, no text prompts were removed and a DDIM sampler [39] with time steps 50 was utilized to generate images. On the Human-Art dataset, we trained all techniques, including ours for 10 epochs to ensure a fair comparison. On the LAION-Human subset, we trained Stable-Pose, HumanSD [16] , GLIGEN [18] and Uni-ControlNet [46] for 10 epochs, while we used released checkpoints from other techniques due to computational limitations. The training was executed using two NVIDIA A100 GPUs, with our method completing in 15 hours for the Human-Art dataset and 70 hours for the LAION-Human subset. This represents a substantial decrease in GPU hours compared to SOTA techniques. For instance, the T2I-Adapter requires approximately 300 GPU hours to train on a large-scale dataset. In contrast, our approach requires less than a quarter of that time and still delivers superior performance. We kept the same seed list for all techniques, including ours, during both training and inference time, to ensure fair comparison and reproducibility. More detailed information is provided in Sec. A.2.Evaluation Metrics. We adopt six metrics for evaluation, covering pose accuracy, image quality, and text-image alignment. For pose accuracy, we employ mean Average Precision (AP), Pose Cosine Similarity-based AP (CAP) [1] , and People Counting Error (PCE) [7] , measuring the accuracy between the provided poses and the pose results extracted from the generated images by the pretrained pose estimator HigherHRNet [6] . For image quality, we use Fréchet inception distance (FID) [10] and Kernel Inception Distance (KID) [5] . Both metrics measure the diversity and fidelity of generated images and are widely used in image synthesis tasks. For text-image alignment, we include the CLIP score [30] that indicates how well the CLIP model believes the text describes the image. Details of the evaluation metrics can be found in Sec. A.4. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We commit to release our source code on GitHub upon paper acceptance.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
results: 
 Quantitative and Qualitative Results. Table 1 reports The qualitative results obtained from Human-Art [15] and LAION-Human [16] are illustrated in Figure 4 . Consistent with the quantitative results, Stable-Pose demonstrates superior control compared to the other SOTA methods in both pose accuracy and text alignment, even in scenarios involving complex poses (the first row of Figure 4 , which is a back view of the figure), and multiple individuals (the third row of Figure 4 ), while the other methods fail to consistently maintain the integrity of the original pose instructions. This is particularly evident in dynamic poses (e.g., yoga poses and athletic activities), where Stable-Pose manages to capture the pose dynamism more faithfully than others.Stable-Pose as a generic adapter. Stable-Pose is designed as a lightweight generic adapter that can be easily integrated into any pre-trained T2I diffusion models to effectively enhance pose control. To further validate its generalizability, we conducted additional experiments by applying Stable-Pose on top of a pre-trained HumanSD [16] model. As shown in Table 2 , the inclusion of Stable-Pose considerably improved the baseline HumanSD by over 10% in AP and 12% in KID, highlighting its high generalizability and effectiveness in enhancing both pose control and image quality.Results on Varying Pose Orientations. Our experiments revealed that the current SOTA methods often faltered when tasked with creating images of humans in less common orientations, such as side or back poses. To investigate the capabilities of these methods in rendering atypical poses, we assembled a collection of around 2,650 images from the UBC Fashion dataset [43] , which comprises exclusively front, side, and back poses. We evaluated the checkpoints of each technique from the LAION-Human dataset to assess pose alignment. As reported in Table 3 , Stable-Pose significantly Results on the Outdoor and Indoor Poses. We extend the evaluation on both outdoor and indoor poseguided T2I generation. We selected approximately 2,000 frames from the DAVIS dataset [27] , which comprises videos of human outdoor activities, as our outdoor pose assessment. In addition, we randomly chose around 2,000 images from the Dance Track dataset [40] , which is characterized by its group dance videos where most videos were shot indoors with multiple individuals and complex poses, as indoor pose-alignment evaluation. As shown in Table 4 , the consistently highest AP and CAP scores achieved by Stable-Pose demonstrate its robustness in pose-controlled T2I generation across diverse environments, highlighting its potential as a backbone for pose-guided video generation. Further results can be found in Table A .6.We conducted a comprehensive ablation study of Stable-Pose on the Human-Art dataset, including the effectiveness of pose masks, coarse-to-fine design, pose-mask guidance strength, and the effectiveness of PMSA and its ViT backbone. Further ablations on model parameters can be found in Sec. A.6.Effectiveness of Pose Masks. To evaluate the impact of pose masks as input to our proposed PMSA and pose-mask guided loss function, we compared with removing them from the PMSA and/or from the associated loss function. As shown in Table 5 , incorporating pose masks in both PMSA and loss function significantly enhanced the performance in both pose alignment and image quality. Coarse-to-Fine Masking Guidance. The granularity of pose-masks is specified by the Gaussian kernels in Gaussian Filters, where a larger kernel generates coarser pose-masks. We compared the results of constant granularity, fine-to-coarse as well as coarse-to-fine setting. All experiments are based on a ViT with PMSA and depth of 2 and Gaussian Filters with fixed sigma σ = 3. Details can be found in Sec. A.3. As indicated in Table 6 , the coarse-to-fine approach consistently offers the best performance across metrics for pose alignment and image quality. This improvement is likely due to its progressive refinement from coarser to finer granularity in pose regions. By methodically narrowing the focus to more precise controllable areas, this strategy smoothly enhances the accuracy of pose adjustments and the overall quality of the generated images. Effectiveness of PMSA and its ViT backbone. Our PMSA incorporates additional pose masks, derived from pose skeletons that have been expanded using Gaussian filters. To evaluate the effectiveness of PMSA, we instead only integrated these augmented pose masks into ControlNet without our PMSA block. We explored two configurations: one in which the original pose skeleton was concatenated with one coarsely enlarged pose mask, denoted as ControlNet-PM1, and another where it was concatenated with both the coarsely and finely enlarged pose masks, referred to as ControlNet-PM2. Table 7 indicates that the enlarged pose masks yield only marginal improvements in ControlNet, suggesting that the substantial enhancements observed in Stable-Pose are primarily due to the innovative design of PMSA, rather than the additional pose masks input. Further, to validate the effectiveness of the ViT backbone in PMSA (PMSA w/ ViT), we replaced it with a conventional pose-masked self-attention module operating between residual blocks (PMSA w/ ResNet). We integrated the same pose masks in both configurations to ensure a fair comparison. Table 7 demonstrates that the ViT design in PMSA significantly outperforms the conventional approach. This substantiates the superior capability of ViT to capture long-range, patch-wise interactions among various anatomical parts of human poses to enhance the pose alignment.Pose Encoding in Stable-Pose. To further validate the design of pose encoding in Stable-Pose, we implemented an ablation study by removing either the pose encoder β or PMSA-ViT, retaining only one type of pose encoding. The results in Table 8 show that using only PMSA-ViT yields an AP of 36.48, which is expected due to the absence of color-coding information for distinguishing body parts. While using β alone increases the AP to 45.03. However, the most significant improvement is observed when integrating both local and global information encoding into the Stable-Pose architecture, achieving the highest AP of 48.88. Pose Masks During Inference. We incorporate the pose masks during inference by default to enhance pose control. To further validate their effectiveness, we additionally conducted experiments with removing the pose masks during inference. As shown in Table 9 , this led to approximately a 3% drop in AP. This could be due to two main reasons: 1) The pose masks provided additional guidance, thus enhancing control; 2) The inclusion of pose masks maintains consistency between the model's behavior during training and inference. Thus, including pose masks benefits pose control in the generation. ),'$3 ),' Pose-mask Guidance Strength in Loss. In our proposed loss function in Eq. ( 7), the parameter α, referred to as the pose-mask guidance strength, controls the intensity of penalization applied to the pose regions. We evaluated the impact of varying α from 1 to 10 on pose alignment and image quality, with the results presented in Figure 5 . Increasing α in our proposed loss means putting more attention on the foreground pose regions. However, when α is getting too large, it forces the model to learn irrelevant texture information like clothing, which negatively impacts training and slightly decreases AP. Despite this, Stable-Pose still outperforms others across an α range of 1-10. Notably, increasing α has a significant impact on FID, worsening it from 11.0 at α=5 to 13.0 at α=10. This indicates that focusing solely on the pose regions may decrease the quality of generated content in non-pose regions. Thus there exists a slight trade-off in selecting α to maintain both high pose accuracy and image quality, in which a value around 5 to 6 turns out to be optimal. 
discussion and conclusion: 
 We introduced Stable-Pose, a novel adapter that leverages vision transformers with a coarse-to-fine pose-masked self-attention strategy, specifically designed to efficiently manage precise pose controls during T2I generation. Stable-Pose outperforms current controllable T2I generation methods across five distinct public datasets, demonstrating high generation robustness across diverse environments and various pose scenarios. Notably, in complex scenarios involving rare poses such as side or back views and multiple figures, Stable-Pose exhibits exceptional performance in both pose and visual fidelity. This can be attributed to its advanced capability in capturing long-range patchwise relationships between different anatomical parts of human pose images through our intricate conditioning design. As a result, Stable-Pose holds significant potential in applications demanding high pose accuracy. It can be easily integrated into any pre-trained T2I diffusion models as a lightweight generic adapter to effectively enhance pose control. One limitation of Stable-Pose is its slightly longer inference time (Sec. A.2), primarily due to the integration of self-attention mechanisms within the ViT. In addition, despite excelling in pose control, Stable-Pose has yet to be evaluated with other conditions such as edge maps. Nevertheless, its design allows for straightforward adaptation to various external conditions, suggesting high potential for diverse applications.Broader Impacts: Stable-Pose's excellent pose control makes it a valuable tool in creating diverse artworks, animations, movies, and sports training programs. Additionally, it can be a reliable tool in healthcare and rehabilitation for correcting posture and preventing patients from musculoskeletal issues. It is worth noting that the scale of the LAION-Human subset and its data distribution closely align with those reported in the SOTA works, such as ControlNet [44] , which was trained on 200,000 images sourced from the Internet. Thus, ensuring a fair comparison is possible. Since none of the aforementioned video datasets are annotated with poses or textual descriptions, we employed the GPT-4 API for generating prompts and HigherHRNet [6] for deducing pose labels. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The error bars for our experimental results are not applicable.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Our paper have discussed both potential positive societal impacts and negative societal impacts of the work performed in Sec. 5.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: We commit to provide effective requirement for users to adhere to usage guidelines once we release our source code on GitHub upon acceptance.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have properly cited the original paper of the code package and dataset used in our work. We also included the information of license and terms of use of the datasets in Sec. A.1.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: To ensure a fair comparison, we trained and evaluated all the techniques on Human-Art dataset. On LAION-Human subset, due to computational constraints, we only trained our method, HumanSD, GLIGEN and Uni-ControlNet, while ControlNet and T2I-Adapter were evaluated using the released checkpoints. The rationale behind this was that HumanSD had been previously trained on a segment of LAION-Human, which might overlap with our validation set, leading to potential data leakage. GLIGEN had been trained with considerably more GPU hours compared to other techniques, posing a risk of an unfair comparison. Uni-ControlNet's available checkpoints are tailored for a multicondition model, however, our research is dedicated exclusively to pose control, necessitating further training. Conversely, for ControlNet and T2I-Adapter, the amount of data and GPU hours for training were on par with our methods, allowing for fair comparison using the checkpoints they provided.Training was conducted on two NVIDIA A100 GPUs. Our method required 15 hours to complete training on the Human-Art dataset and 70 hours for the LAION-Human subset. During training, the peak RAM usage for our technique was 25 GB. We reported the number of trainable parameters and training time on Human-Art dataset for each technique in Table A .1. Our technique has comparable number of trainable parameters and training time to ControlNet but produces significantly better results, highlighting the effectiveness of our model design. We also reported the inference speed of each technique in Table A .1, measured by the average seconds needed to generate an image. To ensure a fair comparison, these measurements were conducted on the same NVIDIA A100 GPUs. Due to the self-attention mechanism in the ViT, our method's inference speed is slightly slower than that of ControlNet.A common way to choose the kernel size k of a Gaussian Filter with sigma σ is k = 2 × ⌈3σ⌉ + 1. This ensures that the kernel captures the majority of the Gaussian function's weight. When using the kernels and sigmas as specified in the above equation, we observed that the difference between the generated coarse and fine masks was too large to yield satisfactory results. We measured this via the computation of the activation rate, reported in Table A .2. We applied the pose mask to the patches and then calculated the ratio of pose-masked patches to the total number of patches. We opted for an alternative method to control the blurring effect by fixing σ and adjusting k, which changes the number of pixels to which the Gaussian weights are applied. Larger values for k dilate the mask, resulting in coarser masks.In our experiment, we fix σ at 3 and then change k for obtaining different kernels. The activation rate of patches under the fixed σ and varying k is reported in Table A . 3 , where k = 23 is the largest kernel size for σ = 3, indicating that any larger k does not affect the activation rate. This size was chosen to generate the most coarse mask. To ensure smooth guidance of our pose-mask in the proposed PMSA, we selected kernel sizes with approximately one percent difference in activation rate. As a result, k = 13 was chosen to generate the fine mask. For our ablation study with 3 pose masks, we selected k = 9, which also yielded about a one percent difference in activation rate compared to k = 13. Details of this ablation study can be found in Sec. A.6.We define the evaluation metrics in detail here. FID assumes that the features extracted from real and generated images by the Inception v3 model [41] follow a Gaussian distribution. It measures the Fréchet distance between these distributions. KID, however, relaxes the assumption of a Gaussian distribution and calculates the squared Maximum Mean Discrepancy (MMD) between the Inception features of real and generated images using a polynomial kernel. Mean Average Precision (AP) computes the alignment between keypoints in real images and generated images. Poses of generated images are detected by the same Considering that images may contain multiple persons, we included People Counting Error (PCE) [7] as a metric, measuring the false positive rate when generating images featuring multiple people. The CLIP score measures the similarity between embeddings of generated images and text prompts, both of which are encoded by CLIP.Challenges in the Current Metrics. We followed the common metrics widely adopted in the current generative AI field, however, despite the rapid advancements in generative AI, existing metrics have not evolved to provide a more accurate evaluation [14] . Some of the issues are 1) CLIP score relies on cosine similarity between the model's semantic understanding and the given text, which may not align with pose assessments or the relevance of generated images. Additionally, this score is sensitive to the arrangement and composition of elements in images; even minor changes can result in significant fluctuations in the score, which may not accurately reflect the overall generative quality. [34] suggests a Dino-based score; 2) FID estimates the distance between a distribution of Inception-v3 features of real images and those of images generated by the generative models. However, Inception's poor representation of the rich and varied content generated by modern text-to-image models incorrect normality assumptions and poor sample complexity [14] . Thus, the FID score does not account for semantic correctness or content relevance-specifically pose-in relation to the specified text or conditions. Relying solely on FID and CLIP scores does not provide a comprehensive assessment of the generative model. Therefore, we further evaluated our method with a new state-of-the-art metric CMMD [14] , which is based on richer CLIP embeddings and the maximum mean discrepancy distance with the Gaussian RBF kernel. It is an unbiased estimator that does not make any assumptions on the probability distribution of the embeddings, offering a more robust and reliable assessment of image quality. As shown in Table A .4, our method achieves better CMMD value compared to HumanSD, demonstrating comparably high image quality. We reported detailed results for the UBC Fashion, DAVIS, and Dance Track datasets in Tables A.5&A.6, where KID is multiplied by 100 for readability. The results demonstrate that our Stable-Pose consistently provides better controllability over poses, even for challenging poses such as back poses.ViT Parameters. We studied the impact of different ViT parameters, including depth and patch sizes, as reported in of 2, using coarse-to-fine pose masks as guidance. While these masks provide smooth guidance, a higher depth with overly fine masks lose valuable information from previous attention layers. Table A .8 presents results for a ViT with PMSA of depth 2 and different patch sizes. Our default patch size of 2 yields the best results, likely because the latent encoding has typically lower dimension (here 64 × 64) compared to the dimensions of input high-resolution images. Larger patches may dilute learning of interconnections among anatomical parts for encompassing too much information.In Figure A.2,we present examples of attention maps extracted from our ViT. These maps are derived from the last block of the ViT and represent an average across all attention heads. The attention maps were approximately overlapped with the pose region, which corroborates the objectives of our proposed PMSA. This alignment underscores the efficacy of our model in focusing on relevant pose features, a critical aspect of our approach to improving model interpretability.Stable-Pose may face failure cases when generating the wrong number of people in very crowded scenes. Stable-Pose enhances SD's accuracy in pose-mask regions, whereas a pre-trained SD may still produce human-like figures in the background. For example, in the first row of We include additional qualitative results from the five datasets we evaluated in NeurIPS Paper Checklist 


Decision: This paper designs a coarse-to-fine pose-masked self-attention strategy and proposes a pose-guided image generation pipeline. After rebuttal, it received SA, 2 BA, and BR. 

The initial concerns focused on the design of the conditions (ypoD), performance (qFuv, RKmi), extreme cases (pWbK) and novelty (RKmi).
After going through the paper, the review, and the response, the AC values the key idea of the coarse-to-fine attention masking strategy with ViT, thinks most concerns have been addressed, and recommends the acceptance of the paper. The AC also urges the authors to append the conditions and the performance discussion, and highlight the difference compared to existing works in the revision.

Reviews: review: paper_summary: paper_summary: To obtain more refined pose condition control, especially for challenging condition (pose & text) generation, this paper designs a coarse-to-fine Pose-Masked Self-Attention (PMSA) module with the use of pose masks, fine-tunes Stable Diffusion with pose-mask guided loss, and ultimately achieves more precise control with little sacrifice of visual generation quality.
strength_weakness: strength_weakness: I carefully evaluated this module paper and did not find particular flaws except for no consideration of the multi-person condition. I will refer to the review suggestions of other reviewers and the author's feedback later.

[Method]

- (No consideration of different persons & parts) Surprisingly, the conditions may include multiple person, but PMSA does not explicitly consider modeling different people and different parts (in this sense the DensePose is a better choice), but only models the relationship between patches and semantics are considered by high-level pose decoder $\beta$. (Although the qualitative results shown look fine, there is no wrong association, such as A's hand is generated at B's hand position, or A's hand is generated at A's foot) I hope the author could provide more detailed explanations and discussion, even on the potential improvement to the proposed PMSA.

[Experiments]

- (Inferior PCE) Can the authors try to analyze the disadvantages of PCE (People Count Error) of the image generated by this method (Tab. 1)?

- (Add module architecture ablation studies) Since this is a work that proposes a module, the ablation studies of the module structure may still be helpful, e.g. the number of the PMSA-ViT block.

- (Add FPS results) Because the proposed PMSA is a plugin, to give readers a better comprehension, it is recommended that the author report the inference time (e.g. FPS) together with other metric results (although the author's claim method has disadvantages).

- (Add failures) It is also important to understand the boundaries of a method, e.g. the failure cases. It is recommended to visualize and analyze some of them.

- (Opinions on masked generation) Pose-mask guided is used as training loss in the paper. To further enhance control, it seems that the generation of humans can be similarly replaced within the mask area during the inference time. I am curious about the author’s opinion on this.
questions: questions: Please see the weaknesses.
limitations: limitations: The authors state the limitations in Sec. 5 as 1. longer inference time caused by the Self-Attention, 2. not evaluated with other condition guidances. It is recommended to add the visualization and discussion of failure cases.
paper_summary: paper_summary: The authors proposed a pose guided image generation pipeline. By introducing the PMSA ViT as pose encoder, the pipeline can generate more controllable images with corresponding 2D pose conditions.
strength_weakness: strength_weakness: - The proposed method is more like an incremental work of previous methods instead of a new innovative pipeline.
- Although the generated images follow the pose guidance better, the image quality(FID) and T2I Alignment(CLIP Score) are worse than those SOTA methods.
questions: questions: - Could the authors discuss more about why image quality(FID) and T2I Alignment(CLIP Score) are worse than those SOTA methods?
- Why the proposed methods are better at distinguish the left and right of the pose condition? During PSMA ViT, it seems like the authors treated all the keypoints the same, without any special design to improve the left-right consistency.
limitations: limitations: The limitation has been discussed in the paper.
paper_summary: paper_summary: The authors introduce a novel approach for pose-guided human image generation by developing a new pose encoder block. This block incorporates a coarse-to-fine attention masking strategy within a Vision Transformer architecture, leveraging the self-attention mechanism of Vision Transformers to analyze the interconnections among various anatomical parts of human pose skeletons. Furthermore, their loss function is specifically designed to place greater emphasis on the pose region. The model's performance is assessed both qualitatively and quantitatively, demonstrating promising results.
strength_weakness: strength_weakness: The coarse-to-fine approach does not appear promising based on the ablation study. The improvement from constant variation is negligible.

Limitations and failure cases are not adequately discussed.
questions: questions: The claim in lines 34-35, stating that pose-guided image generation requires paired human data during training, is not entirely accurate. Previous works, such as [1], [2], [3] etc., do not require paired information and can be trained in unpaired and unsupervised settings, though they do need appearance information. This requirement is not necessarily a disadvantage, as it suits applications like animation from a single image. It would be beneficial to revise this section to reflect these nuances and include the relevant citations.

How do the results appear for extreme poses, such as bending the upper body backward?

[1] A variational u-net for conditional appearance and shape generation. CVPR 2018
[2] Learning realistic human reposing using cyclic self-supervision with 3d shape, pose, and appearance consistency. ICCV 2021
[3]  Towards purely unsupervised disentanglement of appearance and shape for person images generation. 2020
limitations: limitations: Limitations are not adequately discussed.
paper_summary: paper_summary: The paper introduces Stable-Pose, a novel adapter model designed to enhance controllable T2I diffusion models by improving pose guidance, particularly in complex human pose conditions. Utilizing a coarse-to-fine attention masking strategy within a Vision Transformer , Stable-Pose effectively refines pose representation and improves performance over existing methods on the LAION-Human dataset.
strength_weakness: strength_weakness: 1. although this method shows good results in condition-following ability, but it has performance drop on image quality and text alignment.
2. compare to controlnet and other pose-guided t2i models, there is no significant contribution both in training recipe and architecture.
questions: questions: 1. It is interesting to see when pose-mask guidance strength is larger than 7, the AP for pose actually got dropped.
2. for all the examples shown in the paper, are they using the same seed for each of the method? (seed is important)
limitations: limitations: Stable-Pose has slightly longer inference times due to its use of self-attention mechanisms within the Vision Transformer (ViT). It has also not been thoroughly evaluated under various conditions, such as edge maps, leaving some robustness aspects unexplored. Further testing is needed to confirm its reliability and effectiveness across diverse scenarios.

Comments: title: Reply
Reply to the AC
Reply to the authors
Reviewer-author discussions
comment: We highly appreciate the efforts of the reviewer and would like to answer the follow-up questions as below:

**C#1**: We did not observe significant failures in associating joints with the wrong person, particularly when compared to other SOTA methods. We believe that our current local and global designs are sufficient for effectively modeling multiple individuals. Furthermore, we consider that explicitly encoding different people would not significantly enhance the performance in this context. For example, GLIGEN is a method that explicitly learns keypoint embeddings for different body parts across different individuals. However, it did not yield promising results in terms of APs or PCEs. Additionally, the explicit modeling approach in GLIGEN substantially increases the training time compared to other methods.

**C#2**: Thank you for the suggestion. We agree with adding quantitative results in the foreground area and will work on reducing the influence of false positives generated in the background in our future release.
I carefully read the reviewers' reviews and the author's careful and detailed rebuttal. Incluing the ablation study of local and global semantic modules, the experiments that can be combined with other backbones for plug-and-play, and the experimental supplement of masking inference. I want to further follow up and add some:

(**C#1**) The author does not seem to answer me directly. Have you observed failures in associating joints with the wrong person when using implicit conditional encoding color? Or does the author think that the current local and global designs are sufficient to avoid this problem? Will explicit consideration of the encoding of different people bring further gains?

(**C#2**) The author is suggested to add PCE or other quantitative evaluation experimental results of the foreground person number in future editions to verify the qualitative background false positive speculation.
We thank you for the reply; however, we respectfully hold different views. Our work fits the scope of NeurIPS, as it invites submissions that present new and original research on topics such as applications in vision, language, and deep learning (https://nips.cc/Conferences/2024/CallForPapers). We introduced a novel and general **adapter** that can be seamlessly integrated into various pre-trained text-to-image diffusion models like Stable Diffusion, HumanSD, and others to effectively enhance pose control, which is unique and not presented in prior works. While Uni-ControlNet may appear to be an incremental improvement over ControlNet, it introduces a significant new feature: achieving all conditions within a single adapter. Notably, Uni-ControlNet was accepted at NeurIPS 2023, highlighting the value of such innovations in NeurIPS.
Thanks for the author rebuttal, it address most of my concern. I would like to adjust my rating to borderline but slightly tend to reject side. The method and motivation in this paper are reasonable, but indeed it is kind of incremental compared to the previous method. I believe this paper is good enough to be accepted by a conference in computer vision, but whether it meets the standards for NeurIPS depends on the judgment of the area chair.
Thank you for your thoughtful feedback and support. We appreciate your careful consideration of the concerns and are glad to hear you view the paper positively.
I have carefully reviewed the other reviewers' evaluations and the full rebuttal from the authors. As I mentioned earlier, the innovative aspect of the paper lies in how it encodes pose information into the diffusion model. The results, including those presented in the rebuttal, demonstrate that their proposed method outperforms the state-of-the-art in adhering to that control signal. I agree with the other reviewers that this approach may cause a marginal degradation in performance in terms of FID evaluation. However, I do not believe that having an FID value approximately 1 point higher than other methods is a significant issue that should penalize the proposed method. In fact, I doubt this difference is even perceptible visually, and there could be several factors contributing to it. Overall, I believe the module proposed by the authors provides effective pose control, which requires a certain level of innovation, and there are sufficient experiments to support its efficacy. Therefore, I would support the acceptance of this paper. But of course, I am open to discussion if there are different opinions.
Thank you for taking the time to reply to my queries. I have carefully read the concerns about the other authors and the rebuttal. I still think this is a good paper to get accepted and would support it.
Dear Reviewers ypoD, qFuv, pWbK and RKmi

Would you please look at the rebuttal, discuss with the authors and finalize your score after discussion?

Thanks,

Your AC