Stable-Pose: Leveraging Transformers for Pose-Guided Text-to-Image Generation
Authors: Jiajun Wang
MORTEZA GHAHREMANI
Yitong Li
Björn Ommer
Christian Wachinger
Venue: NeurIPS 2024 poster (2024)
Keywords: Pose-guided text-to-image (T2I)
Diffusion models
Stable Diffusion
vision Transformers
Primary area: diffusion_based_models
TL;DR: We present Stable-Pose, a novel adapter model that incorporates a coarse-to-fine attention masking strategy into a vision Transformer (ViT) for gaining accurate pose guidance in T2I diffusion models.

Abstract: Controllable text-to-image (T2I) diffusion models have shown impressive performance in generating high-quality visual content through the incorporation of various conditions. Current methods, however, exhibit limited performance when guided by skeleton human poses, especially in complex pose conditions such as side or rear perspectives of human figures. To address this issue, we present Stable-Pose, a novel adapter model that introduces a coarse-to-fine attention masking strategy into a vision Transformer (ViT) to gain accurate pose guidance for T2I models. Stable-Pose is designed to adeptly handle pose conditions within pre-trained Stable Diffusion, providing a refined and efficient way of aligning pose representation during image synthesis. We leverage the query-key self-attention mechanism of ViTs to explore the interconnections among different anatomical parts in human pose skeletons. Masked pose images are used to smoothly refine the attention maps based on target pose-related features in a hierarchical manner, transitioning from coarse to fine levels. 
Additionally, our loss function is formulated to allocate increased emphasis to the pose region, thereby augmenting the model's precision in capturing intricate pose details.    We assessed the performance of Stable-Pose across five public datasets under a wide range of indoor and outdoor human pose scenarios. Stable-Pose achieved an AP score of 57.1 in the LAION-Human dataset, marking around 13\% improvement over the established technique ControlNet.  The project link and code is available at https://github.com/ai-med/StablePose.

Stable-Pose: Leveraging Transformers for Pose-Guided Text-to-Image Generation
Jiajun Wang"!, Morteza Ghahremani Yitong Li Bjöm Ommer and Christian Wachinger
Lab for Al in Medical Imaging, Technical University of Munich (TUM), Germany Comp @ LMU Munich, Germany
Munich Center for Machine Learning (MCML), Germany
Abstract
Controllable text-to-image (T2I) diffusion models have shown impressive per- formance generating high-quality visual content through the incorporation various conditions. Current methods, however, exhibit limited performance when guided by skeleton human poses, especially in complex pose conditions such as side rear perspectives of human figures. To address this issuc, present Stable-Poss Pose, a novel adapter model that introduces a coarse-to-fine attention masking strategy into vision Transformer (ViT) to gain accurate pose guidance for T21 models. Stable-Poss Posee is designed to adeptly handle pose conditions within pre-trained Stable Diffusion, providing refined and efficient way of aligning pose representation during image synthesis. We leverage the query-key self-attention mechanism of VIT' explore the interconnections among different anatomical parts in human pose skeletons. Masked pose images are used to smoothly refine the attention maps based on target pose-related features in hicrarchical manner, transitioning from coarse to fine levels. Additionally, our loss function is formulated to allocate increased emphasis the pose region, thereby augmenting the model's precision capturing intricate pose details. We assessed the performance of Stable- across five public datasets under wide range indoor and outdoor human pose scenarios. Stable- Pose achieved an AP score of 57.1 1 in the LAION- Human dataset, marking around 13% improvement over the established technique ControlNet. The project link and code are available at https: thub. com/ ai diime//StaabePose
Introduction
Pose-guided text-to-image (T2I) generation holds immense potential for swiftly producing photo- realistic images that exhibit contextual relevance and accurate posing through the integration text prompts and pose instructions. The kinematic or skeleton pose provides set of key points (joints) that represent the skeletal framework of the human body (shown Figure A.I Despite the sparsity, skeleton-pont pose data offers sufficient details of human poses with high flexibility and computational efficiency for T2I generation in various applications such as animation, robotics, sports training, and e- e-commerce, making it user-friee seer-friendly and ideal for real-time applications (2I). Juxtaposed with other forms of pose information like volumetric pose with dense content, skeleton pose capable of conveying heightened articulation information, facilitating intuitive interpretation and flexible manipulation human poses (28 20
Traditional pose-guided human image generation methods require a source image during training Such methods, while offering
for dictating the style of the generated images [22 23
42
38
47
Equual Contribution. (jiajun. wang, morteza. ghahremani, yi_tong. 1i)m
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Orienaatio mismatch
fjjuriinn jjjuttionrrraattin prosittatinn auaiest a with fackinny, fucuring a modi in Sack.
Tats yook sleer Jairr-iin Sojures.
2014a1 carfo-boad, wfut wear mard!
vaatuue purformin it cary nore
Inaccurate pose
Mosiin ieronte body parts
Slae posing sqainut a Sackiings.
fisters jylttg Baxing ring
(x) Pose
T21-Acapter
(s) ControlNie
Statle-Pose (Ouns
(x) Pose
12h-Adapper
(c) ControlNe
(d) Statle-Poee (Duns)
Figure I: Stable-Pose leverages the patch-wise attention of VITs to address the complex pose condi- tioning problem in T2I generation, showing superior performance compared to current techniques.
control over the appearance, limit the flexibility and diversity of the output and depend heavily on the need for paired source-target data during training. contrast, recent advancements in controllable T2I diffusion models have shown the potential to eliminate the need for source images and allowed for higher creative freedom by relying on text prompts and external conditions 44 46 25 I8].
Enabling more versatile visual content creation, these methods often face challenges in precisely aligning conditional images with sparse representations such skeleton pose data, especially when dealing with complex pose scenarios like those depicting the back or side views of human figures (Figure T). Moreover, existing methods may also fail to maintain accurate body proportions, resulting in an unnatural appearance of the body.
To address the insufficient binding of sparse pose data in T2I generation models, a potential strategy is to capture long range patch-wise relationships among various anatomical parts of human poses. In this paper, we introduce Stable-Pose that integrates vision Transformers (VTT) into pre-trained T2I diffusion models like Stable Diffusion (SD) [33, with the goal of improving pose control by capturing patch-wis wise relationships from the specified pose. In Stable-Pose, the learnable attentions adhere an innovative coarse-to- fine masking approach, ensuring that pose conditioning is directed toward the relevant pose areas while preserving the diversity of the overall image. To further enhance this effect, pose- bose-mask guided loss is introduced to optimize the fidelity of the generated images in adherence to the given pose instructions. We evaluated Stable-Poss Pose across five distinct datasets, covering indoor and outdoor image/video datasets. Compared to the state-of-the-art methods, Stable- achieved the highest accuracy and robustness in pose adherence and generation fidelity, making a promising solution for enhancing pose control in T2I generation. We further performed comprehensive ablation studies to demonstrate the effectiveness our design. In summary, our contributions are:
Addressing the challenge of generating photo-realistic human images in pose-guided T21 by integrating a novel ViT, achieving highly accurate synthesis in pose adherence and image fidelity, even under challenging conditions.
Introducing hierarchical integration of pose masks for coarse-to-fine guidance, with a novel masked self self-attention mechanism and pose-mask guided loss function. Stable- is designed as lightweight adapter that can be easily integrated into any pre-trained T2I diffusion models to effectively enhance pose control.
Stable-Pose effectively learns to preserve intricate human shape structures and accurate body proportions, achieving exceptional performance across five publicly available datasets, encompassing both image and video data.

---

Text
X+
Diffusion Process
Denoising U-Net
Cy
Sa
PASSA 3
21
2p
2--1
Computation of Masked Attention F7
Frines TTonssTTTiiaaeee Troinable
Consodation
Zero
Attenion
Query
Misked Posc Phich
The Stable Diffusion architecture with Stable-Pose: operating on the pose skeleton image, Stable-Pose integrates a trainable VIT unit into the frozen-weight Stable Diffusion [ to improve the generation of pose-get aase-guided human images.
Figure 2:
2 Related Work
Pose-Guided Human Image Generation: Traditional pose-guided human image generation takes source image and pose input, aiming to generate photo-realistic human images in specific
poses while preserving the appearance from the source image. Prior works (22] 23 42 38
primarily utilized generative adversarial network (GAN) variational autoencoder (VAE), treating the synthesis task as conditional image generation. Zhu et al. [47] integrated attention mechanism for appearance optimization in Pose Attention Transfer Network (PATN). Zhang al. 45 implemented a Dual-task Pose Transformer Network (DPTN), using Transformer module to incorporate features from two tasks: auxiliary source image reconstruction task and the main pose-guse pose-guided target image generation task, thereby capturing the dual-task correlation. With the recent emergence of diffusion models [2). Bhunia et al. 4 proposed a texture diffusion module transfer texture patterns from the source image to the denoising process. Moreover, classifier-free guidance [ is applied to provide disentangled guidance for style and pose. Shen al. (37 proposed a three-stage synthesis pipeline which progressively performs global feature extraction, target prediction, and refinement with three diffusion models. While the source image offers control over appearance, a limitation arises from the necessity of paired source-target data during training. Text, however, obviates this need and offers higher flexibility and diversity for the synthesis. Thus, incorporating text conditions for pose- possesskeleton- akeleton-guided guided human image generation shows significant promise [19] [6].
Controllable Diffusion Models: Large-scale T2I diffusion models (33
31
32
35
26 excel
creating diverse and high-quality images, yet they often lack precise control with solely text-based prompts. Recent studies aim enhance the control of T2I models using various conditions such
as canny edge, sketch, and human pose 13 44 25 broadly classified into two groups: training the entire T2I model or developing plug-in adapters for pre-trained T2I models. As in the first group. Composer [3] trains a diffusion model from scratch with a decomposition
18 46 16:
24 29]. These approaches can be
paradigm, enabling multi-control capability. HumanSD [16] fine-tunes the entire SD model using deatmap-guidedd loss tailored for pose control. In contrast, T21-Adapter (25] and GLIGEN (18) train lightweight adapters whose outputs are incorporated into the frozen SD. Similarly, ControlNet 44 employs trainable copy of the SD encoder to encode conditions for the frozen SD. Uni-ControlNet [46) introduces uni-adapter for multiple conditions
injection to the trainable branch in multi-scale manner. ControlNet+ (24 proposes to improve controllable generation by explicitly optimizing the cycle consistency between generated images and conditional controls. Our method aligns with the latter category, it is characterized by reduced training time, cost-effecttiveness, and generalizability.
3 Proposed Method
Stable-Pose has a trainable VIT unit that integrated into the pre-trained T2I diffusion models to direct diffusion models toward the conditioned pose. In latent diffusion models (LDMs) 33] pre-trained encoder & and decoder D are employed transform an input RGB image x E R #x with height H and width W into latent space with reduced spatial dimensions and vice versa. The diffusion process then efficiently conducted in the down-scaled latent space. During the training, the forward process in diffusion models adds noise to the encoded RGB image Zo E(x) to generate

---

Pose go
PMSS WT
Time Enbbeddng
Pose- Masked (PMSA)
Projection
K
Gasaa Filter
4
Q
F
L
C
Pese masked Self-Attertion
1
Charaa-do-rrw
Feed Forward
Pose maskeed Seft Attention
Pous Pedder
Feed Forward
Partter Enbeeinng
Mente Eoddas
Figure 3: Stable-Pose consists of pose encoder 3g and coarse-to-fine Pose-Masked Self-Aitention (PMSA) ViT Fg for seeking the patch-wise relationship of human parts. PMSA restricts attention to embedding tokens within specific attention mask to ensure that each embedding token can only attend to pose embedding tokens, not non-pose ones.
its noisy sample z, Rxxxx with height h, width w, and channel C via
2, 0g + v I0, 1=
where ā₂ is pre-determined hyperparameter that controls the noise level at step t. The reverse process of diffusion models, so-called denoising, learns the statistics of the Gaussian distribution at each time step. The reverse process is formulated as:
pe (z)) = (Z t), Eg (Z)
(2)
As shown in Figure 2, the denoising network En adopts UNet backbone, which is equipped with Stable-Pose for augmenting the latent encoding given the input conditional pose image. Let (o(2,, p). & E (1.... ... , T), represent a T-step denoising UNet with gradients V0 over a batch and input text prompt p. The conditional LDM is learned through T steps by minimizing L (Zz, To (p) Up (Ze, ()) [13]. where Stable-Pose Up conditions the latent
Ez, (x...))) [lle encoding on the input pose skeleton yp E Rhxxx3, and To is a text encoder that maps the text prompt p to an intermediate sequence. The proposed framework detailed Figure 3 Stable-Pose aims at improving the frozen decoder in the UNet of SD to condition the input latent encoding 2, on the conditional pose image yp:
z + Up Uo(Z), :p).
In Stable-Pose the pose image and the given latent encoding Z, are processed by two main blocks named Pose-Masked Self-Attention (PMSA) Fa and pose encoder Ae in such way that
vo(2,, :p) = Fo(z,, (o(())
where the pose encoder 3e provides high-level features for the input pose while PMSA Fg explores the patch-wise relationship across input z, using self-attention mechanism and the binary-masker version of the pose image. PMSA employs a coarse-to-fing framework that provides additional guidance the latent encoding, directing it towards attending the conditioned pose. We detail each block the subsequent sections. The updated latent encoding z is subsequently fed through the encoder SD, followed by a series of zero convolutional blocks, a structural resemblance the architecture employed in ControlNet [44] for ensuring a robust encoding of conditional images.
Pose encoder 3g R xWxx3 x3 R xxxx a trainable encoder that maps input pose skeleton image into a feature with height h, width w, and channel f. To this end, we employ combination of six convolutional layers with SiLU activation layers (8), downsampling the input pose image by factor of 8. A convolooolltl layer is added the end. As the input pose image contains sparse information, this straightforward pose encoder is sufficient for accurate encoding of the skeleton pose.
PMSA Fø Ri xhxw + R/ xxxxx seeks the potential relationships between patches within the latent encoding 2,. The interconnections of various parts of the human body suggest the presence of

---

Algorithm Generation of the attention mask for PMSA
1: procedure ATTNMASK(ma)
2: my: Patchified and binarized pose masks of size (N, H, W)
3: My f- Flatten mg along the last two dimensions
4: N < ma.phappe()
5:
L - mm..hhpeee
6:
attn attn_mas mosk initialize tensor of size (N, L, L) with entries set fin
7:
for i E [0, N do
8:
indices find indices where m& [i]
9:
attn_mase [i]] [indices + 0
attn_maskl [i] indices 11: end for
return altn_mask
13: end procedure
cohesive relationships among them. To capture this, leverage the self-attentio mechanism. We divide Ze into L non-overlapping patches of size p X i.c. 2t {z; .....
Zz Ze E IRF xin
(L) (1)
PMSA projects the patch embeddings into Query Q We, Key K W K. and Value V = 2, W v via three learnable weight matrices Wa E f WR K E B/axf and Wv E Blaxl,, respectively. Then it computes the attention scores between all patches via
QK T
As = attention (Q, K, V, ms) sof tmax
+ AttnMask(ms) V.
(5)
Vfg
In this equation, my denotes a binary mask derived from the input pose image, which is expanded by AttnMask(-) for being used in the self-attention computation (see Figure 3p. m, is obtained by converting the pose image into a binary mask and downsampling it into the same size of the latent vector zt. The resultant mask is then dilated by a Gaussian kernel of length k. The dilated-binary mask my is then partitioned into L non-ovenne non-overlapppnn patches of size p x p. The patches containing pose are labelled as I while others are marked as We form resulting L L attention mask based on these L patches through the function A AttnMask(-) )) As illustrated in Algorithm ! for patch entries that correspond to pose regions, both the respective row and column in the mask are set For all other regions not associated with pose, we assign an extremely small integer value. The attention mask helps to enhance the focus of PMSA on the pose-specific regions.
We implement a sequence of N blocks of ViTs, each associated with a unique pose mask, arranged in coarse-to-fine progression the latent encoding. This approach gradually steers the latent encoding towards conforming to the specified pose condition. If = {ki,.... JiN) } denotes a set of Gaussian kernels where b > > KN. then the coarse-to-ing self-attention obtained via
A attention (Q, K. V, V,m,)
(6)
A, attention (QA,-1: K KAn-19 VAn-1, MR.)) for (2.....,M) .N--
Each encoding An undergoes further processing by a Feed Forward unit, with the resulting AN integrated into the feature from the pose encoder Sg, as shown in Figure 3 The Feed Forward block consists of two linear transformations
[ combined with dropout layers, followed by ReLU
non-linear activation functions
2
Pose-mask guided loss criterion: The training of SD models requires high costs in hardware and datasets. Therefore, plug-in adapters for the frozen SD models, such as Stable-Pose, can enhance training efficiency by eliminating the need compute gradients
maintain optimizer states for SD parameters. Instead, the optimization process focuses on improving Stable-Pose parameters. The loss in Stable-Pose aligns with the coarse-to-fine approach and is defined follows:
E
(E - eo (Zt, t, T (p) Ug (Ze, pp) @ m)
a. 2.p.p.ee
(7)
E N11.
[I] (z,, t, (p) vo (Ze, ()) mkx
Here, a 2 represents predetermined pose-mask guidance hyperparameter that emphasizes the significance of masked region contents.
5

---

Table Results on Human-Art dataset and LLAIIN-Humane subset. Methods with * are evaluated on released checkpoints.
Pose Accuracy
Image Quality
T2I Alignment
Dataset
Method
AP
C
PCE 4
FID
KID U
CLIP-score t
SD*
0.24
55.71
2.30
11.53
3.36
33.33
T2I-Adapter
27.22
65.65
1.75
11.92
2.73
33.27
ControlNet
39.52
69.19
1.54
11.01
2.23
32,65
Human-Art
Uni Co-CControlNet
41.94
69.32
1.48
14.63
2.30
32.51
GLIGEN
18.24
69.15
1.46
32.52
HumanSD
44.57
69.68
1.37
10.03
2.70
32.24
Stable-Pose (Ours)
48.88
70.83
1.50
11.12
2.35
32.60
SD*
0.73
44,47
2.45
4.53
4,80
32.32
T2I-Adapter*
36.65
63.64
1.62
6.77
5.44
32.30
ControlNer*
44.90
66.74
1.55
7.53
6.53
32.31
LAION-Human
Uni-ControlNet
50.83
66.16
1.41
6.82
4.52
32.39
GLIGEN
19.65
66.29
1.40
32.04
HumanSD
50.95
65.84
1.25
5.62
7.48
30.85
Stable-Pose (Ours)
57.11
67.78
1.37
6.25
4.50
32.38
4 Experimental Results
Datasets. We assessed the performance of the proposed Stable-Pose as well competing methods on five large-scale human-centric datasets including Human-Art (15), LAION-Human [16), UBC Fashion 43 Dance Track [40] and DAVIS 27 dataset. Details of the datasets and processing steps can be found in Sec. A.
fine-tuned model SD with version 1.5. We utilized Adam [17 optimizer with a learning rate of x 10 For our proposed PMSA ViT module, we adopted depth of 2 and a patch size of where coarse-to-fine pose masks were generated using two Gaussian filters, each with a sigma value 3 but with differing kernel sizes of 23 and 13, respectively. We will explore the effects of these hyperparameters with more details in Sec.
Implementation Details. Similar to previous work [44 25 46).
4.2 In the pose-mask guided loss function, we set an a of as the guidance factor. We also followed
to randomly replace text prompts empty strings a probability of 0.5, which aims to strengthen the control of the pose input. During inference, no text prompts were removed and a DDIM sampler [39] with time steps 50 was utilized to generate images. On the Human-Art dataset, we trained all techniques, including ours for 10 epochs to ensure a fair comparison. On the LAION- Human subset, we trained Stable-Pose, HumanSD [16), GLIGEN (18) and Uni-ControlNet [46] for 10 epochs, while we used released checkpoints from other techniques due to computational limitations. The training was executed using two NVIDIA A100 GPUs, with our method completing in 15 hours for the Human-Art dataset and 70 hours for the LAION- subset. This represents substantial decrease GPU hours compared to SOTA techniques. For instance, the -21-Aeepter requires approximately 300 GPU hours to train a large-scale dataset. In contrast, our approach requires less than a quarter of that time and still delivers superior performance. We kept the same seed list for all techniques, including ours, during both training and inference time, to ensure fair comparison and reproducibility. More detailed information is provided in Sec. A.2
4
Evaluation Metrics. We adopt six metrics for evaluation, covering pose accuracy, image quality, and text-image alignment. For pose accuracy, we employ mean Average Precision (AP), Pose Cosine Similarity- -based AP (CAP) [1]. and People Counting Error (PCE) 77,, measuring the accuracy between the provided poses and the pose results extracted from the generated images by the pretrained pose estimator HigherHRNet [6]. For image quality, we use Fréchet inception distance (FID) [IO] and Kernel Inception Distance (KID) [.. Both metrics measure diversity and fidelity of generated images and are widely used in image synthesis tasks. For text-image alignment, we include the CLIP score (30) that indicates how well the CLIP model believes the text describes the image. Details of the evaluation metrics can be found in Sec. A.4

---

Garage fitt, fegurine of girl girlwwtti gum toy of small platform.
Nigital girl yollow dress flying through the with butteflies.
Btuldiin boy and gir'te's tranc's fouthh strony Early Bird Mott
4 fiafefiy yoga exrretso on Feach, Stook Phieto.
Pose
12llAAdaptor
ControlNat
Uni-ContraiNed
GLIGEN
(1) HumanSD
Figure 4: Qualitative results of SOTA techniques and our Stable-Pose Human-Art (first rows) and LAION-Human (last two rows). An illustration of the pose input is shown in Figure A.
4.1 Results
Quantitative and Qualitative Results. Table reports the quantitative results on both datasets among different methods. We reported the mean Average Precision (AP), Pose Cosine Similarity- based AP (CAP), People Count Error (PCE). Fréchet Inception Distance (FID), Kernel Inception Distance (KID), and the CLIP Similarity (CLIP-score) KID is multiplied by 100 for Human Art and 1000 for LAION-Humal for readability. Table [ (48.87 on Human-Art and 57.41 LAION-Human) and CAP (71. .04 Human-Art and 68.06 LAION-Human), surpassing the SOTA methods by more than 10%. This highlights Paable-Pose's superiority in pose alignment. In terms of image quality and text-image alignment, Stable-Poss achieved comparable results against other methods, with only marginal discrepancy in FID/KID scores, yet the difference is negligible and the resulting quality remains high. Overall, these results underscore Stable-Pose's exceptional accuracy and robustness in both pose control and visual fidelity.
shows that Stable- -Poe achieved the highest AP
The qualitative results obtained from Human-Art [1] and LAION Human [16] illustrated in Figure Consistent with the quantitative results, Stable-Pos Pose demonstrates superior control compared to the other SOTA methods in both pose accuracy and text alignment, even in scenarios involving complex poses (the first row of Figure 4 which is back view of the figure), and multiple individuals (the third row of Figure 4). while the other methods fail to consistently maintain the integrity of the original pose instructions. This is particularly evident in dynamic poses (e.g., yoga poses and athletic activities), where Stable-Pose manages capture the pose dynamism more faithfully than others.
Stable-Pose as generic adapter. Stable-Pose is designed as a lightweight generic adapter that can be easily integrated into any pre-trained T2I diffusion models to effectively enhance pose control. To further validate its generalizability, we conducted additional experiments by applying Stable-Pose on top of pre-trained HumanSD [16] model. As shown in Table 2 the inclusion of Stable-Poss considerably improved the baseline HumanSD by over 10% in AP and 12% KID, highlighting its high generalizability and effectiveness in enhancing both pose control and image quality.
Results on Varying Pose Orientations. Our experiments revealed that the current SOTA methods often faltered when tasked with creating images of humans in less common orientations, such as side or back poses. To investigate the capabilities of these methods rendering atypical poses, we assembled a collection of around 2,650 images from the UBC Fashion dataset 43 which comprises exclusively front, side, and back poses. We evaluated the checkpoints each technique from the LAION- Human dataset to assess pose alignment. As reported in Table Stable-Pose significantly

---

Table 2: Stable-Pose lightweight adapter pre- pre-trained HumanSD model.
Method
AP 1
CAP 1
PCE 4
FID 4
KID 4
CLIP-score f
HumanSD
44.57
69.68
1.37
10.03
2.70
32.24
Stable-Pose
48.88
70.83
1,50
11.12
2.35
32.60
HumanSD+Stable-Ponn
49.24
71.01
1.42
10.42
2.37
32.16
outperforms other methods in recognizing and generating humans all pose orientations, especially for rarer poses in side and back views, which surpasses the other methods by around 20% in AP. This further validates the robust controllability of Stable-Pose.
Table 3: Results of varying pose orientations on the UBC Fashion dataset. We report mean Average Precision (AP) across different methods for three orientations: front, side, and back.
Orientation T21-Adapter ControlNet Uni-ControlNet GLIGEN HumanSD Stable-Pose (Ours)
Front
72.20
74.64
79.47
73.97
76.83
87.269.80851
8008
Side
36.80
52.83
58.26
45.32
57.09
69.7619
19.7%%
Back 6.03
23.68
19.97
4.45
11.05
29.08 22.80%
Results on the Outdoor and In- door Poses. We extend the evalua- tion on both outdoor and indoor pose- guided T2I generation. We selected approximately 2,000 frames from the DAVIS dataset [27], which comprises videos of human outdoor activities, our outdoor pose assessment. In addition, we randomly chose around 2,000 images from the Dance Track dataset [40], which is characterized by its group dance videos where most videos were shot indoors with multi- ple individuals and complex poses, as indoor pose-alignment evaluation. As shown Table the consistently highest AP and CAP scores achieved by Stable- demonstrate its robustness in pose-controlled T2I generation across diverse environments, highlighting its potential as a backbone for pose-guided video generation. Further results can be found in Table
Table 4: Results on the outdoor poses from the DAVIS dataset and the indoor poses from the Dance Track dataset.
DAVIS
Dance Track
Method
AP
CAP 1
AP1
CAP f
T2I-Adapter
20.28
60.76
10.36
72.38
ControlNet
30.13
60.81
16.45
73.16
Uni-ControlNet
37.64
60.57
25.22
73.62
GLIGEN
12.17
59.87
5.61
72.57
HumanSD
38.32
60.59
24.13
69.93
Stable-Pose (Ours)
42.87
62.43
28.58
74.97
4.2 Ablation Study
We conducted comprehensive ablation study of Stable-Pose on the Human-A Art dataset, including the effectiveness of pose masks, coarse-to-fine finfine design, pose-mask guidance strength, and the effectiveness of PMSA and its VIT backbone. Further ablations on model parameters can be found in Sec. A.6
Effectiveness of Pose Masks. To evaluate the impact of pose masks input to our proposed PMSA and pose-mate guided loss function, we compared with removing them from the PMSA and/or from the associated loss function. As shown in Table incorporating pose masks in both A and loss function significantly enhanced the performance both pose alignment and image quality.
Table 5: Results of the pose mask ablation on Human-Art dataset.
Pose mask
AP
CAP
PCE 1
FID
KID
CLIP-score
in PMSA
loss
X
39.40
69.18
1.55
13.94
2.56
32.63
X
44.50
70.51
1.51
14.24
2.61
32.58
X
V
45.39
70.18
1.56
13.17
2.62
32.65
/
48.88
70.83
1.50
11.12
2.35
32.60
Coarse-to-Fine Masking Guidance. The granularity pose-masks is specified by the Gaussian kernels in Gaussian Filters, where larger kernel generates coarser pose-masks pose-masks. We compared the

---

results of constant granularity, tine-to-coarss as well as coarse-to-fine setting. All experiments are based on a ViT with PMSA and depth of 2 and Gaussian Filters with fixed sigma 3. Details can be found in Sec. A.3 As indicated in Table 6 the coarse-to-fine approach consistently offers the best performance across metrics for pose alignment and image quality. This improvement is likely due to its progressive refinement from coarser to finer granularity in pose regions. By methodically narrowing the focus to more precise controllable areas, this strategy smoothly enhances the accuracy
of pose adjustments and the overall quality of the generated images.
Table 6: Results of different Gaussian kernels k used for pose masks in Stable-Pose.
Pose mask granularity
AP
CAP 1
PCE 4
FID 4
KID 4
CLIP-score f
Constant (23, 23)
48.10
70.77
1.59
12.55
2.59
32.62
Fine-to-coarse (13, 23)
47.86
70.84
1.57
12.48
2.54
32.54
Coarse-to-fine (23, 13)
48.88
70.83
1.50
11.12
2.35
32.60
Effectiveness of PMSA and its VIT backhone. Our PMSA incorporates additional pose masks, derived from pose skeletons that have been expanded using Gaussian filters. To evaluate the ef- fectiveness PMSA, we instead only integrated these augmented pose masks into ControlNet without our PMSA block. We explored two configurations: one in which the original pose skeleton was concatenated with one coarsely enlarged pose mask, denoted as ControlNet-PMM and another where it was concatenated with both the coarsely and finely enlarged pose masks, referred to as ControlNet-PM2 Table 7 indicates that the enlarged pose masks yield only marginal improvements in ControlNet. suggesting that the substantial enhancements observed in Stable-Pose primarily due to the innovative design of PMSA, rather than the additional pose masks input.
Table 7: Ablation study on the effectiveness of PMSA and its ViT design. We show the performance of ControlNet with additional pose masks input, and PMSA with ResNet or ViT as backbone.
Method
AP1
CAP t
PCE
FID +
KID 4
CLIP-score
ControlNet
39.52
69.19
1.54
11.01
2.23
32.65
ControlNet-PMM
39.24
68.45
1.50
11.52
2.26
32.70
ControlNet-PMP
40.73
69.27
1.49
11.63
2.24
32.67
PMSA w/ ResNet
45.24
70.09
1.56
13.48
2.60
32.58
PMSA w/ VIT (Ours)
48.88
70.83
1.50
11.12
2.35
32.60
Further, to validate the effectiveness of the ViT backbone in PMSA (PMSA w/ ViT), we replaced it with conventional pose-masked self-attention module operating between residual blocks (PMSA w/ ResNet). We integrated the same pose masks in both configurations to ensure a fair comparison. Table demonstrates that the ViT design in PMSA significantly outperforms the conventional approach. This substantiates the superior capability of ViT to capture long-range, pisch-wise interactions among various anatomical parts of human poses enhance the pose alignment.
Pose Encoding in Stable-Pose. To further validate the design of pose encoding in Stable-Pose. we implemented ablation study by removing either the pose encoder 3 PMSA-VES VIT, retaining only one type of pose encoding. The results in Table 8 show that using only PMSA- RMSA-ViT yields an AP of 36.48, which is expected due the absence of color-coding information for distinguishing body parts. While using 3 alone increases the AP to 45.03. However, the most significant improvement is observed when integrating both local and global information encoding into the Stable-Pose architecture, achieving the highest AP of 48.88.
Table 8: Ablation study on the pose encoding design in Stable-
Method
AP t
CAP t
PCE 4
FID 4
KID 4
CLIP-score f
w/o Encoder
36.48
68,91
1.55
11.17
2.76
31.90
w/o PMSA-ViT Encoder w/ both 8 & PMSA-VIT Encoder
45.03
70.38
1.52
13.67
2.49
32.53
48.88
70.83
1.50
11.12
2.35
32.60
Pose Masks During Inference. We incorporate the pose masks during inference by default to enhance pose control. To further validate their effectiveness, we additionally conducted experiments with removing the pose masks during inference. As shown in Table 9 this led to approximately

---

3% drop in AP. This could be due to two main reasons: 1) The pose masks provided additional guidance, thus enhancing control: 2) The inclusion of pose masks maintains consistency between the model's s behavior during training and inference. Thus, including pose masks benefits pose control in the generation.
Table 9: Ablation study on the presence of pose masks during inference.
Method
AP
CAP
PCE 4
FID
KID J
CLIP-score 1
w/o mask in inference
45.93
70.51 70.83
1.52 1.50
13.11
2.55
32.68
w/ mask in inference
48.88
11.12
2.35
32.60
Pose-mask Guidance Strength in Loss. In our proposed loss function in Eq. (7). the parameter a, referred to as the pose-mask guidance strength, controls the intensity of penalization applied to the pose regions. We evaluated the impact of varying a from to 10 on pose alignment and image quality, with the results presented in Figure Increasing a in our proposed loss means putting more attention on the foreground pose regions. However, when a is getting too large, forces the model to learn irrelevant texture information like clothing, which negatively impacts training and slightly decreases AP. Despite this, Stable-Pose still outperforms oth- ers across an a range of 1-10. Notably, increasing has significant impact on FID, worsening it from 11.0 at a=5 13.0 a=10. This indicates that focusing solely on the pose regions may decrease the quality of generated content in non-pose regions. Thus there exists slight trade-off in selecting a to maintain both high pose accuracy and image quality, in which a value around 5 to 6 turns out to be optimal.
45-
Pose-masks Guidance Strangth (e)
Figure 5: strength the loss function.
Ablation on pose-mask guidance
5 Discussion and Conclusion
We introduced Stable-Pose, a novel adapter that leverages vision transformers with a coarse-to- fine pose-masked self-attention strategy, specifically designed to efficiently manage precise pose controls during T2I generation. Stable-Poss outperforms current controllable T2I generation methods across five distinct public datasets, demonstrating high generation robustness across diverse environments and various pose scenarios. Notably, in complex scenarios involving rare poses such as side or back views and multiple figures, Stable-Pose exhibits exceptional performance in both pose and visual fidelity. This can be attributed to its advanced capability in capturing long-range patch- wise relationships between different anatomical parts of human pose images through our intricate conditioning design. As a result, Stable-Pose holds significant potential in applications demanding high pose accuracy. It can be easily integrated into any pre-trained T2I diffusion models lightweight generic adapter to effectively enhance pose control. One limitation of Stable-Pose is its slightly longer inference time (Sec, A.2), primarily due the integration of self-attention mechanisms within the VIT. In addition, despite excelling in pose control, Stable-Pose has yet to be evaluated with other conditions such as edge maps. Nevertheless, design allows for straightforward adaptation to various external conditions, suggesting high potential for diverse applications.
Broader Impacts: Stable-Pose's excellent pose control makes it valuable tool in creating diverse artworks, animations, movies, and sports training programs. Additionally, it can be a reliable tool in healthcare and rehabilitation for correcting posture and preventing patients from musculoskeletal issues.
Acknowledgements
This work was supported by the Munich Center for Machine Learning (MCML) and the German Re- search Foundation (DFG). The authors gratefully acknowledge the computational and data resources provided by the Leibniz Supercomputing Centre.
10

---

References
[1] Posenet similarity.
Ott.tttppr/ggttt...o/o//////aa/a similarrtt
[2] A. F. Agarap, Deep learning using rectified linear units (relu). arXiv preprint arXiv: 000.00880..
[3] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint
2016.
[4] A. K. Bhunia, S. Khan, H. Cholakkal, R. M. Anwer, J. Laaksonen, M. Shah, and F. S. Khan. Person image synthesis via denoising diffusion model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5968 5976, 2023.
[5] M. Binkowski, D. J. Sutherland, M. Arbel, and A. Gretton. Demystifying mmd gans. arXiv preprint arXiv: TeXivvI801.00 1001001000.,,,,,, 2018.
[6] B. Cheng, B. Xiao, Wang, H. Shi, S. Huang, and Zhang. Higherhrnet: Scale- representation learning for bottom- up human pose estimation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5386-5395, 2020.
[7] S. Cheong, A. Mustafa, and Gilbert. Kpe: Keypoint pose encoding transformer-based image generation. arXiv preprint Xiv:2203.00907, 2022.
[8] S. Flfwing, F. Uchibe, and K. Doya. Sigmoid- linear units for neural network function approxi- mation in reinforcement leamning Neural networks, 107:3-11, 11, 2018.
[9] P. Esser, E. Sutter, and B. Ommer. A variational u-net for conditional appearance and shape generation, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8857-8866, 2018.
[10] M. Heusel, H. Ramsauer. T. Unterthiner, B. Nessler, and S. Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[11] J. Ho and T. Salimans, Classifier-free diffusion guidance. arXiv preprint vrXiv:2207.125p90 2022.
[12] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 6851, 2020.
[13] L. Huang, D. Chen, Y. Liu. Y. Shen, D. Zhao, and J. Zhou. Composer: Creative and controllable image synthesis with composable conditions. arXiv preprint arXiv:2302.09777. 2023.
[14] S. Jayasumana, S. Ramalingam, A. Veit, D. Glasner, A. Chakrabarti, and S. Kumar. Rethinking fid: Towards better evaluation metric for image generation. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9307-9315, 2024.
[15] X. Ju, A. Zeng, J. Wang, Q. Xu, and I.. Zhang. Human-art: A versatile human-centric dataset bridging natural and artificial scenes. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 618-629, 2023.
[16] X. Ju. A. Zeng. C. Zhao, J. Wang. L. Zhang, and Q. Xu. Humansd: native skeleton skeleton-guided diffusion model for human image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15988-15998, 2023.
[17] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint 1rXiv:1412.6600 2014.
[18] Li, H. Liu, Q. Wu, F. J. Yang. J. Gao, C. Li, and Y. J. Lee. Gligen: Open-set grounded text-to-image generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22511-22521, 2023.
[19] D. Liu, L. Wu, F. Zheng, L. Liu, and M. Wang. Verbal-person nets: Pose-guided multi-granularity language-to-peroon generation. IEEE Transactions on Neural Networks and Learning Systems, 34(11): 8589-8601, 2023.
[20] Liu, G. Wang, L.-Y. Duan, K. Abdiyeva, and A. C. Kot. Skeleton- based human action recognition with global context-aware attention lstm networks. IEEE Transactions on Image Processing, 2764):1586-159 2017.
[21] Z. Liu, J. Zhu, J. Bu, and C. Chen. A survey of human pose estimation: the body parts parsing based methods. Journal of Visual Communication and Image Representation, 32:10-19, 2015.
11

---

[22] L. Ma. X. Jia. Q. Sun. B. Schiele. T. Tuytelaars, and L. Van Gool. Pose guided person image generation. Advances in neural information processing systems, 30, 2017.
[23] Y. Men, Y. Mao, Y. Jiang, W.-Y. Ma. and Z. Lian. Controllable person image synthesis with attribute- decomposed gan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 5084-5093, 2020.
[24] L. Ming, Y. Taojiannan, K. Huafeng, W. Jie, W. Zhaoning. X. Xucfeng, and C. Chen, Controlnct++: Improving conditional controls with efficient consistency feedback. arXiv preprint arXiv:2404.07987, 2024.
[25] C. Mou, X. Wang, L. Xie, Y. Wu, J. Zhang, Z. Qi. Y. Shan, and X. Qie. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models, arXiv preprint arXiv:2302.088: 08.00843. 2023.
[26] A. Nichol, P. Dhariwal, A. Ramesh, P. Shyam, P. Mishkin, B. McGrew, Sutskever, and M. Chen. Glide: Towards photorealistic image generation and editing with text- gutde diffusion models. arXiv preprint arXiv:21 10741, 2021.
[27] F. Perazzi, J. Pont- Tuset, B. McWilliams, L. V. Gool, M. Gross, and A. Sorkine Sorkine-Horuung. A benchmark dataset and evaluation methodology for video ohject segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[28] C. R. Qi, H. Su, M. Nießner, A. Dai, M. Yan, and L. J. Guibas. Volumetric and multi-view cnns for object classification on 3d data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648-5656, 2016.
[29] Z. Qiu, W. Liu. H. Feng, Y. Xue, Y. Feng. Z. Liu, D. Zhang, A. Weller, and B. Schölkopf. Controlling text-to-image -mmge diffusion by orthogonal finetuning. In NeurlPS, 2023.
[30] A. Radford, J. W. Kim, C. Hallacy, A, Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMI.R, 2021.
[31] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pages 8821-8831. Pmlr, 2021.
[32] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 2rXiv:2204.06125, 1(2):3, 2022.
[33] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 10684-10695, 2022.
[34] N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman. Dreambooth: Fine tuning text-to- image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22500-22510, 2023.
[35] C. Saharia, W. Chan, Saxena, L. Li, J. Whang. E. L. Denton, K. Ghasemipour, R. Gontijo Lopes. B. Karagol Ayan, Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36499-36494. 2022.
[36] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti. T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. 5aion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 01.255778-5294 2022.
[37] F. Shen, H. Ye, J. Zhang, C. Wang. X. Han, and W. Yang. Advancing guided image synthesis with progressive conditional diffusion models. arXiv preprint arXiv:2310.0633/ 2023.
[38] A. Siarohin, E. Sangincto, S. Lathuilicre, and N. Sebe. Deformable gans for pose-based human image generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3408-3416, 2018.
[39] J. Song, C. Meng, and S. Ermon. Denoising diffiusion implicit models. arXiv preprint xrXiv:2010.02502. 2020.
[40] P. Sun, J. Cao. Y. Jiang, Z. Yuan, S. Bai, K. Kitani, and P. Luo. Dancetrack: Multi-object tracking in uniform appearance and diverse motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
12

---

[41] C. Szegedy, V. Vanhoucke, S. loffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2818-2826, 2016.
[42] H. Tang. S. Bai, L. Zhang, P. H. Torr, and N. Sebe. Xinggan for person image generation. In Computer 2020: 16th European Conference, Glasgow, UK, August 23-28. 2020, Proceedings, Part XXV 16. pages 717-734. Springer, 2020.
[43] Zablotskaia, Siarohin, Zhao, and Sigal. Dwnet: Dense warp-based network pose-guie pose-guided human video generation. arXiv preprint 1rXiv:1910.09139, 2019.
[44] L. Zhang, Rao, and M. Agrawala. Adding conditional control texi-to-image diffusion models. Proceedings of the IEEE/CVE International Conference on Computer Vixion, pages 3836-3847, 2026-3847.0023..
[45] Zhang, L. Yang, J.-H. Lai, and Xie. Exploring dual bual-task correlation for pose guided person image generation. In Proceedings the pages 7713-7722, 2022.
the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
[46] S. Zhao, D. Chen, Y.-C. Chen, J. Bao, S. Hao, L. Yuan, and K.-Y. K. Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
[47] Z. Zhu, T. Huang, B. Shi, M. Yu, B. Wang, and X. Bai. Progressive pose attention transfer for person image generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2347-2356, 2019.
13

---

A
Appendix supplementarry material
Index
Keypoint
Nose
Left-eye
Right-eye
Left-ear
4
Right-ear
Left-shoulder
Right-shoulder
7
Left-elbow
Right-elbow
Left-wrist
10
Right-wrist
11
Lefl-hip
12
Right-hip
Left-knee
14
Right-knee
15
Left-ankle
16
Right-ankle
(a) Table keypoints.
(b) Sample pose with keypoints.
(c) Sample image with pose.
Figure A.I: Illustration of pose input, where each body part is marked in different color. The pose-image pair is from UBC Fashion dataset
A.1 Datasets and Preprocessing
Our method and the other techniques are trained and evaluated on five distinct datasets as below:
Human-Art [15]: The Human-Art dataset comprises 38,000 images distributed across 19 scenarios, encompassing natural scenes, 2D artificial scenarios, and 3D artificial scenarios. We adopt the same vaainvvaidationn split as the authors suggested. The annotations in Human-Art belong to the International Digital Economy Academy (IDEA) and are licensed under the Attribution-Non Commercial-Shee Alike 4.0 International License (CC-BY-NC-SA 4.0).
The LAION-Human derived from the LAION-5B dataset [36], consisting of approximately million images filtered by human estimation confidence scores. We randomly selected subset of 200,000 images for training and 20,000 images for validation. The dataset licensed under the Creative Common CC-BY 4.0 license, which poses no particular restriction. The images are under their copyright.
LAION-Huuan [6]
UBC Fashion 43 The UBC Fashion dataset comprises sequences showcasing fashion models executing turns. We extracted frames representing various orientations: front, side, and back, to rigorously test our model's aptitude for handling both complex and infrequently encountered poses. The dataset yields approximately 1100 front, 450 side, and 1100 back frames. The dataset is licensed under the Creative Commons Attribution-Non Commercial 4.0 International Public License.
Dance Track 40 The Dance Track dataset presents group dance footage, typified by multiple subjects and intricate postures. We curated 20 videos from the Dance Track validation set and extracted a total of 2000 frames assess our model. The annotations of TrackTrack are licensed under Creative Commons Attribution 4.0 License and the dataset of DanceTTrach is available for non-commercial research purposes only.
The DAVIS dataset is a widely used dataset for video-related tasks. We randomly chose 26 human-centric scenarios from the DAVIS Test-Dev 2017 set and the DAVIS Test- Test-Challenge 2017 set, which provided around 2000 frames for evaluation. The DAVIS dataset is released under the BSD License.
DAVIS 27:
All datasets adhere to standardized protocol featuring 17 keypoints and maximum of 10 persons per image, following COCO and Human-Art. Despite the original checkpoints of ControlNet and
14

---

Table A.1: Trainable parameters (millions), training time on Human-Ar (hours), and inference time (seconds per image) for each technique.
Stable-Pose
Method
T21-Adapter
ControlNet
Uni-ControlNet
GLIGEN
HumanSD
(Ours)
Train. params.
77
361
411
209
859
364
Train. time
12.9
14.0
14.2
20.2
19.6
14.3
Infer. time
4.45
5.76
6.20
9.48
4.20
6.37
T2I-Adapter being anchored in the OpenPose keypoint protocol, it is feasible to convert keypoints between different styles without loss of accuracy. preprocessing the datasets for network input, we applied a score threshold of 0.05 to filter out keypoints and connected corresponding keypoints, following the procedure outlined by the authors of Human-Art. Each connecting line is depicted in distinct colors, illustrated in Figure A.I enabling the network to learn associations between each line and specific body parts. Please note that the colors the pose skeleton presented this paper solely for visualization purposes and do not correspond those used in the experiments. During training, we employed consistent data augmentations, including random cropping at 512 pixels, random rotation, random color shift, and random brightness contrast. These augmentations were applied uniformly across all techniques.
It is worth noting that the scale of the LAION-Human subset and its data distribution closely align with those reported in the SOTA works, such as ControlNet [44), which was trained on 200,000 images sourced from the Internet. Thus, ensuring fair comparison is possible. Since none of the aforementioned video datasets are annotated with poses or textual descriptions, we employed the GPT-4 API for generating prompts and HigherH/RNet (6 for deducing pose labels.
A.2 Training and Evaluation
To ensure fair comparison, we trained and evaluated all the techniques on Human dataset. On subset, due to computational constraints, we only trained our method, HumanSD, GLIGEN and Uni-ControlNet, while ControlNet and T21-Adapter were evaluated using the released checkpoints. The rationale behind this was that HumanSD had been previously trained on a segment of LAAIO--Human, which might overlap with our validation set, leading to potential data leakage. GLIGEN had been trained with considerably more GPU hours compared to other techniques, posing risk of unfair comparison. Uni- Coii-oontooNettss available checkpoints condition model, however, our research is dedicated exclusively to pose control, necessitating further training. Conversely, for ControlNet and T21- T21-Adapter, the amount of data and GPU hours for training were on par with our methods, allowing for fair comparison using the checkpoints they provided.
tailored for a multi-
Training was conducted on two NVIDIA A100 GPUs. Our method required 15 hours to complete training on the Human- Art dataset and 70 hours for the LAION -umman subset. During training, the peak RAM usage for our technique was 25 GB. We reported the number of trainable parameters and training time on Human-Art dataset for each technique in Table A Our technique has comparable number of trainable parameters and training time to ControlNet but produces significantly better results, highlighting the effectiveness of our model design. We also reported the inference speed of each technique in Table A.I. measured by the average seconds needed to generate an image. To ensure fair comparison, these measurements were conducted on the same NVIDIA A A00 GPUs. Due to the self-attention mechanism in the ViT, our method's 's inference speed is slightly slower than that ControlNet.
A.3 Selecting Gaussian Filters in PMSA
common way to choose the kernel size of Gaussian Filter with sigma is This ensures that the kernel captures the majority of the Gaussian function's 's weight. When using the kernels and sigmas as specified in the above equation, we observed that the difference between generated coarse and fine masks was too large to yield satisfactory results. We measured this via the computation of the activation rate, reported in Table A.2 We applied the pose mask to the patches and then calculated the ratio of pose-masked patches to the total number of patches. We opted for
[3o]] +
15

---

Table A.2: Ratios of unmasked patches (%) under different sigma o and kernel size k in coarse-to-
fine manner.
0 1,
25
0 3,
19
0 = 2,
13
0 1,
15.83
14.12
12.38
10.48
Table A.3: Ratios of unmasked patches (%) under fixed sigma and varying kernel size & in coarse-to-fine manner. Selected ones are in bold.
23
21
19
17
15
13
11
7
3
14.26
14.17
14.12
13.98
13.77
13.21
12.78
12.14
11.53
10.81
10.06
9.09
an alternative method to control the blurring effect by fixing and adjusting k which changes the number of pixels to which the Gaussian weights are applied. Larger values for le dilate the mask, resulting in coarser masks.
In our experiment, we fix a at 3 and then change for obtaining different kernels. The activation rate of patches under the fixed and varying k is reported in Table where & = 23 is the largest kernel size for o = 3, indicating that any larger does not affect the activation rate. This size was chosen to generate the most coarse mask. To ensure smooth guidance of our pose-mask in the proposed PMSA, we selected kernel sizes with approximately one percent difference in activation rate. As a result, & = 13 was chosen to generate the fine mask. For our ablation study with 3 pose masks, we selected & = 9, which also yielded about a one percent difference in activation rate compared to k = 13. Details of this ablation study can be found in Sec. A.6
A.4 Evaluation Metries
We define the evaluation metrics in detail here. FID assumees that the features extracted from real and generated images by the Inception v3 model [41] follow a Gaussian distribution. It measures the Fréchet distance between these distributions. KID, however, relaxes the assumption of a Gaussian distribution and calculates the squared Maximum Mean Discrepancy (MMD) between the Inception features of real and generated images using a polynomial kernel. Mean Average Precision (AP) computes the alignment between keypoints real images and generated images. Poses of generated images are detected by the same Considering that images may contain multiple persons, we included People Counting Error (PCE) 7 as a metric, measuring the false positive rate when generating images featuring multiple people. The CLIP score measures the similarity between embeddings of generated images and text prompts, both of which are encoded by CLIP.
Challenges in the Current Metrics We followed the common metrics widely adopted in the current generative Al field, however, despite the rapid advancements in generative AI, existing metrics have not evolved to provide a more accurate evaluation [4]. Some of the issues are 1) CLIP score relies on cosine similarity between the model's semantic understanding and the given text, which may not align with pose assessments or the relevance of generated images. Additionally, this score is sensitive to the arrangement and composition of elements in images; even minor changes can result in significant fluctuations in the score, which may not accurately reflect the overall generative quality. Dino-based score; 2) FID estimates the distance between distribution of Inception-v3 features of real images and those of images generated by the generative models. However, Inception's 's poor representation of the rich and varied content generated by modern ext-too-mmae models incorrect normality assumptions and poor sample complexity [14). Thus, the FID score does not account for semantic correctness or content relevance-sspecffcalll pose-in relation to the specified text or conditions. Relying solely on FID and CLIP scores does not provide a comprehensive assessment of the generative model. Therefore, we further evaluated our method with a new sfate-of of-te-o-thheaa t-tthh-an art metric CMMD [4]. which is based on richer CLIP embeddings and the maximum mean discrepancy distance with the Gaussian RBF kernel. is an unbiased estimator that does not make any assumptions on the probability distribution of the embeddings, offering a more robust and reliable assessment of image quality. As shown in Table A.4 our method achieves better CMMD value compared to HumanSD, demonstrating comparably high image quality.
34 suggests
16

---

Table A.4: CMMD evaluation for HumanSD and Stable-Pooe
HumanSD
Stable-Pose
CMMD
5.027
5.025
Table A.5: Detailed results UBC Fashion dataset with front, side, and back orientations.
Pose Accuracy
Image Quality
T2I Alignment
Orientation
Method
AP
CAP
PCE L
FID 4
KID 4
CLIP-score f
T2I-Adapter
72.20
50.16
0.61
6.57
3.95
32.11
ControlNet
74.64
48.11
0.65
4.50
3.91
32.06
Uni-ControlNet
79.47
51.21
0.57
6.14
4.40
32.14
Front
GLIGEN
73.97
50.95
0.44
8.94
4.49
30.35
HumanSD
76.83
51.45
0.60
6.55
4.89
31.14
Stahle-Pose (Ours)
87.26
51.90
0.56
6.05
4.57
32.05
T2I-Adapter
36.80
44.38
0.75
6.10
3.81
32.16
ControlNet
52.83
46.16
0.80
5.05
3.95
32.13
Uni- Uni-ControlNet
58.26
46.70
0.70
5.73
4.28
32.15
Side
GLIGEN
45.32
45.68
0.69
6.58
3,94
30.75
HumanSD
57.09
45.85
0.78
5.14
4.75
31.95
Stable-Pose (Ours)
69.76
47.00
0.71
5.10
4.43
32.25
T21-Adapter
6.03
22.38
0.91
8.49
4.22
32.20
ControlNet
23.68
24.61
0.97
7.53
4.01
32.15
Uni-ControINet
19.97
23.79
0.93
7.20
4.37
32.19
Back
GLIGEN
4.45
18.36
0.90
10.83
4.32
30.29
HumanSD
21.15
0.91
9.12
4.99
32.01
Stable-Pose (Ours)
29.08
25.21
0.93
6.24
4.56
32.11
Table A.6: Detailed results DAVIS dataset and Dance Track dataset.
Pose Accuracy
Image Quality
T21 Alignment
Dataset
Method
AP t
CAP
PCE 4
FID
KID +
CLIP-score 1
T21-Adapter
20.28
60.76
1.90
3.69
2.19
31.05
ControlNet
30.13
60.81
1.67
3.81
2.21
30.72
Uni- Oni-ControlNet
37.64
60.57
1.56
3.60
1.83
30.63
DAVIS
GLIGEN
12.17
59.87
1.83
3.72
1.69
30.16
HumanSD)
38.32
60.59
1.37
2.94
1.82
28.49
Stable-Pose (Ours)
42.87
62.43
1.45
3.54
1.84
30.56
T21-Adapter
10.36
72.38
4.82
19.76
5.64
33.01
ControlNet
16.45
73.16
5.78
20.92
5.13
32.76
Uni-Cont Costrollt
25.22
73.62
4.79
21.89
4.75
32.83
Danee Track
GLIGEN
5.61
72.57
4.72
20.13
4.53
32.37
HumanSD
24.13
69.93
3.78
23.15
8.58
31.64
Stable-Pose (Ours)
28.58
74.97
4.81
17.28
4.74
32.90
A.5
Detailed Results
We reported detailed results for the UBC Fashion, DAVIS, and Dance Track datasets in Ta- bles A.5 where KID is multiplied by 100 for readability. The results demonstrate that our Stable-Pose consistently provides better controllability over poses, for challenging poses such as back poses.
A.6 Additional Ablation Studies
ViT Parameters. We studied the impact of different VIT parameters, including depth and patch sizes, reported in Table A.7 and A.8 Table A.7 shows results for varying depths with fixed patch size
17

---

Table A.7: Comparison of different depths in our proposed VIT. Kernel sizes of Gaussian Filters are noted.
Depth
AP1
CAP
PCE 4
FID
KID 4
CLIP-score f
(23)
47.96
70.75
1.57
11.94
2,45
32.55
2 (23, 13)
48.88
70.83
1,50
11.12
2.35
32.60
3 (23, 13, 9)
47.64
71.13
1.55
12.22
2.65
32.58
Table A.8: Comparison of different patch sizes in our proposed VIT.
Patch size
AP
CAP t
PCE
FID +
KID 4
CLIP-score t
2
48.88
70.83
1.50
11.12
2.35
32.60
4
47.51
70.85
1.53
13.39
2.49
32.58
8
45.69
70.44
1.59
13.14
2.47
32.64
of using coarse-to-fine pose masks as guidance. While these masks provide smooth guidance, higher depth with overly fine masks might lose valuable information from previous attention layers. Table A.8 presents results for ViT with PMSA of depth and different patch sizes. Our default patch size of yields the best results, likely because the latent encoding has typically lower dimension (here 64 x 64) compared to the dimensions of input high-resolution images. Larger patches may dilute learning of interconnections among anatomical parts for encompassing too much information.
A.7 Attention Maps of PMSA
In Figure A.2 we present examples of attention maps extracted from our VIT. These maps are derived from the last block of the ViT and represent average across all attention heads. The attention maps were approximately overlapped with the pose region, which corroborates the objectives of our proposed PMSA. This alignment underscores the efficacy of our model in focusing on relevant pose features, a critical aspect our approach to improving model interpretability.
A.8
Failure Case
Stable-Pose may face failure cases when generating the wrong number of people in very crowded scenes. Stable-Pose enhances SD's accuracy in pose-mask regions, whereas a pre-trained SD may still produce human-like figures in the background. For example, in the first row of Figure A.3 Stable-Pose generates an half-shaped person on the very right side.
A.9 Extreme cases
The generation of some extreme pose cases is shown in Figure A.4, such as bending the upper body backward in some dancing poses. As shown the figures. Stable-Pose still maintains very high pose accuracy on generated images under these challenging scenarios, whereas ControlNet fails to depict the correct pose and body structures.
A.10 More Visualization Results
We include additional qualitative results from the five datasets we evaluated in Figure
A.5 A.6 A.7
A.8
A.9
18

---

(a) Original image
(b) Attention map
(a) Original image
(b) Attention map
Figure A.2: Samples from WAION-Human dataset with original images and attention maps.
Ballet performers in white tutus gracefully danciig on stage with winter background.
(a) Pose
(b) ControlNet
(c) Stable-Pose (Ours)
(d) True Image
Dancers in silver dresses performing sroniizeeedd routine stage.
uum
BANC
(a) Pose
(b) ControlNet
(c) Stable-Pose (Ours)
(d) True Image
Figure A.3: Failure sampled from Dance Track dataset. Unexpected humans generated marked in red circles.
19

---

Acrobatics, a man and woman performing aerial acrobatics.
(a) Pose
(b) ControlNet
(c) Stable-Pose (Ours)
(d) True Image
Acrobatics, a man doing handstand on a white background.
(a) Pose
(b) ControlNet
(c) Stable-Pose (Ours)
(d) True Image
Figure A.4: Extreme poses sampled from the Human-Art dataset. ControlNet creates wrong limbs in
extreme cases while Stable-Pose achieves accurate target poses.
Garage kits, figurine with blue hair and wings.
Kids drawing, painting of ballerina in blue and yollow.
Digital art, rainy day by Kazuhiro Kawai.
(a) Pose
(b) T21-Adapter
(c) ControlNet
(d) Stable-Pose (Ours)
Figure A.5: Results Human-Art dataset.
20

---

Gueci Spring 2016 -Ready-to-
Reeaay-t--Wear Wear Fashion Show Eline Klein Onstenk
Flower girl with ballerina dress pink and ivory colour by angel's face.
The drawing of little girl with daisy painting by Anna Abramska.
(a) Pose
(b) T21-Adapter
(c) ControlNet
(d) Stable-Pose (Ours)
Figure A.6: Results
dataset.
A model wearing floral knee-length dress.
A model showcasing sleeveless dress in studio.
A fashion model studio, showcasing light pink sleeveless dress.
(a) Pose
(b) T21-Adapter
(c) ControlNet
(d) Stable-Pose (Ours)
Figure A.7: Results on UBC Fashion dataset with side orientation.
21

---

A model showcasing a floval dress against plain background.
A model posing in studio with plafn background, wearing brown dress.
Full-body poses of a model, showcasing black bodycon dress.
(a) Pose
(b) T21-Adapter
(c) ControlNet
(d) Stable-Pose (Ours)
Figure A.8: Results on UBC Fashion dataset with back orientation.
Dance crew performing synchromized routine with artistic masés,
A man riding unique green four-wheefed bicycle indoors.
A rider on horse shows jumping in an equestrian area.
(a) Pose
(b) T21-Adapter
(c) ControlNet
(d) Stable-Pose (Ours)
Figure A.9: Results on Dance Track (first row) and DAVIS dataset (the rest rows).
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims in the abstract and introduction accurately match the theoretical and experimental results of our methodology.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discussed the limitations of our method in Sec. 5: Discussion and Conclusion. We also discussed the computational efficiency in Sec. A.2 and Table A.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer:
[NA]
Justification:
Our method does not need theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We included detailed information the structure of our model well as the introduced algorithms and training objective in Sec. 3. We also included implementation details in Sec. 4 and Sec. A.2 to fully promote the reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We commit to release our source code on GitHub upon paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipsscc//pill wttpcccpppppppppeeeeeeeeeeeeeeee
httaa//gguudeee/eeeeebiiiisssss /Cc////oddiiisssseeeeeee ssiiiiiiccyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We provided all the training and test details in the Implementation Details in Sec. 4 and Sec. A.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
The error bars for our experimental results are not applicable.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
25

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provided detailed information on the computer resources Sec. A.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and confirm that we conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our paper have discussed both potential positive societal impacts and negative societal impacts of the work performed Sec. 5.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [Yes]
Justification: We commit to provide effective requirement for users to adhere to usage guidelines once we release our source code on GitHub upon acceptance.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have properly eited the original paper of the code package and dataset used in our work. We also included the information of license and terms of use of the datasets in See. A.I.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
27

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We commit to release the source code of our work GitHub with well- structured documentation upon acceptance.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
28

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "3ilcIVImOu", "reviewer": "Reviewer_ypoD", "summary": "To obtain more refined pose condition control, especially for challenging condition (pose & text) generation, this paper designs a coarse-to-fine Pose-Masked Self-Attention (PMSA) module with the use of pose masks, fine-tunes Stable Diffusion with pose-mask guided loss, and ultimately achieves more precise control with little sacrifice of visual generation quality.", "questions": "Please see the weaknesses.", "limitations": "The authors state the limitations in Sec. 5 as 1. longer inference time caused by the Self-Attention, 2. not evaluated with other condition guidances. It is recommended to add the visualization and discussion of failure cases.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper shows a good accurate control effect consistent with the pose and text guidance.\n\n- The method makes sense because pose is a more sophisticated control condition than text, so it requires a refined and localized design. It is simple and relatively easy to understand.\n\n- There are some experimental verifications in terms of controllability and robustness.", "weaknesses": "I carefully evaluated this module paper and did not find particular flaws except for no consideration of the multi-person condition. I will refer to the review suggestions of other reviewers and the author's feedback later.\n\n[Method]\n\n- (No consideration of different persons & parts) Surprisingly, the conditions may include multiple person, but PMSA does not explicitly consider modeling different people and different parts (in this sense the DensePose is a better choice), but only models the relationship between patches and semantics are considered by high-level pose decoder $\\beta$. (Although the qualitative results shown look fine, there is no wrong association, such as A's hand is generated at B's hand position, or A's hand is generated at A's foot) I hope the author could provide more detailed explanations and discussion, even on the potential improvement to the proposed PMSA.\n\n[Experiments]\n\n- (Inferior PCE) Can the authors try to analyze the disadvantages of PCE (People Count Error) of the image generated by this method (Tab. 1)?\n\n- (Add module architecture ablation studies) Since this is a work that proposes a module, the ablation studies of the module structure may still be helpful, e.g. the number of the PMSA-ViT block.\n\n- (Add FPS results) Because the proposed PMSA is a plugin, to give readers a better comprehension, it is recommended that the author report the inference time (e.g. FPS) together with other metric results (although the author's claim method has disadvantages).\n\n- (Add failures) It is also important to understand the boundaries of a method, e.g. the failure cases. It is recommended to visualize and analyze some of them.\n\n- (Opinions on masked generation) Pose-mask guided is used as training loss in the paper. To further enhance control, it seems that the generation of humans can be similarly replaced within the mask area during the inference time. I am curious about the author’s opinion on this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our sincere appreciation to the reviewer for providing valuable comments and suggestions. We have addressed the comments point-by-point, as outlined below.\n\n*__C#1__: (No consideration of different persons & parts)*\n\nThanks for the insightful discussion. The pose information is encoded in two ways in Stable-Pose: \n1) *The pose encoder (β)*: It encodes the RGB pose skeleton input into high-level contextual information. Through color encoding, β captures semantic information such as the differentiation of limbs and the number of people.  \n2) *PMSA-ViT*: the patchified latent features are processed through several coarse-to-fine pose-masked self-attentive ViTs, which focus on accurately modeling the poses shown in the pose skeleton and the interaction between different pose parts. \n\nThus, through these two modules, Stable-Pose synergistically integrates both local and global contextual information (including the number of people and body parts) from the pose skeleton naturally. \n\nTo further validate the design, we specifically add an ablation study by removing either β or PMSA-ViT, retaining only one type of encoding. The results in the table below show that using only PMSA-ViT yields an AP of 36.48, which is expected due to the absence of color-coding information for distinguishing body parts. While using β alone increases the AP to 45.03. However, the most significant improvement is observed when integrating both local and global information encoding into the Stable-Pose architecture, achieving the highest AP of 48.88.\n|Method|AP ↑|CAP ↑|PCE ↓|FID ↓|KID ↓|CLIP score ↑|\n|-|-|-|-|-|-|-|\n|w/o β Enc.|36.48|68.91|1.55|11.17|2.76|31.90|\n|w/o PMSA-ViT Enc.|45.03|70.38|1.52|13.67|2.49|32.53|\n|w/ both β & PMSA-ViT Encs.|48.88|70.83|1.50|11.12|2.35|32.60|\n\nThanks to this comment, we will add these results and discussion to our final manuscript and we appreciate this valuable comment.\n\n---\n\n*__C#2__: (Inferior PCE)*\n\nWhile PCE is useful for evaluating the number of humans in generated images, it has limitations such as the sensitivity to outliers. Given that Stable-Pose is designed to enhance SD's ability to generate accurate content in pose-mask regions, a pre-trained SD may still produce human-like figures in the background. For example, in the first row of Fig. R1 of our attached PDF, Stable-Pose generates an additional half-shaped person in the background on the right side. Such cases affect the overall PCE measurement. Therefore, foreground counting should be prioritized over background counting. However, PCE treats both equally. \n\nDespite this, Stable-Pose still reaches the best PCE score among adapter-based methods (e.g., ControlNet) in LAION-Human. In addition, applying Stable-Pose on top of a pre-trained HumanSD model (a fine-tuning-based method) considerably improves the PCE score compared to using Stable-Pose with SD. While it results in a marginal error of only 0.05 more/less people per image compared to the HumanSD method, it substantially improves the pose accuracy in AP by 10%.\n|Method|AP ↑|CAP ↑|PCE ↓|FID ↓|KID ↓|CLIP score ↑|\n|-|-|-|-|-|-|-|\n|HumanSD|44.57|69.68|1.37|10.03|2.70|32.24|\n|Stable-Pose|48.88|70.83|1.50|11.12|2.35|32.60|\n|HumanSD+Stable-Pose|49.24|71.01|1.42|10.42|2.37|32.16|\n\nThanks to this comment, we further discuss this aspect of PCE in Appendix A.4 and would love to hear your opinion about it.\n\n---\n\n*__C#3__: (Add module architecture ablation studies) & __C#4__: (Add FPS results)*\n\nThanks for your advice in improving the paper. In fact we have addressed these concerns in our supplementary material as detailed below: \n1) In Table A.6 and A.7, we demonstrated ablation studies of the module structure, including the number of the PMSA-ViT blocks and different patch sizes in the ViT, respectively. \n2) In Table A.1, we reported the inference time (FPS) and training duration (in hours) of our method compared to the other state-of-the-art approaches. \n\nDue to the page limit we could not include these results in the main manuscript, but only in the appendix. That said, we plan to move some parts mentioned above to the main draft, as there is an additional page available for the final submission.\n\n---\n\n*__C#5__: (Add failures)*\n\nThank you for your insightful comment. We attached a PDF file with some failure cases in Fig. R1, such as generating the wrong number of people in very crowded scenes. Stable-Pose enhances SD's accuracy in pose-mask regions, but a pre-trained SD may still produce human-like figures in the background. For example, in the first row of Fig. R1, Stable-Pose generates an extra half-shaped person on the very right side. We will add a dedicated subsection (A.9) in the Appendix to address this issue.\n\n---\n\n*__C#6__: (Opinions on masked generation)*\n\nWe highly appreciate this insightful comment. In our implementation, we indeed incorporate the pose masks during inference by default to enhance control. To further validate their effectiveness, we additionally conducted experiments with removing the pose masks during inference. As shown in the table below, this led to approximately a 3% drop in AP. This could be due to two main reasons:\n1. The pose masks provided additional guidance, thus enhancing control.\n2. The inclusion of pose masks maintains consistency between the model's behavior during training and inference.\n\n|Method|AP ↑|CAP ↑|PCE ↓|FID ↓ |KID ↓|CLIP score ↑|\n|-|-|-|-|-|-|-|\n|w/o mask in inference|45.93|70.51|1.52|13.11|2.55|32.68|\n|w/ mask in inference|48.88|70.83|1.50|11.12|2.35|32.60|\n\nThus, including pose masks benefits pose control in the generation. Thanks to this comment, we will include this result in our final manuscript. \n\n---\n\nWe highly appreciate the valuable suggestions and input from the Reviewer. We hope our responses address the reviewer’s concerns about Stable-Pose, with details covered in the manuscript and appendix. We look forward to any further questions during the reviewer-author discussion period from August 7-13.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cWEeVSuC8t", "reviewer": "Reviewer_qFuv", "summary": "The authors proposed a pose guided image generation pipeline. By introducing the PMSA ViT as pose encoder, the pipeline can generate more controllable images with corresponding 2D pose conditions.", "questions": "- Could the authors discuss more about why image quality(FID) and T2I Alignment(CLIP Score) are worse than those SOTA methods?\n- Why the proposed methods are better at distinguish the left and right of the pose condition? During PSMA ViT, it seems like the authors treated all the keypoints the same, without any special design to improve the left-right consistency.", "limitations": "The limitation has been discussed in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- According to Fig 4, the proposed method is the only one which distinguish the front and back side of the 2D pose condition, which is much better than other SOTAs.\n- The generated images follows the pose condition better than other methods.", "weaknesses": "- The proposed method is more like an incremental work of previous methods instead of a new innovative pipeline.\n- Although the generated images follow the pose guidance better, the image quality(FID) and T2I Alignment(CLIP Score) are worse than those SOTA methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the Reviewer’s comments and insightful suggestions. Below, we provide a point-by-point response to each comment.\n\n*__C#1__: The proposed method is more like an incremental work…instead of a new innovative pipeline.*\n\nWe respectfully hold a different perspective on the novelty assessment. Stable-Pose is a light-weight **adapter** that can be easily integrated into any pre-trained T2I diffusion models to effectively enhance pose control, even in challenging conditions. It achieves so by synergistically capturing both local pose details with a coarse-to-fine PMSA ViT, and global semantics via a pose encoder. Its novelty and efficiency are confirmed by other reviewers (pWbK), and Reviewer RKmi noted ‘the robustness, generalizability, and collaborative potential of the proposed method’. \n\nTo further validate its generalizability, we conducted additional experiments by applying Stable-Pose on top of a pre-trained HumanSD model. As shown in the table below, the inclusion of Stable-Pose considerably improved the baseline HumanSD by over 10% in AP and 12% in KID, highlighting its high generalizability and effectiveness in enhancing both pose control and image quality.\n|Method|AP ↑|CAP ↑|PCE ↓|FID ↓|KID ↓|CLIP score ↑|\n|-|-|-|-|-|-|-|\n|HumanSD|44.57|69.68|1.37|10.03|2.70|32.24|\n|Stable-Pose|48.88|70.83|1.50|11.12|2.35|32.60|\n|HumanSD+Stable-Pose|49.24|71.01|1.42|10.42|2.37|32.16|\n\n---\n\n*__C#2__: Although the generated images follow the pose guidance better, the image quality (FID) and T2I Alignment (CLIP Score) are worse…*\n\n*& __Q#1__: Could the authors discuss more about why image quality (FID) and T2I Alignment (CLIP Score) are worse than those SOTA methods?*\n\nWe thank the reviewer for this insightful comment. We followed the common metrics widely adopted in the current generative AI field, however, despite the rapid advancements in generative AI, existing metrics have not evolved to provide a more accurate evaluation [ref1]. We summarize some of the issues below.\n1) CLIP score: it relies on cosine similarity between the model's semantic understanding and the given text, which may not align with pose assessments or the relevance of generated images. Additionally, this score is sensitive to the arrangement and composition of elements in images; even minor changes can result in significant fluctuations in the score, which may not accurately reflect the overall generative quality. [ref2] suggests a Dino-based score.\n2) FID: it estimates the distance between a distribution of Inception-v3 features of real images and those of images generated by the generative models. However, Inception's poor representation of the rich and varied content generated by modern text-to-image models incorrect normality assumptions and poor sample complexity [ref1]. Thus, the FID score does not account for semantic correctness or content relevance—specifically pose—in relation to the specified text or conditions. \n\nRelying solely on FID and CLIP scores does not provide a comprehensive assessment of the generative model. Therefore, in light of your comment and similar concerns from the other Reviewers, we further evaluated our method with a new state-of-the-art metric CMMD [ref1], which is based on richer CLIP embeddings and the maximum mean discrepancy distance with the Gaussian RBF kernel. It is an unbiased estimator that does not make any assumptions on the probability distribution of the embeddings, offering a more robust and reliable assessment of image quality. As shown in the table below, our method achieves better CMMD value compared to HumanSD, demonstrating comparably high image quality. \n||HumanSD|Stable-Pose|\n|--|--|--|\n|**CMMD ↓**|5.027|5.025|\n\nThanks to this comment, we add a subsection in Appendix A.4 listing the current challenges associated with pose assessment via these metrics and the additional results using CMMD in our final manuscript.\n\n[ref1] Jayasumana et al. \"Rethinking fid: Towards a better evaluation metric for image generation.\" In CVPR 2024.\n\n[ref2] Ruiz, et al. \"Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR 2023.\n\n---\n\n*__Q#2__: Why the proposed methods are better at distinguish the left and right of the pose condition?*\n\nThanks for this comment. Indeed, the PMSA-ViT focuses on exploring relationships between human parts using Transformers, capturing local pose details without differentiating body parts. Yet simultaneously, Stable-Pose includes a pose encoder (β) to encode color-coded pose skeleton information, capturing semantic details like limb positions and the number of people. This enables Stable-Pose to naturally and flexibly enhance pose accuracy by synergistically integrating both local pose details and global human conditions.\n\nTo further validate the design, we specifically add an ablation study by removing either β or PMSA-ViT, retaining only one type of encoding. The results in the table below show that using only PMSA-ViT yields an AP of 36.48, which is expected due to the absence of color-coding information for distinguishing body parts. While using β alone increases the AP to 45.03. However, the most significant improvement is observed when integrating both local and global information encoding into the Stable-Pose architecture, achieving the highest AP of 48.88.\n|Method|AP ↑|CAP ↑|PCE ↓|FID ↓|KID ↓|CLIP score ↑|\n|-|-|-|-|-|-|-|\n|w/o β Enc.|36.48|68.91|1.55|11.17|2.76|31.90|\n|w/o PMSA-ViT Enc.|45.03|70.38|1.52|13.67|2.49|32.53|\n|w/ both β & PMSA-ViT Encs.|48.88|70.83|1.50|11.12|2.35|32.60|\n\nThanks to this comment, we add these results and discussion to our final manuscript and we appreciate this valuable discussion.\n\n---\n\nWe thank you once again for every comment. Our responses aim to address all concerns. If there are further questions or suggestions, we invite the Reviewer to discuss them during the August 7-13 reviewer-author period. We value your input and look forward to productive discussions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2VpCufpzht", "reviewer": "Reviewer_pWbK", "summary": "The authors introduce a novel approach for pose-guided human image generation by developing a new pose encoder block. This block incorporates a coarse-to-fine attention masking strategy within a Vision Transformer architecture, leveraging the self-attention mechanism of Vision Transformers to analyze the interconnections among various anatomical parts of human pose skeletons. Furthermore, their loss function is specifically designed to place greater emphasis on the pose region. The model's performance is assessed both qualitatively and quantitatively, demonstrating promising results.", "questions": "The claim in lines 34-35, stating that pose-guided image generation requires paired human data during training, is not entirely accurate. Previous works, such as [1], [2], [3] etc., do not require paired information and can be trained in unpaired and unsupervised settings, though they do need appearance information. This requirement is not necessarily a disadvantage, as it suits applications like animation from a single image. It would be beneficial to revise this section to reflect these nuances and include the relevant citations.\n\nHow do the results appear for extreme poses, such as bending the upper body backward?\n\n[1] A variational u-net for conditional appearance and shape generation. CVPR 2018\n[2] Learning realistic human reposing using cyclic self-supervision with 3d shape, pose, and appearance consistency. ICCV 2021\n[3]  Towards purely unsupervised disentanglement of appearance and shape for person images generation. 2020", "limitations": "Limitations are not adequately discussed.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well-written and has a clear motivation.\n\nIt offers an interesting method for pose-guided human image generation. The innovative aspect of the paper is how it encodes pose information into the latent vector during the denoising process, using the PMSA ViT block. The design of this block is novel and seems effective, as demonstrated by the ablation study in Table 4 and the attention maps shown in Figure A.2.\n\nThe qualitative and quantitative results look promising, and the ablation study appears thorough.\n\nOverall, this paper is solid and could be considered for acceptance.", "weaknesses": "The coarse-to-fine approach does not appear promising based on the ablation study. The improvement from constant variation is negligible.\n\nLimitations and failure cases are not adequately discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate the Reviewer for the valuable comments and suggestions. We have carefully addressed each comment, providing a point-by-point response below.\n\n*__C#1__: The coarse-to-fine approach does not appear promising…The improvement from constant variation is negligible.*\n\nThanks for this comment. As the coarse-to-fine approach is one part of the PMSA design, the strength of PMSA-ViT lies in its synergistic integration of the masking algorithm and kernel designs. As shown in Table 4 of our manuscript, using coarse-to-fine pose masks in PMSA improves the baseline results by approximately 7% in AP. Table 5 also highlights the significance of the order of kernels and how it can significantly impact the results. We believe that this coarse-to-fine approach can also be effectively generalized and incorporated into the loss definition of future diffusion models. Thank you again for the feedback and we will include the discussion in this regard in our final manuscript.\n\n---\n\n*__C#2__: Limitations and failure cases are not adequately discussed.*\n\nThank you for the comment. We have attached a PDF file that includes some failure cases of our method, shown in Fig. R1, such as the incorrect number of generated humans in very crowded scenes. Given that Stable-Pose is designed to enhance SD's ability to generate accurate content in pose-mask regions, a pre-trained SD may still produce human-like figures in the background. For example, in the first row of Fig. R1 of our attached PDF, Stable-Pose generates an additional half-shaped person in the background on the right side. We will also include a dedicated subsection (A.9) in the Appendix to address this issue with the visualization. \n\nAdditionally, in Sec. 5, we discussed some limitations of Stable-Pose, including its slightly longer inference time due to self-attention (illustrated in the appendix Table A.1), and the need for evaluation under other conditions, such as edge maps. Due to the page limit we could not elaborate it extensively in the main manuscript, however, we plan to include deeper discussion on the limitations and failure cases in the final manuscript as there is an additional page available for the improvement of the final submission.\n\n---\n\n*__Q#1__: The claim…stating that pose-guided image generation requires paired human data during training, is not entirely accurate. Previous works, such as [1], [2], [3] etc., do not require paired information… It would be beneficial to revise this section to reflect these nuances and include the relevant citations.*\n\nThank you for providing the insightful information and feedback. We agree with the Reviewer that the requirement of paired information is not necessarily a disadvantage in some applications. In our work, we would like to differentiate from this line of methods by focusing on a pose-guided T2I model that offers higher flexibility in training and content generation without constraints on the appearance. We also acknowledge the importance of discussing unsupervised training methods with appearance constraints, as highlighted by the Reviewer. We will include the relevant citations and revise our final manuscript accordingly. Thank you again for your valuable input.\n\n---\n\n*__Q#2__: How do the results appear for extreme poses, such as bending the upper body backward?*\n\nThank you for this insightful suggestion. We have illustrated some extreme pose cases in Fig. R2 in our attached PDF file, such as bending the upper body backward in some dancing poses. As shown in the figures, Stable-Pose still maintains very high pose accuracy on generated images under these challenging scenarios, whereas ControlNet fails to depict the correct pose and body structures. We will also add a dedicated subsection (A.10) in the Appendix to address this concern.\n\n---\n\nWe deeply appreciate every comment and suggestion of the Reviewer. We hope we have satisfactorily addressed every point. Your input is valued, and we invite the reviewer to raise any additional questions during the upcoming discussion period from August 7-13.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f0CAFBo82c", "reviewer": "Reviewer_RKmi", "summary": "The paper introduces Stable-Pose, a novel adapter model designed to enhance controllable T2I diffusion models by improving pose guidance, particularly in complex human pose conditions. Utilizing a coarse-to-fine attention masking strategy within a Vision Transformer , Stable-Pose effectively refines pose representation and improves performance over existing methods on the LAION-Human dataset.", "questions": "1. It is interesting to see when pose-mask guidance strength is larger than 7, the AP for pose actually got dropped.\n2. for all the examples shown in the paper, are they using the same seed for each of the method? (seed is important)", "limitations": "Stable-Pose has slightly longer inference times due to its use of self-attention mechanisms within the Vision Transformer (ViT). It has also not been thoroughly evaluated under various conditions, such as edge maps, leaving some robustness aspects unexplored. Further testing is needed to confirm its reliability and effectiveness across diverse scenarios.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper proposes an adapter model using a coarse-to-fine attention masking strategy within a Vision Transformer (ViT) to significantly enhance pose guidance for text-to-image (T2I) diffusion models. This approach not only refines pose representation but also demonstrates performance improvement over existing methods on the LAION-Human dataset. Additionally, the comprehensive evaluation across multiple datasets underscore the robustness, generalizability, and collaborative potential of the proposed method.", "weaknesses": "1. although this method shows good results in condition-following ability, but it has performance drop on image quality and text alignment.\n2. compare to controlnet and other pose-guided t2i models, there is no significant contribution both in training recipe and architecture.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments and feedback. We have addressed each comment carefully and provided a point-by-point response below.\n\n*__C#1__: although this method shows good results…it has performance drop on image quality and text alignment.*\n\nWe thank the reviewer for this insightful comment. We followed the common metrics widely adopted in the current generative AI field, however, despite the rapid advancements in generative AI, existing metrics have not evolved to provide a more accurate evaluation [ref1]. We summarize some of the issues below.\n1) CLIP score: it relies on cosine similarity between the model's semantic understanding and the given text, which may not align with pose assessments or the relevance of generated images. Additionally, this score is sensitive to the arrangement and composition of elements in images; even minor changes can result in significant fluctuations in the score, which may not accurately reflect the overall generative quality. [ref2] suggests a Dino-based score.\n2) FID: it estimates the distance between a distribution of Inception-v3 features of real images and those of images generated by the generative models. However, Inception's poor representation of the rich and varied content generated by modern text-to-image models incorrect normality assumptions and poor sample complexity [ref1]. Thus, the FID score does not account for semantic correctness or content relevance—specifically pose—in relation to the specified text or conditions. \n\nRelying solely on FID and CLIP scores does not comprehensively assess the generative model. Therefore, in light of your comment and similar concerns from the other Reviewers, we further evaluated our method with a new state-of-the-art metric CMMD [ref1], which is based on richer CLIP embeddings and the maximum mean discrepancy distance with the Gaussian RBF kernel. It is an unbiased estimator that does not make any assumptions on the probability distribution of the embeddings, offering a more robust and reliable assessment of image quality. As shown in the table below, our method achieves better CMMD value compared to HumanSD, demonstrating comparably high image quality. \n||HumanSD|Stable-Pose|\n|-|-|-|\n|**CMMD↓**|5.027|5.025|\n\nThanks to this comment, we add a subsection in Appendix A.4 listing the current challenges associated with pose evaluation via these metrics and additional results using CMMD in our final manuscript.\n\n[ref1] Jayasumana et al. Rethinking fid: Towards a better evaluation metric for image generation. In CVPR 2024.\n\n[ref2] Ruiz, et al. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR 2023.\n\n---\n\n*__C#2__: compared to controlnet and other…no significant contribution both in training recipe and architecture.*\n\nWe respectfully hold a different perspective on the assessment of our contributions. Stable-Pose is a light-weight **adapter** that can be easily integrated into any pre-trained T2I diffusion models to effectively enhance pose control, even in challenging conditions. It achieves so by synergistically capturing both local pose details with a coarse-to-fine PMSA ViT, and global semantics via a pose encoder. Its novelty and efficiency are confirmed by other reviewers (pWbK), and Reviewer RKmi noted ‘the robustness, generalizability, and collaborative potential of the proposed method’.\n\nTo further validate its generalizability, we conducted additional experiments by applying Stable-Pose on top of a pre-trained HumanSD model. As shown in the table below, the inclusion of Stable-Pose considerably improved the baseline HumanSD by over 10% in AP and 12% in KID, highlighting its high generalizability and effectiveness in enhancing both pose control and image quality.\n|Method|AP ↑|CAP ↑|PCE ↓|FID ↓|KID ↓|CLIP score ↑|\n|-|-|-|-|-|-|-|\n|HumanSD|44.57|69.68|1.37|10.03|2.70|32.24|\n|Stable-Pose|48.88|70.83|1.50|11.12|2.35|32.60|\n|HumanSD+Stable-Pose|49.24|71.01|1.42|10.42|2.37|32.16|\n\n---\n\n*__Q#1__: It is interesting to see when pose-mask guidance strength is larger than 7, the AP for pose actually got dropped.*\n\nWe share the reviewer’s perspective on the slight decrease in AP for higher values of the pose-mask guidance strength (α), which is less than 1%. Increasing α in our proposed loss means putting more attention on the foreground pose regions. However, if α is too large, it forces the model to learn irrelevant texture information like clothing, which negatively impacts training and slightly decreases AP. Despite this, Stable-Pose still outperforms others across an α range of 1-10. Notably, increasing α has a significant impact on FID, worsening it from 11.0 at α=5 to 13.0 at α=10. This indicates that focusing solely on the pose regions may decrease the quality of generated content in non-pose regions. Thus there exists a slight trade-off in selecting α to maintain both high pose accuracy and image quality, in which a value around 5 to 6 turns out to be optimal. Thanks for this insightful comment, we will include it in our final manuscript.\n\n---\n\n*__Q#2__: for all the examples shown in the paper, are they using the same seed for each of the methods? (seed is important)*\n\nThank you for pointing out this very important aspect in current generative AI. Yes, we indeed always kept the same seed list for all techniques, including ours, during both training and inference time, to ensure fair comparison and reproducibility. We appreciate your attention to this matter and will add this information to our final manuscript to make it more clear. We will also make our code available on GitHub upon acceptance to support the reproducibility of our results.\n\n---\n\nOnce more, we appreciate the Reviewer's insightful suggestions and hope our detailed responses have addressed all concerns. If any further questions arise, we invite the Reviewer to discuss them during the August 7-13 discussion period. We value your comments and look forward to further engagement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the Reviewers for their insightful and constructive feedback. We have incorporated their suggestions and responded comprehensively to their comments. We have addressed the comments of the Reviewers individually. We have incorporated a Q&A section in the Supplementary file that concisely summarizes the discussion related to specific questions raised by the Reviewers. We hope that our study will be well-received as a valuable contribution to the NeurIPS' focus on theory & application. We are available for any further discussions or inquiries the reviewers may have during the reviewer-author discussion period.

It is important to mention that the attached PDF file includes figures that have been prepared to address some of the comments from two respected reviewers with the IDs: ypoD and pWbK.

Best regards, 

Authors

Meta-review: This paper designs a coarse-to-fine pose-masked self-attention strategy and proposes a pose-guided image generation pipeline. After rebuttal, it received SA, 2 BA, and BR. 

The initial concerns focused on the design of the conditions (ypoD), performance (qFuv, RKmi), extreme cases (pWbK) and novelty (RKmi).
After going through the paper, the review, and the response, the AC values the key idea of the coarse-to-fine attention masking strategy with ViT, thinks most concerns have been addressed, and recommends the acceptance of the paper. The AC also urges the authors to append the conditions and the performance discussion, and highlight the difference compared to existing works in the revision.
Decision: Accept (poster)