UniMC: Taming Diffusion Transformer for Unified Keypoint-Guided Multi-Class Image Generation
Authors: Qin Guo
Ailing Zeng
Dongxu Yue
Ceyuan Yang
Yang Cao
Hanzhong Guo
Fei Shen
Wei Liu
Xihui Liu
Dan Xu
Venue: ICML 2025 poster (2025)
Keywords: Diffusion Models
Controllable Generation
Large-Scale Dataset
Primary area: applications->computer_vision

Abstract: Although significant advancements have been achieved in the progress of keypoint-guided Text-to-Image diffusion models, existing mainstream keypoint-guided models encounter challenges in controlling the generation of more general non-rigid objects beyond humans (e.g., animals). Moreover, it is difficult to generate multiple overlapping humans and animals based on keypoint controls solely. These challenges arise from two main aspects: the inherent limitations of existing controllable methods and the lack of suitable datasets. 
First, we design a DiT-based framework, named UniMC, to explore unifying controllable multi-class image generation. UniMC integrates instance- and keypoint-level conditions into compact tokens, incorporating attributes such as class, bounding box, and keypoint coordinates. This approach overcomes the limitations of previous methods that struggled to distinguish instances and classes due to their reliance on skeleton images as conditions. 
Second, we propose HAIG-2.9M, a large-scale, high-quality, and diverse dataset designed for keypoint-guided human and animal image generation. HAIG-2.9M includes 786K images with 2.9M instances. This dataset features extensive annotations such as keypoints, bounding boxes, and fine-grained captions for both humans and animals, along with rigorous manual inspection to ensure annotation accuracy. 
Extensive experiments demonstrate the high quality of HAIG-2.9M and the effectiveness of UniMC, particularly in heavy occlusions and multi-class scenarios.

UNIMC: Taming Diffusion Transformer for Unified Keypoint-Guided
Multi-Class Image Generation
Qin Guo' '
Ailing Zeng²
Dongxu Yue 3 Ceyuan Yang 4 Yang Cao
Hanzhong Guo Fei Shen
Wei Liu
Xihui Liu
Dan Xu
A Semee coealll coecsed Weats Med rE storrt
appeeners augerysa ya dee thi MaAe Ty calr jark
ANEC prseece PRINEE To
DOE M FO Toe UNIT WORCI SOA WIT
WETT wal kast ot
OF PO THEEE andd Tie Eili of cilres iide
UAROO Ei dredaa Tio FNCE DIO0
BRE SErre LITEE VRT S
FRIN DAINO AADO AND BIR
womaa IN Hlaingg dress stands peacefully with liovs and two sheep
A man in green swit sits beside large gorilla
AA young woman in cowboy hat rides horse
(b)
person
liont
Aprilla
horse
Por
SEP
Sheep
Figure 1. Top: We establish HAIG 2. 9M, a large-scale, high-quality, and highly diverse dataset with joint keypoint-level. instance-level, and densely semantic annotations for both humans and animals. Bottom: Based BAIG-2. 9M, design UNIMC, controllable DiT-based framework for keypoint-guider image generation, especially for multi-class (e.g.. (a), (b), (c)) and heavy occlusion scenarios (e.g., (a), (b)). The bottom part of the figure showcases samples generated by UNIMC.
Abstract
datasets. First, we design a DiT-based frame- work, named UNIMC, explore unifying control- lable multi-class image generation. UNIMC in- tegrates instance- and keypoint-level conditions into compact tokens, incorporating attributes such as class, bounding box, and keypoint coordinates. This approach overcomes the limitations of pre- vious methods that struggled to distinguish in- stances and classes due to their reliance on skele- ton images conditions. Second, we propose HAIG-2.. 9M. a large-scale, high-quality, and di- verse dataset designed for seypoint-guided human and animal image generation. HAIG-2. 9M in- cludes 786K images with 2.9M instances This dataset features extensive annotations such as key- points, bounding boxes, and fine finennri captions for both humans and animals, along with rigor- manual inspection ensure annotation curacy. Extensive experiments demonstrate the high quality of HAIG-2. 9M and the effective-
Although significant advancements have been achieved in the progress of keypoint-guided Text- tm-lmage diffusion models, existing mainstream keypoint-guided models encounter challenges in controlling the generation of more general non-rigid objects beyond humans (e.g., animals). Moreover, it difficult generate multiple over- lapping humans and animals based keypoint controls solely. These challenges arise from two main aspects: the inherent limitations of exist- ing controllable methods and the lack of suitable
'The Hong Kong University of Science and Technology Tenneett 'Peking University "The Chinese University of Hong Kong 'The University of Hong Kong National University of Sin- gapore. Correspondence to: Dan Xu <danxu@csc.ust..he
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

UNIMC: Taming Diffusion Transformer for Unified Keypoint-Guidee Multi-Class Image Generation
ness of UNIMC, particularly in heavy occlusions and multi-class scenarios.
condition signal, thus addressing class binding confusion and instance binding confusion. We then propose unified keypoint encoder, a lightweight keypoint encoder that en- codes keypoints of different species into a universal token representation space, avoiding the need for training separate encoders for each species and enhancing both training and inference efficiency. Next, we introduce timestep-aware keypoint modulator, where keypoint tokens are injected into the DiT-based generator to achieve keypoint-level control for multi-class, multi-instance image generation.
1. Introduction
In recent years, diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020) have gained significant attention due to their remarkable performance in image generation. The generation of non-rigid objects, such as humans and animals, is vital across various domains, including ani- mation (Zhang al., 2024; Hu al., 2023), artistic cre- ation (Midjourney, 2024), and animal-related perception tasks (Yu et al., 2021; Yang et al., 2022; Shooter et al., 2021). While class- condiioondi (Dhariwal & Nichol, 2021) and text-conditioned (Rombach et al., 2022; Ramesh et al., 2022) image generation has achieved great success, more controllable generation desirable, especially controlling the structure of non-rigid objects like humans and animals. Previous works on keypoint-guie guided human image genera- tion (Zhang et al., 2023; Ju et al., 2023b; Liu et al., 2024b) have utilized skeleton images as structural conditions. How- ever, it is difficult for those methods to generate multi-class, cullti-instance images of humans and/or animals, especially in scenarios with overlapping instances.
To overcome the limitations of the dataset, we propose HAIG-2. 9M, which includes 786K images with an average aesthetic score (christophschubmann, 2023) of 5.91, cover- ing 31 species classes, and annotated with 2.9M instance- level bounding boxes, keypoints, and captions, averaging 3.66 instances per image. Following (Yang et al., 2022), these 31 classes can be further classified into 15 animal fam- ilies following taxonomic rank to facilitate the evaluation of inter-species and inter-family generalization ability of generative models. Among these, nearly half of the images contain animals, and one-quarter of the images feature both humans and animals. Specifically, ensure data quality and diversity, we crawl 460K images from four high-quality non- commercial data websites (Pexels, 2024; Pixabay, 2024; stocksnap, 2024; unsplash, 2024) and filter 2.07M images from four high-quality datasets (Schuhmann et al., 2024b;a; Sun et al., 2024; Kirillov et al., 2023). We annotate bound- ing boxes, human and animal keypoints, and captions using selection of of (SOTA) models for each an- notation type. First, we label 5% of randomly sampled data, then manually evaluate the annotations, selecting the top-performing model for each type.
We identify two primary reasons for this challenge: 1) Con-
ditional formulation limitations: Most methods plot key- points on images as control signal (Zhang et al., 2023;
Ju et al., 2023b; Liu et al., 2024b), as shown in Fig. 2. 2) Dataset limitations: Most existing keypoint-annotated datasets focus on humans (Lin et al., 2014; Ju et al., 2023a), with only few annotating animals (Yu et al., 2021; Yang et al., 2022). None provide keypoint annotations for images featuring both humans and animals. These datasets are de- signed for perception tasks, lacking the quantity and quality needed for current generative models.
To summarize, our main contributions are three-fold:
We introduce unified DiT-based framework, UNIMC, designed for multi-class, multi-object image generation using keypoint conditions. This framework includes the unified keypoint encoder that encodes keypoints of differ- ent species into the shared representation space and the timestep-aware keypoini modulator for effective keypoint- level control.
Such skeleton image conditions face two main issues: 1) Class binding confusion. The keypoints plotted on the image lack category information. For example, we can- not distinguish if the keypoints should depict cat or dog. 2) Instance binding confusion. For images with overlapping instances, it is challenging to distinguish which instance a keypoint belongs to in the overlapping regions. Moreover, variations in skeleton rendering (e.g., color, line width) further hinder the generation quality. Consequently, using skeleton images as conditions leads to suboptimal performance for multi-class, multi-instance generation.
We propose a unified human and animal image dataset, HAIG-2. 9M, for keypoint-guided image generation. This large-scale and diverse dataset includes 786K im- ages and 2.9M instances with comprehensive annotations such as keypoints, bounding boxes, and captions, cover- ing 31 species classes and 15 animal families, facilitating multi-class, multi-object image generation.
To overcome the condition formulation limitations, we propose UNIMC, unified Diffusion Transformer (DiT)- based (Peebles & Xie, 2023) framework designed for keypoint-guidee guided generation of humans and animals. Specif- ically, we use each instance's 's class name, bounding box and keypoint coordinates instead of skeleton images as the
Extensive experiments demonstrate the high quality of HAIG-2. 9M and the effectiveness and efficiency of UNIMC.
2

---

UNIMC: Taming Diffusion Transformer for Unified Keypoint- Multi-Class Image Generation
Table 1. Comparison of Mainstream Human and Animal Keypoint-Inan Image Paired Datasets (Yang et al., 2022; Lin et al., 2014; Ju et al., 2023a) and General T21 Datasets (Schuhmann et al., 2024b;a; Sun et al., 2024; Kirillov et al., 2023) with HAIG-2. 9M. For COCO, we only calculate statisties of images that contain human keypoint annotations. For general T21 datasets, we exclusively report the scene types and the number of images.
Dataset
Scene Type
Image
Instance
Human
Keypoints
Auimal
Average Aesthetic Score
Caption Length
Average Resolution
Joumey (Sun 2024)
Artistic
4,738,554
SAIB (Kirillov et al., 2023)
Real World
11,000,000
X
LAION-POP (Schuhmann al., 2024k)
Diverse
LAION-AeS Acc--V (Schulmazn et al., 202
Diverse
625,000
X
X
COCO (Lin et al.. 2014)
Real World
58,945
156,165
X
4,981
10.5
578
484
Human- Art (Ju et al., 2023s)
Artistic
50,000
123,131
V
X
5,409
10.5
1297
1298
APT36K (Yang et 2022)
World
53,006
4,516
1080
BAIG-2. SM (ours)
Diverse
786,394
2,874,773
V
5,914
77,48
1775 x 1602
pose UNIMC, which encodes keypoints of different species into a unified representation, enabling keypoint-level control within single framework.
Diffusion Transformer. In this work, we use DiT as our backbone. For details on DiT, see Apx. B.
(1) Class binding confusion
(b) Instance binding confusion
Figure 2. Skeleton image conditions face two main issues: (a) Class binding confusion: difficult to distinguish classes from skeleton images alone; (b) Instance binding confu- sion: challenging to distinguish keypoints of overlapping instances under occlusions (e.g., parts in red box).
Datasets for Human and Animal Image Generation. Large-scale and high-quality datasets are crucial for im- age generation (Chen et al., 2024c). However, existing human-centric datasets often encounter issues such as low quality (Lin al., 2014; Zheng et al., 2015; Han al., 2018), limited diversity (Fu et al., 2022; Liu et al., 2016), limited size (Ju et al., 2023a; Gong et al., 2017), lack of open ac- cess (Liu et al., 2024b), a focus on few person scenes (Qin et al., 2024: Li et al., 2024a), and only skeleton image annotations (Qin et al., 2024). Similarly, animal-centric datasets (Yu et al., 2021; Yang et al., 2022; Ng et al., 2022), most designed for perception tasks. also suffer from very poor quality, small size, and low diversity. Furthermore, none of these datasets offer keypoint annotations for both humans and animals co-existing, hindering the development of seypoint-guides human and animal image generation. In response, we propose HAIG-2 2 9M, as shown in Tab. I, which contains more than ten times the number of images and instances compared to commonly used datasets (Yang et al., 2022: Lin et al., 2014), with significantly improved aesthetic scores. It includes comprehensive annotations for both humans and animals, along with high-quality captions.
2. Related Work
Guided Image Generation. Controllable image generation is critical research direction, evolving from early Generative Adversarial Networks (Creswell et al., 2018; Ma et al., 2017) and Auto-Encoders (Ren et al., 2020; Esser et al., 2021) to the recent diffusion models (Zhang et al., 2023; Mou et al., 2024; Qin et al., 2024; Zhao al., 2024; Li et al., 2023). Keypoint-Guided image generation is vital branch of controllable generation, with recent dif- fusion model-based works like ControlNet (Zhang et al., 2023), T2I-Adapter (Mou et al., 2024), HumanSD (Ju et al., 2023b), and HyperHuman (Liu et al., 2024b) enabling high- quality controllable human synthesis. These methods typi- cally input skeleton images as control signal, but this form faces class binding confusion and instance binding con- fusion as discussed in Sec. 1 and shown in Fig. 2, leading to significant information loss. GLIGEN (Li et al., 2023) initially explored using keypoint positions for human image generation, and subsequent works (Wang et al., 2024: Zhou et al., 2024) adopted similar architectures for instance-level control. However, previous methods focused exclusively on keypoint-guicde human generation, overlooking more general applications such as keypoint-guided animal gener- ation. As demand for flexible control in image generation grows, especially given the importance of animals, pre- cise keypoint-basee based control over both human and animal generation becomes crucial. Additionally, varied keypoint definitions across classes introduce semantic and structural gaps, complicating unified generation. In response, we pro-
3. The Proposed Method
To achieve unified human and animal generation, we present UNIMC, framework that generates keypoint-guided hu- man and animal images within a framework. We intro- duce the preliminaries in Apx. C.I and the problem setting in Sec. 3.1. Then, we introduce UNIMC in Sec. 3.2, which consists of unified keypoint encoder and timestep-aware keypoint modulator. The former is a lightweight module that encodes keypoints from different species into a unified representation space. addressing semantic and structural differences across classes. The latter models keypoint and backbone tokens using self-attention to enable keypoint-
3

---

UNIMC: Taming Diffusion Transformer for Unified Keypoint-Cuided Multi-Class Image Generation
level control.
unified keypoint feature tokens and backbone feature to- kens, enabling keypoint-level control. Fig. Right presents the modulator under different configurations, where is summation function. Configuration (a) works best, named timestep-aware keypoint modulator.
3.1. Problem Setting
We aim to achieve multi-class keypoint-level control in image generation using three conditioning inputs: key- point position p. class name n, and bounding box b. More formally, we aim to learn an image genera- tion model F(cy. [(1 Pr, bı)..... (n,, Pn, b₂)]) condi- tioned on global text prompt cg and per-instance condi- tions (n,, Pi, bi), to generate instances at corresponding positions with various keypoints and classes.
Diffusion models exhibit different spatial structural features at different timesteps (Hertz et al., 2022; Tumanyan et al., 2023). To better utilize keypoint features, we propose an all blocks shared timestep adapter:
F = (Tihhore: Pshare) = Tadosper (t)
3.2. UNIMC Framework
This adapter shares the structure with PIXART-a's AdaLN- single module, enabling the module to be aware of both the structure and the timestep condition. Each added module's features are modulated by the scale Yskare and shift Bshare-
3.2. 1. UNIFIED KEYPOINT ENCODER
Unlike the prevalent approach (Zhang et al., 2023; Ju et al., 2023b) of using skeleton images as conditions, we utilize more compact and informative condition signals. Our con- ditions are the keypoint coordinates, bounding box coor- dinates, and class names of each instance. This compact condition allows class- and instance-awe aware unified encoding and avoids the class binding confusion and instance binding confusion issues that are common in previous approaches. Specifically, each instance's keypoints Pa are expressed as yk, vx)], where (mg, yk) denotes the 2D coordinates and Uy represents the visibility of the k-th keypoint. We parameterize bounding boxes by their top-left and bottom-right corners.
Specifically, we denote the tokens from the backbone as V. We concatenate the tokens g for instances into G. We apply Layer Norm (LN) (Ba et al., 2016) and Multi- Head Self-AAtention (MHSA) to G and V, then select the first m tokens:
= MHSA(LN((V, G))[; m]
(2)
a sequence of tuples [[x1, y/, .i),..., .., (I
Each timestep-aware keypoini modulator incorporates block- wise learnable scale and shift E; (7%, 8.). where i repre- sents the layer index within the backbone. The following operations are performed in this layer:
We convert the 2D point coordinates for each instance's 's key- points P; and bounding boxes b; using a Fourier mapping r() (Mildenhall et al., 2021). Additionally, we encode the class name n, using a T5 (Chung et al., 2024) text encoder and PIXART-a's pretrained text projector, obtaining the class embedding en,, Finally, we concatenate the Fourier embedding and en, and feed them to an MLP to obtain the corresponding keypoints or bounding boxes token g, for the instance i: g = MLP((en,, (p,)))). We use different MLPs for keypoints and bounding boxes. Thus, for each instance i, we obtain g; and g! g.box, If an instance has only one condition, such as keypoint or bounding box, or if some points are not visible, we use learnable mask token (He et al., 2022) e; to represent the invisible parts:
V = (7% + Ysharee V + (3: + +share))
(3)
To preserve the model's initial capabilities, V is passed through zero-initialized linear layer and then added as a residual to V. We denote the structure described above as Config a, and we also explore three other structures:
Config b: Naive Gated SA. We experiment with using the Naive Gated SA architecture directly from GLIGEN (Li et al., 2023). The structure is illustrated in Fig. 3 (b). The learnable scalars % and qa are initialized to zero.
Config c: Time Embedding Using Pretrained. Building on Config a, we explore using the output of the first layer of the pretrained time encoder from PIXART-a to replace "shhrre and Bskare, denoting it as Ep. The structure is il- lustrated in Fig. 3 (c). The aim is to explore whether the pretrained timestep features are effective in fimestep-aware keypoint modulator.
g: MLM((een, s (p:)) + (1 e),)
(1)
where s a binary vector indicating the presence of a spe- cific condition's 's part. Finally, we reorder and concatenate the tokens to group each instance's keypoint token g. kpt with its bounding box token g.ox,, forming a combined token g."!. This process makes the model class- enabling it to en- code keypoints from different classes into the shared space.
Config d: Block-Wis adaLN-Zero Time Embedding. Building on Config a, we omit Tadapter and the bise Iearnable parameters 7/2 and 3,. Instead, we use aeeo for each block as in DiT (Peebles & Xie, 2023). The time encoder for the i-th layer is denoted as Te, and its output (t)). The structure of this module is illustrated in Fig. 3 (d).
3.2.2. TIMEETEP-AWARE KEYPOINT MODULATOR As shown in Fig. 3 Left, we insert self-attentoon layers into the blocks of PIXART-Q to model the relationship between
4

---

UNIMC: Taming Diffusion Transformer for Unified SccyoonntGGuied Multi-Class Image Generation
e lock
xN
Franpt
Different' Thnestep-buare Keypolnnt based Modhlator
Stale
IEN
Text Eanader
Zerol
Zero
S skint
Text
Musi-Tooe Crows Lleetien
The Alkget
Stall, Skilt eif Eu)
Scalt sain att.
Scala, Skill Tul: (hlack wile))
Timesiep-awaa Awaare Keypont baso Madsilttrr
Kepskkss Fealure
Legend
Caffied Erxoder
Sei-Atiention
Sal-Atteelien
Sall-ttweelen
teikAttertion
Laper Norm
Norn
Optioaa Blick
Scale & Saih
Keypaltts Bloucs,
Keypaie Tilom
Figure 3. Timestep-Aware Keypoint based Modulator. Left: The block of PiXART-a. Right: Four modulator variants are proposed to control DiT (detailed in Sec. 3.2.2). All variants leverage self-attention to model backbone and keypoint tokens: (a) and (c) use global-wise timestep modulation (w/o and w pretrained timestep encoder, respectively), (b) adopts GLyGEN-style gated
(Li et al., 2023), and (d) applies block-wise timestep modulation. (a) works best.
①
Collection
?
Filtering
0 Organized Images
P
Conkeining larget animals
Containing humurs animass
Delection Models
Random check
Too liaved humuure or animas
Arstittio Soorm Models
V Qualified
Pexels
Uasplash
Teo low aestielics score
Pillewe
Tep low msolutionn
Safeey
Bounding annotation
>16M Images
NSFW
/ Quplifiee
Rules
Tools
Keypoänts annotation
LAION Ac
D 1.1M Images
V Quelifire
Caption annotation
high quality image websiles
3 P Postprocessing
Qualifirei
high quality dalasets
Balanze of (ilfieeent species
Dillieeent scere byee
Well annotated images
706K Images
(a) colloctto proosses
Annotation
Figure 4. Dataset Collection and Annotation Pipeline. (a) We source over 16M images from high-quality datasets and websites. After rigorous filtering and processing, we retain 786K images. (b) Our data annotated by multiple expert models and subjected to strict manual review.
4. The Proposed HAIG-2. 9M Dataset
mann et al., 2024a). The four datasets originally contain over 16M images. By filtering captions to include our de- sired classes, retain 2.07M images.
In this section, describe the data collection, filtering, and annotation process, illustrated in Fig. 4, to construct HAIG-2. 2. 9M. We collect 31 non-rigid classes, including humans and 30 animal classes, from APT36K (Yang et al., 2022), we refer to these 31 non-rigid classes as our target classes. Each category has 17 corresponding keypoints. All annotation procedures involve 5% random human check and corresponding adjustments to the annotation methods.
Then, we need to filter out images that do not contain the target classes. Specifically, we use two lightweight open- vocabulary object detection models, Grounding-Dinn (Liu et al., 2023) and YOLO- -ooldd (Cheng et al., 2024), to check if the images contain the desired classes. Only when both models detect the target class do we retain the image. Addi- tionally, we exclude images with resolution smaller than 512 x 512 or an aesthetic score (christophschuhman, 2023) below 5.0. We also remove images with overly blurred humans or animals, detected using a combination of edge detection (Canny, 1986) and blurriness metrics based on the Laplacian variance method (Pertuz et al., 2013). Further- more, all images undergo safety checker (CompVVi, 2024) to filter NSFW (Laborde, 2024) content. We ensure bal- ance in the number of instances across different classes by dropping classes with excessive quantities. Ultimately, we obtain 786K images after filtering, with 82. 22.6K filtered from
4.1. Data Collection and Filtering
To ensure data quality and scale, we crawl 460K images from four high-quality image websites: Pexels (Pexels, 2024), Pixabay (Pixabay, 2024), Stocksnap (stocksnap, 2024), and Unsplash (unsplash, 2024). Specifically, we search and crawl images using class names as tags, and rigorously check to ensure all downloaded images are non- commercial. Additionally, we filter images from four large- scale, high-quality image datasets: JourneDDB (Sun al., 2024), SA1B (Kirillov et al., 2023), LAION-PoP (Schuh- mann et al., 2024b), and -AJON-Acstheticss (Schuh-
5

---

UNIMC: Taming Diffusion Transformer for Unified Keypoint-Guided Multi-Class Image Generation
web crawling and 703.4K filtered from existing datasets.
HAIG-2. 9M, 22.6% of the images contain only animals, 24.6% contain both humans and animals, and 52.8% contain only humans. More statistics about HAIG-2 2 9M can be found in Apx. A. Fig. 5(a) illustrates the diversity of styles in n. 9M, while Fig. 5(b) displays an example of our annotation. Additionally, Fig. 1 Top also showcases several examples.
4.2. Data Annotation
To ensure high-quality annotations on such a large-scald dataset, we randomly sample 5K images. For each annota- tion type, we select multiple expert models to annotate the chosen data. We then conduct a user study involving well- trained 10 users to compare the outputs of different models, allowing them choose the one with the best annotation quality. The model with the highest score is subsequently used to annotate the entire dataset.
Table 2. Split of HAIG-2. 9M.
Dataset
Image
Instance
Class
Multi-Class
Multi-Instance
Triining Set
745,828
2,725,484
31
183,376
361,816
Validation Set
39,342
145,504
9,560
19,068
Testing Set
1,224
3,785
31
656
748
Bounding Box Annotation. We compare five expert models: YOLO-World (Cheng et al., 2024), Grounding- DINO (Liu et al., 2023), DITO (Kim et al., 2024), OWI- ViT (Minderer al., 2024), and CoDet (Ma et al., 2024a). Ultimately, we select YOLO- Lorld as our annotation model, as shown in the voting results in Apx. A.3 (a).
Total
786,394
2,874,773
193,592
381,632
Dataset Split. Detailed statisties for each subset of the dataset are provided in Tab. 2. First, for the testing set, we select 40 images for each class, ensuring that each class of images contains multiple classes. Then, we split the remaining images into training and validation sets at an approximately 20 1 ratio. We adopt a class-level partition to ensure the class proportions are balanced between the training and validation sets. The training set comprises 745K images and 2.7M instances, while the validation set consists of 39K images and 145K instances.
Human Keypoint Annotation. We evaluate three expert models: DWPose (Yang et al., 2023), RTMPose (Jiang et al., 2023), and OpenPose (Cao et al., 2017). We choose DWPose as our annotation model, with the voting results presented in Apx. A.3 (b).
Animal Keypoint Annotation. We assess three expert models: ViTPose++H (Xu et al., 2022), X-Pose (Yang et al., 2025), and SuperAnimals (Ye et al., 2024). ViTPose++H is selected as our annotation model, and the voting results can be found in Apx. A.3 (c).
5. Experiments
Implementation Details. We use PIXART-a-1024px (Chen et al., 2024c) as backbone. We use the AdamW opti- mizer (Loshchilov & Hutter, 2017) with a weight decay of 0.03 and a fixed learning rate of 2e-5, we only train the unified keypoint encoder and the timestep-aware key- point modulator. We train at 1024 x 1024 resolution for 8K steps with a batch size of 256 using 8 A800 GPUs. During training, we drop the bounding box condition with 50% probability, the keypoint condition with 15% probability, and the prompt with 10% probability. For evaluation, we utilize the testing set of HAIG-2. 9M.
Caption Annotation. We compare four expert models: 4P (OpenAI, 2024b), CogVLM2 (Wang et al., 2023), LLaVA-1.6-38 1-6-4B (Liu et al. 2024a), and Qwen-VL (Bai et al., 2023). We choose GPT4o and CogVLM2 as our annotation models, using 1:4 ratio for the number of samples annotated. The voting results are shown in Apx. A.3 (d).
The model comparisons and manual checks of annotation results involved approximately 200 person-hours.
Comparison Methods. We compare UNIMC with three cat- egories of representative methods: 1) Base model PIXART- o; 2) ControlNet, which uses skeleton images as conditions; 3) GLIGEN, which uses keypoint coordinates as conditions.
4.3. Data Statisties and Visualization
Statistics Overview. Tab. I provides a comparative anal- ysis of mainstream human and animal keypoint-image paired datasets and HAIG-2 2. 9M, as well as comparisons with mainstream high-quality T2I datasets. Unlike typical datasets such as COCO (Lin et al., 2014), Human-Art (Ju et al., 2023a), and APT36K (Yang et al., 2022), which are exclusively annotated for either humans or animals, our dataset, HAIG-2 2. 9M, uniquely annotates both. Moreover, HAIG 2. 9M covers a more diverse range of scene types and significantly surpasses previous datasets in terms of the number of images and instances, with over ten times the number of images and instances compared these datasets. Additionally, our dataset features higher aesthetic scores, longer caption lengths, and higher average resolution. In
Comparison Datasets. We experiment with four dataset configurations for training: 1) HAIG-2.. 9M; 2) Human dataset COCO; 3) Animal dataset APT36K; and 4) combi- nation of COCO and APT36K.
Evaluation Metries. We adopt commonly-used metrics for comprehensive comparisons from five perspectives: 1) Im- age Quality. FID (Heusel et al., 2017) and KID (Bińkowski et al., 2018) reflect quality and diversity; 2) Text- Text-Image Alignment. CLIP (Radford et al., 2021) text-image similar- ity is reported: 3) Class Accuracy. YOLO-World detects each class and calculates the maximum IoU between detec- tion boxes and the Ground Truth box. If the IoU is higher than 0.5, the class is considered correctly generated; 4) Pose
6

---

UNIMC: Taming Diffusion Transformer for Unified Keypoint-Guides Multi-Class Image Generation
Realistic Style
Cartoos Style
Overall Dense Image Capdion
riker fove abong dee loacf sossct croaing and pisteringe Dresed essoel amon
per'sRn)
combiraabl along shoroles The ornon
exgttrrs slowg and
printed with and Ale Jove This image of's beach Agthlghhiig boad
rder
horse
Person's Caption
The mas drosed as ladt any rabit
foose, dosk Seoae wiile pasdios
Sketch Style
Japanese Style
and slols Roes unay Sion ile cansos gorigg sncomde the horiee Hike
rest-innises cosipeed audile
sall slon
mixcad possee sugenos cployiyy
Jeniny
segoss
scnse maiing
montinn manquo
Class name
person
Class name
herse
Person's ids
Home's Mss
bbax
Hone's
Aesthetic score
5.13
Safety check
pass
(a) Visualization of our dataset
(b) Example of our dataset's 's annotation
Figure 5. Dataset Visualization. (a) HAIG-2 2. 9M contains images of various styles. (b) shows an example from our annotated dataset. For visualization, we render the bounding box and keypoint as images overlaid on the original image. The top right corner displays our detailed captions, including a overall dense image caption, person's caption, and horse's caption. The two boxes in the middle respectively show class name, ID, bounding box, and keypoint annotations for the human and horse separately. For brevity, we omit the specific data for bounding boxes and keypoints. the bottom, we indicate the aesthetic score and whether the image passed the safety check.
"Indoor garden scene with a woman gazing at a small giraffe, natural light streaming in."
orafe
"A young, wet-haired wo moman smiles into the camera, encireled by a chimpanzee..."
sarsol
chimpn BBGG
man and woman traditional dress ride an elaborately painted elephant on a hillside..."
AOOSSOI
"A blonde girl in a pattemed ed dress walks city streets with black bear and a trailing raccoon...
blacke
(a)
(b)
(c)
(c)
(e)
(f)
(g)
Figure 6. Qualitative Comparisons. We provide all methods with required condition formats. From left to right: (a): Input condition, (b): UNIMC trained on HAIG-2. 9M, (c): UNIMC trained on COCO, (d): UNIMC trained on APT36K, (e): UNIMC trained on COCO+APT36K, (f): ControlNet, (g): GLIGEN.
Accuracy. We use DWPose to extract human poses and ViTpose++H to extract animal poses from synthetic images and compare them with the input pose conditions. Average Precision (AP) (Lin et al., 2014) is reported. 5) Human
Subjective Evaluation. We present the human preference study in Apx. D.3.

---

UNIMC: Taming Diffusion Transformer for Unified Keypoint-Guided Multi-Class Image Generation
Table 3. Quantitative Results on Different Methods. We compare our model with PIXART-a (Chen et al., 2024c) and reypoint-guided guided methods (Zhang et al., 2023; Li et al., 2023). For PIXART-a and UNIMC, we first generate 1024 x 1024
results, then resize back to 512 x 512.
Image Quality
Alignment
Class Accuracy (%)
Pose Accuracy (AP)
Methods
FID 4 KID +
C
Human ?
Animal ?
Human t
Animal t
PIXART-a
23.50
7.52
32.17
23.06
13.79
0.19
0.06 20.29
ControlNet
26.72
8,90
32.20
90.67
52.03
29.01
GLIGEN
31.01
11.15
31.44
82,06
9.90
26.58
3.96
UNIMC (ours) 23.63
7.57
32.28
0.236
93.55
3.29
91.71
76.3
30.01
3 3.4%6
28,38
3a. 8%
Tuble 4. Quantitative Results on Different Training Datasets. We bold the textbfbest and underline the second results.
Our improvements over the second method are shown in red.
Image Quality
Alignment
Class Accuracy (%)
Pose Accuracy (AP)
Dataset
FID |
KIDxık J
CLIP f
Human f
Animal
Human f
Animal f
COCO
45.08
18.75
28.07
45.33
4.63
19.25
1.52 19.52
APT36K
43.72
18.20
29.00
21.96
48.95
5.64
COCO+APT36K
33.79
13.55
32.10
67.97
67.83
25.69
21.07
HAIG-2. 9M (ours) 23.63 30.1
7.57
44.1914
32.28
0.09
93.55
37.6%1
91.71
25.2%
30.01
10.8901
28.38
34.7961
5.1. Comparison with Prior Methods
on HAIG-2. 9M significantly improves generation quality and enhances class and pose control capabilities.
Qualitative Analysis. As shown in Fig. 6 (a). (f) and (g),
ControlNet sometimes fails to distinguish classes (e.g., row
Quantitative Analysis. As shown in Tab. 4, training only on COCO results in very poor animal generation capabilities. Since most of the testing set includes animals, the model struggles with animal generation, leading to decline in overall quality, Similarly, training only on APT36K leads to poor human generation capabilities. However, the base model's richer human priors help maintain some human generation ability. Joint training on COCO and APT36K improves the ability to generate both humans and animals but still results in low-quality generation. The model trained on HA AAIG.. 9M outperforms all other configurations across all metrics.
1(f) and struggles with accurate animal pose control (e.g.,
row 2 and 5(f)). GLIGEN tends to generate humans (c.g., row 2(g)) and has difficulty controlling animal poses, lead- ing to significantly degraded image quality (e.g., rows I and 4(g)). In contrast, UNIMC achieves fine-grained control of poses for different classes and instances while maintain- ing high-quality generation, even in the presence of heavy occlusion (e.g.. rows 2 and 3(b)).
Quantitative Analysis. As shown in Tab. 3, PIXART-a and UNIMC outperform ControlNet and GLIGEN in terms of image quality and text-image alignment. UNIMC signifi- cantly surpasses the baseline models in class accuracy and pose accuracy, especially for animal pose accuracy.
5.3. Abalation Study on Model Design
As shown in Tab. 7 of appendix, urations mentioned in Sec. 3 The performance of Config is noticeably superior to the others. This demonstrates that: 1) The commonly used GLIGEN-style gated self-attention in Unet-based diffusion models is not suitable for DiT, leading to the worst performance. 2) Global-wise timestep modu- lation outperforms the more parameter-heavy block-wise timestep modulation. 3) Using newly introduced, non- pretrained timestep encoder is essential. Additionally, we compare the impact of inserting the controller at differ- ent layers under Contig a. By default, the controller is inserted in all 28 layers. Config a(b) involves inserting the controller in the first 14 layers, Config a(C) in the last 14 layers, and Config a(d) in both the first 7 and the last 7 layers. The results indicate that inserting the controller in all layers yields the best performance.
compare the four config-
Both qualitative and quantitative results demonstrate that: 1) unified keypoint encoder outperforms previous methods by encoding richer class and keypoint information into a compact and informative feature space. 2) timestep-ave aware keypoint modulator effectively leverages keypoint tokens to modulate backbone, enabling fine-grained keypoint control.
5.2. Comparison with Different Training Dataset
Qualitative Analysis. As shown in Fig. 6(a)-(e), training only on COCO struggles to generate animals, and when it does, the poses are incorrect. Similarly, training only on APT struggles to control human generation. Training on COCO+АPT36K can generate both humans and animals, but often generates incorrect class (e.g., rows 4(e)) and struggles with aie-grained pose control. All three scenarios degrade generation quality significantly. In contrast, training
8

---

UNIMC: Taming Diffusion Transformer for Unified Ecypoint-Guided Multi-Class Image Generation
6. Conclusion
generation with latent consistency models, arXiv preprint arXiv:2401.05252, 2024b.
In this paper, we present UNIMC, a DiT-based framework im-
specifically designed for high-quality age generation both humans and animals. UNIMC is ca- pable of generating realistic, multi-class, and multi-instance images, leveraging keypoint-level, instance-level, and dense semantic conditions to achieve precise and controllable gen- eration of non-rigid objects. Additionally, we introduce HAIG-2. 2. 9M, the first large-scale, high-quality, and diverse dataset annotated with keypoint-level, instance-level, and densely semantic labels for both humans and animals, de- signed for keypoint-guides guided human and animal image gener- ation. We will discuss limitations in Apx. E.
Chen, J., YU, J., GE, C., Yao, L., Xie, E., Wang. Z., Kwok, J., Luo, P., Lu, H., and Li, Z. Pixart-S aalpaa. Fast training of diffusion transformer for photorealistic text-to-image synthesis. In International Conference Learning Representations, 2024c. URL https: / //openreview. net for umm weeuummidddeeee ie=iii==eeeeee 3m1.
Chen, S., Xu, M., Ren, J., Cong, Y., He, S., Xie, Y., Sinha, A., Luo, P., Xiang. T., and Perez-Rua, J.-M. Gen- trom: Delving deep into diffusion transformers for image and video generation. arXiv preprint arXiv:3312005557 2023.
Impact Statement
Cheng, T., Song, L., Ge, Y., Liu, W. Wang, X., and Shan, Y. Yolo-world: Real-time open-vocabular object detection. arXiv preprint 1rXiv:2401.172700 2024.
This paper presents work whose goal to advance the field of Machine Learning. Our model enhances creativity with precise keypoint-level control but carries misuse risks, we emphasize responsible and transparency.
christophschuhmann.
Aesthetic pre-
dictor v2, 2023.
URL https://
github com/ http://c.ccccpsshhhhhhieeeo wwristtppshhhhhaaan
References
J. Kiros, R., and Hinton, G. E. Layer normalization. arXiv preprint 1rXiv:1607.0eee 1607.06450, 2016.
Chung, H. W., Hou, L.,, Longpre, S., Zoph, B., Tay, Y., Fedus, W. Li, Y., Wang. X., Dehghami, M., Brahma, S., al. Scaling mnstruction-finttune language models. JMLR, 25(70): 2024.
Bai, J., Bai, S., Yang, S., Wang. S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint arXiv:2308. 1 12966, 2023.
Comp checker, 2024.
Stable
diffusion
safety
URL https://
fitcub.com. wtt.m//hrrristtphhhhhmmmm. improvedd-asstteece--peeeee
Bao, F. Nie, S., Xue, K., Cao, Y., Li, C., Su, H., and Zhu, J. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 22669- 22679, 2023.
Creswell, A., White, T., Dumoulin, V., Arulkumaran, K., Sengupta, B., and Bharath, A. A. Generative adversarial networks: An overview. IEEE signal processing maga- zine, 35(1):53336)) 65, 2018.
Binkooski,, M., Sutherland, D. J., Arbel, M., and Gret- ton, A. Demystifying mmd gans. arXiv preprint 2018.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances neural information processing systems, 34:8780-888 01998008794, 2021.
Canny, J. A computational approach to edge detection. IEEE TPAMI, PAMI-8(6)679 1986.
Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12873-12883 2021.
Cao. Z.. Simon. T., Wei. S.-E.. and Sheikh, Y. Realtime 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vi- sion and pattern recognition, pp. 7291-7299 2017.
Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. Scaling rectified flow transformers for high-resolution image synthesis. arXiv preprint arXiv:2403.032000 2024.
Chen, J., Ge, C., Xie, E., Wu, Y.. Yao, L., Ren, X., Wang. Z., Luo, P., Lu, H., and Li, Z. Pixart- Pixart-\sigmaa sigma: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXiv:2403.04692 2024a.
Fu, J., Li. S., Jiang, Y., Lin, K.-Y,, Qian, C., Loy, C. C., Wu, W., and Liu, Z. Stylegan-human: A data-centric odyssey of human generation. In European Conference on Computer Vision, pp. 1-19. Springer, 2022.
Chen, J., WU, Y., Luo, S., Xie, E., Paul, S., Luo, P., Zhao, H., and Li, Z. Pixart- :deeaa: Fast and controllable image

---

UNIMC: Taming Diffusion Transformer for Unified Keypoint-Cuided Multi-Class Image Generation
Gao, P., Zhuo, L.,, Lin, Z., Liu, C., Chen, J., Du, R., Xie, E., Luo, X., Qiu, L.., Zhang, Y., et al. Lumina-t2x: Transform- ing text into any modality, resolution, and duration via flow-based large diffusion transformers. arXiv preprint mrXiv:240500994. 2024.
Conference on Computer Vision and Pattern Recognition, pp. 618-629 629, 2023a.
Ju, X., Zeng, A., Zhao, C., Wang, J., Zhang, L., and Xu, Q. Humansd: A native skeleton- -iriioon-gude diffusion model for human image generation. In International Conference Computer Vision, pp. 15988-15555 15998, 2023b.
Gao, S., Zhou, P., Cheng, M.-M., and Yan, S. Masked diffusion transformer is strong image synthesizer. In International Conference on Computer Vision, pp. 231 3164- 23173, 2023.
Kim, D., Angelova, A., and Kuo, W. Region-centric image- language pretraining for open-vocabulary detection. Eut ropean conference on computer vision, 2024.
Gong, K., Liang, X., Zhang, D., Shen, X., and Lin, L. Look into person: Self-supervised structure-senssttvv learning and new benchmark for human parsing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 932-940, 2017.
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y,, et al. Segment anything. In International Confer- ence on Computer Vision, pp. 4015-4026, 2023.
Han, X., Wu, Z., Wu, Z., Yu, R., and Davis, L. S. Viioo: An amage-based virtual try-on network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 7543-7552, 2018.
Laborde, G. Deep nn for nsfw detection, 2024. URL
model. mmmddel...
Li, S., Fu, J., Liu, K., Wang. W., Lin, K.-Y,, and Wu, W. Cos- micman: text-to-image foundation model for humans. arXiv preprint arXiv:2404.01294 2024a.
Hatamizadeh, A., Song, J., Liu, G., Kautz, J., and Vahdat, A. Diffit: Diffusion vision transformers for image generation. 2023.
arXiv preprint arXiv:2312.021:::
Li, Y. Liu, H., Wu, Q., Mu, F., Yang. J., Gao, J., Li, C., and Lee, Y. Gligen: Open-set grounded text-to-imaae generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22511- 22521, 2023.
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 16000-16009, 2022.
Hertz, A., Mokady, R. Tenenbaum, J., Aberman, K., Pritch, Y., and Cohen-Or, D. Prompt-to-promy im- age editing with eross attention control. arXiv preprint Stii:22008.06220 2022.
Li, Z., Zhang, J., Lin, Q., Xiong. J., Long, Y., Deng, X., Zhang, Y., Liu, X., Huang, M., Xiao, 7.,, et al. Hunyuan- dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding. arXiv preprint WrXXiv2405.06 087488, 2024b.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. European Conference on Computer Vision, pp. 740-755. Springer, 2014.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-688 6851, 2020.
Liu, H., Li, C., Wu, Q., and Lee, J. Visual instruction tun- ing. Advances neural information processing systems, 36, 2024a.
Hu, L., Gao, X., Zhang. P., Sun, K., Zhang, B., and Bo, L. Animate anyone: Consistent and controllable image-to- video synthesis for character animation. arXiv preprint arXiv:2311.17111 2023.
Liu, S., Zeng. Z., Ren, T., Li, F., Zhang, H., Yang. J., Li, C., Yang, J., Su, H., Zhu, J., al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023.
Jiang, T., Lu, P., Zhang, L., Ma, N., Han, R., Lyu, C., Li, Y., and Chen, K. Rtmpose: Real-time multi-person pose estimation based on mmpose. arXiv preprint 1rXiv:2303.07399, 2023.
Liu, X., Ren, J., Siarohin, A., Skorokhodov, I., Li, Y., Lin, D., Liu, X., Liu, Z.,, and Tulyakov, S. Hyperhuman: Hyper-realistic human generation with latent structural diffusion. In International Conference on Learning Rep- resentations, 2024b. URL https: /penreviev... net/ net/forum?2 (tt.oormmmmdduyy44221....
Ju, X., Zeng, A., Wang, J., Xu, Q., and Zhang, L. Human- art: A versatile human-centric dataset bridging natural and artificial scenes. In Proceedings of the IEFE/CVF
10

---

UNIMC: Taming Diffusion Transformer for Unified Keypoint-Cuided Multi-Class Image Generation
Liu, Z., Luo, P., Qiu, S., Wang, X., and Tang. X. Deepfash- ion: Powering robust clothes recognition and retrieval with rich annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1096-1104, 2016.
OpenAL. hello-gpt-4o, 2024b. URL https://openai: http://inxxehelloooeeeee
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In International Conference on Computer Vision, pp. 4195-4205, 2023.
Loshchilov, I. and Hutter, F. Decoupled weight decay regu- larization. In arXiv, 2017.
Pertuz, S., Puig, D., and Garcia, M. A. Analysis of focus measure operators for shape-from- PR, 46(5):1415- 1432, 2013.
Lu, Z., Wang, Z., Huang, D., Wu, C., Liu, X., Ouyang, W., and Bai, L. Fit: Flexible vision transformer for diffusion model, arXiv preprint arXiv:2402. 222.224222.2766, /2376, 2024.
Pexels. Pexels, 2024. URL attps://www.pexet com.
Ma, C., Jiang, Y., Wen, X., Yuan, Z., and Qi, X. Codet: Co-occurrence guided region-wool alignment for open- vocabulary object detection. Advances in neural informa- tion processing systems, 36, 2024a.
Pixabay. Pixabay, 2024. URL https::/pp/xaaaaya/..aaama com.
Podell, D., English, Z., Lacey, K., Blattmann, A., Dock- horn, T., Müller, J., Penna, J., and Rombach, R. SDXL: Improving latent diffusion models for high-resolution im- age synthesis. In International Conference on Learning Representations, 2024. URL https: openreviev. net http/oormmmmii sidiiiddi2200 d12228xg..
Ma, L., Jia, X., Sun, Q., Schiele, B., Tuytelaars, T., and Van Gool, L. Pose guided person image generation. Advances in neural information processing systems, 30, 2017.
Qin, C., Zhang, S., Yu, N., Feng, Y., Yang, X., Zhou, Y., Wang, H., Niebles, J. C., Xiong, C., Savarese, S., et al. Unicontrol: A unified diffusion model for controllable visual generation in the wild. In Advances in neural information processing systems, volume 36, 2024.
Ma. N., Goldstein, M., Albergo, M. S., Boffi, N. M., Vanden- Eijnden, E., and Xie, S. Sit: Exploring flow and diffusion- based generative models with scalable interpolant trans- formers. arXiv preprint arXiv:2401.087400 2024b.
Midjourney. Midjourney, 2024. URL https: : /www.
Radford, A., Kim, J. W., Hallacy, C.,, Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763, PMLR, 2021.
midjourney. com.
Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng, R. Nerf: Representing scenes as neural radiance fields for view synthesis. Communica- tions of the ACM, 65(1))99-100, 2021.
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., and Chen, M. Hierarchical text-conditional image generation with clip latents. arXiv preprint 2rXiv:2204.06125. 1(2):3, 2022.
Minderer, M., Gritsenko, A., and Houlsby, N. Scaling open-vocabulary object detection. Advances in Neural Information Processing Systems, 36, 2024.
Ren, Y., Yu, X.,, Chen, J., Li, T. H., and Li, G. Deep image spatial transformation for person image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7690-7699, 2020.
Mou, C., Wang, X., Xie, L., Wu, Y., Zhang, J., Qi, Z., and Shan, Y. T2i-adapter: Learning adapters dig out more controllable ability for text-to-image diffusion models. In Association for the Advancement of Artificial Intelligence, volume 38, pp. 4296 4304, 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Nair, N. G., Valanarasu, J. M. J., and Patel, V. M. Diff- scaler: Enhancing the generative prowess of diffusion transformers. arXiv preprint urXiv:2404.09977 2024.
Ng, X. L., Ong, K. E., Zheng, Q. Ni, Y., Yeo, S. Y., and Liu, J. Animal kingdom: A large and diverse dataset for animal behavior understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19023-19034, June 2022.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolu- tional networks for biomedical image segmentation, In Medical Image Computing and Computer Assisted Inter- vention, pp. 234 2411. Springer, 2015.
Schuhmann, C., Beaumont, R., Vencu, R., Gor- don, C., Wightman, R. Cherti, M., Coombes, T. Katta, A., Mullis, C. Wortsman, M., al.
OpenAl. Sora: Creating video from text. https:// openai com/ sora, 2024a.

---

UNIMC: Taming Diffusion Transformer for Unified Keypoint-Cuided Multi-Class Image Generation
Laion-aesthetics v2, 2024a.
URL https: Xu, Y., Zhang, J., Zhang, Q. and Tao, D. Vitpose: Simple vision transformer baselines for human pose estimation. Advances in neural information processing systems, 35: 38571-38584, 2022.
#laion aestoonttssttthtis--eee
Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C., Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., et al. Laion pop: 600,000 high- resolution images with detailed descriptions, 2024b. URL https://ai //laion ai /blog//ll///ee wallog/laioone laion-pop/.
Yang. J., Zeng, A., Zhang, R., and Zhang, L. X-pose: Detecting any keypoints. In European Conference on Computer Vision, pp. 249-268. Springer, 2025.
Yang, Y., Yang, J., Xu, Y., Zhang, J., Lan, I.., and Tao, D. Apt-36k: A large-scale benchmark for animal pose estimation and tracking. Advances in neural information processing systems, 35:17301-17711 2022.
Shooter, M., Malleson, C., and Hilton, A. Sydog: A syn- thetic dog dataset for improved 2d pose estimation. arXiv preprint arXiv:2108.00224 2021.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequi- librium thermodynamics. International conference on machine learning, pp. 2256-2265, PMLR, 2015.
Yang, Z., Zeng. A., Yuan, C., and Li, Effective whole- body pose estimation with two-stages distillation. In International Conference on Computer Vision, pp. 4210- 4220, 2023.
stocksnap. stocksnap, 2024.
URL https://
Ye, S., Filippova, A., Lauer, J., Schneider, S., Vidal, M., Qiu, T.. Mathis, A., and Mathis, M. W. Superanimal pretrained pose estimation models for behavioral analysis. Nature Communications, 15(1):5165, 2024.
stocksnap.io/.
Sun, K., Pan, J., Ge, Y., Li, H., Duan, H., Wu, X., Zhang, R., Zhou, A., Qin, Z., Wang, Y., et al. Journeydb: A benchmark for generative image understanding. Advances in neural information processing systems, 36, 2024.
Yu, H., Xu, Y., Zhang, J., Zhao, W., Guan, Z., and Tao, D. Ap- Ap-10k: benchmark for animal pose estimation in the wild. In Advances in neural information processing systems, 2021.
Tian, Y., Tu, Z., Chen, H., Hu, J., Xu, C., and Wang, Y. U-dits: Downsample tokens in u-shaped diffusion trans- formers. arXiv preprint 2rXhv:2055.02730, 2024.
Zhang. H., Chang, D., Li, F., Soleymani, M., and Ahuja, N. Magicpose4d: Crafting articulated models with appearance and motion control. arXiv preprint 2rXh::2405.14017 2024.
Tumanyan, N., Geyer, M., Bagon, S., and Dekel, T. Plug- and-play diffusion features for text-driven image-to- image translation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp- 1921-1930, 2023.
Zhang, L., Rao, A., and Agrawala, M. Adding conditional control to text-to-image diffusion models. In Interna- tional Conference on Computer Vision, pp. 3836-3847, 2023.
unsplash. unsplash, 2024. URL https://unsplash. com/.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E., and Polosukhin, I. At- tention is all you need. Advances in neural information processing systems, 30, 2017.
Zhao, S., Chen, D., Chen, Y.-.,, Bao, J., Hao, S., Yuan, L., and Wong, K.-Y. K. Uni-controlnet: All-in-one control text-to-image diffusion models. In Advances in neural information processing systems, volume 36, 2024.
Wang, W., Lv, Q. Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J.. Yang, Z., Zhao, L., Song, X., et al. Cogylm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.0307 2023.
Zheng, H., Nie, W., Vahdat, A., and Anandkumar, A. Fast training of diffusion models with masked transformers. In TMLR, 2024.
Zheng, L.. Shen, L., Tian, L., Wang. S., Wang, J., and Tian, Q. Scalable person re-identification: A benchmark. In International Conference on Computer Vision, pp. 1116- 1124, 2015.
Wang, X., Darrell, T., Rambhatla, S. S., Girdhar, R., and Misra, I. Instancediffusion: Instance-level control for im- age generation. arXiv preprint arXiv:2402.033290 2024.
Xie, F., Yao, L.,, Shi, H., Liu, 7., Zhou, D., Liu, 7.,, Li, J., and Li, Z. Difffit: Unlocking transfcrability of large diffu- sion models via simple errfiietr-efficient fine-tuning, In International Conference on Computer Vision, pp. 4230- 4239, 2023.
Zhou, D., Li, Y., Ma, F.. Yang, Z., and Yang, Y. Mige: Multi- instance generation controller for text-to-image synthesis. arXiv preprint 240:::2000004000 2024.
12

---

UNIMC: Taming Diffusion Transformer for Unified Keypoint-Guides Multi-Clan Image Generation
A. Dataset
A.1. Licenses
Image Websites:
Pexels! (Pexels, 2024): Creative Commons CCO license.
Pixabay² (Pixabay, 2024): Creative Commons CCO license.
Stocksnap" (stocksnap, 2024): Creative Commons CCO license.
Unsplash (unsplash, 2024): Unsplash+ License
Image Datasets:
JourneDD (Sun al., 2024): JourneDBB customized license?.
SA-IB* (Kirillov et al., 2023): SA-IB Dataset Research License".
LAION-PoPIO (Schuhmann al., 2024b): Creative Common CC-BY 4.0 license.
LAION-Aestherics-V22 (Schuhmann et al., 2024a): Creative Common CC-BY 4.0 license.
Our License: Creative Common CC-BY 4.0 license.
A.2. Statistics
The number of instances per class in HAIG-2. 9M shown in Tab. 6. Notably, because difficult to obtain a large number of images for "black bear" and "polar bear", we use "bear" as supplement to the 31 classes. Due to the scarcity of some classes (e.g., rhino, hippo, and buffalo), our data has a long-tail distribution problem. However, unlike perception tasks that require balanced number of classes, leveraging the priors of large-scale T2I models (e.g., gorilla in Fig. 1 bottom, chimpanzee in Fig. 6 row 4, cheetah in Fig. 6 row 5, buffalo in Fig. row 6, rhino in Fig. 8 row 1 and 2, and hippo in Fig. 8 row 3).
can effectively control the poses of classes with few training samples by
A.3. Annotation Comparison
We present the model comparison voting results mentioned in Sec. 4.2 in Apx. A.3, where the Votes column represents the total number of votes.
B. Related Works
Diffusion Transformer. The Transformer architecture (Vaswani et al., 2017) has achieved success in various domains. In diffusion models, DiT (Peebles & Xie, 2023) and UVTT (Bao et al., 2023) pioneer the Transformer-based diffusion model, with subsequent works refining the architecture (Hatamizadeh et al., 2023; Ma al., 2024b; Lu al., 2024: Tian al., 2024) or improving training efficiency (Gao et al., 2023; Zheng et al., 2024). For Text-to-Image (T2I) synthesis, the
http:////wwwppxels...o con.
"attps://pixabay .cmm/
"attps://stockenapp..//
lbttps://umsplashh.cmm/
aattpe//rmmpplaaa.com///eeeee
hhttps://oo.mmeyy...i.tbbbbbbb
of_
13

---

UNIMC: Taming Diffusion Transtormer for Unified Keypoint- Guided Multi-Class Image Generation
Table 5. Voting Results for Model Selection
(a) Bounding Box
(d) Caption
Annotation
(b) Human Keypoint
(c) Animal Keypoint Annotation
Annotation
Annotation
Model
Votes
Model
Votes
YOLO-Ware World
26199
Model
Votes
Model
Votes
GPT4o
18875
Grounding-DINo
3469
DWPose
36709
ViTPose++H
32092
CogVLM2
18270
DITO
8055
RTMPose
10271
X-Pose
11089
LLaVA-1.6-380
4096
OWL-ViT
4574
OpenPose
3020
SuperAnimals
6819
Qwen- VI.
8759
CoDet
7703
Table 6. Class Counts in the HAIG-2. 9M, Sorted Alphabetically
Class
Count
Class
Count
Class
Count
Class
Count
Antelope
2755
Deer
20701
Monkey
5658
Pig
4018
Bear
22833
Dog
114502
Orangutan
126
Polar Bear
5625
Black Bear
304
Elephant
49457
Panda
1521
Rabbit
623
Buffalo
1219
Fox
7536
Person
2159681
Raccoon
1962
Cat
36396
Giraffe
8508
Pig
4018
Rhino
178
Cheetah
2249
Gorilla
513
Sheep
94183
Tiger
4344
Chimpanzee
321
Hippo
1080
Howling Monkey
1721
Wolf
8976
Cow
100689
Horse
205616
Lion
4400
Zebra
5444
PIXART series (Chen et al., 2024c;b;a) demonstrated more efficient training and higher quality image generation compared to UNet-based models (Rombach et al., 2022; Podell et al., 2024). Other works (Esser et al., 2024; Chen et al., 2023; OpenAI, 2024a; Gao et al., 2024; Li et al., 2024b; Xie et al., 2023; Nair et al., 2024) have also proven the efficiency, potential and scalability DiT. this work, we use PIXART-a as our backbone, which variant of DiT (Peebles & Xie, 2023). We explore various DiT control variants to achieve unified keypoint- control.
C. Method
C.1. Preliminaries
Diffusion Probabilistic Models. Diffusion models (Ho et al., 2020; Sohl-Dickstein et al., 2015) define forward diffusion process to gradually convert the sample x from a real data distribution Pdam (x) into noisy distribution, and learn the reverse process in an iterative denoising way "Sohl-Dickster et al., 2015; Ho et al., 2020). During the sampling process, the model can transform Gaussian noise of normal distribution to real samples step-by-step. The denoising network e. estimates the additive Gaussian noise, which is typically structured as DiT (Peebles & Xie, 2023) or UNet (Ronneberger al., 2015) minimize the mean-squared error:
min Exc,,, [|lc [lle-col (ic-colVa,x+ - ā,c, c,c,c,t
(4)
where x, C ~ Pdata are the sample-condition pairs from the training distribution; e <~ (O is the ground-trufe trnndd-truthho noise; L ((1,T) the time-step and T the predefined diffusion steps; de, is the coefficient decided by the noise scheduler.
Latent Diffusion Model & PIXART-a. The widely- latent diffusion model (LDM) (Rombach et al., 2022), performs the denoising process in separate latent space reduce the computational cost. Specifically, pre-trained VAE (Esser et al., 2021) first encodes the image x to latent space z E(x) for training. At the inference stage, we can reconstruct the generated image through the decoder & (22). In this work, we use PIXART-a (Chen et al., 2024c) as our backbone. which is basic T2I model based on the Latent Diffusion Transformer. achieves higher generation quality while the model parameters and training data are much smaller than those the UNet-based SD series models (Rombach et al., 2022; Podell al., 2024).
14

---

UNIMC: Taming Diffusion Transformer for Unified Keypoint- Multi-Class Image Generation
D. More Results
D.I. Ablation Study of Different Configs of Timestep-aware Keypoint based Modulator
As shown in Tab. 7, we compare the four configurations mentioned in Sec. 3, from Config a to Config d. The performance of Config a is noticeably superior to the others. This demonstrates that: 1) The commonly used GLIGEN-style gated self-attention in Unet-based diffusion models is not suitable for the DiT architecture, leading to the worst performance. 2) Global-wise timestep modulation outperforms the more parameter-heavy block-wise timestep modulation. 3) Using newly introduced, non-pretrained timestep encoder is essential.
Additionally, we compare the impact of inserting the controller at different layers under Config a. By default, the controller inserted in all 28 layers. Config a(b) involves inserting the controller in the first 14 layers, Config a(c) in the last 14 layers, and Config a(d) in both the first and the last 7 layers. The results indicate that inserting the controller in all layers yields the best performance.
Table 7. Ablation on Different Configs of Timestep-Aware Keypoint based Modulator.
Image Quality
Alignment
Class Accuracy (%)
Pose Accuracy (AP)
Methods
FID +
KID xik +
CLIP t
Human f
Animal f
Human
Animal
Config b
42.97
15.60
28.02
51.97
53.44
21.10
18.05
Config
38.66
13.58
29.30
56.92
54.20
21.99
19.26
Config 33,90
12.70
30.39
62.45
64.37
24.38
23.55
Config a(b)
27.93
10.04
30.12
75.33
75.01
28.09
26.55
Config a(c)
26.00
9.28
31.29
82.40
79.85
28.81
27.95
Config a(d)
27.09
9.62
31.81
80.07
77.40
28.46
27.28
Config 23.63
9.184
7.57
18.4%)
32,28 1.5%1
93.55 13.5%1
91.71
14.9501
30.01 4.2%1
28.38 1.3%
D.2. More Qualitative Comparisons
More qualitative comparisons shown in Fig. 7 and Fig. 8.
D.3. Human Preference Study
The human preference study is a crucial subjective metric for evaluating the quality of generative models. We present the user study results from two aspects: 1) generation quality and 2) control ability. We randomly selected 10 samples from the test set and asked 10 participants to evaluate them from two perspectives. For generation quality, participants assessed which method produced the highest quality images. For class and keypoint control ability, they selected the method that generated images where the class and corresponding keypoints most accurately matched the given conditions. We then recorded the number of participants who chose each option. Tab. 8 presents the user study results across different methods, and Tab. 9 shows the user study results across different training datasets.
Table 8. Comparison of human evaluations across different methods, supplementing Table 4 in the main paper.
Methods
Generation Quality
Class and Keypoint Control Ability
Pixart ControlNct
47
0
11
9
GLIGEN
2
4
UniHA (ours)
87
E. Limitations
Similar to AP10K (Yu et al., 2021), our dataset also faces long-tail distribution problem. Additionally, our dataset has a limited number of categories, and the controllable generation of arbitrary non-rigid objects remains a challenging task.
15

---

UNIMC: Taming Diffusion Transtormer for Unified Keypoint- Guided Multi-Class Image Generation
"Orange kitten with bright blue eyes on rocky ground."
Gart
"Couple enjoying pizza and guitar with their dog in cozy...
"Woman and majestic lion face to face in ethereal glow."
porsol
"Medieval couple in elaborate red attire, posing with fox." "
"Man and majestic tiger resting by a serene waterfall."
persso
Higer
"Armored soldier and a massive, armored buffalo through forest."
pe 5DH vnFfalo
"Young woman and majestic tiger sharing a moment in jungle."
porsonn
tiger
"Rider on a galloping horse silhouetted against vibrant sunset."
person
hnbrsc
Figure 7. Additional Qualitative Comparisons (I). From left to right: (a): Input condition, (b): UNIMC trained on HAIG-2. 9M, (c): UNIMC trained on COCO, (d): UNTMC trained on APT36K, (e): UNIMC trained COC0+АРT36K,
(I) ControlNet, (g): GLIGEN.
16

---

UNIMC: Taming Diffusion Transformer for Unified Keypoint-Guidet Multi-Class Image Generation
"Two massive rhinos standing in a golden autumn forest." "
"Close-up of a rhino's face in dense green jungle."
Pilnoo
"Two hippopotamuses wading peacefully in a tranquil river."
kipppo
"Two cowboy riders riding horses on a dusty trail."
Viorse
"Smiling woman holding her small dog against graffiti wall."
DR
doo
"Tribal woman and small raccoon in a mystical forest."
"Child and gentle fox sitting in an enchanted blue forest." "
person fox
"Dedicated artist painting a tiger in lush green jungle." "
porsoo
tiger
(a)
(b)
(c)
(d)
(e)
if
(g)
Figure 8. Additional Qualitative Comparisons (II). From left to right: (a): Input condition, (b): UNIMC trained HAIG-2. 2. 9M, (c): UNIMC trained COCO, (d): UNIMC trained on APT36K, (e): UNIMC trained CоCO+АРT36K,
(f): ControlNet, (g): GLIGEN.
17

---

UNIMC: Taming Diffusion Transformer for Unified Keypoint-Guides Multi-Class Image Generation
Table 9. Comparison of human evaluations for our method using different training datasets, supplementing Table in the
main paper.
Datasel
Generation Quality
Class and Keypoint Control Ability
COCO
2
APT36K
COCO+APT36K
4
HAIG-2.9M (ours)
93
96
F. Significance of Including Animals for Conditional T2I Generation
The inclusion of animals is crucial for enhancing the diversity and generalization capabilities of conditional text-to-image generation models. Animals, like humans, are significant part the biological world and can easily represented using keypoints, Animals are non-rigid and their deformations can be controlled with keypoints. They introduce broader range of pose variations, textures, and anatomical structures, which can greatly benefit models that need generalize across different classes. This contributes to the development of more controllable foundational visual generation models.
18


Reviews:
[{"review_id": "gUBO81y5hV", "reviewer": "Reviewer_kiWJ", "summary": "The paper introduces UNIMC, a unified Diffusion Transformer framework for keypoint-guided multi-class image generation, and HAIG-2.9M, a large-scale dataset with 786K images and 2.9M instance annotations covering humans and 30 animal classes. UNIMC addresses limitations in existing approaches by using explicit class names, bounding boxes, and keypoint coordinates instead of skeleton images, which solves \"class binding confusion\" and \"instance binding confusion\" problems. The framework employs a unified keypoint encoder that maps different species' keypoints into a shared representation space and a timestep-aware keypoint modulator that injects keypoint tokens into the DiT backbone for fine control. Experiments demonstrate UNIMC outperforms previous methods like ControlNet and GLIGEN in image quality metrics, class accuracy, pose accuracy, and handling complex scenarios with multiple overlapping humans and animals.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer kiWJ,**\n\nThank you for your review and constructive comments. During the rebuttal period, we have made every effort to address your concerns. The detailed responses are provided below:\n\n> Q1: Details about the computational requirements of UNIMC compared to baseline methods.\n> \n\nEach method is trained using different datasets and GPU configurations, and their reported training costs are measured in different formats. For example, **ControlNet** reports GPU-hours, while **GLIGEN** uses steps * batch size. Therefore, directly comparing training costs across methods is not meaningful. In the table below, we report the **officially reported training costs of baseline methods**, along with our method’s training cost in both formats. Despite achieving significantly better performance than the baselines, our method maintains a relatively low training cost. For inference, all methods are evaluated on a **single RTX 3080 Ti** with **FP16 precision**, using a **50-step DDIM sampler** and **Flash-Attention-v2**. The generation resolution is **512×512**.\n\nOur method is more efficient than ControlNet, and although the inference time and memory usage are slightly higher than those of GLIGEN, the performance improvements over GLIGEN are substantial.\n\n| **Method** | **Training cost** | **Inference time (s/image)** | **Inference memory (GB)** |\n| --- | --- | --- | --- |\n| ControlNet | 300 GPU-hours with Nvidia A100 80G | 4.51 | 5.79 |\n| GLIGEN | Steps = 500K, Batch size = 32 | 4.01 | 4.90 |\n| UNIMC (Ours) | Steps = 8K, Batch size = 256 ≈ **576 GPU-hours** with Nvidia A800 80G | 4.23 | 5.08 |\n\n> Q2: How well does the model generalize to unseen animal species not present in the HAIG-2.9M dataset?\n> \n\nAlthough our model is trained on only **31 classes**, it generalizes well to unseen categories due to two key factors:\n\n1. the **structural similarity across different species**, and\n2. our use of a **text encoder** for class representation, which allows for extension to arbitrary categories.\n\nIn addition, the underlying **pretrained T2I model** is inherently capable of class-to-image generation for a wide range of categories. Together, these components enable our method to theoretically scale to any class.\n\nTaking **kangaroo** as an example, we observe that its keypoint structure is similar to that of humans. Thus, we reused human keypoints to generate a kangaroo. As shown in [Fig.1](https://ibb.co/9mfwbtc8), the generated pose and spatial structure roughly align with the keypoints, suggesting that our model possesses a certain degree of generalization to **unseen species**.\n\nTherefore, we believe that as long as the training set covers a sufficient diversity of **structurally representative animal classes**, our model can generalize to a broad range of animal species.\n\n> Q3: Failure cases or limitations of UNIMC.\n> \n\nWe observed that for some categories, a few keypoints are not perfectly controllable. For example, in the case of **cat** shown in [Fig.2](https://ibb.co/qLYjrjxn), the tail fails to be accurately generated. Beyond such rare cases, we did not observe significant failure cases in the test set.\n\n> Q4: What would be the minimum computational setup required to fine-tune UNIMC on a smaller dataset for a specific application?\n> \n\nOur use of 8 A800 GPUs was intended to accelerate training on a large-scale dataset. However, the UNIMC model itself is lightweight, with only **0.93B parameters** (and **0.35B trainable**). Under FP16 precision and batch size = 1, it requires only **12GB of GPU memory**.\n\nWe fine-tuned UNIMC on a dataset of **1,000 cat images** using a **single RTX 3090**, with **batch size = 4** and **2K training steps**, taking about **4 hours**. The resulting **AP for the cat category reached 28.05**, and qualitative results are shown in [Fig.3](https://ibb.co/d4659mqZ).\n\n> Q5: How did you ensure the quality of keypoint annotations in HAIG-2.9M?\n> \n\nAs described in Section 4.2, we first randomly sampled **5K images** and annotated them using several expert models. We then conducted a **user study** to select the best-performing model. As a result, our keypoint annotations reach the upper bound achievable by current pretrained models.\n\nMoreover, the animal species in our dataset are already included in the training data of the selected **pretrained keypoint estimator**. During annotation, we manually **sample and discard images with low-quality annotations**, and we also filter out images where keypoint predictions have **low confidence**.\n\nTo further reduce **ambiguous keypoint placements**, we feed the  **bounding box** **detection results** into the keypoint estimator and **restrict keypoint detection to the corresponding bounding boxes.** This ensures that the estimator focuses only on the relevant regions, thereby improving annotation accuracy and reliability.", "claims_and_evidence": "The primary claims and their supporting evidence include:\n\n1. **UNIMC's effectiveness for keypoint-guided generation**: The authors provide extensive quantitative metrics (FID, KID, CLIP scores, class accuracy, pose accuracy) in Tables 3 and 4 showing UNIMC outperforms baseline methods. This is reinforced by qualitative examples in Figures 6-8 that visually demonstrate the model's capabilities.\n\n2. **Superiority of HAIG-2.9M dataset**: Table 1 provides a clear comparison with existing datasets, showing HAIG-2.9M's advantages in scale, diversity, and annotation quality. The benefits of training on this dataset are evidenced in Table 4, showing significant performance improvements compared to training on COCO, APT36K, or their combination.\n\n3. **Effectiveness of the unified keypoint encoder and timestep-aware keypoint modulator**: The ablation studies in Table 7 methodically compare different configurations, showing Config a (the proposed approach) consistently outperforms alternatives across all metrics.\n\n4. **Solution to class binding and instance binding confusion**: Qualitative examples in Figures 6-8 demonstrate the model's ability to correctly generate the appropriate class and manage multiple instances, especially in rows with overlapping subjects.\n\n5. **Human preference**: Tables 8 and 9 provide human evaluation results that align with the quantitative metrics, strengthening the claim that UNIMC produces better results.\n\nThe only claim that could benefit from stronger evidence is the model's efficiency, which is mentioned but not extensively benchmarked in terms of computational requirements or training/inference times compared to alternatives.", "methods_and_evaluation_criteria": "The methods and evaluation criteria in this paper are well-aligned with the problem of keypoint-guided multi-class image generation.\n\n**Methods:**\n- Using explicit class names, bounding boxes, and keypoint coordinates instead of skeleton images directly addresses the identified limitations of previous approaches\n- The DiT-based architecture is appropriate for high-quality image generation\n- The unified keypoint encoder sensibly maps different anatomies into a shared representation space\n- The HAIG-2.9M dataset is particularly well-suited as it directly addresses the identified gap in existing datasets by providing joint annotations for both humans and animals across diverse scenes.\n\n**Evaluation Criteria:**\n- Image quality metrics (FID, KID) are standard and appropriate\n- Text-image alignment (CLIP) sensibly evaluates prompt faithfulness \n- Class accuracy using YOLO-World effectively assesses whether correct subjects were generated\n- Pose accuracy (AP) appropriately measures keypoint control fidelity\n- Human preference studies validly capture subjective quality assessments\n- Comparison against relevant baselines (ControlNet, GLIGEN) provides meaningful context\n- Ablation studies effectively isolate component contributions", "theoretical_claims": "The paper doesn't contain any formal mathematical proofs or theoretical claims requiring verification.", "experimental_designs_or_analyses": "The experimental designs and analyses appear sound and appropriate. The paper uses relevant baselines (PIXART-α, ControlNet, GLIGEN), comprehensive metrics (image quality, alignment, class/pose accuracy), systematic ablation studies, cross-dataset comparisons, and human evaluations. The experiments effectively isolate component contributions and demonstrate UNIMC's advantages.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "## Other Weaknesses:\n\n1. **Computational Requirements**: The paper lacks discussion of computational costs and efficiency. Information about training time, inference speed, and resource requirements would help assess practical applicability.\n2. **Limited Diversity of Animal Species**: While 30 animal classes is a good start, it still represents a limited subset of the animal kingdom. Discussion of how the approach might generalize to unseen species would strengthen the paper.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Could you provide details about the computational requirements of UNIMC compared to baseline methods? Specifically, what are the training time, memory requirements, and inference speed differences between UNIMC and alternatives like ControlNet and GLIGEN?\n\n2. How well does the model generalize to unseen animal species not present in the HAIG-2.9M dataset? For instance, if provided with keypoints for a species like a kangaroo or platypus that wasn't in the training data, how would UNIMC perform?\n\n3. Did you observe any systematic failure cases or limitations of UNIMC? Understanding typical failure modes would help assess the robustness of the approach.\n\n4. The paper mentions using 8 A800 GPUs for training. What would be the minimum computational setup required to fine-tune UNIMC on a smaller dataset for a specific application?\n\n5. How did you ensure the quality of keypoint annotations in HAIG-2.9M, particularly for animal species where keypoint definitions might be less standardized than for humans? What was the process for handling edge cases or ambiguous keypoint placements?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A4Gx39ACje", "reviewer": "Reviewer_2vGV", "summary": "This paper proposes a framework (UNIMC) that generates images containing multiple objects (including humans and various other entities) by leveraging joint keypoints and introduces a large-scale dataset (HAIG-2.9M) to support this approach. Unlike conventional keypoint-based image control methods, UNIMC utilizes a unified keypoint encoder that encodes keypoints of various classes and instances into a shared representation space, enabling more precise multi-class and multi-instance image generation. By leveraging a large-scale dataset, the proposed method enhances the performance of keypoint-based image generation models, particularly in multi-object and occlusion-heavy scenarios, while maintaining high image quality.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer 2vGV,**\n\nThank you for your review and constructive comments. During the rebuttal period, we have made every effort to address your concerns. The detailed responses are provided below:\n\n> Q1: Applicability to other models.\n\nOur method is designed to be compatible with **almost all DiT-based architectures**, due to the following reasons:\n\n1. The **Unified Keypoint Encoder** is model-agnostic and does not rely on any specific architectural design.\n2. The **Timestep-aware Keypoint Modulator** is implemented using only self-attention and does not depend on any model-specific components.\n\nTo demonstrate the general applicability of our method, we conducted experiments on [Hunyuan-DiT](https://github.com/Tencent/HunyuanDiT). The specific experimental results are shown in the table below. Our method successfully equips **Hunyuan-DiT** with the ability to perform **keypoint-conditioned multi-class image generation**.\n\n| **Methods**           | **Human Class ⬆️** | **Animal Class ⬆️** | **Human AP ⬆️** | **Animal AP ⬆️** |\n| --------------------- | ----------------- | ------------------ | -------------- | --------------- |\n| Hunyuan-DiT           | 25.49             | 16.77              | 0.26           | 0.14            |\n| **Hunyuan-DiT+UNIMC** | **94.06**         | **91.89**          | **30.92**      | **28.75**       |\n\n> Q2: Analysis of training and inference speed.\n\nPlease refer to **R1-Q1** and **R1-Q4**. Our method is highly efficient in both training and inference.\n\nSpecifically, it requires only **576 A800 GPU-hours** to complete training. At inference time, it is **more efficient than GLIGEN** and **comparable to ControlNet**. Moreover, our method can quickly achieve strong performance even when fine-tuned on small-scale datasets.\n\n> Q3: Comparison with various text-based and Visual-Language Model (VLM)-based image generation models.\n\nOur method is orthogonal to general text-to-image (T2I) or Visual-Language Model (VLM)-based image generation approaches. These models lack the ability to generate images conditioned on **keypoints from different object categories**. In contrast, we introduce a framework that enables **keypoint-conditioned multi-class image generation**.\n\nThe table below provides a comparison between **UNIMC** and recent state-of-the-art T2I models. Notably, our method achieves **substantial improvements in Human/Animal class accuracy and keypoint-level AP**, demonstrating its superiority in controllability and structural alignment.\n\n| **Methods** | **Human Class ⬆️** | **Animal Class ⬆️** | **Human AP ⬆️** | **Animal AP ⬆️** |\n| ----------- | ----------------- | ------------------ | -------------- | --------------- |\n| Flux.1-dev  | 25.47             | 20.01              | 0.27           | 0.13            |\n| Kolors      | 24.71             | 18.90              | 0.20           | 0.15            |\n| **UNIMC**   | **93.55**         | **91.71**          | **30.01**      | **28.38**       |\n\n[Fig.4](https://ibb.co/Tq4WQWqv) presents a qualitative comparison of **UNIMC** with **Flux** and **Kolors**. As shown, the latter two models, relying solely on the prompt, fail to generate images that align with the provided keypoints.\n\n> Q4: Why our method outperforms prior work in keypoint-conditioned multi-class image generation?\n\nThe core objective of our method is to enable keypoint-conditioned multi-class image generation.  \nTo this end, we introduce a **Unified Keypoint Encoder** that explicitly models the relationship between object categories and their corresponding keypoints, and a **Timestep-aware Keypoint Modulator** that enables fine-grained, keypoint-level feature modulation.\n\nIn contrast to prior methods, our approach effectively addresses the challenges of **class binding fusion** and **keypoint binding fusion**, allowing the model to learn precise keypoint control across multiple classes simultaneously.  \nAs a result, **UNIMC achieves significant improvements over baseline methods** on this task.\n\n> Q5: Additional contributions beyond HAIG-2.9M and UNIMC.\n\nOur primary contribution is extending **keypoint-conditioned image generation** from a single class (human) to a **general multi-class setting**. To achieve this goal, we propose both the **HAIG-2.9M dataset** and the **UNIMC model**, which together enable **keypoint-conditioned multi-class image generation** across diverse object categories.", "claims_and_evidence": "• The paper claims that the proposed method outperforms existing keypoint-conditioned image generation models such as GLIGEN and ControlNet, which generate images containing multiple objects. This claim is supported by quantitative evaluations in the experimental results, where various metrics demonstrate the superiority of the proposed approach.\n\n• In occlusion scenarios, qualitative evaluation results show that objects in the generated images maintain accurate poses by leveraging per-object keypoints.\n\n• The proposed dataset can be effectively utilized for keypoint-based image generation and contains over ten times more images and objects compared to major existing datasets such as COCO and Human-Art, while also maintaining high quality. This is detailed in Table 1 of the paper.", "methods_and_evaluation_criteria": "The proposed method introduces a unified keypoint encoder trained to generate high-quality images that accurately distinguish various classes and multiple objects. Previous approaches struggled with keypoint-based control due to inconsistencies in keypoint formats across classes and challenges in differentiating objects within the same class. This method effectively addresses these issues, marking a significant improvement.\n\nUnlike existing models that rely solely on joint keypoints for generating multi-class and multi-object images, the proposed approach takes a different direction. By integrating both keypoint and class information during training, it offers a more structured and intuitive solution. This innovation appears both practical and effective. However, a limitation of the study is that it does not verify whether the method can be applied to other models, leaving room for further validation.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The paper effectively employs performance metrics commonly used in generative models and conducts both quantitative and qualitative evaluations. By comparing various datasets and models, the experiments are well-structured and appear valid. The results demonstrate that UNIMC maintains accurate poses using per-object keypoints, even in occlusion scenarios, highlighting its robustness.\n\nHowever, the study lacks an analysis of training and inference speed, which are crucial factors for practical application. Providing additional details on these aspects would enhance the clarity of the research. Furthermore, examining whether this method can be applied to other diffusion models would help establish it as a more generalized approach.", "relation_to_broader_scientific_literature": "The paper presents a detailed comparison with ControlNet and GLIGEN, effectively illustrating the limitations of existing methods. It directly evaluates the proposed model against PIXART-α, its base model, as well as other keypoint-conditioned image generation models like ControlNet and GLIGEN. However, a broader comparison with various text-based image generation models would have further strengthened the study.", "essential_references_not_discussed": "While the paper compares various models and their generative performance, it does not provide a thorough comparison with the latest Visual-Language Model (VLM)-based image generation approaches. Incorporating both qualitative and quantitative evaluations against state-of-the-art models in recent research trends would have broadened the study’s scope and enhanced its credibility.", "other_strengths_and_weaknesses": "Strengths\n\nThis study introduces a new dataset designed for multi-class and multi-object image generation based on joint keypoints, addressing the traditional limitation where such datasets were primarily designed for recognition tasks. To ensure accurate annotations, even in images with overlapping objects, bounding boxes, keypoints, and captions were incorporated, ultimately enhancing the quality of image generation.\n\nWeaknesses\n\nWhile the paper demonstrates the superiority of the proposed method through comparative experiments with GLIGEN and ControlNet, it lacks a detailed analysis explaining why it outperforms existing approaches. Additionally, a comparison with Fusion Encoder-based Text-to-Image models, which align with recent research trends, would have further strengthened the study’s impact. Furthermore, while the HAIG-2.9M dataset and the UNIMC model are repeatedly emphasized as key contributions, additional novel elements would help further differentiate this work from prior research.", "other_comments_or_suggestions": "• The contributions of the paper are somewhat repetitive, and it would be beneficial to highlight additional novel contributions.", "questions_for_authors": ".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TpuTQyMIyR", "reviewer": "Reviewer_Rf97", "summary": "The paper proposes a DiT based framework UniMC for keypoint guided multi-instance image generation and introduces HAIG-2.9M dataset designed for keypoint-guided human and animal image generation. Experiments are conducted on COCO, APT36K, and HAIG-2.9M datasets to show the efficacy of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer Rf97,**\n\nThank you for your review and constructive comments. During the rebuttal period, we have made every effort to address your concerns. The detailed responses are below:\n\n> Q1: The need for a joint human-animal keypoint-annotated dataset.\n> \n\nAs shown in **Table 4** and **Figure 6** of the main paper, training on separate human and animal keypoint datasets makes it difficult to generate images with multiple classes, limiting the development of more general keypoint-conditioned generation.\n\n> Q2: Results on multi-instance subset.\n> \n\nWe present results on the **multi-instance subset** below. Our method still performs well, while baseline models decline, demonstrating that **UNIMC** performs better in multi-class scenarios.\n\n| Methods | FID ⬇️ | KID ×1k ⬇️ | CLIP **⬆️** | Human Class **⬆️** | Animal Class **⬆️** | Human AP **⬆️** | Animal AP **⬆️** |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| **ControlNet** | 27.03 | 9.15 | 31.96 | 90.65 | 31.51 | 26.47 | 9.86 |\n| **GLIGEN** | 32.51 | 12.05 | 31.04 | 80.19 | 6.31 | 25.44 | 1.58 |\n| **UNIMC** | **23.51** | **7.59** | **32.20** | **93.56** | **91.50** | **29.39** | **28.67** |\n\n> Q3: Results on COCO+APT testing datasets.\n> \n\nWe have added results on the **COCO+APT testing set**, and similar to HAIG, our method demonstrates strong performance and surpasses all baseline methods.\n\n| Methods | FID ⬇️ | KID ×1k ⬇️ | CLIP **⬆️** | Human Class **⬆️** | Animal Class **⬆️** | Human AP **⬆️** | Animal AP **⬆️** |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| **ControlNet** | 25.00 | 9.12 | 30.02 | 90.01 | 30.09 | 26.92 | 9.50 |\n| **GLIGEN** | 32.41 | 12.09 | 29.32 | 80.10 | 5.09 | 24.49 | 0.90 |\n| **UNIMC** | **24.09** | **8.01** | **30.06** | **93.42** | **92.01** | **29.65** | **28.30** |\n\n> Q4: Performance with and without multiple instances in the dataset.\n> \n\nThe training performance of HAIG with and without multiple instances shows that adding multi-instance images improves all metrics.\n\n| Variants | FID ⬇️ | KID ×1k ⬇️ | CLIP **⬆️** | Human Class **⬆️** | Animal Class **⬆️** | Human AP **⬆️** | Animal AP **⬆️** |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| **W/O multiple instances** | 25.00 | 8.06 | 31.00 | 91.04 | 89.09 | 24.44 | 22.65 |\n| **W multiple instances** | **23.63** | **7.57** | **32.28** | **93.55** | **91.71** | **30.01** | **28.38** |\n\n> Q5: Comparison with **[1]MIGC** and **[2]InstanceDiffusion**.\n> \n\nThe table below presents the **quantitative comparison** with **MIGC** and **InstanceDiffusion**. **MIGC** controls via bounding boxes and classes, while **InstanceDiffusion** can control through bounding boxes, points, and classes. Compared to previous baseline methods, both of these models show significant improvements in class control accuracy. InstanceDiffusion, which adds point control, also improves keypoint control accuracy.\n\n| Methods | FID ⬇️ | KID ×1k ⬇️ | CLIP **⬆️** | Human Class **⬆️** | Animal Class **⬆️** | Human AP **⬆️** | Animal AP **⬆️** |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| **MIGC** | 23.78 | 8.06 | 31.29 | 92.06 | 88.04 | 17.04 | 15.99 |\n| **InstanceDiffusion** | 24.06 | 9.17 | 30.99 | 89.70 | 89.00 | 21.06 | 20.04 |\n| **UNIMC** | **23.63** | **7.57** | **32.28** | **93.55** | **91.71** | **30.01** | **28.38** |\n\nThe **quantitative comparison** can be seen in [Fig.5](https://ibb.co/mFCJztLW).\n\n> Q6: What is the number of images used for FID comparison?\n> \n\nAs mentioned in the **Implementation Details** section, we used the **HAIG-2.9M testing set** for evaluation, with **1,224** images.\n\n> Q7: Why not annotate with the ensemble of models?\n> \n\nRelevant works, such as Pixart-sigma, have shown that using a better annotation model improves the model's performance. Additionally, animal pose estimation is not as mature as human pose estimation, and only selecting the best model ensures that we can achieve usable results.\n\n> Q8: Bias in Evaluation.\n> \n\nThank you for your insightful question. Please refer to **Q3** — our method also achieves strong performance on human-annotated datasets, which supports the **validity of our evaluation results** and the **reliability of our evaluation methodology**.\n\n> Q9: Difference from coordinate regression tasks.\n> \n\nOur task is to extend **keypoint-conditioned image generation** from a single category (human) to a **multi-class setting**. To achieve this, we use keypoint **coordinates as conditional inputs** to guide image generation. In contrast, prior works on **coordinate regression** use coordinates as **outputs**, not as conditioning inputs. While these works demonstrate that **DiT** can model coordinate distributions, they do not show that DiT can **generate images conditioned on coordinates**, which is the core challenge of our work. Therefore, the prior use of DiT for coordinate regression is only loosely related to our task and does not diminish the novelty of our proposed approach.", "claims_and_evidence": "The paper motivated the absence of a joint Human-animal keypoint datasets for training a unified model and introduced the HAIG-2.9M dataset. However, the need for a joint dataset is not justified. An ablation study on the performance for a subset of testing images that contains only multiple instances per image in HAIG-2.9M is not shown.", "methods_and_evaluation_criteria": "Table 4 shows the results of using different datasets for training and evaluating on the HAIG-2.9M testing dataset. However, this is not a fair comparison since it is essentially comparing cross dataset generalization to in dataset generalization. It should also show evaluation on COCO+APT testing datasets.", "theoretical_claims": "There are no theoretical claims in the paper.", "experimental_designs_or_analyses": "Missing comparisons on related prior works such as\n\n[1] Dewei Zhou, You Li, Fan Ma, Xiaoting Zhang, Yi Yang. MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis, CVPR 2024\n\n[2] Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, Ishan Misra. InstanceDiffusion: Instance-level Control for Image Generation, CVPR, 2024\n\nHow does the proposed method compare qualitatively and quantitatively against [1,2]?", "relation_to_broader_scientific_literature": "The proposed method is not entirely novel as Diffusion Transformers have shown to be useful for coordinate regression in prior works [1-4]", "essential_references_not_discussed": "[1] Naoto Inoue, Kotaro Kikuchi, Edgar Simo-Serra, Mayu Otani, Kota Yamaguchi LayoutDM: Discrete Diffusion Model for Controllable Layout Generation, CVPR 2023.\n\n[2] Yilin Wang, Zeyuan Chen, Liangjun Zhong, Zheng Ding, Zhuowen Tu, Dolfin: Diffusion Layout Transformers without Autoencoder, ECCV 2024\n\n[3] Runyang Feng, Yixing Gao, Tze Ho Elden Tse, Xueqing Ma, Hyung Jin Chang DiffPose: SpatioTemporal Diffusion Model for Video-Based Human Pose Estimation, ICCV 2023\n\n[4] Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection, ICCV 2023.", "other_strengths_and_weaknesses": "The paper uses DiffPose for annotating the HAIG-2.9M dataset and also evaluate using the same model. This may introduce a bias in the evaluation.", "other_comments_or_suggestions": "The major concern is the missing comparisons with closely related prior work [1,2].", "questions_for_authors": "What is the number of images used for FID comparison?\n\nWhat is the performance of HAIG with and without multiple instances in the dataset? It will be better to show the performance improvement coming from multi-instance images.\n\nWhy not annotate with the ensemble of models instead of the best model (Sec 4.2, L275).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hjcNiJYlI0", "reviewer": "Reviewer_u8PW", "summary": "This paper proposes a dataset of human and animal images, and their keypoints, bounding boxes, and fine-grained captions. The dataset includes 786K images with 2.9M instances, averaging 3.66 instances per image. The annotations are obtained from the best among several candidates, different annotations having different best choices. Based on this dataset, the paper also proposes a controllable DiT-based framework for keypoint-guided image generation. The model can generate multiple subjects using their keypoints, class, bounding box, and a global text prompt. Both qualitative and quantitative results shows that the model significantly outperforms the previous work.\n\n## update after rebuttal\nThe rebuttal addresses my most concerns. Since I already gave accept, I will keep my rating.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Dear Reviewer u8PW**,\n\nThank you for your review and constructive comments. During the rebuttal period, we have made every effort to address your concerns. The detailed responses are below:\n\n> Q1: Why human shows mild improvement compared with ControlNet?\n> \n\nWhile our method shows only mild improvement on the *human* class compared to ControlNet, it brings **substantial gains across other categories**, especially animals, as shown in Table 3 of the main paper. This highlights the strength of our approach in handling diverse object classes.\n\nOur core objective is to solve the task of **keypoint-conditioned multi-class image generation**, where *human* is just one of many categories. In contrast, ControlNet is specifically designed for **human-centric** keypoint-conditioned generation, and thus naturally excels in that narrow domain.\n\nThis difference is analogous to a **specialist vs. generalist** comparison. A specialist may outperform others on a single task, but a well-designed generalist can achieve strong performance across a broader spectrum. UNIMC is purposefully built as a generalist model that scales across multiple categories, not just humans.\n\n> Q2: Explanation of dropout ratio choices.\n> \n\nIn our initial experiments, we applied a **15% dropout ratio** to both bounding boxes and keypoints. However, we observed that the model quickly overfit to the bounding box information—likely because **coarse object localization is much easier to learn than fine-grained keypoint patterns**. As a result, the model tended to ignore the keypoint inputs, and the pose accuracy plateaued below 20%, regardless of the number of training iterations.\n\nTo address this issue, we increased the dropout ratio for bounding boxes, forcing the model to rely more on the keypoints. The table below shows how different dropout ratios for bounding boxes affect pose accuracy:\n\n| Bounding box dropout ratio | Human AP**⬆️** | Animal AP**⬆️** |\n| --- | --- | --- |\n| 0% | 14.70 | 13.92 |\n| 15% | 17.85 | 18.21 |\n| 50% | 30.01 | 28.38 |\n| 100% | 26.42 | 26.09 |\n\nThese results suggest that a **moderate dropout (around 50%)** strikes a good balance, encouraging the model to make better use of keypoint information without completely discarding spatial cues from bounding boxes.", "claims_and_evidence": "The dataset is for generation purpose, and thus the average aesthetic score is higher than other keypoint datasets, such as COCO.\n\nThe proposed model is tested and ablated, showing better performance than baselines.", "methods_and_evaluation_criteria": "1. This paper directly use keypoint and bounding box coordinates as controlling signal without converting them into spatial representations, such as heatmaps, but into Fourier signals, and it works better than previous work using spatial representations. It slightly counters my intuition, but also makes sense because processing in the implicit way may benefit interation better. This setting could inspire future work on image/video mulit-object interaction generation.\n2. The detailed dataset processing and comparison of different methods for different annotations can benefit the community.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "1. From Table 3, the class and pose accuracy is significantly improved for animals, but those metrics for human have only  mild improvement compared with ControlNet. What might be the reason? Can it be visualized by some attention maps or feature maps?\n2. Some of the magic number should be explained. For example, why use 50% dropout for bbounding box while only 15% for keypoints? I understand it must be a performance choice, but is there any insight why? How does the dropout ratio affect the quantitative and qualitative results?", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Overall, the reviewers acknowledge the superiority of the proposed method and its effectiveness in addressing the target problem. While initial concerns were raised regarding the justification for the new dataset, the lack of model efficiency analysis, and limited comparisons with existing methods, these issues were largely addressed in the rebuttal. The reviewers were generally positive in their final recommendations. Although some concerns remain—particularly around comparison with related methods—they do not significantly impact the perceived contribution of this work.
Decision: Accept (poster)