Action Imitation in Common Action Space for Customized Action Image Synthesis
Authors: Wang Lin
Jingyuan Chen
Jiaxin Shi
Zirun Guo
Yichen Zhu
Zehan Wang
Tao Jin
Zhou Zhao
Fei Wu
Shuicheng YAN
Hanwang Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: customized action; diffusion model; text to image generation
Primary area: diffusion_based_models
TL;DR: We customize a text-guided diffusion model for generating customized action images in a variety of different contexts.

Abstract: We propose a novel method, \textbf{TwinAct}, to tackle the challenge of decoupling actions and actors in order to customize the text-guided diffusion models (TGDMs) for few-shot action image generation. TwinAct addresses the limitations of existing methods that struggle to decouple actions from other semantics (e.g., the actor's appearance) due to the lack of an effective inductive bias with few exemplar images. Our approach introduces a common action space, which is a textual embedding space focused solely on actions, enabling precise customization without actor-related details. Specifically, TwinAct involves three key steps: 1) Building common action space based on a set of representative action phrases; 2) Imitating the customized action within the action space; and 3) Generating highly adaptable customized action images in diverse contexts with action similarity loss. To comprehensively evaluate TwinAct, we construct a novel benchmark, which provides sample images with various forms of actions. Extensive experiments demonstrate TwinAct's superiority in generating accurate, context-independent customized actions while maintaining the identity consistency of different subjects, including animals, humans, and even customized actors.

Action Imitation in Common Action Space for Customized Action Image Synthesis
Wang Lin Jingyuan Chen Jiaxin Shi Zirun Guo Yichen Zhu', Zehan Wang',
Tao Jin' Zhou Zhao', Fei Wu', Shuicheng YANo, Hanwang Zhang'
Zhejiang University! Xmax.A12 Skywork AI Singapore", Nanyang Technological University {linwanglw, jingyuanchen, gzr, yc_zhu, wangzehan01, jint_zju, zhaozhou, wufei) @ani)(/z @zju.edu.cn shi.j112gmaill com, shuich waapheenggyn@knnmmaa yan@kunlun----ae com, hanwangzhang@ntu. edu.s..ttttteeuu.gggg........e..
Abstract
We propose novel method, TwinAct, tackle the challenge of decoupling tions and actors in order to customize the text-guided diffusion models (TGDMs) for few-shot action image generation. TwinAct addresses the limitations of ex- isting methods that struggle to decouple actions from other semantics (e.g., the actor's appearance) due to the lack of effective inductive bias with few ex- emplar images. Our approach introduces a common action space, which is a textual embedding space focused solely on actions, enabling precise customization without actor-related details. Specifically, TwinAct involves three key steps: 1) Building common action space based on a of representative action phrases; Imitating the customized action within the action space; and 3) Generating highly adaptable customized action images in diverse contexts with action similarity loss. To comprehensively evaluate TwinAct, we construct novel benchmark, which provides sample images with various forms of actions. Extensive experiments demonstrate TwinAct's superiority in generating accurate, context-independent customized actions while maintaining the identity consistency of different sub- jects, including animals, humans, and even customized actors. Project page: https://twiincttoffficalgggguuld wwt.:fiiiiiaiiituuuiboo////iaiii
Introduction
We are interested in customizing Text-Guided Diffusion Models (TGDMs) (e.g., Stable Diffusion 21
to generate customized actions specified by few user-provided images, For example, given a few images of Black Widow's signature three-point landing action, we can represent this action unique token V* and generate imaginary creation by the prompt like "Leonardo V* (the third row in Figure [T.. One might question why we do not condition TGDMs on precise textual descriptions [2] or sketch images [31] to generate the desired action. The reason is that actions are often unutterable, and even with finely detailed descriptions (SD column in Figure T or skeleton images (ControlNet column in Figure I.. TGDMs cannot accurately follow these instructions.
customize actions? Unfortunately, the answer is still no. This is because these methods fine-tune the textual embeddings of V* in prompts like "Black Widow V*" by reconstructing the corresponding image. But, except for actions, there are many other semantics in the exemplar images that are coupled in V* like the characteristics of the specific actors. Thus, the few-shot action images in those methods
Further, can
existing few-shot customized TGDMs 4
10
14
27
Equal Contribution.
Corresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Action
Cantrollia
DreanBecth
Roveraon
Custem
ABI
TatAct(Aur)
A A
Tim perter
The gor Bo.
Figure I: Qualitative comparisons of TwinAct with other methods. TwinAct preserves the identity consistency of actors while allowing customized actions to be accurately generalized across different actors by effectively decoupling actions and actors.
lack an effective inductive bias to achieve decoupled action tokens V from actors, which results in action-irrelevant information like Black Widow's red curly hair being confounded in the generated images as shown in Figure Although some methods mitigate the effects of missing inductive bias through contrastive learning on textual embeddings or visual images, they still struggle to exclude all action-irrelevant semantics. For example,
g uses prepositions as positive samples and randomly selects negative samples from limited set of POS words (i.e. nouns, adjectives). However, this approach can not guarantee the exclusion of all vetiionirrelevant information from exemplar images for V*, resulting in the inclusion of irrelevant semantics like manicure and red curl hair (the first and third rows in Figure MT confounding the customized tokens. On the other hand, [7] and [14] utilize contrastive training data involving the same subject with and without specific customized concepts (e.g, melted and closed eyes, etc.) to decouple the subject and customized concepts. However, the creation of extensive contextual training data is costly, particularly when dealing with customized actions that involve multiple subjects.
By observing the real-world process of director instructing an actor to perform a specific action, such as the signature action of the character Black Widow which involves set of action phrases including "left hand on the ground", "right hand raised behind the back". "left leg on the ground", and "right leg straight out", we find that these phrases concentrate solely actions without providing any specific details about the actor involved. Moreover, these phrases can be modified or combined to create new actions, as illustrated in Figure 2 Inspired by these observations, we propose method called TwinAct, which aims to decouple the action from the actor by constructing an effective Common Action Space. This space is exclusively for actions, eliminating any confusion with actor-specific characteristics and enabling the generation of diverse and creative actions that can be applied across various actors and scenarios.
Our approach involves three steps. 1) We collect 832 action phrases that cover a range of movements from finger gestures to full body poses, and extract the corresponding textual embeddings using

---

the tokenizer of CLIP [8] These textual embeddings are used to build a common action space (Section[3.2) through Principal Component Analysis (PCA (): 2) We employ Multilayer Perceptron (MI.P) to modulate the PCA coefficients to imitate the customized actions within the action space (Section 3.3p; and 3) We introduce an action similarity loss to provide semantic similarity between the exemplar images and generated images, resulting in synthesizing highly adaptable customized action images in diverse contexts (Section 3.3). Extensive experiments demonstrate the superiority of our method over the existing method. Finally, the main contributions of this paper are summarized as:
We propose TwinAct to generate customized action images, which can synthesize novel renditions of user-specific action in different contexts including animals, humans, and even customized actors. We introduce a common action space that focuses solely on the action itself, without any actor- specific details. By combining action bases in this space for customized action imitation, we are able to effectively decouple action from actor, allowing for greater flexibility in customization. Through extensive experiments, we demonstrate that TwinAct outperforms previous methods in maintaining the fidelity of customized actions while also preserving the consistency of actor identities.
2
Related Work
2.1 Image Generation and Customization
The text-guided diffusion models (TGDMs) [5 20 21 23
have emerged a powerful paradigm in the domain of image generation, enabling the creation of image variations that are aligned with specific textual descriptions. Recently, the customized TGDMs 4 450 10 13 22 aim encoder the customized concept within the textual embedding, often represented by a unique token and decode the token into pixels by the cross- tross-attention in the U-net decoder (24). Textual Inversion 4 optimizes textual embedding and synthesizes personalized images by integrating the concept token with the target prompt. DreamBooth [22] extends this concept by proposing framework that optimizes all parameters of the denoising U-Net architecture, based on a specific token and the class category of the subject. Several other works [0] have focused on optimizing subsets of weights introducing additional adapters to achieve more efficient optimization and better conditioning of the generated images. For instance. Custom Diffusion [10) fine-tunes only the cross- cooss-attnttion layers within the U-Net, while P+ [27] expands the lextual-conditioning space with per-layer tokens to allow for greater disentanglement and control over the generation process. Despite these methods achieving commendable results in customized actors, they struggle with generating customized action images, as shown in Figure due the lack of an effective inductive bias with few exemplar images.
2.2 Customized Action Image generation
In contrast to the rapid progress in customized actors, customized action has received less attention in the community. Since the complex details such as specific angles, body positions, and motion trajectories, describing specific action is more challenging than describing specifie actor. Even with finely detailed descriptions (see Appendix II TGDMs still cannot accurately follow these textual descriptions. A straightforward approach to generate specific action image is ControlNet [31] which is conditional on a given skeleton image to generate images. However, they only provide a rough approximation of action consistency (see Appendix A.3). Particularly, when it comes to capturing fine details such as fingers, skeleton images may lack the necessary detail to accurately reproduce these actions, leading to noticeable visual defects in the generated images. In addition, when conditioned on detailed images (see Appendix AA3)). they suffer from limited diversity and ficxibility. Recent advancements for customized TGDMs preliminary explore action-based customization. Reversion [8] proposes relation-steering -tterinngntennnggg contrastive learning that aims to guide the relation prompt towards relation- dense regions within the text embedding space, thereby disentangling the learned relation from other concepts like appearances. Lego [1] and ADI 7 construct contexts involving the same subject with and without a specific concept, to decouple the subject and specific concepts. However, these methods by employing contrastive learning on textual embeddings or visual images can not effectively decouple actions from actors. This is because they cannot exclude all irrelevant concepts with limited number of negative samples.
3

---

Filtering Process
Tokenizing & Embedding
Adjustment
TGUM--now Actions pushi-up
someane XXX
Arms crossed
Action Embeddlings
Leaning on tiptoe
Stable Diffasion
TGDMFunkoow Actions
Crab Walk
Reverse Plank
Tensing muscles
PCA
Action Phrases Generated GPT-4
Common Action Space
Head:
Foot:
Looking down
Crossing ankles
Action Composition
Turning head left
Leaning on tiptoe
Tilting head sideways
Squathing down
Hond:
Body:
Arms crossed
Flipping backward
Hands raised parallel
Lying back
Palm facing up
Bouncing excitedly up
Figure
The construction of Common Action Space.
Methods
Imagine the following scenario: When the character Black Widow performs her signature three-point landing action, the director may give the following instruction "Bend knees and lower body down into a squatting position; Place right hand forward, and right arm should be fully extended". We can see that when people involve the composition of various body movements and the adjustment angles and amplitudes as shown in the right part of Figure 2 Inspired by the way in which humans perform specific actions, we propose TwinAct which applies similar principles to action customization. Specifically, we first collect set of basic actions and encode them using a pre-trained text encoder to build common action space represented by a set action bases. Subsequently, we imitate the customized action by decomposing into combination of action bases with corresponding weights. Finally, utilize both the reconstruction loss and action similarity loss to fine-tune the customized token embedding in the text encoder and the LoRA 6 layer in the text-to-image decoder generate high-fidelity images of the customized action.
asked to perform specific action, it is often accompanied by instructions that
3.1 Preliminaries
Our study is based on the text-guided diffusion models (TGDMs) which consist textual encoder and text-to-image decoder D. Given few sample action images and textual prompt with customized token V*, such as "a photo of V*", we aim to fine-tune the embedding of customized token V* & and the LoRA layer in realize action customization. The textual encoder first tokenizes the prompt into set of token embeddings h, which are then used to generate a textual condition Ta(c) with the text transformer in E. Subsequently, the textual condition Te(c) is utilized by the conditional denoising diffusion model ea in D to generate images. The commonly used training objective is:
Cree E,,t,xo. [|le Ex [c)
where is the timestep, Xy is the original image and x; is a noisy image constructed by adding noise (0, to Xo- After training, any textual condition Te(c) with V can generate customized action in a new context defined by prompt
3.2 Step-1: Building Common Action Space
The initial step involves identifying the set of action bases that defines the common action space within the ext-to-image diffusion model.
First, we task GPT-4 [ with generating a range of common action phrases based on different body parts, including head, fingers, hands, and full body (see Appendix B.2). 2). Then, we carefully sift through the generated action phrases eliminate duplicates. Subsequently, we build manual filter based on the pre-trained text-to-image diffusion model by constructing prompts and synthetic images

---

A photo of
Tokenizer
Text Trunsformers
Teal-te-lmage Decoder
Action Enceder
Denoise
Latent
MLP
Norm
Ayyae
Common
Encoder
Action
LoRA
Figure 3: The overview of the proposed TwinAct. We optimize the coefficients of the action bases to avoid encoding the action- iirellianeeeeaattt features. After training, we combine the learned coefficients and shared action base to generate images with the customized action.
of each action phrase to remove those action phrases that are unknown to the text-to-image diffusion model, e.g., "Reverse Plank" shown in Figure 2
After the filtering process, we identify a total of m = 832 action phrases. Each phrase ag where i E [l,...,,, is tokenized and encoded into an action embedding group A; [ dg. It is important to note that the length & of each action embedding group A, varies, as each action may consist of different number of words. Through empirical observation, it was found that action phrases with three words or fewer are the most common, prompting the designation of each A as containing three embeddings (i.e.,
for all m actions). For simplicity, we denote the three embeddings of all A, as Q1, Q2, and Q3 respectively, where Qx = [a,....,a[]]] a%).
Inspiring by prior research [2 employing PCA to transform high-dimensional facial data into lower-dimensional space, we calculate the PCA transformation matrix B₃ for each set VQs using the function PCA(Q:: p). This transformation plays crucial role in reducing redundant information within the data and enhancing the efficiency of model optimization. Specifically, PCA (Q p) reduce dimensionality of Qk E mdd and generate principal components as Bg = [b ., b%] E RPxp, which captures the essential features of data in more compact form. Our experimental findings indicate that optimal results are achieved when = 512 (see Section 4.3) Finally, we obtain a set of action bases as B1, B2. Bz], which define action space that does not contain any action- information thus providing a strong inductive bias to achieve decoupled action token embeddings.
3.3
Step-2: Imitating Action via Action Bases
This step involves searching for the optimal parameters combine the action bases for imitating customized actions.
Our experiments (see Figure have shown that existing methods, which aim to identify optimal personalized token embeddings within the vast text space of pre-trained text encoders, struggle with the inclusion of non-action information. And even with contrastive learning, they are still unable to exclude all irrelevant information. Fortunately, action bases offer an alternative approach for optimizing the customized token embedding. By combining action bases [B1, B2, Bs] with corresponding coefficients [W1, W2, Wa]. where W4 = fwu...... wf), new actions can be imitated as shown in Figure 2 To achieve this, we first utilize Multi-Layered Perceptron (MLP) to combine the action bases. While backpropagation is a common method for determining optimal coefficients, experiments indicate its limited effectiveness due the scarcity of user-proviin provided images (see Section (4.).. Therefore, we use CLIP as an action encoder to extract semantic features as a prior. Specifically, given action sample image x, first extract the action features Xp with action encoder sb (x), then MLP is used to map X, into the modulating coefficients [W1, W2, Wa] as:
W MLP
(2)
Finally, for customized action, we can combine the action bases to imitate the action as:
V j01, θ2, 02), Ox
w b2
(3)
5

---

where is the embedding of the customized tokens V In this way, V is linear combination of the action bases. Consequently, the vector interpolated in the action space would not contain any action-irrelevant features, allowing us to achieve the objective of decoupling the action and the actor.
3.4 Step-3: Generating Customized Action
The final step involves optimizing the embeddings of the customized token V*, which is derived from the action base through linear combinations and synthesis customized action image.
Training. Initially, few aser-provided images and corresponding textual prompts containing V are used to apply the pixel reconstruction- based loss Crees similar to previous customized TGDMs (in [. However, the traditional Lree in existing TGDMs aims to faithfully reconstruct the sample image appearance, focusing on low-level pixel features. This narrow focus limits the model's 's ability to recognize and invert high-level action features in the images, shown in Figure (b).
To address this limitation, we introduce an action similarity loss to enhance the model's grasp of high-level action semantics. Beginning with input image and corresponding prompt randomly initialize a latent variable Xy. This variable progressively denoised until reaches a randomly selected timestep t, at which point the denoised image x is predicted directly from X The goal is to evaluate the action similarity between these two images. Specifically, by leveraging action encoder & (-), the encoded action features for both the reference action Xp and the generated action xpr are extracted. We then calculate the cosine similarity between these action features to measure action consistency during the generation process. This similarity is formally defined as:
Lact Be-p(e) Bess((ate) [1 cos sops_simmpppp xpprd]
By integrating the action similarity loss, the model's focus shifts from detailed pixel information to high-level action semantics. The final training loss for TwinAct can be expressed as:
L = Cree + Lact
(5)
Testing, After training, the three groups of coefficients [W1, W2. Ws] and the LoRA parameters in the text-to-iman image decoder are saved. Users can generate images depicting customized actions in different contexts by providing prompts such as "Leonardo V*"
Experiment
4.1 Experiment Setup
Dataset. We tackle the challenge of decoupling specific actions from the actors in user-provided images. Since there are no publicly available customized action datasets, we introduce novel benchmark, consisting of 12 actions involving multiple body parts, such as fingers, arms, legs, and full-body motions. Each action contains approximately 10 sample images. In addition to actions performed by single actor, we incorporate more complex actions that involve multiple actors.
Metrics. To comprehensively validate the efficacy of TwinAct, we incorporate objective metrics to assess the quality of the generated images. We calculate the consistency between the action in the sample image and the generated image through the CLIP score, which is denoted as "S Action" Additionally, ensuring identity consistency a crucial aspect our task, so we evaluate the actor's identity similarity using pre-trained face recognition encoder [3], which is denoted as "S "Attrr" ". In addition, we also conduct user studies including "U Actson" " and "U Actor", representing user preferences for action fidelity and actor identity consistency in the generated images.
4.2 Comparison with State-of-the-art Methods
Qualitative Comparison. We illustrate the advancements of our method compared to previous approaches in handling various forms of actions, including hand, body, single-actor, and multi-actor actions, as shown in Figure [ Initially, we evaluate two methods for generating customized action images without fine-tuning, generated from detailed textual prompts (SD) frequently exhibit misalignment with the actions the sample images. On the other hand, Controlled Generative Models (ControlNet), which depend on
shown in the column of SD and ControlNet in Figure []] Images

---

Sample Action
A Leonardo l
Barack Obama V*
Batman V
A Polar Bear V
Corgi V
y
Sample Action
Leonardo 1
Barack Obama Y
A Batman V
Polar Bear y
Corgi V
Sample Action
Leonardo V*
Barack Obama y
A Batman V
Polar Bear V
Corgi V
Sample Action
Batman and Superman Y
Barack Obama Leonardo
Leonardo and polar bear
Two gorillas y
Two polar bear V
Sample Action
Batman and Spiderman V*
Barack Obama Leonardo
Borbie doll and Batman
A man and gorillas V A cat and A dog V
V
Sample Action
Batman and Superman
Barack Obama Leonardo
Barack Obama and Batman bear Y*
A panda and lion 5
Two gorillas V*
V
Sample Action
Barack Obama and Leonardo y*
Barack Obama and Michael J Jackson y*
gorilles Barbie dail y*
Leonardo polar bear Y
Superman y*
Figure 4: The results of eustomized action generation with TwinAct. TwinAct generates images of different actors performing customized actions such as celebrities and animals, and maintains the consistency of the action and identity of the subject.

---

Somple Action
Sample
Vitruce
Sample Action
Sumps
Vinselaa
Figure 5: The results of customized actors performing customized actions generated by TwinAct.
Methods
Text Inversion
DreamBooth
Reversion
Custom Diffusion
P+
ADI
Ours
S Action
9.12
12.23
18.73.
26.83
33.95
45.32
69.47
Objective
S Actor
33.85
38.53
45.58
41.58
46.12
48.76
73.34
User
Aetson
0.22
0.19
0.89
1.16
1.30
2.25
3,86
Study
Unstor
0.53
0.70
1.05
1.31
1.23
2.08
4.12
Table 1: Quantitative comparisons. We evaluate TwinAct based on both objective metrics and user study. The results indicate that TwinAct surpasses all baseline methods.
skeleton images, tend only imitate the basic structures of the sample images and lack a profound understanding of the contextual dependencies required for action customization. This deficiency leads to significant body deformations and flaws, particularly when animals are involved in the action. For example, in the fifth and sixth rows of Figure] I both the human and the gorilla mistakenly raise their hands instead of their feet.
Subsequently, we conduct comparisons with statc-of-the the-----tttt-ttt baselines such as Textual Inversion 4 DreamBooth 222. Reversion (8. Custom Diffusion [10] P+ [27). and ADI (7). Textual Inversion and DrcamBooth tend to capture minimal action-related information, primarily because of their limited tunable parameters. On the other hand, Custom Diffusion, which fine-tunes cross-attention in U-net, and Pt, which utilizes per-layer token embeddings, manage to retain more information from the sample images. However, they uncontrollably invert action-irrelevan details in the sample images and bring these details into the generated image, such as the human hand and the manicure in the first and second rows shown in Figure [ In addition, Reversion and ADI try to exclude action-irrelevant information from the images by contrastive learning but face challenges in comparing all negative samples. Therefore, the images they generate still contain some confounded details, such as the red curly hair of the Black Widow in the third row and the yoga pants in the sixth row. In contrast, TwinAct shows the best customized action generation results. Through the acquisition of a customized action token embedding, which is derived from linear combination of action bases within the common action space, TwinAct accurately capture the action information the sample images while excluding irrelevant details with good generalization.
Quantitative Comparison. In addition the visual quality, we also present a numerical comparison between TwinAct and the baselines in Table [ As can be seen, our approach outperforms in terms of both action and actor similarity, demonstrating its effectiveness imitating customized actions and filtering out irrelevant information that could potentially confuse the portrayal of the new actor. Furthermore, the performance across four representative types of action customization With-Animal) is illustrated in Figure 6 Our
tasks (i.e., Single-Actor, Two-Actor, method consistently shows superior performance and the least variability across all types. Notably, the method labeled as P+ experiences the most significant performance drop when involving animals, and ADI exhibits a considerable performance drop when handling actions involving two actors.
Additionally, following the methodology outlined in 330, we conduct a quantitative comparison involving human evaluators to further validate the effectiveness of TwinAct. We invite 100 users to rate each pair of actor-action images generated by TwinAct and baselines on scale ranging from 1 (worst) to (best), focusing on both action and actor consistency. The collective feedback, as presented in Table [ clearly indicates strong preference for TwinAct among the users.
8

---

Methods
Actor
Artion
w/o action space
58.86
52.18
w/ 50% action phrases
68.39
61.86
w/o action encoder
67.27
59.06
w/o Lact
70.23
63.58
64
72.47
54.27
256
73.12
61.66
p 768
71.62
67.63
Ours
73.34
69.47
Figure 6: Numerical analysis in of the actor and action similarity on four prompt types.
Table 2: Ablation studies.
(ssumplo imago)
('(samplo imago)
V'Samplc image)
A balman and spiderman V'*
A Burnck Obama ! 50% action pimoes
A Weish Corgi V
Wio action similarit loss
256
p 768
front of Difiell tower
(a) Action Space
(b) Action Encoder and Lact
(c) Number of Principal Components
Figure 7: Visualization of ablation studies. The results of TwinAct are highlighted in green box.
4.3
Ablation Study
Impact of Action Space. We assess the impact of the number of action phrases for building the common action space, as shown in Figure a). In extreme cases, if there exists no action space and we directly learn the customized token embedding from the action encoder (w/o action space), the model can not generate customized actions and is overfitted to the input image details, such as the actor's clothes and the background. When utilizing smaller number of action phrases (w/ 50% action phrases), the generated quality is not good as TwinAct.
Impact of Action Encoder and Action Similarity Loss. We assess the impact of the action encoder and action similarity loss in Figure (])) and Table The experimental results reveal that employing an action encoder can enhance discriminative capabilities regarding actions and yield satisfactory outcomes. In contrast, without an action encoder, optimizing the coefficients W, of the action bases Bg through back- preck-ppppgttionn (w/o action encoder) is ineffective. Furthermore, similarity loss can guide model towards more accurately imitating the customized actions by drifting the model's focus from low-level pixel features action-related high-level semantic features.
can find action
Impact of the Number of Principal Components. We explore the effect of the number of principal components p model performance. With 768 dimensions in the CLIP text embeddings. conduct experiments with different values of p from {64, 256, 512, 768). The experimental results, outlined in Table 2 indicate that the optimal results are achieved when showcase the variations in the generated images for different p values in Figure c).
512, Additionally,
4.4 Combinations of Customized Actors and Customized Actions
Our method demonstrates strong adaptability to integrate with existing subject-driven customized generation methods[[I], shown in Figure It effectively adapts to customized actors, which were unknown by TGDMs, and generates visually engaging images showcasing these actors performing customized actions. In contrast, as depicted in Figure TO in Appendix, existing methods struggle to accurately capture action-related features, resulting unsatisfactory generation of customized actions and lack of identity consistency for the customized actors.

---

5 Conclusion
In this paper, we introduce a novel approach, TwinAct, for customized action image generation, which aims to preserve the fidelity of the action and the consistency of the actor's identity. TwinAct consists of common action space and can create new actions by adjusting or combining the action bases in the space. By imitating action via action bases, TwinAct effectively decouples action from the actor. Extensive experiments demonstrate TwinAct's superiority generating accurate, context- independent customized actions across various subjects, including humans, animals, and customized actors. Furthermore, TwinAct's ability to preserve the high fidelity and identity consistency in generated images highlights its robustness and adaptability. The potential applications of TwinAct in areas such as animation, gaming, and visual effects are vast, offering new opportunities for creative and personalized content generation.
6 Acknowledgements
This work was supported by the National Natural Science Foundation of China 23..6237733. No.62037001) and the Key Research and Development Program of Zhejiang Province (No. 2023C03192).
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 23XI::23033.8774,, 2023.
[2] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 157-164. 2023.
[3] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4690-4699 2019.
[4] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint ar arXiv:2208. 016188 2022.
[5] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen- Or. Encoder-base based domain tuning for fast personalization of text-to-image models. ACM Transactions on Graphics (TOG), 42(4): 1-33, 2023.
[6] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 2rXiv:2106.09685 2021.
[7] Siteng Huang, Biao Gong, Yutong Feng, Xi Chen, Yuqian Fu, Yu Liu, and Donglin Wang. Learning disentangled identifiers for action-customized text-to-image generation, arXiv preprint arXiv:231 1.1111.18411,1 2023.
[8] Ziqi Huang. Tianxing Wu, Yuming Jiang, Kelvin CK Chan, and Ziwei Liu. Reversion: Diffusion- based relation inversion from images. arXiv preprint arXiv:2303. 13495, 2023.
[9] LIII KPFRS. On lines and planes of closest fit to systems of points in space. In Proceedings of the 17th ACM SIGACT-SIGMOD -SIGRTT symposium on Principles of database systems (SIGMOD), page 19, 1901
[10] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi- concept customization of texl-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1931-1944 2023.
[11] Wang Lin, Jingyuan Chen, Jiaxin Shi, Yichen Zhu, Chen Liang, Junzhong Miao, Tao Jin, Zhou Zhao, Fei Wu, Shuicheng Yan, et al. Non-confusing generation of customized concepts in diffusion models. arXiv preprint arXiv:2405.06914 2024.
10

---

[12] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03983, 2016.
[13] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprini arXiv: 0n::2108..0007 2021.
[14] Saman Motamed, Danda Pani Paudel, and Luc Van Gool. Lego: Learning to disentangle and invert concepts beyond object appearance text-to-image diffusion models. arXiv preprint arXiv:2311. 13833, 2023.
[15] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Iya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprin! 22Xiv:2112.107411 2021.
[16] Anton Pelykh, Ozge Mercanoglu Sincan, and Richard Bowden. Giving a hand diffusion models: a two-stage approach to improving conditional human image generation. arXiv preprint arXiv:2403. 10731, 2024.
[17] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint MxXiv:337.001522 2023.
[18] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8763 PMLR, 2021.
[19] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.0611 06125, 1(2):3 2022.
[20] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pages
Pmlr, 2021.
[21] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695 2022.
[22] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22500-225115 2023.
[23] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36494, 2022.
[24] Nahian Siddique, Sidike Paheding, Colin P Elkin, and Vijay Devabhaktuni. U-net and its variants for medical image segmentation: A review of theory and applications. leee Access, 9:82031-82057, 2021.
[25] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. 2020.
[26] Thanh Van Le, Hao Phung, Thuan Hoang Nguyen, Quan Dao, Ngoc N Tran, and Anh Tran. Anti-dreambooth: Protecting users from personalized text-to-image synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2116-2127, 2023.
(27] Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman, p+: Extended textual conditioning in rext-to-image generation. arXiv preprint arXiv:2303.09552, 2023,
11

---

[28] Chengrui Wang, Pengfei Liu, Min Zhou, Ming Zeng, Xubin Li, Tiezheng Ge, et al. Rhands: Refining malformed hands for generated images with decoupled structure and style guidance. arXiv preprini arXiv:2404. 13984, 2024.
[29] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
[30] Ge Yuan, Xiaodong Cun, Yong Zhang, Maomao Li, Chenyang Qi, Xintao Wang, Ying Shan, and Huicheng Zheng. Inserting anybody in diffusion models via celeb basis. arXiv preprint arXiv:2306.00926, 2023.
[31] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-imaae diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer 2023.
Vision, pages
12

---

In this appendix, we first provide an in-depth analysis of Stable Diffusionin in Section A.1 and
ControlNet in Section A.3] and additional results of TwinAct in Section A.3 and A.44 demonstrating
its effectiveness in generating customized action images. Then we introduce the evaluation setting in Section [B.1] the details the action space in Section[..] and the implementation details in Section C
Finally, we discuss the limitations in Section and societal implications in Section F. for TwinAct.
A
More Qualitative Results
A.1
Comparison with Stable Diffusion
We first utilize GPT-4 generate detailed description of the actions in the sample image provided by the user, as shown in Figure [2] We can find that on the one hand, although GPT-4's description highly sufficiently, it still inevitably loses many details of the action. On the other hand, existing text-guided image diffusion generation models have difficulty in understanding such detailed textual descriptions, like "heart" in the fourth row in Figure 12 Both reasons result in the generation of action images that are far from the sample image. In contrast, our approach accurately captures reltted features, and the user only needs provide textual prompts like "a polar bear V*". and then customized action images can be generated in the desired contexts.
A.2 Comparison with ControlNet
In addition to controlling the generation of models through textual conditioning, current work such as ControlNet supports extracting spatial constraints from reference images as conditions to generate images. We tried two reference images, skeleton images, and sketches, conditions to control the generative model to generate customized action images. The results are shown in Figure 9 We can find that the skeleton image-bases approach loses a of details of the action, such as fingers, and it is difficult to imitate the customized action. The other sketch-based approach provides more details, but this limits the diversity of the model generation results, e-g., difficult to generate images of animals performing customized actions since the sketches contain human bodies. Also, the sketches contain some additional action-irrelevant information such as the background, the actor's s hairstyle, and so More importantly, the controlled generation model lacks the understanding of the contextual dependency of the action, as shown the third row in Figure / It is difficult for the generation model to understand the relationship between the hands and the feet in the customized action, which leads to serious body distortions and visual deficiencies, and this becomes even more obvious when animals are involved.
A.3 Comparison with Baseline on Customized Actor
In this section, we compare the results of existing customized TGDMs and TwinAct in generating images containing customized actors and customized actions. As shown in Figure IO] our approach significantly outperforms the previous methods in maintaining the fidelity of the action and consistency of the character's identity. Textual Inversion, Dreambooth, and Revrsion struggle to imitate the customized action. As for, Custom Diffusion, P+, and ADI, although they can roughly maintain the consistency of the actions specified in the sample image, fail maintain the identity information of the customized actors. This is because their customized action tokens are confounded with information irrelevant the actions the sample image, e.g., Black Widow's red curly hair in the first row, and the yoga pants in the second row.
A.4 Additional Results
In this section, show customized action images generated by TwinAct, shown in Figure covering the rest of the 5 actions within our benchmark.
B Experiment Design
B.1 Evaluation Setting
Following 77 provide 25 subjects for evaluation as follows:
13

---

Semple
Inee-Grained Testual Destriptions
butth
fingers sightll been sprreeed
sperl,
(o1) ground truuching the
jedy's lega focused ditermined, givinng ready
[ol] and sem ing floor tocing each ether their legs touching
age straight mn holdinng
fest
suppont
[o4] and (52) Each one has are anm around the other's shoulder, and thair free amns are ralted and surved above their heads form the shape o heart
Figure 8: Comparing the results generated by TwinAct and fine-grained textual descriptions. The results show that customized actions are difficult to describe and that existing text-to-image generation models do not accurately follow textual instructions even when fine- ane-grained textual descriptions are provided.
Sample image
Skerch image
ConrrelNer
image
Figure 9: Comparison of results generated by TwinAct and sketch-based generation models. The results show that it is difficult for the user to provide a suitable sketch for generating customized action images. The results generated with skeleton images show it is difficult to capture the details of the action such as fingers, while the results generated with line images are limited in generalization, especially when it involves animals.
generic human: "A boy", "A girl", man", "A woman", "An old man" well-known personalities: "Barack Obama", "Michael Jackson", "David Beckham", "Leonardo DiCaprio", "Messi", "Spiderman", "Batman" animals: "A dog". "A cat", "A lion", "A tiger", "A bear", "A polar bear", "A fox", "A cheetah", "A monkey", "A gorilla", "A panda"
customized actors: "Cruze", "Imelda"
where the inclusion of diverse and previously unseen actors, as well as animals and customized characters, requires models to retain pre-trained knowledge while also generating accurate and undistorted representations these actors.
B.2 Action phrases for Action Sapce
As stated in the manuscript, we devise set of action phrases to build action space. Specifically, we collect comprehensive list about 1,200 action phrases with the help GPT-4 [ After obtaining these action phrases, in order to filter out those action phrases that are not recognized by the TGDMs.
14

---

Sample Actor
Sample Action
Textuo Inversion
DrecnBooth
ReVersion
Custom Diffusion
AbI
Our
A photo of yruuce y
Cruze
photo limeida V'andy
Vimelda
A photo of ynan front of Eifiel
Figure Comparing the results of generating customized actors perform customized actions generated by TwinAct and other methods. TwinAct can better maintain the identity consistency of the subject in the generated image, and the fidelity the action.
we synthesize the image result by providing 2 types of prompts, i.e. "A photo of (Action]", "A photo of someone /Action]", to the Stable Diffusion XL [17]. where (Action] represents an action phrases (e.g. "raise hand"). Those action phrases that do not generate the correct action, such as "reverse plank", or "crab walk", are filtered out. We ended up with 832 valid common action phrases, which were then further tokenized and encoded into the action space.
C More Implementation Deatils
C.1 Implementation Details about TwinAct
Implementation Details. For our method, of 2e-4. We use CLIP as a preprocessor to estimate the action of the given reference image. Unless otherwise specified, Stable Diffusion XI. is selected the default pre-trained model, and images are generated at a resolution of 1024x1024. All experiments are conducted on A-100 GPUs. We fine-tuning text embedding along with the LoRA layer. We integrate the LoRA layer into the linear layer within all attention modules of the U-net, utilizing a rank of = 8.
the AdamW[[2 optimizer with learning rate
Sample Details. All experiments and evaluations make use of the DDPM 25 with 50 sampling steps with a scale of 7.5 for all methods. To ensure consistency and filter out undesired variations in diffusion models, we follow the approach outlined in [11] by employing the same negative prompt for both our method and the comparison methods during sampling. The negative prompt used is "long body, low-res, bad anatomy, bad hands, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality."
Running Times. the LoRA layer in the text-to-image decoder typically requires approximately 15 minutes using an Nvidia-A 100 GPU.
The process of tuning customized action token embedding in the text encoder and
C.2 Implementation TwinAct with Customized Actor
Thanks to the rapid development of subject-driven customized text-guided diffusion models for image generation, they achieved an exciting proceeding in generating high-fidelity customized actor images. We combine TwinAct with the existing state-of-the -f-t-o--arrrrr aar subject- driven customized TGDMs [1] to generate images of customized actors performing customized actions. To achieve this, we only need to apply the action base alone to the embedding of the customized action tokens, while the embedding of the customized actor's tokens is obtained as in [I]] The action similarity loss can be applied to the optimization of both the customized action tokens and the customized actor tokens, and our
generation results show that the action similarity loss does not affect the learning of the customized actor tokens.
15

---

D Limitation and Future Work
In this work, our goal to generate images of customized actions. There are limitations to TwinAct. First, TwinAct obtains the embedding of customized action toknes by combining action bases. Although the action base carefully defined, it is still possible to miss some actions, which limit the expressiveness of TwinAct's output domain. This can be overcome by further expanding the action space by adding more common action phrases. Another limitation comes from the physical defects of TGDMs in generating human bodies, we mitigate this problem by providing some negative prompts like "long body, low-res, bad anatomy, bad hands, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality". Furthermore, there some works [29 16 28 designed to solve the problem. Moreover, TwinAct is not restricted image generation models; it can also be used for customizing text-guided diffusion models to generate videos or 3D assets, leave it as future work.
Expected Societal Implications
We aim to address the challenge of decoupling actions and actors customize the text-guided diffusion model for few-shot action image generation. A primary ethical concern is the potential misuse of this technology, notably in creating deepfakes, which can result in misinformation, privacy violations, and other harmful outcomes. To mitigate these risks, the establishment of robust ethical guidelines and continuous monitoring is essential.
The concern raised here is a common one, not just for our method but across various multi-concept customization techniques. A viable strategy lessen these risks might be the implementation of tactics akin to those used in anti-dreambooth (66). This approach involves adding minor noise disturbances the shared images, thereby hindering the customization process. Furthermore, embedding invisible watermarks in the generated images can serve as a deterrent against misuse and ensure that they used without due acknowledgment.
16

---

V
Sample Action
A Leonardo
Barack Obama
Batman V
Polar Bear V
A Corgi
r
Sample Action
A Leonardo V*
Barack Obama l
Batman /
A Polar Bear y
A Corgi 4
Sample Action
A Leonardo y*
Barack Obama r
A Batman V*
A Polar Bear y
A Corgi V
V
Sample Action
Barbie doll and Superman
Barack Obama and Leonardo 5*
Batman and Superman /
Two gorillas 4
gorillas panda
VA
Sample Action
Batman and Superman
Barack Obama and Leonardo
Two foxes V*
Two pandas V*
Two gorillas V
Figure 11: More TwinAct generated results.
17

---

Hands
Finger
Full -ollyBody
on shoulder
Fingers crassed
Nodding head yos Shaking head no Tilting sidewaye Looking up
Crouching stealthily Sprawling out Pauncing forward
Waving goodbye
Finger snapping
Slapping forehead
up
Covering mouth
Spinning finger
Embracing warmly
heart
Finger gun
Looking down
Shoving forcefully
Casting spell
Tapping fingers
Turning head left
Expanding chest
Raising hand
bump
Tuminng head right
Contracting abdomen
folding
Palm up asking
Tucking chin
Twirling gracefully
Crossing
Praying hands
Raising eyebrows
Piroustting steadily
Pointing forward
Knuckle cracking
Furrowing
Bearing
Punching
Hand shoulder
Blinking eyes
Leaping high up
Hugging self
Peace sign
Winking eye
Ducking quickly down
Pushing away
Pinching
Squinting eyes
Tumbling forward roll
Pulling rope
Wrist flex
Gazing forward
Flipping backward somersault
Lifing weights
Slapping forehead
Glancing sideways
Shielding face hands
Clapping hands
Grabibing wrist
Staring intently
Kneeling ground
Shaking hands
Finger circling
Glaring angrily
Squatting rest
Patting back
Clenched fist
Smiling broadly
Push-up
Swinging arm
Open palm
Frowning deeply
Plank
Stretching arm
Pointing down
Puckering lips
Sit-up
Waving hand
Thumbs down
Biting
Crunch
Shaking fist
Palm facing down
Pressing lips
Burpes
Brushing hair
Finger tips touch
Opening mouth
Squat
Scratching head
Holding object
Closing mouth
Lunge
Lifing arm
Boxing punch
Yawning widely
Deadlift
Grabbing shoulder
Counting fingers
Chewing food
High knees
Reaching out
Wagging finger
Gritting teeth
Butt kick
Twisting arm
Wiping tears
Pouting lips
Handstand
Flexing wrist
Typing gesture
Sticking tongue
Arching body backward
Hlipping hand
Finger pointing up
Licking lips
Curling body forward
Pulling back arm
Finger pointing sideways
Tilting head
Leg swings
Swinging arm forward
Finger pinching
Tilting head down
Leg lifts
Swinging arm backward
Thumb flexing
Tilting head left Tilting head right
Leg drops
Throwing arm forward
Finger curling
Bicycle kicks
Pulling arm up
Finger pressing
Rotating head clockwise Jarking head back
Flttter kicks
Pushing arm down Circling arm
Thumb pressing
Crisscrass kicks
Finger rubbing
Jerking head forward
Scissr kickss
Shielding with arm
Finger crossing over
Jerking head sideways
Side kicks
Lowering hand
Finger crossing under
Soratching chin
Front kicks
Shaking arm
Finger forming heart
Puffing cheeks
Beck kicks
Resting arm on hip
Finger forming cirdle
Gnashing teeth
Inward heel taps
Leaning on elbow
Thumb pointing lef:
Stroking chin
Outward heel taps
Draping arm over head
Thumb pointing right
Lowering eyelids
Toe touches
Tucking arm behind back
Thumb sliding down
Sucking on a tooth
Walking
Swinging arm side to side
Thumb sliding up
Pinching nose
Running
Lifting arm over head Lowering arm
Finger forming
Widening eyos Narrowing eyes
Jogging
Sprinting
Extending forward
palms together
Rubbing nose
Skipping
Resting arm
Finger snap
Rubbing eyes
Jumping
Figure 12: Some examples of action phrases.
18

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
In Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification: In Section 3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In Section 4 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification: In Appendix
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cpipp/.ccppullc//uuuii wwwiiicgguie//eeeeeeeeeic CoodeSubeiiie ssi wsionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In Section 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
21

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttpas:///uripps/ wwwpeerwww....uuu.reeepp httppp///ccppp...i.tccccssttteea
Answer: [Yes]
Justification: In Appendix
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
22

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: In Reference
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthhhe....c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
23

---

Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: In Appendix
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "nKhnutgyJz", "reviewer": "Reviewer_BXKD", "summary": "Problem: Preserve the consistency of the reference action when generating images with new actors by decoupling the action and actor properly.\n\nKey Idea:\nThe authors propose a method to disentangle actions from actors in text-guided diffusion models and generate new images that exactly replicate the action pose in the given few-shot inputs. The main contributions are:\n\n-\tGenerating action phrases using GPT-4 and filtering 832 action phrases know to the model. These are embedded into a common action space with a set of action bases, thus containing no distractions from non-action related information. \n\n-\tThe action bases are then combined using a multi-layer perceptron to create new customized actions. Instead of backpropagation to determine optimal coefficients, the authors use CLIP as an action decoder to extract semantic features.\n\n-\tAn action similarity loss is introduced between the encoded features of the reference and generated action to imitate the high-level action more accurately. \n\n-\tA novel benchmark consisting of 12 actions to compare with existing methods. The authors also provide extensive analysis and ablations.", "questions": "- How is the CLIP score exactly being calculated for SAction and SActor? Is it simply an image-to-text matching score?\n\n- For the human evaluation, what is the degree of knowledge of the 100 users (average or expert)?\n\n- How do you plan to scale this method for a higher number of action variations? Just generating more actions using GPT-4 and adding their embedding to the action space might not be enough.\n\n- Is it possible to show examples of what happens if we vary the value of coefficients in the action bases and how they impact the generated image? For example, a crouching position might be a combination of action bases related to sitting and standing. So, if the coefficient of sitting is 0, the generated image should only contain standing. Again, the image might only contain sitting in the opposite case. The intermediate values of these will show various angles of crouching.\n\n- Providing a TSNE visualization of the action space might also help in clarifying.\n\n- For multiple actors, the authors show examples of images containing only two subjects. Can this be increased to more subjects?\n\n- Also, for multiple actors, how is the role in the image defined? For example, in Figure 4 last row, how will we generate Leonardo carrying Obama or Barbie carrying gorilla?\n\n-- POST rebuttal: the authors have addressed all the raised concerns, increasing my rating.", "limitations": "The authors addressed the concerns of limited output possibilities of their common action space and physical defects of TGDMs. They have not discussed any other limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThe proposed method replicates the sample action more precisely with new actors whereas existing methods make mistakes in the details. This shows that the method has successfully isolated the action information to apply it for new image generation. \n\n-\tThe examples in the provided figures give a clear idea where other methods are failing while the proposed one successfully replicates the input action with a new actor.\n\n-\tThe authors report higher scores than existing methods based on both objective and user study.\n\n-\tThe ablation studies show the impact of each contribution and varying the number of principal components (p).", "weaknesses": "-\tThe identity of actors is preserved is a strong claim which is not reflected in qualitative evaluation. For example, in Figure 4 row 4: the faces of Batman and Superman look similar. In rows 5 and 6, the subject does not exactly look like Leonardo. In the last row, we see Spiderman in place of Superman.\n\n-\tIn some cases, the generated image contains a mirrored action of the sample image. For example, in Figure 5 row 2, the generated action is performed with the opposite hand than the sample action in the 3rd and 4th images. \n\n-\tThe output generation space is limited, so some complicated customized actions might not be replicable by combining the bases. \n\n-\tIt is not clearly shown how changing the value of coefficients for the same action bases impact the generated image.\n\n-\tCurrently the model has pre-trained knowledge of only 25 actors and cannot be easily adapted to new ones without retraining.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **For Reviewer** **BXKD**\n\nThank you for recognizing our paper and recommending it for acceptance. Now, we will address the key arguments raised in the reviews.\n### **Q1.The identity of actors**\n\nIt should be noted that two factors influence the identity of the actor. \n\n(1) The first factor is **the actor identification from the reference image**. In this paper, the customized action is decoupled from the actor in the reference image via a common action space (Figure 1, second and third rows). \n\n(2) Another factor is the **confusion between actors in the new context**, as previously mentioned with regard to Batman and Superman. Even in the absence of consideration for customized actions, the stable diffusion model will nevertheless result in confusion regarding the identity information when generating multi-subject [1,2]. The issue of how to eliminate confusion when generating multiple concepts represents a significant challenge, although it is not the primary focus of this paper. \n\nIn order to maintain the rigor of the paper, we will modify the description of how the actor's identity is maintained in the paper.\n\n[1]Easing Concept Bleeding in Diffusion via Entity Localization and Anchoring\n\n[2]FineControlNet: Fine-level Text Control for Image Generation with Spatially Aligned Text Control Injection\n### **Q2.The generated image contains a mirrored action of the sample image**\n\nThe reason for generating the action containing the mirror is twofold:\n\n(1) One is that the actions in different directions are **contained in the reference image**.\n\n (2) The other is because we applied **image augmentation** (including rotation) to the training data.\n\nWe consider that the mirrored actions are, at the very least, not semantically incorrect. However, should one desire to generate actions that are entirely customized, it would be advisable to consider removing the aforementioned two factors.\n### **Q3.Some complicated customized actions might not be replicable**\n\n(1)In this paper, we have endeavored to conduct a comprehensive evaluation of TwinAct's capabilities, encompassing a range of scenarios. These scenarios **include detailed hand movements, full-body movements involving multiple body parts, and movements requiring the coordination of multiple individuals.** \n\n(2)We added **a complex action as shown in Figure 2** in Rebuttal PDF and the results demonstrate the superiority of the TwinAct method. \n\n(3)In the case of particularly intricate actions, the complexity of the customization process can be mitigated by **broadening the scope of the common action space** and **augmenting the number of reference images**.\n### **Q4.Changing the value of coefficients for the same action bases**\nWe have made adjustments to the coefficients as suggested and have generated intriguing findings as shown **in Figure 1 in Rebuttal PDF**. \n### **Q5.** **Easily adapted to new ones**\n\nAlthough the focus of this paper is on the generation of customized actions, rather than customized actors, it also illustrates the capacity of TwinAct to synthesize SD-known actors but also presents experimental findings on a personalized actor dataset (**Figure 5 in the paper**), demonstrating that TwinAct can be easily integrated into existing methods for customizing actors. \n\nFurthermore, the customization of an action by TwinAct is a relatively expeditious process, requiring approximately only **15** minutes.\n### **Q6.  Calculated for SAction and SActor**\nIn the case of S_action, our approach aligns with that of previous customization work[1,2]. The custom diffusion process employs the image from the clip and the image similarity metric for evaluation. In the case of S_actor, the pre-trained face encoder is employed to ascertain the degree of similarity.  Furthermore, to provide a more comprehensive evaluation of the generated results, we have conducted a user study.\n[1] Multi-Concept Customization of Text-to-Image Diffusion\n[2] DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation\n### **Q7. The degree of knowledge of the 100 users**\nWe employ 100 users from the outsourcing platform to conduct user research. The knowledge level of users includes junior college and undergraduate, and the ratio of junior college to undergraduate college is close to 2:1. We also provided detailed documentation and examples to ensure that all participants clearly understood the purpose and requirements of the task.\n### **Q8. Scale this method for a higher number of action variations**\n\nIf the action space is to be further extended:\n\n(1) one feasible way is to add new action phrases through expert knowledge. For example, it is possible to expand the action space by dividing the body parts at a more detailed level of granularity, such as the eyes, the mouth, the thumbs, and so forth, in order to realize a greater number of combinations of actions.\n\n(2) another possible way is to add customized actions token embedding generated by our methods to the action space as well, which we leave as an exploration for future work.\n### **Q9. TSNE visualization of the action space** \nWe have provided visualization results of T-SNE for the action space. The objective of our work is to generate customized actions through action-based combination, with the optimal action base being orthogonal. As shown **in Figure 4 in Rebuttal PDF**, the action base does not exhibit discernible clusters, and the distribution is relatively uniform, which aligns with the nature of our action base.\n### **Q10. Increased to more subjects**\nThanks for your suggestion. We additionally present the generated results of customized actions involving 3 people, as shown **in Figure 2 in Rebuttal PDF**. These results effectively demonstrate TwinAct's capability to manage multiple actors.\n### **Q11. how is the role in the image defined?**\nIf the role in the image needs to be specified, it can be specified by prompts, as shown **in the Figure 3 in Rebuttal PDF.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TXLkhyz3Q8", "reviewer": "Reviewer_3NDQ", "summary": "This paper introduced an text-to-action generation framework. In this framework, the authors first abstracted the actions (represented by natural language phrases) with PCA technique into a common action space. Then these high-level action features are fed into a text-to-image transformer network. In order to optimize the whole pipeline, the authors used a CLIP encoder to measure the cosine similarity between generated action images and reference action images.", "questions": "Please refer to weakness for more details.", "limitations": "The authors provide no limitations in the submission.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The visualization results seem promising. Compared to other approaches in the submission, the introduced approach obtain better qualitative results.\n2. The paper is well organized and easy to follow.", "weaknesses": "1. The authors claim that PCA is leveraged for feature compression when establishing common action space. Moreover, the authors borrow this technique from facial feature representations. It is known to us all that PCA is somehow an old-fashioned technique in the machine learning community. The authors are expected to provide more detailed explanations on why they choose this technique.\n2. The network architecture is quite simple and lacks technical contributions. To the reviewer, the authors simply apply the off-the-shelf text-to-image framework in their pipeline. In the meantime, other components of the framework seems simple and weak in novelty. \n3. The authors provide a similarity score as the evaluation metric. However, this score is computed by CLIP, which also serves as the action encoder during training. To the reviewer, the comparison seems not so fair. The consistency in IDs and actions should be more carefully evaluated to demonstrate the efficacy of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **For Reviewer** **3NDQ**\n\nWe appreciate your time and effort in reviewing our work, and we have carefully considered your comments. We will be sure to incorporate your suggestions to enhance the overall quality of the paper. We hope the following clarifications can address the reviewer's concerns:\n\n### **Q1. More detailed explanations about PCA**\n\nWe appreciate your insightful comments and concerns regarding our choice of Principal Component Analysis (PCA) to construct a common action space.  While PCA is a traditional technique, it is **still widely used** [1,2] today and has several advantages that make it appropriate for our study.\n\n(1) Its robustness to linearly correlated data, computational efficiency, and ease of interpretation fit well with our needs for feature compression and action space generalization. \n\n(2) **PCA is only used as part of the data preprocessing** in our method, and can be easily replaced by more advanced methods if needed.\n\n(3) To some extent, the superior performance of the common action space constructed based on the \"old-fashioned\" PCA algorithm in the experiment also verifies the effectiveness of our method.\n\n[1] Dual Prior Unfolding for Snapshot Compressive Imaging (cvpr2024)\n\n[2] Makeup Prior Models for 3D Facial Makeup Estimation and Applications (cvpr2024)\n\n### **Q2. The network architecture**\n\nA straightforward and impactful approach is the goal we strive for in our paper. The innovative of this work can be summarized as follows:\n\n(1) **Insights into the relationship between action and actor confusion**: The domain of customized action generation remains relatively unexplored. In contrast to existing paradigms of contrastive learning, our approach introduces an action space as an effective method of biased induction, which decouples actions and actors.\n\n(2) **A novel custom token inversion method**: In contrast to typical customization techniques such as TI, dreambooth, and custom diffusion, our approach does not directly learn the parameters of custom tokens through inversion. Instead, we focus on learning the coefficients of action bases.\n\n(3) **Overcome the shortcomings in the understanding of reconstruction loss**: The original diffusion loss is not well-suited to the task of learning abstract action semantics from pixel reconstruction. Therefore, we introduce a loss function that measures image similarity, and the experimental results demonstrate its efficacy in improving customized action generation.\n\nIn summary, our objective is to fine-tune a text-guided to image generative model. However, we propose new designs in multiple aspects, including **parameter initialization, fine-tuning manner, training paradigm**, and provide **detailed motivation, analysis, and ablation experiments** for each component. Extensive experiments also prove that TwinAct can achieve the best performance with its simple and intuitive design.\n\n### **Q3. The evaluation metric**\n\nThanks for your suggestion, we actually used two clips of different sizes. During training, we use **OpenAI CLIP ViT-L (246M)** as the action encoder, and **OpenCLIP ViT-bigG (1.39G)** to evaluate the similarity between generated and reference images. As you suggested, we supplemented the results used for the evaluation using other models (**Align [1]**) as shown in the Table 1 in Rebuttal PDF and below.\n\n\n\n| S_action | Text Inversion | DreamBooth | Reversion | Custom Diffusion |  P+   |  ADI  | Ours  |\n| :------: | :------------: | :--------: | :-------: | :--------------: | :---: | :---: | :---: |\n|   CLIP   |      9.12      |   12.23    |   18.73   |      26.83       | 33.95 | 45.32 | **69.47** |\n|  Align [1]   |     10.05      |   11.67    |   17.37   |      28.33       | 30.45 | 44.93 | **70.82** |\n\n\n\nThe experimental results show that TwinAct **still achieves the best results** among the Align-based evaluation results. In addition, we also constructed **user study (Table 1 in main paper)** and **4-dimensional error analysis (Figure 4 in main paper)**. These experimental results also prove the superiority of TwinAct.\n\n[1] Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ejFxnzHJz9", "reviewer": "Reviewer_Q179", "summary": "This paper aims to tackle the few-shot action image generation problem by customizing text-conditioned diffusion models. To decouple action from actor, the proposed method introduces a common textual action feature space to avoid the interference of the actor's visual semantics during action generation. The experimental results demonstrate the proposed method is effective in generating customized actions and preserving actor identities.", "questions": "Please see the Weaknesses section.", "limitations": "The limitations are discussed and there is no potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work focuses on customizing text-conditioned diffusion models for few-shot action image generation, which is an interesting and meaningful topic;\n- The motivation of decoupling action and actor for action image generation is good, and the proposed textual action space is novel and effective in achieving the decoupling.  \n- The proposed method is easy to understand, and the writing is good.", "weaknesses": "The overall method is technically sound and the demonstrated results are great, but the reviewer still has some concerns about the method designs,\n\n- The reviewer thinks it is reasonable to use three embeddings (i.e., k=3) to encode each action. However, it is hard to understand why the PCA is separately applied along the token embedding axis (e.g., when k=1, only all the first embeddings from each action are processed by PCA). \n- The collected image action dataset is very small (12 actions only). Why not use some image/video action datasets? And, more importantly, how to make sure the learned/customized model can be generated to unseen action image generation.\n- Regarding the action similarity loss, it is hard to ensure that the high-level action semantics will be learned, as there are no explicit constraints.\n- minor issue: the high cost of action phrases filtering: the authors generate 1.2K action phrases using GPT-4 and a total of 2x1.2K images are generated using Stable Diffusion XL; The generation is expensive and the manual filtering process is labor-intensive.\n\n/I will improve my rating if the author's response addresses all my concerns well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **For Reviewer** **Q179**\n\nWe are sincerely grateful to the reviewers for dedicating their time and effort to review our work, and we appreciate the recognition of the novelty of our approach and the significance of our given the impressive result. We will try to address  reviewer's comments in detail below.\n\n\n### **Q1. why the PCA is separately applied along the token embedding axis**\n\nWe are grateful for your valuable suggestions. In response, we have included additional experimental results in the following table.  In our supplementary experiments, we focused on encoding action phrases at the phrase level, as opposed to the token level, using the CLIP model and leveraging the output of the final layer pool.\n\n| Methods    | S_Actor | S_Action |\n| ---------- | ------- | -------- |\n| w/ token   | 73.34   | 69.47    |\n| w/ phrases | 60.27   | 58.66    |\n\n\n\nThe experimental results presented in Table 2 of the Rebuttal PDF indicate that conducting principal component analysis (PCA) on the embedding of individual tokens is more effective than performing PCA on the embedding of the entire action phrase. This is attributed to the greater flexibility afforded by analyzing the token dimension. For instance, when analyzing the phrase \"raising hand,\" distinct coefficients can be assigned to the words \"raising\" and \"hand\" when PCA is applied to the token dimension. Conversely, applying varying coefficients to verbs and nouns across the entire phrase dimension is not feasible.\n\n\n### **Q2.** **Enhancing Action Dataset for Unseen Action Image Generation**\n\n(1) Thank you for your suggestion, **w**e acknowledge that the current dataset is limited in scale. We will expand it further in subsequent work. However, we have made a concerted effort to ensure the diversity of the data by including various types of movements, such as localized movements, whole-body movements, and both single-person and two-person movements.\n\nFurthermore, the current dataset includes some actions that Stable Diffusion (SD) has not encountered before, such as those shown in **rows 1-4 of Figure 4 in the main paper**. Additionally, we have incorporated a more challenging action **in Figure 2 of the Rebuttal PDF.**\n\n(2) Collecting data from image or video action datasets is a good idea, but we find some datasets such as Stanford 40 Actions [1], ADHA [2], CMU MOCAP [3], tend to be designed for action recognition, meaning that most of them only contains **common actions** **(SD-know)** such as **running, jumping, playing guitar, chopping vegetables, etc.** Our goal is to fine-tune the stable diffusion model to generate images of **unseen actions**, so we focus on collecting images of actions that are **outside the distribution.**\n\n(3) For how to generate images of unseen actions, we have experimented in our paper. These experiments are illustrated **in rows 1-4 of Figure 4 in main paper and Figure 2 in the Rebuttal PDF.** These actions are unknown actions (as evidenced by the results shown in Figure 8 in the Appendix). The experimental results show that TwinAct is capable of generating these unseen actions effectively. Moreover, TwinAct can combine these actions with previously unknown actors, as demonstrated **in Figure 5 of the main paper (Actor OOD + Action OOD).**\n\n[1]http://vision.stanford.edu/Datasets/40actions.html\n\n[2]http://[www.mvig.org/research/adha/adha.html](https://www.mvig.org/research/adha/adha.html)\n\n[3]http://mocap.cs.cmu.edu/\n\n### **Q3.The action similarity loss**\n\n(1)  The reconstruction loss is designed to **focus on the low-level details** of an image (see ADI[1], Section 1). However, it is prone to **overfitting** to the reference image (see Dreambooth[2]). To address this, we propose incorporating supplementary supervisory signals, specifically the semantic similarity of images.\n\n(2) CLIP, which has been pre-trained through image-text contrastive learning, has demonstrated **robust performance in numerous action recognition tasks** [3, 4]. Therefore, we utilize it as the action encoder. \n\n(3) The outcomes of the ablation experiments, as illustrated **in Figure 7(b) and row 4 of Table 2 in the main paper**, also indicate that the inclusion of non-low-level visual similarity loss can enhance the generation of customized actions.\n\n[1] Learning Disentangled Identifiers for Action-Customized Text-to-Image Generation\n\n[2] DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation\n\n[3] FROSTER: Frozen CLIP Is A Strong Teacher for Open-Vocabulary Action Recognition\n\n[4] ActionCLIP: A New Paradigm for Video Action Recognition\n\n\n\n\n### **Q4. The high cost of action phrases filtering**\n\nThe objective of our filtering process is to eliminate certain actions that are incapable of being accurately generated by SD. Consequently, our STEP for generating these images necessitates **25 steps** to ascertain the viability of action generation. In comparison, the typical number of STEPs for generating an image is 50, which expedites the generation process to a certain extent. The generation of the images for filtering required a total of **2.5 hours on an A100 GPU**.\n\nIn the context of a manual filtering process, there are some possible **ways to reduce the cost**. The deployment of a multimodal large language model, such as GPT4, can serve as a filtering process. This approach entails the generation of images, which can then be subjected to filtering by GPT4. An additional method for reducing costs is to utilize an image description model. The filtering process entails the generation of corresponding text descriptions and the determination of whether these descriptions contain action phrases or a resemblance to the text prompt utilized for the generation of action images.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MIwqlUumh9", "reviewer": "Reviewer_2ZgR", "summary": "The paper introduces “TwinAct”, a novel method for separating actions from actors in few-shot action image generation using text-guided diffusion models (TGDMs). It creates a “common action space” to focus on actions alone, allowing for precise customization without actor-specific details. The process is streamlined into three main steps: constructing the action space from key phrases, imitating the action within this space, and generating varied, context-adaptive images using an action similarity loss. And the results demonstrate its robustness and versatility for personalized and creative content generation.", "questions": "Please check the detailed comments in the weaknesses part.", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  The idea of imitating actions through a combination of action bases within this space is interesting and provides a new way to achieve few-shot customization.\n2.  The paper is well-structured and clearly articulated. The method is technically sound, with a clear explanation of the steps involved, from building the action space to generating the final images.\n3.  The experimental results are impressive, especially the robustness of TwinAct. The potential applications for both customized characters and actions are of great interests.", "weaknesses": "1.  In Figure 6, why do all the methods show a performance degradation when two people are involved?\n2.  Can the authors explain in detail how the model can be customized for both character and action？\n3.  Could the authors provide examples and analyses of where TwinAct fails or performs suboptimally? What steps can be taken to address these failure modes?\n4.  Adding something like Figures 8 and 9 from the appendix would make the results more convincing for readers who do not venture to the appendix to search for plots.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **For Reviewer** **2ZgR**\n\nWe are sincerely grateful to the reviewers for dedicating their time and effort to review our work, and we appreciate the recognition of the novelty of our approach and the significance of our given the impressive result. We will try to address  reviewer's comments in detail below.\n\n### **Q1. Performance degradation when two people are involved**\n\nIt should be noted that two factors influence the identity of the actor. \n\n(1) The first factor is **the actor identification from the reference image**. In this paper, the customized action is decoupled from the actor in the reference image via a common action space (Figure 1, second and third rows). \n\n(2) Another factor is the **confusion between actors in the new context**, as previously mentioned with regard to Batman and Superman. It should be noted that even in the absence of consideration for customized actions, the stable diffusion model will nevertheless result in confusion regarding the identity information when generating multi subject[1,2]. The issue of how to eliminate confusion when generating multiple concepts represents a significant challenge, although it is not the primary focus of this paper. \n\nAs a result, all models show performance degradation with two actors **due to the confusion of the multi-subject generation**.\n\nThe issue of how to eliminate the confusion represents a significant challenge, although it is not the primary focus of this paper. \n\n[1]Easing Concept Bleeding in Diffusion via Entity Localization and Anchoring\n\n[2]FineControlNet: Fine-level Text Control for Image Generation with Spatially Aligned Text Control Injection\n\n### **Q2. How the model can be customized for both character and action**\n\nThe implementation of custom action generation can be integrated into existing custom production frameworks in an seamless manner. In the event that both the actor and the action are to be customized, it is only necessary to update the tokens of the action and the actor using different strategies. In particular, the actor tokens are updated directly by inversion, while the action tokens are generated by PCA coefficients, which are also updated by inversion.\n\n### **Q3. Bad case analyses**\n\nOne of the suboptimal results is the problem of confusion that arises when the generation process is conducted by multiple individuals. One potential solution is to introduce spatial constraints, such as bounding boxes[1] or manipulation attention maps, as discussed in reference [2]. It is also noteworthy that it is challenging to generate customized actions in certain specific contexts, such as when dealing with reptiles, such as snakes.\n\n[1] FineControlNet: Fine-level Text Control for Image Generation with Spatially Aligned Text Control Injection\n\n[2] Divide & Bind Your Attention for Improved Generative Semantic Nursing\n\n### **Q4. Adding Figures**\n\nThank you for your suggestions. We will readjust the layout in the final version to include the supplement of Figure 8 and Figure 9 in the paper, so that readers can have a better understanding", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### For ALL Reviews

We sincerely thank all the reviewers for their thoughtful feedback. We are glad to see that most of the reviews have recognized our work:

**[ALL Reviews]** **Robust and Superior Results:** We are grateful for the reviewers’ positive feedback on our impressive experimental results, particularly our method's ability to consistently outperform existing approaches in both objective metrics and user studies.

**[BXKD, Q179, 2ZgR]** **Innovative Approach:** We are delighted that most reviewers acknowledge our motivation is good and interesting, and the novelty and effectiveness of our method in action image generation.

**[ALL Reviews]** **Detailed and Accessible Presentation:** We appreciate the reviewers highlighting the comprehensiveness of our explanations and the ease of understanding our paper.

We will now address the key points raised in the reviews and provide detailed responses to each reviewer. We highly recommend that reviewers take the time to **check PDF for additional visualization of results**.

Once again, we extend our thanks for the reviewers’ time and valuable insights, and we look forward to any additional feedback or questions regarding our work.

Meta-review: The paper proposes a method to decouple actions and actors for few-shot action image generation in text-guided diffusion models. It has three key steps: building the common action space based on representative action phrases, imitating the customized action within it, and generating customized action images with action similarity loss. Three of the reviewers' concerns are addressed and one reviewer's most concerns are addressed. After reading the paper, reviews, and responses, the AC agrees that this paper has proposed a sound method, and the method has shown interesting improvements.
Decision: Accept (poster)