Action Imitation in Common Action Space for Customized Action Image Synthesis

Authors: wang  lin
jingyuan  chen
jiaxin  shi
zirun  guo
yichen  zhu
zehan  wang
tao  jin
zhou  zhao
fei  wu
shuicheng  yan
hanwang  zhang
zhejiang  university
xmax  ai
skywork ai singapore
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 07:49:36

Abstract: We propose a novel method, \textbf{TwinAct}, to tackle the challenge of decoupling actions and actors in order to customize the text-guided diffusion models (TGDMs) for few-shot action image generation. TwinAct addresses the limitations of existing methods that struggle to decouple actions from other semantics (e.g., the actor's appearance) due to the lack of an effective inductive bias with few exemplar images. Our approach introduces a common action space, which is a textual embedding space focused solely on actions, enabling precise customization without actor-related details. Specifically, TwinAct involves three key steps: 1) Building common action space based on a set of representative action phrases; 2) Imitating the customized action within the action space; and 3) Generating highly adaptable customized action images in diverse contexts with action similarity loss. To comprehensively evaluate TwinAct, we construct a novel benchmark, which provides sample images with various forms of actions. Extensive experiments demonstrate TwinAct's superiority in generating accurate, context-independent customized actions while maintaining the identity consistency of different subjects, including animals, humans, and even customized actors.

Main paper: 
 introduction: 
 We are interested in customizing Text-Guided Diffusion Models (TGDMs) (e.g., Stable Diffusion [21] ) to generate customized actions specified by a few user-provided images. For example, given a few images of Black Widow's signature three-point landing action, we can represent this action as a unique token V * and generate imaginary creation by the prompt like "Leonardo V * " (the third row in Figure 1 ). One might question why we do not condition TGDMs on precise textual descriptions [21] or sketch images [31] to generate the desired action. The reason is that actions are often unutterable, and even with finely detailed descriptions (SD column in Figure 1 ) or skeleton images (ControlNet column in Figure 1 ), TGDMs cannot accurately follow these instructions.Further, can we use existing few-shot customized TGDMs [4, 7, 8, 10, 14, 22, 27] to customize actions? Unfortunately, the answer is still no. This is because these methods fine-tune the textual embeddings of V * in prompts like "Black Widow V * " by reconstructing the corresponding image. But, except for actions, there are many other semantics in the exemplar images that are coupled in V * , like the characteristics of the specific actors. Thus, the few-shot action images in those methods lack an effective inductive bias to achieve decoupled action tokens V * from actors, which results in action-irrelevant information like Black Widow's red curly hair being confounded in the generated images as shown in Figure 1 . Although some methods mitigate the effects of missing inductive bias through contrastive learning on textual embeddings or visual images, they still struggle to exclude all action-irrelevant semantics. For example, [8] uses prepositions as positive samples and randomly selects negative samples from a limited set of POS words (i.e., nouns, adjectives). However, this approach can not guarantee the exclusion of all action-irrelevant information from exemplar images for V * , resulting in the inclusion of irrelevant semantics like manicure and red curl hair (the first and third rows in Figure 1 ) confounding the customized tokens. On the other hand, [7] and [14] utilize contrastive training data involving the same subject with and without specific customized concepts (e.g, melted and closed eyes, etc.) to decouple the subject and customized concepts. However, the creation of extensive contextual training data is costly, particularly when dealing with customized actions that involve multiple subjects.By observing the real-world process of a director instructing an actor to perform a specific action, such as the signature action of the character Black Widow which involves a set of action phrases including "left hand on the ground", "right hand raised behind the back", "left leg on the ground", and "right leg straight out", we find that these phrases concentrate solely on actions without providing any specific details about the actor involved. Moreover, these phrases can be modified or combined to create new actions, as illustrated in Figure 2 . Inspired by these observations, we propose a method called TwinAct, which aims to decouple the action from the actor by constructing an effective Common Action Space. This space is exclusively for actions, eliminating any confusion with actor-specific characteristics and enabling the generation of diverse and creative actions that can be applied across various actors and scenarios.Our approach involves three steps. 1) We collect 832 action phrases that cover a range of movements from finger gestures to full body poses, and extract the corresponding textual embeddings using the tokenizer of CLIP [18] . These textual embeddings are used to build a common action space (Section 3.2) through Principal Component Analysis (PCA [9] ); 2) We employ a Multilayer Perceptron (MLP) to modulate the PCA coefficients to imitate the customized actions within the action space (Section 3.3); and 3) We introduce an action similarity loss to provide semantic similarity between the exemplar images and generated images, resulting in synthesizing highly adaptable customized action images in diverse contexts (Section 3.3). Extensive experiments demonstrate the superiority of our method over the existing method. Finally, the main contributions of this paper are summarized as:• We propose TwinAct to generate customized action images, which can synthesize novel renditions of user-specific action in different contexts including animals, humans, and even customized actors. • We introduce a common action space that focuses solely on the action itself, without any actorspecific details. By combining action bases in this space for customized action imitation, we are able to effectively decouple action from actor, allowing for greater flexibility in customization. • Through extensive experiments, we demonstrate that TwinAct outperforms previous methods in maintaining the fidelity of customized actions while also preserving the consistency of actor identities.2 Related Work 
image generation and customization: 
 The text-guided diffusion models (TGDMs) [15, 19, 20, 21, 23] have emerged as a powerful paradigm in the domain of image generation, enabling the creation of image variations that are aligned with specific textual descriptions. Recently, the customized TGDMs [4, 5, 8, 10, 13, 11, 22] aim to encoder the customized concept within the textual embedding, often represented by a unique token and decode the token into pixels by the cross-attention in the U-net decoder [24] . Textual Inversion [4] optimizes textual embedding and synthesizes personalized images by integrating the concept token with the target prompt. DreamBooth [22] extends this concept by proposing a framework that optimizes all parameters of the denoising U-Net architecture, based on a specific token and the class category of the subject. Several other works [5, 10] have focused on optimizing subsets of weights or introducing additional adapters to achieve more efficient optimization and better conditioning of the generated images. For instance, Custom Diffusion [10] fine-tunes only the cross-attention layers within the U-Net, while P+ [27] expands the textual-conditioning space with per-layer tokens to allow for greater disentanglement and control over the generation process. Despite these methods achieving commendable results in customized actors, they struggle with generating customized action images, as shown in Figure 1 , due to the lack of an effective inductive bias with few exemplar images.In contrast to the rapid progress in customized actors, customized action has received less attention in the community. Since the complex details such as specific angles, body positions, and motion trajectories, describing a specific action is more challenging than describing a specific actor. Even with finely detailed descriptions (see Appendix A.1) TGDMs still cannot accurately follow these textual descriptions. A straightforward approach to generate a specific action image is ControlNet [31] which is conditional on a given skeleton image to generate images. However, they only provide a rough approximation of action consistency (see Appendix A.3). Particularly, when it comes to capturing fine details such as fingers, skeleton images may lack the necessary detail to accurately reproduce these actions, leading to noticeable visual defects in the generated images. In addition, when conditioned on detailed images (see Appendix A.3), they suffer from limited diversity and flexibility. Recent advancements for customized TGDMs preliminary explore action-based customization. Reversion [8] proposes relation-steering contrastive learning that aims to guide the relation prompt towards relationdense regions within the text embedding space, thereby disentangling the learned relation from other concepts like appearances. Lego [14] and ADI [7] construct contexts involving the same subject with and without a specific concept, to decouple the subject and specific concepts. However, these methods by employing contrastive learning on textual embeddings or visual images can not effectively decouple actions from actors. This is because they cannot exclude all irrelevant concepts with a limited number of negative samples. 3 MethodsImagine the following scenario: When the character Black Widow performs her signature three-point landing action, the director may give the following instruction "Bend knees and lower body down into a squatting position; Place right hand forward, and right arm should be fully extended". We can see that when people are asked to perform a specific action, it is often accompanied by instructions that involve the composition of various body movements and the adjustment of angles and amplitudes as shown in the right part of Figure 2 . Inspired by the way in which humans perform specific actions, we propose TwinAct which applies similar principles to action customization. Specifically, we first collect a set of basic actions and encode them using a pre-trained text encoder to build a common action space represented by a set of action bases. Subsequently, we imitate the customized action by decomposing it into a combination of action bases with corresponding weights. Finally, we utilize both the reconstruction loss and action similarity loss to fine-tune the customized token embedding in the text encoder and the LoRA [6] layer in the text-to-image decoder to generate high-fidelity images of the customized action.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: In Appendix Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 Our study is based on the text-guided diffusion models (TGDMs) which consist of a textual encoder E and a text-to-image decoder D. Given a few sample action images x and a textual prompt c with a customized token V * , such as "a photo of V * ", we aim to fine-tune the embedding of customized token V * in E and the LoRA layer in D to realize action customization. The textual encoder E first tokenizes the prompt c into a set of token embeddings h, which are then used to generate a textual condition τ θ (c) with the text transformer in E. Subsequently, the textual condition τ θ (c) is utilized by the conditional denoising diffusion model ϵ θ (•) in D to generate images. The commonly used training objective is:EQUATION)where t is the timestep, x 0 is the original image and x t is a noisy image constructed by adding noise ϵ ∼ N (0, 1) to x 0 . After training, any textual condition τ θ (c) with V * can generate customized action in a new context defined by prompt c.The initial step involves identifying the set of action bases that defines the common action space within the text-to-image diffusion model.First, we task GPT-4 [1] with generating a range of common action phrases based on different body parts, including head, fingers, hands, and full body (see Appendix B.2). Then, we carefully sift through the generated action phrases to eliminate duplicates. Subsequently, we build a manual filter based on the pre-trained text-to-image diffusion model by constructing prompts and synthetic images of each action phrase to remove those action phrases that are unknown to the text-to-image diffusion model, e.g., "Reverse Plank" as shown in Figure 2 .After the filtering process, we identify a total of m = 832 action phrases. Each phrase a i , where i ∈ {1, ..., m}, is tokenized and encoded into an action embedding groupA i = [a i 1 , ..., a i k ].It is important to note that the length k of each action embedding group A i varies, as each action may consist of a different number of words. Through empirical observation, it was found that action phrases with three words or fewer are the most common, prompting the designation of each A i as containing three embeddings (i.e., k = 3 for all m actions). For simplicity, we denote the three embeddings of all A i as Q 1 , Q 2 , and Q 3 respectively, whereQ k = [a 1 k , ..., a m k ].Inspiring by prior research [2] employing PCA to transform high-dimensional facial data into a lower-dimensional space, we calculate the PCA transformation matrix B k for each set V Q k using the function PCA(Q k , p). This transformation plays a crucial role in reducing redundant information within the data and enhancing the efficiency of model optimization. Specifically, PCA(Q k , p) reduce dimensionality of Q k ∈ R m×d and generate p principal components asB k = [b 1 k , . . . , b p k ] ∈ R p×p, which captures the essential features of data in a more compact form. Our experimental findings indicate that optimal results are achieved when p = 512 (see Section 4.3). Finally, we obtain a set of action bases as [B 1 , B 2 , B 3 ], which define an action space that does not contain any action-irrelevant information thus providing a strong inductive bias to achieve decoupled action token embeddings.This step involves searching for the optimal parameters to combine the action bases for imitating customized actions.Our experiments (see Figure 1 ) have shown that existing methods, which aim to identify optimal personalized token embeddings within the vast text space of pre-trained text encoders, struggle with the inclusion of non-action information. And even with contrastive learning, they are still unable to exclude all irrelevant information. Fortunately, action bases offer an alternative approach for optimizing the customized token embedding. By combining action bases[B 1 , B 2 , B 3 ] with corresponding coefficients [W 1 , W 2 , W 3 ], where W k = [w 1 k , . . . , w p k ],new actions can be imitated as shown in Figure 2 . To achieve this, we first utilize a Multi-Layered Perceptron (MLP) to combine the action bases. While backpropagation is a common method for determining optimal coefficients, our experiments indicate its limited effectiveness due to the scarcity of user-provided images (see Section 4.3). Therefore, we use CLIP as an action encoder to extract semantic features as a prior. Specifically, given an action sample image x, we first extract the action features x p with action encoder ψ (x), then MLP is used to map x p into the modulating coefficients [W 1 , W 2 , W 3 ] as:EQUATIONFinally, for a customized action, we can combine the action bases to imitate the action as:EQUATION)where V is the embedding of the customized tokens V * . In this way, V is a linear combination of the action bases. Consequently, the vector interpolated in the action space would not contain any action-irrelevant features, allowing us to achieve the objective of decoupling the action and the actor.The final step involves optimizing the embeddings of the customized token V * , which is derived from the action base through linear combinations and synthesis customized action image.Training. Initially, a few user-provided images and corresponding textual prompts containing V * are used to apply the pixel reconstruction-based loss L rec , similar to previous customized TGDMs (in Eq 1). However, the traditional L rec in existing TGDMs aims to faithfully reconstruct the sample image appearance, focusing on low-level pixel features. This narrow focus limits the model's ability to recognize and invert high-level action features in the images, as shown in Figure 7 (b).To address this limitation, we introduce an action similarity loss to enhance the model's grasp of high-level action semantics. Beginning with an input image x and a corresponding prompt c, we randomly initialize a latent variable x T . This variable is progressively denoised until it reaches a randomly selected timestep t, at which point the denoised image x prd is predicted directly from x t .The goal is to evaluate the action similarity between these two images. Specifically, by leveraging action encoder ψ (•), the encoded action features for both the reference action x p and the generated action x prd p are extracted. We then calculate the cosine similarity between these action features to measure action consistency during the generation process. This similarity is formally defined as:EQUATIONBy integrating the action similarity loss, the model's focus shifts from detailed pixel information to high-level action semantics. The final training loss for TwinAct can be expressed as:EQUATIONTesting. After training, the three groups of coefficients [W 1 , W 2 , W 3 ] and the LoRA parameters in the text-to-image decoder are saved. Users can generate images depicting customized actions in different contexts by providing prompts such as "Leonardo V * ".Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] • The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
experiment setup: 
 Dataset. We tackle the challenge of decoupling specific actions from the actors in user-provided images. Since there are no publicly available customized action datasets, we introduce a novel benchmark, consisting of 12 actions involving multiple body parts, such as fingers, arms, legs, and full-body motions. Each action contains approximately 10 sample images. In addition to actions performed by a single actor, we incorporate more complex actions that involve multiple actors.Metrics. To comprehensively validate the efficacy of TwinAct, we incorporate objective metrics to assess the quality of the generated images. We calculate the consistency between the action in the sample image and the generated image through the CLIP score, which is denoted as "S Action ". Additionally, ensuring identity consistency is a crucial aspect of our task, so we evaluate the actor's identity similarity using a pre-trained face recognition encoder [3] , which is denoted as "S Actor ".In addition, we also conduct user studies including "U Action " and "U Actor ", representing user preferences for action fidelity and actor identity consistency in the generated images.Qualitative Comparison. We illustrate the advancements of our method compared to previous approaches in handling various forms of actions, including hand, body, single-actor, and multi-actor actions, as shown in Figure 1 . Initially, we evaluate two methods for generating customized action images without fine-tuning, as shown in the column of SD and ControlNet in Figure 1 . Images generated from detailed textual prompts (SD) frequently exhibit misalignment with the actions in the sample images. On the other hand, Controlled Generative Models (ControlNet), which depend on skeleton images, tend to only imitate the basic structures of the sample images and lack a profound understanding of the contextual dependencies required for action customization. This deficiency leads to significant body deformations and flaws, particularly when animals are involved in the action. For example, in the fifth and sixth rows of Figure 1 , both the human and the gorilla mistakenly raise their hands instead of their feet.Subsequently, we conduct comparisons with state-of-the-art baselines such as Textual Inversion [4] , DreamBooth [22] , Reversion [8] , Custom Diffusion [10] , P+ [27] , and ADI [7] . Textual Inversion and DreamBooth tend to capture minimal action-related information, primarily because of their limited tunable parameters. On the other hand, Custom Diffusion, which fine-tunes cross-attention in U-net, and P+, which utilizes per-layer token embeddings, manage to retain more information from the sample images. However, they uncontrollably invert action-irrelevant details in the sample images and bring these details into the generated image, such as the human hand and the manicure in the first and second rows shown in Figure 1 . In addition, Reversion and ADI try to exclude action-irrelevant information from the images by contrastive learning but face challenges in comparing all negative samples. Therefore, the images they generate still contain some confounded details, such as the red curly hair of the Black Widow in the third row and the yoga pants in the sixth row. In contrast, TwinAct shows the best customized action generation results. Through the acquisition of a customized action token embedding, which is derived from a linear combination of action bases within the common action space, TwinAct accurately capture the action information in the sample images while excluding irrelevant details with good generalization.Quantitative Comparison. In addition to the visual quality, we also present a numerical comparison between TwinAct and the baselines in Table 1 . As can be seen, our approach outperforms in terms of both action and actor similarity, demonstrating its effectiveness in imitating customized actions and filtering out irrelevant information that could potentially confuse the portrayal of the new actor. Furthermore, the performance across four representative types of action customization tasks (i.e., Single-Actor, Two-Actor, With-Human, With-Animal) is illustrated in Figure 6 . Our method consistently shows superior performance and the least variability across all types. Notably, the method labeled as P+ experiences the most significant performance drop when involving animals, and ADI exhibits a considerable performance drop when handling actions involving two actors.Additionally, following the methodology outlined in [30] , we conduct a quantitative comparison involving human evaluators to further validate the effectiveness of TwinAct. We invite 100 users to rate each pair of actor-action images generated by TwinAct and baselines on a scale ranging from 1 (worst) to 5 (best), focusing on both action and actor consistency. The collective feedback, as presented in Table 1 , clearly indicates a strong preference for TwinAct among the users. Impact of Action Space. We assess the impact of the number of action phrases for building the common action space, as shown in Figure 7 (a). In extreme cases, if there exists no action space and we directly learn the customized token embedding from the action encoder (w/o action space), the model can not generate customized actions and is overfitted to the input image details, such as the actor's clothes and the background. When utilizing a smaller number of action phrases (w/ 50% action phrases), the generated quality is not as good as TwinAct.Impact of Action Encoder and Action Similarity Loss. We assess the impact of the action encoder and action similarity loss in Figure 7 (b) and 2 , indicate that the optimal results are achieved when p is set to 512. Additionally, we showcase the variations in the generated images for different p values in Figure 7 (c).Our method demonstrates strong adaptability to integrate with existing subject-driven customized generation methods [11] , as shown in Figure 5 . It effectively adapts to customized actors, which were unknown by TGDMs, and generates visually engaging images showcasing these actors performing customized actions. In contrast, as depicted in Figure 10 in Appendix, existing methods struggle to accurately capture action-related features, resulting in unsatisfactory generation of customized actions and a lack of identity consistency for the customized actors. 
conclusion: 
 In this paper, we introduce a novel approach, TwinAct, for customized action image generation, which aims to preserve the fidelity of the action and the consistency of the actor's identity. TwinAct consists of a common action space and can create new actions by adjusting or combining the action bases in the space. By imitating action via action bases, TwinAct effectively decouples action from the actor. Extensive experiments demonstrate TwinAct's superiority in generating accurate, contextindependent customized actions across various subjects, including humans, animals, and customized actors. Furthermore, TwinAct's ability to preserve the high fidelity and identity consistency in generated images highlights its robustness and adaptability. The potential applications of TwinAct in areas such as animation, gaming, and visual effects are vast, offering new opportunities for creative and personalized content generation.In this appendix, we first provide an in-depth analysis of Stable Diffusionin in Section A.1 and ControlNet in Section A.3, and additional results of TwinAct in Section A.3 and A.4, demonstrating its effectiveness in generating customized action images. Then we introduce the evaluation setting in Section B.1, the details of the action space in Section B.2, and the implementation details in Section C. Finally, we discuss the limitations in Section D and societal implications in Section E for TwinAct.A More Qualitative Results 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: N/A Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: In Reference Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: N/A Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We first utilize GPT-4 to generate a detailed description of the actions in the sample image provided by the user, as shown in Figure 12 . We can find that on the one hand, although GPT-4's description is highly sufficiently, it still inevitably loses many details of the action. On the other hand, existing text-guided image diffusion generation models have difficulty in understanding such detailed textual descriptions, like "heart" in the fourth row in Figure 12 . Both reasons result in the generation of action images that are far from the sample image. In contrast, our approach accurately captures action-related features, and the user only needs to provide textual prompts like "a polar bear V * ", and then customized action images can be generated in the desired contexts.In addition to controlling the generation of models through textual conditioning, current work such as ControlNet supports extracting spatial constraints from reference images as conditions to generate images. We tried two reference images, skeleton images, and sketches, as conditions to control the generative model to generate customized action images. The results are shown in Figure 9 . We can find that the skeleton image-based approach loses a lot of details of the action, such as fingers, and it is difficult to imitate the customized action. The other sketch-based approach provides more details, but this limits the diversity of the model generation results, e.g., it is difficult to generate images of animals performing customized actions since the sketches contain human bodies. Also, the sketches contain some additional action-irrelevant information such as the background, the actor's hairstyle, and so on. More importantly, the controlled generation model lacks the understanding of the contextual dependency of the action, as shown in the third row in Figure 9 . It is difficult for the generation model to understand the relationship between the hands and the feet in the customized action, which leads to serious body distortions and visual deficiencies, and this becomes even more obvious when animals are involved.In this section, we compare the results of existing customized TGDMs and TwinAct in generating images containing customized actors and customized actions. As shown in Figure 10 , our approach significantly outperforms the previous methods in maintaining the fidelity of the action and consistency of the character's identity. Textual Inversion, Dreambooth, and Revrsion struggle to imitate the customized action. As for, Custom Diffusion, P+, and ADI, although they can roughly maintain the consistency of the actions specified in the sample image, fail to maintain the identity information of the customized actors. This is because their customized action tokens are confounded with information irrelevant to the actions in the sample image, e.g., Black Widow's red curly hair in the first row, and the yoga pants in the second row.In this section, we show more customized action images generated by TwinAct, as shown in Figure11, covering the rest of the 5 actions within our benchmark.Following [7] , we provide 25 subjects for evaluation as follows: The results show that it is difficult for the user to provide a suitable sketch for generating customized action images. The results generated with skeleton images show it is difficult to capture the details of the action such as fingers, while the results generated with line images are limited in generalization, especially when it involves animals.• generic human: "A boy", "A girl", "A man", "A woman", "An old man"• well-known personalities: "Barack Obama", "Michael Jackson", "David Beckham", "Leonardo DiCaprio", "Messi", "Spiderman", "Batman" • animals: "A dog", "A cat", "A lion", "A tiger", "A bear", "A polar bear", "A fox", "A cheetah", "A monkey", "A gorilla", "A panda" • customized actors: "Cruze", "Imelda" where the inclusion of diverse and previously unseen actors, as well as animals and customized characters, requires models to retain pre-trained knowledge while also generating accurate and undistorted representations of these actors.As stated in the manuscript, we devise a set of action phrases to build action space. Specifically, we collect a comprehensive list of about 1,200 action phrases with the help of GPT-4 [1] . After obtaining these action phrases, in order to filter out those action phrases that are not recognized by the TGDMs. we synthesize the image result by providing 2 types of prompts, i.e. "A photo of {Action}", "A photo of someone {Action}", to the Stable Diffusion XL [17] , where {Action} represents an action phrases (e.g. "raise hand"). Those action phrases that do not generate the correct action, such as "reverse plank", or "crab walk", are filtered out. We ended up with 832 valid common action phrases, which were then further tokenized and encoded into the action space.C.1 Implementation Details about TwinAct Implementation Details. For our method, we use the AdamW [12] optimizer with a learning rate of 2e-4. We use CLIP as a preprocessor to estimate the action of the given reference image. Unless otherwise specified, Stable Diffusion XL is selected as the default pre-trained model, and images are generated at a resolution of 1024×1024. All experiments are conducted on A-100 GPUs. We fine-tuning text embedding along with the LoRA layer. We integrate the LoRA layer into the linear layer within all attention modules of the U-net, utilizing a rank of r = 8. Sample Details. All experiments and evaluations make use of the DDPM [25] with 50 sampling steps with a scale of 7.5 for all methods. To ensure consistency and filter out undesired variations in diffusion models, we follow the approach outlined in [11] by employing the same negative prompt for both our method and the comparison methods during sampling. The negative prompt used is "long body, low-res, bad anatomy, bad hands, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality."Running Times. The process of tuning a customized action token embedding in the text encoder and the LoRA layer in the text-to-image decoder typically requires approximately 15 minutes using an Nvidia-A100 GPU.Thanks to the rapid development of subject-driven customized text-guided diffusion models for image generation, they achieved an exciting proceeding in generating high-fidelity customized actor images. We combine TwinAct with the existing state-of-the-art subject-driven customized TGDMs [11] to generate images of customized actors performing customized actions. To achieve this, we only need to apply the action base alone to the embedding of the customized action tokens, while the embedding of the customized actor's tokens is obtained as in [11] . The action similarity loss can be applied to the optimization of both the customized action tokens and the customized actor tokens, and our generation results show that the action similarity loss does not affect the learning of the customized actor tokens.In this work, our goal is to generate images of customized actions. There are two limitations to TwinAct. First, TwinAct obtains the embedding of customized action toknes by combining action bases. Although the action base is carefully defined, it is still possible to miss some actions, which limit the expressiveness of TwinAct's output domain. This can be overcome by further expanding the action space by adding more common action phrases. Another limitation comes from the physical defects of TGDMs in generating human bodies, we mitigate this problem by providing some negative prompts like "long body, low-res, bad anatomy, bad hands, missing fingers, extra digit, fewer digits, cropped, worst quality, low quality". Furthermore, there are some works [29, 16, 28] designed to solve the problem. Moreover, TwinAct is not restricted to image generation models; it can also be used for customizing text-guided diffusion models to generate videos or 3D assets. we leave it as future work.We aim to address the challenge of decoupling actions and actors to customize the text-guided diffusion model for few-shot action image generation. A primary ethical concern is the potential misuse of this technology, notably in creating deepfakes, which can result in misinformation, privacy violations, and other harmful outcomes. To mitigate these risks, the establishment of robust ethical guidelines and continuous monitoring is essential.The concern raised here is a common one, not just for our method but across various multi-concept customization techniques. A viable strategy to lessen these risks might be the implementation of tactics akin to those used in anti-dreambooth [26] . This approach involves adding minor noise disturbances to the shared images, thereby hindering the customization process. Furthermore, embedding invisible watermarks in the generated images can serve as a deterrent against misuse and ensure that they are not used without due acknowledgment. • The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper proposes a method to decouple actions and actors for few-shot action image generation in text-guided diffusion models. It has three key steps: building the common action space based on representative action phrases, imitating the customized action within it, and generating customized action images with action similarity loss. Three of the reviewers' concerns are addressed and one reviewer's most concerns are addressed. After reading the paper, reviews, and responses, the AC agrees that this paper has proposed a sound method, and the method has shown interesting improvements.

Reviews: review: paper_summary: paper_summary: Problem: Preserve the consistency of the reference action when generating images with new actors by decoupling the action and actor properly.

Key Idea:
The authors propose a method to disentangle actions from actors in text-guided diffusion models and generate new images that exactly replicate the action pose in the given few-shot inputs. The main contributions are:

-	Generating action phrases using GPT-4 and filtering 832 action phrases know to the model. These are embedded into a common action space with a set of action bases, thus containing no distractions from non-action related information. 

-	The action bases are then combined using a multi-layer perceptron to create new customized actions. Instead of backpropagation to determine optimal coefficients, the authors use CLIP as an action decoder to extract semantic features.

-	An action similarity loss is introduced between the encoded features of the reference and generated action to imitate the high-level action more accurately. 

-	A novel benchmark consisting of 12 actions to compare with existing methods. The authors also provide extensive analysis and ablations.
strength_weakness: strength_weakness: -	The identity of actors is preserved is a strong claim which is not reflected in qualitative evaluation. For example, in Figure 4 row 4: the faces of Batman and Superman look similar. In rows 5 and 6, the subject does not exactly look like Leonardo. In the last row, we see Spiderman in place of Superman.

-	In some cases, the generated image contains a mirrored action of the sample image. For example, in Figure 5 row 2, the generated action is performed with the opposite hand than the sample action in the 3rd and 4th images. 

-	The output generation space is limited, so some complicated customized actions might not be replicable by combining the bases. 

-	It is not clearly shown how changing the value of coefficients for the same action bases impact the generated image.

-	Currently the model has pre-trained knowledge of only 25 actors and cannot be easily adapted to new ones without retraining.
questions: questions: - How is the CLIP score exactly being calculated for SAction and SActor? Is it simply an image-to-text matching score?

- For the human evaluation, what is the degree of knowledge of the 100 users (average or expert)?

- How do you plan to scale this method for a higher number of action variations? Just generating more actions using GPT-4 and adding their embedding to the action space might not be enough.

- Is it possible to show examples of what happens if we vary the value of coefficients in the action bases and how they impact the generated image? For example, a crouching position might be a combination of action bases related to sitting and standing. So, if the coefficient of sitting is 0, the generated image should only contain standing. Again, the image might only contain sitting in the opposite case. The intermediate values of these will show various angles of crouching.

- Providing a TSNE visualization of the action space might also help in clarifying.

- For multiple actors, the authors show examples of images containing only two subjects. Can this be increased to more subjects?

- Also, for multiple actors, how is the role in the image defined? For example, in Figure 4 last row, how will we generate Leonardo carrying Obama or Barbie carrying gorilla?

-- POST rebuttal: the authors have addressed all the raised concerns, increasing my rating.
limitations: limitations: The authors addressed the concerns of limited output possibilities of their common action space and physical defects of TGDMs. They have not discussed any other limitations.
paper_summary: paper_summary: This paper introduced an text-to-action generation framework. In this framework, the authors first abstracted the actions (represented by natural language phrases) with PCA technique into a common action space. Then these high-level action features are fed into a text-to-image transformer network. In order to optimize the whole pipeline, the authors used a CLIP encoder to measure the cosine similarity between generated action images and reference action images.
strength_weakness: strength_weakness: 1. The authors claim that PCA is leveraged for feature compression when establishing common action space. Moreover, the authors borrow this technique from facial feature representations. It is known to us all that PCA is somehow an old-fashioned technique in the machine learning community. The authors are expected to provide more detailed explanations on why they choose this technique.
2. The network architecture is quite simple and lacks technical contributions. To the reviewer, the authors simply apply the off-the-shelf text-to-image framework in their pipeline. In the meantime, other components of the framework seems simple and weak in novelty. 
3. The authors provide a similarity score as the evaluation metric. However, this score is computed by CLIP, which also serves as the action encoder during training. To the reviewer, the comparison seems not so fair. The consistency in IDs and actions should be more carefully evaluated to demonstrate the efficacy of the method.
questions: questions: Please refer to weakness for more details.
limitations: limitations: The authors provide no limitations in the submission.
paper_summary: paper_summary: This paper aims to tackle the few-shot action image generation problem by customizing text-conditioned diffusion models. To decouple action from actor, the proposed method introduces a common textual action feature space to avoid the interference of the actor's visual semantics during action generation. The experimental results demonstrate the proposed method is effective in generating customized actions and preserving actor identities.
strength_weakness: strength_weakness: The overall method is technically sound and the demonstrated results are great, but the reviewer still has some concerns about the method designs,

- The reviewer thinks it is reasonable to use three embeddings (i.e., k=3) to encode each action. However, it is hard to understand why the PCA is separately applied along the token embedding axis (e.g., when k=1, only all the first embeddings from each action are processed by PCA). 
- The collected image action dataset is very small (12 actions only). Why not use some image/video action datasets? And, more importantly, how to make sure the learned/customized model can be generated to unseen action image generation.
- Regarding the action similarity loss, it is hard to ensure that the high-level action semantics will be learned, as there are no explicit constraints.
- minor issue: the high cost of action phrases filtering: the authors generate 1.2K action phrases using GPT-4 and a total of 2x1.2K images are generated using Stable Diffusion XL; The generation is expensive and the manual filtering process is labor-intensive.

/I will improve my rating if the author's response addresses all my concerns well.
questions: questions: Please see the Weaknesses section.
limitations: limitations: The limitations are discussed and there is no potential negative societal impact.
paper_summary: paper_summary: The paper introduces “TwinAct”, a novel method for separating actions from actors in few-shot action image generation using text-guided diffusion models (TGDMs). It creates a “common action space” to focus on actions alone, allowing for precise customization without actor-specific details. The process is streamlined into three main steps: constructing the action space from key phrases, imitating the action within this space, and generating varied, context-adaptive images using an action similarity loss. And the results demonstrate its robustness and versatility for personalized and creative content generation.
strength_weakness: strength_weakness: 1.  In Figure 6, why do all the methods show a performance degradation when two people are involved?
2.  Can the authors explain in detail how the model can be customized for both character and action？
3.  Could the authors provide examples and analyses of where TwinAct fails or performs suboptimally? What steps can be taken to address these failure modes?
4.  Adding something like Figures 8 and 9 from the appendix would make the results more convincing for readers who do not venture to the appendix to search for plots.
questions: questions: Please check the detailed comments in the weaknesses part.
limitations: limitations: NA

Comments: title: Reviewer BXKD
After author response
Reminder: author-reviewer discussion will end soon!
Thanks to Reviewer Q179
post-rebuttal-1
Author-Reviewer Discussion Begins!
comment: Thank you for your thoughtful consideration and for taking the time to engage with our responses. We are grateful for your recommendation for the acceptance of our paper.
We greatly appreciate your openness and are glad that our clarifications were helpful. We are grateful for your support and feedback.
Thank you so much for your understanding and support! We are glad to hear that all your questions have been addressed.
Thank you for clarifying the doubts. All my concerns have been addressed. I will increase my rating.
Thanks for the authors' responses. I found myself learning towards the responses, despite my initial hesitations.
Thank you for your thoughtful comments. We sincerely appreciate your feedback. To address your queries, we have provided further explanations below:


## **Q1 The clustering in t-sne**



We must point out that the purpose of an action base is to combine new actions. This suggests that it is desirable for these action bases to **be orthogonal to each other**.  In other words, **each action base should be unique** in the action space so that more new actions can be combined. 

Therefore, when constructing the action space, we **filter repetitive or similar action phrases** to ensure the independence/diversity of the action base.



For orthogonal bases, t-SNE projects them to different locations (e.g., uniform distributions) so that these basis vectors are separated from each other in low-dimensional space. Thus, these basis vectors can be more flexibly combined to generate new vectors(i.e. new actions).



On the other hand, if the action bases form mixed clusters in the action space, indicating that they are highly correlated with each other, such basis vectors will show poorer combining ability. Thus our clustering is not bad.






## **Q2 The stability of the generated samples**



In addition to the qualitative visualization results, we also present **quantitative experimental results** in the paper, and several experimental results demonstrate the superiority of TwinAct. In the **user study**, users also expressed a clear preference for images generated by TwinAct.



Due to the limitation of the discussion, we cannot add more visualization results, but we will **add the results generated by multiple random seeds** in the final paper to further prove the stability of the generated results. Finally, our dataset and code will be open source.





## **Q3 Identify preservation**



Thank you for your suggestion. We will revise the presentation about identify preservation in the final version. **Decoupling the action from the actors** may be more accurate.





## **Q4 About the rotation augmentation**



We apologize for the loose wording, but the "rotation" we refer to in data augmentation includes both small rotations (**RandomRotation(degrees=30)**) and large rotations (**RandomHorizontalFlip**). 

Specifically, for image augmentation, we use the following techniques: **RandomHorizontalFlip, RandomRotation, RandomAffine, and ColorJitter**, which are randomly selected with a probability of 50%.





## **Q5 About role assignment**



For the carry example, since it has an **explicit active-passive relationship**, we can specify it through language. Specifically, we have a mixture of "A v_carry B" and "B is v_carry A" in our dataset, so the model can learn that **"v_carry" means that the front character is always standing**, and **"is v_carry" means that the front character is always lying down**. So we can use "Leonardo v_carry Obama" or "Obama v_carry Leonardo" to specify who is standing in the back and who is lying in the front



However, we recognize that it is difficult to linguistically constrain the spatial constraints for other examples that do not have an obvious active-passive relationship, such as the actions in rows 3 and 4 in Figure 1 of the main paper, and we tried "in the left" and "in the right," but SD does not work for this type of spatial constraint. We tried "in the left" and "in the right", but SD has a limited understanding of this type of spatial relationship. The generated roles are random.



A possible solution is to **add layout information** such as a bounding box for control, and our method can seamlessly access similar methods such as controlnet. **We will add the generation results of introducing ControlNet[1] in the final version.**

[1] Adding Conditional Control to Text-to-Image Diffusion Models



Thank you for your valuable feedback on our paper, we will carefully consider your suggestions and revise the final manuscript.  In addition, we would sincerely appreciate it if you could possibly raise your score.

Finally, please feel free to contact us if you have any further questions. We look forward to your response.
Thank the reviewers for the rebuttal. 
After a careful reading of the author's response as well as the comments from the other reviewers, my concerns have been clearly addressed. I decide to maintain the recommendation for acceptance of this paper.
Thank you for providing a very detailed response, it addressed most of the concerns. I do have a couple of more questions/ just for clarification,
- The shown t-sne in the rebuttal pdf does not look good, it is not clear, with such good generated samples, why clustering would be so bad? Are the shown samples cherry picked? or I am missing something here?
- I think, it will not be good to claim identify preservation, considering it is not the main focus and also there is no strong evidence.
- It is not clear why rotation augmentation will lead to flipped actions?
- I think, claiming the proposed approach can do complex actions will be a stretch, although the shown samples look reasonable, it is not clear how the spatial ordering, role, etc, can be assigned, even using language, the shown samples seems like random assignment.
Dear reviewers,

The authors have responded to your questions and comments. Please propose your post-rebuttal comments.

Thank Reviewer Q179 for your positive discussions!

Best, Your AC
Dear Reviewer 3NDQ:

Thank you for your feedback. We have carefully addressed your comments in our rebuttal and kindly ask you to review them at your earliest convenience. 

If you have further questions, please do not hesitate to contact us—we are committed to providing thorough responses.

Thank you once again for your time and effort in reviewing our work. We look forward to your response.

Best regards
Thank you sincerely for your thoughtful consideration of our rebuttal and the feedback from the other reviewers. We are pleased to hear that our responses have addressed your concerns and that you are willing to improve your rating.  
Please feel free to reach out if any further questions arise. We truly value your feedback and support throughout this process.
Thanks to the authors for their efforts during the rebuttal. After carefully reading the responses and comments from other reviewers, most of my concerns are well resolved, and I am willing to improve my rating.
Dear authors,

Thanks for your hard work and detailed responses.

Dear reviewers,

the authors have responded to your questions and comments, please read them and further provide feedback: Whether your concerns addressed? After reading all the reviews and responses, are there more questions about the clarity, contributions, results, etc?

Thanks!

Best, Your AC