OneActor: Consistent Subject Generation via Cluster-Conditioned Guidance

Authors: jiahao  wang
caixia  yan
haonan  lin
weizhan  zhang
mengmeng  wang
tieliang  gong
guang  dai
hao  sun
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 16:11:32

Abstract: Text-to-image diffusion models benefit artists with high-quality image generation. Yet their stochastic nature hinders artists from creating consistent images of the same subject. Existing methods try to tackle this challenge and generate consistent content in various ways. However, they either depend on external restricted data or require expensive tuning of the diffusion model. For this issue, we propose a novel one-shot tuning paradigm, termed OneActor. It efficiently performs consistent subject generation solely driven by prompts via a learned semantic guidance to bypass the laborious backbone tuning. We lead the way to formalize the objective of consistent subject generation from a clustering perspective, and thus design a cluster-conditioned model. To mitigate the overfitting challenge shared by one-shot tuning pipelines, we augment the tuning with auxiliary samples and devise two inference strategies: semantic interpolation and cluster guidance. These techniques are later verified to significantly improve the generation quality. Comprehensive experiments show that our method outperforms a variety of baselines with satisfactory subject consistency, superior prompt conformity as well as high image quality. Our method is capable of multi-subject generation and compatible with popular diffusion extensions. Besides, we achieve a $4\times$ faster tuning speed than tuning-based baselines and, if desired, avoid increasing the inference time. Furthermore, our method can be naturally utilized to pre-train a consistent subject generation network from scratch, which will implement this research task into more practical applications. (Project page: https://johnneywang.github.io/OneActor-webpage/)

Main paper: 
 introduction: 
 Diffusion probabilistic models [13, 31, 34] have shown great success in image generation. As a prominent subset of them, text-to-image (T2I) diffusion models [14, 28] significantly improve artistic productivity. Designers can simply describe a subject (e.g. a character, an object, an art style) that they desire and then obtain high-quality images of the subject. However, relying on the random sampling, diffusion models fail to maintain a consistent appearance of the subject across the generated images. Taking Fig. 1 (a) as an example, ordinary diffusion models denoise a random noise from the noise space to a clean latent code guided by the given prompt, where the clean code corresponds to a Figure 1 : For every subject in the latent space, there are identity sub-clusters within the subject base cluster. (a) Given different prompts and initial noises, ordinary diffusion models generate inconsistent images from different identity sub-clusters of the "hobbit" base cluster. (b) While our OneActor, after a quick tuning, provides an extra cluster guidance and thus generates images from the same target sub-cluster that show a consistent identity. Different colors denote different identity sub-clusters. salient image in the image space. As shown, if given 4 different prompts of the same subject "hobbit" and random noises, ordinary models will generate 4 hobbits with different identities. In other words, the generation of the ordinary models lacks subject consistency.The subject consistency is a necessity in practical scenarios such as designing an animation character, advertising a product and drawing a storybook protagonist. As diffusion models prevail, many works try to harness the diffusion models to generate consistent content through the following paths. Personalization [8, 33, 42, 39] learns to represent a new subject from several given images and thus generates images of that. Storybook visualization [23, 30] manages to generate consistent characters throughout a story. However, they require external data to function, either a given image set or a specific storybook dataset. Such a requirement not only complicates practical usage, but also limits the ability to illustrating imaginary, fictional or novel subjects. More recently, consistent subject generation [4] is first proposed to generate consistent images of the same subject only driven by prompts. Thus, artists can easily describe the subject and start their creation, which is a more intuitive manner. Despite the pioneering work, their repetitive tuning process of the backbone model leads to expensive computation cost and possible quality degradation. Later, a tuning-free approach [37] equips the backbone model with handicraft modules to eliminate the tuning process. Yet, the extra modules double the inference time and exhibit inefficiency when generating a large number of images.In fact, diffusion models can generate two consistent images albeit with low probability, demonstrating their inherent potential for consistent subject generation. Motivated by this, we believe all the ordinary model needs is a learned semantic guidance to tame its internal potential. To this end, we propose a novel one-shot tuning paradigm, termed as OneActor. We start from the insight that in the latent space of a pre-trained diffusion model, samples of different subjects form different clusters [4, 41] , i.e. base cluster. In a specific base cluster, samples that share common appearance gather into the same sub-cluster, i.e. identity sub-cluster. In the latent space of Fig. 1 , for example, a "hobbit" base cluster contains four different identity sub-clusters. Ordinary generations spread out into four different sub-clusters, causing subject inconsistency. While in our paradigm of Fig. 1(b) , users first choose one satisfactory image from the generated proposals as the target. Then after a quick tuning, our OneActor quickly learns to find the denoising trajectories towards the target sub-cluster, generating images of the same subject. Throughout the process, we expect to learn in the semantic space and avoid harming the inner capacity of the latent space.To achieve high-quality and efficient generation, we pioneer the cluster-guided formalization of the consistent subject generation task and derive a cluster-based score function. This score function underpins our novel cluster-conditioned paradigm which leverages posterior samples as cluster repre-sentations. To systematically address overfitting, we enhance the tuning with auxiliary samples to robustly align the representation space to the semantic space. We further develop the semantic interpolation and cluster guidance scale strategies for more nuanced and controlled inference. Extensive experiments demonstrate that with superior subject consistency and prompt conformity, our method forms a new Pareto front over the baselines. Our method requires only 3-6 minutes for tuning, which is at least 4× faster than tuning-based pipelines. This efficiency gain is achieved without necessarily increasing the inference time, making our method highly suitable for large-scale image generation tasks. Additionally, our method's inherent flexibility allows consistent multi-subject generation and seamless integration into various workflows like ControlNet [43] .Our main contributions are: (1) We revolutionize the consistent subject generation by pioneering the cluster-guided task formalization, which supersedes the laborious backbone tuning with a learned semantic guidance to perform efficient generation. (2) We introduce a novel cluster-conditioned generation paradigm, termed as OneActor, to pursue more generalized, nuanced and controlled consistent subject generation. It highlights an auxiliary augmentation during tuning and features the semantic interpolation and cluster guidance scale strategies. (3) Extensive experiments demonstrate our method's superior subject consistency, excellent prompt conformity and the 4× faster tuning speed without inference time increase. (4) We first establish the semantic-latent guidance equivalence of T2I models, offering a promising tool for precise generation control. 
preliminaries: 
 Before introducing our method, we first give a brief review of ordinary text-to-image diffusion models. To start with, Gaussian diffusion models [34, 13] assume a forward Markov process that gradually adds noise to normal image x 0 :EQUATIONwhere t ∈ [0, T ], ϵ ∼ N (0, I) and ᾱt are a set of constants. Meanwhile, a denoising network ϵ θ , usually a U-Net [32] , is trained to reverse the forward process by estimating the noise given a corrupted image:L(θ) = E t∈[1,T ],x0,ϵt ∥ϵ t -ϵ θ (x t , t)∥ 2 .(2) Once trained, we can sample images x 0 from a Gaussian noise x t by gradually removing the noise step by step with ϵ θ . To introduce conditional control, classifier-free guidance [14] trains ϵ θ in both unconditional and conditional manners: ϵ θ (x t , t, c ∅ ) and ϵ θ (x t , t, c), where c is the given condition and c ∅ indicates no condition. Images are then sampled by the combination of two types of outputs:EQUATIONwhere s is the guidance scale. For text control, conditions c are generated by a text encoder E t , which projects text prompts p into semantic embeddings: c = E t (p). To accelerate the pipeline, latent diffusion models [31] pre-train an autoencoder to compress images into latent codes: z = E a (x), x = D a (z). Thus, the whole diffusion process can be carried out in the latent space instead of the salient image space. 
method: 
 In our task, given a user-defined description prompt p tar (e.g. a hobbit with robes), a user-preferred image x tar is generated by an ordinary diffusion model ϵ θ and chosen as the target subject. Our goal is to equip the original ϵ θ with a supportive network ϕ, formulating ϵ θ,ϕ . After a quick one-shot tuning of ϕ, our model can generate consistent images of the same subject with any other subject-centric prompt p sub = {p tar , p} (e.g. a hobbit with robes + walking on the street). To accomplish this task, We first give mathematical analysis in Sec. 3.1. Then we construct a cluster-conditioned model and tune it in a generalized manner in Sec. 3.2. During inference, we generate diverse consistent images with the semantic interpolation and cluster guidance in Sec. 3.3.To start with, let's revisit the ordinary generation process. Given N initial noises and the same subject prompt, generations of ϵ θ fail to reach one specific sub-cluster, but spread to a base region S base of The key to consistent subject generation is to guide the denoising trajectories of ϵ θ towards the expected target sub-cluster S tar . From a result-oriented perspective, we expect to increase the probability of generating images of the target sub-cluster S tar and reduce that of the auxiliary sub-clusters S aux i . Thus, if we consider the original diffusion process as a prior distribution p(x), our expected distribution can be denoted as:EQUATIONWe take the negative gradient of the log likelihood to derive:EQUATIONWith the reparameterization trick of [13] , we can further express the scores as the predictions of the denoising network θ from a latent diffusion model [31] in a classifier-free manner [14] :EQUATIONwhere η 1 , η 2 are guidance control factors. If we introduce the concept of score function [35] , Eq. ( 6) can be regarded as a combination of target attraction score and auxiliary exclusion score. This formula, termed as cluster-guided score function, is the core of our method. We will manage to realize it in our subsequent parts. The detailed derivation is shown in Appendix F.In accordance with theoretical analysis of Eq. ( 6), we propose a novel tuning pipeline as shown in Fig. 2 . In general, we incorporate the cluster representations to construct a cluster-conditioned model ϵ θ (z t , t, S) with a supportive network ϕ. We then prepare the data and tune the model with the help of auxiliary samples.Cluster-Conditioned Model. In this model, the supportive network ϕ is designed to process the posterior samples into cluster representations of the semantic space. To specify, for each samplex, ϕ transforms its latent code z and prompt embedding c into a subject-specific vector c ∆ , i.e. c ∆ = ϕ(z, c). For the framework of ϕ, a naive way is to use a feature extractor and a space projector. Since the original U-Net encoder E u is already well-trained to extract features from the latent codes, we use it directly as the extractor. However, the U-Net extractor may cause extra computational burden. To bypass the extractor, we approximate the features of z 0 with that of z 1 :EQUATIONThus, we save the intermediates h of the U-Net encoder during data generation and then directly feed them into the projector for the semantic output: c ∆ = ϕ(h, c) = ϕ(h, E t (p)). This approximation condenses ϕ to only a projector and lowers the computational cost by 30%. The output vector c ∆ represents the semantic direction of the sample's sub-cluster. We then split c into word-wise embeddings c i to locate the base word embedding c b and offset it by:EQUATIONThe modified embedding c ′ b later replaces c b to form c ′ and guides the noise predictions. By now, all the factors (i.e. p, h) that could determine a sub-cluster are involved in the cluster-conditioned model, so we can express the cluster-related terms in Eq. ( 6) as:EQUATIONGeneralized Tuning with Auxiliary Samples. The major challenge of one-shot tuning is overfitting because insufficiency of data may lead to severe bias and limited diversity of generated images.To overcome this challenge, we tune the model with not only target samples, but also auxiliary samples. To elaborate, as shown in Fig. 2 (a), given p tar (e.g. a hobbit with robes) which contains the base word p i (e.g. hobbit), we first input it into the ordinary diffusion model for N base images and the corresponding intermediates to form a base set:X base = {x base i , h base i } N i=1. We randomly choose one image as the target sample x tar , h tar and gather the others to form an auxiliary set:X aux = {x aux i , h aux i } N -1 i=1 .We apply face crop and image flip to target image for an augmented set:X tar = {x tar i , h tar } M i=1 .Then in every tuning step, we randomly select 1 target and K auxiliary samples to form a batch of data: B = {x tar , h tar } ∪ {x aux i , h aux i } K i=1 . Thus, more data contributes to the optimization and the batch normalization can be applied to the projector, resulting in a more generalized projection. To carry out the tuning, in Fig. 2 (b), we first get the latent codes by: z = E a (x), x ∈ B and add noise ϵ t to them. Then we input the noisy latent z t , prompt p and feature h to the cluster-conditioned model with the projector ϕ. The prompt p is a random template filled with the base word (e.g. a portrait of a hobbit). We apply the standard denoising loss for both target and auxiliary samples:EQUATIONEQUATION)Simplifying into Average Condition. For the auxiliary exclusion score in Eq. ( 6), it's laborious to calculate the noise predictions of N -1 auxiliary conditions in every inference step. To simplify, we substitute with an average condition c aver ∆ , which is derived by averaging the semantic vectors of the auxiliary instances:EQUATIONIntuitively, this average condition indicates the center of all the auxiliary sub-clusters and repels the denoising trajectories away. The denoising loss is also used in this condition:EQUATION)Note that the data z aver is randomly chosen from target and auxiliary set. This average condition will act as the empty condition of classifier-free guidance [14] later in Sec. 3.3. During tuning, we only fire the projector from scratch and freeze all other components. As shown in Fig. 2 (c), the light-weight projector comprises a ResNet [11] network, linear and AdaIN [16] layers. The complete tuning objective consists of the above 3 losses weighted by hyper-parameters λ 1 , λ 2 :EQUATION)Figure 3 : The observation of semantic-latent guidance equivalence property. We vary the latent guidance scale on the left side and the semantic interpolation scale on the right, respectively. The semantic and latent manipulations show the same effect, which proves our argument.In the inference stage, traditional tuning-based pipelines exhibit the imbalance between consistency and diversity and fail to extend to multiple subjects generation. To address these challenges, we propose a semantic interpolation strategy and a latent guidance scale strategy to achieve more nuanced and controlled inference. We further develop two variants to perform multiple subjects generation.Proof and Implementation of Semantic Interpolation. Classifier-free guidance [14] proves that the conditional interpolation and extrapolation in the latent space, controlled by a guidance scale, reflect the same linear effect in the image space. We argue that the semantic space of the prompt embbedings also shares this property. To prove it, we carry out a toy experiment on SDXL [28] .Since the empty condition c ∅ is also a semantic embedding obtained by encoding an empty string, we perform interpolation in the semantic space:c ′ = c ∅ + g • (c -c ∅ ).Then we replace c with c ′ in the inference process to generate the images in Fig. 3 . As shown, for the latent interpolation (left), when we increase the guidance scale s in Eq. ( 3) within a proper range, the generated image will be more and more compliant with the prompt. While for the semantic interpolation (right), with g increasing, we can observe the same liner reflection in the image space. This proves that the semantic space also possesses the ability of guidance interpolation. We believe that this is because the semantic and latent space are entangled by the denoising network and thus share some properties in common. With this key insight, we can precisely control the offset guidance effect of c ∆ by replacing Eq. ( 8) with:EQUATION)where v is a controllable semantic scale. We further investigate its effect in Appendix A. 4 . In short, a moderate v leads to an optimal balance of consistency and diversity.Sample with Cluster-Guided Score. Given a subject-centric prompt p sub in the inference stage, we first input its semantic embedding c sub to the tuned projector ϕ * for two representations:EQUATIONEQUATIONThen we approximate the N-1 auxiliary predictions in Eq. ( 6) with one prediction under the average condition:EQUATIONwhere the cluster-conditioned terms are written in the form of Eq. ( 9). With this approximation, Eq. ( 6) can be further transformed into:EQUATIONNow we are ready to sample consistent images with the cluster-guided score of Eq. ( 19). Since η 1 and η 2 play the same role as s in Eq. ( 3), we can manipulate them for an optimal performance. For large-quantity image generation, we can set η 2 = 0 to avoid inference time increase at the expense of a slight quality decrease. Besides, since our method acts by guiding the latent code to a specific cluster step by step, we can loosen the restrictions by applying cluster guidance to certain steps instead of all steps. The effects of these inference strategies are also discussed in Appendix A.4.Generating Multiple Subjects via Two Variants. To extend the generation of single subject to multiple subjects, we develop two different variants. In the first variant, we treat the multi-subject generation as a joint distribution problem and assume that an image of two subjects belongs to the intersection sub-cluster S tar = S tar 1 ∩ S tar 2 . Given multiple target prompts P = {p tar j } L j=1 (e.g. {a hobbit with robes, a white dog}), we combine the prompts together to one prompt p tar (e.g. a hobbit with robes and a white dog) containing multiple base words. We generate multi-subject images with p tar and modify the projector to output multiple semantic vectors C = {c ∆,j } L j=1 . Thus, we can run the tuning and inference process above to generate consistent multi-subject images. However, with all subjects fixed from the beginning, this variant is unable to perform continual subject addition.In the second variant, for each p tar j , we treat it as an individual and perform single-subject tuning for multiple projectors ϕ * j . Then given multi-subject-centric prompt during inference, we offset the embeddings of multiple base words with the semantic outputs C = {c ∆,j } L j=1 of the projectors. To avoid intersecting leakage of semantic information, we constrain the effect of every c ∆,j to the subject's corresponding spatial mask in the denoising process. Since all subjects are independent in this variant, we can continually add new subjects without affecting existing ones. 
experiments 4.1 baselines: 
 To evaluate the performance of OneActor, we compare it with a wide variety of baselines: (1) tuningbased personalization pipelines, Textual Inversion (TI) [8] and DreamBooth (DB) [33] in a LoRA [15] manner; (2) encoder-based personalization pipelines, IP-Adapter (IP) [42] , BLIP-Diffusion (BL) [20] and ELITE (EL) [39] ; (3) consistent subject generation pipelines, TheChosenOne (TCO) [4] and ConsiStory (CS) [37] . We will denote them by the abbreviations for simplification. For personalization pipeline, we first generate one target image with the target prompt. Then we use the target image as the input to perform the personalization. Our method and the tuning-based pipeline are implemented on SDXL [28] . Note that TCO and CS have not offered official open-source codes, so we compare to the results and illustrations from their written materials for fairness. We also construct 3 ablation models: original SDXL, our model excluding L aver and our model excluding L aux , L aver . More implementation details and ablation study are presented in Appendix A.3 and H.Single Subject Generation. We illustrate the visual results of personalization baselines and our OneActor in Fig. 4 . As shown, TI generates high-quality and diverse images but fails to maintain the consistency with the target image. With global tuning, DB shows, albeit a certain degree of consistency, limited prompt conformity. IP maintains consistency in some cases, but is unable to adhere closely to the prompt. BL suffers from inferior quality during generation. In contrast, benefiting from the intricate cluster guidance and the undamaged denoising network, our method demonstrates a balance of superior subject consistency, content diversity as well as prompt conformity. We specifically compare our OneActor with other consistent subject generation pipelines. Fig. 5 shows that all pipelines manage to generate consistent and diverse images. Yet our method preserves more consistent details (e.g. the gray shirt of the man, the green coat of the boy). Additionally, we illustrate more examples in Fig. 6 , which reaffirms the excellent performance of our method.Multiple Subjects Generation. We show double subjects generation comparison between baselines and the two variants of our OneActor in Fig. 7 . Most pipelines, represented by DB and TCO, fail to perform multiple subjects generation. CS extends generation of single subject to multiple subjects. However, CS shows mediocre layout diversity, probably due to their unstable SDSA operations [37] . In comparison, both variants of our method excellently maintain the appearances of two subjects and display superior image diversity. In general, our method is able to generate different types of subjects including humans, animals and objects (or their combinations). Also, our method is versatile for arbitrary styles and expressions ranging from realistic to imaginary. More qualitative results including triple subjects generation and amusing applications are illustrated in Appendix B and C. Metrics. To provide an objective assessment, we carry out comprehensive quantitative experiments. We incorporate multi-objective optimization evaluation and encompass two objectives, identity consistency and prompt similarity, which are popular in personalization tasks [8, 33] . We instruct ChatGPT [25] to generate subject target prompts and templates. For a fair and comprehensive comparison, we adopt two metric settings from the two consistent subject generation pipelines [4, 37] . For the first setting of [4] , the identity consistency is CLIP-I, the normalized cosine similarity between the CLIP [29] image embeddings of generated images and that of target image, while the prompt similarity is CLIP-T, the similarity between the CLIP text embeddings of the inference prompts and the CLIP image embeddings of the corresponding generated images. For the second setting of [37] , the identity consistency is the DreamSim score [7] which focuses on the foreground subject and the prompt similarity is the CLIP-score [12] . Results. In Fig. 8 (a), we can observe that the baselines originally form a Pareto front in the first setting. To specify, DB exhibits highest identity consistency at the cost of lowest prompt similarity and diversity. Quite the contrary, TI and BL sacrifice identity consistency for prompt similarity. TCO and IP form a moderate region with balanced identity consistency and prompt similarity. In contrast, since our method makes full use of the capacity of the original diffusion model instead of harming it, it achieves a significant boost of prompt similarity as well as satisfactory identity consistency. Note that our method Pareto-dominates IP, TI and BL. In Fig. 8 (b), the original Pareto front is determined by IP and three variants of CS in the second setting. Our method pushes the front forward and shows dominance over EL, TI, DB and two variants of CS. In summary, whether in the first or second setting, our OneActor establishes a new Pareto front with a significant margin. These results correspond to the qualitative illustrations above. More experiments are shown in Appendix A. 
related work: 
 Consistent Text-to-Image Generation. A variety of works aim to overcome the randomness challenge of the diffusion models and generate consistent images of the same subject. To start with, personalization, given several user-provided images of one specific subject, aims to generate consistent images of that subject. Some works [8, 38] optimize a semantic token to represent the subject, while others [33, 3] tune the entire diffusion model to learn the distribution of the given subject. Later, [1, 10, 18, 36] find that tuning partial parameters of the diffusion model is effective enough. Apart from the tuning-based methods above, encoder-based methods [9, 5, 2, 42, 39] carefully encode the given subject into a representation and manage to avoid user-tuning. While multi-concept composition [19, 6] goes a step further to display multiple custom subjects on the same generated image via specially designed attention mechanisms. However, depending on external given images, they fail to generate imaginary or novel subjects. Recently, a new consistent subject generation [4] task is proposed, which aims to generate images of one subject given only its descriptive prompts. Though, its tuning of the whole diffusion model is expensive and may degrade the generation quality. Later, a new approach [37] designs handicraft modules to avoid the tuning process and extends consistent generation from single subject to multiple subjects. Yet the extra modules significantly increase the inference time of every image. Besides, based on the localization of the subject, it is incapable of abstract subject generation like artistic style. For these issues, we design a new paradigm from a clustering perspective. Learning only in the semantic space, it requires shorter tuning without necessarily increasing inference time. Compared to the training-free pipelines, our pipeline can be naturally utilized to pre-train a consistent subject generation network from scratch.Semantic Control of Text-to-Image Generation Models. The semantic control starts from classifierfree guidance [14] , which first proposes a text-conditioned denoising network to perform text-to-image generation. It entangles the semantic and latent space so the prompt can guide the denoising trajectory towards the expected destination. Since then, many works manage to manipulate the semantic space to accomplish various tasks. For personalization, they either learn an extra semantic token to guide the latent to the subject cluster [8] or push the token embedding to its core distribution to alleviate the overfitting [45] . For image editing, they calculate a residual semantic embedding to indicate the editing direction [27, 24] . Besides, previous works [40, 21] utilize the semantic interpolation of two conditional embeddings for a mixed visual effect. These works repeatedly confirm that solely manipulating the semantic space is an effective way to harness diffusion models for various goals. Hence, in our novel cluster-guided paradigm, we transform the cluster information into a semantic representation. This representation will later guide the denoising trajectories to the corresponding cluster. 
conclusion: 
 This paper proposes a novel one-shot tuning paradigm, OneActor, for consistent subject generation. Leveraging the derived cluster-based score function, we design a cluster-conditioned pipeline that performs a lightweight semantic search without compromising the denoising backbone. During both tuning and inference, we devise several strategies for better performance and efficiency. Extensive and comprehensive experiments demonstrate that intricate semantic guidance is sufficient to maintain superior subject consistency, as achieved by our method. In addition to excellent image quality, prompt conformity and extensibility, our method significantly improves efficiency, with an average tuning time of just 5 minutes and avoidable inference time increase. Notably, the semantic-latent guidance equivalence property proven in this paper is a potential tool for fine generation control. Furthermore, our method is also feasible to pre-train a consistent subject generation network from scratch, which implements this research task into more practical applications.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We describe the details in Appendix H. We also submitted codes in the supplementary materials.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We report the standard deviation as the error bars in Sec. 4.3 and Appendix A.5. Partial results are not reported with error bars because some baseline works didn't offer codes or error bars in such setting.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: We discuss the societal impacts in Appendix E. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: The models will be protected by the safeguards of the Stable Diffusion, as descibed in Appendix E.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We try to credit all the assets in the reference and list the used packages in Appendix H.2.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: A.1 Motivation Verification Figure 9 : T-SNE illustration of the target sample, auxiliary samples, generated samples with and without our method.Since our method is built on the clusters of the latent space, we conduct an intuitive experiment to verify our motivation. To specify, We run the whole process to generate samples, choose a target and tune the projector. Then we execute the inference twice, with and without the tuned projector. We collect the latent codes of the target sample, auxiliary samples and two rounds of generated samples. We perform t-SNE analysis to them and illustrate the results in Fig. 9 . It clearly demonstrates the cluster structure in the latent space. The target (yellow) and auxiliary (green) samples together form a base cluster (the purple dotted line). Without our cluster guidance, the generated samples (blue) spread out in the base cluster region. In contrast, with our cluster guidance, the generated samples (orange) gather into a sub-cluster (the golden dotted line) centered around the target sample. Notably, some samples without cluster guidance fall into the target sub-cluster, which verifies that the original model has the potential to generate consistent images. In general, the result confirms our motivation and proves an expected function of our cluster guidance. To evaluate the efficiency of consistent subject generation pipelines, we illustrate the generation latency of each pipeline in Fig. 10 . The generation latency consists of the average tuning time and inference time. On a single NVIDIA A100, TheChosenOne [4] needs 20 minutes of tuning. After that, it takes about 8.3 seconds to generate an consistent image. ConsiStory [37] requires no tuning time, but needs about 21.3 seconds to sample one image. While our OneActor needs averagely 5 minutes of tuning. With η 2 = 0, the inference time is 8.3 seconds, which is the same as the original model. When η 2 > 0 and applying cluster guidance to steps 1-20, the inference time will rise to about 11.0 seconds. From a user's perspective, within 20 images, ConsiStory is the fastest, followed by our OneActor and then TheChosenOne. However, considering the application scenarios, the user is very likely to have a demand for large-quantity generation. In this case, our method demonstrates definite advantages. For example, when generating 100 images, ConsiStory takes about 35 minutes while our method only needs about 23 minutes. If we set η 2 = 0, with an affordable compromise of generation details, our method will reduce the total time to about 18 minutes. With the generation number increasing, our advantages will continue to grow. Figure 12 : Illustration of the effect of semantic scale v. We find v = 0.8 is optimal because larger v damages the content diversity while smaller v degrades the character consistency.The key to our tuning method lies in the three objectives in Sec. 3.2. To evaluate the effect of every functional component, we carry out a series of ablation experiments. We also construct three ablation models: original SDXL, our model excluding L aver and our model excluding L aux , L aver . We run the process on these models to obtain character-centric images. For a more intuitive demonstration, we input the same initial noise and prompt to all the models and show the visual results in Fig. 11 . We can observe that, first, without L aux and L aver , the model can maintain consistency to a certain extent. However, it exhibits serious bias issue and damages the diversity of the generated images. Next, including L aux helps stabilize the tuning and significantly improve the image quality and diversity.Yet the identity consistency is not satisfying. Finally, in the full model, all components add up to function and generate consistent, diverse and high-quality images. The ablation experiments further validate our analysis in Sec. 3.2. Semantic Interpolation Scale. We specially add semantic interpolation to enhance the flexibility and controllability during inference in Sec. 3.3. To investigate their influence on the generations, we conduct a parameter analysis of semantic scale v. As shown in Fig. 12 , with v increasing, the generated images progressively become more and more consistent with the target image. However, too large v may distort the semantic embeddings and thus damage the content diversity. Hence, we set v to 0.8 for a moderate performance. It's worthy noting that the results correspond with the experiments in Sec. 3.3, which proves that the interpolation is feasible to the learned ∆c.Cluster Guidance Scale. Since η 1 and η 2 in Eq. ( 19) are two crucial guidance scales for consistent subject generation, we conduct experimental analysis on them. In Fig. 13a , we set η 2 to 0 for an isolated experiment of η 1 . We can observe that when η 1 increases, the image aligns more with prompt descriptions. Hence, η 1 acts like the conditional scale s of the original model in Eq. ( 3). We use η 1 = 7.5 as the standard setting. In Fig. 13 (b), we keep η 1 -η 2 = 7.5 and change η 2 to explore its effect. It turns out that the performance under η 2 = 0 is already satisfying. As η 2 grows, the images exhibit more consistent details such as the facial attributes. These results align with our motivation and designs. It also proves that η 1 and η 2 can be manipulated to achieve our intended effect.Cluster Guidance Steps. During inference, our cluster guidance functions like a navigator. Since the inference is step-wise, we change the inference steps to which we apply the cluster guidance, to explore the influence. Fig. 13(c ) shows one interesting fact of the denoising process that, applying guidance to early steps plays a more significant role. It indicates that the early steps control the primary direction of the trajectory and determine which sub-cluster the trajectory falls in. Its influence is so deterministic that guidance in later steps can hardly change it. Hence, if not specified, we apply cluster guidance to steps 1-20.Subject consistency To further evaluate the performance, we devise automatic quantitative experiments on the baselines and our OneActor. The basic settings remain the same with Sec. 4. We segment the foregrounds (fg) and backgrounds (bg) of the images by SAM [17] . For the metrics, we generally report the cosine similarity between the visual embedings of the target image and the generated images. These visual embeddings are extracted by DINO [26] , CLIP [29] and LPIPS [44] . We focus on three dimensions of the metrics: (1) subject consistency: we calculate the similarity on the image foregrounds to obtain DINO-fg, CLIP-I-fg and LPIPS-fg; (2) prompt similarity: we calculate the CLIP-T-Score [12] on the whole images; (3) background diversity: we calculate the similarity on the image backgrounds to obtain DINO-bg, CLIP-I-bg and LPIPS-bg. We use the standard deviation as the error bar throughout this paper.DINO-fg(↑) CLIP-I-fg(↑) LPIPS-fg(↑)The results are presented in Tabs. 1 and 2. In summary, our method achieves the highest DINO-fg score and the second-highest CLIP-I-fg score, showcasing superior subject consistency. Additionally, our method ranks second across various metrics in prompt similarity and background diversity. These evaluation results highlight our method's comprehensive performance in generating consistent images.Based on a mild guidance rather than a tuned backbone or attention manipulations, our OneActor can be naturally extended to various application scenarios.For spatial location-based pipelines like ConsiStory [37] , the inherent flaw is that they can't deal with abstract subjects ( e.g. art styles, brushstrokes, photo lighting). In contrast, relying on the semantic guidance, our OneActor is able to deal with any type of subject as long as it can be described. For example in Fig. 14 , we perform style transfer using OneActor. To specify, we first generate an image with a target style with the prompt "a picture in a style" and choose style as the base word. Then after tuning, we generate styled images with style-centric prompts (e.g. a fancy car in a style).Since OneActor is able to perform multiple subjects generation, we can extend the ability to story visualization. As shown in Fig. 15 , we first generation three target characters with three descriptive prompts. Then we perform multiple subjects generation with the second variant of our method. After tuning, the pipeline is ready to generate consistent images of one specific character or the combination of the three characters. By describing the plots with the base words, we can obtain a consistent story revolving around the three characters. Note that different from traditional story visualization pipelines, the characters in our story are arbitrary generations by diffusion models with prompts.Avoiding tuning of the denoising backbone, our method is naturally compatible with diffusion-based extensions. Here we show the integrations of StableDiffusion V1.5, StableDiffusion V2.1 and ControlNet XL in Fig. 16 . The illustrations show that our method is robust across different versions of diffusion models. Besides, combined with ControlNet, our method is able to generate consistent images given different pose conditions, which highlights the compatibility of our method. Figure 14 : Style transfer using our OneActor. We generate the target image with a prompt and choose "style" as the base word. Then we perform our method to generate images in a consistent style. Figure 20 : Limitations of OneActor. Our method struggles with generating numerous subjects on the same image and is restricted by the latent properties of diffusion models, causing bias and OOD.Despite the above mentioned strengths, our method exhibits several limitations. One the one hand, though able to generate multiple subjects, our method struggles to simultaneously generate numerous subjects (usually > 4) on one image. As shown in Fig. 20 (ii), in the 5 subjects generation case, the subject tie is neglected. On the other hand, based on a guidance manner, our method is affected by the latent properties of the original diffusion models. First, the latent clusters are entangled, which leads to inevitable bias. For the example in Fig. 20 (iii), the umbrella causes bias against indoor environment, resulting in a mismatch with prompts. Second, as illustrated in Figs. 20(v) and (vi), if the target prompt is a man, its base cluster region will expand so further that it's difficult to precisely navigate to the target sub-cluster, causing inconsistency of some details. Yet, this problem can be solved by using more detailed prompts. Third, a sub-cluster distribution is rigid and incomplete, causing out of distribution (OOD) problem. In Figs. 20(vi) and (vii), swim is not within the scope of the target sub-cluster, which leads to mismatches with prompts. Addressing these challenging issues will be the focus of our future work.The advancements in consistent subject generation present significant societal impacts across various domains. Our work, which enhances subject consistency and efficiency in text-to-image generation, democratizes artistic creation, enabling artists and designers of all skill levels to produce consistent and high-quality visual content. It streamlines creative processes in animation, advertising and publishing, leading to substantial time and cost savings. The ability to maintain visual consistency in media and entertainment may revolutionize animation and storybook illustration.Yet, the rapid development of creative diffusion models may lead to job displacement for artists and designers who rely on traditional methods, creating economic challenges and necessitating reskilling. Besides, ethical concerns arise around the misuse of generated content, such as the creation of deepfakes or misleading images, which can harm public trust and spread misinformation. Additionally, ensuring that generative models are trained on unbiased datasets is crucial to prevent reinforcing harmful stereotypes or biases. For these issues, since our model is based on the Stable Diffusion, we will inherit the safeguards such as the NSFW detection and the usage guideline.Given N initial noises and the same subject prompt, generations of ϵ θ fail to reach one specific sub-cluster, but spread to a base region S base of different sub-clusters. If we choose one sub-cluster as the target S tar and denote the rest as auxiliary sub-clusters S aux i , then S base = S tar ∪ {S aux i } N -1 i=1 . The key to consistent subject generation is to guide ϵ θ towards the expected target sub-cluster S tar . From a result-oriented perspective, we expect to increase the probability of generating images of the target sub-cluster S tar and reduce that of the auxiliary sub-clusters S aux i . Thus, if we consider the original diffusion process as a prior distribution p(x), our expected distribution can be denoted as:EQUATIONWe take the negative gradient of the log likelihood to derive:EQUATIONFrom the score function-based perspective [35] , Eq. ( 21) includes one unconditional score and one target conditional scores and N -1 auxiliary scores. With the reparameterization trick of x in terms of ϵ-prediction [13] : x θ (x t ) = (x t -σ t ϵ θ (x t , t))/α t , where t ∈ [0, T ], ϵ ∼ N (0, I) and α t ,σ t are a set of constants, the unconditional can be estimated by:EQUATIONWe follow [14] to approximate the conditional scores in a classifier-free manner:EQUATIONwhere η is a guidance control factor. Then Eq. ( 21) can be transformed into:EQUATIONwhere η 1 , η 2 are two scaled guidance control factors. Since we use a latent diffusion model in this paper, we transfer the above process from the image space to the latent space to derive:EQUATIONG Pseudo-Code ← -ϕ(h tar , c) {c aux ∆,i } K i=1 ← -{ϕ(h tar , c)} K i=1 c aver ∆ ← -1 K K i=1 c aux ∆,iCalculate loss L(ϕ) using Eqs. ( 10)-( 14) L(ϕ).backward () end endJustification: We submitted codes in the supplementary materials. We will release the codes officially online with detailed instructions after preparations.• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 


Decision: The work received positive scores from reviewers. They listed a number of strengths, including the practical importance of the problem, thorough derivations, and sufficient experiments. There were certain questions and concerns raised, which were adequately addressed during the discussion period. AC agrees with the reviewers, and believes that the problem to keep the subject while changing the prompts is indeed of utmost importance. The decision is to accept the manuscript.

Reviews: review: paper_summary: paper_summary: This paper proposes to formalize the consistent content generation problem from a clustering perspective. By designing a one-shot tuning paradigm with a cluster-conditioned model, the proposed pipeline OneActor can achieve faster tuning while maintaining superior subject consistency. Extensive experiments show that the semantic interpoloation and cluster guidance can contribute to the high quality of consistent subject generation.
strength_weakness: strength_weakness: There is a typo in line 119 for the unconditional manner being repeated twice.
questions: questions: How do you design the projector network? What's the motivation behind the current design?
limitations: limitations: The authors have adequately addressed the limitation and social impact in Appendix D and E, respectively.
paper_summary: paper_summary: This paper proposes OneActor, a one-shot tuning paradigm for consistent subject generation in text-to-image diffusion models, driven solely by prompts and utilizing learned semantic guidance to bypass extensive backbone tuning.

A cluster-conditioned model is introduced to formalize consistent subject generation from a clustering perspective. To mitigate overfitting, the tuning process is augmented with auxiliary samples and employs two inference strategies: semantic interpolation and cluster guidance, significantly enhancing generation quality.

Comprehensive experiments demonstrate that this method outperforms various baselines, providing superior subject consistency, prompt conformity, and high image quality. OneActor supports multi-subject generation, is compatible with popular diffusion extensions, offers faster tuning speeds, and can avoid increased inference time. Additionally, this paper proves that the semantic space of the diffusion model has the same interpolation properties as the latent space, presenting a promising tool for fine generation control.
strength_weakness: strength_weakness: - All demos are in cartoon or artistic styles; no photorealistic results are shown. The objects are limited, raising questions about the fairness of the quantitative results.
- The CLIP score may be insufficient to measure ID similarity.
- Figure 2 is somewhat cluttered. Please focus on highlighting the most important technical contribution.
- No qualitative ablation study is provided.
- Efficiency is a concern. Few quantitative results on efficiency are provided. A processing time of 3-6 minutes may be too long for a text-to-image pipeline.
questions: questions: Please refer to the weakness.
limitations: limitations: Well discussed.
paper_summary: paper_summary: This Study proposes a one-shot tuning paradigm for efficient and consistent subject generation. They introduce two inference strategies to mitigate overfitting, improving generation quality significantly. The semantic space of diffusion mode trained under the proposed method has the same interpolation property as the latent space, offering enhanced control over generation. Comprehensive quantitative and qualitative experiments prove that the method is effective.
strength_weakness: strength_weakness: Some of the related techniques aren't as novel as the author argued. Such as “We are the first to prove that the semantic space of the diffusion model has the same interpolation property as the latent space does.”. As far as I know, it is quite common to do identity mixing[1,2,3,4] in the semantic space in human photo customization. [3,4] also use  two inference strategies including classifier-free guidance and semantic interpolation, which is the same as the method in the paper

1. FaceStudio: Put Your Face Everywhere in Seconds
2. PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding
3. InstantID: Zero-shot Identity-Preserving Generation in Seconds
4. FlashFace: Human Image Personalization with High-fidelity Identity Preservation
questions: questions: 1. The cases provided in the paper are relatively simple. Can the algorithm handle more complex multi-entity scenes, such as those involving more than one person or cat?
2. How does this compare to a seemingly simpler approach [1]?

[1] StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation
limitations: limitations: None
paper_summary: paper_summary: The authors present a generation paradigm called “OneActor” to generate consistent subject in text-to-image generating tasks. The core of this algorithm is called as “cluster-guided score function”, which is based on the concept of score function and created to maintain the consistency of generated images. Additionally, the superiorities of this method for consistency performance and faster tuning are shown quantitatively and qualitatively through various experiments.
strength_weakness: strength_weakness: The overall quality of the paper is quite good, but some problems still exist:

1. The core of this method seems to split the scores of CFG into a target and an auxiliary part for customized generation and maintain consistent subject, which is also derived in detail in Appendix F. The novelty is straightforward, and it would be better to compare the results with the research of multi-concept text-to-image tasks, for example:

Kumari, Nupur, et al. "Multi-concept customization of text-to-image diffusion." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

2. There are some typos. For instance, as shown in line 119, the conditional and unconditional scores have the same notation ϵ_θ (x_t,t,c_∅ ).
	
3. Some experimental results are not good enough. For example, the beard of an old man shown in Figure.4 is not consistent. Additionally, a hobbit generated by DB look better than the “OneActor”.
questions: questions: 1. The authors present a framework to maintain consistency property. How to connect the insights shown in this paper with score-sde? In other words, can we use score-matching to understand clusters?  

2. As shown in Eq.(5), since it is an expectation why is the expression not as follows?
p(x)∙p(S^tar |x)/(∏_(i=1)^(N-1) p(S_i^aux |x)+p(S^tar |x) )

3. Since the average condition indicates the center of all the auxiliary sub-clusters, is there a strategy to make sure a conception like “radius” for the clusters?

4. What are the advantages over methods based on the attention layer of neural networks?
limitations: limitations: 1.	As shown in weakness, the method cannot capture all the details of a given target image.

2.	The subject-centric shown in this paper can be described by a word so that it would be meaningful to further explore more complex and diverse cases.

Comments: title: comment
comment: Thank you so much for your further review and we would like to offer some clarifications regarding your concern.

Our method is a **one-shot tuning** pipeline based on the pre-trained text-to-image generation model StableDiffusionXL (SDXL). The whole procedure involves: 
1) generating an **initial target image using SDXL**; 
2) performing our one-shot tuning pipeline to **produce consistent images according to the target image**. 

In each set of examples shown in our submission and rebuttal, the first image is the target generated by SDXL. For instance, in the 'dog' example in the rebuttal pdf (Fig. 21 middle top), the image of 'playing in a park' is generated by the base model SDXL. Our method then performs one-shot tuning based on this image to produce the subsequent images of 'standing by a fence', 'hopping in a meadow', and 'resting on a sofa'. 

**What our method achieves is to maintain excellent consistency with the target image. The artistic influence you observed in images like the 'dog' and 'coffee cup' is due to the training bias of the SDXL model, rather than a limitation of our method.**  During our experiments, we also observed that SDXL often produces animation-like images even when we use prompts like 'photorealistic', 'a photo of', and 'photography'.

In fact, when SDXL does produce truly photorealistic images like the ***'ring in a packing box', 'wallet on the silk fabric',*** and ***'man reading newspaper'*** in the rebuttal pdf (Fig. 21), **our method effectively maintains this photorealistic style**. According to the rebuttal rules, we are not able to provide additional illustrations, but we do believe the current qualitative examples sufficiently demonstrate both the theoretical contributions and practical values of our method.

We sincerely hope this response helps address your concerns. Thank you again for your thoughtful feedback and your efforts!
I continue to observe a data bias in this method, as most examples are in cartoon or artistic styles. The new examples provided by the author have intensified my concerns. Although the newly added examples feature real objects, they still exhibit a strong artistic influence (coffee, dog). This application is important, especially in real-world scenarios, but I believe the current presentation of the paper remains somewhat skewed. The author's explanation of limited space doesn't convince me. If a method is truly general, it should ideally demonstrate its effectiveness across a wider range of domains. I will keep my score unchanged temporarily, but I think we need to discuss the practical significance of this paper.
Dear Reviewer 7cPE,

We sincerely thank you for your valuable reviews. To each of the question, we have provided detailed respones in the rebuttal stage, which we hope could address your concerns. If you have any further questions or there is any omission in our responses, please feel free to let us know. And we would be extremely grateful if you could consider increasing the rating after reviewing our responses. 

Thanks again for your great efforts!
Dear Reviewer SGyo,

We sincerely thank you for your valuable reviews. To each of the question, we have provided detailed respones in the rebuttal stage, which we hope could address your concerns. If you have any further questions or there is any omission in our responses, please feel free to let us know. And we would be extremely grateful if you could consider increasing the rating after reviewing our responses. 

Thanks again for your great efforts!
Dear Reviewer iFKj,

We greatly appreciate your thoughtful comments and are pleased that you found our responses satisfactory. We will incorporate these important discussions to revise the final manuscript.

Thank you once again for your valuable suggestions and feedback. We really enjoy discussing with you and sincerely appreciate your effors.
I have read all the reviewers' comments and the authors' responses. My concerns have been addressed by the authors. I will not reduce my score.
Dear Reviewer BPQh,

We greatly appreciate your satisfaction with our responses and will make revisions to our work based on your comments in order to further improve the quality of our work. 

Thanks again for your valuable suggestions and comments. We enjoy communicating with you and appreciate your efforts!
I think the responses of the authors have addressed my concerns, and the submitted experimental results reflect the effectiveness of the method. I will not reduce my score.