OneActor: Consistent Subject Generation via Cluster-Conditioned Guidance
Authors: Jiahao Wang
Caixia Yan
Haonan Lin
Weizhan Zhang
Mengmeng Wang
Tieliang Gong
Guang Dai
Hao Sun
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion Model
Image Generation
Score-based Generative Model
Primary area: diffusion_based_models
TL;DR: An efficient text-driven consistent subject generation paradigm via intricate semantic guidance to bypass the laborious backbone tuning.

Abstract: Text-to-image diffusion models benefit artists with high-quality image generation. Yet their stochastic nature hinders artists from creating consistent images of the same subject. Existing methods try to tackle this challenge and generate consistent content in various ways. However, they either depend on external restricted data or require expensive tuning of the diffusion model. For this issue, we propose a novel one-shot tuning paradigm, termed OneActor. It efficiently performs consistent subject generation solely driven by prompts via a learned semantic guidance to bypass the laborious backbone tuning. We lead the way to formalize the objective of consistent subject generation from a clustering perspective, and thus design a cluster-conditioned model. To mitigate the overfitting challenge shared by one-shot tuning pipelines, we augment the tuning with auxiliary samples and devise two inference strategies: semantic interpolation and cluster guidance. These techniques are later verified to significantly improve the generation quality. Comprehensive experiments show that our method outperforms a variety of baselines with satisfactory subject consistency, superior prompt conformity as well as high image quality. Our method is capable of multi-subject generation and compatible with popular diffusion extensions. Besides, we achieve a $4\times$ faster tuning speed than tuning-based baselines and, if desired, avoid increasing the inference time. Furthermore, our method can be naturally utilized to pre-train a consistent subject generation network from scratch, which will implement this research task into more practical applications. (Project page: https://johnneywang.github.io/OneActor-webpage/)

OneActor: Consistent Subject Generation via Cluster-Conditioned Guidance
Jiahao Wang!?, Caixia Yan!.", Haonan Lin', Weizhan Zhang Mengmeng W Tieliang Gong', Guang Dai', Hao Sun'
School of Computer Science and Technology, MOEKLINNS, Xi'an Jiaotong University State Key Laboratory of Communication Content Cognition
CCollege of Computer Science and Technology, Zhejiang University of Technology SSIT AI Lab, State Grid Corporation of China
"China Telecom Artificial Intelligence Technology Co.Ltd
uguisuâstu.xjtu .ew.eedesst..tt.ee.... en
senjjuueedde edu.
Abstract
Text-to-image diffusion models benefit artists with high-quality image genera- tion. Yet their stochastic nature hinders artists from creating consistent images of the same subject. Existing methods try to tackle this challenge and generate consistent content in various ways. However, they either depend external stricted data or require expensive tuning of the diffusion model. For this issue, we propose novel one-shot tuning paradigm, termed OneActor. It efficiently performs consistent subject generation solely driven by prompts via a learned semantic guidance to bypass the laborious backbone tuning. We lead the way to formalize the objective of consistent subject generation from a clustering per- spective, and thus design a cluster-conditioned model. To mitigate the overfitting challenge shared by one-shot tuning pipelines, we augment the tuning with aux- illary samples and devise inference strategies: semantic interpolation and cluster guidance. These techniques are later verified to significantly improve the generation quality. Comprehensive experiments show that our method out- performs variety of baselines with satisfactory subject consistency, superior prompt conformity as well high image quality. Our method capable of multi- subject generation and compatible with popular diffusion extensions. Besides, we achieve a 4x faster tuning speed than tuning-based baselines and, if desired, avoid increasing the inference time. Furthermore, our method can be naturally utilized to pre-train a consistent subject generation network from scratch, which will implement this research task into more practical applications. Project page: https: whnneywang github. io/ /nneeeetor...... -webpggeb.b...e...
Introduction
Diffusion probabilistic models [13, 31, 34] have shown great success image generation. As prominent subset of them, text-to-image (T2I) diffusion models [14, 28] significantly improve artistic productivity. Designers can simply describe subject (e.g. a character, an object, an art style) that they desire and then obtain high-quality images of the subject. However, relying the random sampling, diffusion models fail to maintain a consistent appearance of the subject across the generated example, ordinary diffusion models denoise random noise from the noise space to a clean latent code guided by the given prompt, where the clean code corresponds to a
images. Taking Fig. 1(a)
Corresponding authors.
This work was completed during internship at SGIT Lab, State Grid Corporation China.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Noise space Prompt guided
Lateni space
Image space
denoising trajectory
holdole reless cen breakfesh
holesir rokes
A hobbit with roies ranking
A kobbit with rolees
"Hobbit" base cluster
(a) Ordinary model
Different identiiees
Noise space...
Prompt-guided cluster-guied denoising infjectory
Latent space
Image space
Target sub-cluster
A Hisdoit roees det breakfastt
hobbitt
WiBi rores ranning
hobiit with robes
"Hobbit" base cluster
(b) Tuning 3-6 minutes using our OneActor
Consistent identity
Figure I:
For every subject in the latent space, there are identity sub-clusters within the subject base cluster. (a) Given different prompts and initial noises, ordinary diffusion models generate inconsistent images from different identity sub-clusters of the "hobbit" base cluster. (b) While our OneActor, after a quick tuning, provides an extra cluster guidance and thus generates images from the same target sub-cluster that show a consistent identity. Different colors denote different identity sub-clusters.
salient image in the image space. As shown, if given 4 different prompts of the same subject "hobbit" and random noises, ordinary models will generate 4 hobbits with different identities. In other words, the generation of the ordinary models lacks subject consistency.
The subject consistency is necessity in practical scenarios such as designing an animation character, advertising a product and drawing storybook protagonist. As diffusion models prevail, many works try harness the diffusion models to generate consistent content through the following paths. Personalization [8, 33, 42, 39] learns to represent new subject from several given images and thus generates images of that. Storybook visualization [23, 30] manages generate consistent characters throughout story. However, they require external data to function, either a given image set or specific storybook dataset. Such requirement not only complicates practical usage, but also limits the ability to illustrating imaginary, fictional or novel subjects. More recently, consistent subject generation [4] first proposed to generate consistent images of the same subject only driven by prompts. Thus, artists can easily describe the subject and start their creation, which is more intuitive manner. Despite the pioneering work, their repetitive tuning process of backbone model leads to expensive computation cost and possible quality degradation. Later, tuming- free approach [37] equips the backbone model with handicraft modules climinate the tuning process. Yet, the extra modules double the inference time and exhibit inefficiency when generating large number of images.
In fact, diffusion models can generate two consistent images albeit with low probability, demonstrating their inherent potential for consistent subject generation. Motivated by this, we believe all the ordinary model needs is learned semantic guidance to tame its internal potential. To this end, we propose novel one-shot tuning paradigm, termed as OneActor. We start from the insight that in the latent space pre-trained diffusion model, samples of different subjects form different clusters [4, 41), i.e. base cluster. In a specific base cluster, samples that share common appearance gather into the same sub-cluster, i.c. identity sub-cluster. In the latent space of Fig. 1, for example, "hobbit" base cluster contains four different identity sub-clusters. Ordinary generations spread out into four different sub-clusters, causing subject inconsistency. While in our paradigm of Fig. 1(b), users first choose one satisfactory image from the generated proposals as the target. Then after quick tuning, OneActor quickly learns to find the denoising trajectories towards the target sub-cluster, generating images of the same subject. Throughout the process, expect learn in the semantic space and avoid harming the inner capacity of the latent space.
To achieve high-quality and efficient generation, pioneer the cluster-guided formalization of the consistent subject generation task and derive cluster-based score function. This score function underpins our novel cluster-conditioned paradigm which leverages posterior samples as cluster repre-

---

sentations. To systematically address overfitting, we enhance the tuning with auxiliary samples to robustly align the representation space to the semantic space. We further develop the semantic inter- polation and cluster guidance scale strategies for more nuanced and controlled inference. Extensive experiments demonstrate that with superior subject consistency and prompt conformity, our method forms new Pareto front over the baselines. Our method requires only 3-6 minutes for tuning, which at least 4x faster than duning-based pipelines. This efficiency gain is achieved without necessarily increasing the inference time, making our method highly suitable for large-scale image generation tasks. Additionally, our method's inherent flexibility allows consistent multi-subject generation and seamless integration into various workflows like ControlNet [43].
Our main contributions are: (1) We revolutionize the consistent subject generation by pioneering the cluster-guide guided task formalization, which supersedes the laborious backbone tuning with a learned semantic guidance to perform efficient generation.
(2) We introduce a novel cluster-conditioned generation paradigm, termed as OneActor, to pursue more generalized, nuanced and controlled consistent subject generation. highlights an auxiliary augmentation during tuning and features the semantic interpolation and cluster guidance scale strategies. (3) Extensive experiments demonstrate our method's ' superior subject consistency, excellent prompt conformity and the 4x faster tuning speed without inference time increase. (4) We first establish the semantic-latent guidance equivalence of T2I models, offering a promising tool for precise generation control.
Preliminaries
Before introducing our method, first give brief review of ordinary text-to-image diffusion models. To start with, Gaussian diffusion models [34, 13] assume a forward Markov process that gradually adds noise to normal image Zo!
V + v1 OcC.
(1)
where t E [0, e ~ NN I) and ār are a set of constants. Meanwhile, a denoising network ca, usually a U-Net [32], is trained to reverse the forward process by estimating the noise given corrupted image:
(())= = ExETI,T)..... [lle co (z t)
(2)
Once trained, we can sample images T6 from a Gaussian noise æ, by gradually removing the noise step by step with ep. To introduce conditional control, classifier-free guidance [14] trains ea in both unconditional and conditional manners: é (#t, I, Co) and and ep indicates condition. Images are then sampled by the combination of two types of outputs:
c), where is the given condition
co(#,, 1, 0g) + s (2o L, c) - c0 (c)):
where is the guidance scale. For text control, conditions are generated by a text encoder E,, which projects text prompts p into semantic embeddings: c = (p). To accelerate the pipeline, latent diffusion models [31] pre-train autoencoder to compress images into latent codes: = E, (æ), = (z). Thus, the whole diffusion process can be carried out in the latent space instead of the salient image space.
3 Method
In our task, given user-defined description promptp" pda (e.g. a hobbit with robes), a user-preferred image "!" is generated by ordinary diffusion model eg and chosen the target subject. Our goal is to equip the original ég with a supportive network Φ, formulating 60.0- After a quick one-shot tuning of o. our model prompt n = (p""r .p} (c.g. a hobbit with robes + walking on the street). To accomplish this task, We first give mathematical analysis in Sec. 3.1. Then we construct cluster-conditioned model and tune it in a generalized manner in Sec. 3.2. During inference, we generate diverse consistent images with the semantic interpolation and cluster guidance in Sec. 3.3.
generate consistent images of the same subject with any other subject-centric
3.1 Derivation of Cluster-Guided Score Function
To start with, let's revisit the ordinary generation process. Given initial noises and the same subject prompt, generations of eg fail to reach one specific sub-cluster, but spread to base region Shase of
3

---

Aortaa sfa Disker) Prosger hompliess
glar
Teut
Zigerent FONNON
Auckay Jeguor (xt") You
E.layd
Projector
Approdinatod by
U-Net D2
E
U-Net
Elles)
Lakow fummer (2
Solor
(apon)
No need to implement in practiic
Augrontt
Moslfied pronge endhedalop
Data Preparation
Pypar
Encoder E.
Decader D,
Apantee ResNiet
Laiant
Add Arise
Natry Jasaaee
Prodlesed entrr
(c) Internal Framework of Projector
(b) Tuning Chattr-Contttionee Model
Compute Cisr.
Legner
The overall architecture of our method. (a) We first generate base images and construct the target and auxiliary set. (b) We design data. (c) The projector consists of ResNet network, linear and AdaIN layers. Tuning and freezing weights are denoted by fire and snowflake marks. The items used compute different objectives are outlined in different colors. The unimplemented theoretical models are semi semi-transparent
Figure 2:
conditioend model and tune the projector with batched
different sub-clusters. we choose one sub-cluster as the target S and denote the rest as auxiliary sub-clusters Sqme then Stase = S ( 1 The key to consistent subject generation to guide the denoising trajectories of towards the expected target sub-cluster S"". From result-oriented perspective, we expect to increase the probability of generating images of the target sub-cluster S and reduce that of the auxiliary sub-clusters S.tu Thus, we consider the original diffusion process as prior distribution p()), our expected distribution can be denoted as:
(Sau)
S(Star x3)
p(x))
(4)
((S x)
We take the negative gradient the log likelihood to derive:
N-1
V log)(x) V logp(S
[x)+
+
V log p(S.
x).
(5)
With the reparameterization trick of [13], we can further express the scores as the predictions of the denoising network 0 from latent diffusion model [31] classifier-fres manner [14]:
N
c0(2,, 1) +m [<a(z,,1,5 co co ())
T)2
[eo(z,, S. - c (z(2), O).
(6)
target ottroction score
auxiliaar exclusion
where 71, 72 are guidance control factors. If we introduce the concept of score function [35], Eq. (6) can be regarded as combination of target attraction score and auxiliary exclusion score. This formula, termed as cluster-guided score function, is the core of our method. will manage to realize it in our subsequent parts. The detailed derivation is shown in Appendix F.
3.2 Generalized Tuning of Cluster-Conditione Model
In accordance with theoretical analysis Eq. (6), we propose a novel tuning pipeline as shown in Fig. 2. In general, we incorporate the cluster representations to construct cluster-conditione model c2 S) with a supportive network Φ We then prepare the data and tune the model with the help of auxiliary samples.
Muster--ondditined Model. this model, the supportive network designed process the posterior samples into cluster representations of the semantic space. To specify, for each sample

---

z, Φ transforms its latent code and prompt embedding c into a subject-specific vector ca. i.c. CA 0(z c). For the framework of Φ, a naive way is to use a feature extractor and a space projector. Since the original U-Net encoder F., is already well-trained to extract features from the latent codes, we use directly as the extractor. However, the U-Net extractor may cause extra computational burden. To bypass the extractor, we approximate the features of 20 with that of 21:
h = Eu( c) 2 (zo).
Thus, we save the intermediates of the U-Net encoder during data generation and then directly feed them into the projector for the semantic output: ca ( c) = Φ(h, F2 (p)). This approximation condenses & to only a projector and lowers the computational cost by 30%. The output vector ca represents the semantic direction of the sample's sub-cluster. We then split into word-wise embeddings &; to locate the base word embedding c and offset it by:
df
cb + CA
(8)
The modified embedding of later replaces to form c' and guides the noise predictions. By now, all the factors (i.e. p, h) that could determine a sub-cluster are involved in the cluster-conditioned model, we can express the cluster-related terms in Eq. (6) as:
co(z,,1,5 = 60, ca) = <0,0 Ex(p), Φ(h, (e())))
Generalized Tuning with Auxiliary Samples. The major challenge of one-shot tuning overfitting because insufficiency of data may lead to severe bias and limited diversity of generated images. To overcome this challenge, we tune the model with not only target samples, but also auxiliary samples. To elaborate, as shown in Fig. 2(a), given pd" (e.g. hobbit with robes) which contains the base word pt (e.g. hobbit), we first input it into the ordinary diffusion model for N base images and the corresponding intermediates to form a base set: & {a h base N Ji=1- We randomly choose one image as the target sample x""r, ht" and gather the others to form auxiliary set:
N -1 We apply face crop and image flip to target image for augmented set: Then in every tuning step, we randomly select target and K auxiliary samples to form a batch of data: B = (x" h"") U {xr 11 ] Thus, more data contributes to the optimization and the batch normalization can be applied to the projector, resulting in a more generalized projection. To carry out the tuning, in Fig. 2(b), we first get the latent codes by: Em (z)) x E B and add noise & to them. Then we input the noisy latent 2t, prompt p and feature h to the with the base word (e.g. a portrail of a hobbit). We apply the standard denoising loss for both target and auxiliary samples:
x {arqux h J Ji=1-
uar = -=x!"0 I far M Ji=1-
model with the projector Ø. The prompt p is a random template filled
(g) = "EE(I,TT, ie [lke - (0,0(2t" .t, Ex(p), ((h f0r .3)
(10)
(a)(() EE [lle: e0,0 mi t, ED), ((h oux .))
(11)
Simplifying into Average Condition. For the auxiliary exclusion score in Eq. (6), it's laborious to calculate the noise predictions of N auxiliary conditions in every inference step. To simplify, we substitute with an average condition eaver which is derived by averaging the semantic vectors of the auxiliary instances:
1 K
s(h
aur
c).
(12)
Intuitively, this average condition indicates the center of all the auxiliary sub-clusters and repels the denoising trajectories away. The denoising loss is also used in this condition:
Laver (Φ) = EE [ller (2 (zvver .t, ( 1.]
(13)
Note that the data 2 aver is randomly chosen from target and auxiliary set. This average condition will act as the empty condition of classifier- free guidance [14] later in Sec. 3.3. During tuning, we only fire the projector from scratch and freeze all other components. As shown in Fig. 2(c), the weight projector comprises a ResNet [11] network, linear and AdaIN [16] layers. The complete tuning objective consists of the above losses weighted by parame-as d1;Aa:
C(Φ) C²"(()) + Ca (Φ) + Lar (t).
(14)
5

---

Input prompt: A 2D-ycar-old girl, beautifull, withowt backgrooud.
Perform interpolation in the latent space.
Perform interpolation semantic space.
au
4G Gu 7.
0.2
0d
0.6 Da 1.0
the latent guidance scale, defanlt the semantic interpolation scale definlt 1.0
Figure 3: The observation of semantic-latent guidance equivalence property. We vary the latent guidance scale on the left side and the semantic interpolation scale on the right, respectively. The semantic and latent manipulations show the same effect, which proves our argument.
3.3 Inference Strategies for Versatile Generation
In the inference stage, traditional tuning-based pipelines exhibit the imbalance between consistency and diversity and fail to extend to multiple subjects generation. To address these challenges, propose a semantic interpolation strategy and latent guidance scale strategy to achieve more nuanced and controlled inference. We further develop two variants perform multiple subjects generation.
Proof and Implementation of Semantic Interpolation. Classifier guidance [14] proves that the conditional interpolation and extrapolation in the latent space, controlled by guidance scale, reflect the same linear effect in the image space. We argue that the semantic space of the prompt embbedings also shares this property. To prove it, we carry Since the empty condition cy is also semantic embedding obtained by encoding an empty string, we perform interpolation in the semantic space: e! CD + 9 (c cg). Then replace e with c' in the inference process to generate the images in Fig. 3. As shown, for the latent interpolation (left), when increase the guidance scale in Eq. (3) within a proper range, the generated image will be more and more compliant with the prompt. While for the semantic interpolation (right), with increasing, we can observe the same liner reflection in the image space. This proves that the semantic space also possesses the ability of guidance interpolation. We believe that this is because the semantic and latent space are entangled by the denoising network and thus share some properties common. With this key insight, we can precisely control the offset guidance effect of ca by replacing Eq. (8) with:
toy experiment
SDXL [28].
c, -.- CA.
(15)
where v is a controllable semantic scale. We further investigate its effect in Appendix A.4. In short, moderate v leads an optimal balance of consistency and diversity.
Sample with Cluster-Guided Score. Given a subject-centric prompt Td1 in the inference stage, we first input its semantic embedding bu to the tuned projector & for two representations:
CA
(16)
N-1
Σ
80 (h c.
(17)
Then we approximate the N-1 auxiliary predictions in Eq. (6) with one prediction under the average condition:
N
Σ (00 (Ze, 1, dh ,( c0 Co)) RV c0. (Zi, 1, sub (eaeer - co),
(18)
where the cluster-conditioned terms are written in the form Eq. (9). With this approximation, Eq. (6) can be further transformed into:
c0 <0 ca) m [<e (Z L, mub (gr) (,)
(19)
1/2 (co. (Ze, t, sub cgverr - co cg)].
Now we are ready to sample consistent images with the cluster-guided score of Eq. (19). Since m and play the same role as in Eq. (3), we can manipulate them for an optimal performance. For large-quantity image generation, we can set f/2 = 0 to avoid inference time increase at the expense of slight quality decrease. Besides, since method acts by guiding the latent code to specific

---

cluster step by step, we can loosen the restrictions by applying cluster guidance to certain steps instead of all steps. The effects of these inference strategies are also discussed in Appendix A.4.
Generating Multiple Subjects via Two Variants. To extend the generation of single subject to multiple subjects, develop two different variants. In the first variant, we treat the multi-subject generation as a joint distribution problem and assume that an image of two subjects belongs to the intersection sub-cluster S S n S Given multiple target prompts P {p Jj=1 (e.g. {a hobbit with robes, a white dog )) we combine the prompts together to one prompt p (e.g. hobbit with robes and a white dog) containing multiple base words. We generate multi-subject images with p and modify the projector to output multiple semantic vectors C = و Thus, we can run the tuning and inference process above to generate consistent multi-subject images. However, with all subjects fixed from the beginning. this variant is unable to perform continual subject addition.
In the second variant, for each pl"", we treat it as an individual and perform single-subject tuning for multiple projectors $3. Then given multi-subject-centric prompt during inference, we offset the embeddings of multiple base words with the semantic outputs C [c of the projectors. To avoid intersecting leakage of semantic information, we constrain the effect of every ca., to the subject's 's corresponding spatial mask in the denoising process. Since all subjects are independent in this variant, we can continually add new subjects without affecting existing ones.
4 Experiments
4.1 Baselines
To evaluate the performance OneActor, we compare it with wide variety of baselines: (1) tuning-
based personalization pipelines, Textual Inversion (TT) [8] and DreamBooth (DB) [33] in LoRA
[15] manner: (2) encoder-based personalization pipelines, IP-Adapter (IP) [42], BLIP-Diffusion
(BL) [20] and ELITE (EL) [39]; (3) consistent subject generation pipelines, TheChosenOne (TCO) [4] and ConsiStory (CS) [37]. We will denote them by the abbreviations for simplification. For personalization pipeline, we first generate one target image with the target prompt. Then we use the target image as the input to perform the personalization. Our method and the tuning- based pipeline implemented on SDXL [28]. Note that TCO and CS have not offered official open-source codes, so we compare to the results and illustrations from their written materials for fairness. We also construct 3 ablation models: original SDXL, our model excluding Laver and our model excluding Lawus Laver- More implementation details and ablation study are presented in Appendix A.3 and H.
4.2 Qualitative Illustration
Single Subject Generation. We illustrate the visual results of personalization baselines and our OneActor in Fig. 4. As shown, TI generates high-quality and diverse images but fails to maintain the consistency with the target image. With global tuning. DB shows, albeit a certain degree of consistency, limited prompt conformity. IP maintains consistency in some cases, but is unable to adhere closely to the prompt. BL suffers from inferior quality during generation. In contrast, benefiting from the intricate cluster guidance and the undamaged denoising network, our method demonstrates balance of superior subject consistency, content diversity as well as prompt conformity. We specifically compare our OneActor with other consistent subject generation pipelines. Fig. 5 shows that all pipelines manage to generate consistent and diverse images. Yet our method preserves more consistent details (e.g. the gray shirt of the man, the green coat of the boy). Additionally, we illustrate more examples in Fig. 6, which reaffirms the excellent performance of our method.
Multiple Subjects Generation. We show double subjects generation comparison between baselines and the two variants of our OneActor in Fig. 7. Most pipelines, represented by DB and TCO, fail to perform multiple subjects generation. CS extends generation of single subject to multiple subjects However, CS shows mediocre layout diversity, probably due to their unstable SDSA operations [37]. In comparison, both variants of our method excellently maintain the appearances of two subjects and display superior image diversity. In general, our method is able to generate different types of subjects including humans, animals and objects (or their combinations). Also, our method is versatile for arbitrary styles and expressions ranging from realistic to imaginary. More qualitative results including triple subjects generation and amusing applications are illustrated in Appendix B and C.
7

---

Target
Targei
Target
A halbit withi roless
ebi vue wi black het
rotnning In parlic
cathig insulefast
reading
walkingg
rowning
dotlng
reading
walling
detting browkfire
resddngg book
on stroet
Inol
2e Street
le pverls
a booli
park
OneActor
DB
IP
BL
Figure 4: The qualitative comparison between personalization pipelines and our OneActor. TI lacks consistency, while DB and IP exhibit limited prompt conformity and diversity. BL suffers from poor quality in certain cases. In contrast, our method shows superior consistency, diversity as well as stability. Target prompts and base words are marked blue and red, respectively.
in with resiiaine and Het, Fraefon
50 star's old with hoir
+1 Hite hece
realing Vize
at loldius mricado
holdhe
Time Sahare
Fas
rear +he Vatul
Ieedd
Purk
bede
leadh
miscalo
lace
Omccctor
100
d plg-allpeteengeof homsage bog unth Harkiiaare sight
Watoroaor Ubottartio of parpe hra hamea
Weltee wHlt wishight tes and endrolleer here Mandon
laitiddoo
conder
oetirrrat
moold
OmcActor
2
Figure 5: The qualitative comparison between consistent subject generation methods. Though all methods generate consistent images given different prompts, our OneActor refines more details such the characters' clothes.
4.3 Quantitative Evaluation
Metrics. To provide an objective assessment, we carry out comprehensive quantitative experiments. We incorporate multi-objective optimization evaluation and encompass two objectives, identity consistency and prompt similarity, which are popular in personalization tasks [8, 33J. We instruct ChatGPT [25] to generate subject target prompts and templates. For fair and comprehensive comparison, we adopt two metric settings from the two consistent subject generation pipelines [4, 37]. For the first setting of [4], the identity consistency CLIP-I, the normalized cosine similarity between the CLIP [29] image embeddings of generated images and that of target image, while the prompt similarity CLIP-T, the similarity between the CLIP text embeddings of the inference prompts and the CL.PP image embeddings of the corresponding generated images. For the second setting of [37], DreamSim score [7] which focuses on the foreground subject and the prompt similarity is the CLIP-score [12].
the identity consistency

---

A yaing woman withi long, suavi hair, woaring
dag with Baffy puloo weigging firl
senin Mot witl Stort darty Hulr, searing whire
wolcing on
hoepiin
wildfonodrs
cafe
sillage
syacz Fell
Acarems coffee with hautee pieto realfont: Aersore
silver duresd
black loody plato realark
pitoro roallsti
ploed
the rode
Rower Adid
the sitt faiii
the drt greied
Boot the wattrr
ic the Shewiere
dsounentts
Incidow
woodon vible
perling kox
Vanket
Figure 6: More qualitative results of OneActor. Our method demonstrates excellent subject consis- tency across characters, animals and objects.
JOINT TARGET: Ilsstration of geogles
Insperemoalls patlrrin of fuitiie inowe digle
Paper moal ota paintting
"I TARGET2:
The & wewdphoor
The wife geogles
The wolfis is googfit
Targer
schnosing lnatal masder (SYOc))
wolf
DB
DB
goggles
CIt
CS
walf
TC0
girl
goggles
OneAdton (varinnt
wolf
One One-Actor Actor
girl
goggles
Figure 7: Double subjects generation comparison between baselines and the two variants of our OneActor. Only ConsiStory and our method are able maintain consistency of multiple subjects.
Results. In Fig. 8(a), we can observe that the baselines originally form Pareto front in the first setting. To specify, DB exhibits highest identity consistency at the cost of lowest prompt similarity and diversity. Quite the contrary, TI and BL sacrifice identity consistency for prompt similarity. TCO and IP form a moderate region with balanced identity consistency and prompt similarity. In contrast, since our method makes full use of the capacity of the original diffusion model instead of harming it achieves significant boost of prompt similarity as well as satisfactory identity consistency. Note that our method Pareto-dominates IP, TI and BL. In Fig. 8(b), the original Pareto front is determined by IP and three variants of CS in the second setting. Our method pushes the front forward and shows dominance over EL, TI, DB and two variants of CS. summary, whether the first or second setting, our OneActor establishes a new Pareto front with a significant margin. These results correspond to the qualitative illustrations above. More experiments are shown in Appendix
Related Work
Consistent Text-to- Text-to-Image Generation. A variety of works aim to overcome the randomness challenge of the diffusion models and generate consistent images of the same subject. To start with, personalization, given several user-provided images of one specific subject, aims to generate consistent images of that subject. Some works [8, 38] optimize semantic token to represent the subject, while others [33, 3] tune the entire diffusion model to learn the distribution of the given subject. Later, [1, 10, 18, 36] find that tuning partial parameters of the diffusion model is effective enough. Apart from the tuning- maseeg-based methods above, encoder-base based methods [9,5, 2,2,, 39] carefully encode the given subject into a representation and manage to avoid user- user-tuning. While
composition [19, 6] goes step further to display multiple custom subjects on the same generated image via specially designed attention mechanisms. However, depending they fail to generate imaginary or novel subjects. Recently, a new consistent subject generation [4] task is proposed, which aims to generate images of one subject given only its descriptive prompts.
external given images,

---

0.9
Prompt similarity
Monity consistency
0.85
Parele fromt
Prompt similarity
0.38
DB
Ablatioa model
Daseline model
Original Pareto frant
Ours
Our AneAct
0.8
Ouax was Cewe
(d
EL
TCO
Idemi
IP
Ours
New Pareta Dront
Dream score
0.75
(PELLES)
34 0.5
0.82
Ours Cype
w.t. lax Sere
EL
0.7
0.8
DB
0.78
Baseline model
0.76
BL
Ablation model
Orginal SDXL
Our OneActur
Orginal SDXL
0.14
0.16
0.18
0.2
0.22
0.24
0.26
0.28
0.3
0.32
0.5
0.55
0.5
0.65
CLIP-T
CLIP-score (-)
(a) Quantitative results TheChosenOne setting.
(b) Quantitative results in ConsiStory setting.
Figure 8: The quantitative comparison between baselines and our OneActor in (a) TheChosenOne setting and (b) ConsiStory setting. Either way, our method establishes a new Pareto front with superior subject consistency and prompt conformity.
Though, its tuning of the whole diffusion model is expensive and may degrade the generation quality. Later, a new approach [37] designs handicraft modules avoid the tuning process and extends consistent generation from single subject to multiple subjects. Yer the extra modules significantly increase the inference time of every image. Besides, based the localization of the subject, it is incapable of abstract subject generation like artistic style. For these issues, we design a new paradigm from a clustering perspective. Learning only in the semantic space, requires shorter tuning without necessarily increasing inference time. Compared to the training-free pipelines, our pipeline can be naturally utilized to pre-train a consistent subject generation network from scratch.
Semantic Control of Text-to-Iman Image Generation Models. The semantic control starts from classifier- free guidance [14], which first proposes lext-conditioned denoising network to perform text-to-image generation. It entangles the semantic and latent space so the prompt can guide the denoising trajectory towards the expected destination. Since then, many works manage manipulate the semantic space to accomplish various tasks. For personalization, they either learn an extra semantic token to guide the latent the subject cluster [8] or push the token embedding to its core distribution to alleviate the overfitting [45]. For image editing, they calculate a residual semantic embedding to indicate the editing direction [27, 24]. Besides, previous works [40, 21] utilize the semantic interpolation of two conditional embeddings for mixed visual effect. These works repeatedly confirm that solely manipulating the semantic space is an effective way harness diffusion models for various goals. Hence, in our novel cluster- cluster-guided paradigm, we transform the cluster information into semantic representation. This representation will later guide the denoising trajectories to the corresponding cluster.
Conclusion
This paper proposes a novel one-shot tuning paradigm, OneActor, for consistent subject generation. Leveraging the derived cluster-based score function, we design cunster-oonditioned pipeline that performs lightweight semantic search without compromising the denoising backbone. During both tuning and inference, we devise several strategies for better performance and efficiency. Extensive and comprehensive experiments demonstrate that intricate semantic guidance is sufficient to maintain superior subject consistency, as achieved by our method. In addition to excellent image quality, prompt conformity and extensibility, our method significantly improves efficiency, with an average tuning time of just minutes and avoidable inference time increase. Notably, the semantic-lateni guidance equivalence property proven in this paper is a potential tool for fine generation control. Furthermore, our method is also feasible to pre-train consistent subject generation network from scratch, which implements this research task into more practical applications.
10

---

Acknowledgements
This work supported by the National Natural Science Foundation of China under Grant 62192781, Grant 62172326, Grant 62137002 and Grant 62302384, China Postdoctoral Science Foundation under Grant 2023M742790, Research Project Funded by the State Key Laboratory of Communication Content Cognition under Grant No. A202403, and by the Project of China Knowledge Centre for Engineering Science and Technology.
References
[1] Yuval Alaluf, Elad Richardson, Gal Metzer, and Daniel Cohen-Or. A neural space-time representation for text-to-image personalization, ACM Trunsactions Graphics, 2023.
[2] Moab Arar, Rinon Gal, Yuval Atzmon, Gal Chechik, Daniel Cohen-Or, Ariel Shamir, and Amit H. Bermano. Domain-agnostic tuning-encoder for fast personalization of text-to-image models. In ACM SIGGRAPH Asia, 2023.
[3] Omri Avrahami, Kfir Aberman, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. Break-a- scene: Extracting multiple concepts from a single image. In ACM SIGGRAPH Asia, 2023.
[4] Omri Avrahami, Amir Hertz, Yael Vinker, Moab Arar, Shlomi Fruchter, Ohad Fried, Daniel Cohen-Or, and Dani Lischinski. The chosen one: Consistent characters in text-to-image diffusion models. ACM SIGGRAPHI, 2024.
[5] Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wes Wei Chang, and William W. Cohen. Subject-driven text-to-image generation via apprenticeship learning. In Advances in Neural Information Processing Systems, 2023.
[6] Ganggui Ding, Canyu Zhao, Wen Wang, Zhen Yang, Zide Liu, Hao Chen, and Chunhua Shen. Freecustom: Tuning-free customized image generation for multi-concept composition. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024.
[7] Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning dimensions of human visual similarity using synthetic data. In Advances in Neural Information Processing Systems, 2023.
[8] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In International Conference on Learning Representations, 2023.
[9] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H. Bermano, Gal Chechik, and Daniel Cohen- Or. Encoder-based domain tuning for fast personalization of text-to-image models. ACM Transactions Graphics, 2023.
[10] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris N. Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. In IEEE/CVF International Confer- ence on Computer Vision, 2023.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[12] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. In Conference on Empirical Methods in Natural Language Processing, 2021.
[13] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
[14] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. CORR, 2022.
[15] Edward J. Hu, Yclong Shen, Phillip Wallis. Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference Learning Representations, 2022.
[16] Xun Huang and Serge J. Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. IEEE International Conference on Computer Vision, 2017.
[17] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan- Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. In IEEE/CVF International Conference Computer Vision, 2023.

---

[18] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun Yan Zhu. Multi- concept customization of text-to-image -ixaate- diffusion. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[19] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi- concept customization of text-to-image diffusion. In IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2023.
[20] Dongxu Li, Junnan Li, and Steven C. H. Hoi. Blip-diffusion: Pre-trained subject representation for controllable -it-to--mmae generation and editing. In Advances in Neural Information Processing Systems, 2023.
[21] Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan. Photomaker:
Customizing realistic human photos via stacked ID embedding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[22] Haonan Lin, Mengmeng Wang, Jiahao Wang. Wenbin An, Yan Chen, Liu Yong, Feng Tian, Guang Dai, Jingdong Wang, and Qianying Wang. Schedule your edit: A simple yet effective diffusion noise schedule for image editing. CoRR, 2024.
[23] Adyasha Maharana, Darryl Hannan, and Mohit Bansal. Storydall-e: Adapting pretrained text- to-image transformers for story continuation. In European Conference on Computer Vision, 2022.
[24] Thao Nguyen, Yuheng Li, Utkarsh Ojha, and Yong Jae Lee. Visual instruction inversion: Image editing via image prompting. In Advances in Neural Information Processing Systems, 2023.
[25] OpenAl. Chatgpt. https://chat.opemaa.com// Accessed: 2023-10-15.
[26]
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Yao Huang, Shang-Wen Li, Ishan Misra, Michael G. Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jégou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. CoRR, 2023.
[27] Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li. Jingwan Lu, and Jun- Yan Zhu. Zero-shot image-to-image translation. In ACM SIGGRAPH, 2023.
[28] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: improving latent diffusion models for high-resolution image synthesis. CoRR, 2023.
[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In In- ternational Conference on Machine Learning, Proceedings of Machine Learning Research, 2021.
[30] Tanzila Rahman, Hsin- Ying Lee, Jian Ren, Sergey Tulyakov, Shweta Mahajan, and Leonid Sigal. Make-a-story: Visual memory conditioned consistent story generation. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
[31] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser. and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[32] Olaf Ronneberger, Philipp biomedical image segmentation. In Medical Image Computing and Computer-Assisted Inter- vention, 2015.
Fischer, and Thomas Brox. U-net: Convolutional networks for
[33] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[34] Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, 2015.
12

---

[35] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[36] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to-image personalization. In ACM SIGGRAPH, 2023.
[37] Yoad Tewel, Omri Kaduri, Rinon Gal, Yoni Kasten, Lior Wolf, Gal Chechik, and Yuval Atzmon. Training-free consistent text-to-image generation. ACM Transactions on Graphics, 2024.
[38] Yael Vinker, Andrey Voynov, Daniel Cohen-Or, and Ariel Shamir. Concept decomposition for visual exploration and inspiration. ACM Transactions on Graphics, 2023.
[39] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. ELITE: encoding visual concepts into textual embeddings for customized text-to-image generation. In IEEE/CVF International Conference on Computer Vision, 2023.
[40] Yuxuan Yan, Chi Zhang, Rui Wang, Yichao Zhou, Gege Zhang, Pei Cheng, Gang Yu, and Bin
Fu. Facestudio: Put your face everywhere in seconds. CoRR, 2023.
[41] Mingzhao Yang. Shangchao Su, Bin Li, and Xiangyang Xue. Exploring one-shot semi- supervised federated learning with pre-trained diffusion models. In AAAI Conference on Artificial Intelligence, 2024.
[42] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. CoRR, 2023.
[43] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In IEEE/CVF International Conference on Computer Vision, 2023.
[44] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang- The unreason- able effectiveness of deep features perceptual metric. In IEEE Conference on Computer Vision and Pattern Recognition, 2018
[45] Xulu Zhang, Xiao-Yor Yong Wei, Jinlin Wu, Tianyi Zhang, Zhaoxiang Zhang, Zhen Lei, and Qing
Li. Compositional inversion for stable diffusion models. In AAAI Conference on Artificial Intelligence, 2024.
13

---

Appendix
Table of Contents
A Extra Experiments
15
A.1 Motivation Verification
15
A.2
Efficiency Analysis
15
A.3 Ablation Study
16
A.4 Parameter Analysis
17
A.5 More Quantitative Evaluation
18
B
Applications
19
B.1 Style Transfer
19
B.2 Storybook Generation
19
B.3 Integration with Diffusion Extensions
19
More Qualitative Results
22
D Limitations
25
Societal Impacts
25
F Detailed Derivation of Cluster-Condittoned Score Function
25
Pscudo-Code
26
H Experiment Details
27
H.I Implement Details..
27
H.2 Packages License
27
H.3 Prompts for Evaluation
27
14

---

A Extra Experiments
A.1 Motivation Verification
Since our method is built on the clusters of the latent space, we conduct an intuitive experiment to verify our motivation. To specify, We run the whole process to generate samples, choose target and tune the projector. Then we exe- cute the inference twice, with and without the tuned projector. We collect the latent codes of the target sample, auxiliary samples and two rounds of generated samples. We perform t- SNE analysis to them and illustrate the results in Fig. 9. It clearly demonstrates the cluster structure in the latent space. The target (yellow) and auxiliary (green) samples together form base cluster (the purple dotted line). Without our cluster guidance, the generated samples (blue) spread out in the base cluster region. In con- trast, with our cluster guidance, the generated samples (orange) gather into sub-cluster (the golden dotted line) centered around the target sample. Notably, some samples without cluster guidance fall into the target sub-cluster, which verifies that the original model has the potential generate consistent images. In general, the result confirms our motivation and proves an expected function of our cluster guidance.
Targot sample
Sumples w. cluster guidance
Auxiliary stmples
Bisc clustcr boundary
Samples w.o, claster guidance
Target cluster boundaey
-100
-200
100
200
Figure 9: T-SNE illustration of the target sam- plc, auxiliary samples, generated samples with and without our method.
A.2 Efficiency Analysis
To evaluate the efficiency of consistent subject generation pipelines, we illustrate the generation latency of each pipeline in Fig. 10. The gener- ation latency consists of the average tuning time and inference time. On a single NVIDIA A A100, TheChosenOne [4] needs 20 minutes of tuning. After that, it takes about 8.3 seconds to generate an consistent image. ConsiStory [37] requires no tuning time, but needs about 21.3 seconds to sample one image. While OneActor needs averagely minutes of tuning. With 7)2 = O, the inference time is 8.3 seconds, which is the same as the original model. When 1/2 > and apply- ing cluster guidance steps 1-20, the inference time will rise to about 11.0 seconds. From a user's perspective, within 20 images, ConsiS- tory is the fastest, followed by our OneActor and then TheChosenOne. However, considering the application scenarios, the user is very likely to have demand for aaggequantitty generation. In this case, our method demonstrates definite advantages. For example, when generating 100 images, ConsiStory takes about 35 minutes while our method only needs about 23 minutes. If we set 1/2 = 0. with an affordable compromise of generation details, our method will reduce the total time to about 18 minutes. With the generation number increasing, advantages will continue to grow.
CunsiSiory
TheChesenQue
OseActor (1,20)
30
OneAstor
3nneratinn time (tiin)
255
5
0
10a
10
10
the number of generation images (log scalod)
Figure 10: The latency comparison between con- sistent subject generation baselines and OneActor.
15

---

Orginal SDXI.
feuee
Ours model)
eating
breakfast
a girl with golden hair whitee dress
+ reading book
Target image
reuninng in the park
Figure 11: Illustration the component ablation study. It shows that the proposed objective component significantly contribute the enhanced character consistency and content diversity.
V
Corginal SDXL)
0.2
v
y 0.6
+ holding umbrella
セ= 0.8
セ= 1.0
v= 1.2
ツ= 1.4
girl with goldon hair in white dress
v (orginal SDXL)
v 0.2
v == 0.4
v= 0.6
Target image
+ playing the plano
1.0
y 1.4
Figure 12: Illustration of the effect of semantic scale u. We find v 0.8 is optimal because larger damages the content diversity while smaller degrades the character consistency.
A.3 Ablation Study
The key to our tuning method lies in the three objectives in Sec. 3.2. To evaluate the effect of every functional component, we carry out a series of ablation experiments. We also construct three ablation models: original SDXL, our model excluding Laver and our model excluding Lauxs Luver- We run the process on these models to obtain character-centrie images. For a more intuitive demonstration, we input the same initial noise and prompt to all the models and show the visual results in Fig. 11. We can observe that, first, without L. and Lover, the model can maintain consistency to certain extent. However, it exhibits serious bias issue and damages the diversity of the generated images. Next, including Law helps stabilize the tuning and significantly improve the image quality and diversity.
16

---

Yet the identity consistency is not satisfying. Finally, in the full model, all components add up to function and generate consistent, diverse and quality images. The ablation experiments further validate our analysis in Sec. 3.2.
A.4 Parameter Analysis
122
7/1
1 3,5
M 5.5
7/+
3 9.5
f/= 11.3
playing the plano
holding an umbrella
(a) Effect of 171
7.5
girl golden hair white dross
7/2
712
112
112
1/2
playing the piano
holding aw umbrella
Target image
(b) Effect of 1/2
Cluster guidance steps:
1- 10
6 15
11-20
16 25
21 30
rumming in the park
1
1 10
1 15
-
-5
1 30
(c) Effect of cluster guidance steps
Figure 13: Illustration of parameter analysis of (a) guidance scale 771 (b) guidance scale 7/2 and (c) cluster guidance steps.
Semantic Interpolation Scale. We specially add semantic interpolation to enhance the flexibility and controllability during inference in Sec. 3.3. To investigate their influence on the generations, we conduct a parameter analysis semantic scale v. As shown Fig. 12, with increasing, the generated images progressively become more and more consistent with the target image. However, too large v may distort the semantic embeddings and thus damage the content diversity. Hence, set to 0.8 for moderate performance. It's worthy noting that the results correspond with the experiments in Sec. 3.3, which proves that the interpolation is feasible to the learned Ac.
Cluster Guidance Scale. Since and f22 in Eq. (19) are two crucial guidance scales for consistent subject generation, we conduct experimental analysis on them. In Fig. 13a, we set 1/2 to an isolated experiment of ?/1 We can observe that when 7n increases, the image aligns more with prompt descriptions. Hence, 7/1 acts like conditional seale s of the original model in Eq. (3). We use 771 7.5 as the standard setting. In Fig. 13(b), keep 7/1 722 7.5 and change 7/2 to explore its
17

---

effect. It turns out that the performance under n2 = 0 is already satisfying, As 72 grows, the images exhibit more consistent details such as the facial attributes. These results align with our motivation and designs. It also proves that and 1/2 can be manipulated to achieve our intended effect.
Cluster Guidance Steps. During inference, cluster guidance functions like navigator. Since the inference is step-wise, we change the inference steps to which we apply the cluster guidance, to explore the influence. Fig. 13(c) shows one interesting fact of the denoising process that, applying guidance to early steps plays more significant role. It indicates that the early steps control the primary direction of the trajectory and determine which sub-cluster the trajectory falls in. Its influence is so deterministic that guidance in later steps can hardly change it. Hence, if not specified, we apply cluster guidance steps 1-20.
A.5 More Quantitative Evaluation
Subject consistency
Method
DINO-fg(1)
CLIP-I-fg(t)
LPIPS-fg(T)
Textual Inversion
0,452=+0.130
0.644-0.082
0.322
DreamBooth-LoRA
0.735=+0.078
0.833=0.091
0.296+00004
BLIP- LifP--Diffusio
0.668=00.118
0.750=0.067
0.308
IP-Adapter 0.655=00.099
0.698=0.073
0.289
OneActor(ours))
0.756:10.073
0.821
0.299
Table Subject consistency scores of the baselines and OneActor. The best and second-best results are marked bold and underlined, respectively.
Prompt similarity
Background diversity
Method
CLIP-T-score(t)
DINO-bg(L)
CLIP-I-bg(L)
LPIPS-bg(L)
Textual Inversion
0.654+10.069
0.318=0.074
0.412=0.089
0.467=0.063
DreamBooth-Lood
0.594++0.058
0.441
0.438=0.044
0.482++0.097
Diff-Diffsion
0.547=000774
0.362+00.03
0.446=0.064
0.423:0.077
IP-Adapter
0.446:10.051 0.620++0.038
0.388
0.494 0.403
OneActor(ours)
4.0066
0.426
0.453
Table 2: Prompt similarity and background diversity scores of the baselines and OneActor.
To further evaluate the performance, we devise automatic quantitative experiments on the baselines and our OneActor. The basic settings remain the same with Sec. We segment the foregrounds (fg) and backgrounds (bg) of the images by SAM [17]. For the metrics, we generally report the cosine similarity between the visual embedings of the target image and the generated images. These visual embeddings are extracted by DINO [26], CLIP [29] and LPIPS [44]. We focus on three dimensions of the metrics: (1) subject consistency: we calculate the similarity on the image foregrounds obtain DINO-fg. CLIP-I-fg and LPIPS-fg: (2) prompt similarity: we calculate the CLIP-T-Score [12] on the whole images; (3) background diversity: we caleulate the similarity on the image backgrounds to obtain DINO-bg, CLIP-I-bg and LPIPS-bg. We use the standard deviation as the error bar throughout this paper.
The results are presented in Tabs. 1 and 2. In summary, our method achieves the highest DINO-fg score and the second-highest CLIP-I-fg score, showcasing superior subject consistency. Additionally, our method ranks second across various metrics in prompt similarity and background diversity. These evaluation results highlight our method's 's comprehensive performance generating consistent images.
18

---

B Applications
Based mild guidance rather than a tuned backbone attention manipulations, our OneActor can be naturally extended to various application scenarios.
B.1
Style Transfer
For spatial location-based pipelines like ConsiStory [37], the inherent flaw is that they can't deal with abstract subjects e.g. art styles, brushstrokes, photo lighting). In contrast, relying on the semantic guidance, our OneActor is able deal with any type of subject as long example in Fig. 14, we perform style transfer using OneActor, To specify, we first generate an image with target style with the prompt "a picture in style" and choose style as the base word. Then after tuning, we generate styled images with style-centric prompts (e.g. a fancy car in style).
can be described. For
B.2 Storybook Generation
Since OneActor is able to perform multiple subjects generation, visualization. As shown in Fig. 15, we first generation three target characters with three descriptive prompts. Then we perform multiple subjects generation with the second variant of our method. After tuning, the pipeline is ready to generate consistent images of one specific character or the combination of the three characters. By describing the plots with the base words, we can obtain consistent story revolving around the three characters. Note that different from traditional story visualization pipelines, the characters in our story are arbitrary generations by diffusion models with prompts.
extend the ability to story
B.3 Integration with Diffusion Extensions
Avoiding tuning of the denoising backbone, our method naturally compatible with diffusion-basee extensions. Here we show the integrations of StableDiffusion V1.5, StableDiffusion V2.1 and ControlNet XL in Fig. 16. The illustrations show that our method is robust across different versions of diffusion models. Besides, combined with ControlNet, our method is able generate consistent images given different pose conditions, which highlights the compatibility of our method.
19

---

picture a styll a fanoy
fearless warrior
magnificent chirrh
majesticc Mountain
Target style
Figure 14: Style transfer using OneActor: We generate the target image with a prompt and choose "style" as the base word. Then we perform our method to generate images in a consistent style.
Characterl:
Character2:
Character3:
vicrtek of priccos in bradie dross
sicrtll whith wolf with sharee
slcctiit of whi builey welf
hoppered along
pilly
child sunikkal doep
grandeslyy
One day, an amy from the niihhooring ccuntry imvdedd the polace.
Though hepolas, the prinons hald the boby wolf in sems, trying the hee protect it
At this ortical moment, the varieted mother well led a walf pack and defeased the amy.
The mother will frand the peinonss and exprosee thanks her help for baby wall!
To be continued.
Figure 15: Story Visualization using our OneActor. We generate 3 target characters with prompts and perform the second variant of method for multiple subjects generation. Different target characters are marked with different colors.
20

---

OneActor x Stable Diffusion v1.5
50-year-old maan with curll hair and white shirt
romilig
eatting food
walking on the
reoding book
the park
elever doe, with white furn
OneActor X Stable Diffusion v2.
An old man with glosses and black coat walking on the stroott
remning in
cating food the house
reading book
the park
A cirtta cat with whire furn
OneActor X ControlNet-SDDXL Input different poses
An añien with blue skin, pop art
work hyperrellism style EARE
toowagor skirt's the park
shoos, digital palnting stvlo the leeach
the street
the fun house
Figure 16: Integration results of our OneActor and several diffusion--ased extensions. The results demonstrate the robust adaptability and compatibility of our method.
21

---

C
More Qualitative Results
In order to comprehensively showcase the performance of our method, we provide more generation samples covering single subject, double subjects and triple subjects in Figs. 17, 18 and 19.
+ the Eiffel Tower as background
+ the Statue of Liberty as background
city backgound
cating burget
wearing sunglasses
Target
An alien with blue skin, pop art style
A sophisticated gentleman in tailored suit, charcoal sketch
A cute white baby cat, realistic
A carefree hippic with long flowing hair, tie-dye style
A rugged adventurer with tousled hair and stubble, comic book style
A handsome teenager wearing shirts and canvas shoes, digital painting
Figure 17: Illustrations of single subject generation.
22

---

boy & dog:
Neonpuink style of boy with black spiky hair, with his dog
+ sittiing in the house
walking in the park with a ball
playing
walking in the snow
dog hat:
A realistic photo of a dog
&cating food with a hat
+ sittting the mat, wifh a hat
jumping over puddle
+ walking on the street with hat
wizard & warrior:
An old wizard and a young boy warrior, digital painting
walking in the sHow
+ climbing on top of the mountain
resting inside the desort
cautiously get through forest
explorer & lion:
An watercolor style of an wild explorer with blue coat + drinking water near lake
getting off from his car
meeting big lion
+ giving water to the lion
Figure 18: Illustrations of double subjects generation.
23

---

A digital painting of a happy girl, with a black dog and a white cat
holding together
+ sitting on the street
playing in the garden
+ walking the street
A realistic photo of man in black suit
walking the beach with sunglasses
+ running in rain with sunglasses and an umbrella
+ sitting in café with the umbreall
+ drinking beer in a vílla
watercolor image of a mother with her son and daughter
playing in the garden
+ eating dinner the house
camping besides a láke
walking the gr grassland
A manga page of warrior
+ fighting with a monster barehanded
+ walking on the road with the sword and a shield
holding a sword under the sun
resting wihle hoding the shield
Figure 19: Illustrations of triple subjects generation.
24

---

D Limitations
man, wearing watch, sunglasses, tie, holding an umbrella sitting in the house
A Mndid
walkking on the street
Target
Target
swinmming a pool
missing tie (ii)
bias against house (iii)
inconsistency & swim is OOD
(iv)
(v)
(vi)
Limitations of OneActor. Our method struggles with generating numerous subjects the same image and is restricted by the latent properties of diffusion models, causing bias and OOD.
Figure 20:
Despite the above mentioned strengths, our method exhibits several limitations. One the one hand, though able to generate multiple subjects, our method struggles to simultancously generate numerous subjects (usually >
one image. As shown in Fig. 20(ii,, in the 5 subjects generation case, the subject tie is neglected. On the other hand, based on a guidance manner, our method is affected by the latent properties of the original diffusion models. First, the latent clusters are entangled, which leads to inevitable bias. For the example in Fig. 20(ii), the umbrella causes bias against indoor environment, resulting
mismatch with prompts. Second, illustrated Figs. 20(v) and (vi), the target prompt is man, its base cluster region will expand so further that it's difficult to precisely navigate the target sub-cluster, causing inconsistency of some details. Yet, this problem can be solved by using more detailed prompts. Third, sub-cluster distribution is rigid and incomplete, causing out of distribution (OOD) problem. In Figs. 20(vi) and (vii), swim is not within the scope of the target sub-cluster, which leads to mismatches with prompts. Addressing these challenging issues will be the focus of our future work.
E Societal Impacts
The advancements in consistent subject generation present significant societal impacts across various domains. Our work, which enhances subject consistency and efficiency in text-to-image generation, democratizes artistic creation, enabling artists and designers of all skill levels to produce consistent and high-quality visual content. It streamlines creative processes in animation, advertising and publishing, leading to substantial time and cost savings. The ability to maintain visual consistency in media and entertainment may revolutionize animation and storybook illustration.
Yet, the rapid development of creative diffusion models may lead to job displacement for artists and designers who rely on traditional methods, creating economic challenges and necessitating reskilling. Besides, ethical concerns arise around the misuse of generated content, such the creation of deepfakes or misleading images, which can harm public trust and spread misinformation. Additionally, ensuring that generative models trained unbiased datasets is crucial prevent reinforcing harmful stereotypes or biases. For these issues, since our model is based on the Stable Diffusion, we will inherit the safeguards such as the NSFW detection and the usage guideline.
F
Detailed Derivation of Cluster-oonditioned Score Function
Given N initial noises and the same subject prompt, generations of ce fail to reach one specific sub-cluster, but spread to base region Share different sub-clusters. If we choose one sub-cluster as the target Star and denote the rest as auxiliary sub-clusters S then Sbaze S (Squ)) ) The key to consistent subject generation is to guide eg towards the expected target sub-cluster S From a result-oriented perspective, we expect to increase the probability of generating images the target sub-cluster Star and reduce that of the auxiliary sub-clusters Saic Thus, if we consider the original
25

---

diffusion process prior distribution p(z), our expected distribution can be denoted as:
((Star 2
p(x))
(20)
p(S x
We take the negative gradient of the log likelihood to derive:
Vx log Vn log
x+)
+
Vn log
x))
(21)
From the score function-based perspective [35], Eq. (21) includes one unconditional score and one target conditional scores and N - auxiliary scores. With the reparameterization trick of æ in terms
of e-prediction [13]: (x;,) = (æ, - (2) ())/a, where i E [J, NNO,, I) and ag, are of constants, the unconditional can be estimated by:
c0( 2 V mæ log)(
(22)
follow [14] to approximate the conditional scores in classifier-free manner:
co co t, co(x,, t) not V me log p(S xx),
(23)
where 7) is a guidance control factor. Then Eq. (21) can be transformed into:
N
eg @er t) + (,, c () 7/2
Σ
(ea(xr, L, S,"))) ea (a, t)},
(24)
where 71, 1/2 are two scaled guidance control factors. Since we use a latent diffusion model this paper, we transfer the above process from the image space the latent space to derive:
N-1
e0(z,,t) + m (2 c0(ze, t)] 1/2
(eo (, =0(2,, (25)
G
Pseudo-Code
Algorithm 1: Tuning Process of OneActor
Input :Original diffusion model 0, projector Φb target prompt p"",, base word w, prompt templates P. max tuning step T
Output: Tuned projector e" begin
Input rd" into 0 for dataset {2 h"r U (2qua h ii=1
for s s 1 to T do
Randomly choose E P and fill it with
Obtain the embedding c p using the text encoder of 0
for 20 in 2 muxx do
21 t- Va + vi die, where E (0, ~N(0,) and a are constants
end
Ca t- th .c)
2a Σ Ca.
Calculate loss L(x)) using Eqs. (10)-(14) L(Φ)) backward ()
end
end
26

---

H Experiment Details
H.1 Implement Details
General Settings. We implement our method based on SDXL, which is a common choice by most of the related works. All experiments are finished on single NVIDIA A GPU. All images are generated in 30 steps. During tuning, we generate N = 11 base images. We use K = 3 auxiliary images each batch. The projector consists of 5-layer ResNet, linear layer and AdaIN layer. The weight hyper-paramen parameters λ1 and λ are set to 0.5 and 0.2. We use the default Adam W optimizer with learning rate of 0.0001, weight decay 0.01. We tune with convergence criterion, which takes 3-6 minutes in most cases. During inference, we set the semantic interpolation scale to 0.8 if not specified. We set the cluster guidance scale 7/) and 7/2 to 8.5 and 1.0. We apply cluster guidance to the first 20 inference steps and normal guidance to the last 10 steps.
Multi-Word Subject. Since we apply semantic offset to the base word of the target prompt, multi-word subject (c.g. pencil box) may cause problem. For this case, our method automatically adapts the projector to output multiple offsets for the words, while other parts remain the same.
Subject Mask Extraction. When generating multiple subjects, our method perform mask extraction to the involved subjects. To specify, in each generation step, collect all the cross-attention maps that correspond to the token of each subject. We apply Otsu's 's method to the average the maps for a binary mask for each subject.
H.2
Packages License
SDXL [28] implementation
https://huuggingface oo///aablittya//ttallddddffffffa xl- wal-base11- 1.0.
DreamBooth-LoRA [33] and Textual Inversion [8] implementations on SDXL
https://hugggnggffcc.co//// co/diffusers.
ELITE [39] |39
httpp://githuub.com//axxxett
BLIP-diffusion implemmnntaoooonn at
IP-Adapater [42] [4=Adapater implementation at
https://gittuub.com/reeee www.eeettt-aaaaa------ia--
CLIP [29], DINO [26] and SAM [17] implementation
htt.gs//gtttab..coo//ggggggggg/g
LPIPS [44] implementation
H.3
Prompts for Evaluation
Target prompts, where the base words are denoted bold:
A sophisticated gentleman in tailored suit, charcoal sketch.
A graceful ballerina in flowing tutu, watercolor.
A carefree hippie with long flowing hair, tic-dye style.
A wise elder with a wrinkled face and kind eyes, pastel portrait.
A rugged adventurer with tousled hair and stubble, comic book style.
A trendy hipster with thick-rimmed glasses and beanie, street art.
A spirited athlete in mid-stride, captured in a dynamic sculpture.
A curious child with wide eyes and messy hair, captured in whimsical caricature.
A playful panda with black patches and fluffy ears, cartoon animation.
A majestic tiger with orange stripes and piercing gaze, photorealistic illustration.
27

---

A graceful deer with antlers and doe eyes, minimalist vector art.
A curious raccoon with bandit mask and bushy tail, pastel-colored sticker.
A wise owl with feathers and intense stare, ink brushstroke drawing.
A sleek dolphin with gray skin and playful smile, digital pixel art.
A mischievous monkey with long tail and banana, clay sculpture.
A colorful parrot with vibrant feathers and beak, paper cut-out animation.
A cuddly koala with fuzzy ears and button nose, felt fabric plushie.
A majestic cagle with sharp talons and soaring wings, metallic sculpture.
A noble griffin in a tapestry, its presence commanding attention with intricate detail and grandeur.
A mischievous imp captured in charcoal, its sly grin dancing off the page.
A radiant unicorn in stained glass, casting prisms of light with its ethereal beauty.
An ominous kraken in sculpture, its tentacles frozen in time, threatening the bravest souls.
A resplendent phoenix in metalwork, its fiery wings ablaze with eternal flames.
An enigmatic sphinx in relief, guarding ancient secrets with stoie grace.
A whimsical faun in woodcarving, its playful spirit captured in intricate detail.
A formidable werewolf in digital art, its primal fury palpable contained.
A mystical mermaid in underwater mural, her siren song echoing through ocean depths.
A legendary dragon castle tapestry, its scales shimmering with timeless majesty and power.
A majestic lion in bronze sculpture, its mane flowing with regal power.
A stealthy ninja in ink drawing, moving silently through the shadows.
A serene monk in oil painting, meditating amidst tranquil surroundings.
A futuristic robot in 3D rendering, with sleek lines and glowing eyes.
Subject-centric prompts, where [b] indicates the base word:
bb]. at the beach.
bb. in the jungle.
bb. in the snow.
[b) in the street.
(b). with city in the background.
[b). with mountain in the background.
[b). with the Biffel Tower in the background.
(b. near the Statue of Liberty.
[b], near the Sydney Opera House.
66. floating on top of water.
[). eating a burger.
[b. drinking a beer.
[b]. wearing blue hat.
[b]. wearing sunglasses.
b). playing with ball.
b). as a police officer.
[b. on rooftop overlooking city skyline.
bb. in library surrounded by books.
[b). on farm with animals in the background.
28

---

[6] in a futuristic cityscape.
[b] on skateboard performing tricks.
[b]. in a traditional Japanese garden.
[6] in medieval castle courtyard.
[b]. in science laboratory conducting experiments.
[b]. in yoga studio practicing poses.
[b]. at carnival with colorful rides.
[b]. in a cozy coffee shop sipping a latte.
[6]. in a sports stadium cheering for a team.
[b]. in space station observing Earth.
[6], in submarine exploring the depths of the ocean.
[6]. on film set directing movie scene.
[b]. in an art gallery admiring paintings.
[b) in a theater rehearsing for play.
[b]. at protest holding sign.
(6] in classroom teaching students.
b). in space observatory studying the stars.
b). walking on the street.
(b. eating breakfast in the house.
bb. running in the park.
bb. reading book in the library.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We accurately reflect our contributions and scope in the abstract and introduc-
tion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in Appendix D.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
30

---

Justification:
We provide the theoretical analysis in Appendix F.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We submitted codes in the supplementary materials. We try to describe all the necessary details in Sec. 4 and Appendix H.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
31

---

Answer: [Yes]
Justification: We submitted codes in the supplementary materials. We will release the codes officially online with detailed instructions after preparations.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (uttps://nips.. cc/
http://uuiies//ooeeuuiisssssssss wowooicoollllllooooooooooooiilll for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https: /nips. wwtpppcwcceeeeeecccceccccceeeeee wtwggg/iicccsssai wwteSSubmisiooPooo piioicccc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We describe the details in Appendix H. We also submitted codes in the supplementary materials.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report the standard deviation the error bars in Sec. 4.3 and Appendix A.5. Partial results are not reported with error bars because some baseline works didn' It offer codes error bars in such setting.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
32

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information the compute resources in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : //tprr/pe..p//eeeee oc Spppblic//thicsuuiiiiipe
Answer: [Yes]
Justification:
We try our best to conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the socictal impacts in Appendix E.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
33

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: The models will be protected by the safeguards of the Stable Diffusion, as descibed in Appendix E.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We try to credit all the assets in the reference and list the used packages in Appendix H.2.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twwode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
34

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We will provide documentation when officially release the codes.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "KBJpPp7zpi", "reviewer": "Reviewer_iFKj", "summary": "This paper proposes to formalize the consistent content generation problem from a clustering perspective. By designing a one-shot tuning paradigm with a cluster-conditioned model, the proposed pipeline OneActor can achieve faster tuning while maintaining superior subject consistency. Extensive experiments show that the semantic interpoloation and cluster guidance can contribute to the high quality of consistent subject generation.", "questions": "How do you design the projector network? What's the motivation behind the current design?", "limitations": "The authors have adequately addressed the limitation and social impact in Appendix D and E, respectively.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The formalization of consistent subject generation from a clustering perspective is novel and elegant. \n2. This one-shot tuning paradigm with cluster guidance is creative and could pave a new path of fine control with diffusion models.\n3. Experiments are extensive and solid, showing the superior performance compared to baseline methods.\n4. This paper is well organized and of high quality.", "weaknesses": "There is a typo in line 119 for the unconditional manner being repeated twice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your commendation of our work. It is without doubt a tremendous encouragement for us. For the weakness and question, we will response to each of them individually.\n***\n## Weakness: Typo\nWe will carefully review our paper sentence-by-sentence and try to correct every type error. Thanks for your meticulous review!\n## Question: The design of the projector network\nThis is a very meaningful question which we've discussed a lot ourselves. In our pipeline, the role of the projector network is to transform a token embedding to another adjusted token embedding with respect to the visual feature. Its essence is a transformation inside one modality (semantic) conditioned on another modality (visual). It's similar to the style transfer task, which transfers one image style to another style with respect to the image elements. Inspired by this, we utilize a projector based on AdaIN, which is a popular transformation layer in the style transfer task. Then the ResNet and linear layer are naturally chosen to extract the visual feature and output the $\\beta$ and $\\gamma$ of the AdaIN layer. According to the experiments, our design functions well in the one-shot tuning setting. Furthermore, our method is potential to be utilized to pre-train a consistent subject generation model from scratch. In thus setting, we do believe there might be a more perfect network design for the projector.\n***\nWe believe the rebuttal stage is highly meaningful, and we will add the rebuttal discussion to the new version of our paper as much as possible. If there is any omission in our response or if you have any additional question, please feel free to let us know. Thanks again for your great efforts!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8icjA6SwCN", "reviewer": "Reviewer_SGyo", "summary": "This paper proposes OneActor, a one-shot tuning paradigm for consistent subject generation in text-to-image diffusion models, driven solely by prompts and utilizing learned semantic guidance to bypass extensive backbone tuning.\n\nA cluster-conditioned model is introduced to formalize consistent subject generation from a clustering perspective. To mitigate overfitting, the tuning process is augmented with auxiliary samples and employs two inference strategies: semantic interpolation and cluster guidance, significantly enhancing generation quality.\n\nComprehensive experiments demonstrate that this method outperforms various baselines, providing superior subject consistency, prompt conformity, and high image quality. OneActor supports multi-subject generation, is compatible with popular diffusion extensions, offers faster tuning speeds, and can avoid increased inference time. Additionally, this paper proves that the semantic space of the diffusion model has the same interpolation properties as the latent space, presenting a promising tool for fine generation control.", "questions": "Please refer to the weakness.", "limitations": "Well discussed.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The problem of changing the text prompt while preserving the ID is important.\n- Learning semantic guidance to harness its internal potential is insightful.\n- The auxiliary augmentation in one-shot tuning is novel in text-to-image applications.\n- The code is provided.\n- Quantitative results are clearly presented in Figure 7.\n- According to Figure 6, this paper also considers multiple subjects.", "weaknesses": "- All demos are in cartoon or artistic styles; no photorealistic results are shown. The objects are limited, raising questions about the fairness of the quantitative results.\n- The CLIP score may be insufficient to measure ID similarity.\n- Figure 2 is somewhat cluttered. Please focus on highlighting the most important technical contribution.\n- No qualitative ablation study is provided.\n- Efficiency is a concern. Few quantitative results on efficiency are provided. A processing time of 3-6 minutes may be too long for a text-to-image pipeline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your valuable reviews. For the weakness and questions, we will response to each of them individually.\n***\n## Weakness 1: Limited styles and objects in the main part of the paper\nThe 9-page main part of the submission is limited in space so we placed the most crucial comparisons with the baselines in it and placed more visual results in the Appendix of the submission. Because the baselines (TheChosenOne and ConsiStory) have not yet open-sourced their codes, we can only compare to the results of their written materials, which accidentally limits the variety of the style and central subject in the main part. We showed a little more results of photorealistic styles and objects in the Appendix. B and C. Additionally, we provide more visual results concentrating on objects and photorealistic style in Figure 21 of the global rebuttal pdf. The results demonstrate that our method performs well in this setting.\n## Weakness 2: CLIP score is not insufficient\nCLIP-I-score is the metric that the baselines utilize so we followed and reported it in the main part of our submission. Yet we completely agree that CLIP-I score is insufficient to measure the subject ID. Thus in addition, we provided a more comprehensive quantitative evaluation in the Appendix A.5, where we reported DINO score and LPIPS score as well. Beside these common metrics, we also notice that recently some human-aligned benchmarks are proposed to specially evaluate the image consistency, such as DreamBench++[1]. Due to the time limit, we are not able to perform comprehensive experiments on such benchmarks in the rebuttal stage, but we will try to add such experiments in the new version of our paper. \n\n[1] DreamBench++: A Human-Aligned Benchmark for Personalized Image Generation, arXiv2406.16855\n## Weakness 3: Cluttered figure\nWe will revise the Figure 2 to highlight the projector and the cluster guidance methodology. Thanks for your valuable advice!\n## Weakness 4: Qualitative ablation study\nSince there is no space in the main text, we provided the illustration of component ablation study in the Appendix A.3 and parameter analysis in the Appendix A.4. Please refer to it for further review.\n## Weakness 5: Quantitative efficiency results\nWe provided an efficiency analysis in the Appendix A.2 and add an efficiency statistics in Table 3 of the rebuttal pdf. Though our method takes averagely 5 minutes to tune, it doesn’t increase the inference time as training-free baselines do. From the user’s perspective, when generating <20 images for one subject from scratch, ConsiStory is the fastest, followed by our method and then TheChosenOne. However, considering the application scenarios, the user is very likely to have a demand for large-quantity generation (>50 images). In this case, our method demonstrates definite advantages. For example, when generating 100 images, ConsiStory takes about 35 minutes while our method only needs about 23 minutes in total. If we set $\\eta_2=0$, with an affordable compromise of generation details, our method will reduce the total time to about 18 minutes. With the generation number increasing, our advantages will continue to grow.\n***\nWe believe the rebuttal stage is highly meaningful, and we will add the rebuttal discussion to the new version of our paper as much as possible. If there is any omission in our response or if you have any additional question, please feel free to let us know. Thanks again for your great efforts!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6sFc4G33Mx", "reviewer": "Reviewer_7cPE", "summary": "This Study proposes a one-shot tuning paradigm for efficient and consistent subject generation. They introduce two inference strategies to mitigate overfitting, improving generation quality significantly. The semantic space of diffusion mode trained under the proposed method has the same interpolation property as the latent space, offering enhanced control over generation. Comprehensive quantitative and qualitative experiments prove that the method is effective.", "questions": "1. The cases provided in the paper are relatively simple. Can the algorithm handle more complex multi-entity scenes, such as those involving more than one person or cat?\n2. How does this compare to a seemingly simpler approach [1]?\n\n[1] StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The motivation and the methods proposed in the entire article are justifiable. The illustrations are incredibly exquisite, and the presented results effectively prove the efficacy of their method\n2. For practitioners in the field of AIGC, I indeed believe it holds practical value.", "weaknesses": "Some of the related techniques aren't as novel as the author argued. Such as “We are the first to prove that the semantic space of the diffusion model has the same interpolation property as the latent space does.”. As far as I know, it is quite common to do identity mixing[1,2,3,4] in the semantic space in human photo customization. [3,4] also use  two inference strategies including classifier-free guidance and semantic interpolation, which is the same as the method in the paper\n\n1. FaceStudio: Put Your Face Everywhere in Seconds\n2. PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding\n3. InstantID: Zero-shot Identity-Preserving Generation in Seconds\n4. FlashFace: Human Image Personalization with High-fidelity Identity Preservation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your valuable reviews. For the weakness and questions, we will response to each of them individually.\n***\n## Weakness: Novelty of semantic interpolation\nWe believe our discovery is fundamentally different from previous works. Ordinary diffusion process can be denoted as $p(\\boldsymbol{z}\\mid \\boldsymbol{c},s)$, where $\\boldsymbol{c}$ is the semantic condition and $s$ is the weight of classifier-free guidance score. Previous works [1,2,3,4] prove that the semantic interpolation $\\boldsymbol c'=\\boldsymbol c_1+\\alpha\\cdot(\\boldsymbol c_2-\\boldsymbol c_1)$ between two conditional embeddings $\\boldsymbol c_1$ and $\\boldsymbol c_2$ shows a mixed visual effect and a gradual visual change, which can be written as: $p(\\boldsymbol z\\mid\\boldsymbol c’,s)\\approx p(\\boldsymbol z\\mid\\boldsymbol c_1,s)\\cdot p(\\boldsymbol z\\mid\\boldsymbol c_2,s)$. This discovery is utilized to mix different input conditions for a fancy hybrid output image. By contrast, our contribution lies in proving that the semantic interpolation $\\boldsymbol c'=\\boldsymbol c_\\emptyset+\\alpha\\cdot(\\boldsymbol c_1-\\boldsymbol c_\\emptyset)$ between the unconditional embedding $\\boldsymbol c_\\emptyset$ and conditional embedding $\\boldsymbol c_1$ has the same effect with the guidance interpolation $s'=0+\\beta\\cdot(s_1-0)$ between no guidance $s=0$ and standard guidance scale $s=s_1$. The above conclusion can be written as: $p(\\boldsymbol z\\mid \\boldsymbol c',s_1)\\approx p(\\boldsymbol z\\mid \\boldsymbol c_1,s')$. It indicates that semantic manipulation has the same effect with the sampling guidance strength, which we believe is essentially different from the conclusion of previous works. Yet we apologize for the misinformation caused by our inappropriate wording and we will polish our phrasing to be more precise and objective. We will also add the above-mentioned works to the Related Work section in the new version of our paper. Thanks so much for your nuanced review!\n\n[1] FaceStudio: Put Your Face Everywhere in Seconds\n\n[2] PhotoMaker: Customizing Realistic Human Photos via Stacked ID Embedding\n\n[3] InstantID: Zero-shot Identity-Preserving Generation in Seconds\n\n[4] FlashFace: Human Image Personalization with High-fidelity Identity Preservation\n## Question 1: Multi-entity scenes\nWe provided illustrations of double subjects generation and triple subjects generation in Figures 17 and 18 in the Appendix C of the submission. The results show that our method is able to deal with the complex 2-entity and 3-entity scenarios. The storybook generation in the Appendix B.2 also proves that our method is capable of complex sequential generation. Yet when dealing with more numerous subjects (usually >4), as discussed in the Appendix D, some subjects might be neglected. We also observe this defect in the original SDXL model and addressing this foundational limitation will be the focus of our future work.\n## Question 2: Comparison with StoryDiffusion\nWe carry out qualitative comparison with the contemporaneous work StoryDiffusion [5] in Figures 20 and 21 in the global rebuttal pdf. Generally, we think our method performs better in terms of subject consistency and background diversity. In a broader context, compared with the training-free pipelines like StoryDiffusion, our tuning-based pipeline has the following advantages: \n- Our method can be naturally utilized to pre-train a consistent subject generation network from scratch, which implements this research task into more practical applications.\n- Based on the stable tuning process of an auxiliary network rather than the U-Net backbone, our method demonstrates more robust performance across different settings. By contrast, the attention manipulation to the U-Net backbone utilized by training-free baselines exhibits occasional degradations of the image quality (see the first and fourth images of the ‘*man*’ case generated by StoryDiffusion in the Figure 21 of the global rebuttal pdf).\n- Though our method takes averagely 5 minutes to tune, it doesn’t increase the inference time as training-free baselines do. Thus, as the generation number increases, our method outperforms the training-free methods in terms of total generation time (see the Table 3 in the global rebuttal pdf).\n\n[5] StoryDiffusion: Consistent Self-Attention for Long-Range Image and Video Generation\n***\nWe believe the rebuttal stage is highly meaningful, and we will add the rebuttal discussion to the new version of our paper as much as possible. If there is any omission in our response or if you have any additional question, please feel free to let us know. Thanks again for your great efforts!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XlQGkuTyio", "reviewer": "Reviewer_BPQh", "summary": "The authors present a generation paradigm called “OneActor” to generate consistent subject in text-to-image generating tasks. The core of this algorithm is called as “cluster-guided score function”, which is based on the concept of score function and created to maintain the consistency of generated images. Additionally, the superiorities of this method for consistency performance and faster tuning are shown quantitatively and qualitatively through various experiments.", "questions": "1. The authors present a framework to maintain consistency property. How to connect the insights shown in this paper with score-sde? In other words, can we use score-matching to understand clusters?  \n\n2. As shown in Eq.(5), since it is an expectation why is the expression not as follows?\np(x)∙p(S^tar |x)/(∏_(i=1)^(N-1) p(S_i^aux |x)+p(S^tar |x) )\n\n3. Since the average condition indicates the center of all the auxiliary sub-clusters, is there a strategy to make sure a conception like “radius” for the clusters?\n\n4. What are the advantages over methods based on the attention layer of neural networks?", "limitations": "1.\tAs shown in weakness, the method cannot capture all the details of a given target image.\n\n2.\tThe subject-centric shown in this paper can be described by a word so that it would be meaningful to further explore more complex and diverse cases.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe authors creatively present an insight that samples of different subjects form different clusters, and analyze it in detail, which is the inspiration of their method.\n\n2.\tThe derivation of formulas is in complete detail without errors and codes are committed.\n\n3.\tThe experiments are sufficient and solidly conducted, including both qualitative and quantitative comparisons. The ablation studies are sensible and include user studies as well.", "weaknesses": "The overall quality of the paper is quite good, but some problems still exist:\n\n1. The core of this method seems to split the scores of CFG into a target and an auxiliary part for customized generation and maintain consistent subject, which is also derived in detail in Appendix F. The novelty is straightforward, and it would be better to compare the results with the research of multi-concept text-to-image tasks, for example:\n\nKumari, Nupur, et al. \"Multi-concept customization of text-to-image diffusion.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n2. There are some typos. For instance, as shown in line 119, the conditional and unconditional scores have the same notation ϵ_θ (x_t,t,c_∅ ).\n\t\n3. Some experimental results are not good enough. For example, the beard of an old man shown in Figure.4 is not consistent. Additionally, a hobbit generated by DB look better than the “OneActor”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your insightful review of our work. For the weakness and questions, we will response to each of them individually.\n***\n## Weakness 1: Comparison with multi-concept customization pipeline\nWe provide a qualitative comparison with FreeCustom[1], the current state-of-the-art multi-concept customization method, in the Figure 22 of the global rebuttal pdf. Note that FreeCustom is officially implemented on StableDiffusion V1.5 and thus exhibits averagely lower image quality. In terms of subject consistency and prompt conformity, which are backbone-irrelevant, our method still outperforms FreeCustom. This reaffirms the effectiveness of the proposed cluster guidance method. Due to the time limit, we are not able to perform more comprehensive experiments of multi-concept customization methods. We will add these methods to the Related Work section and add more experiments in the new version of our paper.\n\n[1] FreeCustom: Tuning-Free Customized Image Generation for Multi-Concept Composition, CVPR 2024\n## Weakness 2: Typo\nWe will carefully review our paper sentence-by-sentence and try to correct every type error. Thanks for your meticulous review!\n## Weakness 3: Imperfect consistency\nFor the \"hobbit\" case comparison between DB and our method, solely from the perspective of subject consistency, DB indeed beats our method. Yet we believe our method achieves a more balanced performance of subject consistency, background and layout diversity. We believe this balanced performance is more meaningful in the consistent subject generation task. Due to the non-ideal distribution of the clusters, there will be some inconsistent minor details such as the beard or clothing decorations. Addressing this challenge will be the focus of our future work.\n## Question 1: Connection with score-SDE\nIn our opinion, the basic logic of our method is the same as that of the score-SDE. The methodology of score-SDE [2] is to transform the log-likelihood into the predicted score of the denoising network in a score-matching manner. We are inspired and employ the transformation in the derivation from Equation (5) to Equation (6) in our submission. The training of $\\epsilon_{\\boldsymbol \\theta,\\boldsymbol \\phi^*}(\\boldsymbol z_t,t,\\boldsymbol c^{sub},c_\\Delta^{tar})$ and $\\epsilon_{\\boldsymbol \\theta,\\boldsymbol \\phi^*}(\\boldsymbol z_t,t,\\boldsymbol c^{sub},c_\\Delta^{aver})$ are essentially two score-matching processes of the target sub-cluster score $\\nabla_{\\boldsymbol{x}}\\log p(\\mathcal{S}^{tar}\\mid\\boldsymbol{x})$ and the auxiliary sub-cluster score $\\sum_{i=1}^{N-1}\\nabla_{\\boldsymbol{x}}\\log p(\\mathcal{S}_i^{aux}\\mid\\boldsymbol{x})$.\n\n[2] Score-based generative modeling through stochastic differential equations\n## Question 2: Formula expression\nThere are many different ways to express the expectation of the task including $p(\\boldsymbol{x})\\cdot\\frac{p(\\mathcal{S}^{tar}\\mid\\boldsymbol{x})}{\\prod_{i=1}^{N-1}p(\\mathcal{S}_i^{aux}\\mid\\boldsymbol{x})+p(\\mathcal{S}^{tar}\\mid\\boldsymbol{x})}$. We eventually adopted the simplified Equation (5) for the following reasons:\n- Retaining only the multiplication and division terms simplifies the subsequent formula transformation.\n- $p(\\boldsymbol{x})\\cdot\\frac{p(\\mathcal{S}^{tar}\\mid\\boldsymbol{x})}{\\prod_{i=1}^{N-1}p(\\mathcal{S}_i^{aux}\\mid\\boldsymbol{x})+p(\\mathcal{S}^{tar}\\mid\\boldsymbol{x})}$ can be transformed into \n\n   $p(\\boldsymbol x)\\cdot\\frac{1}{\\prod_{i=1}^{N-1}p(\\mathcal{S}_i^{aux}\\mid\\boldsymbol x)/p(\\mathcal{S}^{tar}\\mid\\boldsymbol x)+1}$. \n\n   Thus, the core expectation is also to increase $\\frac{p(\\mathcal{S}^{tar}\\mid\\boldsymbol x)}{\\prod_{i=1}^{N-1}p(\\mathcal{S}_i^{aux}\\mid\\boldsymbol x)}$ as the Equation (5) does.\n- During implementation, the target and auxiliary samples all contribute to the average condition, which completes the center of the whole cluster. So the simplification is more like a theoretical trick.\n## Question 3: Radius concept\nWe believe there exists the concept of *radius* in the latent space, yet it’s hard to precisely determine the *radius*. Because if we consider the $(\\boldsymbol z_t,t)$ as a whole, the latent space is complicated and might not be a Euclidean space. The *radius* might appear to be a geometric manifold that extends continuously over $t$. It will be a interesting yet challenging future topic to develop a strategy to determine the *radius*.\n## Question 4: Advantages over attention manipulation methods\nCompared with the training-free pipelines based on attention manipulation, our tuning-based pipeline has the following advantages: \n- Our method can be naturally utilized to pre-train a consistent subject generation network from scratch, which implements this research task into more practical applications.\n- Based on the stable tuning process of an auxiliary network rather than the U-Net backbone, our method demonstrates more robust performance across different settings. By contrast, the attention manipulation to the U-Net backbone utilized by training-free baselines exhibits occasional degradations of the image quality (see the first and fourth images of the ‘*man*’ case generated by StoryDiffusion in the Figure 21 of the global rebuttal pdf; also see the first image of the ‘*girl-cat*’ case generated by FreeCustom in the Figure 22).\n- Though our method takes averagely 5 minutes to tune, it doesn’t increase the inference time as training-free baselines do. Thus, as the generation number increases, our method outperforms the training-free methods in terms of total generation time (see the Table 3 in the global rebuttal pdf).\n***\nWe believe the rebuttal stage is highly meaningful, and we will add the rebuttal discussion to the new version of our paper as much as possible. If there is any omission in our response or if you have any additional question, please feel free to let us know. Thanks again for your great efforts!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers, ACs, SACs and PCs for reviewing so many papers and providing insightful and objective opinions.

We are so grateful that:
- __All the reviewers give our work positive ratings (7556)__, which is a tremendous encouragement for us.
- Reviewer iFKj highly recommends our work for the novel and elegant formalization, the creative methodology and the solid experiments. 
- Reviewer SGyo confirms the importance of our work, the insightful guidance strategy and the novel augmentation approach.
- Reviewer 7cPE appreciates the justifiable motivation, the exquisite illustrations and the practical value of our work.
- Reviewer BPQh highlights the creative insights, the detailed derivation and the comprehensive experiments of our work.

Meanwhile, all the reviewers offer valuable questions and suggestions. We have responsed to every reviewer individually in each separate rebuttal, accompanied by __a global rebuttal pdf__ of figures and tables down below. Please refer to the pdf as well.

We believe the rebuttal stage is highly meaningful, and we will add the rebuttal discussion to the new version of our paper as much as possible. If there is any omission in our response or if you have any additional question, please feel free to let us know. Thanks again for your great efforts!

Meta-review: The work received positive scores from reviewers. They listed a number of strengths, including the practical importance of the problem, thorough derivations, and sufficient experiments. There were certain questions and concerns raised, which were adequately addressed during the discussion period. AC agrees with the reviewers, and believes that the problem to keep the subject while changing the prompts is indeed of utmost importance. The decision is to accept the manuscript.
Decision: Accept (poster)