Subject-driven Text-to-Image Generation via Preference-based Reinforcement Learning
Authors: Yanting Miao
William Loh
Suraj Kothawade
Pascal Poupart
Abdullah Rashwan
Yeqing Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Diffusion Models
Text-to-Image generation
Reinforcement Learning
Primary area: diffusion_based_models
TL;DR: We present the $lambda$-Harmonic reward function and Reward Preference Optimization (RPO) for the subject-driven text-to-image generation task.

Abstract: Text-to-image generative models have recently attracted considerable interest, enabling the synthesis of high-quality images from textual prompts. However, these models often lack the capability to generate specific subjects from given reference images or to synthesize novel renditions under varying conditions. Methods like DreamBooth and Subject-driven Text-to-Image (SuTI) have made significant progress in this area. Yet, both approaches primarily focus on enhancing similarity to reference images and require expensive setups, often overlooking the need for efficient training and avoiding overfitting to the reference images. In this work, we present the $\lambda$-Harmonic reward function, which provides a reliable reward signal and enables early stopping for faster training and effective regularization. By combining the Bradley-Terry preference model, the $\lambda$-Harmonic reward function also provides preference labels for subject-driven generation tasks. We propose Reward Preference Optimization (RPO), which offers a simpler setup (requiring only 3\% of the negative samples used by DreamBooth) and fewer gradient steps for fine-tuning. Unlike most existing methods, our approach does not require training a text encoder or optimizing text embeddings and achieves text-image alignment by fine-tuning only the U-Net component. Empirically, $\lambda$-Harmonic proves to be a reliable approach for model selection in subject-driven generation tasks. Based on preference labels and early stopping validation from the $\lambda$-Harmonic reward function, our algorithm achieves a state-of-the-art CLIP-I score of 0.833 and a CLIP-T score of 0.314 on DreamBench.

Subject-driven Text-to-lmage Generation via Preference-based Reinforcement Learning
Yanting Miao
William Loh Depart of Computer Science
Department of Computer Science University of Waterloo, Vector Institute y43miao@@uaterloo.co
University of Waterloo, Vector Institute
smlohhuwaterloo.
Suraj Kothawade
Pascal Poupart
Google
Depart of Computer Science University of Waterloo, Vector Institute
stothawade@googlee.com
ppoupart@uvaterloo..
Abdullah Rashwan
Yeqing Li Google yeqingegoogle.com
Google
arashvan@google.com com
Abstract
Text-to-image generative models have recently attracted considerable interest, enabling the synthesis of high-quality images from textual prompts. However, these models often lack the capability to generate specific subjects from given reference images or to synthesize novel renditions under varying conditions. Methods like DreamBooth and Subject-driven Text-to-Image (SuTT) have made significant progress in this area. Yet, both approaches primarily focus enhancing similarity to reference images and require expensive setups, often overlooking the need for efficient training and avoiding overfitting the reference images. In this work, we present the A-Harmonic reward function, which provides a reliable reward signal and enables early stopping for faster training and effective regularization. By combining the Bradley-Teree preference model, the X-Harmonic reward function also provides preference labels for subject-driver generation tasks. We propose Reward Preference Optimization (RPO), which offers a simpler setup (requiring only 3% of the negative samples used by DreamBooth) and fewer gradient steps for fine-tuning. Unlike most existing methods, our approach does not require training text encoder optimizing text embeddings and achieves text-image alignment by fine-tuning only the U-Net component. Empirically, A-Harmonic proves to be reliable approach for model selection in subject-driven generation tasks. Based on preference labels and early stopping validation from the A-Harmonic reward function, our algorithm achieves a tate--ff-hhe-art CLIP-I score of 0.833 and a CLIP-T score of 0.314 on DreamBench. Our Pytorch implementation is available at htttps://githhbb com/ andrew- miao/RPO
Introduction
In the evolving field generative AI, text-to-image diffusion models (28 have demonstrated remarkable abilities in rendering scenes that are both imaginative and contextually
13 26 29
27 23 24
"This project was completed during work at Google
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

duck boy
À Har momioii Roward
*... VIDI Pass Linn Seple
Rood"
with the beskground
Mith Ieagreen
dog
Preference Labels
Diflusion Models
style"
vaur somrean eotfel"
with Eppotian Paramiis coo
resditon
the backgruudd
Figure I: We illustrate the X-Harmonic reward function applied to the subject-driven generation task. Leveraging preference labels produced by the X-Harmonic reward function, alongside few reference images, our preference-based algorithm efficiently generates unseen scenes that are both faithful to the reference images and the textual prompts.
appropriate. However, these models often struggle with tasks that require the portrayal of specific subjects within text prompts. For instance, if provided with photo of your cat, current diffusion models are unable to generate image of your cat situated in the castle of your childhood dreams. This challenge necessitates deep understanding subject identity. Consequently, subject-driven text-to-image generation has attracted considerable interest within the community. Chen et al. [7] have noted that this task requires complex transformations of reference images. Additionally, Ruiz
et al. (23) have highlighted that detailed and descriptive prompts about specific objects can lead to varied appearances in subjects. Thus, traditional image editing approaches and existing text-to-image models are ill- suited for subject-driven tasks.
Current subject-driven text-to-image generation methods less expressive and expensive. Textual Inversion [12 performs poorly due to the limited expressiveness of frozen diffusion models. Imagic [15] is both cune-con and resource-intensive during the fine-tuning phase. requires text- embedding optimization for each prompt, fine-tuning of diffusion models, and interpolation between optimized and target prompts. The training process is complex and slow. These text-based methods require 30 to 70 minutes to fine-tune their models, which is not scalable for real applications. SuTI proposes an in-context learning method for subject-driven tasks. However, SuTI demands half million expert models for each different subject, making it prohibitively expensive. Although SuTI can perform in-context learning during inference, the setup of expert models remains costly. DreamBooth 23 provides simpler method for handling subject-driven tasks. Nevertheless, DreamBooth requires approximately 1000 negative samples and 1000 gradient steps, and also needs fine-tuning of text encoder to achieve state-of-the -ott----tttt-tttt performance. Therefore, it is worthwhile to explore more efficient training methods: the setup should be as simple as possible. First, training should not include multiple optimization phases. Second, text-to-image alignment should fine-tur tune as few components as possible such as the UNet, but not the text encoder for each prompt. Third, faster evaluation and regularization should be enabled by early stopping based on model selection.
In this paper, we propose X- A-Harmonic reward function that enables early stopping and accelerates training. In addition, we incorporate the Bradley- preference model to generate preference labels. We utilize preference-basee reinforcement learning algorithms to finetune pre-trained diffusion models and to achieve text-to-image alignment without optimizing any text encoder or text embedding. The whole finetuning process including setup, training, validation, and model saving only takes to 20 minutes Cloud TPU V4. Our method, Reward Preference Optimization (RPO), only requires

---

few input reference images and the finetuned diffusion model can generate images that preserve the identity specific subject while aligning well with textual prompts (Figure T..
To show the effectiveness of our X-Harmonic reward function, we evaluate RPO on diverse subjects and text prompts on DreamBench [2]] and we report the DINO and CLIP-I/CLIP-T of RPO's generated images on this benchmark and compare them with existing methods. Surprisingly, our method requires simple setup (3% of DreamBooth configuration) and with fewer gradient steps, but the experimental results outperform or match SOTA.
In summary, contributions are as follows:
We introduce the A-Harmonic reward function, which permits early-stopping to alleviate overfitting in subject-driven generation tasks and to accelerate the finetuning process.
By combining the X-Harmonic reward function and a preference model, we present RPO, which only requires a cheap setup, but still can provide high quality results.
We evaluate RPO and show the effectiveness of the X-Harmonic function with diverse subjects and various prompts on DreamBench. We achieve results comparable to SOTA.
2 Related Works
Ruiz et al. 23 formulated a class of problems called subject-driven generation, which refers to preserving the appearance of a subject contextualized in different settings. DreamBooth [23] solves the issue of preserving the subject by binding it in textual space with a unique identifier for the subject in the reference images, and simultaneously generating diverse backgrounds by leveraging prior class-specifee information previously learned. A related work that could possibly perform the the
same task is textual inversion [12]. However, its original objective is to produce a modification subject property marked by unique token in the text. While it can be used to preserve the subject and change the background or setting, the performance underwhelming compared to DreamBooth
in various metries (23
The prevalent issue in DreamBooth and textual inversion is the long training time [23 12 since gradient-based optimization has to be performed on their respective models for each subject. Subject- driven sot-too-mage generator (SuTT) by 7 aims to alleviate this issue by employing apprenticeship learning. By scraping millions of images online, many expert models are trained for different clusters of images centered around different subjects, which allows an apprentice to learn quickly from the experts [7]. However, this is an incredibly resource intensive task with massive computational overhead during training.
In the field of natural language processing, direct preference optimization has found great success in large language models (LLM) [2I]. By bypassing reinforcement learning from human feedback and directly maximizing likelihoods using preference data, LLMs benefit from more stable training and reduced dependency on an external reward model. Subsequently, this inspired Diffusion-DPO by (30], which applies a similar technique to diffusion models. However, this relies on preference labelled dataset, which can be expensive collect or not publicly available for legal reasons.
Fortunately, there are reward models that can serve as functional substitutes such as CLIP 19 and ALIGN [1]. ALIGN has a dual encoder architecture that was trained on large dataset. The encoders can produce text and image embeddings, which allows us to obtain pairwise similarity scores by computing cosine similarity. There are also diffusion modelling techniques that can leverage reward models. An example is denoising diffusion policy optimization (DDPO) by Black et al. [3] that uses a policy gradient reinforcement learning method encourage generations that lead higher rewards.
3 Preliminary
In this section, we introduce the notation and some key concepts about text-to-image diffusion models and reinforcement learning.
are a family of latent variable models of the form po (xx) fxx Po (Xq:1 )dx..7. where the Xi. XT are noised latent variables of the same dimensionality as the input data xo (xx)). The diffusion or forward process
Text-to-Image Diffusion Models. Diffusion models 13
26 28
3

---

is often Markov chain that gradually adds Gaussian noise to the input data and each intermediate sample x, can be written
X Xo + v1 0x for all (I,...,7), T)
where 04 refers to the variance schedule and e, NO I). Given conditioning tensor (often a text embedding), the core premise of text-to-image diffusion models is to use a neural network (x(,, c, () that iteratively refines the current noised sample x, obtain the previous step sample X,-1, This network can be trained by optimizing a simple denoising objective function, which the time coefficient weighted mean squared error, the derivation is shown in Appendix A.1
I extt, [x=())]eoox+ c, 1) cell2]
and w()) is time dependent weight that can be where t is uniformly sampled from {1, ,l, . simplified to according to [1 27
Reinforcement Learning and Diffusion DPO Reinforcement Learning for diffusion models aims to solve the following optimization problem:
Exa c)
Σ R(x,, xt- c) =ID (xt-1 xx, c) Plasse (x1-1 X c))
(3)
where is a hyperparameter controlling the KL-divergence between the finetuned model pe and the pre-trained base model Phase- In Equation (14) from Diffusion-DPO [30), the optimal can be approximated by minimizing the negative log-likelihood:
log (3 ((1ehmee (xt, ,c,t) -,, ||oo(xxx c, E E+12
E x x 1xxt.xx x
Cl] ,c,t) - c.. 1|co(x; c,t) -
(4)
where et and e are independent samples from Gaussian distribution, x, and x; are perturbed versions of x and x, that depend on ett and e and x is preferred to Xg- A detailed description given Appendix A.1
Additional notation. We use X and Xgen to represent the reference image and generated image, respectively. Iet denotes the set of reference images, and Igen is the set of generated images. P(x > x) represents the probability that x is more preferred than &.
Method
We present X-Harmonic reward function that provides reward signals for subject-driven tasks to reduce the risk that the learned model will overfit to the reference images. Based on this reward function, use the Bradley-Terry model to sample preference labels and preference algorithm to finetune the diffusion model by optimizing both similarity loss and a preference loss.
4.1 Reward Preference Optimization
there is no human feedback in the subject-driven text-to-image generation task. The model only receives few reference images and prompt with specific subject. Hence, we first propose the X-Harmonic reward function that can leverage the ALIGN model [14] to provide feedback based on the generated image fidelity: similarity to the given reference images and faithfulness the text prompts.
In contrast to other fine-tuning applications 30
18
21
20
X-Harmonic Reward Function.
The normalized ALIGN-I and ALIGN-T scores be denoted CosSim( (x), (x)) +1
ALIGN-I(x, Iat)
(Image alignment)
Iietl
2
TET
ALIGN-T(x,C c) = (x)) (c)) +
(Text alignment),
2

---

where CosSim is the cosine similarity, (x) is the image feature extractor and ( is the text encoder the ALIGN model. Given a reference image set Tee the X-Harmonic reward function can be defined by weighted harmonic mean of the ALIGN-I and ALIGN-T scores,
rxx, C; ITett
X
1-X
ALIGN-I(X, Inet) ALIGN-T(xxC)
Compared the arithmetic mean, there to AM-GM-HM inequalities [10], the harmonic mean is a lower bound of the arithmetic mean and maximizing this "pessimistic" reward can also improve the arithmetic mean of ALIGN-I and ALIGN- T scores; (2) the harmonic mean is more sensitive to the smaller of the two scores, i.c., a larger reward only achieved when both scores are relatively large.
advantages using the harmonic mean: (1) according
For a simple example, consider À = 0.5. If there are two images, x and x, where the first image achieves an ALIGN-I score of 0.9 and ALIGN-T score of 0.01, and the second image receives an ALIGN-I score of 0.7 and an ALIGN-T score of 0.21, we may prefer the second image because has high similarity to the reference images and is faithful to the text prompts. However, using the arithmetic mean would assign both images the same reward of 0.455. In contrast, the harmonic mean would assign the first image reward of 0.020 and the second image reward of 0.323, aligning with our preferences. During training, we set Again = O, which means the reward model will focus solely on text-to-image alignment because the objective function consists only of loss for image-to-image alignment. Note that we set Aval to a different value for validation, which evaluates the fidelity of the subject and faithfulness of the prompt. Details can be found Section 5
Dataset. The set of images for subject-driven generative tasks can usually represented as I = -eet U Igen. where Igen is the image set generated by the base model. DreamBooth [23] requires two different prompts, and Cprs where includes a reference the prior class of the subject but not the subject. For example, с can be "a photo of [V] dog" and Cpr can be "a photo
the subject while Cpr refers to
a dog", where "[V]" a unique token that refers to the subject and dog is the prior class of the subject. DreamBooth then uses Cpr to generate variety of images ggnn in the prior class to avoid overfitting to the reference images via a regularizer. Typically, the size of the set of generated images is around 1000, i.c., Igenl 1000, which is time-consuming and space-intensive in real applications. However, the diffusion model can only maximize the similarity score and still receives high reward based this uninformative prompt Cpr- Our method aims to balance the trade-off between similarity and faithfulness. Thus, for efficiency, we introduce 8 novel training prompts, Cmod of the form "a [V] [class noun] [modifi cation]" where modification includes artistic style transfer. cecooextuullization, and accessorization. For example, Cmad can be "a [V] dog is the Moon'. These training prompts can be pre-specified or generated by other Large Language Model The full list of training prompts is provided in the supplementary material (Figure 6). We feed these training prompts the base model and generate 4 images for each training prompt, i.e., IZgen| = 32.
Once we obtain reward signals, adopt the Bradley-Terry model 4 generate preference labels. In particular, given a tuple (X Xgen: Cmod), we sample preference labels y from the following probability model:
exp(r Cmod X, Iret))
P(xref > Xgen) = exp(r (((xee's Cmooti X, Iner)) +
Gmodi X, Irett
Learning. The learning objective function consists of two parts - similarity loss and preference loss. The similarity loss is designed to minimize the KL divergence between the distribution of reference images and the learned distribution (x), which is equivalent to minimizing:
Ssim C.t.el [ll [l
c, 1
t1 ... T), Eat ~ VO. (7)
The preference loss aims to capture the preference signals and fit the preference model, Eq. Therefore, we use binary cross-entropy as the objective function for the preference loss. Combining
SSTTI 7 utilizes PaLM 9 generate unseen prompts during training
5

---

Reference Images
Generated Images
Similar Loss
Diffusion Models
Preference Loss
i-Harmonic Reward
"A [V] dog is on the Moon"
Base Models
Figure 2: Overview of the finetuning phase for RPO. First, the base diffusion model generates a few images based novel training prompts. Second, we compute the rewards for both reference and generated images using Equation . Then, preference labels are sampled according to the preference distribution, defined in Equation (6) Finally, the diffusion model is trained by minimizing both the similarity loss (Equation and preference loss (Fquation g
the DPO objective function in Eq. 4 the loss function can be written as follows:
(pre(())
R Rx Caed C. y ylog a (Blo (xto: Xgen; Canod: Us L Gref' Egen)
(1 3) log
BBg (Neet: Xgens Gmods y, t, Cref: Egen)
(8)
where
to (x Xgan Cmod: Y, t, Eyets Egen)
|| Cbaso (Xeet Gmod, t) Crer ||co (Xret,e Cmod. Cxet
(]) Ebase &gen,ts Gmod, () Egen 12 ( Gmod, () Egenl 1eall
and t ~ uf1,..., T} and Gre's Egen ~ _N I). Combining these two loss functions together, the objective function for finetuning is written as
((0) = Csim (0)
Cper()) (O)
Figure presents an overview of the training method, which includes the base model generated samples, the ALIGN reward model, and the preference loss. Note that Lpeet serves as a regular- izer for approximating the text-to-image alignment policy. Conversely, DreamBooth 23 adopts x,, x,,c) c)) as its regularizer, which cannot guarantee faithfulness to
DKL (Ptasc X2, c)e the text-prompt. Based on this loss function and preference model, only need few hundred gradient steps and small set size of Igeal to achieve results that are comparable to, or even better than, the state of the art. The fine-tuning process, which includes generating images, training, and validation, takes about 5 to 20 minutes on a single Google Cloud Platform TPUv4-8 (32GB) for Stable Diffusion.
5 Experiments
In this section, we present the experimental results demonstrated by RPO. We investigate several questions. First, can our algorithm learn to generate images that are faithful both to the preference images and the textual prompts, according preference labels? Second, RPO can generate high-quality images, which part is the key component of RPO: the reference loss or the early stopping by the X-Harmonic reward function? Third, how different Aval values used during validation affect performance in RPO? We refer readers to Appendix A.2 for details on the experimental setup, Appendix A.7 for the skill set of RPO, Appendix A.4 for the Timitations of the RPO algorithm, and Appendix A.5 for future work involving RPO.

---

5.1 Dataset and Evaluation
DreamBench. In this work, we use the DreamBench dataset proposed by DreamBooth 23 This dataset contains 30 different subject images including backpacks, sneakers, boots, cats, dogs, and toy, etc. DreamBench also provides 25 various prompt templates for each subject and these prompts are requiring the learned models to have the following abilities: re-contextualization, accessorization, property modification, and attribute editing.
Evaluation Metrics. We follow DreamBooth 23 and SuTI 7 to report DINO [5] and CLIP-I [19] for evaluating parage-to-image similarity score and CLIP-T 19 for evaluating the text-to-image similarity score. We also use our A-Harmonic reward as a evaluation metric for the overall fidelity and the default value of X = 0.3. For evaluation, we follow DreamBooth [23] and Su'Il 7 to generate 4 images per prompt, 3000 images total, which provides a robust evaluation.
Baseline algorithms. DreamBooth 23 test-time fine-tuning method. This algorithm requires approximately Zgen| 1000 and 1000 gradient steps to finetune the UNet and text-encoder com- ponents. SuTT 77] A preitre method that requires half million expert models and introduces cross-attention layers into the original diffusion models. Textual Inversion [12]: A text-based method that optimizes the text embedding but freezes the diffusion models. Re-Imagen (8) An information retrieval-based algorithm that modifies the backbone network architectures and introduces cross- attention layers into the original diffusion models. DisenBooth 6 test-time fine-tuning method that generates subject-driven images by optimizing textual
embeddings. Custom Diffusion [16] a test-time fine-tuning method, focused on efficient training by optimizing the key and value projection matrices in the cross-attention layers of diffusion models. ELITE [31]: A test-time fine-tuning approach consisting of two stages: one for training the textual embeddings and another for preserving identity. IP-Adapter [ a pretrained method where the decoupled cross-attention layer captures the textual signal while integrating reference images. SSR-Encoder [33 also a pretrained method that highlights selective regions and extracts detailed features for subject-driven image generation.
5.2 Results
Quantitative Results. We begin by addressing the first question. We use quantitative evaluation to compare RPO with other existing methods on three metries (DINO, CLIP-I, CLIP-T) in DreamBench to validate the effectiveness of RPO. The experimental results on DreamBench is shown in Table We observe that RPO can perform better or on par with existing works on all three metrics. Compared DreamBooth, RPO only requires 3% of the negative samples, but RPO can outperform DreamBooth on the CLIP-I and CLIP-T scores by 3% given the same backbone. Our method outperforms all baseline algorithms in the CLIP-T score, establishing a new SOTA result. This demonstrates that RPO, by solely optimizing UNet through preference labels from the X-Harmonic reward function, can generate images that are faithful to the input prompts. Similarly, our CLIP-I score is also the highest, which indicates that RPO generate images that preserve the subject's 's visual features. In terms of the DINO score, our method is almost the same as DreamBooth when using the same backbone. We conjecture that the reason RPO achieves higher CLIP scores and lower DINO score is that the X-Harmonic reward function prefers to select images that are semantically similar to the textual prompt, which may result in the loss of some unique features in the pixel space.
Qualitative Results. We use the same prompt as SuTI 77. and the generated images are shown in Figure 3 RPO generates images that are faithful to both reference images and textual prompts. We noticed a semantic mistake in the first prompt used by SuTI 7 it should be A dog eating a cherry from a bowl. Furthermore, each reference bowl image contains blueberries, and the ambiguous prompt caused the RPO-trained model to become confused during the inference phase. However, RPO still preserves the unique appearance of the bowl. For instance, while the text on the bowl is incorrect or blurred in the SuTI and DreamBooth results, RPO accurately retains the words Bon Appetit from the reference bowl images. Although existing methods can produce images highly faithful to the reference images, they may not align as well with the textual prompts. We also
'DINO encodes only images and makes a better fine- ane-ggrined understanding of images than CLIP. But it cannot provide signals text-image alignment.
7

---

A dog esting cherry bowl
Bay
Kopp
Two thin-neck vases on table
A tiny dog having hairout
SuIT
Re-Imagen
DreamBBooti
RPO
Figure 3: Qualitative comparison with other subject-driven text-to-image methods, adapted from
Table 1: Quantitative comparison for the number of iterations, subject fidelity and prompt fidelity.
Method
Backbone
Iterations
DINO
CLIP-I
CLIP-T
Reference Images
N/A
N/A
0.774
0.885
N/A
DreamBooth 23
Imagen
24
1000
0.696
0.812
0.306
DreamBooth
SD
22
1000
0.668
0.803
0.305
Textual inversion
12
SD 222
5000
0.569
0.780
0.255
SuTI 7
Imagen
2
1.5 x 10s
0.741
0.819
0.304
Re-Imagen
Imagen
24
x 105
0.600
0.740
0.270
DisenBooth
6
SD 22
3000
0.574
0.755
0.255
Custom Diffusion 16
SD 22
500
0.695
0.801
0.245
ELETE [3]
SD 22
3000
0.652
0.765
0.255
IP-Adapter 32
SD SD 22
106
0.608
0.809
0.274
SSR- ESR-Encoder
SD
108
0.612
0.821
0.314
Ours: RPO
SD 22
400
0.652
0.833
0.314

---

Table 2: Ablation study on regularization to evaluate fidelity across multiple subjects and prompts. Standard deviation included.
Method
DINO f
CLIP-I
CLIP-T
0.3-Harmonic
Pure Csim
0.695 + 0.077
0.852 + 0.043
0.285 + 0.027
0.660 + 0.016
pees w/o early-stopping
0.688 + 0.082
0.845 + 0.042
0.296 + 0.027
0.663 + 0.014
Early-stopping w/o Sprel
0.575 + 0.124
0.799 + 0.052
0.323 + 0.025
0.672 + 0.013
RPO (Aval 0.3)
0.581 + 0.113
0.798 + 0.039
0.329 0.021
0.673 0.013
Table 3: Different validation Ava- Harmonic reward comparison for evaluating fidelity over multiple subjects and prompts. Standard deviation is included.
Configuration
DINO f
CLIP-I
CLIP-T
0.3- ..3-Hammonic t
Aval 0.3
0,581 + 0.113
0.798 + 0.039
0.329 + 0.021
0.673 + 0.013
Aval 0.5
0.652 0.082
0.833 0.041
0.314 # 0.022
0.671 0.008
Aval = 0.7
0.679 + 0.085
0.850 + 0.045
0.304 + 0.023
0.667 + 0.011
provide an example in Appendix A.3 that shows how RPO can handle the failure case observed in DreamBooth and SuTI. Additionally, we find that the RPO model can generate reasonable images even for highly imaginative prompts, as shown in Appendix A.7 (Figure [6 and Figure [7). These images demonstrate that the RPO-trained model does not overfit the training data. Instead, the X- A-Harmonic function provides method for selecting model capable of text-to-image alignment, even when the textual prompts are highly imaginative.
5.3 Ablation Study and Method Analysis
Preference Loss and X-Harmonic Ablation. We investigate the second question through an ablation study. Two regularization components are introduced into RPO: reference loss as regularizer and early stopping by Ava Harmmonc reward function. Consequently, we compare four methods: (I) Pure fsim. which only minimizes the image-to- tange-to--mag similarity loss Lsim: (2) ferett w/o early stopping. which employs peer as regularizer but omits early stopping by -armonic reward function; (3) Early stopping w/o Lprl which uses -armooic reward function
regularization method
but excludes peets (4) Full RPO, which utilizes both pee and early stopping by the -armooii reward function. We choose the default value Aval 0.3 in this ablation study.
the evaluation results of these four methods DreamBench. We observe that without early stopping, Lpret can still prevent overfitting to the reference images and improve text-to-image alignment, though the regularization effect is weak. Specifically, the -3-Harmonic only shows
Gradient steps (GS)
Reference Images
GS=80, Reward Reward Reward 0.661
GS=160.
GS=240,
GS=320,
GS=400,
0.654
0.648
the Moon"
backpack
GS=80, Reward 0.660
GS=160, Reward 0.657
GS=240, Reward 0.653
GS=320, Reward 0.652
GS=400, Reward 0.652
".. In Vincent Van Gogh
cat
Figure 4: Changes in the 0.3 Harmonic reward value during RPO training process.

---

Reference images
with the CN Tower the background
dog
Aval 0.3
Aval 0.5
Aval = 0.7
Figure 5: Different Aval's 's will lead to different results. small Aval assigns higher a weight for Akt-to-image alignment and leads to diverse generation. A large Aval may also cause overfitting.
marginal improvement of 0.003 over pure Sii and 0.001 over early stopping without Spret. The early stopping facilitated by the Ava-- reward function plays a crucial role in counteracting overfitting, helping the diffusion models retain the ability generate high-quality images aligned with textual prompts. To provide a deeper understanding of the X-Harmonic reward validation, we present two examples from during training in Figure 4 covering both objects and live subjects. We found that the model tends to overfit at a very early stage, i.c., within 200 gradient steps, where X-Harmonic can provide correct reward signals for the generated images. For the backpack subject, the generated image receives a low reward at gradient step 80 due to its lack of fidelity to the reference images. However, at gradient step 400, the image is overfitted the reference images, and the model fails to align well with the input text, resulting in another low reward. À A-Harmonic assigns a high reward images that are faithful both the reference image and textual prompts.
Impact of Aval- We examine the third question by selecting different Aval values from the set {0.3, 0.5, 0.7) as the validation parameters for the X-Harmonic reward. According Equation 5 we believe that as Ayal increases, the A-Harmonic reward function will give higher weight to the srage-to-image similarity score. This will make the generated images more closely resemble the reference images, however, there is also risk of overfitting. Table 3 shows the results of three different Aaa values on DreamBench. As we expected, a larger Aval makes the images better preserve the characteristics of the reference images, but it also reduces the text-to-image alignment score. Figure Shswws us an example. this example, different Aval values lead different outcomes due to varying strengths of regularization. A smaller Aval 0.3 can generate more varied results, but seems somewhat off from the reference images. Aval = 0.5 preserves some characteristics beyond the original subject, such the sofa, but also maintains alignment between text and image. However, when Aval = 0.7 is chosen as an excessively large value, the model actually overfits to the reference images, ignoring the prompts. We have additional comparisons, including different Auain different Pythagorean means, and aesthetic scores 25 in Appendix A.3
Conclusion
We introduce the X-Harmonic reward function derive preference labels and employ RPO finetune the diffusion model for subject- text-to-image generation tasks. Additionally, the X-Harmonic reward function serves validation method, enabling early stopping to mitigate overfitting to reference images and speeding the finetuning process.
Acknowledgments and Disclosure of Funding
We thank Shixin Luo and Hongliang Fei for providing constructive feedback. This work was supported by Google grant with Cloud TPUs from Google's TPU Research Cloud (TRC). We also thank the Vector Institute, the Canada CIFAR AI Chair program and the Natural Sciences and Engineering Research Council of Canada for their support.
References
[1] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless
10

---

assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05880 2022.
[2] Charlotte Bird, Eddie Ungless, and Atoosa Kasirzadeh. Typology of risks of generative text-to- image models. In Proceedings of the 2023 AAAI/ACM Conference on Al, Ethics, and Society, pages 396-410, 2023.
[3] Kevin Black, Michael Janner, Yilun Du, 1lya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint (xXiv:2305.13311 2023.
[4] Ralph Allan Bradley and Milton E Terry Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):324 345, 1952.
[5] Mathilde Caron, Hugo Touvron. Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski. and Armand Joulin. Emerging properties in self-supervise vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650-9660, 2021.
[6] Hong Chen, Yipeng Zhang, Simin Wu, Xin Wang. Xuguang Duan, Yuwei Zhou, and Wenwu Zhu. Disenbooth: identity-preservinn disentangled tuning for subject-driven sext-to-image generation. arXiv preprint arXiv:2305.03:: 2023.
[7] Wenhu Chen, Hexiang Hu, Yandong Li, Natamiel Ruiz, Xuhui Jia, Ming-We Wei Chang, and William W Cohen. Subject-driven text-to-image generation via apprenticeship learning. Ad- vances in Neural Information Processing Systems, 36, 2024.
[8] Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen. Re-imagen: Retrieval- augmented text-to-image generator. arXiv preprint arXiv:2209. /4491, 2022.
[9] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1- 113, 2023.
[10] Dušan Djukić, Vladimir Janković, Ivan Matić, and Nikola Petrović. The IMO compendium: collection of problems suggested for the International Mathematical Olympiads: 1959-2004, volume 119. Springer, 2006.
[11] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine- tuning text-to-image diffusion models. In Whirty-sevente Conference on Neural Information Processing Systems, 2023.
[12] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashník, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.0166 2022.
[13] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-66 (1668006651,,,0 2020.
[14] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuae Sung. Zhen Li, and Tom Duerig. Scaling up visual and languae representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4916 PMLR, 2021.
[15] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. Proceedings of the IEEF/CVF Conference Computer Vision and Patlern Recognition, pages 6007-6017, 2023.
[16] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi- concept customization of text-to-image diffusion. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1931-1941, 2023.

---

[17] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.051011, 2017.
[18] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[19] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 7748-8763. PMLR, 2021.
[20] Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q": Your language model is secretly a q-function. arXiv preprint htti::404.1223555 2024.
[21] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[22] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10684-10695, 2022.
[23] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22500-22510, 2023.
[24] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36494, 2022.
[25] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25221 2022.
[26] Jascha vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages
Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper-
PMLR, 2015.
[27] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations, 2021.
[28] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[29] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXivv2011.13356 2020.
[30] Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. arXiv preprint 1. 2023.
[31] Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for eustomized text-to-image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15943- 15953, 2023.
[32] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06722 2023.
12

---

[33] Yuxuan Zhang, Yiren Song, Jiaming Liu, Rui Wang. Jinpeng Yu, Hao Tang, Huaxia Li, Xu Tang. Yao Hu, Han Pan, al. Ssr-encoder: Encoding selective subject representation for subject- driven generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8069-8078, 2024.
13

---

A Appendix
A.1 Background
Reinforcement Learning. In Reinforcement Learning (RL), the environment can be formalized as Markov Decision Process (MDP). An MDP is defined by tuple (S, A, R. po. T), where S is the state space, A is the action space, P is the transition function, R is the reward function, Po the distribution over initial states, and T is the time horizon. At each timestep t, the agent observes state S and selects an action a, according to policy π ( and obtains a reward R(s, at), and then transit to next state St+1 P(s St: az). As the agent interacts with the MDP, it produces sequence of states and actions, which is denoted as a trajectory T = (So. ag: $1, , -1 ar-1). The RL objective is to maximize the expected value of cumulative reward over the trajectories sample from its policy:
T-1 Σ
B ET (T)
R(s a.)
Diffusion models.
26
13 29
consider a variance schedule (3.)T-1- For each data point
Xp Prista Xo), a Markov chain is constructed by transition probability p(x; x(-1) =
(x;; 11 Consequently, Pa. (x, Xo) = (xxx; VarXo. (1 a T)) where
a IT I 1 (1 3.). 28 29
show that training diffusion models, co c t), is equivalent to score matching, i.c., training a score function se (x,, t):
Ex-u(,.....T..x (.xo.........mmmm........ Xiompum.Xirrp (xi|x0)
[I] Vx, logPa, (xx, xo) so sx(x:: 1)
|| ||--- (xt vaixo) - so (xi, ()
27
= -Erul( T) No No rppa (xx)xx0)
22
=Berd((i... N. Nom ((.1 ||-ví - OrE - -so(x,,)) '' = VaiX0 + v1 04E
-B No Pam ((o) [11 - (x))lEa((x:))) (x:: - let ED 50(xx,1) == S0 (o(x::1 1)
vI at
=Be-U( (a e- [= 1) - <| where w()) == C2z-
Diffusion MDP We formalize the denoising process as the following Diffusion MDP:
(x Xe, c). Pu p(c) N.O,I) R(s R(x, x1
ST = (x, aT Xx-1,
s)) (aT
r (((x0, c)
ift=
otherwise
where r(xo: c) be a reward signal for the denoised image. The transition kernel is deterministic, i.e. P( St-t, a, = (ar-1, (x,-1, c). For brevity, the trajectory T is defined by (Xo. XI...., XT). Hence, the trajectory distribution for given diffusion models can be denoted as the joint distribution pe (Xo:T | c). In particular, the RL objective function for finetuning diffusion models can be re-written the following optimization problem:
Σ R(x,,x,-11 1, c) 3I (x- Xx, c) [/Phuse (X:-1 Xx, c))
(xa e)
where hyperparameter controlling the KL-divergence between the finetune model pe and the base model Pbase. This constraint prevents the learned model from losing generation diversity and falling into 'mode collapse' due single high cumulative reward result. In practice, this KL-divergence has become a standard constraint in large language model finetuning
18 21
Given preference labels and following the Direct Preference Optimization (DPO) framework 30 21 20 we can approximate the optimal policy p" by minimizing the upper bound:
Exp No sxx! x,
loog
[ (( [x]]E)onsc(x/ (xj, c),)
< 112 ||E c) -e =
-[] ( - -6; -l,eell
-llllxxxx .c,)) - 113) --)))
14

---

[s raken) (snject, Tokee) Viscent Vas reNdicJN',
1a toke)
(sabjett (eken)
outtit'
(enge, (4km) (snject toke)) the Rooe's
11a outtern tokan)
(ustjert_coemn coken) in Vincent Van Sogh rendition's
grean (rttp (sxt)ect tanen))
sakpert__caen with China Great Hell, backgraze')
(stton_sste)) (snjectt tokee)
taken) [sebjort Teken) with Stutes of Liberty backgreesd"
platform',
ra (u8tge resmn) (sajjet_ with the Shamae Liberty the backgraund"
Fa (unlore, tokes)
(entpa teken) (saject, tokee) with wolcase in tockyround', sibwey platforre,,
11a (onke)s_
lasbjert token) en the Rean"s
(18:9)_(**0)) (snject,
11a green (untevt_ tokas) (sukject_teken))
Object prompts
Live subject prompts
Figure 6: Training prompts for objects and live subjects.
Table 4: RPO Hyperparameters
Parameter
Value
Optimization
optimizer
Adam [17]
learning rate
5 x 10-6
weight decay
0.01
gradient clip norm
1.0
RPO
regularizer weight, |3
1.0
gradient steps
400
training preference weight, Again validation preference weight (default), Aval
0.0
0.3
where e+ and 6 are independent samples from a Gaussian distribution, x and x; are perturbed versions of x and x that depend on et and e and xf is preferred to x6- A detailed derivation can be found in 330
A.2 Experimental Details
Training Prompts. We collect 8 training prompts: re-contextuullzzatonn, I property modification and I artistic style transfer for objects. 5 re-contextualization, 1 attribute editing, artistic style transfer and accessorization for live subjects. The trainig prompts are shown in Figure 6
Experimental Setup During training we use Amain = O for the X-Harmonic reward function to generate the preference labels. We evaluate the model performance by ~var-Harmonic per 40 gradient steps during training time and save the checkpoint that achieve the highest validation reward. Table f lists the common hyperparameters used the generating skill set and the Aval used the default setting.
A.3 Additional Comparisons
Comparison to DreamBooth and SuTI. We observe RPO that can be faithful both reference images and the input prompt. To investigate whether RPO can provide better quality than DreamBooth and SuTI, we follow SuTI paper and pick the robot toy as an example demonstrate the effectiveness of RPO (Figure 7). In this example, DreamBooth is faithful to the reference image but it does not provide a good text-to-image alignment. SuTI provides an result that is fidelty textual prompt. However, SuTI lacks fidelity to the reference image, i.e., the robot should stand with its wheels use DreamBooth finetune SuTI (Dream-Sul)) SuTTl further to solve this failure case. Instead, RPO can generate an image not only faithful to the reference images but also align well with the input prompts.
instead of legs.
Comparison to different Aval- We have also added more samples for comparison of different Aval values (see Figure We find that \ = 0.5 encourages the learned model to retain output diversity while still aligning with textual prompts. However, the generated images invariably contain a sofa, which is unrelated the subject images. This occurs because every training image includes a sofa. large A weakens the regularization strength and leads to overfitting. Nevertheless, small value of
15

---

Table 5: Different Harm-no reward comparison for evaluating fidelity over multiple subjects and prompts.
Configuration
DINO
CLIP-I1
CLIP-T
Arain = 0.0 (default)
0.581 + 0.113
0.798 + 0.039
0.329 + 0.021
Arain = 0.3
0.646 = 0.083
0.815 = 0.037
0.315 + 0.026
Arain = 0.5
0.649 = 0.080
0.829 = 0.039
0.314 - 0.026
rain 0.7
0.651 + 0.088
0.831 + 0.033
0.314 + 0.026
Table 6: Comparison for harmonic mean and arithmetic mean reward function.
Contiguration
DINO t
CLIP-I
CLIP-T
Aval 0.3 (arithmetic mean)
0.638 + 0.083
0.823 + 0.037
0.318 + 0.027
Aval 0.5 (arithmetic mean)
0.702 + 0.078
0.857 + 0.047
0.295 + 0.017
Aval 0.7 (arithmetic mean)
0.678 + 0.085
0.851 + 0.041
0.299 + 0.026
Aval 0.3 (harmonic mean)
0.581 + 0.113
0.798 0.039
0.329 + 0.021
Aval 0.5 (harmonic mean)
0.652 + 0.082
0.833 + 0.041
0.314 + 0.022
Aval 0.7 (harmonic mean)
0.679 + 0.085
0.850 0.045
0.304 0.023
Aval can potentially climinate background bias. We highlight that this small Ayal not only encourages diversity but also mitigates background bias in identity preservation and enables the model to focus on the subject.
Comparison to different Atrain During our experiments, we use the default value for A 0. Thus, we also investigate how different Auman will effect the performance of RPO. We test three different values of Amain {0.3, 0.5, 0.7} and report their results in DreamBench (Table We observe that the image-to-image alignment increases with larger, but the text-to-image alignment decreases because the preference model tends to favor alignment with reference images and ignores the prompt alignment.
Comparison to different Pythagorean means. Further, we examine different Pythagorean means of ALIGN-I and ALIGN-T scores. We replace the harmonic mean as the arithmetic mean and test three different Aval (0.3, 0.5, 0.7). We report the results of the arithmetic mean reward in Table 6 The arithmetic mean not very sensitive to smaller values; tends to maximize higher values to achieve better final score. In practice, ALIGN-I will receive a higher value (this effect can be seen from CLIP-I and CLIP-T in Tables I to 3). Thus, the model will tend to optimize image-to-image alignment and achieve good results on DINO and CLIP-I but have lower score for text-to-image alignment.
Comparison to DPO The original DPO is not suitable for subject-driven tasks because the datasets do not contain preference labels. We introduce the X-harmonic function and design a variant of DPO for this task. We implement DPO diffusion 30 (without similarity loss) using preference labels for image-to-image similarity and text-to-image alignment. We choose Araai = 0.5 since this value assigns equal weights to the image-to-image similarity and text-to-image alignment. For a fair comparison, we also repost the results from RPO with the same Auain value. The results on DreamBench for these two methods are shown in Table 7 These results show that DPO can capture the text-to-image alignment from the preference labels. However, without Lsims DPO faces a significant overfitting problem; i.e., it achieves high text-to-image alignment but cannot preserve the subject's 's unique features.
Aesthetic comparison. Limited evaluation metrics are a common issue in subject-driven tasks. In T we report the average aesthetic scores (25 of the real reference images in DreamBench and the average aesthetic scores obtained with the best CLIP I/T À configuration (Aval = 0.5) in DreamBench. We observe that RPO does not decrease the quality of images. Instead, the generated images achieve slightly better quality than the reference images.
16

---

Table 7: DPO comparison for evaluating fidelity over multiple subjects and prompts on DreamBench
Method
DINO t
CLIP-I
CLIP-T1
DPO
0.338
0.702
0.334
Ours: RPO
0.649
0.819
0.314
Table 8: Aesthetic score comparison for the real reference images and RPO generated images
Method
Aesthetic Score f
Real images
5.145 + 0.312
Ours: RPO (Aval 0.5) 5.208 + 0.327
Referencs
Prompt
DreamBooth
SufT
Dream-Sufi
RPO
A robot is standing the streat neon-li lit city with high-rise buildings.
Figure 7: Comparison between DreamBooth, SuTI, Dream-Sui and RPO, adapted from
de نلااتاا
Reference
Aval 0.5
with the CN Tower basigronnd
dog
Figure 8: Additional samples of different Ayal A small Aval leads to more diverse generated images.
17

---

A.4 Limitations
Figure 9 illustrates some failure examples of RPO. The first issue is context- aapexxx-apearancc entanglement. In Figure the learned model correctly understands the keyword blue house; however, the appearance of the subject is also altered by this context, e.g. the color of the backpack has changed, and there is house pattern on the backpack. The second issue is incorrect contextual integration. We conjecture that this failure is due the rarity the textual prompt. For instance, imagining cross between chow chow and tiger is challenging, even for humans. Third, although RPO provides regularization, still cannot guarantee the avoidance overfitting. As shown Figure this may be because, to some extent, the visual appearance of sand and bed sheets is similar, which has led to overfitting issues in the model.
A.5 Future Work
The overfitting failure case leads to future work direction: can online RI. improve regularization and avoid overfitting? The second direction for future work involves implementing the LoRA version for RPO and comparing it to LoRA DreamBooth. Last but not least, we aim to identify construct open-source, subject-driven datasets for comparison. Currently, DreamBench is the only open-source dataset we can access and evaluate for model performance. Nevertheless, we should create larger dataset that includes more diverse subjects to verify the effectiveness of different algorithms.
A.6 Broader Impacts
The nature of generative image models is inherently subjected to criticism on the issue of privacy, security and ethies in the presence of nefarious actors. However, the core of this paper remains purely on an academic mission to extend the boundaries of generative models. The societal consequences of democratizing such powerful generative models is more thoroughly discussed in other papers. For example, Bird et al. outlines the classes of risks in text-to-image models (2] in greater detail and should be directed to such papers. Nevertheless, we play our part in the management of such risks by avoiding the use of identifiable parts of humans in the reference sets.
A.7
Skill Set
The skill set of the RPO-trained model is varied and includes re-contextualizztion (Figure 10 artistic style generation (Figure [1], expression modification (Figure [2]. subject accessorization (Figure 13 color editing (Figure multi-view rendering (Figure 15). novel hybrid synthesis (Figure 16]. and novel prompt generation (Figure 17).
18

---

Context-apperance entanglement
A [V] backpack with blue house in the background
Incorrect contextual integration
cross of a [V] chow chow and a tiger
Overfitting
A [V] pink sunglasses in the Sahara Desert
Figure 9: RPO's fallure modes include: (1) The context and the appearance of the subject becoming entangled. (2) The trained model failing to generate an image with respect to the input prompt. (3) The trained model still overfitting to the training set.
19

---

Reference Images
backpack
[V] backpack the Grand Canyon
A [V] backpack under staryy sky
backpack Times Square
backpack water
sunglasses
A cat wearing a [V] pink
A [V] pink sunglasses
[V] pink sunglasses the microwave
A [V] pink sunglesses under the Tokyo Tower
sunglasses
surrounded by cakes
clay teapot
clay leapat floating in the water
dayy leapot glass table
A clay teapot wooden table together with salmon sushi.
A leapot surrounded by delicate porcelain reacups
Figure 10: Re-contettualization samples from the RPO algorithm. RPO is able to generate images of specific subjects in unseen environments while preserving the identity and details of the subjects and be faithful to the input prompts. Reference images are shown on the left. We display the generated images along with their textual prompts on the right.
Reference Images
Wheent van Gogo
Pablo PICASSO
Rembrand van Rijo
Cluuds Moner
Figure 11: Artistic style rendition samples from the RPO algorithm. These art renditions can be applied dog given prompt of "a [painter] styled painting of [V] dog". The identity the subject is preserved and faithfully imitates the style of famous painters.
Reference Images
deprossed
Jayous
sleepy
Screaming
Figure 12: Expression modification samples from the RPO algorithm. RPO can integrate subject with various unseen expressions in the reference images. We also notice the pose of generated images, e.g., [V] sleepy dog, were not displayed in the training set.
20

---

Reference Images
X
chef outit
nurse outlit
police outfit
Superman out!!
witch outfi
Ironman outfit
angel wings
fre-fighter outfit
Figure 13: Accessories samples from the RPO algorithm. Conditioned the prompts, "aa [V] chow chow wearing [target outfit]", the generated images retains the unique features of the reference images, e.g., the hair color and breed of the subject dog. The interaction between subject dog and the outfits is realistic.
Reference Images
pupple
Figure 14: Color editing samples from the RPO algorithm. We display color modifications using prompts such as "a [target color] [v] monster toy". The identity of the subject pre- served.
Reference Images
bottom
back
Figure 15: Multi-view samples from the RPO algorithm. We generate images from specified viewpoints subject. For the top and bottom views, we use the prompts "a [V] cat looking up/down at the camera', For the back and side views, we apply the prompts "a back/side view of a [V] cat".
21

---

Reference Images
boar
hoals
Jon
Novel hybrid synthesis samples from the RPO algorithm. We apply the prompts cross of [V] chow chow and [target species]" to the RPO-trained model to generate these images. We highlight that RPO can synthesize new hybrids that retain the identity of the subject chow chow and perform property modifications.
Figure 16:
cyberpunk [V] dog with neon fur patterns and robotic limbs walking down rain-soaked alleyway in bustling city"
Reference Images
"a superhero [V] dog soaring through the sky over a city skyline, with cape billowing in the wind"
"an astronaut [V] dog in a sleek space suit floating weightlessly
inside a spacecraft."
"an underwater [V] dog wearing diving suit swimming through a vibrant coral reef surrounded by exotic marine life"
Figure 17: Novel prompts results from the RPO-trained model. We use ChatGPT to create 4 different highly imaginative prompts for specific dog. We select 4 images for each prompt. The results demonstrate the generalization capability of RPO for handling novel prompts.
22

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] pprovided proper justification is given (e.g., "error bars are not reported because would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The scope and contributions are clearly defined in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of our technique discussed the Appendix where we show certain cases that are not desirable.
23

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our works focus on the experimentaal/mmirrcaa performance of subject-driven generation only. Due to the page constraints and the scope, we decided not to pursue theoretical results for now
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The details are mentioned in the Appendix A.2 where states specific configu- rations needed to reproduce the main experiments, including the prompts.
Guidelines:
24

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to hosted model (e.g., g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our paper uses the DreamBench dataset for the experiments, which is publicly available for anyone interested in reproducing the main results. The code will be submitted as a ZIP file.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm ec/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/tipps.cc/public/guudeee hww.bl/ic./giiddee//deeeuuiiiiii /Ct////ooeeeeeaeeaaaeee ssi onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
25

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The details are mentioned in the Appendix A.2 In Table the À which is an important hyperparameter our experiments, is stated along with the metrics.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Does paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Experiments are conducted for each subject across various prompts, where metric computed individually. The standard deviation over all prompts and subjects is provided in the ablation study in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The compute resources used in this paper is described in Method section.
26

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: For our research and experiments, we use public datasets and models that have been used in prior published works. Our research does not involve direct interaction with human subjects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Given the nature of research, it necessitates the general discussion of the potential negative societal impacts which takes place in Appendix A.6
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
27

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification: Safeguards would fall outside the scope of our research and we mainly focus on subjects that are inanimate. No experiments explicitly use personally identifiable features of humans as subject.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The authors of Stable Diffusion, DreamBench and SuTI are eited and credited in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The asset mainly consist of code which contains instruction on how to run it.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
28

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We do not crowdsource images by ourselves; we use images from DreamBench which has been vetted through.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
As mentioned previously, there is no crowdsourcing involved.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
29


Reviews:
[{"review_id": "79td35Mipn", "reviewer": "Reviewer_tGfs", "summary": "This paper proposes a harmonic reward function for text-to-image personalization. Specifically, it uses this reward function to perform early stopping and regularization. Experimental results demonstrate the effectiveness of the proposed method.", "questions": "What are the results when the lambda value of the harmonic reward function is not set to 0 in the preference loss during training?", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* Their proposed reward-based model selection and regularization are reasonable and compelling.\n* The generated images presented in the paper demonstrate superior text fidelity of the proposed method.", "weaknesses": "* More baselines are needed. Specifically, since this paper is similar to DCO [1], a comparison with this method is needed.\n\n[1] Lee, Kyungmin, et al. \"Direct consistency optimization for compositional text-to-image personalization.\" arXiv preprint arXiv:2402.12004 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and thoughtful feedback. Please see our responses to your specific questions below.\n\n**Q: Comparison to DCO.**\n\nLee et al. [1] use SDXL [2] as the backbone model and apply LoRA to fine-tune the pretrained model. For a fair comparison, we only use LoRA to fine-tune the U-Net component for both methods (DCO and RPO). Furthermore, we implement RPO with SDXL as the backbone model, training the LoRA parameters with 1000 gradient steps and saving the checkpoint from the last gradient step, i.e., without early stopping. RPO achieves better image-to-image similarity results than DCO (higher DINO and CLIP-I). Even without early stopping, the CLIP-T score is only slightly lower (by 0.016) than DCO.\n\n| Method | Backbone | DINO  | CLIP-I | CLIP-T |\n| :----- | :------- | :---- | :----- | :----- |\n| DCO    | SDXL     | 0.593 | 0.792  | **0.343**  |\n| RPO (w/o early stopping)   | SDXL    | **0.644** | **0.823** | 0.327 |\n\n\n**Q: What happens when $\\lambda_{\\text{train}} \\neq 0$ during training?**\n\nWe test four different values of $\\lambda_{\\text{train}} = 0.3, 0.5, 0.7$ with the default $\\lambda_{\\text{val}} = 0.3$ and report their results in DreamBench. We observe that the image-to-image alignment increases with larger $\\lambda_{\\text{train}}$, but the text-to-image alignment decreases because the preference model tends to favor alignment with reference images and ignores the prompt alignment.\n\n|                                | DINO      | CLIP-I    | CLIP-T    |\n| :----------------------------- | :-------- | :-------- | :-------- |\n| $\\lambda_{\\text{train}} = 0.0$ | 0.581 $\\pm$ 0.113     | 0.798 $\\pm$ 0.039     | **0.329 $\\pm$ 0.021** |\n| $\\lambda_{\\text{train}} = 0.3$ | 0.646 $\\pm$ 0.083    | 0.815 $\\pm$ 0.037    |  0.315 $\\pm$ 0.026    |\n| $\\lambda_{\\text{train}} = 0.5$ | 0.649 $\\pm$ 0.080     | 0.829 $\\pm$ 0.039     | 0.314 $\\pm$ 0.026     |\n| $\\lambda_{\\text{train}} = 0.7$ | **0.651 $\\pm$ 0.088** | **0.831 $\\pm$ 0.033** | 0.314 $\\pm$ 0.026     |\n\n[1]: Lee, Kyungmin, et al. \"Direct consistency optimization for compositional text-to-image personalization.\" arXiv preprint arXiv:2402.12004 (2024).\n\n[2]: Podell, Dustin, et al. \"Sdxl: Improving latent diffusion models for high-resolution image synthesis.\" arXiv preprint arXiv:2307.01952 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oREL3bKZMY", "reviewer": "Reviewer_Kj9e", "summary": "This paper presents a method for generating personalized images from text using the λ-Harmonic reward function in a system called Reward Preference Optimization (RPO). Specifically, RPO fine-tunes a pre-trained text-to-image diffusion model using a reinforcement-learning-based objective concerning with the harmonic averaged value of text similarity and image similarity. This objective is used in a reinforcement learning framework for diffusion models. The results demonstrate that the proposed techniques improve the performance of the DreamBooth baseline and alleviate the image overfitting problem.", "questions": "* The metrics evaluations in Tables 1 and 2 appear inconsistent, despite seemingly identical parameters (λval = 0.3).\n* Additionally, in the metrics ALIGN-I and ALIGN-T, I am unclear about the rationale behind adding \"+1\" to the similarity calculations.", "limitations": "This paper have discussed it’s limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is well-structured and easy to follow.\n* The proposed λ-Harmonic reward function is reasonable and has achieved better results than the baseline.", "weaknesses": "* The authors could introduce briefly how does the method support adjusting λ at the test time. How about the method comparing with adding classifier-based guidance?\n* The wall-clock time analysis of the proposed method against the baseline DreamBooth in training and sampling is missing.\n* The experimental analysis of why harmonic mean is used instead of arithmetic mean is missing.\n* Comparisons could be made more extensive. There are more state-of-the-art personalized generation methods nowadays like [a,b]. \n* The authors may want to compare with more baselines that also support adjusting the text-image-tradeoff at the test time [c,d,e].\n* The motivation to alleviate the image overfitting and enhance the prompt adherence has been explored in [f]. Comparisons or at least some discussions are necessary.\n\n[a] Disenbooth: Identity-preserving disentangled tuning for subject-driven text-to-image generation, Chen et al., ICLR 2024.\n\n[b] Multi-concept customization of text-to-image diffusion, Kumari et al., CVPR 2023.\n\n[c] ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation, Wei et al., ICCV 2023.\n\n[d] IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models, Ye et al..\n\n[e] SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generation, Zhang et al., CVPR 2024.\n\n[f] PALP: Prompt Aligned Personalization of Text-to-Image Models, Tewel et al..", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and thoughtful feedback. Please see our responses to your specific questions below.\n\n**Q: How does the method allow for adjusting $\\lambda$ during inference, and how does it compare to classifier-guidance inference?**\n\nOur contribution is on the fine-tuning step on a pretrained diffusion model. Currently, our method (RPO) only provides reward or preference signals during the fine-tuning phase. $\\lambda_{\\text{val}}$ allows flexibility in the model-selection process for the trade-off between image-image alignment and text-image alignment for subject-driven tasks (details can be seen in Table 3).  The reward function selects the optimal checkpoint during fine-tuning, and we do not modify the diffusion reverse process for the inference phase. Therefore, our inference is the same as the SD, which is classifier-free.\n\n**Q: Wall-clock comparison between DreamBooth.**\nWe define the fine-tuning time as the sum of the preparation time and training time. The preparation time refers to the time spent generating negative samples. We use RPO and DreamBooth to fine-tune SD on 4 TPU v4 nodes and report the wall-clock fine-tuning time for these two methods in the following table.\n\n| Method      | Aesthetic Score |\n| :---------- | :-------------- |\n| DreamBooth | 28 min 38.87 sec      |\n| Ours: RPO   | **7 min 59.78 sec** |\n\n**Q: Experimental results of arithmetic mean is missing.**\n\nWe set the harmonic mean as the arithmetic mean and test the $\\lambda_{\\text{val}} = 0.3, 0.5, 0.7$. We report the results of the arithmetic mean reward function in the following table. We refer readers to Table 3 in our paper for comparison. We discussed the difference between the harmonic mean and the arithmetic mean from lines 139 to 143. The arithmetic mean is not very sensitive to smaller values; it tends to maximize higher values to achieve a better final score. In practice, ALIGN-I will receive a higher value (this effect can be seen from CLIP-I and CLIP-T in Tables 1 to 3). Thus, the model will tend to optimize image-to-image alignment and achieve good results on DINO and CLIP-I but have a lower score for text-to-image alignment.\n\n|                                | DINO      | CLIP-I    | CLIP-T    |\n| :----------------------------- | :-------- | :-------- | :-------- |\n| $\\lambda_{\\text{val}} = 0.3$ (arithmetic) | 0.638 $\\pm$ 0.083     | 0.823 $\\pm$ 0.037     | 0.318 $\\pm$ 0.027 |\n| $\\lambda_{\\text{val}} = 0.5$ (arithmetic) | 0.702 $\\pm$ 0.078     | 0.857 $\\pm$ 0.047     | 0.295 $\\pm$ 0.017     |\n| $\\lambda_{\\text{val}} = 0.7$ (arithmetic) | 0.678 $\\pm$ 0.085 | 0.851 $\\pm$ 0.041 | 0.299 $\\pm$ 0.026     |\n\n**Q: Extensive comparisons with other baseline methods.**\n\nWe encourage readers to refer to the *Extensive comparisons with other baseline methods* section in the global response.\n\n**Q: Discussion of PALP**\n\nWe encourage readers to refer to the *Discussion of PALP* section in the global response.\n\n**Q: Table values seem to be inconsistent**\n\nTable 1 results from $\\lambda_{\\text{val}} = 0.5$ (Details show in Table 3). Table 2 results from $\\lambda_{\\text{val}} = 0.3$, which is our default setting.\n\n**Q: Unclear about the rationale behind adding \"+1\" to the similarity calculations.**\n\nFirstly, harmonic mean only supports elements that are non-negative [1]. Secondly, our measure of alignment is $\\text{CosSim}(u, v) \\in [-1, +1]$. Therefore, we add 1 then divide the sum by 2 to ensure each element is between 0 and 1.\n\n[1] Scipy Library. https://github.com/scipy/scipy/blob/v1.14.0/scipy/stats/_stats_py.py#L215-L307", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zloAsa4Vf5", "reviewer": "Reviewer_BTXt", "summary": "This paper proposes a Reward Preference Optimization (RPO) method, introducing the 𝜆-harmonic reward function to address overfitting and accelerate the fine-tuning process in subject-driven text-to-image generation. Experimental results demonstrate the effectiveness of the proposed approach.", "questions": "Please see weaknesses.", "limitations": "no", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. By introducing the λ-harmonic reward function, this model can achieve simple and efficient subject-driven text-to-image generation.\n\n2. Quantitative comparisons demonstrate that this model achieves superior text-alignment results.", "weaknesses": "1. The novelty of the proposed method is limited for the following reasons. First, although this paper uses the weighted harmonic mean [ref-1] as the reward function, it is a classical Pythagorean mean. Second, the loss 𝐿_𝑝𝑒𝑟 is a simple variation of the binary cross-entropy and DPO.\n\n2. The experimental results do not effectively demonstrate the advantages of the proposed method. Although the proposed RPO method performs better in text-alignment in quantitative comparisons, this improvement is not evident in Figure 3. Specifically, in the first and second rows, the terms \"eating\" and \"on table\" do not appear to be closely aligned.\n\n3. The comparisons are unfair since this paper does not compare the proposed method with any reinforcement methods, such as DPO.\n\n4.  The ablation studies show the effectiveness of the loss 𝐿_𝑟𝑒𝑓. However, it is not defined in this paper.\n\n5. More details about the CLIP and DINO models need to be provided, as stronger models may yield more detailed results, both in your findings and in compared results.\n\n[ref-1] Weighted Harmonic Means, Complex Anal. Oper. Theory (2017) 11:1715–1728.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and thoughtful feedback. Please see our responses to your specific questions below.\n\n **Q: Limited novelty of reward function and loss.**\n\nYes, the $\\lambda$-harmonic reward function is indeed a classical Pythagorean mean with weights. In fact, when $\\lambda = 0.5$, it is exactly the classical Pythagorean mean. The novelty of our contribution comes from the combination of using this reward function along with reinforcement learning-based finetuning approach, not from the reward function alone. There are two aspects to consider in the design of the  $\\lambda$-harmonic reward function: $\\lambda$ provides flexibility to the user in skewing the relative importance of image-to-image alignment and text-to-image alignment. Also, as mentioned in  lines 139 to 143, the harmonic variant is preferred over geometric and arithmetic counterparts because we do not want the fine-tuning process to ignore text-to-image alignment since they tend to yield lower reward values. \n\nWhile $\\mathcal{L}_{\\text{pre}}$ is a variation of binary cross-entropy (BCE) and DPO, it has an important difference albeit the seemingly unimpressive formulation. Validation in DPO is done using an external model or human evaluation [1], while trained using a different approach using raw preference data. Our approach uses binary labels (hence requiring the BCE loss) sampled from the reward model to fine-tune, but the validation also comes from the same reward model. This reduces distributional shifts between training and validation. \n\nUsing the $\\lambda$-Harmonic function and our simple loss function (Equation 9), RPO allows for early stopping and requires only 3% of the negative samples used by DreamBooth, as well as fewer gradient steps, making it highly efficient.\n\n**Q: Quantitative comparisons do not show the improvement.**\n\nFor Figure 3, as we discussed in our paper (Lines 234 to 236), the first prompt is not grammatically correct: \"A dog eating a cherry bowl.\" The correct prompt should be \"A dog eating a cherry from a bowl.\" The trained model was confused by the original incorrect prompt and generated this image. As for the concern over the misalignment with \"table\" in the second row,  the image generated by our model clearly shows the object standing on a table (at the bottom of the image). Furthermore, we also provide an additional comparison in the Appendix A.3. RPO can address some of the failure cases in DreamBooth and SuTI.\n\n**Q: Comparison to DPO.**\n\nThe original DPO is not suitable for subject-driven tasks because the datasets do not contain preference labels. We introduce the $\\lambda$-harmonic function and design a variant of DPO for this task. To address your concern, we implemented a pure DPO for diffusion [2] (without image similarity loss) using preference labels for image-to-image similarity and text-to-image alignment. We chose $\\lambda_{\\text{train}} = 0.5$ because this value assigns equal weights to image-to-image similarity and text-to-image alignment. For a fair comparison, we also report the results from RPO with the same $\\lambda_{\\text{train}} = 0.5$. The results on DreamBench for these two methods are shown in the following table.\n\n| Method    | DINO      | CLIP-I    | CLIP-T    |\n| :-------- | :-------- | :-------- | :-------- |\n| DPO       | 0.338     | 0.702     | **0.334** |\n| Ours: RPO | **0.649** | **0.819** | 0.314  |\n\nThese results show that DPO can capture the text-to-image alignment from the preference labels. However, without $\\mathcal{L}_{\\text{sim}}$, DPO faces a significant overfitting problem; i.e., it achieves high text-to-image alignment but cannot preserve the subject's unique features.\n\n\n**Q: Undefined function.**\n\nThank you for pointing out the typo. **$\\mathcal{L}_{\\text{ref}}$** should have in fact been **$\\mathcal{L}_{\\text{pre}}$**.\n\n**Q: Details on CLIP and DINO.**\n\nCLIP consists of two encoders: one for the image and one for the text [3]. Suppose a dataset contains pairs of image and caption. The objective is to ensure that the encoding of the image and the encoding of the text (caption) are aligned in terms of cosine similarity. This is achieved by contrastive learning using positive and negative pairs, which naturally proves useful in our case since this cosine similarity can be used as a reward signal for both image-image and image-text alignment. DINO, on the other hand, encodes only images and in a significantly different manner [4]. Image augmentations are made with random perturbation and cropping, passed through the student and teacher models to produce representations that summarize the image. This makes a better fine-grained understanding of images than CLIP in terms of producing accurate rewards, but cannot provide signals on text-image alignment.\n\n\n[1] Rafailov, Rafael, et al. \"Direct preference optimization: Your language model is secretly a reward model.\" *Advances in Neural Information Processing Systems 36*. 2024.\n\n[2] Wallace, Bram, et al. \"Diffusion model alignment using direct preference optimization.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[3] Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" *International conference on machine learning*, PMLR. 2021.\n\n[4] Caron, Mathilde, et al. \"Emerging properties in self-supervised vision transformers.\" *Proceedings of the IEEE/CVF international conference on computer vision*. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "623c5mQ1GJ", "reviewer": "Reviewer_5xjH", "summary": "The paper introduces a novel method for generating text-to-image outputs that incorporate specific subjects from reference images.  The authors propose a λ-Harmonic reward function and a Reward Preference Optimization (RPO) algorithm, which simplifies the training process and enhances the model's ability to maintain subject fidelity while generating diverse and contextually accurate images.  The approach outperforms existing methods like DreamBooth and SuTI in various metrics, demonstrating significant improvements in training efficiency and image quality.", "questions": "How does the proposed method perform with completely unseen or highly imaginative prompts that deviate significantly from the training data?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  **Innovative Reward Function**: The introduction of the λ-Harmonic reward function is a notable advancement, providing a robust reward signal that facilitates early stopping and regularization, thus accelerating the training process.\n2.  **Efficient Training Process**: The proposed RPO method significantly reduces the computational resources required for training by only needing 3% of the negative samples used by DreamBooth and fewer gradient steps, making it highly efficient.\n3.  **Empirical Performance**: The method achieves state-of-the-art performance on the DreamBench dataset, with superior CLIP-I and CLIP-T scores, indicating strong text-to-image alignment and subject fidelity.\n4.  **Simplicity**: The approach simplifies the training pipeline by fine-tuning only the U-Net component of the diffusion model, avoiding the need for optimizing text embeddings or training a text encoder.", "weaknesses": "1.  **Limited Evaluation Metrics**: While the paper uses DINO and CLIP-I/CLIP-T scores, it would benefit from additional evaluation metrics that capture other aspects of image quality, such as perceptual quality or user satisfaction.\n2.  **Overfitting Risk**: Although the λ-Harmonic reward function helps in regularization, there is still a noted risk of overfitting, particularly in generating images with high text-to-image alignment but lower uniqueness in certain features.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and thoughtful feedback. Please see our responses to your specific questions below.\n\n**Q: Limited evaluation metrics.**\n\nLimited evaluation metrics are a common issue in subject-driven tasks. We appreciate you raising this issue and providing reference metrics, such as image quality. In the table below, we report the average aesthetic scores [1] of the real reference images in DreamBench and the average aesthetic scores obtained with the SOTA CLIP I/T lambda configuration ($\\lambda_{\\text{val}} = 0.5$) in DreamBench. We observe that RPO does not decrease the quality of images; instead, the generated images achieve slightly better quality than the reference images.\n\n| Method      | Aesthetic Score |\n| :---------- | :-------------- |\n| Real images | 5.145 $\\pm$ 0.312     |\n| Ours: RPO   | **5.208 $\\pm$ 0.327** |\n\n**Q: Overfitting risk.**\n\nWe observe this problem and propose a hypothesis in lines 228 to 231. To solve this overfitting issue, we may need to use ControlNet [2] or add an additional cross-attention layer to the reference images within the U-Net component [3]. Mathematically, using these methods, the distribution will be changed from $p_{\\theta}(x_{t - 1} | c, x_t)$ to $p_{\\theta}(x_{t - 1} | c, x_t, x_{\\text{ref}}),$ which can capture the information from the reference images during the inference phase. However, RPO has no assumptions about the model architecture. Therefore, in future work, we will combine the ControlNet or cross-attention layer approach with RPO to alleviate this overfitting risk.\n\n**Q: Unseen prompts visualization results.**\n\nWe have added 32 images generated by highly imaginative prompts to the attached PDF on global response.\n\n[1] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. \"Laion-5b: An open large-scale dataset for training next generation image-text models\", 2022.\n\n[2] Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. \"Adding conditional control to text-to-image diffusion models.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[3] Chen, Wenhu, et al. \"Subject-driven text-to-image generation via apprenticeship learning.\" Advances in Neural Information Processing Systems 36 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TtPjk4H4dJ", "reviewer": "Reviewer_u7k9", "summary": "Summary:\nThis paper proposed a method which utilized ALIGN model to calculate $\\lambda$-Harmonic reward function and combine binary cross-entropy function and DPO as the preference loss to supervise image generation models.", "questions": "No question. Just hope the authors could provide more visualization examples and discuss more about DINO scores.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strength:\n1. The method is computation-efficient while achieving the highest scores on CLIP-I and CLIP-T.\n2. Successfully extend preference optimization methods to Subject-driven image generation.", "weaknesses": "Weakness:\n1. I think there should be more visualization results showing the superiority of the proposed method. However, even considering the images in Appendix, the number of generated images is still not enough. \n2. RPO does not achieve satisfactory performances in DINO scores. The authors have explained the reason, but I still wonder whether there is any method that could prevent the loss of detailed pixel-level information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review and thoughtful feedback. Please see our responses to your specific questions below.\n\n**Q: More visual results**\n\nWe have added 32 images generated by RPO to the attached PDF on global response, specifically focusing on prompts that are both unseen in the training data and highly imaginative.\n\n**Q: Improve DINO scores**\n\nTo preserve the details of the reference images, we may need to use ControlNet [1] or incorporate an additional cross-attention layer for the reference images within the U-Net component [2]. Mathematically, these methods will modify the distribution from $p_{\\theta}(x_{t - 1} | c, x_t)$ to $p_{\\theta}(x_{t - 1} | c, x_t, x_{\\text{ref}}),$ allowing the model to capture extra information from the reference images during the inference phase. However, RPO does not have any assumptions about the model architecture. Therefore, in future work, we will integrate the ControlNet or cross-attention layer approach with RPO to improve DINO scores.\n\n[1] Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. \"Adding conditional control to text-to-image diffusion models.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[2] Chen, Wenhu, et al. \"Subject-driven text-to-image generation via apprenticeship learning.\" Advances in Neural Information Processing Systems 36 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global response

We would like to thank all the reviewers for providing high-quality reviews and constructive feedback. We are encouraged that the reviewers think our paper *"generated images presented in the paper demonstrate superior text fidelity of the proposed method. (Reviewer tGfs)"*, *"proposed reward-based model selection and regularization are reasonable and compelling (Reviewer tGfs)"*, *"significantly reduces the computational resources (Reviewer 5xjH)"*, and *"successfully extend preference optimization methods to subject-driven image generation (Reviewer u7k9)"*.

Below, we summarize our major work for this rebuttal.

- **Extensive baseline comparison.**
In response to Reviewers tGfs, Kj9e, and BTXt, we compare RPO to additional baseline methods.

- **Additional visual results.**
We generated 32 additional images for the unseen and highly imaginative prompts to address the concerns of Reviewers 5xjH and u7k9.

## Additional Response to Reviewer Kj9e

Due to the character limit for the rebuttal, we provide extensive comparisons with additional baseline methods and discuss the differences between our approach and another method, PALP [8], which focuses on balancing text-to-image alignment.

**Q: Extensive comparisons with other baseline methods.**

We compared all personalization and text-image trade-off baseline methods Reviewer Kj9e mentioned in the DreamBench dataset, and the results are shown in the following table. We highlight that our method, RPO, still achieves the highest CLIP-I and CLIP-T results across these new baselines.

| Method            | DINO      | CLIP-I    | CLIP-T    |
| :---------------- | :-------- | :-------- | :-------- |
| DisenBooth [1]    | 0.574     | 0.755     | 0.255   |
| Multi-concept [2] | **0.695** | 0.801     | 0.245     |
| ELETE [3]         | 0.652     | 0.762     | 0.255     |
| IP-Adapter [4]    | 0.608     | 0.809     | 0.274    |
| SSR-encoder [5]   | 0.612     | 0.821     | 0.308     |
| Ours: RPO         | 0.652     | **0.833** | **0.314** |

**Q: Discussion of PALP**

The PALP method [6] is designed for balancing the trade-off between personalization and text-to-image alignment. PALP involves two different phases: the personalization phase and the prompt-aligned phase. The first phase minimizes the image similarity loss. During the second phase, PALP optimizes the model by taking the gradient (Equation 6 in [6]). Here we use $\theta$ to represent the fine-tuning parameters, $\phi$ be the parameters of the pre-trained model, $\hat{x}_0(\theta)$ is a prediction of $x_0$ (Equation 15 in [7]), $\hat{x}_t(\theta) = \sqrt{\bar{\alpha}_t} \hat{x}_0(\theta) + \sqrt{1 - \bar{\alpha}_t} \epsilon$, $\epsilon \sim \mathcal{N}(0, \mathbf{I}),$ and $y_p$ is the personalized prompt with an identity token, and $y_c$ is the clean prompt, e.g., **$y_p = \text{``a photo of a [V] dog''}$** and **$y_c = \text{``a photo of a dog''}$**.

Following the idea from [8] (Appendix A.4 in [8]), we can derive the following equation:

$$\nabla_{\theta} L_{PALP} \propto \nabla_{\theta} \mathbb{D}_{\text{KL}}\big(\text{model}(\hat{x}_t(\theta) \mid y_p) \| \text{pre-trained model}(\hat{x}_t(\theta) \mid y_c)\big)$$

Thus, updating parameters by $\nabla_{\theta} L_{PALP}$ is equivalent to minimizing the KL divergence between **$p_{\theta}(\hat{x}_t(\theta) \mid y_p)$** and **$p_{\phi}(\hat{x}_t(\theta) \mid y_c)$**. PALP alleviates the text-to-image alignment by restricting the learned model to be close to the pretrained model. However, in RPO, we optimize the lower bound of RL objective function (Equation 3 in our paper).

RPO does not only include the penalty of the KL divergence between the learned probability distribution and the pretrained probability distribution, but also utilizes the reward signals. Compared to PALP, our method should be more flexible since we have no assumptions about the reward signals, and these reward signals can also be adjusted to other objectives, e.g., aesthetic score.

[1] Disenbooth: Identity-preserving disentangled tuning for subject-driven text-to-image generation, Chen et al., ICLR 2024.

[2] Multi-concept customization of text-to-image diffusion, Kumari et al., CVPR 2023.

[3] ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation, Wei et al., ICCV 2023.

[4] IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models, Ye et al.

[5] SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generation, Zhang et al., CVPR 2024.

[6] Arar, Moab, et al. PALP: Prompt Aligned Personalization of Text-to-Image Models. arXiv preprint arXiv:2401.06105 (2024).

[7] Ho, Jonathan, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems 33 (2020): 6840-6851.

[8] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In The
Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023.

Meta-review: This paper proposes the lambda-harmonic reward function and the reward preference optimization (RPO) algorithm for subject-driven text-to-image generation. Initially, reviewers raised concerns such as the lack of comparisons with more baselines and related works, the lack of evaluation metrics, insufficient visualization results, experimental analysis, limited novelty, marginal improvement, and overfitting risk. The rebuttal successfully addressed most of these concerns and all reviewers recommended positive final ratings. Considering that the proposed method is reasonable and experimental results show its effectiveness and efficiency, as appreciated by reviewers, the AC follows this unanimous recommendation. Reviewers did raise valuable concerns that should be addressed. The authors should ensure that all new experiments and discussions provided during the rebuttal phase are included in the camera-ready version.
Decision: Accept (poster)