Title: Subject-driven Text-to-Image Generation via Preference-based Reinforcement Learning
Venue: NeurIPS 2024
Accepted: yes
Keywords: Diffusion Models

Text-to-Image generation

Reinforcement Learning
Abstract: Text-to-image generative models have recently attracted considerable interest, enabling the synthesis of high-quality images from textual prompts. However, these models often lack the capability to generate specific subjects from given reference images or to synthesize novel renditions under varying conditions. Methods like DreamBooth and Subject-driven Text-to-Image (SuTI) have made significant progress in this area. Yet, both approaches primarily focus on enhancing similarity to reference images and require expensive setups, often overlooking the need for efficient training and avoiding overfitting to the reference images. In this work, we present the $\lambda$-Harmonic reward function, which provides a reliable reward signal and enables early stopping for faster training and effective regularization. By combining the Bradley-Terry preference model, the $\lambda$-Harmonic reward function also provides preference labels for subject-driven generation tasks. We propose Reward Preference Optimization (RPO), which offers a simpler setup (requiring only 3\% of the negative samples used by DreamBooth) and fewer gradient steps for fine-tuning. Unlike most existing methods, our approach does not require training a text encoder or optimizing text embeddings and achieves text-image alignment by fine-tuning only the U-Net component. Empirically, $\lambda$-Harmonic proves to be a reliable approach for model selection in subject-driven generation tasks. Based on preference labels and early stopping validation from the $\lambda$-Harmonic reward function, our algorithm achieves a state-of-the-art CLIP-I score of 0.833 and a CLIP-T score of 0.314 on DreamBench.

Subject-driven Text-to-Image Generation via Preference-based Reinforcement Learning

Abstract

Text-to-image generative models have recently attracted considerable interest, enabling the synthesis of high-quality images from textual prompts. However, these models often lack the capability to generate specific subjects from given reference images or to synthesize novel renditions under varying conditions. Methods like DreamBooth and Subject-driven Text-to-Image (SuTI) have made significant progress in this area. Yet, both approaches primarily focus on enhancing similarity to reference images and require expensive setups, often overlooking the need for efficient training and avoiding overfitting to the reference images. In this work, we present the λ -Harmonic reward function, which provides a reliable reward signal and enables early stopping for faster training and effective regularization. By combining the Bradley-Terry preference model, the λ -Harmonic reward function also provides preference labels for subject-driven generation tasks. We propose Reward Preference Optimization (RPO), which offers a simpler setup (requiring only 3% of the negative samples used by DreamBooth) and fewer gradient steps for fine-tuning. Unlike most existing methods, our approach does not require training a text encoder or optimizing text embeddings and achieves text-image alignment by fine-tuning only the U-Net component. Empirically, λ -Harmonic proves to be a reliable approach for model selection in subject-driven generation tasks. Based on preference labels and early stopping validation from the λ -Harmonic reward function, our algorithm achieves a state-of-the-art CLIP-I score of 0.833 and a CLIP-T score of 0.314 on DreamBench. Our Pytorch implementation is available at https://github.com/andrew-miao/RPO .

1 Introduction

In the evolving field of generative AI, text-to-image diffusion models [28, 13, 26, 29, 27, 23, 24] have demonstrated remarkable abilities in rendering scenes that are both imaginative and contextually

ducktoy

HarmonicReward

"...on the Moon"

...with the China Great

"...

with a volcano in the

6op

Wall in the background "

background“

DiffusionModels

..in ink wash painting

"...is wearing a superman

"...

with Egyptian Pyramids

"... is wearing glasses"

cat

style"

outfit"

in the background"

"...is swimming in the pool"

"...on a subway platform"

with the Statue of Liberty

in the background"

rendition"

appropriate. However, these models often struggle with tasks that require the portrayal of specific subjects within text prompts. For instance, if provided with a photo of your cat, current diffusion models are unable to generate an image of your cat situated in the castle of your childhood dreams. This challenge necessitates a deep understanding of subject identity. Consequently, subject-driven text-to-image generation has attracted considerable interest within the community. Chen et al. [7] have noted that this task requires complex transformations of reference images. Additionally, Ruiz et al. [23] have highlighted that detailed and descriptive prompts about specific objects can lead to varied appearances in subjects. Thus, traditional image editing approaches and existing text-to-image models are ill-suited for subject-driven tasks.

Current subject-driven text-to-image generation methods are less expressive and expensive. Textual Inversion [12] performs poorly due to the limited expressiveness of frozen diffusion models. Imagic [15] is both time-consuming and resource-intensive during the fine-tuning phase. It requires textembedding optimization for each prompt, fine-tuning of diffusion models, and interpolation between optimized and target prompts. The training process is complex and slow. These text-based methods require 30 to 70 minutes to fine-tune their models, which is not scalable for real applications. SuTI [7] proposes an in-context learning method for subject-driven tasks. However, SuTI demands half a million expert models for each different subject, making it prohibitively expensive. Although SuTI can perform in-context learning during inference, the setup of expert models remains costly. DreamBooth [23] provides a simpler method for handling subject-driven tasks. Nevertheless, DreamBooth requires approximately 1000 negative samples and 1000 gradient steps, and also needs fine-tuning of the text encoder to achieve state-of-the-art performance. Therefore, it is worthwhile to explore more efficient training methods: the setup should be as simple as possible. First, training should not include multiple optimization phases. Second, text-to-image alignment should fine-tune as few components as possible such as the UNet, but not the text encoder for each prompt. Third, faster evaluation and regularization should be enabled by early stopping based on model selection.

In this paper, we propose a λ -Harmonic reward function that enables early stopping and accelerates training. In addition, we incorporate the Bradley-Terry preference model to generate preference labels. We utilize preference-based reinforcement learning algorithms to finetune pre-trained diffusion models and to achieve text-to-image alignment without optimizing any text encoder or text embedding. The whole finetuning process including setup, training, validation, and model saving only takes 5 to 20 minutes on Cloud TPU V4. Our method, Reward Preference Optimization (RPO), only requires a

few input reference images and the finetuned diffusion model can generate images that preserve the identity of a specific subject while aligning well with textual prompts (Figure 1).

To show the effectiveness of our λ -Harmonic reward function, we evaluate RPO on diverse subjects and text prompts on DreamBench [23] and we report the DINO and CLIP-I/CLIP-T of RPO's generated images on this benchmark and compare them with existing methods. Surprisingly, our method requires a simple setup ( 3% of DreamBooth configuration) and with fewer gradient steps, but the experimental results outperform or match SOTA.

In summary, our contributions are as follows:

We introduce the λ -Harmonic reward function, which permits early-stopping to alleviate overfitting in subject-driven generation tasks and to accelerate the finetuning process.

By combining the λ -Harmonic reward function and a preference model, we present RPO, which only requires a cheap setup, but still can provide high quality results.

We evaluate RPO and show the effectiveness of the λ -Harmonic function with diverse subjects and various prompts on DreamBench. We achieve results comparable to SOTA.

2 Related Works

Ruiz et al. [23] formulated a class of problems called subject-driven generation , which refers to preserving the appearance of a subject contextualized in different settings. DreamBooth [23] solves the issue of preserving the subject by binding it in textual space with a unique identifier for the subject in the reference images, and simultaneously generating diverse backgrounds by leveraging prior class-specific information previously learned. A related work that could possibly perform the same task is textual inversion [12]. However, its original objective is to produce a modification of the subject or property marked by a unique token in the text. While it can be used to preserve the subject and change the background or setting, the performance is underwhelming compared to DreamBooth in various metrics [23].

The prevalent issue in DreamBooth and textual inversion is the long training time [23, 12] since gradient-based optimization has to be performed on their respective models for each subject. Subjectdriven text-to-image generator (SuTI) by [7] aims to alleviate this issue by employing apprenticeship learning. By scraping millions of images online, many expert models are trained for different clusters of images centered around different subjects, which allows an apprentice to learn quickly from the experts [7]. However, this is an incredibly resource intensive task with massive computational overhead during training.

In the field of natural language processing, direct preference optimization has found great success in large language models (LLM) [21]. By bypassing reinforcement learning from human feedback and directly maximizing likelihoods using preference data, LLMs benefit from more stable training and reduced dependency on an external reward model. Subsequently, this inspired Diffusion-DPO by [30], which applies a similar technique to diffusion models. However, this relies on a preference labelled dataset, which can be expensive to collect or not publicly available for legal reasons.

Fortunately, there are reward models that can serve as functional substitutes such as CLIP [19] and ALIGN [14]. ALIGN has a dual encoder architecture that was trained on a large dataset. The encoders can produce text and image embeddings, which allows us to obtain pairwise similarity scores by computing cosine similarity. There are also diffusion modelling techniques that can leverage reward models. An example is denoising diffusion policy optimization (DDPO) by Black et al. [3] that uses a policy gradient reinforcement learning method to encourage generations that lead to higher rewards.

3 Preliminary

In this section, we introduce the notation and some key concepts about text-to-image diffusion models and reinforcement learning.

Text-to-Image Diffusion Models. Diffusion models [13, 26, 28, 29, 27] are a family of latent variable models of the form p θ ( x 0 ) = ∫ X p θ ( x 0: T ) d x 1: T , where the x 1 , . . . , x T are noised latent variables of the same dimensionality as the input data x 0 ∼ q ( x 0 ) . The diffusion or forward process

is often a Markov chain that gradually adds Gaussian noise to the input data and each intermediate sample x t can be written as

where α t refers to the variance schedule and ϵ t ∼ N ( 0 , I ) . Given a conditioning tensor c (often a text embedding), the core premise of text-to-image diffusion models is to use a neural network ϵ θ ( x t , c , t ) that iteratively refines the current noised sample x t to obtain the previous step sample x t -1 , This network can be trained by optimizing a simple denoising objective function, which is the time coefficient weighted mean squared error, the derivation is shown in Appendix A.1:

where t is uniformly sampled from { 1 , . . . , T } and ω ( t ) is a time dependent weight that can be simplified to 1 according to [13, 27, 22].

Reinforcement Learning and Diffusion DPO Reinforcement Learning for diffusion models [3, 11, 30] aims to solve the following optimization problem:

where β is a hyperparameter controlling the KL-divergence between the finetuned model p θ and the pre-trained base model p base. In Equation (14) from Diffusion-DPO [30], the optimal p θ can be approximated by minimizing the negative log-likelihood:

where ϵ + and ϵ -are independent samples from a Gaussian distribution, x + t and x -t are perturbed versions of x + 0 and x -0 that depend on ϵ + and ϵ -, and x + 0 is preferred to x -0 . A detailed description is given in Appendix A.1.

Additional notation. We use x ref and x gen to represent the reference image and generated image, respectively. I ref denotes the set of reference images, and I gen is the set of generated images. P ( x ≻ ˜ x ) represents the probability that x is more preferred than ˜ x .

4 Method

We present our λ -Harmonic reward function that provides reward signals for subject-driven tasks to reduce the risk that the learned model will overfit to the reference images. Based on this reward function, we use the Bradley-Terry model to sample preference labels and a preference algorithm to finetune the diffusion model by optimizing both a similarity loss and a preference loss.

4.1 Reward Preference Optimization

In contrast to other fine-tuning applications [30, 18, 21, 20], there is no human feedback in the subject-driven text-to-image generation task. The model only receives a few reference images and a prompt with a specific subject. Hence, we first propose the λ -Harmonic reward function that can leverage the ALIGN model [14] to provide feedback based on the generated image fidelity: similarity to the given reference images and faithfulness to the text prompts.

λ -Harmonic Reward Function. The normalized ALIGN-I and ALIGN-T scores can be denoted as

where CosSim is the cosine similarity, f ϕ ( x ) is the image feature extractor and g ϕ ( c ) is the text encoder in the ALIGN model. Given a reference image set I ref , the λ -Harmonic reward function can be defined by a weighted harmonic mean of the ALIGN-I and ALIGN-T scores,

Compared to the arithmetic mean, there are two advantages to using the harmonic mean: (1) according to AM-GM-HM inequalities [10], the harmonic mean is a lower bound of the arithmetic mean and maximizing this 'pessimistic' reward can also improve the arithmetic mean of ALIGN-I and ALIGNT scores; (2) the harmonic mean is more sensitive to the smaller of the two scores, i.e., a larger reward is only achieved when both scores are relatively large.

For a simple example, consider λ = 0 . 5 . If there are two images, x and ˜ x , where the first image achieves an ALIGN-I score of 0.9 and an ALIGN-T score of 0.01, and the second image receives an ALIGN-I score of 0.7 and an ALIGN-T score of 0.21, we may prefer the second image because it has high similarity to the reference images and is faithful to the text prompts. However, using the arithmetic mean would assign both images the same reward of 0.455. In contrast, the harmonic mean would assign the first image a reward of 0.020 and the second image a reward of 0.323, aligning with our preferences. During training, we set λ train = 0 , which means the reward model will focus solely on text-to-image alignment because the objective function consists only of a loss for image-to-image alignment. Note that we set λ val to a different value for validation, which evaluates the fidelity of the subject and faithfulness of the prompt. Details can be found in Section 5.

Dataset. The set of images for subject-driven generative tasks can usually be represented as I = I ref ∪ I gen, where I gen is the image set generated by the base model. DreamBooth [23] requires two different prompts, c and c pr, where c includes a reference to the subject while c pr refers to the prior class of the subject but not the subject. For example, c can be 'a photo of [V] dog' and c pr can be 'a photo of a dog' , where '[V]' is a unique token that refers to the subject and dog is the prior class of the subject. DreamBooth then uses c pr to generate a variety of images I gen in the prior class to avoid overfitting to the reference images via a regularizer. Typically, the size of the set of generated images is around 1000, i.e., |I gen | = 1000 , which is time-consuming and space-intensive in real applications. However, the diffusion model can only maximize the similarity score and still receives a high reward based on this uninformative prompt c pr. Our method aims to balance the trade-off between similarity and faithfulness. Thus, for efficiency, we introduce 8 novel training prompts, c mod of the form 'a [V] [class noun] [modification]' where modification includes artistic style transfer, re-contextualization , and accessorization . For example, c mod can be 'a [V] dog is on the Moon' . These training prompts can be pre-specified or generated by other Large Language Models 2 . The full list of training prompts is provided in the supplementary material (Figure 6). We feed these training prompts to the base model and generate 4 images for each training prompt, i.e., |I gen | = 32 .

Once we obtain reward signals, we adopt the Bradley-Terry model [4] to generate preference labels. In particular, given a tuple ( x ref , x gen , c mod ) , we sample preference labels y from the following probability model:

Learning. The learning objective function consists of two parts - similarity loss and preference loss. The similarity loss is designed to minimize the KL divergence between the distribution of reference images and the learned distribution p θ ( x ) , which is equivalent to minimizing:

The preference loss aims to capture the preference signals and fit the preference model, Eq. (6). Therefore, we use binary cross-entropy as the objective function for the preference loss. Combining

Reference Images

Generated Images

SimilarLoss

Diffusion Models

PreferenceLoss

-Harmonic Reward

"A [M] dog is on

Base Models

the DPO objective function in Eq. 4, the loss function can be written as follows:

where

and t ∼ U{ 1 , . . . , T } and ϵ ref , ϵ gen ∼ N ( 0 , I ) . Combining these two loss functions together, the objective function for finetuning is written as

Figure 2 presents an overview of the training method, which includes the base model generated samples, the ALIGN reward model, and the preference loss. Note that L pref serves as a regularizer for approximating the text-to-image alignment policy. Conversely, DreamBooth [23] adopts D KL ( p base ( x t -1 | x t , c ) | p θ ( x t -1 | x t , c )) as its regularizer, which cannot guarantee faithfulness to the text-prompt. Based on this loss function and preference model, we only need a few hundred gradient steps and a small set size of |I gen | to achieve results that are comparable to, or even better than, the state of the art. The fine-tuning process, which includes generating images, training, and validation, takes about 5 to 20 minutes on a single Google Cloud Platform TPUv4-8 (32GB) for Stable Diffusion.

5 Experiments

In this section, we present the experimental results demonstrated by RPO. We investigate several questions. First, can our algorithm learn to generate images that are faithful both to the preference images and to the textual prompts, according to preference labels? Second, if RPO can generate high-quality images, which part is the key component of RPO: the reference loss or the early stopping by the λ -Harmonic reward function? Third, how do different λ val values used during validation affect performance in RPO? We refer readers to Appendix A.2 for details on the experimental setup, Appendix A.7 for the skill set of RPO, Appendix A.4 for the limitations of the RPO algorithm, and Appendix A.5 for future work involving RPO.

5.1 Dataset and Evaluation

DreamBench. In this work, we use the DreamBench dataset proposed by DreamBooth [23]. This dataset contains 30 different subject images including backpacks, sneakers, boots, cats, dogs, and toy, etc. DreamBench also provides 25 various prompt templates for each subject and these prompts are requiring the learned models to have the following abilities: re-contextualization, accessorization, property modification, and attribute editing.

Evaluation Metrics. We follow DreamBooth [23] and SuTI [7] to report DINO [5] 3 and CLIP-I [19] for evaluating image-to-image similarity score and CLIP-T [19] for evaluating the text-to-image similarity score. We also use our λ -Harmonic reward as a evaluation metric for the overall fidelity and the default value of λ = 0 . 3 . For evaluation, we follow DreamBooth [23] and SuTI [7] to generate 4 images per prompt, 3000 images in total, which provides a robust evaluation.

Baseline algorithms. DreamBooth [23]: A test-time fine-tuning method. This algorithm requires approximately |I gen | = 1000 and 1000 gradient steps to finetune the UNet and text-encoder components. SuTI [7]: A pre-trained method that requires half a million expert models and introduces cross-attention layers into the original diffusion models. Textual Inversion [12]: A text-based method that optimizes the text embedding but freezes the diffusion models. Re-Imagen [8]: An information retrieval-based algorithm that modifies the backbone network architectures and introduces crossattention layers into the original diffusion models. DisenBooth [6]: A test-time fine-tuning method that generates subject-driven images by optimizing textual identity-preserving embeddings. Custom Diffusion [16]: a test-time fine-tuning method, focused on efficient training by optimizing the key and value projection matrices in the cross-attention layers of diffusion models. ELITE [31]: A test-time fine-tuning approach consisting of two stages: one for training the textual embeddings and another for preserving identity. IP-Adapter [32]: a pretrained method where the decoupled cross-attention layer captures the textual signal while integrating reference images. SSR-Encoder [33]: also a pretrained method that highlights selective regions and extracts detailed features for subject-driven image generation.

5.2 Results

Quantitative Results. We begin by addressing the first question. We use a quantitative evaluation to compare RPO with other existing methods on three metrics (DINO, CLIP-I, CLIP-T) in DreamBench to validate the effectiveness of RPO. The experimental results on DreamBench is shown in Table 1. We observe that RPO can perform better or on par with existing works on all three metrics. Compared to DreamBooth, RPO only requires 3% of the negative samples, but RPO can outperform DreamBooth on the CLIP-I and CLIP-T scores by 3% given the same backbone. Our method outperforms all baseline algorithms in the CLIP-T score, establishing a new SOTA result. This demonstrates that RPO, by solely optimizing UNet through preference labels from the λ -Harmonic reward function, can generate images that are faithful to the input prompts. Similarly, our CLIP-I score is also the highest, which indicates that RPO can generate images that preserve the subject's visual features. In terms of the DINO score, our method is almost the same as DreamBooth when using the same backbone. We conjecture that the reason RPO achieves higher CLIP scores and lower DINO score is that the λ -Harmonic reward function prefers to select images that are semantically similar to the textual prompt, which may result in the loss of some unique features in the pixel space.

Qualitative Results. We use the same prompt as SuTI [7], and the generated images are shown in Figure 3. RPO generates images that are faithful to both reference images and textual prompts. We noticed a semantic mistake in the first prompt used by SuTI [7]; it should be A dog eating a cherry from a bowl . Furthermore, each reference bowl image contains blueberries, and the ambiguous prompt caused the RPO-trained model to become confused during the inference phase. However, RPO still preserves the unique appearance of the bowl. For instance, while the text on the bowl is incorrect or blurred in the SuTI and DreamBooth results, RPO accurately retains the words Bon Appetit from the reference bowl images. Although existing methods can produce images highly faithful to the reference images, they may not align as well with the textual prompts. We also

A dog eating a cherry bowl

Two thin-neck vases on table

A tiny dog is having a haircut

SuT1

Re-Imagen

DreamBooth

RPO

provide an example in Appendix A.3 that shows how RPO can handle the failure case observed in DreamBooth and SuTI. Additionally, we find that the RPO model can generate reasonable images even for highly imaginative prompts, as shown in Appendix A.7 (Figure 16 and Figure 17). These images demonstrate that the RPO-trained model does not overfit the training data. Instead, the λ -Harmonic function provides a method for selecting a model capable of text-to-image alignment, even when the textual prompts are highly imaginative.

5.3 Ablation Study and Method Analysis

Preference Loss and λ -Harmonic Ablation. We investigate the second question through an ablation study. Two regularization components are introduced into RPO: reference loss as a regularizer and early stopping by λ val-Harmonic reward function. Consequently, we compare four methods: (1) Pure L sim, which only minimizes the image-to-image similarity loss L sim ; (2) L pref w/o early stopping, which employs L pref as a regularizer but omits early stopping by λ val-Harmonic reward function; (3) Early stopping w/o L pref, which uses λ val-Harmonic reward function as a regularization method but excludes L pref; (4) Full RPO, which utilizes both L pref and early stopping by the λ val-Harmonic reward function. We choose the default value λ val = 0 . 3 in this ablation study.

Table 2 lists the evaluation results of these four methods on DreamBench. We observe that without early stopping, L pref can still prevent overfitting to the reference images and improve text-to-image alignment, though the regularization effect is weak. Specifically, the 0 . 3 -Harmonic only shows a

Gradientsteps(Gs)

ReferenceImages

GS=80,

GS=160,

GS=240,

GS=320,

GS=400,

Reward=0.654Reward=0.666Reward=0.661Reward=0.654Reward=0.648

on theMoon"

backpack

GS=80,

GS=160,

GS=240,

GS=320,

GS=400,

Reward=0.660Reward=0.657Reward=0.653Reward=0.652

Reward=0.652

"... inVincent

VanGogh

rendition"

cat

ReferenceImages

6op

Aval = 0.5

marginal improvement of 0.003 over pure L sim and 0.001 over early stopping without L pref. The early stopping facilitated by the λ val-Harmonic reward function plays a crucial role in counteracting overfitting, helping the diffusion models retain the ability to generate high-quality images aligned with textual prompts. To provide a deeper understanding of the λ -Harmonic reward validation, we present two examples from during training in Figure 4, covering both objects and live subjects. We found that the model tends to overfit at a very early stage, i.e., within 200 gradient steps, where λ -Harmonic can provide correct reward signals for the generated images. For the backpack subject, the generated image receives a low reward at gradient step 80 due to its lack of fidelity to the reference images. However, at gradient step 400, the image is overfitted to the reference images, and the model fails to align well with the input text, resulting in another low reward. λ -Harmonic assigns a high reward to images that are faithful to both the reference image and textual prompts.

Impact of λ val . We examine the third question by selecting different λ val values from the set { 0 . 3 , 0 . 5 , 0 . 7 } as the validation parameters for the λ -Harmonic reward. According to Equation 5, we believe that as λ val increases, the λ -Harmonic reward function will give higher weight to the image-to-image similarity score. This will make the generated images more closely resemble the reference images, however, there is also a risk of overfitting. Table 3 shows the results of three different λ val values on DreamBench. As we expected, a larger λ val makes the images better preserve the characteristics of the reference images, but it also reduces the text-to-image alignment score. Figure 5 shows us an example. In this example, different λ val values lead to different outcomes due to varying strengths of regularization. A smaller λ val = 0 . 3 can generate more varied results, but seems somewhat off from the reference images. λ val = 0 . 5 preserves some characteristics beyond the original subject, such as the sofa, but also maintains alignment between text and image. However, when λ val = 0 . 7 is chosen as an excessively large value, the model actually overfits to the reference images, ignoring the prompts. We have additional comparisons, including different λ train's, different Pythagorean means, and aesthetic scores [25] in Appendix A.3.

6 Conclusion

We introduce the λ -Harmonic reward function to derive preference labels and employ RPO to finetune the diffusion model for subject-driven text-to-image generation tasks. Additionally, the λ -Harmonic reward function serves as a validation method, enabling early stopping to mitigate overfitting to reference images and speeding up the finetuning process.

Acknowledgments and Disclosure of Funding

We thank Shixin Luo and Hongliang Fei for providing constructive feedback. This work was supported by a Google grant with Cloud TPUs from Google's TPU Research Cloud (TRC). We also thank the Vector Institute, the Canada CIFAR AI Chair program and the Natural Sciences and Engineering Research Council of Canada for their support.

References

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless

withtheCNTower inthebackground

assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862 , 2022.

Charlotte Bird, Eddie Ungless, and Atoosa Kasirzadeh. Typology of risks of generative text-toimage models. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society , pages 396-410, 2023.

Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.13301 , 2023.

Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 39(3/4):324-345, 1952.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

Hong Chen, Yipeng Zhang, Simin Wu, Xin Wang, Xuguang Duan, Yuwei Zhou, and Wenwu Zhu. Disenbooth: Identity-preserving disentangled tuning for subject-driven text-to-image generation. arXiv preprint arXiv:2305.03374 , 2023.

Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Ruiz, Xuhui Jia, Ming-Wei Chang, and William W Cohen. Subject-driven text-to-image generation via apprenticeship learning. Advances in Neural Information Processing Systems , 36, 2024.

Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W Cohen. Re-imagen: Retrievalaugmented text-to-image generator. arXiv preprint arXiv:2209.14491 , 2022.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research , 24(240):1113, 2023.

Dušan Djuki´ c, Vladimir Jankovi´ c, Ivan Mati´ c, and Nikola Petrovi´ c. The IMO compendium: a collection of problems suggested for the International Mathematical Olympiads: 1959-2004 , volume 119. Springer, 2006.

Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for finetuning text-to-image diffusion models. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618 , 2022.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning , pages 4904-4916. PMLR, 2021.

Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6007-6017, 2023.

Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multiconcept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1931-1941, 2023.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730-27744, 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. From r to q ∗ : Your language model is secretly a q-function. arXiv preprint arXiv:2404.12358 , 2024.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36, 2024.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22500-22510, 2023.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems , 35:36479-36494, 2022.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems , 35:25278-25294, 2022.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning , pages 2256-2265. PMLR, 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations , 2021.

Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems , 32, 2019.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. arXiv preprint arXiv:2311.12908 , 2023.

Yuxiang Wei, Yabo Zhang, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1594315953, 2023.

Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721 , 2023.

Yuxuan Zhang, Yiren Song, Jiaming Liu, Rui Wang, Jinpeng Yu, Hao Tang, Huaxia Li, Xu Tang, Yao Hu, Han Pan, et al. Ssr-encoder: Encoding selective subject representation for subjectdriven generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8069-8078, 2024.

A Appendix

A.1 Background

Reinforcement Learning. In Reinforcement Learning (RL), the environment can be formalized as a Markov Decision Process (MDP). An MDP is defined by a tuple ( S , A , P, R, ρ 0 , T ) , where S is the state space, A is the action space, P is the transition function, R is the reward function, ρ 0 is the distribution over initial states, and T is the time horizon. At each timestep t , the agent observes a state s t and selects an action a t according to a policy π ( a t | s t ) , and obtains a reward R ( s t , a t ) , and then transit to a next state s t +1 ∼ P ( s t +1 | s t , a t ) . As the agent interacts with the MDP, it produces a sequence of states and actions, which is denoted as a trajectory τ = ( s 0 , a 0 , s 1 , a 1 , . . . , s T -1 , a T -1 ) . The RL objective is to maximize the expected value of cumulative reward over the trajectories sample from its policy:

Diffusion models. [26, 13, 29] consider a variance schedule { β t } T t =1 . For each data point x 0 ∼ p data ( x 0 ) , a Markov chain is constructed by transition probability p ( x t | x t -1 ) = N ( x t ; √ 1 -β t x t -1 , β t I ) . Consequently, p α t ( x t | x 0 ) = N ( x t ; √ α t x 0 , (1 -α t ) I ) , where α t = ∏ t t ′ =1 (1 -β t ) . [28, 29] show that training diffusion models, ϵ θ ( x t , t ) , is equivalent to score matching, i.e., training a score function s θ ( x t , t ) :

Diffusion MDP We formalize the denoising process as the following Diffusion MDP :

where r ( x 0 , c ) can be a reward signal for the denoised image. The transition kernel is deterministic, i.e., P ( s T -t +1 | s T -t , a T -t ) = ( a T -t , c ) = ( x t -1 , c ) . For brevity, the trajectory τ is defined by ( x 0 , x 1 , . . . , x T ) . Hence, the trajectory distribution for given diffusion models can be denoted as the joint distribution p θ ( x 0: T | c ) . In particular, the RL objective function for finetuning diffusion models can be re-written as the following optimization problem:

where β is a hyperparameter controlling the KL-divergence between the finetune model p θ and the base model p base. This constraint prevents the learned model from losing generation diversity and falling into 'mode collapse' due to a single high cumulative reward result. In practice, this KL-divergence has become a standard constraint in large language model finetuning [1, 18, 21].

Given preference labels and following the Direct Preference Optimization (DPO) framework [30, 21, 20], we can approximate the optimal policy p ∗ by minimizing the upper bound:

prompt_list=[

f'a {unique_token}{subject_token} in Vincent Van Gogh rendition',

f'a {unique_token} {subject_token}wearing a green outfit',

f'a {unique_token}{subject_token}on the Moon',

f'a {unique_token} {subject_token} in Vincent Van Gogh rendition',

f'a green {unique_token}{subject_token}',

f'a {unique_token} {subject_token}with a China Great Wall in the background',

f'a{unique_token} {subject_token}with a China Great Wall in the background',

f'a {unique_token} {subject_token} is swimming in a pool',

f'a {unique_token} {subject_token}on top of wooden roof',

f'a

a{unique_token} {subject_token}on a subway platform',

f'a{unique_token}(subject_token}with the Statue of Liberty in the background'

f'a {unique_token} {subject_token}with the Statue of Liberty in the background'

f'a{unique_token}{subject_token}with a volcano in the background',

f'a {unique_token} {subject_token}on the Moon',

f'a {unique_token}{subject_token}on a subway platform',

f'a green {unique_token}{subject_token}',

Objectprompts

Livesubjectprompts

where ϵ + and ϵ -are independent samples from a Gaussian distribution, x + t and x -t are perturbed versions of x + 0 and x -t that depend on ϵ + and ϵ -, and x + 0 is preferred to x -0 . A detailed derivation can be found in [30].

A.2 Experimental Details

Training Prompts. We collect 8 training prompts: 6 re-contextualization, 1 property modification and 1 artistic style transfer for objects. 5 re-contextualization, 1 attribute editing, 1 artistic style transfer and 1 accessorization for live subjects. The trainig prompts are shown in Figure 6.

Experimental Setup During training we use λ train = 0 for the λ -Harmonic reward function to generate the preference labels. We evaluate the model performance by λ val-Harmonic per 40 gradient steps during training time and save the checkpoint that achieve the highest validation reward. Table 4 lists the common hyperparameters used in the generating skill set and the λ val used in the default setting.

A.3 Additional Comparisons

Comparison to DreamBooth and SuTI. We observe RPO that can be faithful to both reference images and the input prompt. To investigate whether RPO can provide better quality than DreamBooth and SuTI, we follow SuTI paper and pick the robot toy as an example to demonstrate the effectiveness of RPO (Figure 7). In this example, DreamBooth is faithful to the reference image but it does not provide a good text-to-image alignment. SuTI provides an result that is fidelty to textual prompt. However, SuTI lacks fidelity to the reference image, i.e., the robot should stand with its wheels instead of legs. [7] use DreamBooth to finetune SuTI (Dream-SuTI) further to solve this failure case. Instead, RPO can generate an image not only faithful to the reference images but also align well with the input prompts.

Comparison to different λ val . We have also added more samples for comparison of different λ val values (see Figure 8). We find that λ val = 0 . 5 encourages the learned model to retain output diversity while still aligning with textual prompts. However, the generated images invariably contain a sofa, which is unrelated to the subject images. This occurs because every training image includes a sofa. A large λ val weakens the regularization strength and leads to overfitting. Nevertheless, a small value of

λ val can potentially eliminate background bias. We highlight that this small λ val not only encourages diversity but also mitigates background bias in identity preservation and enables the model to focus on the subject.

Comparison to different λ train . During our experiments, we use the default value for λ train = 0 . Thus, we also investigate how different λ train will effect the performance of RPO. We test three different values of λ train = { 0 . 3 , 0 . 5 , 0 . 7 } and report their results in DreamBench (Table 5). We observe that the image-to-image alignment increases with larger , but the text-to-image alignment decreases because the preference model tends to favor alignment with reference images and ignores the prompt alignment.

Comparison to different Pythagorean means. Further, we examine different Pythagorean means of ALIGN-I and ALIGN-T scores. We replace the harmonic mean as the arithmetic mean and test three different λ val = { 0 . 3 , 0 . 5 , 0 . 7 } . We report the results of the arithmetic mean reward in Table 6. The arithmetic mean is not very sensitive to smaller values; it tends to maximize higher values to achieve a better final score. In practice, ALIGN-I will receive a higher value (this effect can be seen from CLIP-I and CLIP-T in Tables 1 to 3). Thus, the model will tend to optimize image-to-image alignment and achieve good results on DINO and CLIP-I but have a lower score for text-to-image alignment.

Comparison to DPO The original DPO is not suitable for subject-driven tasks because the datasets do not contain preference labels. We introduce the λ -harmonic function and design a variant of DPO for this task. We implement a DPO diffusion [30] (without similarity loss) using preference labels for image-to-image similarity and text-to-image alignment. We choose λ train = 0 . 5 since this value assigns equal weights to the image-to-image similarity and text-to-image alignment. For a fair comparison, we also repost the results from RPO with the same λ train value. The results on DreamBench for these two methods are shown in Table 7. These results show that DPO can capture the text-to-image alignment from the preference labels. However, without L sim, DPO faces a significant overfitting problem; i.e., it achieves high text-to-image alignment but cannot preserve the subject's unique features.

Aesthetic comparison. Limited evaluation metrics are a common issue in subject-driven tasks. In Table 8, we report the average aesthetic scores [25] of the real reference images in DreamBench and the average aesthetic scores obtained with the best CLIP I/T λ configuration ( λ val = 0 . 5 ) in DreamBench. We observe that RPO does not decrease the quality of images. Instead, the generated images achieve slightly better quality than the reference images.

Reference

Prompttext

DreamBooth

SuTI

Dream-SuTI

RPO

Arobotisstanding

onthestreetofa

neon-litcitywith

high-risebuildings.

Aval = 0.3

ReferenceImages

Aval= 0.5

..withtheCNTower

inthebackground

6op

Aval = 0.7

A.4 Limitations

Figure 9 illustrates some failure examples of RPO. The first issue is context-appearance entanglement. In Figure 9, the learned model correctly understands the keyword blue house ; however, the appearance of the subject is also altered by this context, e.g. the color of the backpack has changed, and there is a house pattern on the backpack. The second issue is incorrect contextual integration. We conjecture that this failure is due to the rarity of the textual prompt. For instance, imagining a cross between a chow chow and a tiger is challenging, even for humans. Third, although RPO provides regularization, it still cannot guarantee the avoidance of overfitting. As shown in Figure 9, this may be because, to some extent, the visual appearance of sand and bed sheets is similar, which has led to overfitting issues in the model.

A.5 Future Work

The overfitting failure case leads to a future work direction: can online RL improve regularization and avoid overfitting? The second direction for future work involves implementing the LoRA version for RPO and comparing it to LoRA DreamBooth. Last but not least, we aim to identify or construct open-source, subject-driven datasets for comparison. Currently, DreamBench is the only open-source dataset we can access and evaluate for model performance. Nevertheless, we should create a larger dataset that includes more diverse subjects to verify the effectiveness of different algorithms.

A.6 Broader Impacts

The nature of generative image models is inherently subjected to criticism on the issue of privacy, security and ethics in the presence of nefarious actors. However, the core of this paper remains purely on an academic mission to extend the boundaries of generative models. The societal consequences of democratizing such powerful generative models is more thoroughly discussed in other papers. For example, Bird et al. outlines the classes of risks in text-to-image models [2] in greater detail and should be directed to such papers. Nevertheless, we play our part in the management of such risks by avoiding the use of identifiable parts of humans in the reference sets.

A.7 Skill Set

The skill set of the RPO-trained model is varied and includes re-contextualization (Figure 10), artistic style generation (Figure 11), expression modification (Figure 12), subject accessorization (Figure 13), color editing (Figure 14), multi-view rendering (Figure 15), novel hybrid synthesis (Figure 16), and novel prompt generation (Figure 17).

Context-apperance entanglement

Incorrect contextual integration

A crossofa[M] chow

chow and a tiger

Overfitting

A [M] pink sunglasses in

theSaharaDesert

Reference Images

backpack

A[M]backpackinthe

A[M]backpackunder

A[M]backpackin

GrandCanyon

starrysky

TimesSquare

clayteapot

A[M]clayteapotfloating

A[M]clayteapotona

A[M]clayteapotona

inthewater

glasstable

woodentabletogether

withsalmonsushi.

ReferenceImages

VincentvanGogh

PabloPicasso

ClaudeMonet

ReferenceImages

depressed

joyous

sleepy

screaming

ReferenceImages

XNURSE

chefoutfit

nurseoutfit

policeoutfit

Supermanoutfit

witchoutfit

Ironmanoutfit

angelwings

fire-fighteroutfit

ReferenceImages

女

blue

pink

purple

yellow

ReferenceImages

bottomview

backview

sideview

topview

ReferenceImages

lion panda

NeurIPS Paper Checklist

The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.

Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:

You should answer [Yes] , [No] , or [NA] .

[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.

Please provide a short (1-2 sentence) justification right after your answer (even for NA).

The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.

The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.

IMPORTANT, please:

Delete this instruction block, but keep the section heading 'NeurIPS paper checklist" ,

Keep the checklist subsection headings, questions/answers and guidelines below.

Do not modify the questions and only use the provided macros for your answers .

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The scope and contributions are clearly defined in the abstract and introduction. Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: The limitations of our technique is discussed in the Appendix where we show certain cases that are not desirable.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our works focus on the experimental/empirical performance of subject-driven generation only. Due to the page constraints and the scope, we decided not to pursue theoretical results for now.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The details are mentioned in the Appendix A.2 where it states specific configurations needed to reproduce the main experiments, including the prompts.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Our paper uses the DreamBench dataset for the experiments, which is publicly available for anyone interested in reproducing the main results. The code will be submitted as a ZIP file.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The details are mentioned in the Appendix A.2. In Table 3, the λ which is an important hyperparameter in our experiments, is stated along with the metrics.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Experiments are conducted for each subject across various prompts, where a metric is computed individually. The standard deviation over all prompts and subjects is provided in the ablation study in the experiment section.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The compute resources used in this paper is described in Method section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: For our research and experiments, we use public datasets and models that have been used in prior published works. Our research does not involve direct interaction with human subjects.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Given the nature of our research, it necessitates the general discussion of the potential negative societal impacts which takes place in Appendix A.6.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [No]

Justification: Safeguards would fall outside the scope of our research and we mainly focus on subjects that are inanimate. No experiments explicitly use personally identifiable features of humans as a subject.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The authors of Stable Diffusion, DreamBench and SuTI are cited and credited in this paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The asset mainly consist of code which contains instruction on how to run it.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: We do not crowdsource images by ourselves; we use images from DreamBench which has been vetted through.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: As mentioned previously, there is no crowdsourcing involved.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
Summary:
This paper proposed a method which utilized ALIGN model to calculate $\lambda$-Harmonic reward function and combine binary cross-entropy function and DPO as the preference loss to supervise image generation models.

STRENGTHS:
Strength:
1. The method is computation-efficient while achieving the highest scores on CLIP-I and CLIP-T.
2. Successfully extend preference optimization methods to Subject-driven image generation.

WEAKNESSES:
Weakness:
1. I think there should be more visualization results showing the superiority of the proposed method. However, even considering the images in Appendix, the number of generated images is still not enough. 
2. RPO does not achieve satisfactory performances in DINO scores. The authors have explained the reason, but I still wonder whether there is any method that could prevent the loss of detailed pixel-level information.

QUESTIONS:
No question. Just hope the authors could provide more visualization examples and discuss more about DINO scores.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel method for generating text-to-image outputs that incorporate specific subjects from reference images.  The authors propose a λ-Harmonic reward function and a Reward Preference Optimization (RPO) algorithm, which simplifies the training process and enhances the model's ability to maintain subject fidelity while generating diverse and contextually accurate images.  The approach outperforms existing methods like DreamBooth and SuTI in various metrics, demonstrating significant improvements in training efficiency and image quality.\n\nSTRENGTHS:\n1.  **Innovative Reward Function**: The introduction of the λ-Harmonic reward function is a notable advancement, providing a robust reward signal that facilitates early stopping and regularization, thus accelerating the training process.\n2.  **Efficient Training Process**: The proposed RPO method significantly reduces the computational resources required for training by only needing 3% of the negative samples used by DreamBooth and fewer gradient steps, making it highly efficient.\n3.  **Empirical Performance**: The method achieves state-of-the-art performance on the DreamBench dataset, with superior CLIP-I and CLIP-T scores, indicating strong text-to-image alignment and subject fidelity.\n4.  **Simplicity**: The approach simplifies the training pipeline by fine-tuning only the U-Net component of the diffusion model, avoiding the need for optimizing text embeddings or training a text encoder.\n\nWEAKNESSES:\n1.  **Limited Evaluation Metrics**: While the paper uses DINO and CLIP-I/CLIP-T scores, it would benefit from additional evaluation metrics that capture other aspects of image quality, such as perceptual quality or user satisfaction.\n2.  **Overfitting Risk**: Although the λ-Harmonic reward function helps in regularization, there is still a noted risk of overfitting, particularly in generating images with high text-to-image alignment but lower uniqueness in certain features.\n\nQUESTIONS:\nHow does the proposed method perform with completely unseen or highly imaginative prompts that deviate significantly from the training data?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper proposes a Reward Preference Optimization (RPO) method, introducing the 𝜆-harmonic reward function to address overfitting and accelerate the fine-tuning process in subject-driven text-to-image generation. Experimental results demonstrate the effectiveness of the proposed approach.

STRENGTHS:
1. By introducing the λ-harmonic reward function, this model can achieve simple and efficient subject-driven text-to-image generation.

2. Quantitative comparisons demonstrate that this model achieves superior text-alignment results.

WEAKNESSES:
1. The novelty of the proposed method is limited for the following reasons. First, although this paper uses the weighted harmonic mean [ref-1] as the reward function, it is a classical Pythagorean mean. Second, the loss 𝐿_𝑝𝑒𝑟 is a simple variation of the binary cross-entropy and DPO.

2. The experimental results do not effectively demonstrate the advantages of the proposed method. Although the proposed RPO method performs better in text-alignment in quantitative comparisons, this improvement is not evident in Figure 3. Specifically, in the first and second rows, the terms "eating" and "on table" do not appear to be closely aligned.

3. The comparisons are unfair since this paper does not compare the proposed method with any reinforcement methods, such as DPO.

4.  The ablation studies show the effectiveness of the loss 𝐿_𝑟𝑒𝑓. However, it is not defined in this paper.

5. More details about the CLIP and DINO models need to be provided, as stronger models may yield more detailed results, both in your findings and in compared results.

[ref-1] Weighted Harmonic Means, Complex Anal. Oper. Theory (2017) 11:1715–1728.

QUESTIONS:
Please see weaknesses.

LIMITATIONS:
no
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method for generating personalized images from text using the λ-Harmonic reward function in a system called Reward Preference Optimization (RPO). Specifically, RPO fine-tunes a pre-trained text-to-image diffusion model using a reinforcement-learning-based objective concerning with the harmonic averaged value of text similarity and image similarity. This objective is used in a reinforcement learning framework for diffusion models. The results demonstrate that the proposed techniques improve the performance of the DreamBooth baseline and alleviate the image overfitting problem.\n\nSTRENGTHS:\n* The paper is well-structured and easy to follow.\n* The proposed λ-Harmonic reward function is reasonable and has achieved better results than the baseline.\n\nWEAKNESSES:\n* The authors could introduce briefly how does the method support adjusting λ at the test time. How about the method comparing with adding classifier-based guidance?\n* The wall-clock time analysis of the proposed method against the baseline DreamBooth in training and sampling is missing.\n* The experimental analysis of why harmonic mean is used instead of arithmetic mean is missing.\n* Comparisons could be made more extensive. There are more state-of-the-art personalized generation methods nowadays like [a,b]. \n* The authors may want to compare with more baselines that also support adjusting the text-image-tradeoff at the test time [c,d,e].\n* The motivation to alleviate the image overfitting and enhance the prompt adherence has been explored in [f]. Comparisons or at least some discussions are necessary.\n\n[a] Disenbooth: Identity-preserving disentangled tuning for subject-driven text-to-image generation, Chen et al., ICLR 2024.\n\n[b] Multi-concept customization of text-to-image diffusion, Kumari et al., CVPR 2023.\n\n[c] ELITE: Encoding Visual Concepts into Textual Embeddings for Customized Text-to-Image Generation, Wei et al., ICCV 2023.\n\n[d] IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Models, Ye et al..\n\n[e] SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generation, Zhang et al., CVPR 2024.\n\n[f] PALP: Prompt Aligned Personalization of Text-to-Image Models, Tewel et al..\n\nQUESTIONS:\n* The metrics evaluations in Tables 1 and 2 appear inconsistent, despite seemingly identical parameters (λval = 0.3).\n* Additionally, in the metrics ALIGN-I and ALIGN-T, I am unclear about the rationale behind adding \"+1\" to the similarity calculations.\n\nLIMITATIONS:\nThis paper have discussed it’s limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

comments: SUMMARY:
This paper proposes a harmonic reward function for text-to-image personalization. Specifically, it uses this reward function to perform early stopping and regularization. Experimental results demonstrate the effectiveness of the proposed method.

STRENGTHS:
* Their proposed reward-based model selection and regularization are reasonable and compelling.
* The generated images presented in the paper demonstrate superior text fidelity of the proposed method.

WEAKNESSES:
* More baselines are needed. Specifically, since this paper is similar to DCO [1], a comparison with this method is needed.

[1] Lee, Kyungmin, et al. "Direct consistency optimization for compositional text-to-image personalization." arXiv preprint arXiv:2402.12004 (2024).

QUESTIONS:
What are the results when the lambda value of the harmonic reward function is not set to 0 in the preference loss during training?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13