Confidence-aware Reward Optimization for Fine-tuning Text-to-Image Models
Authors: Kyuyoung Kim
Jongheon Jeong
Minyong An
Mohammad Ghavamzadeh
Krishnamurthy Dj Dvijotham
Jinwoo Shin
Kimin Lee
Venue: ICLR 2024 poster (2024)
Keywords: text-to-image generation
overoptimization
confidence calibration
Primary area: generative models
TL;DR: We explore the issue of reward overoptimization in text-to-image generation and propose a simple uncertainty estimation-based method for mitigating the risk.

Abstract: Fine-tuning text-to-image models with reward functions trained on human feedback data has proven effective for aligning model behavior with human intent. However, excessive optimization with such reward models, which serve as mere proxy objectives, can compromise the performance of fine-tuned models, a phenomenon known as reward overoptimization. To investigate this issue in depth, we introduce the Text-Image Alignment Assessment (TIA2) benchmark, which comprises a diverse collection of text prompts, images, and human annotations. Our evaluation of several state-of-the-art reward models on this benchmark reveals their frequent misalignment with human assessment. We empirically demonstrate that overoptimization occurs notably when a poorly aligned reward model is used as the fine-tuning objective. To address this, we propose TextNorm, a simple method that enhances alignment based on a measure of reward model confidence estimated across a set of semantically contrastive text prompts. We demonstrate that incorporating the confidence-calibrated rewards in fine-tuning effectively reduces overoptimization, resulting in twice as many wins in human evaluation for text-image alignment compared against the baseline reward models.

Published as a conference paper at ICLR 2024
CONFIDENCE-AWARY REWARD OPTIMIZATION FOR TEXT-TO-IMAGE MODELS
Kyuyoung Kim
Jongheon Jeong
Minyong An' Krishnamurthy Dvijotham
Mohammad Ghavamzadeh*
Jinwoo Shin' Google DeepMind
Kimin Lee'
KAIST
Korea University
Yonsei University *Amazon
ABSTRACT
Fine-tuning text-to-ine image models with reward functions trained on human feed- back data has proven effective for aligning model behavior with human intent. However, excessive optimization with such reward models, which serve as mere proxy objectives, can compromise the performance of fine-tuned models, a phe- nomenon known reward overoptimization. To investigate this issue in depth, we introduce the Text-Image Aligmment Assessment (TIA2) benchmark, which com- prises diverse collection of text prompts, images, and human annotations. Our evaluation of several state-of-the-art reward models on this benchmark reveals their frequent misalignment with human assessment. We empirically demonstrate that overoptimization occurs notably when poorly aligned reward model is used as the fine-tuning objective. To address this, we propose TextNorm, a simple method that enhances alignment based on measure of reward model confidence estimated across a set of semantically contrastive text prompts. We demonstrate that incorporating the confidence-calibrate rewards in fine-tuning effectively re- duces overoptimization, resulting in twice as many wins in human evaluation for text-image alignment compared against the baseline reward models.
INTRODUCTION
Large-scale text-to-image models have been successful in generating high-quality and creative im- ages given text prompts as input (Saharia et al., 2022; Rombach et al., 2022; Yu et al., 2022). However, current models have several weaknesses (Hu et al., 2023), including limited ability in compositional generation (Feng et al., 2023), inaccurate text rendering (Liu et al., 2022), and diffi- culty with spatial understanding (Gokhale al., 2022). Moreover, large-scale datasets used train state-of-the-arr text-to-image models often contain malicious content (Schuhmann et al., 2021) and undesirable biases (Fan al., 2023) that models can potentially learn from during training.
Learning from human feedback has emerged as powerful method for addressing these limitations and aligning text-to-image models with human intent (Fan et al., 2023; Lee et al., 2023; Xu et al., 2023). This method involves learning reward function that approximates human objective (i.e.. the true objective) from human feedback data, followed by optimizing the model against the learned reward enhance alignment. However, optimizing too much against proxy objectives can hinder the true objective, a phenomenon commonly known as reward overoptimization (Gao et al., 2023).
In this work, investigate the issue of overoptimization in text-to-image generation, evaluating several state-of-the-art reward models. To facilitate our evaluation, we introduce the Text-Image Alignment Assessment' (TIA2) benchmark, a diverse compilation of text prompts, images, and hu- man annotations. Our findings indicate that reward models fine-tuned on human feedback data, such as IewgeReward (Xu al., 2023) and PickScore (Kirstain et al., 2023), exhibit stronger correlations with human assessments compared to pre- are-trained models like CLIP (Radford et al., 2021). Never- theless, all of the models struggle to fully capture human preferences. As Figure demonstrates, excessive optimization can compromise both text-image alignment and image fidelity. We show that this is particularly the case when the reward signal is poorly aligned with human judgment.
Work done at KAIST.
'The henchmark data available https: / tiggthhbbooo com www..o/////k.o...... //

---

Published as a conference paper at ICLR 2024
Original
CLIP
BLIP-2
ImageReward
PickScore
TextNorm
realistic photo laptop and suitcase
a realistic photo of three horses
blue bird and brown bear
Figure 1: Images generated using Stable Diffusion v2.1 (Rombach et al., 2022) fined-tuned with CLIP (Radford et al., 2021), BLIP-2 (Li et al., 2023), ImageReward (IR; Xu et al. 2023), and PickScore (PS; Kirstain et al. 2023). Both text-image alignment and image fidelity exhibit degrada- tion when subjected to excessive optimization. Our proposed method (TextNorm) demonstrates its robustness against overoptimization, illustrated in the last column.
To alleviate overoptimization, we propose textual normalization (TextNorm), simple method to enhance the alignment of reward models by calibrating rewards based measure of model con- fidence. Specifically, we consider set of semantically contrastive prompts and adjust the reward conditioned on the input prompt relative to those conditioned on the contrastive prompts. The key idea is to leverage the relative comparison of the rewards as measure of model confidence to cal- ibrate rewards. In constructing the set of contrastive prompts, show that both simple rule-based approaches and leveraging large language models (LLMs)) such as ChatGPT (OpenAI, 2022) can be effective. We also propose and demonstrate that ensemble methods, which combine multiple reward models, can be used to achieve further improvement. Our experimental results demonstrate that TextNorm significantly enhances alignment with human judgment on the TIA2 benchmark. This improvement renders the fine-tuning of text-to-ine to--ttoimmgeeee models more robust against overoptimization, a conclusion supported by human evaluations.
Our main contributions are as follows:
We introduce a benchmark for evaluating reward models in text-to-image generation, providing key insights into the alignment of several state-of-the-art models with human judgment.
We empirically demonstrate the adverse effects of excessive optimization against learned reward models. Importantly, we show that overoptimization is conceivable, even for reward models trained on extensive human preference data.
We propose TextNorm, a simple method for enhancing alignment by calibrating rewards based measure of model confidence. Through extensive experiments, demonstrate that approach could substantially mitigate overoptimization.
RELATED WORK
Text-to-image generation. Diffusion model of generative models that have achieved ofate-of-the-- results across various domains, including image synthesis (Dhariwal & Nichol, 2021), 3D synthesis (Poole et al., 2023), and robotics (Chi al., 2023). Different conditioning mechanisms guide the diffusion process allowing, for instance,
et al., 2015; Ho et al., 2020) is a family
2

---

Published as a conference paper at ICLR 2024
generation of images from textual descriptions. This has led to the development of many of the recent state-of-the-ar eoxt-to-imace models (Rombach et al., 2022; Ramesh et al., 2022; Saharia et al., 2022). However, generating images that fully respect the input prompt remains challenge.
Evaluating text-to-image generation. Evaluation of text-to-image models often requires consider- able human effort, prompting a rising interest in the development of automatic evaluation methods. Vision-language models (VLMs), such as CLIP (Radford et al., 2021) and BLIP (Li et al., 2023), measure text-image alignment using the cosine similarity between their embeddings. Methods such as DALL-Eval (Cho et al., 2023) and VISOR (Gokhale et al., 2022) utilize object detection to de- termine whether the objects are present in the correct quantities and locations. ImageReward (Xu et al., 2023) and PickScore (Kirstain et al., 2023) fine-tune VLMs on a large set of human preference data to directly predict preference. this work, we examine multiple of these models and propose a method for improving their alignment with human judgment.
Reward overoptimization. Training reward model on human feedback data and fine-tuning LLMs on this reward signal has proven effective in aligning the models with human objectives (Stiennon et al., 2020; Ouyang et al., 2022). Similar methods have been applied to improve text-to-image models (Lee et al., 2023; Xu et al., 2023; Fan et al., 2023), enabling the models to generate more human-prec -umae-preferred images. However, excessive optimization against a reward model, which is an imperfect proxy, can degrade model quality (Gao et al., 2023; Black et al., 2023). In this study, present a comprehensive empirical demonstration of overoptimization in text-to-image generation and show that employing better calibrated rewards can effectively alleviate the issue.
TIA2: A BENCHMARK FOR TEXT-IMAGE ALIGNMENT ASSESSMENT
3.1 DATA COLLECTION AND STATISTICS
We collect 100 text prompts sampled from diverse sources, including DrawBench (Saharia et al., 2022), PartiPrompt (Yu et al., 2022), ImageRewardDB (Xu et al., 2023), and Pick-a-Pic (Kirstain et al., 2023), to construct the comprehensive set. We additionally create synthetic prompts describ- ing quantities of objects or combinations of two distinct objects to form the counting and compo- sition sets. For the synthetic prompts, we utilize a total of 25 object classes, comprising 10 from CIFAR-10 (Krizhevsky et al., 2009) and 15 from MS-COCO (Lin et al., 2014). The counting set contains prompts describing the quantity of objects from I to 6 for each object class, such as "three deer," while the composition set comprises prompts describing combinations of two distinct object classes, such as "a cat and a dog." " For each prompt, we generate 50 images using Stable Diffusion v2.1 (Rombach et al., 2022) to include in the benchmark. Table provides basic statistics on the benchmark. More details on the dataset can be found in Appendix B.
To evaluate text-image alignment, gather binary feedback (good or bad) from three human notators for each text-image pair. We consolidate annotators' responses by assigning a good label at least two out of three are positive; otherwise, we assign bad label. Figure 2 illustrates two sets of images, highlighting instances where the reward models fail to fully capture human assessment.
3.2 BENCHMARKING REWARD MODELS
Problem setup. Our benchmark presents a dataset D := {(xi, Vi, x)}{= comprising triplets y, =) of text prompt x X, an image y E V, and a binary human label {0, 1} indi- cating whether z and are semantically consistent ("1"") or not ("0"). The goal of reward modeling for predicting text-image semantic consistency is then to derive a function r r y) E based on which we can infer the label for the prompt and the image y. In this view, we can frame reward modeling as binary classification task and assess reward models based on their performance as bi- nary classifiers of human labels. For instance, rewards computed using r can be converted to binary predictions based on a chosen threshold, and then compared to the human labels.
Evaluation metrics. We employ standard metrics for binary classification, including the Area Un- der the Receiver Operating Characteristic (AUROC) and the Area Under the (AUPRC), to evaluate the ability of reward models as classifiers in distinguishing between seman- tically consistent and inconsistent samples. These amrehhod--nddpeenee the overall alignment with human labels. When reward models used optimization objective,
Curve
metrics are used to assess
3

---

Published as a conference paper at ICLR 2024
Table 1: Statistics on the TIA2 benchmark. The benchmark consists of total of 550 text prompts, 27,500 images, and a set of three hu- man annotations for every text-image pair. See Appendix B for more details on the benchmark.
Prompt: in apple and
Prompt: a bosk and ially bear
libel
Human
Good
Bid
Good
Hnd
CLIP
CИP
Counts
Labels (%)
0.379
0.360
0,324
0,352
BLJP2
Category
Texts
Images
Good
Bad
0.386
ImagReward
ImageeBrarr
Comprehensive
100
5,000
47.22
52.78
0,300
1453
1,288
1.450
Counting
150
7,500
43.27
56.73
PirkSrane
0.208
PickSeun
Composition
300
15,000
38.97
61.03
Total
550
27,500
41.64
58.36
Figure 2:
Sample images for which the reward models do not fully agree with human labels.
ensuring that the high-scoring samples align closely with human assessment becomes particularly important. To gauge this alignment, we use average precision (AP) at Is E {5, 10, 25) to evaluate how well the rewards match the human labels for the top-k samples. Lastly, we consider rank corre- lation statistics, namely Spearman's 's and Kendall's т, as additional metrics to assess the similarity between the rewards and human labels. For more fine-grained comparison, individual human labels for each example. We convert a "good" label to I, a "bad" label to 0, and an "inconclusive" label 0.5. The score is then computed as the mean of these three numbers, and rank correlations are calculated between the rewards and these aggregate scores. We compute the metries for each prompt using the 50 samples provided in the benchmark.
aggregate the three
-CLLIRAATD REWARD DESIGN
In this section, explore methods that enhance the alignment of reward models based on measure of model confidence. We quantitatively assess a range of reward models on our benchmark, showing that even those fine-tuned human feedback data often inadequately capture human preferences. Subsequently, we demonstrate the effectiveness of our method in comparison.
4.1 TEXTUAL NORMALIZATION
Given a reward model r(zo, v) E E that measures text-image semantic consistency between a text prompt 2o and an image y, we propose a simple method that leverages a measure of reward model confidence for enhancing its alignment. Specifically, we consider set of alternative prompts X, and normalize the reward r(zo, y) using the softmax function based on the rewards conditioned on these prompts. The idea to view the relative comparison of the rewards as a measure of model confidence in r(zo, y) and calibrate the reward accordingly. For instance, if the rewards conditioned on the alternative prompts are relatively close to r(Fo, y), suggests that the model is less confident. in r(zo, y). prompting more significant adjustment in the reward. This softmax-bate based approach, while simple, can be effective in estimating model confidence (Hendrycks & Gimpel, 2017).
With access to all semantically distinct prompts, we could compute this model confidence precisely. However, computing the softmax function over this prompt set would involve an infinite sum. In- stead, we approximate this sum by considering finite set X of prompts each of which shares syntactic similarity with 2o while being semantically distinct. The rationale behind this approach is based the hypothesis that prompts that differ from 20 in both syntax and semantics are unlikely to yield high reward values r(x, y). thus making negligible contributions to the softmax score. We empirically evaluate this hypothesis and confirm that it holds in practice. Using the of prompts X = (エ5}ر we define the following score for the reward model r:
exp r(0o: w)) x (Io. = exp ( r(zo, + Emn exp ( -(x,,))))
(I)
where 7 > 0 is temperature scale.
To illustrate the types of prompts in X, consider the text prompt Zo = "a photo of two dogs". We could include prompts such "a photo of three dogs" and "a photo of three cats" that
4

---

apply TextNorm to derive corresponding rewards (ri",..., y.X } normalized over the set of prompts Published as a conference paper at ICLR 2024
Scoring
Softmax
Input imuge
Imogee Reward
Input prompt
four borses
Ensemble
Reward
LLM
Pickscoore
Contrastive prompts
horses
Figure 3: TextNorm normalizes rewards set of contrastive prompts generated using LLM. Combining an ensemble of normalized rewards multiple models can further enhance alignment.
syntactically similar to but semantically different from In contrast, prompts such as "describe colorful abstract painting" would not be as useful to be part of X.
Prompt set synthesis via language models. An important step in implementing TextNorm involves creating contrastive prompts over which to normalize rewards. For simple input prompts, rule- based approach can often be effective. For instance, when the input prompt describes quantity of an object, prompts that describe various other quantities of the same object can be used. For more linguistically complex input prompts,
leverage LLMs generate contrastive prompts, providing appropriate few-shot examples to guide the generation:
"Create captions that different from the original input used for the text-to- itto-maage generation model, referencing the provided failure cases [few-shot examples] [inpul prompt] ..."
When composing the LLM prompt, analyze common failure scenarios of text-to-image models for the type of the input zo and present their textual descriptions as few-shot examples. For instance, observed that models often generate images containing incorrect quantities types of objects compared to those described in zo. Incorporating these prompts into X results in better calibrated rewards, particularly when the image is indeed inconsistent with the input prompt. Also, depending on the nature of Zu, we include few-shot examples considering other relevant aspects such as spatial relationship, color, and material. See Appendix D for further details on the use of LLMs.
4.2 REWARD MODEL ENSEMBLE
On our benchmark, we found PickScore be overall better aligned with human judgment than other baselines (see Section 4.3 for a more comprehensive evaluation). However, we noted instances where competitive baselines such IewaeReward showed stronger alignment, shown in the examples in Appendix B.4. Motivated by the observation, we propose combining an ensemble of reward models to further enhance alignment. Given set of reward models {rı,..., Fik, }. first
X. This ensemble of normalized rewards then be combined in several natural ways. See Figure for an illustration of an overview of the method.
Mean ensemble. method simple averaging can be effective, partieularly when the reward models within the ensemble are of comparable quality.
(Fo, v)
Σ
r (2o. v),
where zo is the prompt, and y denotes the image.
Uncertainty-regulrrinn ensemble. If the reward models in the ensemble disagree significantly, penalty based on the variance the normalized rewards can act as form of regularization. That is,
x (Fo. u) = r (wo, v)
X Te Σ (No. y) ri (xo, x))²
where zo is the prompt, y denotes the image, and is the weight of the variance penalty.
5

---

Published as a conference paper at ICLR 2024
Table Alignment evaluation on each of the comprchensive (C100), counting (C150), and compo- sition (C300) sets. Prompts for which all human labels are identical are excluded.
Reward
AUROC
AUPRC
AP@S
AP@
2
Spearman
Kendall
CLIP
0.671
0.611
0.674
0.667
0.635
0.235
0.187
BLIP-2
0.683
0.603
0.648
0.642
0.616
0.233
0.185
C100
ImageReward
0.761
0.674
0.755
0.740
0.708
0.361
0.289
PickScore
0.747
0.675
0.779
0.763
0.706
0.325
0.258
TextNorm
0.788
0.727
0.841
0.820
0.765
0.372
0.298
CLIP
0.595
0.522
0.622
0.600
0,550
0.136
0.107
BLIP-2
0.565
0.499
0.570
0.556
0.519
0.072
0.056
C150
Iewae
0.657
0,543
0.611
0.605
0.570
0.208
0.166
PickScore
0.731
0.614
0,711
0.702
0.650
0.340
0,273
TextNorm
0.807
0.719
0.877
0.841
0.763
0.443
0.356
CLIP
0.607
0.737
0.699
0.641
0,431
BLIP-2
0.506
0.579
0.554
0.516
0,246
0,186
C300
ImageReward
0.774
0.675
0.785
0.747
0.699
0,538
0,423
PickScore
0.785
0.696
0.848
0.811
0.741
0.521
0.407
TextNorm 0.844
0.787
0.944
0.906
0.828
0.622
0,495
Note that while the ensemble methods above may visually resemble those proposed in the concurrent work of Coste et al. (2023), consider ensemble of reward models not necessarily trained identical data or with identical hyperparameters. Moreover, we combine rewards normalized accord- ing our proposed TextNorm method and demonstrate the approach in the context of text-to-image generation, while Coste et al. (2023) focuses on language modeling. Evaluating the effectiveness of various ensemble techniques across diverse domains would be an interesting future research.
4.3 QUANTITATIVE EVALUATION
We evaluate TextNorm using the metrics outlined in Sec- tion 3.2, averaged across prompts within each set in the benchmark, with the following four baselines: CLIP, BLIP- 2, ImageReward, and PickScore. ImageReward and PickScore with appropriately constructed prompt sets which we release alongside the benchmark. To the
Table Ablation study results.
X
AUROC
AUPRC
use an ensemble of
X
X
0.764
0.669
Rand LLM
X
0.641
0.540
X
0.780
0.704
demonstrate both ensemble methods proposed, mean ensemble for the composition set prompts and the uncertainty ensemble for prompts from the counting and comprehensive sets.
LLM
0,822
0.752
Table 2 summarizes the quantitative results. We note that ImageReward and PickScore, which have been trained considerable human feedback data, generally outperform other baseline models. Nevertheless, they exhibit weaker alignment, particularly on certain prompt types, such as those in the counting set. comparison, TextNorm notably improves aligmment across all three sets of prompts on the benchmark, as evidenced by improvements in all measured metrics.
We also conduct ablation studies assess TextNorm based on the prompt set type used and whether an ensemble method is utilized. Table 3 summarizes the results in terms of AUROC and AUPRC averaged over all prompts the benchmark, with PickScore as the baseline. The column X denotes the use of TextNorm and specifies whether it is applied, and if whether random prompt set or a set constructed with an LLM used. The column denotes whether an ensemble method additionally used. The results indicate that (a) leveraging TextNorm with suitable prompt set can enhance alignment, and (b) using an ensemble method can yield further improvement.
EXPERIMENTS
In this section, we assess the effectiveness of TextNorm in mitigating overoptimization across three optimization methods: best-of-n sampling, supervised fine-tuning (SFT), and policy gradient-based reinforcement learning (RL). We first present qualitative comparisons in Sections 5.2 and 5.3, fol- lowed by quantitative results from human evaluations in Section 5.4.

---

Published as a conference paper at ICLR 2024
5.1 SETUP
In the experiments, we use Stable Diffusion v2.1 (SD v2.1) as our base text-to-image model. For best-of-n sampling, generate set of images using this base model and then select subset based on the reward models for comparison. For SFT and RL fine-tuning, we use low-rank adapta- tion (LoRA; Hu al. 2022) to fine-tune SD v2.1 with the reward models and compare the images generated using the fine-tuned text-to-image models. We choose the carliest checkpoint at which the number of generated images with higher scores, as measured by the reward model, than those generated using SD v2.1 reaches the maximum. Hence, better aligned reward model improves fine-tuning by providing (a) more aligned signals for optimization and (b) a better measure based on which to select checkpoints. We apply the same parameters used for the quantitative evaluation in Section 4.3 to TextNorm for these experiments. We select and use 30 prompts from each of the comprehensive, counting, and composition sets in our benchmark for best-of-n sampling, and 10 prompts from the 30 for the SFT and RL fine-tuning experiments. Further experimental details can be found in Appendix A.
5.2 BEST-OF-N SAMPLING
We begin by evaluating all reward models using best-of-n sampling, which is simple yet effective inference-time algorithm that selects the optimal sample from a set of n candidates based given reward model. This allows us to assess the alignment of the rewards in isolation without involving fine-tuning. Specifically, for a given prompt, we generate a of E {16, 64, 256} images using the text-to-image model and select the image with the highest reward.
Iraagefloward
TocHorr
CLP
BLP-2
Iragefoonrd
PiaScor
TodWom
Prompt: realisic photo tour faddy bears
Prompt: a roalistcc photo of bid and a cup
Figure 4: Images sampled using best-of-n for n E {16, 64, 256) with the five reward models.
Figure illustrates best-of-n samples selected based on the five reward models. It highlights that se- lecting the best image from a larger pool of samples may be less desirable, especially when poorly aligned reward model used. Specifically, the quality of images chosen using BLIP-2 degrades as the value of n increases. Other reward models generally yield better results; however, with baseline models, some selected images often depict objects that are only partially visible (see ImageReward on the left), arguably less realistic for higher values of n (see CLIP on the right).
5.3 FINE-TUNING WITH REWARD MODELS
We further our evaluation by fine-tuning SD v2.1 using the reward models and comparing the images generated using the resulting models. We consider both SFT and RL-based fine-tuning.
Supervised fine-tuning. We adopt setup similar to that in Lee al. (2023), using reward-weighter regression (RWR) with the reward model scores weights for fine-tuning. For each text prompt, we generate 100 images, then select the top 10 based on the reward model to create mindel and fine-tune SD v maximize the seward-weightn likelihood of the data:
(4) where 3 is the coefficient for the Kullback-Leible (KL) regularizer used prevent excessive devi- ation from the original model pa, (Fan et al., 2023). We experiment with multiple KL coefficients and select the that achieves the most improvement.
J(0) B(z.))=Deaa (r( y) log (y æ) 8 log (po(y 22/Po, (y x))].
7

---

Published as a conference paper at ICLR 2024
mallatic photo of dog and a frog
a realistic photo horse and a ship
realisfic photo of four leddy bears
Greek statue of man tripping over a cat
couch on the left of chair
ImageReward
PlckS
TextNor
(a) Supervised fine-tuning with RWR
mallstic of dog frog
roalistic photo
realistic of leddy
Grock
the left
tripsing
ImageReward
PickScore
TextNorm
(b) fine-tuning with DDPO
Figure 5: Images generated using SD v2.1 fine-tuned with the reward models.
RL fine-tuning, We also consider RL-based fine-tuning, where we iteratively collect data using the current policy and perform optimization, instead of fitting a fixed distribution. We use the denoising diffusion policy optimization (DDPO; Black al. 2023), where the denoising process is treated as multi-step Markov decision process, allowing fine-tuning of models using policy gradient methods. Given a reward model and a prompt distribution p(x), the following objective maximized:
[()) Ex-pp()) (r) (x. + X(Oo. 8 v.,)],
(5)
where À is divergence- based regularizer to penalize deviating too far from the original model dp-
Figure shows sample images generated using the fine-tuned models. Common text-image align- ment issues observe with the models include: (a) missing object (upper row 1, column 2), (b) containing an unmentioned object (lower row 2, column 2), (c) depicting incorrect quantities of objects (upper rows 1 and 2, column 3), and (d) disregarding the relations between objects as
8

---

Published as a conference paper at ICLR 2024
Text-Image Alignment
Text-Image Alignment
Origindl
0.32
0.52
9.16
Original
0.51
0.34
0.15
1R
0:15
0,31
0.21
IR
0.51
0.2
0.29
0.0
0.2
6.6
0.5
0.2
64
0.8
1.0
Image Quality
Image Quality
Original
0.16
0.73
0.11
Original
0.31
0.46
0.33
IR
IR
Ps
0.19
0,68
0.13
0.1
0:13
0:47
0.0
0.2
0.6
0.5
0.2
64
0.8
1.0
Win
Tic
Loss
Win Tie
Loss
(a) Supervised fine-tuning with RWR
RL fine-tuning with DDPO
Figure 6: Human evaluation results. We compare the models fine-tuned with TextNorm against SD v2.1 (Original) and those fine-tuned with ImageReward (IR) and PickScore (PS). TextNorm consistently achieves significantly better alignment with comparable image quality in the case of SFT, though with slightly greater sacrifice in the case of RL.
described in the prompt (lower row 1, column 5). TextNorm, which aligns more closely with hu- man judgment, provides improved rewards for optimization and a more reliable metric for selecting checkpoints, leading to images with better text-image alignment.
5.4 HUMAN EVALUATION
For human evaluation, we generate eight images per prompt using the fine-tuned models and create two sets of four images, resulting a total of 60 sets evaluated by three independent annotators. Fig- 6 reports the overall win, tie, and loss rates of the models to SD 1 and those fine-tuned with leageReward and PickScore. For SFT, TextNorm achieves ap- proximately twice as many wins for text-image alignment, with comparable image quality in all three comparisons. For RL, TextNorm achieves an even more dramatic improve- ment in text-image alignment but at the expense of image quality. We suspect that this is because in constructing the benchmark data, which are used to tune the parameters of TextNorm, the anno- tators primarily considered alignment their assessment. As RL more powerful optimization algorithm, it possibly optimized more for alignment at the expense of image quality. Considering multiple objectives, such as alignment and quality, and leveraging the corresponding reward models in fine-tuning would be a promising approach to achieving better balance among the objectives.
with TextNorm compared
slightly better
The complete human evaluation results for the best-of-n sampling experiment, which also demon- strate the effectiveness of TextNorm, provided in Appendix C.
CONCLUSION
Fine-tuning models on a reward function trained on human feedback data has emerged as a promis- ing method for aligning models with human intent. However, excessive optimization can degrade model performance. In this work, we introduce the TIA2 benchmark to assess several state-of-thee art reward models in text-to-image generation. We find that the reward models, even those trained on human data, are often not well-aligned with human assessment. We demonstrate that overoptimiza- tion occurs particularly when poorly aligned reward models are used for fine-tuning. To address this issue, introduce TextNorm, simple method that enhances alignment through reward calibration using semantically contrastive prompts. We demonstrate both quantitatively and qualitatively that the imffidence-calibrate scores effectively reduces overoptimization.

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
Generative models for image synthesis, like many machine learning models in general, are sus- ceptible to learning biases inherent in the training data (Birhane al., 2021). While fine-tuning pre-trained models with suitable reward can enhance the models, overoptimzing against an im- perfect proxy objective can instead degrade model quality, as we investigate depth in this work. Hence, it is important to understand the limitations of both the pre-trained models and the rewards with which they are fine-tuned. Our work timely exploration of this issue text-to-image generation, examining various state-of-the-an reward models and reporting their limitations both evaluation metrics and training objectives. We also introduce a simple method for better aligning reward models with human intent. While we demonstrate the effectiveness of our method in reduc- ing overoptimization, the implementation relies on external models such as an LLM and a VLM. Therefore, careful selection of these models is crucial, as the success of the implementation hinges on their quality.
REPRODUCIBILITY STATEMENT
We provide implementation details including the experiment setup, the models used, and the training specifications (e.g.. hyperparameters) Section 5 and in Appendix
ACKNOWLEDGMENTS
We thank Seunghwan Lee, Sojeong Kim, Dongjun Lee, Changyeon Kim, Jongjin Park, Yisol Choi, Hyunsub Jeong, Junesuk Choi, and Younghyun Kim for their help with labeling the data of the main experiments of this work.
This work was supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea government (MSIT) (N0.2019-0-00075, Artificial In- telligence Graduate School Program (KAIST); No.2021-0-02068, Artificial Intelligence Innovation Hub; No.2022-0-00955 Few-shot Learning of Casual Inference in Vision and Language for Deci- sion Making).
REFERENCES
Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. Multimodal datasets: misogyny, pomography, and malignant stereotypes. arXiv preprint arXiv:2110.01963. 2021.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305. /3301, 2023.
Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. Robotics: Science and Systems, 2023.
Jaemin Cho, Abhay Zala, and Mohit Bansal. DALL-EVAL: Probing the reasoning skills and social biases of text-to-image generative models. In International Conference on Computer Vision, 2023.
Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help
mitigate overoptimization. arXiv preprint arXiv:2310.02744 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. In Advances in Neural Information Processing Systems, 2021.
Ying Fan, Olivia Watkins, Yuqing Du Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lec. DPOK: Reinforcement learning for fine-tuning text-to-image diffusion models. In Advances in Neural Information Processing Systems, 2023.
Weixi Feng, Xuchai He, Tsu-Jui Fu, Varum Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for
10

---

Published as a conference paper at ICLR 2024
compositional text-to-image synthesis. In International Conference Learning Representations, 2023.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, 2023.
Tejas Gokhale, Hamid Palangi, Besmira Nushi, Vibhav Vineet, Eric Horvitz, Ece Kamar, Chitta Baral, and Yezhou Yang. Benchmarking spatial relationships in text-to-image generation. arXiv preprint arXiv:2212. 2022.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and examples in neural networks. In International Conference on Learning Representations, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, 2020.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation large language models. In International Con- ference on Learning Representations, 2022.
Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. TIFA: Accurate and interpretable text-to-image faithfulness evaluation with question an- swering, In International Conference on Computer Vision, 2023.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-Pic: An open dataset of user preferences for text-to-image generation. In Advances in Neural Information Processing Systems, 2023.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. University of Toronto, 2009.
Kimin Lee, Hao Liu, Moonkyung Ryu, Olivia Watkins, Yuqing Du, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, and Shixiang Shane Gu. Aligning text-to-image models using human feedback. arXiv preprint 12X1v:2302.12192, 2023.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.1259/) 2023.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In European Conference on Computer Vision, 2014.
Rosanne Liu, Dan Garrette, Chitwan Saharia, William Chan, Adam Roberts, Sharan Narang, Irina Blok, RJ Mical, Mohammad Norouzi, and Noah Constant. Character-aware models improve visual text rendering. arXiv preprint arXiv:2212. 10562, 2022.
OpenAI. Introducing ChatGPT. https://opena: www.c//ccc...ooo.............c.a hollog/cchhttt 2022.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama. Alex Ray, al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3D using 2D diffusion. In International Conference on Learning Representations, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with CLIP latents. arXiv preprint 2016:2204.0 2022.
11

---

Published as a conference paper at ICLR 2024
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Conference on Computer Vision and Pattern Recognition, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In Advances in Neural Infor- mation Processing Systems, 2022.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. LAION-400M: Open dataset of CLIP- FLIP-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learn- ing, 2015.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, 2020.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. ImageReward: Learning and evaluating human preferences for text-to-image generation. In Advances in Neural Information Processing Systems, 2023.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang. Vijay Vasudevan, exander Ku, Yinfei Yang, Bureu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. Transactions on Machine Learning Research, 2022. ISSN 2835-8856.
12

---

Published as a conference paper at ICLR 2024
A EXPERIMENTAL DETAILS
A.1 BASELINE REWARD MODELS
We use publicly available implementations of the baseline reward models. For CLIP, CLIP ViT-B/32 model from the official CLIP implementation provided by OpenAI. For BLIP- we use the pretrain version of the blip2 model from the Salesforce LAVIS library. For ImageReward and PickScore, we use the official models provided by the authors of the papers, which are based the BLIP and CLIP ViT-H/14 architectures, respectively.
the
A.2 FINE-TUNING TEXT-TO-IMAGE MODELS
Our fine-tuning implementations are based on publicly available Python libraries. Specifically, we used the diffusers library for the SFT experiments and the trl library" for the RL experiments. For SFT, we used stabilityii/tt for up 1,000 steps. We experimented with KL coefficients ranging from 0.00 1.00, in increments of 0.25, and selected the one that yielded the most improvement. For RL, we used stabilityee www.llitttssallleeeeeeeee faf.ab-dffffsionn---ooo 2-1-baase as base model and fine-tuned for up to 500 epochs over the training prompts. Note that we have conducted minimal hyperparameter tuning and, in many cases, adopted the default values provided by the library code.
as our base model and fine-tuned
Table 4: Summary of hyperparameters used for SFT and RI. fine-tuning.
Parameters
RWR
DDPO
Denoising steps
50
50
Diffusion
Guidance scale
7.5
5.0
Optimizer Learning rate
AdamW
Adam W
1e-5
3e-4
Weight decay
le-2
le-4
Optimization
31
0.9
0.9
32
0.999
0.999
1e-8
1e-8
Max gradient norm
1.0
1.0
Batch size
32
64
Samples per iteration
256
Training
Gradient updates per iteration
1
Mixed precision
fp16
fp16
A.3 PARAMETERS FOR TEXTNORM
The TextNorm score used for both the quantitative analysis in Section 4.3 and the experiments in Section 5 is derived from ensemble of ImageReward and PickScore. To demonstrate both ensemble methods discussed in Section 4.2, we employed the mean ensemble for the composition set prompts and the uncertainty-regulrriii sets. We only lightly tuned the temperature parameter and the uncertainty penalty coefficient based on the average of the three AP metrics considered this work.
ensemble for prompts from the counting and comprehensive
Pattpp:/ggttabb.omm/paaaa///
httt::///ggtbbbb.co///ggggggggge fuse
/https://githhil
13

---

Published as a conference paper at ICLR 2024
B
DETAILS ON THE TIA2 BENCHMARK
B.1 EXAMPLES AND STATISTICS
Table 5 provides sample prompts from each of the three in the benchmark.
Table Sample prompts taken from TIA2 per category.
Category
Examples
minnle mouse and baby shark cartoon A 1960s poster warning against climate change cute bee wearing chef hat colored pencil art
Comprehensive
a realistic photo three dogs
Counting
a realistic photo four deer a realistic photo of six teddy bears
realistic photo of deer and truck realistic photo
Composition
bird an umbrella realistic photo an airplane and a ship
Table reports the complete set of object classes used to create the synthetic prompts. Table provides further statistics the subcategories of the comprehensive
Table 6: Object class names used generate synthetic prompts in TIA2.
Table Statistics on each subcategory of the comprehensive set in TIA2.
Subcategory
Object classes
Counts
Human labels (%)
Accessory
suitcase, umbrella microwave, toaster
Subcategory
Texts
Images
Good
Bad
Appliance
Colors
10
500
91.20
8.80
Animal
bird, cat, dog. horse, deer, frog laptop
Composition
10
500
33.00
67.00
Electronic
Counting
10
500
52.80
47.20
Food
cake, apple, orange, carrot chair book, teddy bear, vase cup, fork automobile, airplane, truck, ship
Creative
10
500
70.00
30.00
Furniture
Location
10
500
84.20
15.80
Indoor
Reddit
20
1,000
36.60
63.40
Kitchen
Spatial
10
500
14.20
85,80
Style
10
500
53.20
46,80
Vehicle
Text
10
500
0.40
99.60
Total
100
5,000
47.22
52.78
B.2
LABELING PROCEDURE
For each text-image pair in the benchmark, we asked three human annotators to assign a binary label indicating whether the text and image are semantically aligned, indicate that the assessment is inconclusive. We asked the annotators to assess text-image alignment first and foremost, and to consider image fidelity their discretion if it was noticeable enough to affect alignment. Table shows an excerpt of the labeling instructions we provided to the annotators for the task. Figure is a screenshot of the labeling interface the annotators used.
Table Excerpt from labeling instructions given to annotators.
Instruction
Options = (1: Good, 2:
Bad and 3: Skip (hard to answer) (1) If the provided prompt aligns well with the image, select 1; otherwise, select 2.
(2) If the quality of the image is poor that it affects the alignment, select 2.
(3) If you feel you mislabeled something, you can relabel it.
14

---

Published as a conference paper at ICLR 2024
[1/3000] Prompt [a realistic photo of an airplane and an apple]
Image quality [1=good, 2=bad, 3=skip(hard to answer)):
Figure 7: Screenshot of the labeling interface.
B.3 ADDITIONAL ANALYSIS
We conduct additional analysis on the reward models, reporting the average AUROC for each syn- thetic prompt included in the benchmark. Since each synthetic prompt corresponds either a pair of two object classes insights into how well the reward models align with human judgment, depending on the type and quantity of object classes considered in the dataset.
combination of an object class and a count, this analysis offers deeper
In the tables below, we present heat maps displaying the per-prompt average AUROC values. Cells with value of 0.75 remain uncolored, while those with higher values are shaded in varying intensi- ties of blue, indicating higher scores. Conversely, cells with lower values are shaded in varying in- tensities of red, denoting lower scores. As the heat maps suggest, both ImwgeReward and PickScore generally attain higher AUROC values compared to CLIP and BIIP-2, with TextNorm consistently outperforming all baselines.
Table 9: Per-prompt AUROC for CLIP the composition set.
alyeess
5
ciplaae
0.73
6.79
10.57
0.6
0.58
0.76
10
0.66
0.54
18.9
0.77
0.54
0.78
969
10.59
0.57
1.3
10.5
0.09
0,22
1157
apple
0.73
0.73
0.15
0.62
0.74
0.02
0..0
1.335
0.37
0.54
1.56
11.1
0.66
0.75
0.68
0.77
0.78
0.72
0.74
066
0.13
0.77
0.75
0.85
0.87
0.71
18:7
bid
0.57
6.7
boot
0.6
0.95
0.73
11.2
0.72
10.9
1157
0.44
0.6
0.29
10:91
0.33
0.68
088
0.6
aike
0.58
00:44
0.6
D.7
0.68
0.84
11.25
10.6
16.06
11-1
0.57
0.73
0.57
0.71
0.37
0.64
0.75
0.75
1887
0.6
0.8
1.97
carmi
0.73
0659
0.79
0.6
0099
0.21
chair
0:75
0.09
0.72
18.56
0.81
0.7
0.7%
6379
0.76
0.73
0.83
0.30
0.96
11.99
cap
0.65
10:07
0.35
0.65
10:00
0.25
0.64
11.55
10.00
0.75
10.2
0..9
1085
0.55
0.02
0.78
0.74
0.67
0.27
11.00
0.7
10.6
0.72
1085
deer
0:84
0.75
0151
0.9%
0.64
0.71
116
0.59
0.73
10.5
0166
0.7
1176
0.71
03:55
11
0.66
0.96
0.77
0.95
Ing
0.65
10.90
187
61
0.75
15774
11
18.05
1..0
0.52
18.9
0.91
0.11
0.97
10.6
13.64
0.775
11:23
0.3
11.99
hores
119
0
10.000
11
0.4
1991
0:11
10:85
10.03
11
lyptp
0.66
0445
0.51
0.75
oraage
0.59
0.75
0.75
0.65
0.79
10.77
0.5
11.50
0.74
0.50
0.53
0.5
0.6
0.7
0.79
0.74
0.64
0.68
0.74
0.09
10.0
11.00
shu
0.59
0.72
1002
0.06
1001
0.64
0.03
1..65
0.53
0.67
0.75
1.14
0.2
0.85
0.33
0.54
0.74
0.71
0.76
0.73
1025
0164
11:00
saloose
0.71
0.54
0.67
10.55
0.87
10:3
0.1
0:66
0.71
0.7
10.34
0.76
0.65
01.64
0.7
0.55
11:9
0.24
0.73
0.02
1090
11:66
ruck
0.69
0.71
C.6
0.7
0.85
0.65
0.71
11.77
11:00
1096
0.88
11:66
1394
0.82
0.69
10:85
0.70
066
1192
0.64
[1.1
unbeella
0.82
0.72
0.63
027
0.
0.7
10446
0.64
0.67
0.64
15

---

Published as a conference paper at ICLR 2024
Table 10: Per-prompt AUROC for BLIP-2 on the composition set.
suplanc
10:7
01.83
11:1
1355
0.35
10:35
0.38
0.98
63
0.71
10:64
1..7
01.55
0.5
0.44
0.7
0.71
0.71
0.61
0.71
0.7
0.58
hnnd
0.7
18.5
10:05
0.55
10.94
0.08
0.74
0.18
0.63
10/11
0/81
0.51
19:35
1190
1394
18:5
1151
0.68
11.9
hoxt
0.56
11:1
1854
00:44
0.11
0.97
0.57
3.99
0.67
1.77
8-48
0.59
0.59
13:56
01.57
1.59
0.57
6.79
0.44
0.66
10:91
0.5
02999
0.55
11.76
6.8
0.67
0.63
0.95
0.57
10.4
0.55
18
11:1
18.97
0.55
15.%
0.74
10:99
0.41
0.55
10:09
11.00
0.71
chair
0.64
0.62
0.64
0.55
0.77
0.75
0.64
0.7
0.45
0.56
0.57
0.51
0.82
0.83
0161
0.45
0.23
0.61
0.78
0.9
0.5
0.54
cap
10.55
0.33
0:67
0.65
0.72
0.8
0.55
deg
0.71
10:55
0.65
10:34
0.30
0.00
10.57
15:05
0.33
0.75
0.6
0.56
0.72
0:41
0.29
0.44
0.64
10.35
0.54
0.46
0.62
0.24
1145
0.75
0.65
0.T
0.74
0.64
0.5
0.57
0.55
0.51
0.6
0.89
0.5
0.36
0.62
0.70
0.77
0.77
0.59
11.7
Imy
0.15
00611
0:33
10.54
0.511
10.65
10:41
10.90
1159
laptop
10:47
0.55
11:1
0.5
0.84
0.99
0.55
10.2
micnoarc
0.71
0048
6.72
0.43
0.52
0.72
0.06
11.43
1152
10.6
(.49
0.29
0.56
1447
0.51
0.54
0.55
0.76
0.81
11.5
oragg
0.
1082
0.34
0.37
01.47
0.:1
0.44
0.62
0.35
0.55
10
0162
0.51
0.39
0.67
01:05
0.85
11.15
0.76
0.54
0.91
0.81
1667
0.226
0.8
1156
sddy Near
1041
0.73
[6:97
0.94
13:5
0.67
0.99
11:00
01
0.71
0.54
0.77
0.6
0.57
0.67
18.5
0.4
0.77
10:50
11:41
11:11
0.66
11.7
0.34
0.64
0.44
0:57
0.67
0.35
0.66
11:1
0:46
0:48
0U61
0.71
0.67
unbooll
0.41
0.71
01:5
0:44
10:34
0.44
0.71
0.66
0.77
0.9
0.11
0.54
0.47
0.67
0.67
10:25
0.6
0.68
0.06
0.45
0.57
1.37
0.75
18.
0.51
0.54
0.60
018
Table 11: Per-prompt AUROC for ImageReward on the composition set.
alyshose
sippaac
0.71
0.72
appl
0062
0.14
0.43
0.67
0.5
0.5
11.54
0152
1150
0.56
0.71
0.33
1.83
0.79
0.69
0.71
9.87
0.77
0.65
0.5
0.73
0.65
10.0
1L05
0.85
0.74
0.58
0:81
0.5
0.63
0.522
0.3
0.50
0.58
11.33
0.8
0.65
0.69
0.85
1.0
0.41
0.82
0.81
0.81
1162
1899
0.70
kid
0.87
10.98
099
039
0:45
0.62
0.65
0.00
10.56
[0.1
(175
0.75
coke
0.79
0..8
0.76
01
1.59
115
0.71
03.67
0..50
0.99
18.9
0.24
0.27
care
0.99
00.6
C.65
0.16
0.69
10.6
10.2
11.5
18:75
10BS
1LY7
0.81
0.72
1.9
10.05
0.66
0.22
0.71
0.20
1081
11.9
0:20
10K22
0.39
0.29
10:02
1081
0.89
cap
0277
0.51
0.65
0.26
10.30
0.81
EL12
0.14
0.96
0.33
0.96
0.95
0.59
0..01
11.91
deer
10055
0:955
0.86
0.1
0.21
11.00
E104
109
088
0./6
0.71
0.64
00.87
0.00
10.04
0.996
0.85
0.55
11.65
dog
0:63
0.71
0.73
10
0.75
0.69
1087
4.76
0.56
0.59
0.66
0.73
0.75
0.75
0.77
0.87
0.67
1.7H
Ing
0:51
0.81
00:11
13.50
11.0
0191
0.50
165
07
0.57
0..3
boren
0.75
0.65
10:00
0:48
0.67
0.307
13.74
11.7%
0.63
0.56
1855
C.M
0.81
13.00
1..0
0.75
0.00
0.77
0.31
18.7
Sypipp
0.77
0.69
10:81
0.35
19:77
76766
1.75
0.67
0.7
0.65
10:81
0.55
0.79
0.72
0.77
6.92
0.62
0.67
0.955
0..79
13.34
13.56
0.57
0.75
0.89
0.33
0.88
0.77
0.94
0.51
01.2
0.75
salkosc
0061
0.65
0.81
0.29
0.25
0.79
0.500
10:42
0.53
0.55
0163
10:6
0.73
045
0.59
0.75
0.87
0.64
0.384
0.73
0.33
0.67
11.77
Jodly lear
0.72
0.73
01.5
0.67
0.94
0.75
0.66
0.51
vick
0.7
0:65
1693
0.63
0.96
0.74
0.74
10.01
18.02
1150
0.55
0.57
0.9
0.82
0.73
0.74
1898
0.7
0.92
0.35
0.77
urrhellaa
0.83
1091
6.75
0.65
0.73
0.27
0.8
11:00
0.
0.81
0.85
B./
0.92
10.8
0.81
10.0
0.91
090
0.25
0.60
11:00
1191
06
0.34
10/5
0.83
0.77
0.67
00
0.79
0.66
0.85
1991
0.88
A99
1434
(087)
0.54
0.76
[0/9
0.67
0.63
Table 12: Per-prompt AUROC for PickScore the composition set.
strplane
0.75
6.72
0.67
0.6
0.20
0.8
0.8
1098
0.64
0.71
0.24
0.69
B.7
10:3
0.68
1:78
apple
0.95
10:88
0:19
01:88
0187
11:4
1..5
FLS1
0.91
0.7
0:18
0351
0.53
0.28
0.55
0161
11:0
kind
0.67
0.76
0.56
0.76
0.56
0.62
0.7
0.79
beot:
0.64
0.75
6.76
0.76
0.79
0.73
0151
0/5
0.85
0.76
0.56
0.56
0.51
0.52
0.65
0.82
0.77
0.71
0.68
1.5
0.74
0.76
0.56
11:2
osko
0.79
0.75
10:801
0.25
0.74
0.77
1087
0.92
1.75
0.76
0.53
8:66
0.87
0.11
0.7
0.77
0.3
1005
0.73
0.67
0.56
0.5
carme
0.73
0.81
0.76
0.53
0.69
0.64
0.31
0.7
chair
0.8
0.87
6.84
0.34
0.0
1190
01776
0.83
0.53
0.55
1.75
0.76
0.71
0.73
1938
0.85
0.30
0.71
0.77
0:9
0.66
113
cap
0.8
0.85
0.5
0.85
0.85
0.25
0.29
0.770
0.54
0.67
IL.4
0082
0.65
0:87
0.79
0.69
0.24
0.80
0.82
1.99
1000
0.39
11.44
deer
(039))
0.76
0.52
0.89
0.34
6:54
1009
0.06
0.35
foci
0.93
0.85
6.76
0.76
0.34
0.66
0.52
0..09
1.55
0.50
0.33
0.71
0396
0.86
0.99
0.92
0.69
10.00
11.00
11.8
log
0053
0.91
0.75
0.54
0.307
0.52
10.00
0.75
0.52
0.56
1L72
0.71
0.93
0.6
0.69
0.72
10:00
0.71
0.20
0.76
02.29
0.35
10.2
hane
0.52
00.02
0.76
0.74
0.55
1996
6.75
0:91
0.96
10:55
0:74
(896)
0.67
0.33
0.80
01:0
11
1179
11.1%
11.9
0394
0.78
11.72
0.25
13.00
13.75
0.75
11:7%
enagg
0.34
(0.1
C.6
0.54
0.77
00:5
0.81
11.16
10:85
1..9
0..79
11.00
0.5
14.2
0.81
0661
0.82
13.79
0.60
13:44
11:77
11.22
11.0
13:74
en
0.99
0.61
0.67
0.71
10:05
0.81
0.
0.50
0:85
6:81
0.72
0.79
0.27
11.00
Tear
0:53
0.6
0.54
0.65
0.42
0.85
(6.58
115
0.74
10:88
0.63
0.775
0.7
1150
0.66
0.76
0.55
0.76
13.
0.77
10:0
0.73
0360
(.0)
11.44
Tick
0:18
0.61
016
0.75
0.76
0.67
0.72
0.7
005
0.96
0.55
0.14
0.79
0.85
0:83
0.73
0.72
0.51
00
0.73
10:00
0.74
0155
wrivella
0.76
0.11
0.75
0.7%
0.71
7.744
6.7%
0.64
16

---

Published as a conference paper at ICLR 2024
Table 13: Per-prompt AUROC for TextNorm on the composition set.
arplane
10.79
0.78
10:1
01.53
18MM
1005
10:00
0.39
18M3
01.25
10:6
apple
0.85
0.77
0.74
10.91
0.9
10.88
10.88
0.56
0.76
10:89
6.77
0.09
0.71
0.71
13.7
1185
18.00
0153
0:48
0.75
0.88
0.77
0.61
10:90
0.11
0.54
0.67
toot
0:51
0.74
0.65
10
0.77
11.3
10.00
0.81
11.55
8.89
0.3
0.48
0.54
01:81
0.85
0.8
0.7
0.77
0.99
0.71
0.87
0.5
0885
0.85
cst
0.83
03:87
0.23
1052
0.72
0.77
0.57
0.89
0..7
1996
6.85
0.72
0.71
0.23
00.83
0.77
0.82
0.00
10.9
0.53
chair
0:94
0..7
0:36
0.92
0.00
03.72
10.8
1US3
10.66
6.75
1005
0.86
1997
0.29
10.00
0.84
0.82
10.8
0.50
00
cap
18.06
0151
0:89
10/97
0.92
0.95
des
2.95
0:92
01.0
10.99
0.75
0.82
0.55
10.99
10.50
0.51
10.52
0.71
0.75
0.75
0.76
1062
0:84
01.
0.60
0.558
0.85
0.89
0.51
0.52
0.89
0.82
0.92
0929
0.77
0..00
13.72
1.57
0.98
DU84
0099
0.89
1094
0.86
0.75
0..11
0.88
10.99
1.59
1US5
my
0-05
13:99
0.8
1116
GA1
0.91
10:98
11.8
0189
11.99
hppip
10.03
0.44
01:22
1189
1..6
11.99
10.5
10.55
00.02
01.66
0.23
00733
(0.84
0.94
10
10.90
18.71
11.30
11:91
01.79
10.2
1L.9
0.85
10594
10.99
10.94
13.96
11:00
11.0
11.8
onags
0:71
13.54
1899
01:22
11.84
1.56
0.51
10:25
10:5
0.77
0.8
0.86
0.75
0.96
10:91
0.76
Islly Tear
0993
01.7
16.85
18:09
0.50
0.54
0.5
10.77
11.54
1.77
10.95
1856
D151
0151
0.83
0.67
0.66
0:85
0.511
0.72
0.72
10.91
0.1
11.75
0:43
0.81
0.07
0.31
0
0183
0.50
0.83
0.66
0155
0.58
0.75
0.71
0.88
0.73
0.1
1150
unbeella
0.96
0.67
1156
0.5
0.59
6.71
0.70
0.9
0396
0.82
0.77
0.82
0.33
88
0.89
0.82
035
0.85
0.9
037
0.88
0.79
0.75
0.80
00:00
0.75
Table 14: Per-prompt AUROC for CLIP on the counting set.
000
DAN
11:0
10:2
0.5
0:ST
100
1151
0.64
041
0.11
lour
0.98
0.57
0.55
0.7
0.35
0:44
0:32
842
0.43
0.46
0.65
0.67
0.65
0.66
6.75
0:55
0.58
0.6
0.54
0.65
0.58
0.72
0.34
the
116
1:63
0.6
0158
5.67
0.29
0.56
0.49
0.1
0.69
0.63
1034
113
0.6
07
0.62
0.58
0.5
0:53
0.38
0.40
1.75
0443
40%
6.91
0.56
0.64
0.52
0.30
0.22
0.5
0.4
0.8
0033
0.
0.4
0.99
0:49
0.38
0.84
0.69
0.59
0.48
0.1
0441
0.1
0.64
0.75
Table 15: Per-prompt AUROC for BLIP-2 on the counting set.
WOlAS Hoar
WOREEE
600
0.8
688
0.76
0.67
Ino:
08
0.64
0.54
0.59
6.19
0.5%
841
0.32
0448
0.36
0.71
0.35
0.6
6.56
0.75
0.5
0.51
0.49
144
06T
0.00
forr
0.75
18.9
0.64
035
0.69
04
0.1
0.2
0.36
0.4
0:45
065
0.55
0:40
0.52
0.64
0.55
0.52
0.51
0.1
0.5
06
0.49
Ine
0.3
0:39
0.63
0.55
0.56
0.55
0.32
0.5
0.46
033
0.27
042
0.55
0.44
0.67
0.66
0.5
0.4
0:22
06
0.34
0.90
0.3h
0.57
0.31
0.28
Table 16: Per-prompt AUROC for ImageReward on the counting set.
5
0.34
[1997]
6.7K
0.81
0.19
1896
0.67
11
0.9K
0/5
0:58
77
033
0.86
0197
1896
0.5K
deee
0.55
0.52
0.54
0.88
0.58
0.71
0.3
064
0.56
0.43
0.62
0.57
0.92
6.74
0.00
0.86
06
0.76
0.45
0.48
0.22
0.58
0.79
0.3
0.86
fnrr
0:33
18:66
0:37
0.54
0.71
0.54
0.82
0.36
0.48
0.43
861
0:41
0.6
0.82
0.6
80
0.69
0.63
0.59
04
844
0.61
88
0:47
0.47
0.4
0.45
0.68
0.57
0.71
0:47
0.45
0.55
0.31
0133
0.33
0.55
0.42
Table 17: Per-prompt AUROC for PickScore the counting set.
0.76
1191
0A6T
0.76
0.88
091
0.72
1w0
0.85
0.9
0.36
109
0.79
0.23
0.65
0.63
11.0
0.74
6.87
0.32
0.95
0.95
0.6
0.99
0.3
0.9
0.5
093
deee
0.31
0.9
0.70
0.57
0.55
0.69
0.94
034
0L77
0.7
IL52
0.25
1000
0,5
0.76
0.75
0.5
0.59
0.7
016
0.25
0.5
0.33
0.0
0.99
knnr
0.76
0:37
8.3
0.7
0.75
0.87
0.6
88
0.8
0.2
Iie
6:42
0.55
0.3
0.96
051
Table 18: Per-prompt AUROC for TextNorm the counting set.
000
6.05
0.7
0.36
0:24
0/22
10:45
0:95
B.97
0.67
0163
0.86
00.3
0.99
0.82
10:6
0.82
0:57
100
0.77
1006
0.5
08L
0.91
0.03
0.34
0.64
0U93
0.49
399
037
0.5
0.83
0.95
0.93
DJ66
0.96
0.98
0.37
0.79
091
fhow
0:8T
9:95
0.85
0.7
0388
RN3
0:8
0.9
0.85
1187
0.98
0.75
0.8
0.4
0.95
0.78
0.8k
Joss
0.7
0.88
0.5
0.93
0.90
0.92
0.67
6.9
6.72
0:48
0.64
0.65
0191
0.6
10.8
0.66
10.84
0.77
0.81
919
0.39
17

---

Published as a conference paper at ICLR 2024
B.4 REWARD MODEL SCORES ON SAMPLE IMAGES
While PickScore generally outperforms the other baselines, there are instances where those baselines exhibit stronger alignment shown in Figure 8. This observation naturally led to considering ensemble methods to achieve further improvement.
Prompt
CLIP
BLIP-2
IR
PS
deer and an orange
0.385
0.407
1.750
0.232
deer
0.314
0.347
-0.028
0.207
a deer and two oranges
0,368
0.391
0.867
0.236
deer-like orange
0.344
0.394
1.454
0.221
an orange-like deer
0.348
0.391
1.023
0.222
Prompt
CLIP
BLIP-2
IR
PS
deer and an orange
0.301
0.411
0.377
0.228
deer
0.314
0.432
1.097
0.220
a deer and two oranges
0.295
0.388
-0.906
0.222
deer-like orange
0.304
0.372
0.125
0.222
an orange-like deer
0.321
0.394
0.225
0.224
Figure 8: Reward model scores sample images and prompts.
C MORE HUMAN EVALUATION RESULTS
For the best-of-n sampling experiment, annotators evaluate the top four images selected from the n samples based on the reward models for each prompt. As illustrated by the evaluation results summarized in Figure TextNorm consistently outperforms all baselines in terms of text-image alignment, often by a substantial margin, across all three values of n. Especially compared to CLIP and BLIP-2, TextNorm achieves notably better text-image alignment well image quality. Even when compared to ImageReward and PickScore, TextNorm achieves two to three times as many wins losses alignment, with comparable or only minor compromises in image quality.
Teal Imnge Alganeni
Teak Imngle Aiganent
Text Eeaii-emagecccccc Agganezi
CLIP
BLB
Image Quilty
Tranee Qualtty
Image Quality
C1.0
CLtr
(a) Bfst-of-16
(b) Best-of-64
(c) Best-of-256
Figure 9: Results of human evaluation of the images selected using best-of-n sampling.
18

---

Published as a conference paper at ICLR 2024
D DETAILS ON PROMPT SYNTHESIS VIA LLMS
Algorithm outlines detailed pseudocode for instructing ChatGPT to generate a contrastive prompt based on given input prompt zo. Depending on the nature of zo. assign appropriate categories and provide an optional list of few-shot examples to guide generation through in-context learning. For example, given the text prompt "A black colored car". we categorize under colors and counting and provide few-shot examples that encompass common failure cases for the two categories. our experiments, we use the cpt-4-0613 model for contrastive prompt generation, configuring the temperature to 0.0 and the frequency penalty to 0.2. These hyperparameters affect the generation process based on the token frequencies.
Table 19 presents the few-shot examples considered in our experiments for eight distinct categories: text, style, composition, counting, creative, location, colors, and spatial, To generate diverse and high-quality prompts, we vary the quantity and type of objects described in the input prompt, along with other relevant properties like spatial relations.
Algorithm Prompt set synthesis via ChatGPT
x 0: input prompt T, temperature, frequency penalty examples per_peereeeeeeeeee _er_cateoor:: few-shot examples for each category
invoke ChatGPT to return chat completion
def model completion (x 0, model prompt, **params):
proopp = model woomppprompp.toommmmmm copy ()
"user", "content"; x_0)) create(
resp = openai..ChttComppeee...
engine="gpt-4406133
messages-model prompt, **params) return resp
prepare the LIM prompt with few-shot examples and invoke the model def synthesize prompts (x_ 0, T, P):
messages = [[
"role": "system",
"content": "Create captions that are different from the original input used for the text-to-image generation model, referencing the provided failure cases. The new captions should offer perspectives that context of the images. Ensure that each contrasting caption provides distinct perspective, while maintaining the integrity of the image's 's subject matters. Let' think step by step."
distinct from the original
}] allocate category to input prompt x_0
categories category allocation (x__0 (x_0)
for category categories:
Lew_shot. few_shot_examppess
category "[An"
for example in examples per category [category]
few wt__sshttt_pppla original *r""p""+*"'n"
"Original prompt:" example("
few shot examples += "Contrasting captions:"
for idx, caption in enumerate (example ["proper__ee [""roporrcandddaaeeeeee candidates"))
few_shot eav_shot__xxmppes += f" (idxt1) ( (caption) \n"
few few_shee shot examples += "J\n"
messages message
[0] ["content += few shot examples params ""temperature": T, "frequency pocaootyeeallt"; P}
return model, m conellli_omleiio((ee (x_0, messages, params)
19

---

Published as a conference paper at ICLR 2024
Table 19: Few-shot examples for each category of prompts used for in-context learning.
Category: Text
Input: "A sign that says "Diffusion""
Output: ["A sign misspelled "Difision.." ",, "A sign containing bizarre 'Difšion'."
Category: Style
Input: "Greek statue of man tripping over a cat."
Output: ["Greek statue of man", "Greek statue of two men", men"," "Greek statue of two men tripping over cat.", "Gree, statue of a man tripping over a dog.", statue of a man tripping over ball.")
"oree. "Greek statue of two men tripping over dog.", "Greek
Category: Composition
Input: "A red car and white sheep."
Output: ["A red and two white sheep." "A red and herd of sheep.", rod and dominant white sheep among the gray ones." "A red with real sheep on the side.")
Category: Counting
Input: "Four cars on street."
Output: ["Two cars on the street.", "trect."""Thro "hhree cars on the strect.", "Five cars on the street.", "trect""""iix cars on the street.") "]
Category: Creative
Input: "A heart made of chocolate"
made of caramel", "a flower made marshmallows," "a diamond made gummy hears"; "a made of licorice",," "a sun made jelly beans"."a butterfly made of lollipops", "a made of cotton candy"," "a rainbow made of skittles"," "a cloud made of cotton candy".'s "a made of chocolate-covere, pret- zels"."a "a snowflake made of peppermint candies". "a fish made of gummies". "a bird made of chocolate- covered almonds", "a made of chocolate bars", "a house made of chocolate cookies",* "a boat made of chocolate-con
Output:
strawberries" "an airplane made of chocolate truffles""," "a guitar made of chocolate wafer sticks"." "a camera made of chocolate coins". "a dinosaur made of chocolate eggs")
Category: Location
Input: "A glowing mushroom the forest"
Output: ['a sparkling flower in the garden", "adeen"""a luminous firefly the night sky", "a shimmering starfish in the occan" radiant sunflower in field" glowing jellyfish the deep sea" sea"."a gleaming diamond in jewelry store". "a luminescent moon the night sky" gloowingg firefly in meadow", "/ sparkling gemstone the cave", "a luminous butterfly in garden". shimmering seashell on the beach", radiant rainbow in the sky", glowing lantern the dark", luminescent lightning bug the field", sparkling erystal in the night sky", "a glowing firefly the park", "a
cave", "a shimmering waterfall in the forest", "a radiant luminous pearl in the oyster", "a sparkling diamond in the jewelry box"]
Category: Colors
Input: "A blue bird and brown bear."
Output: ["A pair of bears, one blue and the other brown." "A blue bear and a hrown hear, no bird in sight.", "Two "Two bears, both brown, no blue "Two "Two bears, one brown and the other unexpectedly blue.")
Category: Spatial
Input: "An umbrella on top of spoon."
Output: ["A spoon.", ", "An umbrella." "An umbrella the right of spoon." "An umbrella the left of spoon." ", "An umbrella the bottom of spoon.", "wwo umbrellas top of spoon.")
20


Reviews:
[{"review_id": "CWiJz5JwQq", "reviewer": "Reviewer_9bKW", "summary": "The paper proposes a simple method to prevent the overoptimization problem in alignment of text-to-image models. They work hypothesizes that the overoptimization problem is mainly due to lack of alignment between the human preference and the learnt reward model. Inorder to solve this issue, they propose two techniques namely textual normalization which better normalizes the reward using contrastive text description. Further they propose textual inconsistency score, which compares the distance between the textual description and the predicted reward . They find that both these objectives help improve the calliberation and alignment with the human reward.", "questions": "- Is there a fundamental reason why Softmax CLIP objective is better at caliberation than Sigmoid CLIP objective(https://arxiv.org/abs/2303.15343) ?\n\n-  \"Hence, we use the insight that the terms that contribute significantly to the denominator of the softmax score are the ones for which rφ(x, y) is sufficiently large. \" -  It's unclear to me how using semantic-contrastive semantic prompts results in finding the terms that have high reward value?\n\n- Can the authors instead of finding contrastive prompts, take random prompts and show an ablation?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The problem that paper picks namely reward overoptimization. Seems to be important with very works addressing it.\n- The paper does a good job at benchmarking previous reward functions on various settings.\n- The paper proposes novel techniques to improve the alignment and caliberation of the current reward models.", "weaknesses": "- The paper talks about overoptimization as the motivation, however doesn't directly evaluate for overoptimization. It instead evaluates for caliberation and human evaluation. To my understanding , current works get away with overoptimization issue with early stopping such as AlignProp (https://arxiv.org/abs/2310.03739) however this is not discussed in the paper. I would expect the right way to evaluate would be to not do early stopping and see the tradeoffs with the proposed solutions. Also talk about the benefits of not doing early stopping.\n- Further AlignProp, says that they only need to do early stopping for the Aesthetics reward function, however not for HPS v2. I think it would be worth comparing againt the HPS v2 reward function and also discuss methods such as AlignProp or DDPO (https://arxiv.org/abs/2305.13301) and the tricks they use to prevent overoptimization. And why using those tricks might not be a good idea.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Eb9N9lBeVP", "reviewer": "Reviewer_exWV", "summary": "This paper proposed a text-image alignment benchmark and reward score normalization methods to evaluate and address the \"overoptimization\" issue when finetuning with a reward model. The proposed benchmark contains 550 prompts covering comprehensive/counting/compostion categories and 27500 generated images. The normalization method consider normalizing text-image reward scores by contrasting to generated texts. The human evaluation and qualitative results showing the proposed method performs better on the \"overoptimization\" issue over other baselines.", "questions": "- Section 5.2. \"select the top 10% of the samples as measured by the chosen reward model\". Is that means the model finetuned with different reward models are finetuned with different data? Why not use the same data for finetuning?\n- Any Failure cases for proposed text norm method? (failure cases for both comprehensive/counting/composition are helpful)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to understand\n- The proposed benchmark showing the drawbacks of different reward models\n- The proposed text normalization method can reduce the overoptimization problem and improves the model finetuning performance", "weaknesses": "- One major contribution is the proposed text-image alignment assessment benchmark. However, the role/completeness of the benchmark are not fully explored. For example: 1. Are the comprehensive/counting/composition enough to compare those reward models? 2. In figure 3 (a), we can see ImageReward > PickScore > CLIP > BLIP-2 in terms of \"comprehensive\". Are these reward model ranking align with human evaluation (Do human also thinks ImageReward > PickScore > CLIP > BLIP-2)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nIYHuXduXb", "reviewer": "Reviewer_fRcb", "summary": "This paper study the problem of reward functions on human feedback data for image diffusion models. It proposes a new text-image alignment assessment benchmark (TIA2) for evaluating the problem of reward designing. It shows that several existing reward-based models are not well-aligned with human assessment after evaluated on the benchmark. The authors propose a method TextNorm to induce confidence calibration in reward models. It shows the proposed score help reduce over-optimization in the human feedback tuning. Experiments are conducted with Stable Diffusion v2.1 and LoRA for parameter-efficient fine-tuning.", "questions": "In the human evaluation, the authors are comparing with “the best-performing baseline reward model” but did not mention which one it is. Please provide the details on which reward function you are comparing with.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThe proposes benchmark for evaluating text-image alignment can help the reward designing in human feedback data. It includes different categories of prompts: comprehensive, counting and composition, which are the key categories that common image reward functions such as CLIP and ImageReward cannot do well on.\n-\tThe proposed TextNorm is novel and shows superior performance on improving calibration errors on different types of prompts. It also improves text-image alignment compared to existing reward functions according to some qualitative results and human evaluation.\n-\tAn ablation study is also conducted to analyze the impact of different aspects: normalization over prompts, prompt synthesis, and the textual inconsistency score.", "weaknesses": "-\tIn the process of prompt synthesis using language model, the authors only ask the LLM to generate variations based on the object category and number of objects. Other important aspects such as spatial relationships and object attributes like material, color, size is not included. This greatly limit the scope of the experiment in this work. I also only saw sampled result images related to object category and object number in this paper. It would be great if more analysis on those other aspects can be studied as well.\n-\tIn figure 5, the difference of TextNorm and existing methods are not significant. BLIP2 has very similar results as the TextNorm on the correctness of the object number and categories.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)