SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Reward Weighted. [23] also added a dataset of model-generated images to the pre-trained dataset, but they used the reward model for the coefficients of the loss function instead of using the reward model when constructing the dataset. Specifically, their loss function is shown in 4.
$$
\mathcal {L} (\theta) = \mathbb {E} _ {(x, z) \sim \mathcal {D} ^ {\text {m o d e l}}} [ - r _ {\phi} (x, z) \log (p _ {\theta} (x | z)) ] + \beta \mathbb {E} _ {(x, z) \sim \mathcal {D} ^ {\text {p r e}}} [ - \log (p _ {\theta} (x | z)) ] \tag {4}
$$
where  $\theta$  denotes parameters of pre-trained LDM,  $\phi$  denotes parameters of the reward model, and  $\beta$  is a penalty parameter. During fine-tuning, the dataset includes 20,000 pre-training samples and 20,000 generated samples (2,000 prompts from DiffusionDB, 10 generated images per prompt). In [23], the reward is constrained within the range [0, 1], while our model follows an approximately normal distribution with a mean of 0 and a variance of 1. Therefore, in order to replicate their methodology, we need to map the scores to the range [0, 1]. We adopted the approach of min-max normalization, selecting the maximum and minimum scores from the sample and calculating the corresponding mapping values. Additionally, we set the penalty coefficient  $\beta = 0.5$ .
RAFT. [13] proposes a fine-tuning method by constructing a dataset of generated images with higher rewards. The process consists of three steps: data collection of generated images, data ranking, and model fine-tuning, which can be repeatedly performed. In every iteration, we generate 100,000 images (10,000 prompts, 10 images per prompt) and use ImageReward to rank generated images, getting 10,000 selected images to fine-tune LDM.
F More Results of ReFL
F.1 Demonstrations of ReFL
Figure 19 shows the insight of ImageReward score during denoising.
Figure 19: (Left) Correlation between Spearman  $\rho$  and denoising step rate. (Right) Correlation between ImageReward score and denoising step rate.
F.2 ReFL compares other fine-tuning methods
Figure 20 shows the comparison between ReFL and other fine-tuning methods. ReFL gets the highest win rate compared to any other method.
More qualitative examples of ReFL are provided in Figure 21 - 22.
G Additional Results of ImageReward Compared to Other Typical Image Scorers
Figure 20: Win rate between different fine-tuning methods.
More qualitative examples of ImageReward can be seen in Figure 23 - 26.
Original
Dataset Filtering
Reward Weighted
RAFT
ReFL (Ours)
Figure 21: More examples for comparing fine-tuning methods.
Figure 22: More examples for comparing fine-tuning methods.
Figure 23: Qualitative comparison with previous typical methods. Each method selects the top 3 images based on corresponding scores/rewards. Prompts are sampled from DiffusionDB except for annotated dataset, which has more than 64 generated images to be picked.
Figure 24: Qualitative comparison of ImageReward to typical image scoring methods.
BLJP Score CLJP Score
BLJP Score
Figure 25: Qualitative comparison of ImageReward to typical image scoring methods.
CLIP Score
Figure 26: Qualitative comparison of ImageReward to typical image scoring methods.
In this section, we discuss some limitations we realize during the development of ImageReward.
Annotation scale, diversity, and quality. Although our annotation data has reached up to about 9k prompts and 137k pairs of expert comparisons, the larger scale of the annotation dataset is still needed for better RM training. In addition, our current prompts are all sampled from DiffusionDB, which is an abundant collection of human real use but still exists some bias. Despite these prompts may close to many real cases, biases exist since the real application when people use the text-to-image model are far beyond trying strange prompts. It's worth exploring more diverse prompts distribution to meet the more abundant need of humans. Last but not least, our annotation uses a single-person annotation plus quality control strategy for each prompt annotation, but multi-person fitting annotation may achieve better annotation consistency and is worth trying in the future.
RM training techniques. As we mentioned in Section 2.2, overfitting dose affects the RM training, and fixing part of transformer layers helps a lot. Nevertheless, we speculate that more advanced techniques (e.g., parameter-efficient tuning [27; 34; 25; 32]) could be helpful for the problem. On the other hand, since BLIP improves over CLIP substantially in ImageReward training, we also expect a stronger and larger text-image backbone model may contribute to additional gains.
Using RM to improve generative models. Though we have proposed ReFL as an effective method to utilize human preference scorers' feedback to optimize LDMs, it remains an approximation of original RLHF algorithms and could be improved fundamentally. It is necessary to develop corresponding unbiased and efficient feedback learning algorithms with solid theory groundings to allow better human alignment.
I Broader Impact
The aim of this paper is to introduce human preference feedback to improve text-to-image generation, which will help image generation to better match the needs of human life and to conform to social norms. Fine-tuning the model with human feedback helps to avoid researchers from over-relying on various types of data with copyright issues for training, and can instead directly improve performance with reward model feedback. A downside is that the preferences of a single reward model are not representative of the multiplicity of human aesthetics, and we can address this by training a variety of reward models and limiting the use of individual reward models. We believe that these benefits outweigh the drawbacks.
J Reproducibility
We have made substantial efforts to guarantee the reproducibility of our assessments. The code and detailed information for the ImageReward model and ReFL algorithm are openly accessible in our repository (Cf. Abstract). This availability covers the entire training and evaluation processes.
Training. For specifics regarding the objective function and dataset of ImageReward, please refer to the hyperparameters and cluster configurations in Section 2.2. Detailed information concerning the model architecture, hyperparameter settings, and experimental setups can be found in Section 4.1.
Evaluation. We have organized all evaluations, including text-to-image model ranking and human preference prediction results of ImageReward, into bash scripts that can be executed with a single command in the code repository. Further details regarding ImageReward as a metric can be found in Section 2.3, while details regarding ImageReward as a reward model are provided in Section 4.1.
ReFL. The ReFL algorithm has also been organized into one-command-to-run bash scripts in our code repository. Detailed insights into ReFL can be located in Section 4.2.