SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \operatorname {K L} \left(p _ {\theta} \left(x _ {0: T} \mid z\right) \| p _ {\text {p r e}} \left(x _ {0: T} \mid z\right)\right) = \int p _ {\theta} \left(x _ {0: T} \mid z\right) \times \log \frac {p _ {\theta} \left(x _ {0 : T} \mid z\right)}{p _ {\text {p r e}} \left(x _ {0 : T} \mid z\right)} d x _ {0: T} \\ = \int p _ {\theta} \left(x _ {0: T} | z\right) \log \frac {p _ {\theta} \left(x _ {T} | z\right) \prod_ {t = 1} ^ {T} p _ {\theta} \left(x _ {t - 1} \mid x _ {t} , z\right)}{p _ {\mathrm {p r e}} \left(x _ {T} | z\right) \prod_ {t = 1} ^ {T} p _ {\mathrm {p r e}} \left(x _ {t - 1} \mid x _ {t} , z\right)} d x _ {0: T} \\ = \int p _ {\theta} \left(x _ {0: T} | z\right) \left(\log \frac {p _ {\theta} \left(x _ {T} \mid z\right)}{p _ {\mathrm {p r e}} \left(x _ {T} \mid z\right)} + \sum_ {t = 1} ^ {T} \log \frac {p _ {\theta} \left(x _ {t - 1} \mid x _ {t} , z\right)}{p _ {\mathrm {p r e}} \left(x _ {t - 1} \mid x _ {t} , z\right)}\right) d x _ {0: T} \\ = \mathbb {E} _ {p _ {\theta} (x _ {0: T} | z)} \left[ \sum_ {t = 1} ^ {T} \log \frac {p _ {\theta} (x _ {t - 1} | x _ {t} , z)}{p _ {\mathrm {p r e}} (x _ {t - 1} | x _ {t} , z)} \right] = \sum_ {t = 1} ^ {T} \mathbb {E} _ {p _ {\theta} (x _ {t: T} | z)} \mathbb {E} _ {p _ {\theta} (x _ {0: t - 1} | x _ {t: T}, z)} \left[ \log \frac {p _ {\theta} (x _ {t - 1} | x _ {t} , z)}{p _ {\mathrm {p r e}} (x _ {t - 1} | x _ {t} , z)} \right] \\ = \sum_ {t = 1} ^ {T} \mathbb {E} _ {p _ {\theta} (x _ {t} | z)} \mathbb {E} _ {p _ {\theta} (x _ {0: t - 1} | x _ {t}, z)} \left[ \log \frac {p _ {\theta} (x _ {t - 1} | x _ {t} , z)}{p _ {\mathrm {p r e}} (x _ {t - 1} | x _ {t} , z)} \right] = \sum_ {t = 1} ^ {T} \mathbb {E} _ {p _ {\theta} (x _ {t} | z)} \mathbb {E} _ {p _ {\theta} (x _ {t - 1} | x _ {t}, z)} \left[ \log \frac {p _ {\theta} (x _ {t - 1} | x _ {t} , z)}{p _ {\mathrm {p r e}} (x _ {t - 1} | x _ {t} , z)} \right] \\ = \sum_ {t = 1} ^ {T} \mathbb {E} _ {p _ {\theta} (x _ {t} | z)} \left[ \mathrm {K L} \left(p _ {\theta} \left(x _ {t - 1} | x _ {t}, z\right) \| p _ {\text {p r e}} \left(x _ {t - 1} | x _ {t}, z\right)\right) \right]. \\ \end{array}
$$
Taking the expectation of  $p(z)$  on both sides, we have
$$
\mathbb {E} _ {p (z)} \left[ \mathrm {K L} \left(p _ {\theta} \left(x _ {0} \mid z\right)\right) \| p _ {\text {p r e}} \left(x _ {0} \mid z\right)\right) ] \leqslant \mathbb {E} _ {p (z)} \left[ \sum_ {t = 1} ^ {T} \mathbb {E} _ {p _ {\theta} \left(x _ {t} \mid z\right)} \left[ \mathrm {K L} \left(p _ {\theta} \left(x _ {t - 1} \mid x _ {t}, z\right) \| p _ {\text {p r e}} \left(x _ {t - 1} \mid x _ {t}, z\right)\right) \right] \right]. \tag {15}
$$
A.3 The gradient of objective Eq. (8)
From Lemma 4.2, for online fine-tuning, we need to regularize  $\sum_{t=1}^{T} \mathbb{E}_{p_{\theta}(x_t)} \left[ \mathrm{KL}\left(p_{\theta}(x_{t-1}|x_t,z) \| p_{\mathrm{pre}}(x_{t-1}|x_t,z)\right) \right]$ . By the product rule, we have
$$
\begin{array}{l} \nabla_ {\theta} \sum_ {t = 1} ^ {T} \mathbb {E} _ {p _ {\theta} (x _ {t})} \left[ \mathrm {K L} \left(p _ {\theta} \left(x _ {t - 1} | x _ {t}, z\right) \| p _ {\mathrm {p r e}} \left(x _ {t - 1} | x _ {t}, z\right)\right) \right] \\ = \sum_ {t = 1} ^ {T} \mathbb {E} _ {p _ {\theta} \left(x _ {t}\right)} \left[ \nabla_ {\theta} \mathrm {K L} \left(p _ {\theta} \left(x _ {t - 1} \mid x _ {t}, z\right) \| p _ {\text {p r e}} \left(x _ {t - 1} \mid x _ {t}, z\right)\right) \right] \tag {16} \\ + \sum_ {t = 1} ^ {T} \mathbb {E} _ {p _ {\theta} (x _ {t})} \left[ \sum_ {t ^ {\prime} > t} ^ {T} \nabla_ {\theta} \log p _ {\theta} \left(x _ {t ^ {\prime} - 1} \mid x _ {t ^ {\prime}}, z\right) \cdot \operatorname {K L} \left(p _ {\theta} \left(x _ {t - 1} \mid x _ {t}, z\right) \| p _ {\text {p r e}} \left(x _ {t - 1} \mid x _ {t}, z\right)\right) \right], \\ \end{array}
$$
which treats the sum of conditional KL-divergences along the future trajectory as a scalar reward at each step. However, computing these sums is more inefficient than just the first term in Eq. (16). Empirically, we find that regularizing only the first term in Eq. (16) already works well, so we adopt this approach for simplicity.
A.4 Lemma 4.3
Similar to derivation in [10], and notice that  $q$  is not dependent on  $z$  (i.e.,  $q(\dots |z) = q(\dots)$ ), we have
$$
\begin{array}{l} \mathbb {E} _ {p (z)} \mathbb {E} _ {p _ {\text {p r e}} (x _ {0} | z)} [ - (r (x _ {0}, z) + \gamma) \log p _ {\theta} (x _ {0} | z) ] \\ \leqslant \mathbb {E} _ {p (z)} \mathbb {E} _ {p _ {\text {p r e}} (x _ {0} | z)} \mathbb {E} _ {q (x _ {1: T} | x _ {0}, z)} \left[ - (r (x _ {0}, z) + \gamma) \log \frac {p _ {\theta} (x _ {0 : T} | z)}{q (x _ {1 : T} | x _ {0} , z)} \right] \\ = \mathbb {E} _ {p (z)} \mathbb {E} _ {p _ {\text {p r e}} (x _ {0} | z)} \mathbb {E} _ {q (x _ {1: T} | x _ {0}, z)} [ - (r (x _ {0}, z) + \gamma) \log \frac {p (x _ {T} | z)}{q (x _ {T} | x _ {0} , z)} \\ - \sum_ {t > 1} (r \left(x _ {0}, z\right) + \gamma) \log \frac {p _ {\theta} \left(x _ {t - 1} \mid x _ {t} , z\right)}{q \left(x _ {t - 1} \mid x _ {t} , x _ {0} , z\right)} \tag {17} \\ \left. - \left(r \left(x _ {0}, z\right) + \gamma\right) \log p _ {\theta} \left(x _ {0} \mid x _ {1}, z\right) \right] \\ = \mathbb {E} _ {p (z)} \mathbb {E} _ {p _ {\text {p r e}} (x _ {0} | z)} [ (r (x _ {0}, z) + \gamma) \mathbb {E} _ {q (x _ {T} | x _ {0}, z)} [ \mathrm {K L} (q (x _ {T} | x _ {0}) \| p (x _ {T})) ] \\ + (r (x _ {0}, z) + \gamma) \sum_ {t > 1} \mathbb {E} _ {q \left(x _ {t} \mid x _ {0}, z\right)} [ \mathrm {K L} (q \left(x _ {t - 1} \mid x _ {t}, x _ {0}\right) \| p _ {\theta} \left(x _ {t - 1} \mid x _ {t}, z\right)) ] \\ - (r \left(x _ {0}, z\right) + \gamma) \mathbb {E} _ {q \left(x _ {1} \mid x _ {0}, z\right)} \left[ \log p _ {\theta} \left(x _ {0} \mid x _ {1}, z\right) \right], \\ \end{array}
$$
where the first inequality comes from ELBO.
Let  $\tilde{\mu}_t(x_t,x_0)\coloneqq \frac{\sqrt{\bar{\alpha}_{t - 1}}\beta_t}{1 - \bar{\alpha}_t} x_0 + \frac{\sqrt{\alpha_t}(1 - \bar{\alpha}_{t - 1})}{1 - \bar{\alpha}_t} x_t$  and  $\tilde{\beta}_t = \frac{1 - \bar{\alpha}_{t - 1}}{1 - \bar{\alpha}_t}\beta_t$ , we have:
$$
\begin{array}{l} \mathbb {E} _ {p (z)} \mathbb {E} _ {p _ {\text {p r e}} (x _ {0} | z)} [ (r (x _ {0}, z) + \gamma) \sum_ {t > 1} \mathbb {E} _ {q (x _ {t} | x _ {0}, z)} [ \mathrm {K L} (q (x _ {t - 1} | x _ {t}, x _ {0}) \| p _ {\theta} (x _ {t - 1} | x _ {t}, z)) ] ] \\ = \mathbb {E} _ {p (z)} \mathbb {E} _ {p _ {\text {p r e}} (x _ {0} | z)} [ (r (x _ {0}, z) + \gamma) \sum_ {t > 1} \mathbb {E} _ {q \left(x _ {t} \mid x _ {0}, z\right)} [ \frac {1}{2 \sigma_ {t} ^ {2}} \| \tilde {\mu} _ {t} \left(x _ {t}, x _ {0}\right) - \mu_ {\theta} \left(x _ {t}, t, z\right) \| ^ {2} ] ] + C, \tag {18} \\ \end{array}
$$
Notice that  $p_{\theta}(x_0|z, x_1)$  is modeled as a discrete decoder as in [10] and  $p(x_T|z)$  is a fixed Gaussian; neither is trainable.
$$
\begin{array}{l} \mathbb {E} _ {p (z)} \mathbb {E} _ {p _ {\text {p r e}} (x _ {0} | z)} [ - (r (x _ {0}, z) + \gamma) \log p _ {\theta} (x _ {0} | z) ] \\ \leqslant \mathbb {E} _ {p (z)} \mathbb {E} _ {p _ {\text {p r e}} \left(x _ {0} \mid z\right)} \left[ \left(r \left(x _ {0}, z\right) + \gamma\right) \sum_ {t > 1} \mathbb {E} _ {q \left(x _ {t} \mid x _ {0}, z\right)} \left[ \frac {1}{2 \sigma_ {t} ^ {2}} \left| | \tilde {\mu} _ {t} \left(x _ {t}, x _ {0}\right) - \mu_ {\theta} \left(x _ {t}, t, z\right) \right| ^ {2} \right] \right] + C _ {1} \tag {19} \\ \end{array}
$$
Furthermore, from triangle inequality we have
$$
\begin{array}{l} \sum_ {t > 1} \mathbb {E} _ {q \left(x _ {t} \mid x _ {0}, z\right)} [ \gamma \frac {\left| \left| \tilde {\mu} _ {t} \left(x _ {t} , x _ {0}\right) - \mu_ {\theta} \left(x _ {t} , t , z\right) \right| \right| ^ {2}}{2 \sigma_ {t} ^ {2}} ] \tag {20} \\ \leqslant \sum_ {t > 1} \mathbb {E} _ {q \left(x _ {t} \mid x _ {0}, z\right)} [ \gamma \frac {\left| \left| \tilde {\mu} _ {t} \left(x _ {t} , x _ {0}\right) - \mu_ {\text {p r e}} \left(x _ {t} , t , z\right) \right| \right| ^ {2} + \left| \left| \mu_ {\text {p r e}} \left(x _ {t} , t , z\right) - \mu_ {\theta} \left(x _ {t} , t , z\right) \right| \right| ^ {2}}{2 \sigma_ {t} ^ {2}} ] \\ \end{array}
$$
So another weaker upper bound is given by
$$
\begin{array}{l} \mathbb {E} _ {p (z)} \mathbb {E} _ {p _ {\text {p r e}} (x _ {0} | z)} [ - (r (x _ {0}, z) + \gamma) \log p _ {\theta} (x _ {0} | z) ] \\ \leqslant \mathbb {E} _ {p (z)} \mathbb {E} _ {p _ {\text {p r e}} (x _ {0} | z)} \left[ \right. \sum_ {t > 1} \mathbb {E} _ {q \left(x _ {t} \mid x _ {0}, z\right)} \left[ \frac {r \left(x _ {0} , z\right)\left|\left| \tilde {\mu} _ {t} \left(x _ {t} , x _ {0}\right) - \mu_ {\theta} \left(x _ {t} , t , z\right)\right|\right| ^ {2}}{2 \sigma_ {t} ^ {2}} \right. \tag {21} \\ \left. + \frac {\gamma (\| \mu_ {\mathrm {p r e}} (x _ {t} , t , z) - \mu_ {\theta} (x _ {t} , t , z) \| ^ {2})}{2 \sigma_ {t} ^ {2}} ] \right] + C _ {2}. \\ \end{array}
$$
Notice that  $p_{\theta}(x_0|z,x_1)$  is modeled as a discrete decoder as in [10] and  $p(x_{T}|z)$  is a fixed Gaussian; neither is trainable. Then the proof is complete.
A.5 Value function learning
Similar to [35], we can also apply the variance reduction trick by subtracting a baseline function  $V(x_{t},z,\theta)$ :
$$
\begin{array}{l} \mathbb {E} _ {p (z)} \mathbb {E} _ {p _ {\theta} \left(x _ {0: T} | z\right)} \left[ - r \left(x _ {0}, z\right) \sum_ {t = 1} ^ {T} \nabla_ {\theta} \log p _ {\theta} \left(x _ {t - 1} \mid x _ {t}, z\right) \right] \tag {22} \\ = \mathbb {E} _ {p (z)} \mathbb {E} _ {p _ {\theta} (x _ {0: T} | z)} \left[ - \sum_ {t = 1} ^ {T} \left(r (x _ {0}, z) - V (x _ {t}, z, \theta)\right) \nabla_ {\theta} \log p _ {\theta} (x _ {t - 1} | x _ {t}, z) \right], \\ \end{array}
$$
$$
V \left(x _ {t}, z, \theta\right) := \mathbb {E} _ {p _ {\theta} \left(x _ {0: t}\right)} [ r \left(x _ {0}, z\right) | x _ {t} ]. \tag {23}
$$
So we can learn a value function estimator by minimizing the objective function below for each  $t$ :
$$
\mathbb {E} _ {p _ {\theta} \left(x _ {0: t}\right)} \left[ \left(r \left(x _ {0}, z\right) - \hat {V} \left(x _ {t}, \theta , z\right)\right) ^ {2} \mid x _ {t} \right]. \tag {24}
$$
Since subtracting  $V(x_{t},\theta ,z)$  will minimize the variance of the gradient estimation, it is expected that such a trick can improve policy gradient training. In our experiments, we find that it can also improve the final reward: For example, for the prompt "A dog on the moon", adding variance reduction can improve ImageReward from 0.86 to 1.51, and also slightly improve the aesthetic score from 5.57 to 5.60. The generated images with and without value learning are shown in Figure 6. We also find similar improvements in multi-prompt training (see learning curves with and without value learning in Figure 7).
With value learning
Figure 6: Text prompt: "A dog on the moon", generated from the same random seeds from the model trained with and without value learning, respectively.
Without value learning
A.6 Importance sampling and ratio clipping
In order to reuse old trajectory samples, we can apply the important sampling trick:
$$
\begin{array}{l} \mathbb {E} _ {p (z)} \mathbb {E} _ {p _ {\theta} (x _ {0: T} | z)} \left[ - \sum_ {t = 1} ^ {T} \left(r (x _ {0}, z) - V (x _ {t}, z, \theta)\right) \nabla_ {\theta} \log p _ {\theta} (x _ {t - 1} | x _ {t}, z) \right] \\ = \mathbb {E} _ {p (z)} \mathbb {E} _ {p _ {\theta_ {\mathrm {o l d}}} (x _ {0: T} | z)} \left[ - \sum_ {t = 1} ^ {T} \left(r (x _ {0}, z) - V (x _ {t}, z, \theta)\right) \frac {p _ {\theta} (x _ {t - 1} | x _ {t} , z)}{p _ {\theta_ {\mathrm {o l d}}} (x _ {t - 1} | x _ {t} , z)} \nabla_ {\theta} \log p _ {\theta} (x _ {t - 1} | x _ {t}, z) \right] \\ = \mathbb {E} _ {p (z)} \mathbb {E} _ {p _ {\theta_ {\text {o l d}}} (x _ {0: T} | z)} \left[ - \sum_ {t = 1} ^ {T} \left(r \left(x _ {0}, z\right) - V \left(x _ {t}, z, \theta\right)\right) \nabla_ {\theta} \frac {p _ {\theta} \left(x _ {t - 1} \mid x _ {t} , z\right)}{p _ {\theta_ {\text {o l d}}} \left(x _ {t - 1} \mid x _ {t} , z\right)} \right]. \tag {25} \\ \end{array}
$$
In order to constrain  $p_{\theta}$  to be close to  $p_{\theta_{\mathrm{old}}}$ , we can use the clipped gradient for policy gradient update similar to PPO [36]:
$$
\mathbb {E} _ {p (z)} \mathbb {E} _ {p _ {\theta_ {\text {o l d}}} \left(x _ {0: T} \mid z\right)} \left[ - \sum_ {t = 1} ^ {T} \left(r \left(x _ {0}, z\right) - V \left(x _ {t}, z, \theta\right)\right) \nabla_ {\theta} \operatorname {c l i p} \left(\frac {p _ {\theta} \left(x _ {t - 1} \mid x _ {t} , z\right)}{p _ {\theta_ {\text {o l d}}} \left(x _ {t - 1} \mid x _ {t} , z\right)}, 1 + \epsilon , 1 - \epsilon\right) \right], \tag {26}
$$
where  $\epsilon$  is the clip hyperparameter.
Online RL training. For hyper-parameters of online RL training used in Section 5.2., we use  $\alpha = 10, \beta = 0.01$ , learning rate  $= 10^{-5}$  and keep other default hyper-parameters in AdamW, sampling batch size  $m = 10$ . For policy gradient training, to perform more gradient steps without the need of re-sampling, we perform 5 gradient steps per sampling step, with gradient norm clipped to be smaller than 0.1, and use importance sampling to handle off-policy samples, and use batch size  $n = 32$  in each gradient step. We train the models till generating 20000 online samples, which matches the number of samples in supervised fine-tuning and results in 10K gradient steps in total. For stable training, we freeze the weights in batch norm to be exactly the same as the pretrained diffusion model. For importance sampling, we apply the clip hyperparameter  $\epsilon = 10^{-4}$ .
Supervised training. For hyper-parameters of supervised training, we use  $\gamma = 2.0$  as the default option in Section 5.2, which is chosen from  $\gamma \in$
$\{0.1, 1.0, 2.0, 5.0\}$ . We use learning rate  $= 2 \times 10^{-5}$  and keep other default hyper-parameters in AdamW, which was chosen from  $\{5 \times 10^{-6}, 1 \times 10^{-5}, 2 \times 10^{-5}, 5 \times 10^{-5}\}$ . We use batch size
Figure 7: Learning curves with and without value learning, trained on the Drawbench prompt set: Adding value learning could result in higher reward using less time.
$n = 128$  and  $M = 20000$  such that both algorithms use the same number of samples, and train the SFT model for 8K gradient steps. Notice that Lemma 4.3 requires  $r + \gamma$  to be non-negative, we apply a filter of reward such that if  $r + \gamma < 0$ , we just set it to 0. In practice, we also observe that without this filtering, supervised fine-tuning could fail during training, which indicates that our assumption is indeed necessary.
C Investigation on ImageReward
We investigate the quality of ImageReward [43] by evaluating its prediction of the assessments of human labelers. Similar to Lee et al. [17], we check whether ImageReward generates a higher score for images preferred by humans. Our evaluation encompasses four text prompt categories: color, count, location, and composition. To generate prompts, we combine words or phrases from each category with various objects. These prompts are then used to generate corresponding images using a pre-trained text-to-image model. We collect binary feedback from human labelers on the image-text dataset and construct comparison pairs based on this feedback. Specifically, we utilize 804, 691, 1154 and 13228 comparison pairs obtained from 38, 29, 41 and 295 prompts for color, location, count, and composition, respectively. Additionally, for evaluation on complex text prompts from human users, we also utilize the test set from ImageReward, which consists of 6399 comparison pairs from 466 prompts. Table 2 compares the accuracy of ImageReward and baselines like CLIP [27] and BLIP [18] scores, which justifies the choice of using ImageReward for fine-tuning text-to-image models.
Color	Count	Location	Composition	Complex
CLIP	82.46	66.11	80.60	70.98	54.83
BLIP	77.36	61.61	78.14	70.98	57.79
ImageReward	86.06	73.65	79.73	79.65	65.13
Table 2: Accuracy (%) of CLIP score, BLIP score and ImageReward when predicting the assessments of human labelers.
D A Comprehensive Discussion of Related Work
We summarize several similarities and differences between our work and a concurrent work [2] as follows:
- Both Black et al. [2] and our work explored online RL fine-tuning for improving text-to-image diffusion models. However, in our work, we provide theoretical insights for optimizing the reward with policy gradient methods and provide conditions for the equivalence to hold.
- Black et al. [2] demonstrated that RL fine-tuning can outperform supervised fine-tuning with reward-weighted loss [17] in optimizing the reward, which aligns with our own observations.
- In our work, we do not only focus on reward optimization: inspired by failure cases (e.g., over-saturated or non-photorealistic images) in supervised fine-tuning [17], we aim at finding an RL solution with KL regularization to solve the problem.
- Unique to our work, we systematically analyze KL regularization for both supervised and online fine-tuning with theoretical justifications. We show that KL regularization would be more effective in the online RL fine-tuning rather than the supervised fine-tuning. By adopting online KL regularization, our algorithm successfully achieves high rewards and maintains image quality without over-optimization issue.
4Full list of text prompts is available at [link].
$^{5}$ https://github.com/THUDM/ImageReward/blob/main/data/test.json
E More Experimental Results
E.1 KL-D vs KL-O: Ablation Study on Supervised KL Regularization
We also present the effects of both KL-D and KL-O with coefficient  $\gamma \in \{0.1, 1.0, 2.0, 5.0\}$  in Figure 8 and Figure 9. We observe the followings:
- KL-D (eq. (11)) can slightly increase the aesthetic score. However, even with a large value of  $\gamma$ , it does not yield a significant improvement in visual quality. Moreover, it tends to result in lower ImageReward scores overall.
- In Figure 9, it is evident that KL-O is more noticeably effective in addressing the failure modes in supervised fine-tuning (SFT) compared to KL-D, especially when  $\gamma$  is relatively large. However, as a tradeoff, KL-O significantly reduces the ImageReward scores in that case.
- In general, achieving both high ImageReward and aesthetic scores simultaneously is challenging for supervised fine-tuning (SFT), unlike RL fine-tuning (blue bar in Figure 8).
We also present more qualitative examples from different choices of KL regularization in Appendix E.5.
(a) ImageReward score
(b) Aesthetic score
Figure 8: (a) ImageReward scores and (b) Aesthetic scores of supervised fine-tuned (SFT) models with different choices of regularization term and coefficient  $(\gamma)$  on text prompt "A green colored rabbit". Each score is averaged over 50 samples from each model. KL-O and KL-D refer to eq. (12) and eq. (11), respectively.
Figure 9: (Left) Sample from the RL model with KL regularization and sample from the SFT model without KL regularization. (Right) Samples from supervised fine-tuned models with different choices of regularization term and coefficient  $(\gamma)$  on "A green colored rabbit". As we increase  $\gamma$  in the SFT case, there is a tradeoff between alignment with the prompt and image quality (i.e. oversaturation). On the other hand, the RL with KL sample is able to retain both alignment and image quality.
E.2 Images from unseen text prompts
Figure 10 shows image samples from the original model, SFT model and RL fine-tuned models on unseen text prompts.
Figure 10: Comparison of images generated by the original Stable Diffusion model, supervised fine-tuned (SFT) model, and RL fine-tuned model. Images in the same column are generated with the same random seed. Images from unseen text prompts: "A green colored cat" (color), "A cat and a cup" (composition), "Four birds in the park" (count), and "A lion on the moon" (location).
E.3 Multi-prompt Training
Here we report more details for multi-prompt training. We still use the same setting in Appendix B, but 1) increase the batch size per sampling step to  $n = 45$  for stable training; 2) use a smaller KL weight  $\beta = 0.001$ ; 3) use extra value learning for variance reduction. For value learning, we use a larger learning rate  $10^{-4}$  and batch size 256. Also, we train for a longer time for multi-prompt such that it utilizes 50000 online samples. See sample images in Figure 11.
E.4 Long and Complex Prompts
We show that our method does not just work for short and simple prompts like "A green colored rabbit", but can also work with long and complex prompts. For example, in Fig. 12 we adopt a complex prompt and compare the images generated by the original model and the supervised fine-tuned model. We can observe that online training encourages the models to generate images with a different style of painting and more fine-grained details, which is generally hard to achieve by supervised fine-tuning only (with KL-O regularization,  $\gamma = 2.0$ ).
E.5 More Qualitative Results from Ablation Study for KL Regularization in SFT
Here we provide samples from more prompts in KL ablation for SFT: see Figure 13.
E.6 Qualitative Comparison
Here we provide more samples from the experiments in Section 5.2 and Section 5.3: see Figure 14, Figure 16, Figure 15, Figure 17, Figure 18 and Figure 19, with the same configuration as in Section 5.2 for both RL and SFT.
Figure 11: Sample images generated from prompts: (a) "A chair in the corner on a boat"; (b) "A dog is laying with a remote controller"; (c) "A cube made of brick"; (d) "A red book and a yellow vase", from the original model and RL model respectively. Images in the same column are generated with the same random seed.
Figure 12: Text prompt: "oil portrait of archie andrews holding a picture of among us, intricate, elegant, highly detailed, lighting, painting, artstation, smooth, illustration, art by greg rutowski and alphonse mucha".
RL with KL
SFT w.o KL
T-TH qaa LDS O-TX qaa LDS
(a) "Four wolves in the park"
(b) "A cat and a dog"
(c) "A dog on the moon"
Figure 13: Samples from supervised fine-tuned models with different KL regularization and KL coefficients.
(a) Original model
(b) SFT model
(c) RL model
Figure 14: Randomly generated samples from (a) the original Stable Diffusion model, (b) supervised fine-tuned (SFT) model and (c) RL fine-tuned model.
(c) RL model
Figure 15: Randomly generated samples from (a) the original Stable Diffusion model, (b) supervised fine-tuned (SFT) model and (c) RL fine-tuned model.
(c) RL model
Figure 16: Randomly generated samples from (a) the original Stable Diffusion model, (b) supervised fine-tuned (SFT) model and (c) RL fine-tuned model.
(c) RL model
Figure 17: Randomly generated samples from (a) the original Stable Diffusion model, (b) supervised fine-tuned (SFT) model and (c) RL fine-tuned model.
(a) RL model without KL regularization
(b) RL model with KL regularization
Figure 18: Randomly generated samples from RL fine-tuned models (a) without and (b) with KL regularization.
(a) SFT model without KL regularization
(b) SFT model with KL regularization
Figure 19: Randomly generated samples from supervised fine-tuned (SFT) models (a) without and (b) with KL regularization.