SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Setup and results of the human evaluation. Table 5 below presents the results of our human evaluation on CNN/DM summarization under the T5-base LM. We generally adopt the protocol in Stiennon et al. [12] to evaluate the overall summary quality. Our model is compared with the baselines Supervised, Supervised+PPO, and Supervised+NLPO in RL4LMs [61]. The result of the reference summaries is also presented, which is intended for sanity check rather than method comparison. In conducting this evaluation, we randomly picked 100 articles in the test split of CNN/DM and showed to 20 qualified evaluators the summaries generated from each method, along with the article. The method names were anonymized. The evaluators were asked to read the article and score each summary. Summaries are scored on a 5-Point Likert Scale  $\{1,2,3,4,5\}$ , where score 5 is the highest and 1 the lowest. From Table 5, it is clear that human evaluation supports the improvements in ROUGE, Meteor, and BertScore by our method in Table 4.
Table 5: Average human ratings on CNN/DM summarization under the T5-base LM. We bold the best result apart from the ground-truth Reference summary. A detailed description on the setup is in the above text.
Supervised	Supervised+PPO	Supervised+NLPO	Ours	Reference
Average Human Rating	2.92	3.17	3.29	3.61	3.88
Table 6: Scores on each ROUGE metric for our method using sequence-level and token-level preference-based guidance in the summarization tasks in Section 4.3 (a). "Seq." denotes our method with sequence-level preference-based guidance, and "Token" denotes our method with token-level preference-based guidance. The reported numbers are mean (standard deviation) over three random seeds. The row "Average" shows the average of the three ROUGE scores, i.e., (ROUGE-1 + ROUGE-2 + ROUGE-L) / 3.
CNN/DM	XSum	CNN/DM (T5-base LM)
Seq.	Token	Seq.	Token	Seq.	Token
ROUGE-1	40.20 (0.07)	40.94 (0.02)	32.56 (0.08)	33.62 (0.03)	42.10 (0.15)	43.09 (0.06)
ROUGE-2	17.80 (0.08)	18.78 (0.03)	9.98 (0.04)	11.17 (0.02)	19.23 (0.11)	20.17 (0.04)
ROUGE-L	37.08 (0.06)	38.17 (0.03)	25.11 (0.07)	26.33 (0.05)	38.09 (0.14)	39.99 (0.07)
Average	31.69	32.63	22.55	23.71	33.14	34.42
Table 7: Scores on each ROUGE metric for our method with and without the reward-function retraining scheme in the summarization tasks in Section 4.3 (b). "Without Retrain" denotes our method without reward-function retraining, and "With Retrain" denotes our method with reward-function retraining. The reported numbers are mean (standard deviation) over three random seeds. The row "Average" shows the average of the three ROUGE scores, i.e., (ROUGE-1 + ROUGE-2 + ROUGE-L) / 3.
CNN/DM	XSum	CNN/DM (T5-base LM)
Without Retrain	With Retrain	Without Retrain	With Retrain	Without Retrain	With Retrain
ROUGE-1	40.83 (0.10)	40.94 (0.02)	33.45 (0.11)	33.62 (0.03)	42.98 (0.08)	43.09 (0.06)
ROUGE-2	18.70 (0.07)	18.78 (0.03)	11.07 (0.06)	11.17 (0.02)	20.09 (0.06)	20.17 (0.04)
ROUGE-L	38.07 (0.09)	38.17 (0.03)	26.23 (0.10)	26.33 (0.05)	39.87 (0.08)	39.99 (0.07)
Average	32.53	32.63	23.58	23.71	34.31	34.42
Table 8: Scores on each ROUGE metric for the summarization task on CNN/DM in Section 4.3 (c), where we vary the number of sequences used to learn the token-level guidance. The reported numbers are mean (standard deviation) over three random seeds. The row "Average" shows the average of the three ROUGE scores, i.e.,  $(\text{ROUGE-1} + \text{ROUGE-2} + \text{ROUGE-L}) / 3$ .
Number of Sequences
2	3	5	7	9
ROUGE-1	40.80 (0.06)	40.94 (0.02)	40.87 (0.09)	40.86 (0.08)	40.95 (0.01)
ROUGE-2	18.70 (0.04)	18.78 (0.03)	18.71 (0.02)	18.74 (0.06)	18.78 (0.01)
ROUGE-L	38.05 (0.03)	38.17 (0.03)	38.09 (0.07)	38.08 (0.08)	38.18 (0.02)
Average	32.52	32.63	32.56	32.56	32.64
(a) Temperature  $\beta$  (SST-2)
(b) Temperature  $\beta$  (CNN/DM)
(c) Balancing Coeff.  $\alpha$  (SST-2)
Figure 6: Line plots comparing the performance under different values of the hyperparameter  $\beta$  in Eq. (4) and  $\alpha$  in Eq. (5). The plotted numbers are mean over three random seeds. Error bars show one standard deviation.
Table 9: Scores on each ROUGE metric for the summarization task on XSum in Section 4.3 (c), where we vary the number of sequences used to learn the token-level guidance. The reported numbers are mean (standard deviation) over three random seeds. The row "Average" shows the average of the three ROUGE scores, i.e.,  $(\text{ROUGE-1} + \text{ROUGE-2} + \text{ROUGE-L}) / 3$ .
Number of Sequences
2	3	5	7	9
ROUGE-1	33.54 (0.06)	33.62 (0.03)	33.56 (0.08)	33.56 (0.02)	33.63 (0.02)
ROUGE-2	11.12 (0.04)	11.17 (0.02)	11.12 (0.05)	11.19 (0.05)	11.20 (0.03)
ROUGE-L	26.26 (0.06)	26.33 (0.05)	26.28 (0.06)	26.34 (0.06)	26.36 (0.03)
Average	23.64	23.71	23.65	23.70	23.73
A.2 Further Ablation Study
In learning the preference-based sequence-level guidance in Section 4.3, the aggregation function  $f(\cdot)$  in Section 2.1 is removed, since it is inapplicable and unnecessary to the sequence-level reward function. For the minimalist LM training objectives Eqs. (5) and (6) in Section 2.2, we change them to the corresponding versions that use sequence-level guidance. Self-normalization in reward-weighted MLE Eq. (6) is removed, since it is again inapplicable and unnecessary to the sequence-level setting.
In this section, we continue our discussion in the Ablation Study (Section 4.3) by answering the following additional questions on our method.
(a): Is our method robust to the hyperparameter(s): temperature  $\beta$  and balancing coefficient  $\alpha$ ?
To study the choice of the temperature parameter  $\beta$  in the soft-maximum/minimum aggregation Eq. (4), we vary the value of  $\beta$  in the MIN variant in Tables 1 and 2 from  $\beta = 2$ . Furthermore, to study the balancing coefficient  $\alpha$  in the REINFORCE-style LM-training approach Eq. (5), we vary the  $\alpha$  parameter in the AVG variant in Table 1 from  $\alpha = 2^{-3}$ . Fig. 6 respectively shows the prompt results on the SST-2 dataset and the summarization results on the CNN/DM dataset. For summarization, we again plot the average ROUGE scores, with the breakdown scores of the three ROUGE metrics in Table 10 below.
Recall that the best baseline result on SST-2 in Table 1 is 90.5, and on CNN/DM in Table 2 is 31.3. We see that our method can achieve competitive results on a relatively wide range of the temperature  $\beta$ . A too-small value of  $\beta$ , such as 0.25 and 0.5, may incur a harder optimization problem and thus an inferior performance on both prompt and summarization tasks.
For the choice of the balancing coefficient  $\alpha$ , we see that our method provides competitive results in a relatively wide range of  $\alpha \in [0.08, 0.15]$ , when compared to the best baseline result of 90.5 in Table 1. A too-small value of  $\alpha$  may not prevent the REINFORCE-style method from pre-mature convergence. The resulting LM therefore may not sufficiently explore the sampling space or capture multiple good behavior-modes, resulting in an inferior and highly varying performance. A too-large value of  $\alpha$  distracts the optimization of the LM, and again leads to a worse result.
Table 10: Scores on each ROUGE metric for the summarization task on CNN/DM, where we vary the temperature parameter  $\beta$  in the soft-minimum aggregation Eq. (4). The reported numbers are mean (standard deviation) over three random seeds. The row "Average" shows the average of the three ROUGE scores, i.e., (ROUGE-1 + ROUGE-2 + ROUGE-L) / 3.
β = 2-2	β = 2-1	β = 20	β = 21	β = 22
ROUGE-1	40.77 (0.11)	40.74 (0.09)	40.79 (0.11)	40.78 (0.06)	40.80 (0.01)
ROUGE-2	18.67 (0.06)	18.68 (0.05)	18.68 (0.09)	18.67 (0.03)	18.71 (0.04)
ROUGE-L	38.00 (0.10)	37.98 (0.08)	38.03 (0.12)	38.01 (0.04)	38.02 (0.01)
Average	32.48	32.47	32.50	32.49	32.51
(b): How does our method perform in generating longer prompts compared with the baseline?
To further validate the harm of the delayed-feedback issue to the related LM-training methods that learn under the sparse sequence-level feedback, we compare our method with RLPrompt [60] on generating prompts with length increased from 5 to 10 and to 20 tokens, on the SST-2 dataset. Table 11 below shows the results.
Table 11: Test accuracy on the prompt task on the SST-2 dataset, for our method and RLPrompt on generating prompts with a length of 5, 10, and 20 tokens. We report the mean and standard deviation over three random seeds.
RLPrompt	Ours (AVG)	Performance Gap
5 Tokens	90.5 (1.5)	92.6 (1.7)	2.1
10 Tokens	75.8 (7.6)	86.0 (2.9)	10.2
20 Tokens	65.2 (6.0)	80.9 (4.5)	15.7
We see that RLPrompt performs worse than our method on generating longer prompts. In particular, the performance gap increases as the prompt length (feedback delaying) increases. This comparison can further demonstrate the harm of the delayed-feedback issue in training text-generation LMs, and that our framework, in particular our preference-grounded token-level guidance for LM training, is a viable solution to it.
It is intrigued that the results of both methods deteriorate with the prompt length. After checking the generated prompts from our method, we find that longer prompts mostly contain many repeated tokens, as shown by the following example prompt of length 20
PerformanceExceptionMovieMovieMovieMovieMovieMovieVideoVideoVideoVideoVideoVideoVideoVideoImageVideoImageImage
which is separated into two lines at the location of “\” due to the page-width limit. In this prompt example, the tokens Movie and Video are each consecutively repeated seven times, and the bigram ImageVideo is repeated two times. Such prompts with heavy repetitions may confuse the downstream classifier.<sup>6</sup> This aligns with our intuition that a clear and succinct instruction is preferable than a long but verbose one.
As a side note, in generating Table 11, we use the default hyperparameters for both our method and RLPrompt. It is possible that RLPrompt requires careful tuning for generating longer prompts, due to the delayed-feedback issue that we try to address. We leave a thorough tuning of RLPrompt on long-prompt generation as a future work.
(c): Is the efficacy of our framework tied to the specific preference sources considered in Section 4?
To investigate whether the performance of our framework is tied to the specific preference-sources considered in the experiment section (Section 4), inspired by RL4LMs [61], we simulate the sequence-level preference on the summarization task by using another two automatic metrics "Rouge-avg" and "Rouge-avg2", rather than the classical Meteor score [90] in Section 4. Table 12 below presents the ROUGE scores of our method under each of the three preference sources on the CNN/DM dataset under the T5-base LM. For a more thorough investigation, we provide the results for our method both with and without the guidance re-estimation scheme. The baseline results in Table 12 below come from the best baseline method in Table 4 of Appendix A.1.
A detailed description of the prompt task is deferred to Appendix D.
Table 12: Results for our method on CNN/DM summarization under T5-base LM when using different automatic metrics to simulate the sequence-level preference. We provide the detailed ROUGE scores for our method both with and without guidance re-estimation. "Baseline" denotes the results of the best baseline method in Table 4 of Appendix A.1. The reported numbers are the mean over three random seeds. The row "Average" shows the average of the three ROUGE scores, i.e., (ROUGE-1 + ROUGE-2 + ROUGE-L) / 3.
Baseline	With Guidance Re-estimation	Without Guidance Re-estimation
Rouge-avg	Rouge-avg2	Meteor	Rouge-avg	Rouge-avg2	Meteor
ROUGE-1	42.9	43.14	43.07	43.09	42.96	42.98	42.98
ROUGE-2	19.4	20.18	20.12	20.17	20.07	20.05	20.09
ROUGE-L	36.1	39.93	39.89	39.99	39.80	39.77	39.87
Average	32.8	34.42	34.36	34.42	34.28	34.27	34.31
Concretely, these two new automatic metrics "Rouge-avg" and "Rouge-avg2" are constructed as
$$
\text {R o u g e - a v g} = 0. 5 \times \text {R O U G E - 1} + 0. 5 \times \text {R O U G E - 2} + 0. 5 \times \text {R O U G E - L},
$$
$$
\text {R o u g e - a v g} 2 = 0. 5 \times \text {R O U G E -} 1 + 0. 5 \times 2 \times \text {R O U G E -} 2 + 0. 5 \times \text {R O U G E - L},
$$
where the "Rouge-avg" metric is exactly the same as that in the RL4LMs [61]. The "Rouge-avg2" metric is constructed by multiplying ROUGE-2 by 2 to make its numerical value similar to the others.
It is clear that changing the preference source from Meteor to these two alternative metrics does not significantly alter the performance of our method, especially when compared to the performance improvement of our method over the best baseline method in Table 4 of Appendix A.1. This set of comparisons confirms that the efficacy of our framework is generally not tied to a specific preference source. It could also further corroborate the effectiveness of our preference-grounding perspective on guiding the LM training.
B Additional Experiment Details
B.1 Prompt Generation
Implementation Details. To ensure a fair comparison, the implementation of our framework is based on the official codebase of RLPrompt available at https://github.com/mingkaid/rl-prompt, and the Hugging Face library [73]. We have provided some implementation details in Section 4.1. Here we continue the discussion.
The LM  $\pi_{\theta}$  is parametrized as a frozen distilGPT-2 model with parameter  $\theta$  being one MLP-layer of size 2048 inserted right before the output head. The token-level reward function  $r_{\phi}$  is implemented as a distilGPT-2 with a two-layer projection-MLP of sizes 2048 and 1 on top. The LM  $\pi_{\theta}$  is trained by a maximum of 12000 steps, with early stopping based on the validation set. The reward training is reconducted every 1000 steps during the first 6000 steps of the LM training process and is (and almost always) early stopped. RoBERTa-large is used [9] as the pre-trained downstream LM  $\pi_{\mathrm{DLM}}$ .
Datasets. We use the standard datasets provided in the RLPrompt codebase [60]. We test on three popular few-shot classification datasets in prior work [e.g., 77, 78], i.e., two sentiment binary-classification datasets SST-2 [79] and Yelp Polarity [81], and the topic four-way-classification dataset AG News [81]. In keeping with the standard few-shot setting [76], both the training and the validation sets have 16 examples per class. To mitigate the randomness in the few-shot setting, each dataset is subsampled into five few-shot training and validation sets, while the test set is standard. We train our models on each few-shot (sub-)dataset with three random seeds and evaluate three generated prompts in each case. For all three tested datasets, we report the average test accuracy and standard deviation across all evaluated prompts in all random seeds and all few-shot (sub-)datasets.
Hyperparameters. Apart from the hyperparameters discussed in the ablation study (Section 4.3 and Appendix A.2), most other hyperparameters as well as the training and evaluation procedures of our framework follow RLPrompt. Additionally, we list the important hyperparameters for training our reward model in Table 13, and important hyperparameters for training our LM in Table 14. The generated prompts have a fixed length of 5. The same hyperparameters are used in all tested datasets.
Baselines. For the baseline results in Table 1, we rerun the codebase of RLPrompt under the same random seeds and evaluation script as our method. Other baseline results are from the literature
[60, 88]. We note that our reported RLPrompt results have some small discrepancies compared to the original paper's results. We have confirmed our reproduced results with RLPrompt's authors and with Table 2 of the recent TEMPERA paper [88].
Table 13: Hyperparameters for training our reward model in the prompt-generation task.
Hyperparameter	Value
Gradient clipping norm	5.0
Max train steps	10000
Steps per epoch	100
Number of epochs	100
Learning rate	5e-5
Batch size	64
Learning-rate decay	0.8
Learning-rate scheduler	ReduceLROnPlateau
Scheduler patience	2
Early-stop count	7
Optimizer	Adam [95]
Backbone	distilGPT-2
Table 14: Hyperparameters for training our LM in the prompt-generation task.
Hyperparameter	Value
Gradient clipping norm	5.0
Max train steps	12000
Steps per epoch	500
Number of epochs	24
Learning rate	5e-5
Batch size	32
Learning-rate decay	0.8
Learning-rate scheduler	ReduceLROnPPlateau
Scheduler patience	2
Early-stop count	7
Optimizer	Adam
Backbone	distilGPT-2
Reward retrain period	1000 steps
B.2 Text Summarization
Implementation Details and Hyperparameters. The implementation of our framework is based on the Hugging Face library [73]. We have provided some implementation details in Section 4.2. The discussion is continued here.
Due to our limited computational resources, unless explicitly mentioned, we use the standard T5-small model [89] for the LM. Similar to the prompt tasks, the token-level reward function is implemented also as a T5-small model, with a two-layer projection-MLP on top with sizes 2048 and 1. The LM  $\pi_{\theta}$  is trained for a standard 5 epochs. Apart from the hyperparameters discussed in the ablation study (Section 4.3 and Appendix A.2), most other hyperparameters as well as the training and evaluation procedure of our framework follow the standard setting of using a T5 model for text summarization on the Hugging Face library. Additionally, we list the important hyperparameters for training our reward model in Table 15, and important hyperparameters for training our LM in Table 16. The same hyperparameters are used in both the CNN/DailyMail and the XSum datasets.
We note that the ROUGE-L metric we report is technically the rougeLsum metric from the Hugging Face interface and in the RL4LMs' codebase [61]. This one matches the result scales in prior work especially on texts with newlines ("\n"), as reported in this GitHub issue.
Baselines. For the baseline methods' results in Table 2, we rerun the codebase of RL4LMs [61] with a T5-small model as our method. We have carefully tuned the (supervised+) PPO/NLPO in RL4LMs on several hyperparameters, such as learning_rate, kl_div:coeff, kl_div:target_kl, and so on. Furthermore, we ran these baseline methods on the same random seeds as our method and we provide error bars. Since we use the T5-small model and the same random seeds for both our method and the baselines, our reported results are therefore (more) fair comparisons.
Table 15: Hyperparameters for training our reward model in the text-summarization task.
Hyperparameter	Value
Gradient clipping norm	5.0
Number of epochs	1
Amount of training data	10% of training set
Learning rate	5e-5
Batch size	32
Optimizer	Adam
Backbone	T5-small
Table 16: Hyperparameters for training our LM in the text-summarization task.
Hyperparameter	Value
Gradient clipping norm	5.0
Number of epochs	5
Learning rate	5e-5
Batch size	32
Optimizer	AdamW [96]
Weight decay	0.0
Backbone	T5-small
Reward retrain period	0.5 epoch
C A Naïve Numeric Example for the Average Aggregation
This section provides a naive numeric comparison that the average aggregation in Section 2.1 will not automatically favor longer sequences, while the classical summation will.
Suppose we have  $K = 2$  sequences  $\tau^1$  and  $\tau^2$  for preference learning, respectively having length  $T^1 = 5$  and  $T^2 = 15$ . For simplicity, assume that all tokens in  $\tau^1$  and  $\tau^2$  are the same and all have reward 1, i.e.,  $r_{\phi}(s_t^k, a_t^k) = 1, \forall k, t$ . The average sequence length  $C$  is then  $C = (1/2) \times (5 + 15) = 10$ . For the first sequence  $\tau^1$ , the average-aggregated sequence-level evaluation  $e_{\phi}^{\mathrm{avg}}(\tau^1) = (10/5) \times \sum_{t=0}^{4} 1 = (10/5) \times 5 = 10$ . And for the second sequence  $\tau^2$ ,  $e_{\phi}^{\mathrm{avg}}(\tau^2) = (10/15) \times \sum_{t=0}^{14} 1 = (10/15) \times 15 = 10$ . Therefore, no sequence will be automatically preferred based only on the length.
By contrast, when using the classical summation as the aggregation function,  $\tau^1$  will be evaluated as  $\sum_{t=0}^{4} 1 = 5$  while  $\tau^2$  will be evaluated as  $\sum_{t=0}^{14} 1 = 15$ . So, indeed, the longer sequence  $\tau^2$  will be automatically preferred.
D Details on the Prompt Generation Task
Task Description. In discrete text-prompt generation [e.g., 10, 74], we input a discrete text-prompt  $\pmb{a}$  and an observation sequence  $o$  to a large pre-trained downstream LM  $\pi_{\mathrm{DLM}}(y_{\mathrm{DLM}} \mid \pmb{a}, o)$  to directly classify text  $o$ , without finetuning  $\pi_{\mathrm{DLM}}$ . Here,  $y_{\mathrm{DLM}}$  denotes the output of the large downstream LM  $\pi_{\mathrm{DLM}}$  on the observation text  $o$  prompted by text  $\pmb{a}$ . We follow the classical prompt setting [e.g., 10, 75, 60] that solves the classification problem by an encoder-only downstream LM via token infilling. Classification is reduced to selecting tokens corresponding to some predefined class labels, known as verbalizers, such as "happy" for positive and "sad" for negative. The set of verbalizers is denoted as  $\mathcal{C}$ . As an example, to classify an observation text  $o$  by prompt  $\pmb{a}$  using an encoder-only downstream LM  $\pi_{\mathrm{DLM}}$ , we input a template such as "[o] [a] [MASK]" to  $\pi_{\mathrm{DLM}}$ , and select the most probable verbalizer token that fills into [MASK].
Setting. In our input-agnostic setting, the generated prompt is independent of the observation text  $o$ . During inference time, only the learned prompts are used and the LM  $\pi_{\theta}$  is discarded. The initial input  $x$  to  $\pi_{\theta}$  is a dummy, and the target  $y$  is the class label in the mask position. We also adopt the few-shot setting, where the training set consists of a small number of samples per class. There is a larger standard test set for evaluation. With a fixed length  $T$ , the goal is to find discrete text-prompts  $\mathbf{a} = (a_0, \dots, a_{T - 1})$  that have high test accuracy.
Source of the Preference. For learning the token-level guidance, we simulate the sequence-level preference by the recently proposed stepwise metric  $\mathcal{R}_{\mathrm{step}}$  in Deng et al. [60], i.e., the higher the metric value the better prompt. This choice ensures a fair comparison with RLPrompt [60] and avoids a potential overfitting that we train and evaluate the LM on the same evaluation metric "accuracy".
Given a prompt  $\pmb{a}$ , observation text  $o$ , and the true class label  $y \in \mathcal{C}$ ,  $\mathcal{R}_{\mathrm{step}}$  measures the gap between the true class's probability and the highest probability in other classes. The gap is defined as
$$
\mathrm {G a p} _ {o} (\pmb {a}, y) = \pi_ {\mathrm {D L M}} (y | \pmb {a}, o) - \max _ {y ^ {\prime} \in \mathcal {C}, y ^ {\prime} \neq y} \pi_ {\mathrm {D L M}} (y ^ {\prime} | \pmb {a}, o),
$$
where  $\mathrm{Gap}_o(a,y) > 0$  when the prediction  $y_{\mathrm{DLM}}(a,o)$  for text  $o$  is correct and  $< 0$  otherwise. Define the indicator for correct prediction for  $o$ ,  $\mathrm{Corr}_o$ , as  $\mathrm{Corr}_o = \mathbf{1}\{\mathrm{Gap}_o(a,y) > 0\}$ . The stepwise metric  $\mathcal{R}_{\mathrm{step}}$  for prompt  $a$  on observation text  $o$  and true class label  $y$  is defined as
$$
\mathcal {R} _ {\mathrm {s t e p}} (y _ {\mathrm {D L M}} (\boldsymbol {a}, o), y) = \lambda_ {1} ^ {1 - \operatorname {C o r r} _ {o}} \lambda_ {2} ^ {\operatorname {C o r r} _ {o}} \times \operatorname {G a p} _ {o} (\boldsymbol {a}, y),
$$
where  $\lambda_1 = 180$  and  $\lambda_{2} = 200$ . In the experiments (Section 4 and Appendix A.2), we report test accuracy as in prior works.
LM Training. Since the prompt-generation task does not assume the availability of supervised data — the ground-truth prompts, the LM  $\pi_{\theta}$  is trained by the REINFORCE-style update in Section 2.2 to automatically discover highly-accurate prompts.