Language Model Self-improvement by Reinforcement Learning Contemplation
Authors: Jing-Cheng Pang
Pengyuan Wang
Kaiyuan Li
Xiong-Hui Chen
Jiacheng Xu
Zongzhang Zhang
Yang Yu
Venue: ICLR 2024 poster (2024)
Keywords: Reinforcement learning; language model self-improvement; text evaluation
Primary area: reinforcement learning
TL;DR: This paper proposes a simple language model self-improvement methods that adapts to various sizes of models.

Abstract: Language model self-improvement (LMSI) techniques have recently gained significant attention as they improve language models without requiring external supervision. A common approach is reinforcement learning from AI feedback (RLAIF), which trains a reward model based on AI preference data and employs a reinforcement learning algorithm to train the language model. 
However, RLAIF relies on the heuristic assumption that an AI model can provide effective feedback and correct wrong answers, requiring a solid capability of the language model. This paper presents a novel LMSI method, Reinforcement Learning Contemplation (RLC). We disclose that it is simpler for language models to evaluate a sentence than to generate it, even for small language models. Leveraging the gap between the evaluation and generation, RLC evaluates generated answers and updates language model parameters using reinforcement learning to maximize evaluation scores. Through testing on various challenging reasoning tasks and text summarization task, our experiments show that RLC effectively improves language model performance without external supervision, resulting in an answering accuracy increase (from 31.23% to 37.09%) for BigBench-hard reasoning tasks, and a rise in BERTScore for CNN/Daily Mail summarization tasks. Furthermore, RLC can be applied to models of different sizes, showcasing its broad applicability.

Published as a conference paper at ICLR 2024
LANGUAGE MODEL SELF-IMPROVEMENT BY REIN- FORCEMENT LEARNING CONTEMPLATION
Jing-Cheng Pang Pengyuan Wang .2. Kaiyuan Li', Xiong-Huo Chen 1,, Jiacheng Xu', Zongzhang Zhang' & Yang 12,.2,3,0
National Key Laboratory for Novel Software Technology. Nanjing University, China & School of Artificial Intelligence, Nanjing University, China;
PPolixinai; Peng Cheng Laboratory, Shenzhen, 518055, China
ABSTRACT
Language model self-improvement (LMSI) techniques have recently gained sig- nificant attention as they improve language models without requiring external supervision. A common approach is reinforcement learning from Al feedback (RLAIF), which trains a reward model based on preference data and employs a reinforcement learning algorithm to train the language model. However, RLAIF relies on the heuristic assumption that an AI model can provide effective feedback and correct wrong answers, requiring solid capability of the language model. This paper presents a novel LMSI method, Reinforcement Learning Contemplation (RLC). We disclose that it is simpler for language models evaluate a sentence than to generate it, even for small language models. Leveraging the gap between the evaluation and generation, RLC evaluates generated answers and updates language model parameters using reinforcement learning to maximize evaluation scores. Through testing on various challenging reasoning tasks and text summarization task, our experiments show that RLC effectively improves language model perfor- mance without external supervision, resulting an answering accuracy increase (31 .3% - 37.09%) for BigBench-hard reasoning tasks, and rise in BERTScore for CNN/Daily Mail summarization tasks. Furthermore, RLC can be applied models of different sizes, showcasing its broad applicability.
INTRODUCTION
Scaling large language models (LLM) has led to remarkable performance in various natural language processing (NLP) tasks, such as language understanding, machine translation, and question answering (Zhao et al., 2023 Liu et al. (023). However, training LLMs scale often requires extensive high- quality supervised datasets (Huang et al. 2022a). To rectify this drawback, unsupervised training of LLMs has gained significant interest recently, offering the potential for training LLMs without relying on external supervision. This paper calls these methods language model self-improvement (LMSI) methods.
Two notable LMSI works are RLCAI (Bai et al. 2022) and RLAIF (Lee et al. 2023 RLCAI employs a model evaluate which of the two answers better adheres subsequently creating a dataset of AI preferences. The system then trains using reinforcement learning (RL), with the preference model as the reward signal. This method aims enhance specific properties of generated text, such as helpfulness and harmlessness. RLAIF, on the other hand, follows a similar training process (i.e., reward model + RL) but further validates that AI feedback performs comparably to human feedback in summarization tasks. However, these approaches rely the heuristic assumption that AI models can provide effective feedback or correct errors, which requires strong capability of the language models and is not sufficiently justified in their works. Besides, the need to collect preference data and train an extra reward model introduces added costs.
of rules or principles,
Compared to these methods, humans possess the capacity for relf-improvement through reflection on past experiences, as indicated by contemplative science within the field of human psychology (Van Gordon al. 2022). This process does not necessitate the explicit learning of an additional pref-
"Equal contribution. Corresponding: yuy @jj..cdu.cn.

---

Published as a conference paper at ICLR 2024
erence model for self-feedback Consequently, we aim to investigate the analogous self- ceff--imrovementt capabilities within LLM. In this paper, we propose a novel LMSI method, which is grounded on the observation that it is simpler for a language model to evaluate a sentence than to generate it. For example, while writing an attractive story can be challenging, identifying the generated text is relatively easy. Fig. iillustrates the dis- parity between text generation and evaluation. We comprehensively investigate the text evalu- ation ability of language models through experi- ments across various NIP tasks. Leveraging this evaluation ability, we present Reinforcement Learning Contemplation (RLC). Given an unla- belled question set, the language model gener- ates answers to unlabeled samples and directly evaluates the quality of these answers. The model is then updated using reinforcement learn- ing optimize for maximum evaluation scores. Our proposed method employs eefluaaalttionn results as rewards and utilizes reinforcement learning to optimize the language model. We refer to this process as reinforcement learning contemplation.
Text generation task
Self-evaluation task
Write a mystery that leaves the render unable to predict ending. The answer
the following artiele a mystery that leaves the reader unable to predict eading? [TEXT], Answer:
Language model
Language model
The answer is: Dotective James had been working on the case for weeks, but he couldn't seem to make any headway.
Answer: Yes, is.
Text generation
Text evaluation
Figure I: A comparison between text generation and evaluation. Text evaluation involves evaluating and analyzing certain properties of existing text, while generation requires the creation of entirely new text.
The contribution of this work can be summarized as follows:
This work explains why the RLAIF functions effectively. We provide evidence that evaluation tasks are simpler than generation tasks for LLMs. This difference can be leveraged to boost the performance of LLMs of various sizes. To our knowledge, this is the first work that presents the gap between text generation and evaluation. Besides, we propose a novel LMSI method for unsupervised training of LLMs, climinating the need for external supervision and training reward models. Finally, we conduct comprehensive experiments to demonstrate that LLM,, trained with our method, can self-improve its ability in solving reasoning and text summarization problems. We also present that RLC can be applied to LLMs with a parameter range of 80M to 780M, and the trained LLM generalizes well to new and unseen datasets, demonstrating the extensive applicability of the proposed method.
2 RELATED WORK
Language model self-improvement. LMSI methods have drawn significant attention recently. These methods can be broadly classified into two categories, namely noninvasive and invasive methods.
Noninvasive methods utilize Al feedback to guide the LLM to revise its output during the generation
process. This approach involves repeated refinement and correction of the model's 's output, based on
the LLM self-evaluation (Huang et al., 2022a Cobbe et al., 2021] Weng et al. 2022 Raman et al.,
2022). For example, the re-prompting (Raman al. [0222 method identifies errors the current
plan generated by the L.M and revises the current step if an error is detected. ReAct (Yao et al.)
2022) and Self-Refine (Madaan et al.) 2023) provides specific design to correct the self-generated error using language models. On the other hand, invasive methods update the LLM parameters using AI feedback as reward signals. Recent advancements in this field include RLAIF (Lee et al. 2023). which trains reward model based on Al preference data and use a reinforcement learning algorithm to train the LLM. In contrast prior studies relied mostly on comprehensive experiment results, this work explains why such self-evaluation mechanism work well: we provide evidence that evaluation tasks are simpler than generation tasks for LLMs, and this difference can be leveraged to boost the performance of LLMs of various sizes.
Train language model with unlabeled data. Learning from unlabelled data is a promising approach for enhancing the model efficiency. Self-training is popular technique in this field, which entails assigning pseudo labels from learned classifier to unlabeled data. These labeled examples then utilized to enhance the initial model training (RoyChowdy al. 2019 Chen et al. [021] He et al. 2020). In recent years (Huang et al. (202a)) self-training has been employed to fine-tune tun- large-scale language models such as Pal.M (Chowdhery al. 2022) with 540B parameters, and it
2

---

Published as a conference paper at ICLR 2024
has shown immense potential. However, it fails to present efficiency smaller size of LLMs. In contrast to methods that utilize LLMs to generate pseudo-labels, our approach employs RL to train LLMs, which has proven more effective than supervised learning (Stiennon al. 2020).
3 PRELIMINARY
We begin with a vocabulary Σ and an LLM M which takes token sequence of the question q = (q...... 9n. as input and predicts the next token using autoregressive modeling: 0++1 M(g. foo..... 0t H), where E n" and on E E. To fine-tune LLM with RL, we can view this problem by Markov Decision Process (MDP) (Puterman 1994. Sutton described as tuple A, R, y, do):
Barto 1998), which is
State space S: the space of input token sequences q U {oo, 01,... or so is defined as the token sequence of question q.
Action space A: the space of tokens Or-
Reward function R(g, fou..... 0r }) score that reflects the quality of the generated answer to the question, which can be obtained from human feedback or a predefined criterion. The reward is typically given when the complete answer has been generated.
Transition P: st+1 U O1+1-
Initial distribution dg: the distribution of question q.
Here, the LLM M acts as policy mapping from state space to the probability space over action space. The objective of RL is train the policy to maximize the expected return:
E Σ
"R((g. do, ~ M(-1s))
In RLC, we formulate the LLM text generation problem as an MDP, analogous to the one described above, with the reward function derived from evaluation results. To simplify the notation, ~ M(q) to represent the autoregressive sampling of a complete answer o from the language model M, based on the input question q.
EVALUATION SIMPLER THAN GENERATION
In this section, we aim to verify the text evaluation ability of LLMs by investigating three key topics subsequent subsections: (1) comparison of LLMs' ability in text generation and evaluation; (2) the correlation between evaluation results and the established evaluation metries; and (3) potential for using text evaluation to improve LLMs.
4.1 COMPARISON OF EVALUATION AND GENERATION
We conduct experiments compare the text generation and evaluation abilities of LLMs using the CommonGen (Lin et al. 2020) task, which involves generating a sentence that describes an everyday scenario based given set of common concepts such /dog. frishee, catch, and throw). Specifically, we use FLAN-T5 (Chung et al. 2022) as the LLM to generate text based on common conceptual questions provided by CommonGen and evaluate the accuracy of the generated text through human evaluation. In addition, we assess the LLM's text evaluation ability by using the same LLM evaluate whether the generated text meets the given concepts. Appendix B shows more experiment details (c.g., the prompts we use). As the experiment results presented in Fig. 2 we observe that the evaluation accuracy exceeds the generation accuracy all scales of models. Especially when the parameter size of the model is small (see FLAN-T5-Large/XLL. the evaluation accuracy significantly outperforms the text generation by 15%. These results indicate that it is simpler to self-evaluate the generated text than to generate high-quality text that meets contextual requirements. However, is essential to note that the evaluation accuracy is affected by the quality of the target of evaluation, and this experiment is only preliminary demonstration of the LLM'' 's ability to self-evaluate. We conduct more experiments to verify the text evaluation ability of LLM further. as presented in the following subsections.
3

---

Published as a conference paper at ICLR 2024
60
geraration Text avaluation
80
06
Metric
BLEU
ROUGE
BERTScore
64
Task
Accurae (4
40
CNN/Daily Mail
0.26
0.16
0.23
IWSLT 2017
0.21
0.28
0.29
20
10
The correlation coefficient between evaluation and different metrics. The evaluation result correlates positively with all three metrics. The experiments are conducted with FLAN-T5-XL.
Table 1:
(2808))
(118)
Figure 2: Comparison of the text generation and evaluation.
4.2 CORRELATION BETWEEN EVALUATION AND ESTABLISHED METRICS
This section provides an analysis of the correlation between text evaluation and established metrics using two datasets: the CNN/Daily Mail dataset (Nallapati et al. 2016) for text summarization and the IWSLT 2017 dataset (Cettolo et al. 2017) for text translation. To find the relationship between evaluation and established metrics, LLM generates two answers for each question in these tasks and then self-evaluates to determine which answer is better. Additionally, utilize three metrics, namely BLEU (Papineni et al. 2002), ROUGE (Lin 2004), and BERTScore (Zhang et al. 2020), to compare the answers, calculate their correlation coefficient with evaluation, and present the findings in Tab. [ As demonstrated by the results, the evaluation exhibits a positive correlation with all the considered metrics, indicating consistent measurement of the quality of the generated text, and it is a reliable way to measure the quality of the generated text.
4.3 POTENTIAL FOR SELF-IMPROVEMENT
The previous section shows positive correlation between evaluation and established metrics. How- ever, it remains be seen whether and how the text evaluation can be utilized to improve text generation. To investigate this problem, we design a text generation strategy that utilizes the text evaluation ability of LLM: the LLM re-samples the answer based the self-evaluation result, which will be explained the following paragraphs. We conduct experiments on two NLP benchmarks.
One such benchmark is Bigbench-hard (Srivastava et al. 2022). which includes multiple reasoning tasks consisting of multiple-choice and street-generation tasks. We compare two answer generation strategies: (1) w/o SE: the answer is directly deterministic output of the LLM, and (2) w/ SE: LLM generates an answer and evaluates its correctness. If the evaluation indicates the answer is incorrect, LLM re-generates answer as the final output. We tested these two strategies multiple Bigbench-hare tasks, and the results are presented in Tab. [ The experiment results demonstrate that with evaluation, the answer accuracy outperforms or is comparable that of direct answer generation on 10 of 12 evaluation tasks. This result justifies using evaluation to help LLM improve answer accuracy.
Reasoning about Logical Deduction (7)
Tracking Shufiled Objects (5)
Object Counting
Tracking Shuffled Geometric Objects (3)
Colored Objects
Shapes
w/o SE
43.2% 44.7%
30.3%
13.7 %
22.6 %
25.7% 28.5 %
9.6%
w/ SE
34.5%
10.6%
22.5 %
10.5%
Sports Understanding
Logical Deduction (3)
Logical Deduction (5)
Penguins in
Web of Lies
Navigate
Table
w/o SE
50.5% 51.0%
55.1 %
50.7%
29.7%
30.8% 34.0 %
51.1 %
w/ SE
55.1 %
57.2%
34.8 %
49.9%
Table 2: Comparison of the answer accuracy between answer generation with/without self-evaluation. Full results on all 27 BigBench tasks are presented in Appendix CK
Furthermore, we also conduct experiments on two text summarization tasks, CNN/Daily Mail and BBC (Gupta et al. 2022). As it is not meaningful to evaluate the correctness of generated summa-
4

---

Published as a conference paper at ICLR 2024
different approach to utilize self-evaluation ability of LLM in this experiment: (I) w/ SE: LLM samples three different answers and evaluates which answer the best one as the final answer, and (2) w/o SE: LLM samples three different answers, and present the average score of the three answers. As shown in Fig. 3 the generated answers have higher scores under all three metrics when self-evaluation is used. This result suggests that evaluation can potentially improve the quality of the generated text, which serves as a stepping stone for building RLC method.
rizations,
0.50
0.70
wi SE
We SE
Who SE
SISE
0,078
0442
0:5
0,410
0.086
0.08
BLEU
0.59
0.626
0.534
0.04
0.50
0.50
0.50
0.26
0445
6BC
CMN
0.20
BBC
CNN
640
BBC
CNN
(a) BLEU
(b) ROUGE
(c) BERTScore
Figure 3: Comparison of text generation with/without self-evaluation text summarization tasks.
5 SELF-IMPROVEMENT BY REINFORCEMENT LEARNING CONTEMPLATION
In the previous section, we observed gap between text evaluation and generation for L.M,, and present opportunity enhance the generation abilities of LLMs. For example, L.LMs can effectively evaluate the sentences they generate. These evaluation results serve as learning signals to guide the LLMs towards improved generation ability.
In this section, we will elaborate on our proposed LMSI method, RLC, that improves LLM ability without external labels based on self-evaluation. We are given pre-trained language model M and an unlabeled training dataset Drrain = (q: |=) The overview of our method is illustrated in Fig.
which iterates through the following steps:
Gather question-answer pair (g. o) via q ~ Dirain o M. Evaluate the question-ansen -awooer and obtains the reward via Eq.
Self-improvement with reinforcement learning.
Answer generation to unlabeled questions. RLC randomly samples batch of questions from Dimin and asks the LLM to generate answers to these questions. For different types of questions, we use different prompts to generate answers, as described in Appendix [.3 For reasoning problems, we use the Chain-of-
(CoT) prompt Wei et al. (022)), which has been shown to effectively improve LLM performance in previous studics (Wang et al. [023 Huang et al. 2022a)).
Self-evaluation the reward. After gathering the question-answe answer pair (g. o), RLC self-evaluate the generated answer to produce the reward for RL training:
R(q. o) = d(M(EE 9,
where @ is processing function that processes the LLM output to the numerical reward and PEP the prompt used for self-evaluation. Compared to previous works that refine/revise the generated text based on the evaluation results, RLC uses an invasive learning style, i.e., updating the language model to improve its generation ability. The advantage of such invasive training is the quick response speed of the trained model: does not need to repeatedly generate and revise the answer.
RLC uses two types of prompts for evaluating the self- aeffggneeated text: (1) Correctness Evaluation Prompt (CEP): "Is the answer to the question correct? The question is: [Q]. The answer is: [A]", and (2) Quality Evaluation Prompt (QEP): "Please evaluate the answer the question and give an evaluation score from to 10. The question is: [Q]. The answer is [A]". Depending on the type of question, either CEP QEP is utilized to evaluate the generated text.
RLC applies CEP to assess the factual accuracy of the generated text, which is essential for tasks such question-answering and reasoning. The CEP prompts L.LMs to verify the answer's correctness and
5

---

Published as a conference paper at ICLR 2024
Unlabeled questions
Evaluation prompt
Generate
Q1: Given a movie review, your task to classify it as positive, negative, or neutral based on the text: The movie is interesting
Answer
Is the answer the question correct? question is: [Q], The answer (A).
Q2: Sort following list words alphabetically: The word list: oven costume counterpart.
Language model
Evaluation prompt
Please evaluate answer question
Q3: Today Christmas Eve 1937.
and give me evaluation score from
What is the date days ago? Options: (A) 12/14/2026 (B) 12/14/2007 (C) 12/14/1937.
(2) Evaluate
question
Reward
[Q]. The answer is [A].
(3) Improve
Figure 4: Overall training procedure of RLC. which iterates through three steps: (1) Answer generation the unlabeled questions. (2) Self-evaluation by asking LM using evaluation prompt, with the evaluation results as the reward. (3) Update the language model to maximize the reward using reinforcement learning algorithms. The solid lines represent the data flow, while the dashed line represents the update of LLM parameters.
identify any factual errors. Given question and the answer o generated by the LLM, the reward R(g, o) is a 0-1 value based on the evaluation result. RLC employs QEP to evaluate the overall effectiveness and coherence of the generated text in terms of its writing, structure, and style. Quality often evaluated fits some given requirements more. The QEP prompts LLMs to rate the text's s quality on a scale of to 10 based how well answers the question. This prompt type helps solve generation tasks like translation and summarization.
scale, with higher scores indicating the text presents better effectiveness and
During training, the reward distribution may change the LLM is updated. We the initial pre-trained LLM M for self-evaluation while keeping it fixed to ensure stable training.
Self-improvement through reinforcement learning. With the evaluation reward, the LLM M can be updated using any RL algorithm to maximize the reward. In our case, we employ the Proximal Policy Optimization (PPO) algorithm (Schulman et al. 2017), which has demonstrated promising results in applications (Huang et al. 2022b). To ensure better exploration, we apply entropy regularization, which prevents the sampling phase from converging too early. Additionally, we use the Kullback-Leibler (KL.) divergence to prevent M from deviating too far from the initial pre-trained LLM.
EXPERIMENTS
We conduct a series of experiments to demonstrate the effectiveness of our proposed RLC method.
Our experiments encompass the following topics: (1) comparison of the proposed method with
baseline methods on various NLP tasks (Section 6.2); : (2) the applicability the proposed method to
different model sizes (Section 6.3); and (3) generalization unseen tasks (Section 6.4 We first. introduce our experimental setup.
6.1 EXPERIMENT SETUP
Dataset for evaluation. We use the BigBench (Srivastava et al. 2022 benchmark to conduct our experiments. BigBench is a challenging reasoning task requiring the language models' complex reasoning capabilities. The tasks in BigBench are pretty diverse, including reasoning the final results of sequence of actions, understanding dates, and completing tasks that require simple arithmetic calculations. In our experiments, we use 12 challenging tasks from the BigBench-Hasd datasets] which covers judgments, multiple choices, and text generation tasks.
Baselines for comparison. We compare RLC with representative baselines that improve L.MM without supervised data. These methods include: (1) RLAIF trains reward model based on Al
Detailed descriptions about the tasks are in main

---

Published as a conference paper at ICLR 2024
Example inputs
Example outputs
Judgement
"Jamey lies. Vernell says Jamey tells the truth. Millicent says Vernell lies. Rashida says Milli- cent Does Vina tell the truth?"
"Vernell is lying because Vernell is telling ... The answer: Yes."
Text generation
"I have chicken, snail, three dogs, snake, and a goat. How many animals do have?"
So I have chicken + snail + four dogs + a snake goat = 7 ani- mals. So the answer is 7."
MultiChoice
"Here table where the first line is header and each subsequent line is a penguin: name,
"There are 4 penguins the first table and 4 gi- raffes in the second table. The answer: (D)."
age, ... How many penguins are there in the tables? Options: (A) I (B) 2 (C) 3 (D) 4 (."
Table 3: Examples of inputs and outputs for the tasks in our experiments.
preference data and employs RL algorithm to train LLM; (2) Reinforcement Learning Muning (RLFT) fine-tunes the LLM using reinforcement learning (RL) and employs oracle metrics as the reward. The metric used is answer accuracy for multiple-choice, judgement, and fill-in-the-blank tasks. For summarization tasks, BERTScore is used as the metric; (3) Self-train (Huang al.) 2022ab utilizes the SC method to generate "righ-confidence" answers for unlabelled questions as the pseudo-labels. then fine-tunes the LLM on the generated data with supervised learning; (4) Direct Generation (DG) directly generates the answer with the deterministic output of the LLM:
(5) Self-consistency (SC) (Wang et al. 2023) samples a diverse set of reasoning paths instead of solely relying on the greedy LLM output. then selects the most consistent answer by marginalizing out the sampled reasoning paths. In our experiments, we use the number of sample paths as 40; (6) Self-refine (Madaan et al., 2023) employs LLM to generate
initial output. This output is evaluated from multiple perspectives and revised by the same LLM. (7) Best-of-N (OpenAI 2022) generates N answers and uses the same model to evaluate and choose the best answer. Some may concern about the fairness of the comparison, as Best-of-N and SC generates multiple answers and selects one best answer from the answer candidates, while other method only generates one answer as the final answer. It is important to highlight that our experimental focus lies in assessing the performance gains of various methods in enhancing language models. Our evaluation primarily measures the performance disparity between DG (which generates one answer during the generation process).
Implementation details. We utilize PPO train the LLM for 6,000 gradient steps for each task, with a batch size 12. The PPO implementation is from the trix repository on GitHub (CarperAT 2020). We implement RLC using CEP in reasoning tasks and QEP for summarization task. Unless otherwise specified, we use PLAN-TS-Large which has 780M parameters, as our L.M in the experiments. All reported results are averaged over three random trials, except for RLAIF with one seed, and the experiments can be conducted using two GTX 3090 graphics cards with 24GB of memory. We provide specific ayper-parameters and more detailed implementation descriptions in Appendix [B] All RL-based methods use the same we consistently employ the same prompts for all baseline models and the RLC. For each method, we utilize the CoT prompts, specifically, "Let's think step by step." A comprehensive list of the prompts used in our experiments can be found in Table of the Appendix.
for training RL algorithms. In our experiments,
6.2 MAIN RESULTS
Results on BigBench. We evaluate RLC extensively using the BigBench dataset. Tab. presents the answer accuracy of the LLMs trained with RLC and other baseline methods. We see RLC outperforms all the baselines without the supervised label. In particular, the RLC improve answer accuracy over DG from 31.23% to 37.09%. On some tasks (c.g., Reasoning about Colored Objects), RLC even catches up with the performance of RLFT, which fine-tunes L.M with Oracle supervision. This result
7

---

Published as a conference paper at ICLR 2024
could be attributed to the fact that the RLC effectively finds the proper direction for improvement, contributing to the significant improvement of LLM. Besides, RLC outperforms all baselines on most tasks, demonstrating the effectiveness of the proposed method. SC presents good performance these tasks and achieves the highest score on 12 tasks. important to note that SC is also compatible with RI.C, which be used to improve the algorithm performance further. Besides, RLC can achieve comparable or better performance than Best-of-N and self-refine, while RLC takes less computation time during the generation process. We also compare with RLAIF (with GPT2 (Radford et al., 2019) as reward model) with RLC. The experiment results show that RLC achieve better performance. This can be attributed that reward model with smaller size may not provide effective preference results. The training curves are depicted in Fig.
located in the Appendix C.I
Reasoning about Colored Objects
Logical Deduction (7)
Tracking Shuffled Objects (5)
Object Counting
Tracking Shuftled Geometric Objects (3)
Shapes
RLFT
32.1%
45.7%
12.4%
42.6%
33.6%
18.9%
DG
32.0%
35.2%
12.4%
31.9%
31.2%
5.2%
SC
39.6%
27.6%
12.4%
24.0%
33.6%
15.6%
Self-train
19.5%
13.1%
15.5 %
11.7%
33.1%
12.4%
Self-refine
25.2%
13.2%
8.0%
18.0%
25.2%
10.0%
Best-of-N 26.8%
12.8%
12.1%
14.0%
30.0%
8.4%
RLAIF
30.4%
36.9%
11.4%
32.5%
32.8%
14.0%
RLC
35.0%
39.2%
12.2%
35.4%
33.6%
17.8% %
Sports Understanding
Logical Deduction (3)
Logical Deduction (5)
Penguins in Table
Web of Lies
Navigate
RLFT
72.2%
68.8%
58.6%
41.9%
44.2%
55.6%
DG
43.6%
53.2%
39.6%
28.4%
15.7%
46.4%
SC
48.4%
53.6%
42.8%
30.8%
35.2%
62.8%
Self-train
51.1%
51.1%
34.0%
18.4%
19.7%
48.7%
Self-refine
47.2%
50.0%
28.4%
17.2%
17.8%
46.0%
Best-of-N
50.0%
59.2%
42.0%
22.0%
17.8%
45.2%
RLAIF 52.1%
56.1%
22.0%
33.7%
19.8%
48.8%
RLC 52.9%
53.5%
44.0%
34.6%
29.8%
57.1%
Table 4: The answer accuracy RLC and baseline methods on the Bigbench-hard tasks. Each value represents the average answer accuracy of the last three training iterations. The highest performing value among methods without external labels highlighted in bold.
Results summarization task. In addition to reasoning tasks, we evaluate the performance of RLC on challenging text generation benchmark: CNN/Daily Mail. Self-train method is implemented follows: We utilize the initial LLM to generate the summarization label to the text in the dataset, after which the L.M is fine-tuned on this newly generated data. As illustrated in Fig. 5 RLC. enhances the BERTScore from 0.886 to 0.899 in the summarization task, while self-train hardly improves the model performance on this dataset. Unlike the BigBench tasks, where the generated text is relatively short, the LLM is required to produce longer text for these two tasks. The experimental results demonstrate that RLC effectively improves the text generation capabilities of the LLM.
0.55
Self-trin
0.886
0.585
0.88
0..80
0.70
Sunmuniones
Figure 5: Performance on sum- marization task.
6.3 EVALUATION ON MODEL SIZE VARIATIONS
conduct experiments to assess the performance of RLC across various language model sizes. We select three distinct models: TLAN-T5-Small, FLAN-T5-Base, and FLAN-TS-Large containing 80M, 250M, and 780M parameters, respectively. We train these models using RLC on three challenging BigBench tasks, with the final scores presented in Fig. 6 In general, RLC effectively enhances performance across different scales language models. Notably, when the parameter size small (80M) and the base score is low, the language model exhibits significant improvement.
8

---

Published as a conference paper at ICLR 2024
ZLC
Ter R14
283
27.4
Neviastas
2427
6
25
16.4
Paurerer Siav (Millios))
Pareete Sox
Pracosner Size (Millce))
(a) Tracking Shuffled Objects (3)
(b) Penguins Table
(c) Object Counting
Figure 6: Performance of RLC on different sizes of language models. RLC can improve the answer accuracy over the initial pre-trained model without supervision.
6.4 GENERALIZATION TO UNSEEN TASKS
The primary objective of an LMSI method is facilitate scalable training and enable the lan- guage model to improve continuously. In this section, we explore the potential for continuous self-improvement of the LLM using the RLC. The ability to generalize to unseen tasks is a crucial aspect of scalability. Thus, we assess whether the improved generative capability can generalize to datasets previously unseen.
Specifically, we utilize RLC to simultaneously train an LLM on five tasks, namely Geometric Shapes, Logical Deduction (3), Logical Deduction (5), Navigate and Tracking Shuffled Objects (3). After that, we test the trained LLM on five new tasks. As the evaluation results depicted in Tab. 5 The increase in performance is not necessarily correlated with the similarity between the training tasks and the evaluation tasks. For instance, Logical Deduction (3) (+1.5) and Tracking Shuffled Objects (5) (-0.1) both have tasks akin to the training tasks. It is important to note that the tasks in question were not seen during the training process. Despite this, the RLC method demonstrates an overall improvement in performance on average. These findings underscore the potential of RLC for application across a more comprehensive array of datasets, thereby augmenting the overall performance of LLMs.
Logical Deduction (7)
Object Counting
Penguins a Table
Sports Understanding
Tracking Shuffled Objects (5)
Average
Acc.
36.7 (+1.5) 32.7 (+0.7)
18 (+2.2)
52.8 (-0.4)
12.3 (-0.1)
30.5 (+0.8)
Table 5: The answer accuracy on five unseen datasets.
7 CONCLUSION
This paper introduces a novel approach to enable LLM to self-improve: reinforcement learning contemplation (RLC). We demonstrate the performance gap of generation and evaluation of LLMs and the potential for evaluation boosting generation through extensive experimental analysis. By utilizing evaluation ability, LLMs can obtain valuable insights into their performance and identify areas for improvement. Our experiments demonstrate that RLC effectively improves LLM performance across various text generation tasks. Moreover, illustrate the potential for the RLC method scale with varying model sizes and training data. However, some things could be improved in our approach. First, like previous unsupervised methods for training language models, RLC requires an unlabeled dataset to generate answers and facilitate self-immros self-improvemmet. It would be worthwhile to explore training LLMs without relying on the datasets. For instance, through self-evaluation, an LLM can improve its specialized capabilities in text generation, such as outputting positive reviews. Additionally, our current implementation utilizes two separate models for evaluation and improvement. It remains to be investigated how the evaluation ability of the LLM will also change it gets updated. Lastly, we primarily evaluate RLC on LLMs with 780M parameters due to computational limitation. Future experiments should include larger language models to better demonstrate the method's application range. We hope this work could provide new insights into LMSI. Future research could address the above questions to develop more efficient and effective training methods for language models.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
This work was supported by the National Science Foundation of China (61921006), the Major Key Project of PCL (PCL The authors would like thank Lei Yuan for discussing the method details, and anonymous reviewers & chairs for providing value comments.
SOCIETAL IMPACT
The introduction of RLC offers a promising solution to the challenges associated with fine-tuning large language models using external labels. This approach can reduce the manual cost and time required for providing supervision, thereby increasing access to NLP technologies for individuals and organizations with limited resourees. Moreover, the enhanced accuracy and effectiveness of LLMs through the use of RLC have implications for wide range of NLP tasks, including machine translation, reasoning problems, and other text generation tasks. As result, RLC potentially influences diverse industries such as healthcare, finance, and education, where an accurate and reliable language model essential. RLC present an exciting advancement the field NLP that could impact society, while must ensure that the self improvemnnt direction of the LLMs remains aligned with human values.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Constitutional Harmlessness from al feedback. arXiv preprint arXiv:22/2.0807. 2022.
CarperAl. Transformer reinforcement learning X. GitHub repository, GitHub, 2020.
Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian Stükerr, Katsuhito Sudoh, Koichiro Yoshino, and Christian Federmann. Overview of the IWSLT 2017 evaluation campaign. In IWSLT, 2017.
Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmenta- tion with cross pseudo supervision. CVPR, 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling cnatuutiion-inetuune language models. arXiv preprint arXiv:2210.11420 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, al. Training verifiers to solve math word problems. arXiv preprint urXiv:2110.14166 2021.
Anushka Gupta, Diksha Chugh, Anjum, and Rahul Katarya. Automated news summarization using transformers. In Sustainable Advanced Computing: Select Proceedings of ICSAC 202/, pp. 249-259. Springer, 2022.
Junxian He, Jiatao Gu, Jiajun Shen, and Marc' Aurelio Ranzato. Revisiting self-training for neural sequence generation. In ICLR, 2020.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprint arXiv:2210.16110, 2022a.
Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 implementation details of proximal policy optimization. In ICLR Blog Track, 2022b.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2339.00267, 2023.
Bill Yuchen Lin, Ming Shen, Wangchunshu Zhou, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. Commongen: constrained text generation challenge for generative commonsense reasoning. In AKBC, 2020.
Chin-Yew Lin. ROUGE: package for automatic evaluation of summaries. TSBO, 2004.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Grabam Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35. 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.1765/1 2023.
Ramesh Nallapati, Bowen Zhou, Cicero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. Abstractive summarization using sequence-to-sequenee RNNs and beyond. In 2016
OpenAl. Measuring goodhart's law, 2022.
URL
beasuring-goodiie

---

Published as a conference paper at ICLR 2024
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In ACL, 2002.
Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley
Series in Probability and Statistics. Wiley, 1994.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, and Stefanie Tellex Planning with large language models via corrective re- promptinng arXiv preprint 1rXiv:221..09955, 2022.
Aruni RoyChowdhury, Prithvijit Chakrabarty, Ashish Singh, Young Jin, Huaizu Jiang, Liangliang Cao. and Erik Learned-Mille Automatic adaptation of object detectors to new domains using self-training. In CVPR, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-llnno, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint 1466,, 2022.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. In NeurIPS, 2020.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks, 9(5): 1054-1044 1998.
William Van Gordon, Supakyada Sapthiang, and Edo Shonin. Contemplative psychology: History, key assumptions, and future directions. Perspectives Psychological Science, 17(1):99-107, 2022.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc V. Le, and Denny Zhou Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS, 2022.
Yixuan Weng, Minjun Zhu, Shizhu He, Kang Liu, and Jun Zhao Large language models are reasoners with self-verification. arXiv preprint arXiv:2212.0985 2022.
Yan Xu, Fuming Sun, and Xue Zhang, Literature survey of active learning multimedia annotation and retrieval. In ICIMCS, 2013.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In ICLR, 2022.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinherger, and Yoav Artzi. Bertscore: Evaluating text generation with BERT. In ICLR, 2020.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjic Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2023.
12

---

Published as a conference paper at ICLR 2024
Appendix
Table of Contents
Discussion
14
The Usage Unlabelled Questions Datasel
A.2
Sampling Strategy
14
B
Experiment Details
14
B.
Tasks for Evaluation
15
B.2
Baselines)
15
B.3
Prompts in Our Experiments
16
Hyper-parameteri
Metrics used Our Experiments
C
Additional Experimental Results
18
Training Curves on BigBench-haide hard Datasets
18
Correlation Between Answer Accuracy and Training Return
19
More Experiment Results about Sell-evaluation Ability Verification
20
4
Self Train on Different Model Sizes
21
Examples the LLM Output
21
D
A Algorithm Description
26
Appendix References
27
13

---

Published as a conference paper at ICLR 2024
A DISCUSSION
A.1 THE USAGE OF UNLABELLED QUESTIONS DATASET
In Section we introduce RLC, assuming the availability of training dataset Dirain consisting of unlabeled questions. This raises two questions: (1) what data type does the dataset contain? and (2) does RLC work without an unlabeled dataset?
Concerning the first question, the training dataset in our experiments involves questions in Dirain that adhere to an instruction + problem format. For example, question might be "Summarize the following article (instruction). [TEXT] (problem)". In this way, RLC can train LLM using any open-source datasets in the NLP community by appending instructional text to the problems within these datasets. Tab. 6 presents the instructions we used in our experiments, where '[TEX]] denotes the questions/text in the original dataset. This way of using a dataset is similar to instruction learning Lou al. (2023). which utilizes task instructions to guide the learning system in rapidly adapting to various new tasks. We posit that as the data in Durain becomes more diverse and abundant, RLC can be employed to train an LLM with generalization ability. Our experiments in Section 6.3 offer preliminary validation of this outcome.
Task
Instruction
BigBench-hard
[TEXT] Let's think step by step.
Text summarization
Please give summary of the following text. Text: [TEXT] Answer:
Table 6: Instructions for different tasks in our experiments.
As for the second question, although our experiments utilize an unlabeled dataset to train the LLM, are interested in exploring whether RLC can be applied without dataset. To achieve this, need an objective, which serves as an evaluation criterion for assessing the quality of the generated text, to train the language model. For example, we might expect an LLM consistently generate positive or polite text. In such a case, we can prompt the LLM to generate text randomly from the start token and evaluate its positivity or politeness using either the CEP the QEP. This approach allows the L.M to optimize toward the desired attribute. However, it is essential to note that some training objectives or text attributes might be challenging to evaluate, making them unsuitable for self-improvement using the reinforcement learning contemplation method.
A.2 SAMPLING STRATEGY
In our experiments, RLC employs a random sampling approach to select questions from the dataset. Although this method is effective, alternative sampling techniques could enhance the process. For instance, prioritizing questions with more uncertainty could be beneficial. The field of active learning offers many practical sampling methods (Xu et al., (013) that could potentially augment learning efficiency. We posit that these active sampling strategies align well with the concept of self-imprrvemment. However, in this study, our primary focus lies in the design of the LMSI method, for which we maintain the use of random sampling.
B EXPERIMENT DETAILS
In our experiments, we utilize the open-sourced RL repository, trix, to implement the reinforcement learning contemplation. All experiments are run three times with different random seeds. This section will present experiment details omitted in the main body due the space limitation. including the evaluation task we use, the basclines, the prompts in different experiments, the pareeeerr for reproducibility.c etc.
14

---

Published as a conference paper at ICLR 2024
B.1 TASKS FOR EVALUATION
In our experiments, we use five challenging benchmarks in the NLP domain to conduct various experiments to support our method. This section gives detailed introduction to these benchmarks, which are omitted in the main body.
CommonGen Lin al. 2020) is task that focuses on constrained text generation and includes benchmark dataset. Its primary objective is to assess machine's ability to generate common sense reasoning. The task requires constructing a coherent sentence that describes everyday scenarios using predefined set of concepts. CommonGen presents significant challenges, it demands two primary skills: (1) utilizing background knowledge for relational reasoning and (2) effectively managing combination generalization of concealed concept combinations. We employ CommonGen to evaluate the text generation and self-evaluation ability of LLM,, wherein the LLM is tasked with generating a sentence based on four concepts simultaneously.
BigBench-hard Srivastava et al. (2022) consists of 27 challenging tasks designed to evaluate the reasoning abilities of language models. These tasks present increased difficulty due to their complexity, In our experiments, we employ 12 challenging tasks that focus on evaluating different aspects of reasoning ability:
Reasoning about Colored Objects (multiple choices): Answer simple questions about the colors of objects on a surface.
Logical Deduction (multiple choices): Deduce the order of sequence of objects.
Tracking Shuffled Objects (multiple choices): Determine the final positions of a set of objects given their initial positions and description of a sequence of swaps.
Object Counting (text generation): Questions that involve enumerating objects of different types and asking the model to count them.
Geometric Shapes (text generation): Name geometric shapes from their SVG paths.
Web of Lies (judgement): Evaluate random boolean function expressed as word problem.
Sports Understanding (judgement): Determine whether an artificially constructed sentence relating to sports is plausible or implausible.
Penguins in Table (text generation): Answer questions about table of penguins and their attributes.
Navigate (judgement): Given series of navigation instructions, determine whether one would end up back at the starting point.
Note that specific tasks (e.g., Logical Deduction) encompass the same topic but are presented at varying difficulty levels, and we introduce these tasks with varying levels together.
The CNN/Daily Mail Nallapptii et al. (016) dataset serves as widely recognized benchmark for text summarization. Comprising over 300,000 news articles from CNN and the Daily Mail and corresponding human-written summaries, this dataset has been instrumental training and evaluating various text summarization models, including extractive and abstractive methods. Each article features several highlights summarizing its main points, making the dataset ideal for training and testing automatic text summarization models that aim generate concise versions of the original text while retaining crucial information.
The BBC Gupta al. (2022) dataset, created by the BBC News website, is another widely used resource for text summarization. The dataset has been pre-processed and annotated with human- generated summaries, encompassing approximately 2.225 news articles on diverse topics such as politics, entertainment, technology, and sports. These concise summaries, typically 3 to 4 long sentences, make the dataset invaluable for training and evaluating text summarization models.
B.2 BASELINES
In our experiments, we compare RLC with representative baseline methods. This section presents these baselines and their implementation details.
15

---

Published as a conference paper at ICLR 2024
(I) SC: Self- Self-Consistency. SC enhances the LLM's answers without updating its parameters. Its motivation is similar to ensemble methods, which sample multiple answers simultaneously and vote among them to select the most consistent one. SC has demonstrated significant capabilities in reasoning problems. In our experiments, the sampling path (i.e., the number of answers that LLM samples) is set to 40.
(2) Self-train: Language Model Self-train. LMSI combines the SC method with supervised learning fine-tuning. It first uses SC generate answers for all questions the dataset and then fine-tunes the LLM using supervised learning. In contrast to LMSI, which employs supervision signals to enhance language models, RLC utilizes reward signals. Reward signals provide more abstract and general performance measure, enabling reinforcement learning algorithms to learn from fewer examples and generalize hetter to new situations. Furthermore, reward signals allow encoding of more complex and diverse settings, potentially leading to more creative and adaptive answers.
(3) RLFT: Reinforcement Learning Fine-Tuning RLFT trains the LLMs using Oracle metrics as rewards, such answer correctness or BERTScore. We include this baseline to verify the effectiveness and performance upper bound RLC. In our experiments, we implement RLFT and RLC using the same code framework. The primary difference is that RLC employs self-evaluation results as rewards.
(4) Self-Refine' Self-Refine is method designed enhance the initial outputs generated by Large Language Models (LLMs) through an iterative feedback and refinement process. The core concept involves generating initial output using LLM, after which the same LLM evaluates and refines its output iteratively. Similarly, Self-Refine operates without the need for supervised training data. However, due Flan-T5-Large being too small generate effective feedback, we have pruned the feedback section and directly inputted the results generated by refinement into LLM to obtain the following result.
(5) Best-of-N: Best-of-N sample n times and take the one that scores the highest according to the proxy objective. In our experiments, the number of answers LLM generates (i.c., N) set to 40. And then, we use the same LLM to evaluate the 40 generated answers as the proxy objective. Then, we choose the answer with the highest score as the final answer.
(6) RLAIF:
The RLAIF framework is similar to RLHF, but the difference is that an off- sffllf LLM, instead of humans, labels the preferences. our experiments, we use GPrg-Large as the off-the-shelf LLM to train the reward model, which is the same as [ee et al. 2023
B.3 PROMPTS IN OUR EXPERIMENTS
In various experiments, different prompts serve specific experimental objectives. A comprehensive overview of these prompts can be found in Tab. [ We employ unique instructional text to generate answers for questions across diverse datasets, as illustrated in Tab.
B.4 HYPER-PARAMETERS
Tab. g presents the
used in our experiments.
B.5 METRICS USED IN OUR EXPERIMENTS
In our experiments, we present various experimental results under different metries. For BigBench, we utilize accuracy to judge the correctness of the generated answer compared to the reference answer. translation and summarization tasks, we consider the following metrics: BLEU is a reference-based metric that evaluates the similarity between a machine-generater output and one or more reference outputs. BERTScore a reference-les metric that assesses the similarity between the embeddings of machine machine-generated output and reference answer. ROUGE is another reference- based metric that measures the overlap between the generated output and one or more reference outputs. BLEU emphasizes precision, while ROUGE focuses recall.
16

---

Published as a conference paper at ICLR 2024
Experiment
Prompt
Comparison of the text genera- tion and self-evaluation (Section
Consider task that needs to generate coherent sentence describing an everyday scenario using all following concepts. You will be given a few concepts and sentence, please tell me whether the task is done. If you think the task is done, reply yes. If you think the task is not done, reply no. Concepts: [CONCEPT], Sentence: [SENTENCE].
Correlation with self-evaluation and established metrics (Transla- tion) (Section 4.2
Suppose you reviewer of the text translation. You will be given two translations of a text, please tell me which one better according to the conciseness and integrality of the trans- lation. If you think Translation (1) is better, reply (1). If you think Translation (2) is better, reply (2). Tex::[TASK] Trans- lation (I): [Translation 1] Translation (2): [Translation 2]. Which one better?
Correlation with self-evaluation and established metrics (Summa- rization) (Section 4.2)
Suppose you are a reviewer of the text summary. You will be given two summaries of a text. Please tell me which one is better according to the conciseness, integrality of the summary. If you think Summary (1) is better, reply (1). If you think Summary (2) is better, reply (2). Text: [TASK] Summary (I):
[Summary_]] Summary (2): [Summary_22 2]. Which one is better.
CEP for BigBench (Section 6.2
Is the answer the question correct? The question is: [Q]. The answer is: [A]
QEP for Summarization (Sec- Please help me evaluate the summary results of the following text. Only give a score from 1 to 10, without explanation. Text: [Q] Summary: [A]
tion 6.2
Table 7: Prompts used in different experiments.
Table 8: Hyper-parameters in our experiments.
Hyper-parameters
Value
PPO epoch
PPO clip ratios
0.2
PPO
0.95
batch size
12
value function coefficient
1.0
learning rate
lc-4
discount factor 7
0.99
temperature for LLM exploration
top_k of LLM
50
top_p of LLM Sampling path of SC/LMSI
0.95
LMSI learning rate
5e-4
LMSI batch size
2
SC sampling path
40
17

---

Published as a conference paper at ICLR 2024
C ADDITIONAL EXPERIMENTAL RESULTS
C.1
TRAINING CURVES ON BIGBENCH-HARD DATASETS
Fig.
presents all the training curves 12 BigBench-hard tasks omitted in the main body.
RLC.
0.50
0.35
RLC
REFT
KLFT
Atewer accuncy
accumcy
0.55
Ateuer accuney
0.334
0.15
0.33
0.10
0.32
RLC
0.05
RLFT
2000
4000
6000
2000
4000
6000
2000
4000
6000
Gradient Skep
Grdiiet Step
Gradient Step
(a) Geometric Shapes
(h) Logical Deduction (3)
(c) Tracking Shuffled Ohjects
0.50
RLC
0.50
RLC
0.70
RLC
0.45
Atewer sccuney
0.45
0.40
Ateaer sccurecy
0.65
0.35
0.40
0.30
0.50
0.35
0.25
0.45
0.20
2000
4000
8000
2000
4000
6000
0.40
2000
4000
6000
Gradicnt Skp
Grddiet Stop
Gradient Stop
(d) Logical Deduction (7)
(c) Logical Deduction (5)
(f) Navigate
0.45
0.5
RTC RLPT
0.80
Answerr accuracy 00.00000000
00.10
Answer acuuracy
0.4
Answerr accuracy
0.75
REFT
0.35
0.30
0.28
RLC
0.2
0.20
REFT
2000
Gradient
Gradicnt
Gradicat
(g) Object Counting
(h) Penguins in Table
(i) Sports Understanding
0.5
RLC
Anower acauncy
RI.FT
Answer scourey
0.4
KLF
menmeyy
0.130
RLFT
0,3
0.120
2000
4000
6000
0.1
20000
2000
6000
2000
4000
6000
Grndient Soep
Gradiens Step
Gradient Step
Web of Lies
(k) Reasoning about Colored Ohjects ()) Tracking Shuffled Objects (5)
Figure 7: Training curves reinforcement learning contemplation on BigBench hard tasks. The shaded area represents the standard deviation over three seeds.
18

---

Published as a conference paper at ICLR 2024
C.2
CORRELATION BETWEEN ANSWER ACCURACY AND TRAINING RETURN
Fig. 8
presents all the changes of answer accuracy and episodic return during RLC training process.
0.46
Anneli Aocersy
Ipiaeds
Retuii
Eplands Beruu
014
Anewer racuraey
0.15
0.42
0.320
0.325
0.10
0.2 0.40
wW
0.38
0.36
0.4
2000
4000
6030
2000
4000
6000
2000
4003
5000
Gradient Step
Grudient Step
Grodient Stop
Geometric Shapes
(b) Logical Deduction (3)
(c) Tracking Shuffled Objects (3)
01.2
Aunnea Anaens
Asanst Accuune
0.
Aaren AECATRY
0.40
Fiiondio Retare
0.35
Egiond Betuo
Egi Egiiodic Ruduri
0.59
0.36
0.50
0.0
0.25
-02
63
0.50
0.4
0.4
0.20
0.30
2505
4000
6005
2000
4000
6000
2000
4000
5000
Gradient Step
Gradiont Step
Gradient Step
(d) Logical Deduction
(c) Logical Deduction (5)
(f) Navigate
0154
Egisclii
Aspeec ACMEAA
1.6
Equeadic
0.
Opicalic Retum
0.39
Egeaddre Retura
0.25
0.2
0.50
0.20
0.53
0.32
0.0
022
0.39
2505
4014
6005
2000
4000
6000
2000
40000
600
0.0
Cradieet Step
Gradiont Step
Gradicnt Step
g Object Counting
(h) Penguins in a Table
(i) Sports Understanding
01.50
Afsover Accunny
18
0.400
Anurer Acconcy
0.130
Auaiarr. Eqiiiddic Reiare
Fysoodil Rurret
Eppodiie Actumn
0.52
0.128
Epi Piliii Rehun
0.50
0.350
0.2
0.126
0.48
02
0.325
0.124
0.46
0.100
0.9
0.122
0.44
0.120
0.42
2000
4000
6000
04
2000
4000
6000
2000
4000
5000
Gradient Step
Gradient Step
Ordient Step
Web of Lies
(k) Reasoning about Colored Ob- (l) Tracking Shuffled Objects (5)
jects
Figure 8: Training curves of the answer accuracy and episodic return on BigBench- hard tasks.
19

---

Published as a conference paper at ICLR 2024
C.3 MORE EXPERIMENT RESULTS ABOUT SELF-EVALUATION ABILITY VERIFICATION
In Section 4.3 we examine the self-evaluation ability of LLMs and their potential for self- improvement. Due space limitations in the main body, we present only subset of the tasks (12 tasks). This section provides the complete results for all 27 tasks on the BigBench-hard bench- mark, as shown in Tab.
Reasoning about Colored Objects
Logical Deduction (7)
Tracking Shufiled Objects (5)
Object Counting
SE
43.2% 44.7%
30.3% 34.4% %
13.7% 10.7%
22.6% 22.5% %
Sports Understanding
Logical Deduction (3)
Logical Deduction (5)
Web of Lies
w/o SE
50.5% 51.0%
55.1%
50.7% 57.2%
29.7% 34.8%
w/ SE
55.1%
Formal Fallacies
Date Understanding
Causal Judgement
Hyperbaton
w/o SE
55.3%
62.8%
29.9%
56.0%
SE
63.1%
60.9%
32.5%
57.4%
Boolcan Exprcssions
Ruin Names
Tracking Shuffled Objects (7)
Temporal Sequences
w/o SE
50.8%
27.1%
9.9% 9.2%
22.8 % 22.8 %
w/ SE
57.5%
26.8%
Tracking Shuffled Objects (3)
Geometric Shapes
Snarks
Navigate
w/o SE
25.7%
9.6%
65.5% 59.4%
51.1% % 49.9%
SE
28.5%
10.5%
Penguins Table
Disambiguation QA
Multistep Arithmetic Two
Word Sorting
w/o SE
30.8%
55.3% 58.5% %
3.5 % 3.5 %
1.1% 1.1 %
w/ SE
34.0%
Dyck Languages
Salient Translation Error Detection
Movie Recommendation
Average
w/o SE
27.6%
26.5% % 34.7% %
35.5% 41.2% %
34.9%
SE
28.4%
36.7 %
Table 9: Accuracy of answer generation with and without self-evaluation on 27 BigBench tasks.
20

---

Published as a conference paper at ICLR 2024
C.4
SELF TRAIN ON DIFFERENT MODEL SIZES
No-Roae ScPliee
20
60.
23
23
(a) Tracking Shuffled Objects (3)
(b) Penguins in a Table
(c) Object Counting
Figure 9: Performance of RLC and Sclf-train on different sizes of language models.
C.5
EXAMPLES OF THE LLM OUTPUT
Tab. 10 13
presents some examples of LLM output before/after training with RLC.
21

---

Published as a conference paper at ICLR 2024
Question
Before training
After training
Please give a summary of the following text. Text: Kyle Naughton has been ruled out for the rest of the season after David Meyler's red card challenge on Saturday. The Hull midfielder challenge that left Naughton
[BELU score: 0.887] David Meyler lunged in high Kyle Naughton and was sent off. The Hull midfielder was shown straight red for the hefty challenge. The Hull midfielder was shown straight red for hefty challenge. The Hull midfielder was shown straight red for the hefty challenge. The Hull midfielder was shown
[BLEU score: 0.904] David Meyler was sent off for the tackle on Kyle Naughton. The Hull midfielder was shown a straight red for the tackle.
shown straight red for the hefty hospital at the weekend, though neither Garry Monk nor Steve Bruce condemned Meyler at the time. A scan on Monday revealed Naughton suffered ligament damage and the right-back will face six- week lay-off. David Meyler (left) lunges in high Kyle Naughton and was sent off the poor challenge on Saturday. The full back writhes in pain following the incident and faces a ex-k lay off with ankle ligament damage. A club statement said: An initial scan Saturday ruled fracture, follow up scan last night confirmed ligament damage that will keep the Jamuary signing from Spuns out for around six weeks. 'With just seven league games left, it's blow for Naughton and the Swans with the right- back continuing impress with every appearance." Angel Rangel will return the side to face Everton on Saturday. Rangel said: 'It's not good because Kyle has an injury and I don't t wish bad on anyone, but from my point of view I always try and be ready if the manager needs me I'll try my best.' Naughton will miss the rest of the season and had helped off the pitch by the Swansea team physios. Meyler looks shocked as referee Andre Marriner shows him straight red, decision that was justified. Answer:
straight red for the hefty challenge. The Swansca.
Please give summary of the following text. Text: The Chief Secretary to the Treasury has been caught knocking back drinks with high-class call girl. But sadly for scandalmongers, the chance encounter between Danny Alexander and Brooke Magnanti better known by her former alias Belle de Jour was innocent. The two were snapped Bar One in Inverness Friday part the Nip festival, which celebrates the gins and whiskies of the Highlands. Chief Secretary to the Treasury Danny Alexander pictured drinking with call girl Brooke Mag- nanti. Mr Alexander was snapped with Ms Magnati also known Belle de Jour the Nip festival. It comes as David Cameron faced a growing revolt night over his 'flat-footed' General Election campaign. Several MPs pointed the finger of blame at the Prime Minister's Aus- tralian Election guru Lynton Crosby. And former Conser- vative Cabinet Minister criticised George Osborne's ' Diick- ensian' message on 'Austerity Britain', The first signs of Tory wobble emerged after last week's 's polls suggested that instead of leaving Labour trailing, as Tory chiefs had ex- pected, Labour slightly ahead. And far from collapsing under the weight of massed Tory attacks, Labour leader Ed Miliband's 's ratings have improved. In an outspoken intervention, retiring Conservative MP Brian Binley said: "With the Election campaign well under way. Mr Cameron still appears be standing apart aloof, almost like spectator.' added: footed and lame, we already look as if we're running to catch up.' Answer:
[BLEU score: 0.853] Danny Alexander pictured drinking with call girl Brooke Mag- nanti.
[BLEU score: 0.854] Danny Alexander was pictured knock- ing back drinks with high-class call girl Brooke Magnanti.
Table 10: Model output examples on summarization task (CNN/Daily Mail). The table presents the output generated by the model before and after RLC training across various text input.
22

---

Published as a conference paper at ICLR 2024
Question
Before training
After training
(Object Counting) I have two fish, four cats, two chickens, mouse, and a goat. How many animals do have? Let's think step by step.
First we add all the animals: two fish, 4 cats, 2 chickens, mouse, and goat= 11. Now we add up all the animals together: 2 x 4 = 8. Now divide this by 4 and multiply by to get the num- ber of animals: 8 x 48. So, the answer is
I have 2 fish, 4 cats. 2 chickens, mouse, and a goat, so have 2 fish + 4 cats + chickens + mouse + 1 goat 10 ani- mals. So, the an- swer is 10.
(Object Counting) have a chicken, a snail, three dogs, a snake, and goat. How many animals do have? Let's think step by step.
Since pound equals 10 ani- mals, the four animals in ques- tion have 10 animals / /4= ani- mal weighings a dog / the dog weighs So dog / the snake weighs and a snake weighs 05. So a snake / 5 three dogs 4 and smake / 05 that's dogs 4. So chicken, snail. 3 dogs / 4 and
have chicken, snail, four dogs, a snake, and a goat. So T have chicken + snail + four dogs + snake + goat
= 7 animals. So the answer is 7.
(Logical Deduction Objects (3)) The following para- graphs each describe set of three objects arranged a fixed order. The statements are logically consis- tent within each paragraph. On a branch, there are three birds: a blue jay, quail, and a falcon. The falcon is to the right of the blue jay. The blue jay is the right of the quail. Options: (A) The blue jay is the second from the left (B) The quail is the second from the left (C) The falcon is the second from the left. Let's think step by step.
The falcon is second from the left. The answer: (C).
The blue jay is the second from the left. The answer: (A).
(Logical Deduction Objects (3)) The following para- graphs each describe set of three objects arranged fixed order. The statements are logically con- sistent within each paragraph. On a shelf, there are three books: a blue book, an orange book, and a red book. The blue book is the rightmost. The orange book is the lefumost. Options: (A) The blue book is the second from left (B) The orange book the second from the left (C) The red book is the second from the left. Let's think step by step.
The orange book is second from the left. The answer: (B).
The red book is the second from the left. The answer: (C).
(Geometric Shapes) This SVG path element path d= "M 21. 10.19.9.99 I. 22. 14,97.44 I. (0.48,73.34 I.
n" "row":len(path
d= "m Trapezoid. So the fi-
2119,9794 I 2214,9744 1
nal answer is (K).
9.43,73.84 I. 21. 10,9100.799 79."9/// draws Options: (A)
1048,7334
943,7384
|
circle (B) heptagon (C) hexagon (D) kite (E) line
2110.9794")S "row":len(pat"
(F) octagon (G) pentagon (H)) rectangle (I) sector (J)
d= "m 2110,97941 11109999412214944441
triangle (K) trapezoid. Let's think step by step.
1048,7334 1 943.73.
(Geometric Shapes) This SVG path element path d= "M 52.00,22.00 L 3.00,95.00 L 53,00,30.00 L
path element (path d="m 52.00.72.00 1 3.00.95.00 1
Path element path d="m 52.00.72.00
52.00,72.00"/> draws Options: (A) circle (B) hep-
53.00.30.00 I 52.00,72.00"/>)
3.00,95.00 I
tagon (C) hexagon (D) kite (E) line (F) octagon (G)
draw octagon the path ele-
53.00.30.00
pentagon (H) rectangle (1) sector (J) triangle. Let's
ment (path d="m 52.00,72.00
52,72.00%/> draws
think step by step.
1 3.00,95.00 1 53.00,30.00 00 52.00.72.00'/> draws octagon. So, the answer is (F).
triangle. So the
final answer is (J).
Table 11: Model output examples on on BigBench-hard. The table presents the output generated by the model before and after RLC training across various tasks.
23

---

Published as a conference paper at ICLR 2024
Question
Before training
After training
(Tracking Shuffled Objects (3)) Alice, Bob, and Claire are friends and avid readers who occasion- ally trade books. At the start of the semester, they each buy one new book: Alice gets Ulysses, Bob gets Frankenstein, and Claire gets Catch-22. As the semester proceeds, they trading around the books. First, Bob and Claire swap books. Then, Claire and Alice swap books. Finally, Alice and Bob swap books. At the end of the semester, Bob has Options: (A) Ulysses (B) Frankenstein (C) Catch 22. Let's think step by step.
Frankenstein 2/ 3 of bob's books then bob gets second book they trade around books bob eventually gets Ulysses. So, the answer is (A).
Frankenstein the end of semester. So the final answer is (B).
(Tracking Shuffled Objects (3)) Alice, Bob, and Claire are dancers at square dance. At the start of a song, they each have partner: Alice is dancing with Izzi, Bob is dancing with Melissa, and Claire is dancing with Lola. Throughout the song, the dancers often trade partners. First, Claire and Alice switch partners. Then, Bob and Claire switch partners, Fi- nally, Alice and Bob switch partners. At the end of the dance, Alice is dancing with Options: (A) Izzi (B) Melissa (C) Lola. Let's think step by step.
At the start of the song, ev- eryone has a partner, hence Alice starts dancing with Izzi: bob dances with Melissa; Claire dances with Lola throughout the song, they usually trade partners. So, at the end of the dance, Alice is dancing with Izzi; at the start, bob is dancing with Melissa; the end, Claire is dancing with Lola. So, the answer (C).
Izzi, So the final swer (A).
(Web of Lies) Audrie the truth. Ryan says Audrie lies. Rashida says Ryan lies. Fletcher says Rashida tells the truth. Shaunda says Fletcher tells the truth. Does Shaunda tell the truth? Let's think step by step.
If Fletcher tells the truth, the third party does not have to be- lieve his words in this scenario, Shaunda does not have to he- lieve his statement. Therefore, answer is No.
Shaunda says Fletcher tells the truth. Shaunda telling the truth because fletcher tells truth. So the answer is Yes.
(Web of Lies) Jamey lics. Vernell says Jamey tells the truth. Millicent says Vernell lics. Rashida says Millicent tells the truth. Vina says Rashida tells the truth. Does Vina tell the truth? Let's think step step.
Verell tells the truth to Vernell, but Millicent tells the truth to Millicent. Vina tells the truth to Rashida. So the answer is No.
Vernell is lying because Vernell is telling the truth. Rashida is lying because Rashida telling the truth. Vina is telling the truth because Rashida is telling the truth. The answer: Yes.
(Sports Understanding) Is the following sentence plausible? "Jakub Vrana skated backwards" Let's think step by step.
The sentence is faase because there is no documentation of vrana skated backwards. So, the answer No.
Jakub Vrana skated backwards in his ca- reer. So, the answer Yes.
(Sports Understanding) Is the following sentence plausible? "David Pastrnak skated backwards" Let's think step by step.
David Pastrnak former pro- fessional skateboarder. So, the answer is No.
David Pastrnak is professional skater and has skated back- wards. So, the an- swer is Yes.
Table 12: Model output examples BigBench-hard, The table presents the output generated by the model before and after RLC training across various tasks.
24

---

Published as a conference paper at ICLR 2024
Question
Before training
After training
(Penguins in Table) Here table where the first line is a header and cach subsequent line is a penguin: name, age, height (cm), weight (kg) Louis, 7. 50, 11 Bernard, 5, 80, 13 Vincent, 9, 60, 11 Gwen, 8, 70. 15. For example: the age of Louis is the weight of Gwen 15 kg. the height of Bermard is 80 em. And here similar table, but listing giraffes: name, age, height (cm), weight (kg) Jody, 5, 430, 620 Gladys, 10, 420, 590 Marian, 2, 310, 410 Donna, 9, 440, 650. How many penguins are there in the tables? Options: (A) (B) (C) (D) 4 (E) 5. Let's think step by step.
Number of girafres twice as many as the number of penguins in the first table. So, there are tables. The an- swer: (C).
There are 4 pen- guins in the first ta- ble and 4 giraffes in the second table. The answer: (D).
(Penguins in a Table) Here a table where the first line is a header and each subsequent line is a penguin: name, age, height (em), weight (kg) Louis, 7, 50, 11 Bernard, 5, 80, 13 Vincent, 9, 60, 11 Gwen, 8, 70, 15 For example: the age of Louis height of Bernard is em. Which penguin is one year younger than Vincent? Options: (A) Louis (B) Bernard (C) Vincent (D) Gwen (E) James. Let's think step by step.
Louis 7*11- 9*60 year younger than Vin- cent. The answer: (A).
Louis
penguin
Louis, 7. 50, 11
Beraard, 5, 80, 13
Vincent, 9, 60, 11
the weight of Gwen 15 kg. the
Gwen, 8, 70, 15. So
the answer (D).
(Logical Deduction Objects (5)) The following para- graphs each describe set of five objects arranged in fixed order. The statements are logically consistent within each paragraph. On a branch, there are five birds: falcon, raven. a cardinal, a hawk, and crow. The cardinal is the second from the right. The hawk is to the right of the falcon. The raven the second from the left. The crow is the rightmost. Options: (A) The falcon is the rightmost (B) The raven is the rightmost (C) The cardinal is the rightmost (D) The hawk is the rightmost (E) The crow is the rightmost. Let's think step by step.
The crow belongs to the rightmost group. The answer: (C).
The crow is the rightmost. The an- swer: (E).
(Logical Deduction Objects (5)) The following para- graphs each describe set of five objects arranged in fixed order. The statements are logically consistent within each paragraph. On a shelf, there are five books: blue book, purple book, yellow book, red book, and gray book. The yellow book is the right of the gray book. The purple book is to the left of the gray book. The red book is to the right of the blue book. The purple book is the third from the left. Options: (A) The blue book is the third from the left (B) The purple book the third from the left (C) The yellow book is the third from the left (D) The red book is the third from the left
Purple book is to the left of the red book yellow book is.
The purple book is the third from the left. The answer: (B).
(E) The gray book is the third from the left. Let's think step by step.
(Tracking Shuffled Objects (5)) Alice, Bob, Claire, Dave, and Eve are playing game. At the start of the game, they are each holding ball: Alice has a brown ball, Bob has a black ball, Claire has white ball, Dave has yellow ball, and Eve has a purple ball. As the game progresses, pairs of players trade halls. First, Claire and Dave swap balls. Then, Alice and Eve swap balls. Then, Dave and Bob swap balls. Then, Alice and Eve swap balls. Finally, Claire and Eve swap balls. At the end of the game, Alice has the Options: (A) brown ball (B) black ball (C) white ball (D) yellow ball (E) purple ball. Let's think step by step.
White ball to Bob = I ball, black ball to Claire ball, white ball to Dave = 1 hall, yellow ball to Dave = ball, purple ball to Eve ball. So the final answer is (D).
Brown ball. So the final answer is (A).
Table 13: Model Output Examples BigBench-hard. The table presents the output generated by the model before and after RLC training across various tasks.
25

---

Published as a conference paper at ICLR 2024
D ALGORITHM DESCRIPTION
We present a summary of the RLC training process in Algorithm
Algorithm 1 Self-Improvement by Reinforcement Learning Contemplation (RLC) Input: pre-trained LLM M, a pre-trained LLM for self-evaluation M M,, an unlabelled dataset
Dirain {q.)/11- Output: the trained LLM M.
1: Initialize the TL generator
2: while training not complete do
3: Sample questions {4:} from dataset Darain
4: Use M to sample answers {or} to the questions, using stochastic output of LLM.
5: Calculate reward R(q, o) according to self-evaluation results (Eq. 2
6: Update M with RL algorithm.
7: end while
8: return the trained LLM M.
26

---

Published as a conference paper at ICLR 2024
APPENDIX REFERENCES
Anushka Gupta, Diksha Chugh, Anjum, and Rahul Katarya. Automated news summarization using transformers. In Sustainable Advanced Computing; Select Proceedings of ICSAC 2021, pp. 249-259. Springer, 2022.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00667 2023.
Bill Yuchen Lin, Ming Shen, Wangchunshu Zhou, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. Commongen: constrained text generation challenge for generative commonsense
reasoning. In AKBC, 2020.
Renze Lou, Kai Zhang, and Wenpeng Yin. Is prompt all you need? no. a comprehensive and broader view of instruction learning. arXiv preprint arXiv:2303. 10475, 2023.
Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dos Santos, Çaglar Gülçehre, and Bing Xiang. rnns and beyond. In CoNLL, 2016.
Abstractive text summarization using sequence-to-sequeee
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprini arXiv:2206.044 04615, 2022.
27


Reviews:
[{"review_id": "IbKeojMTLi", "reviewer": "Reviewer_scBc", "summary": "This paper presents a method for improving a language model using its own output. Specifically, the authors first show that there is an accuracy gap in the output of a language model between generation and evaluation, using Bigbench and summarization tasks. They then present a method for fine-tuning the language model using the evaluation output as the reward in reinforcement learning. The authors use Flan-T5 (780M) to demonstrate the effectiveness of their proposed approach.", "questions": "- My understanding is that Flan-T5 is an encoder-decoder model, which may have helped to boost the accuracy of evaluation since the encoder transformer allows each token to attend to any other token in the input. I am wondering if the big accuracy gap between  generation and accuracy still exists in decoder-only models.\n- Figure 2 suggests that the accuracy gap between generation and evaluation becomes smaller as the model size increases. Is the proposed approach still effective for larger models?\n- Is it true that GPT-2 Large was used in Lee et al. (2023)?  It seems to me that they used PaLM 2.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The fact that evaluation is sometimes easier than generation is well known, but the authors show concrete experimental results that support this in non-trivial settings.\n- Overall, the paper is well written and easy to follow.", "weaknesses": "- The novelty of the work is limited. The overall approach is very similar to RLAIF.\n- The effectiveness of the proposed approach is demonstrated using the 780M model of Flan-T5, but it is not clear how effective it is when other or larger models are used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FOuUhpdha3", "reviewer": "Reviewer_ksaG", "summary": "This paper proposes an approach for language model self-improvement: given an LM, improve it using itself to provide feedback. This paper proposes a method called Reinforcement Learning Contemplation (RLC) to do this, based on the observation that LMs are better at providing feedback versus generating full examples. \n\nThe three stage process is to 1. sample a bunch of QA pairs from the model (the questions come from a training dataset), 2. evaluate the QA pair from the model and transform it into a reward, and then 3. update the base LM accordingly. \n\nThe paper evaluates this over CommonGen (lexically constrained text generation), 12 reasoning tasks from BigBench-Hard, along with CNN-daily mail summarization. The experiments show improvements over baselines like sampling once or N times, as well as self-training approaches without RL. Training on seed data from multiple tasks also shows some generalization to unseen tasks.", "questions": "For the experiments, how do the non-PPO trained models generate answers / what happens if they generate an answer or assign high probability mass outside of the set of candidate options (e.g. for MultiChoice answering something that's not A/B/C/D/E?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall, this paper seems reasonably solid to this reviewer. The approach proposed is simple and general, yet it also seems novel and underexplored at least to this reviewer. The results suggest that it helps on reasoning tasks which could suggest something interesting is happening during the RL process. It's good to see results on a variety of different model sizes too, which suggest also that the gain doesn't go away with scale (at least for e.g. the BigBench task \"Penguins in a Table\")\n\nI have some concerns about evaluation below though I think/hope they can be answered in rebuttal --", "weaknesses": "The main concerns to this reviewer are:\n* the datasets considered are a bit toy. It would be great to see other experiments from other domains (maybe something like math word problems?) or things considered by some of the other common papers in this space.\n* The approach is limited to having a training dataset, though this is addressed appropriately in Appendix A.1 (at least to this reviewer, maybe extending this could be left for future work). However, I am a bit concerned that there might be stronger baselines that make use of that unlabeled training dataset.\n* It's not clear to this reviewer what the model is learning here -- is it learning to truly reason or just to produce better answers (e.g. that mimic the format of the seed dataset)? It would be great to run more controlled experiments studying this. The main comparison that seems to address this is the \"Self-train\" baseline that generates high-confidence answers and then trains on them. It would be great to have a version of Figure 6 (performance of RLC across sizes of LMs) comparing with \"Self-train\" instead of with no finetuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xODpoMgjx9", "reviewer": "Reviewer_BB78", "summary": "This work proposes a method for training language models without the need for external supervision or additional reward models. First, the language model is used for generation. Second, the language model is used to evaluate its own generated outputs. Third, the evaluation result is used to improve the generation. This approach results in some gains on BigBench tasks.", "questions": "What is the statistical significance of the gains in Tables 2 and 4? Several of the columns show what seems to be minor gains (e.g. for Table 2, all except Penguins in a Table). Are these gains within standard deviation?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This work demonstrates empirical gains using self-improvement via evaluating self-generated outputs.", "weaknesses": "1. The approaches demonstrated in this work is not new, and the authors do not discuss prior work. For there is a family of work in evaluating self-generation (https://arxiv.org/abs/2210.03629), how is this work different?\n2. This work uses CoT, do the baselines use CoT as well? How much of the gain is coming from CoT? This work is missing ablations that quantify the gains from the primary contribution.\n3. The majority of this manuscript describe background information. The main contribution of this work starts on page 5. I suggest that the authors drastically truncate the background content and use the remaining room to analyze the primary contribution. First, how does it compare to prior work? What is generated during self-evaluation? Where do they agree/disagree with reference evaluation?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "duPh4oa4ew", "reviewer": "Reviewer_Rzu4", "summary": "The authors introduce a self-improvement method for language models, i.e. a method that enables a language model to improve without any external supervision, as opposed to RLHF. It builds on the intuition that it is easier for language models to evaluate text than to generate it. In section 4, the authors confirm this intuition quantitatively on experiments from standard benchmarks and show that text evaluation can enable non-invasive improvement of language models by post-processing of text generations. \nThey introduce a self-evaluating task-specific reward model (RLC) for “invasive” improvement of the LMs. Similarly to RLAIF (which asks the LM to rank generated text), it tailors a prompt for each task, possibly giving a denser reward signal to PPO: a strong case for RLC is it does not require learning a reward model. Empirical results on a large number of tasks show that RLC performs better than best-of-N, RLHF, plain RLAIF and other baselines to fine-tune small models.", "questions": "- It is not clear to me how accuracy is computed in experiments of section 4.1.\n- What is the difference between best-of-N and w/ SE in section 4.3?\n- Do you have an intuition why preference-based RLAIF is weaker than RLC? Better “reward” prompts in RLC as opposed to the use preferences in RLAIF? Having to explicitly learn a reward model?\n- In Table 4, any idea why RLFT is stronger than RLC on “Reasoning about Colored Objects”?\n- It is often not really clear what model is used (4.2, 4.3, 6) and why results are not reported for different model sizes (as claimed in the abstract)\n\n \nMinor nitpicks:\n- Define Self-Evaluation (SE) in 4.3\n- Define Self-consistency (SC) before self-train as SC is mentioned before definition.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is very well structured and well written. I particularly liked the structure and factorization of the appendix, as well as the experiments from Section 4 that confirms the intuition that LMs are better at evaluation (as humans)\n- Showing that the internal reward / ranking capabilities of LMs are correlated with heuristics metrics (BLEU, BERTScore, ROUGE).\n- Experiments are well-executed and done on a variety of standard tasks.", "weaknesses": "**Weaknesses**\n- Section 4.3 looks a bit weak to me (greedy and 3 samples) to evaluate the accuracy of best-of-N with the best evaluated text versus accuracy of average of LM samples. Furthermore, greedy is evaluated against 1 or 2 samples) \n- Why doing only 3 samples (this experiment does not require any fine-tuning and is done on a small LM whose identity is unknown).\n- The method is simple and not novel: it relies on two “template” reward prompts (defined section 5) and good parsing of LM’s output (as opposed to RLAIF that learns a reward model). This is not a problem to me as long as experiments are done rigorously.\n\n**Additional experiments  (could make me increase a bit my score)**\n- Could you plot the evaluation of the LM when fine-tuning from RLC (for instance on the Figure 7 from appendix), as well as the heuristic metrics (BLEU, BERTScore, ROUGE) when applicable?\n- Would appreciate a discussion (backed by experiments) on the RLC/RLAIF updates compared to doing more sampling and non-invasive self-evaluation. \n\n**Recommendations (could make me increase my score considerably):**\n- From “self”-improvement to “any”-improvement: what if the evaluation model is different from the LM to improve? \nwould be interesting to have a more complete view: e.g. using smaller/bigger evaluation models. Similarly to the RLAIF baseline that uses GPT2 through the RLAIF baseline in 6.2 but not using RLC.\n- The authors mentioned two reward prompts (section 5): it could be interesting to make the LM learn search prompts given a task description.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U2PzVfS8t2", "reviewer": "Reviewer_jzQi", "summary": "The authors present a method Reinforcement Learning Contemplation (RLC) as an alternative to existing RLAIF workflows. Rather than training a preference model + doing PPO using the preference model, they just directly prompt the original model for reward signal, following analysis indicating that evaluation is easier than generation. They evaluate on a number of natural language reasoning tasks and outperform their baselines.", "questions": "--Do the RLAIF and other baselines also get to use chain of thought for the reasoning tasks, like your method? \n\n--I saw you mentioned in the appendix that you use GPT2-Large for the reward model for RLAIF, rather than the same Flan-T5 model you use for the main model. Is it fair to use Flan-T5 to evaluate outputs for your model, while using finetuned GPT2-Large (presumably weaker) to compare outputs for your baseline?\n\n--What are the prompts used for RLAIF / other baselines?\n\n--Using language models to directly output 1-10 scores seems like it might suffer from poor calibration - why not e.g., try to finetune with a regression head on top rather than just using the LM out of the box? Would that improve performance?\n\n--The numbers in 6.4 don't seem very convincing for what you're claiming, if anything it seems like the performance went up more on the unrelated tasks (object counting, penguins)?\n\n--Nit: I don't really see why you distinguish between RLCAI and RLAIF; from what I can tell you basically mean the same approach for both terms? (And the paper you cite for RLAIF attributes the term RLAIF to the original Anthropic paper anyway.)", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "--Comprehensive results on several reasoning tasks, with interesting analyses.\n\n--Interesting to see that it seems to work reasonably at small model scales, unlike the original Anthropic RLAIF paper\n\n--The new pipeline is a bit simpler than RLAIF, not needing to train a preference model as an intermediate step, while still working well", "weaknesses": "--I'm not sure that the idea of evaluation being easier than generation is particularly new; you could view this as the intuition for RLAIF too, where you get the model to evaluate itself and then use the preference model to improve its generations later using PPO. I'm not sure it's the right motivation for your work anyway; if I understood correctly, the main difference between your method and RLAIF is that you *evaluate the absolute quality of only one output* at a time (either binary or 1-10), rather than *evaluating the relative quality of two outputs*. Then naturally you don't bother fitting a preference model since you already have the reward due to predicting absolute quality rather than relative quality.\n\n--On a related note, these tasks seem a bit nonstandard for RLAIF evaluation, are there other works using these too? I'm curious whether the results hold up when you do the classic tasks like harmlessness and helpfulness. In particular, if I understand correctly, one major difference is that the tasks you evaluate seem to have a \"gold\" \"correct\" answer, unlike e.g. harmlessness and helpfulness, which makes it reasonable for your model to label the absolute quality of a given output rather than comparing relative quality between two outputs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)