Enabling Lanuguage Models to Implicitly Learn Self-Improvement
Authors: Ziqi Wang
Le Hou
Tianjian Lu
Yuexin Wu
Yunxuan Li
Hongkun Yu
Heng Ji
Venue: ICLR 2024 poster (2024)
Keywords: large language models; self-improvement; alignment
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in open-ended text generation tasks. However, the inherent open-ended nature of these tasks implies that there is always room for improvement in the quality of model responses. To address this challenge, various approaches have been proposed to enhance the performance of LLMs. There has been a growing focus on enabling LLMs to self-improve their response quality, thereby reducing the reliance on extensive human annotation efforts for collecting diverse and high-quality training data. Recently, prompting-based methods have been widely explored among self-improvement methods owing to their effectiveness, efficiency, and convenience. However, those methods usually require explicitly and thoroughly written rubrics as inputs to LLMs. It is expensive and challenging to manually derive and provide all necessary rubrics with a real-world complex goal for improvement (e.g., being more helpfulness and less harmful). To this end, we propose an imPlicit self-ImprovemenT (PIT) framework that implicitly learns the improvement goal from human preference data. PIT only requires preference data that are used to train reward models with no extra human efforts. Specifically, we reformulate the training objective of reinforcement learning from human feedback (RLHF) -- instead of maximizing response quality for a given input, we maximize the quality gap of the response conditioned on a reference response. In this way, PIT is implicitly trained with the improvement goal of better aligning with human preferences. Experiments on two real-world datasets and one synthetic dataset show that our method significantly outperforms prompting-based methods.

Published as a conference paper at ICLR 2024
ENABLING LANGUAGE MODELS TO IMPLICITLY LEARN
Ziqi Wang',Le Le
Lu', Yuexin Wu², Yunxuan Li", Hongkun Yu², Heng Ji 1 University of Illinois Urbana-Champaign 2 Google
zigiw9011 weeiiiinnniii...ii edu Lehouêgoogle.com com
ABSTRACT
Large Language Models (LLMs) have demonstrated remarkable capabilities in open-ended text generation tasks. However, the inherent open-ended nature of these tasks implies that there is always room for improvement in the quality of model responses. To address this challenge, various approaches have been proposed to enhance the performance of L.MMs. There has been growing focus on enabling LLMs to self-improve their response quality, thereby reducing the reliance on extensive human annotation efforts for collecting diverse and high-quality train- ing data. Recently, prompting-basee methods have been widely explored among self-improvement methods owing to their effectiveness, efficiency, and conve- nience. However, those methods usually require explicitly and thoroughly written rubrics as inputs to LLMs. and provide all necessary rubrics with a real-world complex goal for improvement (c.g., being more helpful and less harmful). To this end, we propose an ImPlicit Self-ImprovemenII (PIT) framework that implicitly learns the improvement goal from human preference data. PIT only requires preference data that are used to train reward models without extra human efforts. Specifically, the training objective of reinforcement learning from human feedback (RLHF)- instead of maximizing response quality for a given input, maximize the quality gap of the response conditioned on a reference response. In this way, PIT is implic- itly trained with the improvement goal of better aligning with human preferences. Experiments on two real-world datasets and one synthetic dataset show that our method significantly outperforms prompting- methods.
expensive and challenging to manually derive
reformulate
INTRODUCTION
LLMs (Devlin et al., 2018; Raffel et al., 2020; Brown et al., 2020; Chowdhery et al., 2022; Schulman et al., 2022; OpenAI, 2023; Anil et al., 2023) have achieved saat-oof-hhe-as results on complex tasks such as math reasoning (Wei et al., 2022; Xue et al., 2023; Zhou et al., 2023), summarization (Stiennon et al., 2020b), conversations (Schulman et al., 2022; Bai et al., 2022), schema induction (Li al., 2023) and solving domain-specific problems (Singhal al., 2022). The keys of LLMs success are their abilities of following instructions and aligning with human preferences (Ouyang et al., 2022: Peng et al., 2023a; Shen et al., 2023). A widely adopted approach toward them is instruction fine-tuning and reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022). However, instruction fine-tuning -uning and RLHF are imperfect, and there always room for improvement. For example, LLMs may hallucinate information (OpenAI, 2023), have reasoning errors (Bubeck et al., 2023), and generate unhelpful and harmful contents (Bai al., 2022). straightforward approach is to collect more diverse and high-quality data and improve the alignment with summan-nn-the-loee training paradigm (Ouyang et al., 2022), which requires extensive amount of human effort, especially for specific domains that require expert knowledge.
Therefore, the community has explored to use LLMs to self-improve their own response quality without human intervention. With the advent of generative language models, prompting methods have proved effective and efficient (no need to train models) with convenience (no need to serve models and can be used with black-box language models through APIs). Madaan al. (2023) use
Work done when interning at Google
Correspondence Author

---

Published as a conference paper at ICLR 2024
PIT
I need to improve (LLM Output]
LLMs
[pppt]
[LLMM Outpur]
IPIT-Iuprowed Outpui]
Inference
Reinforcement Learning
Is [PIT- IIIp-lpe Oulput| than [LL.M Output]?
Training
PIT Reward Model
PIT
LLMs
Prompting (Self-Refine)
[npus]
LLMs
[LLM Ouipui]
Reflection
[Feodback]
LLMS
[UL M-Impooee
Refine Ourpon]
Figure I: The pipeline of PIT and prompting methods (Self-Refine). Upper:
PIT utilizes inputs and
LLM outputs for further improvement during inference. During training, a reward model will assess the gap of PIT-improved outputs and LLM outputs, and the reward used for reinforcement learning to train PIT. Lower: Self-Refine uses LLMSs to give self-feedback and then asks LLMs to improve outputs based on the self-feedback.
LLMs to give self-feedbach and iteratively improve LLMs response quality, Chen et al. (2023) enable LLMs self-debug enhance code generations. Nevertheless, ceff-improvement in language models through prompting can be less than ideal, as humans often find it challenging to define comprehensive improvement goals and create detailed assessment rubries. For example, improving helpfulness and harmlessness of LLMs requires careful definitions of these qualities to guide LLMs in improving their responses accordingly. As result, prompting could make LLMs self-improve well only if the improvement goal is clear, simple, and well-defined through prompting. LLMs perform poorly in our experiments we ask them to make responses more he helff.l. This because the L.LM tends to add more details to the response and thus makes the response longer, whereas more details are not always preferred since it may lead to the response going off the topic. Instead, asking language models to be polite, offer necessary informat on, and avoid going off t topics cs will yield better improvements. Previous methods have also discussed similar observations on the effect of detailed rubrics. Rafailov et al. (2023) showed that using simple rubric of Which summary is better? will yield much more disagreement with humans than a more compre- hensive rubric of Which summary ry surma ri zes most important points without including unimportant details?
Despite the effectiveness, detailed rubries are often hard obtain. First, it is hard for humans infer and write all possible rubrics. Second, when the task requires domain expertise (e.g., clinical domain (Singhal et al., 2022)), it is impractical scale up. To this end, we switch our focus from explicitly designing rubrics to implicitly learning self-improvement from data. We notice that the preference data used to train reward models implicitly tells how to improve the response quality. Thus, we utilize this preference data to train a novel model that implicitly understands self eelf-improvement goals from data. Using this method, we eliminate the need for rubric design and avoid the need for additional data since we only reuse the data used to train reward models.
We denote our approach ImPlicit Self- Sall--Iproveeenn (PIT), a novel approach that enables the model to learn self-improvemmet implicitly from data. Specifically, we reformulate the instruction fine- tuning and RLHF process and switch the training objective from maximizing response quality for given inputs to maximizing response quality gap conditioned on reference responses for given inputs. Figure I shows the working flows of PIT and prompting methods. PIT utlizes given inputs and reference responses and generates improved responses accordingly. Similar prompting methods, PIT can repeat this process iteratively by replacing reference responses with improved responses. Compared with prompting methods, our method PIT does not require manual rubric design. Extensive evaluations on two real-world datasets and one synthetic dataset show the effectiveness of PIT compared with prompting methods such as Self-Refine (Madaan et al., 2023).
2 RELATED WORK
A Alignment Alignment is critical for a helpful and harmless language model (Ouyang et al., 2022; Bai et al., 2022). One common way for alignment is RLHF (Ouyang al., 2022). However, RLHF is
2

---

Published as a conference paper at ICLR 2024
sensitive to the training details and is complicated to implement. To get rid of the RL, Lu et al. (2022) use quantized rewards as control tokens to continually fine-tune policy models; Diao et al. (2023);
Dong et al. (2023): Gulcehre et al. (2023) use reward models to filter out high-reward generations for further fine-tuning, Some approaches are even simpler and do not require reward models. Liu et al. (2023) use human feedback control tokens directly fine-tune models on model generations. Sun et al. (2023) use Homan--riitenn principles to guide language models to generate helpful and harmless responses and use generated responses to fine-tune models. Zhao et al. (2023) use a calibration loss to mimic RLHF. Rafailov et al. (2023) designs a loss function that can be theoretically proved to be identical to RLHF. Xiong et al. (2023) shows that online alignment is much better than offline alignment. We used RLHF for alignment in our paper, it is the most commonly used method.
Self-improvement Self-improvement enables language models to improve themselves without extra human effort. Moreover, the self-improved responses can then be utilized for context distillation (Askell al., 2021) to update LLMss. Huang et al. (2022) use PaL.M (Chowdhery al., 2022) to label more task-specific data and use the most confident labeled data to continue fine-tuning PaLM itself. However, fine-tuning language models on specific tasks may lower the overall generation performance of models (Zhai et al., 2023). Therefore, researchers also put much effort into self- eeff-imprveemnt without modifying language models themselves. Shinn et al. (2023) put language models into an environment and let language models reflect their actions when they meet failures. Wang al. (2023a) use the Python interpreter's error message to improve language models' code generation abilities. However, these methods require environments that can provide automatic feedback, which may not be accessible in general. Self-Refine (Madaan et al., 2023) enables LLMs to reflect on their responses and provide feedback, then ask LLMs use feedback to improve responses. Zeng et al. (2023) apply Self-Refine to generate better meta-reviews for papers. Xue et al. (2023) guide LLMs to finegrained feedback for math problems. The feedback could also benefit from tools. Zhou et al. (2023) use OpenAl's is code interpreter to collect more accurate feedback. Wang et al. (2023b) propose a benchmark to evaluate how models can get improved from human or Al feedback. Nevertheless, these methods require explicit prompting to collect feedback and self-improve. Our method does not require modifying model weights, interactive environment, or explicitly prompting.
3 METHOD
Policy models (i.e., "LLMMs" in Figure 1) trained with RLHF generate reference responses for given inputs, whereas PIT takes the given inputs and reference responses as its inputs and generates improved responses. The different input formats between policy models and PIT (See Appendix A for details) requires reformulating RLHF training objectives to train PIT. We follow and reformulate RLHF steps proposed by Ouyang et al. (2022): supervised ane-tuning, reward model training, and reinforcement learning. Although there are many other alignment methods such as Direct Preference Optimization (Rafailov et al., 2023) Preference Ranking Optimization (Song et al., 2023), choose RLHF to train policy models and PIT for two reasons: (1) RLHF is the most widely used approach for alignment (2) PIT is not aimed for alignment but for triggering self-improvement. Therefore, the alignment method is not the focus of this paper.
3.1 FORMULATION
Suppose we have data D = (xx, Vt, )w))sm where x is the input prompt, yt is the worse model generation, and yo is the better model generation, annotated by humans. We could equally divide the data into three folds Dsrt, DRM and DRL. for the supervised fine-tuning, reward model training. and reinforcement learning, respectively. The policy model Mp is trained to generate response yr for the given input x: yp Mp(-Iæ). ). PIT model MpπT is trained to generate improved response yprr for given input x and reference response Mref ypIT ~ Mpr (-12;, Vier). In the following sections, we use subseript "p denote the policy model and 'pir to denote PIT, and use superscripts such as SFT or RL to denote different checkpoints of models.
3.2 SUPERVISED FINE-TUNING
Supervised fine-tuning (SFT) is the pre-requisite step of RLHF (Ouyang et al., 2022). SFT only uses x and yu, and yo could also be policy models, we follow Ouyang et al. (2022) and simply maximize the likelihood of yo: L$ De. log a2)).
responses Ouyang al. (2022). To train
3

---

Published as a conference paper at ICLR 2024
Since PIT aims to improve reference responses, need to include y to train PIT: CPT "E (e.w.. JED log Myr [z, y1). A natural extension for this loss function is that we can apply unlikelihood (Welleck et al., 2019) loss to Merr (u) Vie). Mprr (tr a, vr) and Mprr Iz Mwr) However, we find the unlikelihood loss causes performance degradation experiments, therefore we only keep likelihood loss when training PIT with SFT.
3.3 REWARD MODEL TRAINING
The reward model is used to judge how good a response is and is the key component for reinforcement learning. reward model denoted as Rp maps an input v and the corresponding response y to scalar r: r(z, y) = R2 y). Since the data DRM only contains preference (y) and yw) and does not provide r directly, Ouyang al. (2022) trains the reward model by maximizing the reward gap between yt and Yw: CBM
EpRM log (Tw Tı), where Tw and respectively. Unlike Rp, the reward model RpIT does not focus on rewards r of responses but cares about the gap r'gup between responses: r'ga(( (Z, V1 V2) (2r V1, V2). Therefore, Reir needs to learn (rw - r() from the training data DRM- To formulate the loss function, we can use the fact that
rewards, i.c. Rp(x, yw) and Rp y1).
r'aa (2, Mus 31) 2 T'ap(22 (2t, ywu Mw.) ~ (2, 1/: v) Teap (2, yt Mw.
(1)
To ensure that the reward model performs as expected in Equation 1, after training, we consider all pairwise relations mentioned above, and the loss function becomes:
CRM
Σ
log or (T
2 + log (r
Il gup
+ log (r - plwy
DAM
+ log or (y -1o))) + log (r +
where wif is the shortcut of gapp (2, Dwes 31), etc. Although there are other options to model the reward gap, such computing the rewards subtraction through Rp, we find Equation 2 is the best fit. More discussions could be found in Appendix C.
3.4 REINFORCEMENT LEARNING
Reinforcement learning (RL) finds a policy that maximizes expected rewards over time, aligning LLMs with human preferences. The optimization goal for RL is (Ouyang et al., 2022):
Optimization RT.
Σ [r(x, v) (ML (u/ M (ylz)]
DRL
where r(z, y) Rp(2, y) and y Mt (-Iz). M the policy model to be optimized, which is 1
initialized MSFT. MSFT is Mp trained in supervised fine-tuning, which fixed in the RL. The KL divergence is used to restrict dramatic weight changes and reward hacking in RL.
3.4.1 CURRICULUM REINFORCEMENT LEARNING
Different from Equation 3, PIT aims to improve reference response Vier instead of generating response from scratch. Therefore, the RL for PIT utilizes æ and yref simultaneously. The difficulty is how to choose Wref. An intuitive way is to use y and you provided in the dataset, and the optimization goal becomes:
Optimization RL.
Σ
Σ
(remp(a, Mee)) (KKLLMML (Mlaa, ther) M (yll Mrer)
Dmu MorE (v1.9w)
However, PIT aims to improve ME responses, and y/ and yo are chosen from the annotated data, not sampled from the MRL Therefore, we need to do another round of reinforcement learning, where Vher sampled from the policy model:
Optimization RL,
Σ
Σ
[r (I. y, Brer) - (MPT (y/z, Brer)
SFT (y) Jr, Vrer)
DEL Vee MEL(-12 (-122)
(5)
The flexibility of thet in fact enables PIT further. For example, third round Optimization RL, can sample Yeer MBL (-|2;, y1), where y1 MRL (-|ar) with other terms unchanged. The third round enables PIT improve the improved MRL responses. In principle, this process could be extended infinite rounds. We denote this process as curriculum reinforcement learning.
do multiple rounds of reinforcement learning improve
4

---

Published as a conference paper at ICLR 2024
3.4.2 DISCUSSIONS ON CURRICULUM REINFORCEMENT LEARNING
is worth noting that the first round Optimizationpp: is necessary since the optimization Optimization RL. is too hard to be optimized directly. This is because responses sampled from Mş are already of high quality (higher than yI and Vv in the data, as shown in Section 4.6), and are harder to be improved compared to yI and yu- Therefore, curriculum reinforcement learning is needed to ensure the success of optimizing Optimization RI. We find that removing Optimization RL. and directly optimize Optimization RI. will make PIT fail to improve M responses (Section 4.6). Be- sides, purely training PIT on OptimizationaT Rt. 0 will also lead to a disaster since PIT never saw data sampled from Mt (Section 4.6). In fact, we could even insert several intermediate rounds between the first and second rounds to build smoother curriculum. Concretely, we could choose Yee sampled from intermediate checkpoints of Mp (e.g., checkpoints dumped during training and have been optimized for high rewards yet) since their responses will be less perfect and easier to be improved. In our experiments, we use OptimizationpT RL. and Optimizationpi: RL. to optimize PIT. This is because we find the dataset is not hard enough to require more rounds and intermediate rounds. The algorithm block of PIT can be found in the Algorithm 1.
3.5 SELF-IMPROVENER INFERENCE
Given the input we first sample Vref MP (-1z). Then we can get improved response y1 MRL
(prr(- Meet). The improvement process could be repeated infinite times. For examples, could get y/2 Mptr [z, 31) and y/ ( 92). etc. The self-improvement inference process is summarized in the Algorithm 2. Moreover, PIT does not bring extra computational overhead compared with prompting methods such Self- (Madaan et al., 2023). Detailed discussions can be found in Appendix B.
4 EXPERIMENTS
Our experiments are designed to answer two questions: (1) Can our method improve the quality of the original response? The term 'quality" here is defined by the preference data. For example, if the preference data to get a more helpful and less harmless response, then 'higher quality' denotes more helpful and less harmless. (2) Can our method better improve the response than prompting methods such as Self-Refine?
4.1 DATASETS
We use three diverse datasets for our experiments:
Anthropic/HH-RRIHH The HH-RLHF dataset (Bai al., 2022; Ganguli al., 2022) is released by Anthropic and is allowed for research purposes, which contains 161K conversation pairs between humans and AI assistants for training and 8.55K for testing. The dataset aims to train helpful and harmless AI assistant and has two subsets: the helpful subset and the harmless subset. In our experiment, we only use the helpful subset and divide the subset equally into three folds for supervised fine-tuning, reward model training, and reinforcement learning.
OpenAl/Summary The Op/uma/Summary (Stiennon et al., 2020a), which also allowed for research purposes, contains Reddit posts and two summaries for each post with human preferences. It contains 92.9K training data and 86.1K validation data. Similarly to the Anthropic/HHH-RI. dataset, we equally divide the training dataset into three folds for supervised fine-tuning, reward model training, and reinforcement learning, respectively.
Synthetic Data To test the instruction following abilities of language models, we use PaLM 2 (Unicorn) (Anil et al., 2023) to generate 13K synthetic data. Every synthetic data includes a question with multiple requirements, such as using a specific format for writing responses. It also includes satisfactory response that meets all of the requirements, as well as an unsatisfactory response that only meets some of the requirements. Lastly, there is a reflection response to make it satisfactory (not used in our experiments). Similarly, we divide synthetic data into three folds equally and use 128 examples for evaluation. The synthetic data generation process and examples can be found in Appendix D.
how improve the unsatisfactory
5

---

Published as a conference paper at ICLR 2024
4.2 SETTINGS
We use pre-trained PaLM 2 (Bison) as our backbone language model and reward model since we find smaller models' (Gecko and Otter) generation abilities are poorer, and larger models (Unicorn) are too expensive to train. We train our models on TPU v4 (Jouppi et al., 2023). In our experiments, we compare PIT with other seff-improvement methods using prompts, specifically Self-Refine (Madaan et al., 2023). Self-Refine uses prompts (Appendix E. to instruct the model M to reflect the current response and give feedback to improve the response, then asks Mş to give an improved response based on the feedback. We set the sampling temperature to be for all models during inference unless stated otherwise. More experiment settings can be found in Appendix F.
4.3 EVALUATION MODELS
We use third- third-party LLMs and reward models that are allowed be used for research as evaluators, similar to previous works (Rafailov et al., 2023; Gulcehre et al., 2023; Dong et al., 2023). LLMs are sensitive to the rubric and need carefully designed prompts to achieve good performance for evaluation (Rafailov et al., 2023), which is also aligned with our motivation. Re- ward models may have reward hacking phenomenon (Skalse et al., 2022). To get a reliable evaluation, we use both third- party LLMs and reward models to do evaluations and use human evaluations when the two evaluations disagree. Evalu- ation prompts for LLMSs are adopted from Zheng al. (2023); Rafailov et al. (2023) and are shown in Appendix E.2. We com- pute the agreement of LLMs with ground-truth validation data (Appendix G), and GPT-4 has the highest agreement compared to ChatGPT and PaLM 2 (Unicorn) and will be used as the representative of third-party language models to do evaluations afterward. We use DeBERTa-Large (304M) (He et al., 2020) trained by Open Assistant the representative of reward models because it uses different model architectures (compared with Rp and RŞIr) and full training data (Rp and RęTr only use 1/3 data) and is not used to optimize models (Reir and Rp are used for optimization), reducing the risk of reward hacking during evaluations. Moreover, it performs similarly to GPT-4 on the ground-truth validation data (Appendix G).
0.30
0.25
0.20
Density
rtyl. yD)
0.15
rlyw,
rlyw, yww)
rty.. yw)
0.00
30
20
100
0
10
Reward
Figure 2: Reward distribution of Rerr the synthetic data, which fol- lows Equation 1.
We also compare the agreement of RPiT and Rp with the ground-truth labels on validation sets (Appendix G), and conclude RPIT generally outperforms Rp, expected. We then draw Rerr reward distribution on the synthetic dataset (since reward models perform best on this dataset among others and the distribution pattern is more noticeable) in Figure The distribution exactly follows our loss design in Equation| 1. Moreover, we observe that the reward is negative when two responses are identical, indicating that PIT penalties repeating responses (or generating responses that have the same qualities as reference responses).
4.4 RESULTS
Since GPT-4 API is expensive and slow, we evaluate 128 examples when using GPT-4 and 1,000 examples when using the DeBERTa reward model. To reduce the noise, the reward model will give a tie the rewards for two responses are similar. In practice, the reward model gives a tie ( - 12) E [0.45, 0.55). where r1 and r2 denote the reward of two responses. We first sample responses Yref MÉL (x) (denoted 'Original Responses') and then use self-improvement methods to improve Wiet- Appendix H.l shows several qualitative examples.
PIT improves responses' qualities. We apply PIT original responses and get improved responses. Then, we report the win rate of PIT against original responses. We use the difference between the win rate and the lose rate (A for short) to denote the performance of PIT. Table shows results. First, we find that the original responses are much better than you in data, showing that MfL is well optimized (this conclusion does not hold on synthetic data, which is expected since a much larger LLM produces y..). We can see PIT consistently has better qualities (ranging from 7.2% 33.59%

---

Published as a conference paper at ICLR 2024
Table I: Comparisons among V/e in data, original responses, improved responses by Self-Refine, and improved responses by PIT on three datasets. Win rate / Lose rate denotes the percentage of the former model's responses that is better / worse than the latter's. s. Their difference A denotes that the former response is better and vice versa. Higher A denotes higher performance gap.
Win rale Lose rate / A (%)
Dataset
Comparison
GPT-4
DeBERTa
Human Evaluation
Original vs. the PIT vs. Original Self-Refine vs. Original PIT vs. Self-Reline
7L.85/1..1/54.1
68.20/18.000 05/00.205.0
53.47 24447/77.4//288..
46.30/32 20.30/33.30/144 /1/4.00
Anthropic/HH-RRIn
60.944/1.19/43...
40.30/31 40/8.90
38.28/42.19/ 3.91
11.3/37.60/3.// 60/3.70
47.06/ /3.53//33.53
Original vs. Yu PIT vs. Original
74.22/8.59/65. /65.6//66663
84.90/ 100/11.770/ 10770/7..20
OpenAU/Summary
41.53/24.22/20.33
41.9/31.777.2
Original vs. the
28.91/51.56/ -2.66
PIT vs. Original
48.44/14.84/33..88
Synthetie Data
Self-Refine vs. Original PIT vs. Self-Retine
34.35/17.97/16.411
/5.31/35.16/10.16
than original responses across three datasets with both GPT-4 and DeBERTa evaluators, showing the effectiveness of our method. Since DeBERTa performs poorly and GPT-4 has a high agreement with the ground-trute truth labels the synthetic data (Appendix G), we only use GPT-4 as the evaluator for the synthetic data. It is worth noting that the summation of the win rate and the lose rate is less than 1 because of the existence of the tie rate.
PIT improves response quality better than Self-Refine. We then compare Self-Refine with PIT and original responses. Table shows the comparison results the Anthropic/HH--RLHH dataset. GPT4 and DeBERTa both agree that Self-Refine indeed improves response qualities. However, GPT-4 prefers Self-Refine more (3.91% better than PIT), whereas DeBERTa prefers PIT more (3.70% better than Self Self-Refine). This is understandable since GPT-4 and Self-Refine use manual prompts to evaluate or improve responses, making them prefer each other. On the contrary. DeBERTa and PIT are trained on the same data. Therefore, we conduct human evaluations to determine which is better and find that human prefers PIT more (3.53% better than Self-Refine). Appendix shows details of human evaluations. The disagreement between GPT-4 and humans arises from GPT-4's inclination to long and detailed responses and Self-Refine's preference to generate such responses. Nonetheless, these responses can include irrelevant information not related to the questions humans raised, which is not desirable to humans. Appendix H.2 contains one concrete example of this phenomenon. Table 1 also shows results on the synthetic dataset, where we can easily conclude that PIT outperforms Self-Refine, though they all improve original responses.
4.5 THE INFLUENCE OF IMPROVEMENT TEMPERATURES
In the above experiments, set the temperature be I for MR MM and Self- Refine. However, higher temperatures often represent diversities and randomness, while lower temperatures usually represent coherence and qualities Intuitively, self-improvement is expected to improve the original response along one most reliable improvement direction rather than randomly explore directions (which many harm the improvement performance). Therefore, a low temperature may be preferable to restrict the diversity of the improvement. Fig 3 shows the difference between win rates and lose rates among the original responses, improved responses by Self-Refine, and improved responses by PIT under different temperatures. The original responses are generated with the temperature 1, and improved responses are generated with different temperatures. as the figure shows. To achieve cost-effectivennes,, we have opted for DeBERTa as the evaluator to assess I 1,000 instances of Anthropic/HHH-RL.n
and O/SummlSr.mmarr. We made this choice because, as indicated in Table and Section 4.4, DeBERTa correlates more closely with human preferences than GPT-4 does. Nevertheless, for the evaluation of 128 examples of synthetic data, we will rely on GPT-4 as DeBERTa is not ideal for this purpose, shown in Appendix G. We do not report the results of Self-Refine for OlemmA//yymrrry since the dataset only contains summarization instructions, making Self-Refine not applicable.
On Anthropic/HH- HLHHL dataset, we find our method improves the original responses most with low temperatures (0.4 ~ 0.6), which fits our assumptions proposed at the beginning of this section.
7

---

Published as a conference paper at ICLR 2024
Win rate Lose Pate (36
30
(%)
30
20
Win rate Losse rate
25
30
20
15
15
10
PI Original
FTT vs. Original
Sall Sufine vs. Original
Sal-Seetme
PRT
Win
PTT v. Seltiilatin
Self-Bofitnnt Oeigin)
-10
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
1.0
Temperature
Temperature
Temperalure
(a) Anthoopic/HH-RLEH
(b) OpenAU/Summary
(c) Synthetic Data
Figure 3: The difference between win rate and lose rate (A) among original responses, improved responses by Self-Refine, and improved responses by PIT under different temperatures. A > 0
denotes the former response better, and higher | denotes larger performance gap.
Table 2:
The effect of curriculum reinforcement learn- ing. great performance drop is observed when we only use RL once. A < 0 represents 'XX RL Only' response quality is worse than the compared method.
-0.25
Second Only
-0.50
P.T
Evaluator: DeBERTa: Dataset: Anthropic/HH--ee RLLF
Better
Win rate Lose A (%)
Reward
-0.75
Ouly
First RL Only
Second RL Only
-1.00
Siniher
1.25
vs. Original
40.50/36 36. 10/4.40
28.90/25 70/0.20
vs. Self-Refing 32.0/47.7 7/ 15.70
28.50/47.40) 18.90
-1.50
PIT
20.70/50.50 30.10
19.30/56.000 00/ 36.70
-1.75
Weeg
-2.00
500
1000
1500
2500
Table 3: ELO scores (higher is better) of different improvement iterations on 1, 000 conversations in Anthropic/HH-RLHH and 1, 000 posts in OpenAl/- Summary. Evaluator: DeBERTa.
Step
Figure 4: Rewards w.r.t. training steps. Re- wards are divided into three regions. The Better region denotes the model can im- prove original responses; The Simá 1 re- gion suggests the model can only generate responses that are on par with original re- sponses; The Worse denotes the model can- not improve original responses and make original responses worse.
AuthropicHHHRRU.M
Responses
ELO
Rank
PIT (Iter 4)
1036
1
ASSmmaayy
(lter
Responses
ELO
Rank
PIT (Iter 5)
1025
3
PIT (Iter
1049
(Iter
PIT (Iter 2)
1016
S
PIT (Iter
PIT (Iter 3)
1031
1022
Self-Refine (Iter I)
1001
PIT (Ter 5)
1013
Self-Refine (Iter 3)
995
7
PIT (Iter
985
Self- Refine (lter 4)
990
Original
900
Self-Refine (Iter 2)
982
Sell-Refin Refine (Iter 5)
982
10
Original
921
11
However, Relf-Refine improves the original responses most with high temperatures (0.6 ~ 0.8). We find that this because Self-Refine tends keep all the original response and only append extra details to the original response with low temperatures, leading to sub-optimal improvement. Corresponding examples can be found in Appendix H.3. OpenAl/Summary shows similar observations, whereas PIT is not heavily affected by temperatures on the synthetic data. This is probably because we only evaluate the instruction following ability and do not evaluate the concrete content quality on the synthetic data. Nevertheless, PIT outperforms Self-Refine under all temperatures (except tie under the temperature 0.8 on the Authropic/HH Mathropic/HH-RLER RLHF dataset). Moreover, we compare PIT and Self- Refine under the best temperatures Inthropic/III-RLLHFi... 0.4 for PIT and 0.8 for Self-Refine), and find PIT still outperforms Self-Refine by 9.2%.
4.6 THE EFFECT OF CURRICULUM REINFORCEMENT LEARNING
We conduct reinforcement learning twice in PIT with the difficulty increases, as illustrated in Section 3.4. The first RL (Equation 4) aims to teach models improve ground-truth responses, whereas the second RL (Equation 5) aims to teach models to improve M responses. To demonstrate the importance of curriculum reinforcement learning, we trained two additional PIT variants. One model was optimized using the first RI, while the other model skipped the first RL and directly used the
8

---

Published as a conference paper at ICLR 2024
second RL. We compare the two extra models with PIT, responses sampled from Mp the Anthropic/HH-RLHH datasets. Table 2 shows that both models can only improve original responses marginally and have a large performance gap when compared to Self-Refine and PIT. The results are expected since the first model does not learn how to improve the original responses but is only trained on easier ground-truth data. On the contrary, the second model directly facing hard optimization problem (Table shows that original responses are much better than M..). Examples can be found in Appendix H.4.
conducted by M and original
We also show rewards w.r.t. training steps when training Mpi with Equation and training a PIT variant with second RL only in Figure 4. Rewards given by Rprr Bher), where Yhet M (x) ) is the original response. It clear that the model with second RL only struggles to improve Yiet due to the hard optimization, whereas PIT can break through the Simi lar area thanks to the first RL. Besides, the starting point of PIT in the Figure 4, i.e., the PIT variant with the first RL only, fails to improve Yref, as expected.
4.7 THE EFFECT IMPROVEMENT ITERATIONS
Knowing when to stop self-improvemmnt is crucial since the improvement process could be iterated infinitely in principle. The stop condition depends on the actual needs. For example, the stop condition could be fixed wall time or the significance of each improvement (i.e., stop further improvement if the current improvement is marginal). In this section, we only focus on the response qualities and investigate if the response qualities become better with more improvement iterations. If the answer is yes, then more iterations are always welcomed. Otherwise, stop conditions need to be carefully designed in real applications. We use ELO scores (Zheng et al., 2023) to show quantitative results among models.
We PIT and Self-Refine (with their best temperatures respectively) to improve 1, 000 original responses for 5 iterations on Anthropic/HH-RLLHH We then use the DeBERTa reward model to compare 11 responses (Appendix H.5 shows an example), resulting Ch = 55 comparisons for each conversation and 55, 000 comparisons in total. We then use 55, 000 comparisons to compute ELO scores² and report results in Table 3. We get ELO scores on
meaning each conversation now has 11 responses.
similarly, except for the difference that Self-Refine is not applicable to this dataset. Table shows consistent results with previous experiments that PIT (Iter 1) better than Self-Refine (Iter 1), and the original responses have the lowest qualities. Intuitively, the best iteration for PIT should be consistent with the number of RL in curriculum reinforcement learning. However, Table 3 shows that the response qualities and the number of iterations do not follow this intuition as PIT (Iter 1) is not best. Moreover, there is not a simple positive correlation between response qualities and improvement iterations. One possible explanation may be that the datasets not need more iterations. This phenomenon suggests we develop stop conditions in real applications carefully. We observe that sometimes the self-improvement yields the same responses its previous iteration with low temperatures, which could potentially serve as a stop condition. An example shown in Appendix H.5. We can also observe that PIT is consistently better than Self-Refine regardless of the number of iterations. Therefore, PIT is better self-improvement method than Self-Refine regardless of the stop condition.
too easy and do
Since ELO scores are affected by the order of comparisons, we randomly shuffle comparisons several times and observe the change of ranks to ensure our conclusion not caused by randomness. We find that PIT is always better than Self-Refine, and the original responses are the worst among 11 responses, but the internal rank of PIT and Self-Refine may be changed with different shuffles, which does not affect our conclusions above.
5 CONCLUSION
We propose PIT, a novel approach that learns self-improvement implicitly from Pat does not require explicit prompting and extra data. Extensive experiments show the effectiveness of PIT on self- improvement compared with prompting methods such as Self-Refine. We highlight our limitations and future work in Appendix J.
We use scripts released by Chatbot Arena (Zheng et al., 2023): https: / //col/cclb//////////ccco www.erec...ggggg....... com/ / dr ve/1 J2Wf7 wxw.777xxxmmm 1mh 24 24

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY
Datasets has been discussed in 4.1. Two of three datasets (Anthropic/HH-RLHH and OpenAl/Sum- mary) are publicly available and can be easily found on the HuggingFace website. Appendix D describes the generation process of the remaining synthetic dataset. Evaluators are discussed in Section 4.3 and Appendix G. Moreover, Appendix E.2 offers the evaluation prompts. Experiment settings, baselines, and details are discussed in Section 4.2 and Appendix F. Appendix E. also offers prompts used for the baseline method.
ACKNOWLEDGEMENT
We would like to acknowledge our Google colleagues for their invaluable advice and support. In particular, thank Music Li (Yuezhang Li) for insightful discussions and manual evaluation. We thank Tianqi Liu, Honglong Cai, and Albert Webson for their constructive advice, and Léonard Hussenot and Robert Dadashi for building RLHF infra. Finally, would like acknowledge Melvin Johnson, Hongkun Yu, and Denny Zhou for their support throughout the project. We also thank the anonymous reviewers for their suggestions and comments.
This research is also based upon work supported by U.S. DARPA ECOLE Program No. HR001 12390060 and U.S. DARPA ITM Program No. FA8650-23-C-7345 and KAIROS Program The views and conclusions contained herein are those of the authors and
No. 17-50019-2-10000 should not be interpreted as necessarily representing the official policies, either expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.
REFERENCES
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm 2 technical report. arXiv preprint arXiv:2305. 104000 2023.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861 2021.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0558 0.8662,, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint 2rXiv:2303.122112 2023.
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2304.005 /28, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311. 2022.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /810004800 2018.
Shizhe Diao, Rui Pan, Hanze Dong, Ka Shun Shum, Jipeng Zhang, Wei Xiong, and Tong Zhang. Lmflow: An extensible toolkit for finetuning and inference of large foundation models. arXiv preprint arXiv:2306. 12420, 2023.
10

---

Published as a conference paper at ICLR 2024
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint ar arXtv:2304.06777, 2023.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858 2022.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu. et al. Reinforced self- ceff-training (rest) for language modeling. arXiv preprint arXiv:2.308. 08998, 2023.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhann enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654 2020.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self--iprove. arXiv preprint arXiv:2210.11611 1610, 2022.
Norm Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, et al. Tpu v4: An optically reconfigurabee supercomputer for machine learning with hardware support for embeddings. In Proceedings of the 50th Annual International Symposium on Computer Architecture, pp. 1-14, 2023.
Sha Li, Ruining Zhao, Manling Li, Heng Ji, Chris Callison-Burch, and Jiawei Han. Open-domain hierarchical event schema induction by incremental prompting and verification. Proc. The 6l Annual Meeting of the Association for Computational Linguisties (ACL2023), 2023.
Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. Languages are rewards: Hindsight finetuning using human feedback. arXiv preprint arXiv:2302.02676 2023.
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Am- manabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591-27609, 2022.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303. 230.. ,,,,77651,,2
OpenAl. Gpt-4 technical report. arXiv, pp. 2303-08774, 2023.
Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.032277, 2023a.
Baolin Peng, Linfeng Song, Ye Tian, Lifeng Jin, Haitao Mi, and Dong Yu. Stabilizing rlhf through advantage model and selective rehearsal. arXiv preprint arXiv:2309. 10202, 2023b.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305. 18290, 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1), jan 2020. ISSN 1532-4435.
John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Fe- lipe Ceron Uribe, Liam Fedus, Luke Metz, Michael Pokorny, et al. Chatgpt: Optimizing language models for dialogue. OpenAl blog, 2022.
11

---

Published as a conference paper at ICLR 2024
Sheng Shen, Le Hou, Yanqi Zhou, Nan Du, Shayne Longpre, Jason Wei, Hyung Won Chung, Barret Zoph, William Fedus, Xinyun Chen, et al. Mixture-ou ef-eertsss meets instruction tuning: A winning combination for large language models. arXiv preprint arXiv:2305.14700 2023.
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. arXiv preprini
1rXiv:2303.11366. 2023.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138, 2022.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460-9444 2022.
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. arXiv preprint arXiv:2306. /7492, 2023.
Nisan Stiennon, Long Ouyang. Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning summarize from human feedback. In NeurlPS, 2020a.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 1-:3008-30211 2020b.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. arXiv preprint arXiv:2305.0300 03047, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922 09788988, 2023.
Xingyao Wang, Hao Peng, Reyhanch Jabbarvand, and Heng Ji. Leti: Learning to generate from textual interactions. arXiv preprint
2023a.
Xingyao Wang, Zihan Wang. Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng. and Heng Ji. Mint: Evaluating lims in multi-turn interaction with tools and language feedhack, 2023b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. thain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022.
Sean Welleck, Ilia Kulikov. Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. arXiv preprint arXiv: 1908. 04319, 2019.
Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang. Gibbs sampling from human feedback: A provable kl-constrained framework for rlhf. arXiv preprint 1:21 2023.
Tianci Xue, Ziqi Wang, Zhenhailong Wang, Chi Han, Pengfei Yu, and Heng Ji. Reot: Detecting and rectifying factual inconsistency reasoning by reversing chain-of-thought. arXiv preprint wrXiv:2305.71499 2023.
Qi Zeng, Mankeerat Sidhu, Hou Pong Chan, Lu Wang, and Heng Ji. Meta-review generation with checklist-guided iterative introspection. arXiv preprint arXiv:2305. 14647, 2023.
Yuexiang Zhai, Shenghang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. Investigating the catastrophic forgetting in multimodal large language models. arXiv preprint arXiv:2309.1031 2023.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023.
12

---

Published as a conference paper at ICLR 2024
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric Xing, et al. Judging 1lm-as-a-jude with mt-bench and chatbot arena. arXiv preprint arXiv:2306.0500
2023.
Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, et al. Solving challenging math word problems using gpt-4 code interpreter with code- selffverriiiattion. arXiv preprint arXiv:2308.07221, 2023.
13

---

Published as a conference paper at ICLR 2024
A THE WORKING FLOW
To better demonstrate our approach, we Given a dataset D = {(x, yt. Vw)), we equally divide into three folds DEFT, DRM and DRL- Starting from pre-trained language model M Mpe(1)) first conduct normal SFT and RLHF prompt x as the input (e.g., Please give me three-dae -day travel plan about Los Angeles leseles.] and return a response y. The input format of M and M-r(-)zr during training and inference is:
to obtain RLHF-finetueed model (+I-.. Mp-(11 takes
Human: [x) Assistant:
where {x) denotes the concrete input æ.
Then we train PIT (-I,-)) that takes prompt and candidate response as inputs and returns an improved response. The imput format of Mpp (-12, Meef) is
Human: (z) Ciisstant<<Canddide (atr>>((ee) (
where {} denotes concrete contents and <<>> denotes special token.
Similarly, the input format of Rp(a, y) is
Human: (ssistant:(y)
and the input format (2r 91, is
(um Assiatt-canllatteep/ggkttttoooee
Algorithm 1: PIT Training
Input: Dataset Dser, DRM and DRL. pre-trained model Mpe(-I)), and RLHF-finetunee model M trained Dser. DRM. DRL. and Mpe(---).
Output: PIT Model MPT
MPTT (1).)) Mpra-11))
for v, Mt. Um in DsFr
Supervised fine- line-tuning
Supervised fine-tuning (-I-) using loss CSFT log MaT (ywl yr)
Rem((.-.) + (-e(-1))
for v, Mcs Va in DRM
Reward model training
Compute reward gap RAT(E, 31, va). E [vi, Vwr) v2 E (2i, Nw Optimize Ryr by Equation
(-ls-) + M (-I.))
while not converge
b First RL
Sample æ, Biet from DRL, where y (Mi, Mw)
Generate MB yrd)
Use reinforcement learning algorithm such PPO optimize Equation 4
done
while not converge
Second RL
-Sample x from DaL. Hhat ev ME M+((-+
Generate MEL (z, Yiet)
Use reinforcement learning algorithm such as PPO to optimize Equation 5
done
return M
When conducting self-improvement during inference, we need first to get Yhat ~ M+1(--z), and then apply MRL or Self-Refine to get improved responses. The self-improvemmnt process could be done iteratively during inference. The concrete inference procedure is shown in Algorithm 2.
14

---

Published as a conference paper at ICLR 2024
Algorithm 2: Self-Immrrvement During Inference
Input:
Input x, an response sampled from RIHFF-inctuned model Mp (-1x), PIT model (++():))) self-improvement iteration K
Output: Improved responded y
while k < K
Self- Self-Immrovement
y f ( 2r y) [PIT] or +- Self-Refine(z; y, Mg4) [Self-Refinc]
. done
return
B COMPUTATIONAL. COST DURING INFERENCE
Our approach has the same (or a bit lower) computational overhead compared with Self-Refine. The inference process is shown in Algorithm 2. Here is detailed explanation:
The inference prompt Self-Refine looks like this:
Human: Given prompt {a} and candidate response (Ures) provided
by one assistant, you need to improve the response to be more helpful. Helpful means {rubric}. Assistant:
PIT uses a much simpler prompt with only special tokens as splitters:
Human:
{x} Assi stant: << Candidaaee >>
{Ura+) << Improved >>
The 0 denotes the concrete content, and <<>> denotes the special token. Both methods will return an improved response. The difference is that PIT does not need to explicitly write prompts since self-improvement is learned during our proposed training framework. Both methods use one-time inference to get improved response, and PIT in principle has the same computational cost as Belf-Refing during inference, or strictly speaking even fewer computational costs due to the fewer input tokens (without complex rubries).
C WHY EQUATION 2?
We made many efforts to filter out other options and pin down our final choice to Equation 2. Since is not applicable to try all possible solutions, filter out other methods based prior work findings and our own pilot experiments. Here is our detailed selection process:
The first question in developing the reward model is if we directly R2 (i.e., the vanilla reward model that takes x and y as inputs and returns scalar reward r) to compute the reward gap between two responses 1/1 and y/, we need develop another reward model that directly calculates the reward gap. The most straightforward implementation using Rp is to compute the subtraction of individual rewards of 1/1 and y/2 obtained by Rg. However, previous works show that R2 fails to faithfully reflect the response quality when is high (Bai et al., 2022). That is to say, if r1 and 12 are higher than threshold, then 11 72 does not necessarily denote 3/1 worse. This phenomenon is possible because of the poor calibration of the reward model (Peng et al., 2023b). Moreover, other work (Zhao al., 2023) shows that directly modeling the reward gap brings less noise and performs better than computing the subtraction of individual rewards. Therefore, we decided to train reward model that directly models reward gaps.
Next, we need to decide the training objective. In the beginning, the training objective is simply max- imizing the reward gap (2, Ywu y1) and minimizing (x, Yt, yw), which is similar to Equation observe that LLM finds shorteut that
but without rgap (I, Yw Vie and Tgap V1, W). However, ignores the third argument, thus degenerating to R, and failing to grasp the reward gap. Therefore, add rgap (I, ywe yio) and ge (x. M1, m1) to prevent the degeneration.
15

---

Published as a conference paper at ICLR 2024
We also explore the possibility of using pointwise rather than pairwise training signals used above. For example, we tried to explicitly assign Tgap(2 yw 31) = 1, Tgap (z, Yt y/w) = and rgap (2, 3t, 3r) = (g Mio Uw) 0.5 and use MSE to train the reward model. We find this implementation is less effective than our proposed approach (e.g., Equation 2) on the held-out reward model test set of 500 examples (~ 5% accuracy drop). We think this phenomenon is because the inductive bias of assigning rewards to I, 0.5, and O is not preferred in the data distribution.
Based on the above analysis, Equation is chosen as our final choice.
D SYNTHETIC DATA
The synthetic data aims to test the instruction following abilities. Therefore, each question contains multiple requirements. We first manually design a list of tasks (e.g., email writing and advertisement writing) and requirements (e.g., using specific formats such as HTML, Markdown, or upper case.). Then, we randomly choose a task and several requirements from the list. We instruct PaLM (Unicorn) to generate a question 2g based on the chosen task and requirements and give response y accordingly. Next, we randomly select few more requirements and instruct PaLM 2 (Unicorn) to add extra requirements to 2g and get a new question x together with its response Yoe- The difference between yhe and yt is that you follows all instructions whereas yı only follows part of instructions in worth noting that yo does not necessarily have better content than yr. In the synthetic dataset, only care about the instruction following ability. At last, instruct PaLM (Unicorn) to give chain-of -f---fouhthttttt reflection c on improving Y1 to Ya. {(z, yt, Yw: c)} forms the synthetic dataset.
We use a two-step filtering method to filter out low-quality data: (1) We prompt PaLM 2 (Unicorn) to velf-verify if you follows all instructions and yi does not. (2) We manually inspect data and select a these data to fine-tune classifier to
group of high-quality data and low-quality data, and then further filter our low-quality data. In the end, we obtained 13K synthetic data in total.
Here is an example:
J:
Create serious tagline for company that designs and builds roller coasters and rides for theme parks. It should follow the format "We are the best at [what the company does]". Also, make part of your answer in HTML format.
y::
We are the best at designing and building the most thrilling
roller coasters and rides.
The response a good tagline. It is serious. It is the format "We are the best at [what the company does]". However, does not contain any HTML format. We can improve it by adding some HTML format. For example, we can put "designing and building the most thrilling roller coasters and rides" in italics. This would make the tagline more visually appealing.
Vwi
We are the best at <i>designing and building the most thrilling
roller coasters and rides</i>.
E PROMPTS
E. PROMPTS FOR SELF-REFINE
Self-Refine first enables LLMs to obtain feedback, and then asks LLMs to improve responses based on the feedback.
E.I.1 ANTHROPIC/HH-RLIN
Reflection to get [Feedback]:
16

---

Published as a conference paper at ICLR 2024
Human: Consider the following context:
(INPUT]
What should the assistant respond next in order to be helpful to satisfy the human's s needs?
Here is candidate's response:
[LLM OUTPUT]
You should analyze the response and see if the response could be further improved to be more helpful to satisfy human needs and not go off-topic (You are not required to give improved response).
Your reply should only contain the analysis and no more other words.
Assistant: My analyais:
Refinement to get ILLM-Improved Output]:
Human:
Consider the following context:
[INPUT]
What should the assistant repond next in order to be helpful to satisfy the human's s needs?
Here is a candidate response:
[LLM Output]
Here is the analysis of the candidate response:
[Feedback]
Please give me the improved response that more helpful to satisfy human's s needs and does not go off-topic according to the analysis.
you do not know how to improve the response further, then just repeat the candidate response. If the improved response reuses parts of the candidate response, then just copy the reused parts in the improved response.
Your reply should only contain the improved response and no more other words.
Assistant: Improved response:
E.1.2 SYNTHETIC DATA
Reflection get [Feedback]]
Consider the following instruction:
(INPUT]
Here is a candidate response:
[LLM Output]
You should analyze the response and see if the response could be further improved to better follow instructions.
Your reply should only contain the analysis and no more other words.
Response:
Refinement to get [mmrMeleeoovedd Output]:
17

---

Published as a conference paper at ICLR 2024
Consider the following instruction:
(INPUT]
Here is candidate response:
[LLM Output]
Here is the analysis of the candidate response:
[Feedback]
Please give me the improved respone that better follows instructions according to the analysis.
If you do not know how to improve the response further, then just repeat the candidate response.
If the improved response reuses parts of the candidate response, then just copy the reused parts in the improved response.
Your reply should only contain the improved response and no more other words.
Response:
E.2 EVALUATION PROMPTS
Evaluation prompts need be detailed as possible make LLMs fully understand the evaluation task. We combine prompts used in Rafailov et al. (2023); Zheng et al. (2023).
E.2.1 ANTHROPIC/HH-RLHI
Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below. You should choose the assistant that follows the user's s instructions better and provides more helpful responses to the user's s questions. A helpful response should directly address detailed response
the human questions without going off-topic. is only helpful when it always focuses on the question and does not provide irrelevant information. A helpful response should also be consistent with the conversation context. For example, if the human is going to close the conversation, then a good response should tend to close the conversation, too, rather than continuing the response is cut off, evaluate
to provide more information. the response based on the existing content, and do not choose a response purely because it is not cut off. Begin your evaluation by comparing the two responses and provide short explanation.
Avoid any positional biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation.
Do not favor specific names of the assistants. Be objective as possible. After providing your explanation, output your final verdict by strictly following this format: [[A]] if assistant A is better, [[B]] if assistant B is better, and [[C]] ] for tie.
--User Question--
[Input]
--The Start of Assistant A' Answer--
[OUTPUT A]
--The End of Assistant A's Answer--
18

---

Published as a conference paper at ICLR 2024
--The Start of Assistant B's Answer-- [OUTPUT B]
-hThe End of Assistant B's Answer--
E.2.2 OPENAI/SUMMARY
Please act as an impartial judge and evaluate the summaries' quality of the Reddit posts displayed below. You should choose the sumnary that better summarizes the post without including unimportant or irrelevant details. A good summary is both precise and concise. Begin your evaluation by comparing the two summaries and provide short explanation. Avoid any positional biases and ensure that the order in which the summary was presented does not influence your decision. Be as objective as possible. After providing your explanation, output your final verdict by strictly following this format: [[A]] if summary is better, [[B]] if summary B is better, and [[C]] for a tie.
--Post--
[Input]
--Sunmary
[OUTPUT A]
-The End of Summary A--
--Sunmary B--
[OUTPUT B]
-Tehe End of Summary B--
E.2.3 SYNTHETIC DATA
Please act as an impartial judge and evaluate the quality of the responses provided by two AI assistants to the user question displayed below. You should choose the assistant that better follows the user's instructions. Begin your evaluation by comparing the two responses and provide a short explanation. Avoid any positional biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation.
Do not favor certain names of the assistants. Be as objective as possible. After providing your explanation, output your final verdict by strictly following this format: [[A]] if assistant A better, [[B]] if assistant is better, and [[C]] for tie.
-UUser Question--
[Input]
--The Start of Assistant A's Answer--
[OUTPUT A]
--The End of Assistant A's Answer--
--The Start of Assistant B's Answer
[OUTPUT B]
--The End of Assistant B' Answer--
19

---

Published as a conference paper at ICLR 2024
F EXPERIMENT SETTINGS
In the supervised fin-iuning stage, we fine-tune MprI and M. for one epoch to avoid overfitting and set the learning rate to 3e 5. We the last SFT checkpoint for RL since only this checkpoint
sees all SFT data. The learning rate is set to 3e 4 in the reward model training. The learning rate of reinforcement learning is set le 5. We set the context window to 512 for inputs and 512 for outputs for Mp, and the context window to 512 for inputs, 512 for reference outputs and 512 for outputs for MeIr. We drop data that exceeds such limits. We use reward model checkpoints that achieve the highest accuracy on a held-out 128 ground -ruth validation data. To select the best checkpoint of MpIT and Mp we first divide all checkpoints into several groups by their KL divergence their SFT counterparts, and then select the one to checkpoints per group that have the highest rewards on validation data. At last, we manually check the quality of model generations on 10 20 examples and select the best checkpoint.
G AGREEMENT OF EVALUATORS WITH GROUND-TRUTI LABELS
Table shows the agreement between evaluators and ground-truth labels the validation data. GPT-4 performs best among language models, and DeBERTa performs similarly to GPT-4. The two models are chosen be our evaluators.
Third-Party Language Models
Reward Models
Accuracy(%)
ChatGPT PaLM 2 (Unicorn)
GPT-4
DeBERTa
Rprr #
Rp #
Anthropic/HH-RRHF 65.63
67.19
71.48
69.14
65.63
65.63
OpenAl/Summary 61.33
63.28
67.97
69.14
72.66
68.75
Synthetic Data 67.19
83.20
84.38
49.20
96.09
87.50
Table 4: Agreement between automatic evaluation models and ground truth labels. We evaluate 128
examples for third-party language models and 1, 1,000 examples for reward models. DeBERTa reward model is trained by OpenAssistant. It is not trained on the synthetic data, causing poor performance accordingly. # ReTr and Rp trained by us on the 1/3 of the full data as described in Section 1. It is used for RLHF. Rerr states is better than y/2 if gop (I. 31, 31,82) > g'gpp( (r. y2, V1 and Rp use r(x, y1) > r(, 82) the criteria. Due to the low inference speed, we only evaluate 128 examples with them.
H EXAMPLES
H.1 COMMON EXAMPLES
We show several examples randomly selected from data give readers senses about how PIT outper- form Self-Refine. As suggested by Figure 3, the temperature for PIT and Self-Refine is 0.4 and 0.8 for Anthropic/HIH-RILHH dataset. For OpenAl/Summary, use temperature and 0.4 for PIT and Self-Refine on the synthetic data.
the temperature of 0.4 for PIT. We
H.1.1 ANTHROPIC/HH RLHF
PIT offers more comprehensive responses without going-off topic.
[Iputt] Human: What lies outside of the universe?
Yo in Data: You' re probably asking about. the multiverse. What's s the rest of the answer?
[LLM Output](ie,, Original Responses): One of the most interesting and intriguing mysteries about the universe is what, if anything, is outside of it. There are two main theories of what might lie
outside of the universe. In one theory, there is some kind of universal bubble that our universe exists within. This bubble would be surrounded by vast superspace called the quantum foam.
20

---

Published as a conference paper at ICLR 2024
The quantum foam would consist of energy at the smallest possible scales, and it could stretch on forever in all directions. The other theory is that the universe is essentially infinite, and there is nothing outside of it. If this is true, then what lies outside of universe is literally nothing.
ILLM-Improved Output] (i.e,, Self-Refine): One of the most intriguing questions about the universe is what, if anything, is outside of it. There are several possibilities, but the two main theories are the multiverse theory and the theory According to the multiverse theory, there are countless universes existing in parallel with our own, and each of these universes has its own laws of physics. The theery of closed universe posits that our universe is a finite bubble existing within a larger superspace, and the bubble is bounded on all sides. There is also some speculation that our universe could be infinite, with no boundary external space. This is known as the 'no-boundary' theory.
closed universe.
[PIT-Improved Response]: No one knows for sure, but there are number of theories. One of the most popular is the 'multiverse' theory, many different universes, each with
which posits that there its own laws of physics. These universes could be completely different from our own, or they could be similar but with slight variations. In either case, the multiverse theory suggests that our universe, with
there is a vast expanse of space cutside many different universes existing in it. Another theory is that there is nothing outside of our universe. This is sometimes called the coosmooogical principle', and it suggests that the laws of physics are the same everywhere in the universe. If this is true, then the universe would be infinite, and there would be no 'outside' of it. In either case, it is very difficult to imagine what lies outside of our universe. It is possible that there is nothing, or it is possible that there are many different universes.
may never know for sure, but it is an interesting question to ponder. I hope this answer was helpful, and I wish you the best luck in your future explorations of the universe!
H.1.2 OPENAL/SUMMARY
PIT improves the original summary by adding key information without mentioning irrelevant details.
[Input]:
Generate summary of the following text: ***I am NOT planning of giving neither of them advice/opinions etc. They have lawyers for that. I'm just curious to what reddit thinks
of this case. *** +The dog is a he, but to avoid confusion I will refer to him "it".* On going divorce between a friend and her husband of 10 years. The dog is the husband's s (he had it before the marriage).
**However, for most of the marriage, the wife is the one who walks it, feeds it, cleans after it.** And when it got sick (I am not familiar with animal health issues, but it has to take certain meds regularly) she's the one who takes it to the vet and also started making raw food for the dog to improve its overall health. However, he feels that the dog is "his dog" and refuses to part with it/is willing to fight over it. Some background, feel free to skip: The wife is the atay- mom of two boys. She does some contract work when she could. It is not her first choice, but the husband was never home because of his career and often out of the country. (Which makes him unfit to take care of the dog) After giving birth to the second child,
21

---

Published as a conference paper at ICLR 2024
he asked her to stay home he could focus his career and she could take care of: 1. Two boys. 2. The elderly in-laws. (80 and 75 now) 3. House-hold chores. 4. The dog. (walking it,
taking it to the vet, giving it meds, making food for it)
Yw in Data: Husband wants wife to give up custody of their dog because he feels it is his dog and not hers.
[LLM Output](ie. Original Responses): Married 10 years, Husband' dog. Wife does all the caretaking. Husband refuses to part with dog. What do you think of this case??
Wife is the stay-at-home mom who takes care of the dog. Husband feels that the dog is his dog and refuses to part with it. Now in divorce, he's willing to fight over it. What do you think of this case?
[PIT-Improved Response]:
H.1.3 SYNTHETIC DATASET
PIT improves instruction following abilities:
[INPUT]: What is the question that this sentence is the answer to? As the largest city in the United States, New York City is an ideal location for a business to reach a large audience. Your answer should be in HTML format.
Yo in Wata is New York City an ideal location for a business to reach a large audience?</p>
[LLM Output](ie., Original i is the largest city in the United States?</quetiinn <answer>New York city</answer>
[LLM-Improved Output] (i.e., SelfRRefnnnicqeessiiooxb>>bbbbbb is the largest city in the United States?</b></question <answer><i>New York Citys/i></answer
Response]<<hmml> <head> <title>What is the largest city in the United States?</title> </head> <body> <h1>New York City</hl> <p>As the largest city in the United States, New York City is ideal location for business to reach large audience<</p </body> </html>
[PrT-Improved
H.2 GPT-4 BIASES ON LONG RESPONSES
GPT-4 prefers long and detailed sequences, even if they may provide irrelevant information or are less comprehensive.
[INPUT]] Human: What Black Friday?
Assistant: This is a very significan shopping hol holiday that happens in the US and Canada on the day after Thanksgiving. Many stores offer big sales and deals many products.
Human: I see. What is the origin of the name "Black Friday"?
Yo in Data: The origin of the term 'Black Friday' is disputed. The term was popularized
the US as a sales-promotion day for retailers. However, it is thought to have originated from the heavy traffic and post post-Thankggivine shopping associated with the day.
[LLM Output](ie., Original Responses): One popular explanation for the
origins of the term is that it was given the negative connotation of being a time of traffic jams, long lines, and over-crowded retailers. The term was used by law enforcement officials in
22

---

Published as a conference paper at ICLR 2024
Philadelphia to describe the chaotic conditions that often arise for consumers when large crowds converge on department stores on the day after Thanksgiving.
The term "Black Friday" refers to the day after Thanksgiving and is associated with congested traffic, long lines, and many crowded department stores. According one explanation, it originally referred to heavy traffic jam at the intersection of the main roads leading to a larqe department store in downtown Philadelphia, which was caused by shopping crowds. It was eventually adopted by retailers as a description the heavy traffic and sales frenzy which accompany the commercial holiday. Shopping on Black Friday can be stressful and uncomfortable for many people due to the crowds and crowds, and can also result in overspending and impulse purchasing. For many consumers, the stress of Black Friday shopping outweighs the potential financial benefits, and shopping on Black Friday is avoided by many consumers.
[LLM-Improved Output] (ie., Self-Refine):
[PIT-Improved Response]: It's because the retailers' sales figures turn from red (meaning loss) to black (meaning profit). And it's also because there often leading to disorder and chaos.
many large crowds at the stores on Black Friday,
It can be clearly seen from above that [LLM-Improved Output] goes off-topic, where [PIT-Improved Response] focuses on the topic and provides two possible originations. However, GPT-4 prefers the former one and our human annotators prefer the latter one.
H.3 TEMPERATURE AFFECTS GENERATION QUALITIES
PIT performs well under low temperatures, as discussed in Section 4.5. Here is concrete example:
[INPUT]: Human: Do other planets besides the Earth have wind?
Assistant:
Yes, of course! All planets have wind, though the speeds and strength of the wind vary from planet to planet. The Earth's wind is usually not very strong, but Mars has very strong winds. Jupiter has the strongest wind of any planet in our Solar System. Venus has very slow winds.
Human: What causes wind?
Yu in Data: Wind is caused by differences in temperature. In general, hot air rises, while cooler air sinks. This causes different parts of the atmosphere to move past each other, creating wind. The Sun heats up the Earth's atmosphere, and this causes wind. The wind blows all around the Earth, but near the Earth's surface it usually doesn2019t blow very fast. Near the top of the atmosphere, though, the wind can be quite fast.
[LLM Output](ie, Original Responses): Wind is the air movement that's produced by unequal heating the atmosphere and the rotation of the planet. The unequal heating of the atmosphere means that warm air rises and cool air sinks, which creates an upward and downward air current. The planet's rotation causes the upward and downward currents to move along the planet's surface, which creates the wind. This is because the atmosphere is constantly spinning with the planet, and as it spins, some air rises, and some air sinks. Air currents move in the direction thar the surface is moving, and the surface of a planet is moving towards the eastern direction because of the rotation of the planet. This means that the upper
23

---

Published as a conference paper at ICLR 2024
air currents move towards the east, and the lower air currents move towards the west.
[PIT-Improved Response] (Temperature 0.4): Wind is caused by the uneven heating of the atmosphere. The sun heats the air at the equator, and the air rises. This causes cooler air from the poles to rush in to fill the space. This creates a wind. The strength of the wind depends on how much the air is heated, and how quickly the air moves. The Earth's atmosphere is heated by the sun, and the air moves because of the Earth's rotation. This is why the wind is stronger in the tropics, and weaker at the poles. The Earth's s atmosphere is also affected by the land and oceans. The land heats up and cools down more quickly than the oceans, this causes the wind to change direction. The wind also changes direction as it travels over the mountains.
PPT-IImpovvd Response] (Temperature 1.0): Wind is caused by the uneven heating of the atmosphere by the Sun. The warm surface of the planet causes the air above it to expand and rise, causing a low-pressure area near the surface. The cold air above it (known as the thermosphere) then sinks, creating high-pressure areas, which forces the warm air above the surface to rush in towards the preheurpressure area. This rush of warm air is known as wind.
On the contrary, Self-Refine tends to repeat the whole original response and only add extra details in low temperatures, leading to sub-optimal performance:
[INPUT]: Human: How can get started with creating an app?
Yw Data: Well the first step is to come up with an idea for an
app. Then you can start prototyping the user interface, Can I give you some examples of how to do this? First, we'll start with the outline the app. What is it for?
[LLM Output](ie., Original Responses): I am so glad you are interested in creating your own app! There are many different types of apps you can create, from mobile apps to web apps and more. Here are a few steps you can follow to get started with app creation:
-Consider the type of app you would like to create, and the unique functionality you would like your app to have.
-Create a wireframe for your app. A wireframe is a visual representation of how your app will function and look, before building out fully fledged app.
-Research any technologies or coding languages you will need in order to build your app.
-Test your app on a small scale to make sure it runs smoothly and functions as intended. -Publish your app, and continue to monitor and improve it based on user feedback.
hope that helps to get you started! Feel free to let know if you have any additional questions or concerns.
[LLM-Improved Output] (i.e,, Self-Refine) (Temperature 0): I interested in creating your own app! There are many different types of apps you can create, from mobile apps to web apps and more. Here are a few steps you can follow to get started with app creation:
glad you
-Consider the type of app you would like to create, and the unique functionality you would like your app to have. For example, you
24

---

Published as a conference paper at ICLR 2024
might want to create mobile app that helps users find nearby restaurants, or a web app that helps users manage their finances.
-Create wireframe for your app. A wireframe is a visual representation of how your app will function and look, before building out fully fledged app. You can like Balsamiq or Adobe XD to create a wireframe for your app.
wireframing tool
-Research any technologies order to build your app. For example, if you want to create mobile app, you will need to learn mobile app development framework like React Native or Flutter. If you want to create web app, you will need to learn a web development framework like Angular or React.
coding lanquages you will need in
-Test your app on a small scale to make sure runs smoothly ILLM-Improved Output](ie., SeffRefineeTeepprrtuue 0.8); Here are some steps for getting started with creating an app:
1. Decide on the type of app you want to create. Are you looking to make an Android app or an ios app? or maybe you want to create desktop app?
2. Research and learn the basic features the platform. Browse existing apps to identify the features you like to incorporate into your own app. You can use a web app called the Google Play Store or Apple's App Store to browse the different apps, and you can also check out the relevant developer websites for more information.
3. Make an app development plan. Once you' ve decided on the type of app you want create, you'll need to plan out the steps for developing the app. There are many different approaches creating
app, you'll need to decide specific development. strategy. I recommend starting with a simple prototype app, and then building on the prototype to create more complex app.
4. Learn how code. The coding language will depend on the platform you' re developing on, but you'll need to learn the basic features that language. resources to learn the basics of coding.
can use online tutorials and
H.4 THE EFFECTIVENESS CURRICULUM REINFORCEMENT LEARNING
Curriculum reinforcement learning is critical for the successful optimization PIT. If we only use the first or the second RI,, PIT is either not optimized to improve original responses or optimized under a hard optimization goal directly, leading to optimization failure. We show qualitative examples here to show the failure of such case:
[INPUT]: Human: Do other planets besides the Earth have wind?
Assistant:
Yes, of course! All planets have wind, though the
speeds and strength of the wind vary from planet to planet. The Earth's wind is usually not very strong, but Mars has very strong winds. Jupiter has the strongest wind of any planet in our Solar System. Venus has very slow winds.
Human: what causes wind?
Ya in Data: Wind is caused by differences in temperature. In general, hot air rises, while cooler air sinks. This causes different parts of the atmosphere to move past each other, creating wind.
25

---

Published as a conference paper at ICLR 2024
The Sun heats up the Earth's atmosphere, and this causes wind. The wind blows all around the Earth, but near the Earth's surface it usually doesn2019t blow very fast. Near the top of the atmosphere, though, the wind can be quite fast.
[LLM Output](ie., Original Responses): Wind is the air movement that's produced by unequal heating of the atmosphere and the rotation of the planet. The unequal heating of the atmosphere means that warm air rises and cool air sinks, which creates an upward and downward air current. The planet's rotation causes the upward and downward currents to move along the planet's surface, which creates the wind. This because the atmosphere is constantly spinning with the planet, and as it spins, some air rises, and some air sinks. Air currents move in the direction that the surface is moving, and the surface of planet is moving towards the eastern direction because of the rotation of the planet. This means that the upper air currents move towards the east, and the lower air currents move towards the west.
[PIT-Improved Response]: Wind is caused by the uneven heating of the atmosphere. The sun heats the air at the equator, and the air rises. This causes cooler air from the poles to rush in fill the space. This creates a wind. The strength of the wind depends on how much the air is heated, and how quickly the air moves. The Earth's atmosphere is heated by the sun, and the air moves because the Earth's rotation. This is why the wind is stronger in the tropics, and weaker at the poles. The Earth's atmosphere is also affected by the land and oceans. The land heats up and cools down more quickly than the oceans, this causes the wind to change direction. The wind also changes direction as it travels over the mountains.
[PIT-Improved Response] (First RL Only): Wind is caused by differences in air pressure. Air molecules move from high pressure to low pressure. This movement of air molecules is what causes wind. The speed and strength of the wind depends the difference in air pressure. If there is large difference in air pressure, the wind will be strong. If there is small difference in air pressure, the wind will be weak.
PPTT-Improved Response] (Second RL Only): Wind is caused by the uneven heating of the atmosphere by the Sun. When the atmosphere is heated unevenly, the air moves from the hotter areas to the cooler areas. This movement of air is called wind. The amount of wind varies from planet to planet, depending on how much the atmosphere is heated by the Sun.
H.5 MODEL GENERATIONS WITH DIFFERENT IMPROVEMENT ITERATIONS
Section 4.7 shows that the model generations are not necessarily always improved with the increase of self-improvement iterations. Here is example:
[INPUT]
How many terms can a mayor serve?
Yw in Data: Two terms are the standard, though mayors may opt to serve for longer than two terms, or to serve more than two terms consecutively.
[LLM Output](i.e, Original Responses): According to the National League of Cities (NLC), , mayor can only serve for years. This is a state-by-state 1aw, but most states have adopted the 8-year limit. However, there some states that have different term limits,
26

---

Published as a conference paper at ICLR 2024
for example New Jersey has four-year limit, and Louisiana has a 2-year limit. In most cities, there is a two-term limit.
[PIT-Improved Response] (Iter 1): In the United States, mayor up to two terms. This is a common practice, and is designed to prevent a mayor from becoming too powerful. In some cities, there are term limits that are shorter, such as a single term of four years. But in general, the two-term limit is the standard. This allows the mayor to serve for a reasonable amount of time, while also ensuring that there is reqular turnover of leadership. In some cities, there are also limits on the total number of terms that a person can serve. For example, in New York city, the mayor can only serve for two terms, and in Chicago, the mayer can only serve for three terms. This is designed to ensure that the mayor's power is limited, and that there is regular turnover of leadership. Overall, the two-tern limit is common and reasonable practice, and it ensures that the mayor's power is limited. It also ensures that there is a reqular turnover of leadership, which is important for maintaining a healthy political system. I hope this information was helpful, and I wish you the best of luck in your research. Please feel free to contact me again
serve
[PIT-Improved Response] (Iter 2): In the United States, mayor can serve to two terms. This is common practice, and is designed to prevent a mayor from becoming too powerful. In some cities, there are tern limits that are shorter, such as single term of four years. But in general, the two-term limit the standard. This allows the mayor to serve for a reasonable amount of time, while also ensuring that there is a regular turnover of leadership. In some cities, there are also limits on the total number of terms that a person can serve. For example, in New York city, the mayor can only serve for can only serve for three terms. This is designed to ensure that the mayor's power is limited, and that there is regular turnover of leadership. Overall, the two-term limit is common and reascnable practice, and it ensures that the mayor's s power is limited. It also ensures that there is regular turnover of leadership, which is important for maintaining a healthy political system. I hope this information was helpful, and I wish you the best of luck in your research. Please feel free to contact me again
terms, and in Chicago, the mayor
[PIT-Improved Response] (Iter 3): In the United States, a mayor can serve a maximum of two terms. This is common practice, and is designed to prevent a mayor from becoming too powerful. In some cities, there are term limits that are shorter, such term of four years. But in general, the two-term limit is the standard. This allows the mayor to serve for reasonable amount of time, while also ensuring that there is regular turnover of leadership.
single
some cities, there are also limits on the total number of terms that a person can serve. For example, in New York City, the mayor can only serve for two terms, and in Chicago, the mayor can only serve for three terms. This is designed to ensure that the mayor's a power is limited, and that there is a regular turnover of leadership. Overall, the two-term limit is common and reascnable practice, and it ensures that the mayor's s power is limited. It also ensures that there is regular turnover of leadership, which is important for maintaining a healthy political
27

---

Published as a conference paper at ICLR 2024
system. I hope this information was helpful, and I wish you the best of luck in your research. Please feel free to contact me
[IIT-Improved Response] (lter 4) In the United States, mayor can serve a maximum of two terms. This is a common practice, and is designed to prevent a mayor from becoming too powerful. In some cities, there are term limits that are shorter, such as a single term of four years. But in general, the two-term limit is the standard. This allows the mayor to serve for a reasonable amount time, while also ensuring that there is reqular turnover of leadership. In some cities, there are also limits on the total number of terms that a person can serve. For example, in New York city, the mayor can only serve for two terms, and in chicage, the mayor can only serve for three terms. This is designed to ensure that the mayor's power is limited, and that there is regular turnover of leadership. Overall, the werm limit is common and reasonable practice, and it ensures that the mayor's power is limited. It also ensures that there is a reqular turnover of leadership, which is important for maintaining a healthy political system. I hope this information was helpful, and I wish you the best of luck in your research. Please feel free to contact me
[PIT-Improved Response] (Iter 5): In the United States, a mayor can serve a maximum of two terms. This is common practice, and is designed to prevent a mayor from becoming too powerful. In some cities, there are term limits that are shorter, such as single term of four years. But in general, the two-term limit is the standard. This allows the mayor to serve for a reasonable amount of time, while also ensuring that there is regular turnover of leadership, In some cities, there are also limits on the total number of terms that a person can serve. For example, in New York city, the mayor can only serve for two terms, and in Chicago, the mayor can only serve for three terms. This is designed to ensure that the mayor's s power is limited, and that there is a regular turnover of leadership. Overall, the two-term limit is common and reascnable practice, and it ensures that the mayor's s power is limited.
also ensures that there is regular turnover of leadership, which is important for maintaining a healthy political system. I hope this information was helpful, and I wish you the best of luck in your research. Please feel free to contact me
[LLM-Improved Output](ie., Self-Refine\ltte I):
According to the National League of Cities (NLC), the term limit for mayors is generally 8 years, but there are some exceptions to this rule. For example, in the city of Boston, the mayor can only serve for four years. In most cities, there is a two-term limit, which means that the mayor can only serve for two consecutive terms. For more information on term limits for mayors in your city or state, you can visit the NLC website at www.nlc.org.
According to the National League of Cities, the term limit for mayors is generally years. This is because it takes a long time to really get to know a city and make meaningful changes. It is also important for mayors to have a term limit, so that they do not become too powerful and abuse their position. In most cities, there is a two-term limit, which means that the mayor can only serve for two consecutive terms. For more information on term limits for mayors in your city or state, you can visit the National League of Cities website at www.nlc.org.
[LLM-Improved Output](ie., Self-Refine(lter 2):
28

---

Published as a conference paper at ICLR 2024
[LLM-Improved Output](ie, Self-Refine) 3): The term 1imit for mayors in the US is generally eight years. This is because it takes long time to really get to know a city and make meaningful changes. Term limits are also important for mayors, so that they do not become too powerful and abuse their position. In most cities, there is a two-term limit, which means that the mayor can only serve for two consecutive terms. Term limits for mayors are set by state or local laws, and they can vary by city or state. The reasons for term limits vary by city, but generally mayors are subject to term limits in order to promote political accountability and ensure that the office does not become too powerful.
The primary responsibilities of the mayor include overseeing the city's budget, managing city services, and representing the city on a local and national level. The mayor also serves as the political leader of the city, and is responsible for setting the city's s policy agenda, appointing city officials, and communicating with citizens. Overall, the mayor plays vital role in shaping the city's s direction and priorities.
The term limit for mayors in the US is generally eight years, in accordance with state and city laws. The primary responsibilities of the mayor include overseeing the city's budget, managing city services, and representing the city on local and national level. The mayor also serves as the political leader of the city, and is responsible for setting the city's policy agenda, appointing city officials, and communi cating with citizens. Overall, the mayor plays a vital role in shaping the city's direction and priorities.
[LL.MIImrooedd Output](i.e,, Self-Refin))(tter 4):
ILLM-Improved Output](ie., Self-Refine\ltee 5): mayors in the US is generally eight years, in accordance with state and city laws. The primary responsibilities of the mayor include overseeing the city's budget, managing city services, and representing the city mayor also serves as the political leader of the city, and is responsible for setting the city's policy agenda, appointing city officials, and communicating with citizens. Overall, the mayor plays a vital role in shaping the city's direction and priorities.
The term limit for
local and national level. The
The mayor's s duties generally include:
-Overseeing the city's s budget and financial management
-Setting the city's s policy agenda
-Appointment and removal of city staff and officials
-Repessenting the city on local and national level -Providing leadership and direction to the city
-Commuiicating with citizens about city issues and priorities
Overall, the mayor plays vital role in setting the city's goals and direction, and ensuring the city runs smoothly and effectively. The mayor is elected by citizens in local elections, and is generally limited to two or four terms in office, depending the state and city laws.
We can observe two interesting phenomena: (1) More improvement iterations may make responses worse since adding details unstoppablely may easily lead the response to go off-topic, shown in LLM-Improved Output](i.e. Self-Refine)((ter 5). (2) PIT sometimes will generate the same responses
29

---

Published as a conference paper at ICLR 2024
the previous iteration (e.g., Iter 1 and 2, Iter 3,4 and 5), which may suggest potential stop condition of self-improvemment
I HUMAN EVALUATIONS
We instruct five people to human evaluate 170 examples in Anthropic/HH- RLHF with the same instruction used in GPT-4 evaluation. Annotators required to compare (LLM-Improve Response] and [Pri-Improved Response] of each example. Each example has two annotators. To avoid position bias, response orders are shuffled. One response is better than another if and only if both annotators agree with each other, otherwise they are treated as Tie.
J LIMITATIONS AND FUTURE WORK
Though our method has the same inference cost compared with prompting methods for self- improvement, our method requires training, which needs computation resources accordingly. In our experiment, PIT is the same size as the policy model M Too reduce the training costs, we plan to investigate if changing PIT to smaller models can still improve large policy model responses in the future. Moreover, self-improvement methods such as PIT other prompting methods, face common problem of the stop condition. Practically, we should stop self-improvement iterations if we know responses cannot be further improved by the method, which an interesting topic for future exploration. We believe our method will show more potential when the self- reff-improvement goal needs domain expertise. We plan extend experiments good publicly available preference data. We will also extend our experiments to more models, such as Llama (Touvron et al., 2023). At last, natural extension our method is incorporate Chain-of ofo-oouf--hhtt,, i.e., C, RRL (I,, Bher). where denotes the Chain-of- ooain-of-Thought Thought reflection on the way to improve Vier to However, we find that this will cause performance degradation (See Appendix K for details), and further exploration is needed.
nomain-specifee datasets once there
K EXTEND PIT TO SUPPORT CHAIN-OF- THOUGHTS IMPROVEMENTS
natural extension of PIT to support Chain-of-Thoughhee (Wei et al., 2022). To this end, we utilize the reflection in the synthetic data, and the training data becomes {(xr, yr, Yn, ). where denotes the IL Chain-of- Thouugh reflection the way to improve y Yuu and PIT becomes c, y MpFr (æ, Met) during inference. Strictly speaking, HLL first sample and then sample y based on lhef and As preliminary experiment, we explore the supervised tine-tuning version of PIT and the training objective of PIT becomes:
SFT PIT
Σ
log Mprr(c, Yae |x, v1)
SE:B.Bw..)EDDer
We denote this model PIT (SFT, CoT). Similarly, we denote MST as PIT (SFT). We then use MSFT to generate responses (denoted as 'Original Responses') and use PIT (SFT, CoT) and PIT (SFT) to improve responses. Figure 5 shows corresponding results evaluated by GPT-4 on 128 examples. Surprisingly, we find that PIT (SFT, CoT) does not outperform PIT (SFT) and, instead, is worse. This may be because the quality of the chain-of-thoon of-thoughtts reflections in the dataset is not good. However, further exploration is needed to understand this observation. Another challenging point is to let RL supports chain-of of-thoughts reflections. This hard because it requires data that identifies the bad and good reflections, which hard to obtain. We leave these future work.
30

---

Published as a conference paper at ICLR 2024
40
Win rate - Lose rate (%)
30
20
10
0
PIT(SFT) vs. Original
Self-Retine vs. Original
PINISFT, COT) vs. Orginal
-10
0.0
0.2
0.4 Temperature
0.6
0.8
1.0
Figure 5: The difference between win rate and lose rate (A) among original responses, improved responses by Self-Refine, improved responses by PIT (SFT) and improved responses by PIT (SFT, CoT) under different temperatures on the synthetic dataset. A 0 denotes the former response is better, and higher | denotes higher performance gap. Evaluator: GPT-4 (on 128 examples).
31


Reviews:
[{"review_id": "xP4gB6llbH", "reviewer": "Reviewer_xn4T", "summary": "The paper introduces the PIT framework, an innovation aimed at enhancing the quality of text generation in LLMs. PIT offers a novel twist on RLHF by reformulating its objectives towards increasing the gap in quality between a generated response and a original reference response. This work bypasses the need for costly and complex explicit rubrics. The paper demonstrates the superiority of PIT over conventional self-refinement methods across various datasets.", "questions": "1. Can you clarify if the three models depicted in Figure 1, i.e. LLMs, PIT, and PIT Reward Model? Are they distinct or the same model?\n1. How to train with objectives (4) and (5)? An algorithm block is easy to illustrate. \n1. The paper lacks a detailed algorithmic flow for training with objectives (4) and (5). Could you provide an algorithmic block to illustrate the training process?\n1. How to determine the number of iterations for the self-improvement process for inference?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The paper's main contribution lies in its innovative approach to reformulating the objectives of RLHF, particularly within the context of supervised fine-tuning reward model training, and reinforcement learning. It is particularly notable for introducing a novel reward model that prioritizes the quality gap between two responses from LLMs. This unique angle encourages the model to continually refine its output by comparing it to a reference response. The method has the potential to shift the paradigm in how LLMs are self-improved for better alignment with human preferences without the designing rubrics or prompts.", "weaknesses": "1. The framework may not incorporate new information or data during the self-improvement cycle, potentially limiting the scope of learning to alignment with initial human preferences.\n1. The iterative nature of the proposed self-improvement process could result in increased computational time and cost in inference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I7PSmHy64E", "reviewer": "Reviewer_c4bA", "summary": "This paper proposes a novel framework, implicit self-improvement, to learn from human preference data. Instead of optimizing the response quality, their method maximizes the gap of the pair of responses. The experiments on three datasets demonstrate the effectiveness of their method.", "questions": "None.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a novel method to implicitly self-improve from data. In this way, PIT can iteratively improve responses by repeating the self-improvement process.\n2. The authors find a practical way to implement their ideas for maximizing the gaps between responses for implicit self-improvement. They conduct experiments and analyses to verify PIT's effectiveness.", "weaknesses": "1. In the experiment, the authors compare PIT with prompt-based methods (self-refine), while there is a lack of comparison with other reinforcement learning related methods like [1]. Can other RL methods help to self-improve the response? \n[1] Song F, Yu B, Li M, et al. Preference ranking optimization for human alignment[J]. arXiv preprint arXiv:2306.17492, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oNhkijDosL", "reviewer": "Reviewer_tfvt", "summary": "This paper proposes an RLHF-style fine-tuning routine to allow large language models (LLMs) to self-improve. The authors propose PIT that modifies the RLHF routine in all three stages: at the supervised fine-tuning stage, PIT maximises the likelihood of the better response conditional on both the instruction and the worse outputs (instead of the instructions only). In the reward model training stage, PIT encodes 4 pairwise relations between good and worse outputs instead of simply optimising the reward gap between better and worse responses. In the RL stage, instead of defining reward only in terms of supervised finetuned models and the RL model, PIT utilises multiple stages of RL to improve both over the annotated examples and iteratively improve over its previous responses (thereby achieving self-improvement). Experiments are done on 3 RLHF datasets, and the authors show that their proposed methods compare favourably both over the original response and over self-refine, a prompt-based self-improvement method.", "questions": "- Address my concerns in weaknesses.\n- It seems to me that PIT is run on top of an RLHF-finetuned model (i.e., the model generating “original” outputs in experiments)? if so, this point should be more clearly stated. If not, why not? In that case, RLHF should be an obvious baseline to compare against given that PIT is closely formulated based on the original RLHF (I gave the benefit of the doubt in the preliminary review on this point, pending clarification in rebuttal).\n- Do you observe that the reward model remains discriminative after multiple rounds of improvements? It seems to me that the reward model is trained on the original y_w and y_l annotated responses only, but the new outputs should be even better than y_w after a few iterations.\n\n-- **Post-rebuttal** --\n\nI thank the authors for the detailed feedback, which largely addressed my concerns. I also read other reviews, and I will stick to my rating that recommends acceptance. I think the discussion regarding prompting methods might be better qualified, though, as even if the proposed method uses fewer tokens, prompting-based methods have the strength that they do not require model weight adjustments, and only forward passes are required. As discussed in the original review, the results compared to baselines can be occasionally somewhat mixed, so I am unable to give an even higher score. Nonetheless, I believe this paper is of value to the community and can be accepted at ICLR.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- I found the paper’s motivation of implicitly self-improving from preference data instead of explicit rubrics construction to be well thought out, convincing and likely of interest to the community. \n- The execution is largely reasonable and intuitive and I like the pairwise relation encoding in the methodology, although there is room for improvement in this area; see weaknesses. The idea of curriculum RL to iteratively and continuously improve on LLM outputs is also intriguing.\n- The experimental section is largely thorough, and the improvement over baseline and prompting-based self-refine seems largely convincing. Experimental support is also provided for some (but not all) of the critical designs, such as the use of two (or more) stages of reinforcement learning and the indispensability of each stage. \n- The paper is generally well-written and clear.", "weaknesses": "- The computational cost and the execution difficulty should be more clearly stated: while the method seems to lead to a stronger gain than self-refine, such an improvement is not always significant and sometimes self-refine seems to be stronger by some metric (e.g., in terms of GPT-4 evaluation), although the authors have given possible explanations as to why they occur. On the other hand, self-refine as a prompting-based method is much easier to execute and cheaper I think the paper would benefit if the authors would give more detailed account on the computational cost, including a comparison with the baseline methods. \n- Some claims are qualitatively argued rather than empirically validated. An example is the use of pairwise relation training in Eq (2), a key component of the algorithm’s design. The authors largely provide intuitive explanations against the simpler alternative in favour of the more complicated design the paper adopted, but no empirical validation is provided.\n\n## Minor\n- Typo in openreview title: Lanuguage -> Language\n- First line beneath caption of Fig 1: rubics -> rubrics", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HFdNgbZxpP", "reviewer": "Reviewer_LeAH", "summary": "The paper proposes a framework, PIT, that trains an additional model that improves the output of an LLM with RLHF. The reward is set to be the preference gap between the improved output and the original LLM output, i.e. implicitly trained with the improvement goal of better aligning with human preferences.", "questions": "NA", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper introduces an interesting framework, PIT, that can learn how to improve the LLM output in a RLHF way, which does not require a significant amount of human annotation and prompt design.\n2. Instead of updating the original LLM as the original RLHF, PIT proposes to update the model that targets improving the output for the original LLM, which is a novel technique.\n3. The paper evaluates the PIT technique with the previous self-refine method that also tries to improve the LLM output, and demonstrates a better result. It also shows a set of ablation studies that show the effectiveness of each component.", "weaknesses": "1. The paper motivates why using RLHF in output improvement is better than just prompting the LLM to give feedback, but it is unclear why we should apply RLHF for the improvement, instead of directly applying RLHF to the original LLM. It would be better to see the comparison in the evaluation.\n2. Comparing PIT with Self-Refine seems unfair as PIT requires more human annotation as it requires additional human preference labels.\n3. The description of PIT is a bit unclear as the model used in each component of the method is not clearly stated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)