Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment
Authors: Geyang Guo
Ranchi Zhao
Tianyi Tang
Xin Zhao
Ji-Rong Wen
Venue: ICLR 2024 poster (2024)
Keywords: large language models
alignment
fine-grained SFT
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Alignment with human preference is a desired property of large language models (LLMs). Currently, the main alignment approach is based on reinforcement learning from human feedback (RLHF). Despite the effectiveness of RLHF, it is intricate to implement and train, thus recent studies explore how to develop alternative alignment approaches based on supervised fine-tuning (SFT). A major limitation of SFT is that it essentially does imitation learning, which can't fully understand what are the expected behaviors. To address this issue, we propose an improved alignment approach named $\textbf{FIGA}$. Different from prior methods, we incorporate fine-grained (i.e., token or phrase level) quality signals that are derived by contrasting good and bad responses. Our approach has made two major contributions. Firstly, we curate a refined alignment dataset that pairs initial responses and the corresponding revised ones. Secondly, we devise a new loss function can leverage fine-grained quailty signals to instruct the learning of LLMs for alignment. Extensive experiments have demonstrated the effectiveness of our approaches by comparing a number of competitive baselines.

Published as a conference paper at ICLR 2024
BEYOND IMITATION: LEVERAGING FINE-GRAINED QUALITY SIGNALS FOR ALIGNMENT
Geyang Guo Ranchi Zhao'; Tianyi Tang', Wayne Xin Zhao' Ji-Rong Wen 1.2 "Gaoling School of Artificial Intelligence, Renmin University of China.
School of Information, Renmin University of China.
guogeyangerue edu. en, zanchizhao@gmail com, steventiany tang@ ook. com, batman y gmaalllll com, jrwendruc cn
ABSTRACT
Alignment with human preference is a desired property large language mod- els (LLMs). Currently, the main alignment approach is based on reinforcement learning from human feedback (RLHF). Despite the effectiveness of RLHF, it is intricate to implement and train, thus recent studies explore how to develop al- ternative alignment approaches based on supervised fine-tuning (SFT). A major limitation of SFT is that it essentially does imitation learning, which cannot fully understand what are the expected behaviors. To address this issue, we propose an improved alignment approach named FIGA. Different from prior methods, we incorporate fine-gres grained (i.e., token or phrase level) quality signals that are derived by contrasting good and bad responses. Our approach has made two ma- jor contributions. Firstly, we curate refined alignment dataset that pairs initial responses and the corresponding revised ones. Secondly, we devise new loss function can leverage fine-graine fine-grained quality signals to instruct the learning of LLMs for alignment. Extensive experiments have demonstrated the effectiveness of our approaches by comparing number of competitive baselines. We release all the above-mentioned resources at httts:/gtthbb.com//UAAABB////CC/
INTRODUCTION
Pre-trained large language models (LLMs) such as LLaMA (Touvron et al.) 2023a) have shown remarkable potentials to solve various downstream tasks by mastering the universal pre-training task of next-token prediction. While after large-scale pre-training, often needs subsequent tuning for enhancing and regulating the behaviors of L.MMs. Two typical approaches tuning (SFT) and reinforcement learning from human feedback (RLHF), which can largely improve LLMs both task solving capacity and human alignment (Ouyang et al. 2022).
supervised fine-
Despite widely explored, SFT and RLHF have their own strengths and weaknesses. On the one hand, SFT is easy implement and effectively boost the general task solving abilities by in- struction based eliciting (Wei et al. 2021 Ouyang et al., 2022; Chung et al., (2022). while it mainly imitates the behaviors experts (essentially doing behavior clone (Wiseman Rush (0166). which are demonstrated by the human annotators or powerful I.LMs such as ChatGPT. T'herefore, the SFT performance highly relies on high-quality demonstration data (Zhou et al., 2023), and might suffer from the huge distribution shifts between its outputs and imitated outputs (Zhang al. [2019] Schul- man 2023 Zhao et al. 2023a). On the other hand, RLHF can better explore the semantic space of LLMs, and identify the optimal policy by encouraging good behaviors and discouraging bad behav- iors during learning. However, is very complicated to effectively implement, often suffering from
training instability issues such reward collapse (Song et al. 2023 Wolf al. 2023).
To leverage the benefits of SFT and RLHF, several recent studies propose to develop alignment ap- proaches without reinforcement learning (RL). These studies typically construct refined instruction data using methods such as quantile ranking (Lu et al. 2022) and rejection-sampling (Touvron et al.)
"quual contribution.
Corresponding author.

---

Published as a conference paper at ICLR 2024
2023b), and then follow or slightly modify the original SFT loss. Another line of research designs alternative optimization approaches that bypasses reward modeling (Rafailv et al.) 2023). To con- duct effeettiee alignment without RL, key issue is how to effectively learn by discriminating good and bad behaviors as that in RLHF (Ouyang et al. 2022), such that LLMs can understand what are good behaviors to follow and what are bad behaviors to avoid. Despite the prior efforts, they largely limited by response-level discrimination signals: they are only aware of the quality label (e.g, good or bad) of a demonstration but not what makes it good or bad. Thus, it can't fully capture the correct alignment behaviors even demonstrated by what are good and bad behaviors.
In this work, we introduce FIGA, a novel method that aligns language models with human prefer- ences. The core idea is to contrast a low-quality initial response from LLM's output with a cor- responding high-quality revised response by another powerful LLM (e.g., ChatGPT), so that LLMs can be noted with what are newly added (good actions) and what are removed or substituted (bad actions) from such a revision process. Such fine-grained quality signals can be more useful than the widely used response-level quality signal. It can instruct LLMs to emphasize the learning of good actions and penalize the bad actions in a single response. To implement our approach, we first cu- rate alignment dataset called SPA that pairs initial response with revised response under the guidance of the ground-trumd demonstrations. We mainly keep the queries that a LLM performs less well on, and perform strict filtering. Further, we design new fine-tuning method that assigns spe- cific token-level weights to different parts (e.g,, good or bad tokens). Our learning loss can directly impose fine-grained reward scores to guide the learning of LLMs for improved alignment.
To the best of our knowledge, it is the first attempt that leverages fine-grained quality signals for improving the alignment of LLMs without RL. Our approach can make LLMs better understand what are good and bad behaviors beyond simple imiiation. By conducting extensive experiments, we demonstrate that FIGA shows promising performance in aligning language models with human preferences: our approach outperform the initial snservised-finetune model by notable 3.2 points and the strong PPO method by 1.8 points.
2 RELATED WORK
In this section, we review the related work in the two aspects, namely reinforcement learning from human feedback and alignment without reinforcement learning.
Large-scale pre-training empowers large lan- guage models (L.LMs) to acquire extensive knowledge, underscoring their remarkable potential across diverse tasks (Brown et al. 2020 Kojima er al., 2022 Zhang et al.] 2022 Chowdhery et al.] 2022]. Nonetheless, models exclusively focus on next token prediction in pre-training phrase, while do not consider human preferences. Consequently, this gives rise to unexpected behaviors like harm- ful or inaccurate information, and emphasizes the necessity to align language models with human preferences. The current mainstream approaches (Ouyang et al. 2022) to better harness the capabili- ties LLMS include supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF). To be specific, this involves three stages: firstly, using SFT to enable the model to better follow human instructions; subsequently, training reward model (RM) using human preference data; and ultimately, tune the model to maximize the reward through the proximal policy optimiza- tion (PPO) (Schulman et al. (2177) algorithm. Furthermore, there are works exploring enhancement for this process (Ramamurthy al. 2022 Lightman et al. 2023 Lee et al.) 2023). However, RLHF selecting. Besides, requires load- ing three to four models simultaneously, resulting in high memory usage. These challenges propel researchers to explore alternative approaches to align language models with human feedback.
Reinforcement learning from human feedback
presents challenges due to complex coding and
Alignment without reinforcement learning Several studies are based on the rationale that lan- guage models have already acquired comprehensive knowledge during the pre-training, and only high-quality supervised fine-tuning data is required for further tuning (Zhou et al. 2023). So these works (Liu et | 2023bb Sun et al. 2023 Bai et al.) 2022b Bhardwaj & Poria 2023 Krishna et al.) 2022 Gulcehre et al. (023) bypass reward modeling, and instead concentrate on the construction of datasets that align well with human preferences. Other works are directed towards exploring sub- stitutes for the intricate PPO algorithm. These efforts employ diverse approaches to learn from the preference data, encompassing the creation of a supervised fine-tuning training dataset enriched with
2

---

Published as a conference paper at ICLR 2024
human preference data Liu al. 2023a Zhang al. 2023; Dong et al. 2023), the integration of
preferences for different outputs into the loss function Yuan et al. 2023 Rafailov et al. 2023 Zhao
et al.
2023b Liu et al., (203d]e) c), and the utilization of controllable text generation techniques Lu
et al. 2022). However, the human preference information used in these methods is at the sentence
level, lacking more fine-grained supervision signals.
APPROACH
In this section, we present the proposed alignment approach FIGA by leveraging fine-grained qual- ity signals. Our approach is developed based on a specially curated alignment dataset called SPA (Section 3.1). where each low-quality initial response is paired with high-quality revised response. Based on such an alignment dataset, we further develop new loss function that incorporates fine- grained quality signals derived by contrasting good and bad responses (Section
3.2) Our approach is easy to implement (similar to SFT) and can capture the underlying effect to generate high-quality responses instead of simply imitating them (similar RLHF), which are discussed in Section 3.3 The overall framework of our FIGA pipeline is shown in Figure
Rollout
Reward
Revise
Levenshtein Distance
x
X
What ie the best way te get from Takye
X
way get
Instonces Pool
The best way to oen Tolayo
Y
Cheke by toking the divekkassnn bullet trae With the ballet frzin, you con neach Cueka Takyo jurt over hours.
Y
The Desired words
The best way to get from Tokyy to Cooks by trais jearney frains
best way
trom Tolyyo tokes
y
aneer B lours gest there sesers 19R8 por say
Initial Model
Undestned words
Align the Initial Model
Reward Model
LLM
Figure 1: The overall illustration of our alignment approach FIGA.
3.1 CURATED ALIGNMENT DATASET
From the perspective of dataset, the novelty of our alignment approach can be given in two major aspects. Firstly, we don't directly aggregate all the available instruction data, but instead focus on high-quality instruction data that LLM performs less well It enables LLMs specially improves their weaknesses, reducing the cost of replicate learning. Secondly, we don't 't take what human annotators write powerful LLMMs (e.g., ChatGPT GPT--4) generate training targets, but instead seek more similar surrogate that is derived based on its own output by a LLM. It can largely reduce the distribution shift between the LLM to be aligned and the ground-trutt demonstrations.
We carefully construct the SubPar Alignment (SPA) dataset, a curated collection of query, model's initial response, and the corresponding improved response (with minor revision). Compared with prior work (Ouyang et al. 2022 Yuan et al. 2023 Liu et al. 2023a). we mainly consider the queries where LLMs' performance are not satisfactory and aim correct these bad cases via specific train- ing. Moreover, we refine the initial response of L.M that is to be aligned as training target, which can effectively reduce the distribution shifts from the ground-truth demonstrations.
Formally, we denote the initial model as wo, which can be supervised-finetueed model (e.g., Al- paca (Taoii et al. 2023)) or a pre-trained base model (e.g., LLaMA (Touvron et al. 2023a)). To assume that a reward model for assessing the alignment level is available. In practice, a number of reward models have been released publicly (e.g.. DeBERTa (OpenAssiis)
construct our dataset,
tant 2023)), which can be used for approach. Given query X and response Y, we leverage a reward model RM to compute the reward score Ry = RM(X, Y). which reflects how well the response Y aligns with given query X. Below, detail the construction procedure.
3

---

Published as a conference paper at ICLR 2024
Rollout for initial response generation We first broadly collect existing paired datasets encom- passing a wide range of wealdworld tasks, and construct the instances pool D Y To better align with human value, select preference datasets (e.g.. HH-RLHF (Bai et al. 2022a)
that adhere to the 3H principle ei.e. helpfulness, honesty, and harmlessness) in this work. Further- more, we also include instruction dataset (e.g. OpenOrea (Mukherjee et al. (023)) preserve the task solving abilities of LLMs. We aim to train a both capable and safe model like ChatGPT, rather than only focusing on alignment while sacrificing the task solving abilities. Based on these datasets, we employ the rollout model Ty to generate initial responses Y = (X for the given queries.
Identifying the queries to be enhanced After obtaining the model's initial response Ý and the human-preferrees response Y. we next identify the queries where the model requires further im- provement to better align with human intent through the reward score RM(-). Following existing work (Ouyang et al., 2022). we employ the reward model as a surrogate of human preferences, and design filtering process based on the calculated reward score Ry and Ry for all the instances. We only keep the instances that meet all the three following restrictions: (1) Ry < 1)1 (a subpar initial performance, i.e., bad cases), (2) Ry > 1/2 (high-quality demonstrations), and (3) Ry Ry > Tss (clear quality difference), where 7h. 722, and 17s are three threshold values for filtering. we will set them according the reward score distribution. The details can be found in Section 4.1.2 With the above filtering mechanism, we ensure the quality and usefulness of our SPA dataset. We target at bad case correction of the rollout model, which is more directed and effective than existing methods that directly trains the model on the whole collected dataset.
Revising initial responses for reducing the distribution shifts To align a LLM, a basic principle is to ensure that the distribution of the model should not experience significant shifts during the alignment process (Bai et al. 2022a). Despite that the ground-truth demonstration (Y;) is human preferred, it is likely span very different semantic distribution as the LLM be aligned. Our solution is to revise the initial response (Ý) by referring to the ground-truth demonstration (Y;). In this way, we can effectively reduce the distribution shifts as well as obtaining demonstrations similar to the original output. Specially, we generate pscudo reference Y based the target Y, making minor adjustments to the Y and enhance its quality, i.e. modifying y as minimally as possible based on Y, Such a generation process is conducted by prompting the powerful ChatGPT. To facilitate the generation process, we further manually inspect the low-quality responses that we have previously filtered and identify four major low-quality reasons: (1) lack of detail, (2) inaccuracy in response, (3) the need for structural adjustments, and (4) other factors (off-topic or harmful content). In detail, we leverage ChatGPT to determine, given Y, which of the four reasons Y is associated with. Afterwards, we design different prompts for the four reasons and instruct the LLM to make minor correction to the initial response based on Y.. We denote the revised response as Y. The details of our process and prompts be found in Appendix B
Finally, we obtain the SPA dataset {X, Y) for subsequent training. Our construction method has dual merits: it not only aligns the reference output with human preferences but also preserves the inherent linguistic style and overall semantic distribution of the model to be aligned. Note that we keep both the initial and revised responses in a contrastive form, because they are jointly used for deriving fine-gratn -ine-graieed quality signals in subsequent training.
3.2 FINE-GRAINED QUALITY-AWARD ALIGNMENT TUNING
As described above, our fine-tuning dataset for alignment contains both low-quality initial responses (Y) and high-quality revised responses (Y). Instead of directly learning from these high-quality responses (similar to rejection sampling (Tourron et al. 2023b)), it is important for LLMs to under- stand why such revisions are useful to produce the high-quality responses. Furthermore, LLMs can improve the alignment capacity from the contrast between good and bad responses.
Motivated by previous work (Liu et al. 2022), we utilize Levenshtein distance to quantify the simi- larity between of Ý and Y. Levenshtein distance is dynamic programming algorithm to obtain the minimal edit distance between two sentences through three operations: addition, deletion, and sub- stitution. Comparing the initial and revised response, the involving tokens can be generally divided into three types: newly added, deleted, or substituted. We consider assigning different weights to
4

---

Published as a conference paper at ICLR 2024
these three types of tokens. We reward the tokens that added substituted in the revised sponse Y, penalize the tokens that deleted substituted in the original response Y, and tend to overlook the rest tokens that remain the same after the revision process. Formally, we introduce two token-level weighting functions characterize the above ideas:
F(Tr, 1) ax if is added substituted 7: otherwise
{ [, if De is deleted or substituted
fiee, ()
0, otherwise
where 0, O, and y 2 0 three coefficients to control the encouraged, discouraged, and ignored parts, which can be empirically set or learned from tuning data.
In this way, then encourage the model "imitate" the desired actions that have greater impact on enhancing quality, discourage the model from emulating the undesired actions that lead poor performance in quality. The final training loss can be formulated as:
L=
Σ F(Ur,t) log To (Welice X) JEYY
Σ t( No (0 (ct. B.EP
increase the probubility of desired wards
decrease the probability undesired words
I The major advantages of FIGA over typical SFT (Ouyang et al. 2022) is that can learn from emneggrinee contrast between good and bad responses, which is essentially similar to that in reinforcement learning (discussed in Section 3.3). In addition, by explicitly modeling the revision effect, such an approach can naturally zoom into crucial words or phrase, making the model better zoom into fine-graine semantics.
The overall FIGA pipeline is illustrated in Algorithm]
Algorithm 1: FIGA Leveraging Fine-grained Quality Signals for Alignment
Input: Instance pool = {X, y) initial model To, revision model (ChatGPT), reward function R(--- ### SPA Dataset Construction for each instance {X, Y) in 'D do
1. Rollout for initial generation. Generate Y ~ (X) and compute Ry, Ry:
2. Reward filtering. if Rg > Yn Ry < 1/2 or Ry Ry < then
L Discard the current instance;
Response Revision. Analyze the reason for the poor performance of y, and generate the corresponding revision Y ~ LLM(L, Y) based the identified reason.
Construct the SPA dataset S = [X., Y. f.
### Alignment Learning 10 for epoch e 1,... do
I1
for each instance [X, V, }} in SPA S do Locate the crucial parts with Levenshtein distance using Equation | and assign weights according to r( F(Mr.)) L)::
13
Update The using the fine-grained quality-aware learning objective in Equation
3.3 DISCUSSION
In this part, we discuss how the proposed FIGA approach relates to existing fine-tuning approaches, namely SFT and RLHF.
Relationship with SFT SFT can be viewed as special case of our FIGA method without revision, when training is performed with the higher-quality instance Y, and each token Y considered equally important. Compared to SFT. FIGA has the following two advantages: (1) only consider the inferior part the bad case that the initial model does not perform well; (2) we explicitly enforce the model understand what good and bad behaviors in the loss function. It inherits the merits of SFT, and further leverages fine-fined quality signals for improving the alignment.
Relationship with RL Our method can be considered as simplified but efficient version of RL. Using typical PPO method (Schulman et al., 2017) as an example, its objective to optimize the actor model (i.e,, the initial model nτ) maximize the expected reward score, formally given
5

---

Published as a conference paper at ICLR 2024
CPPO = E(
To To(ii Dee. x)
Ag.
Tead (it Vice X)
where Ag. is the advantage function of the de token returned by the critic model given the reward score Ry. Tia is the model before the previous parameter update. Here, we ignore the clipping function and KL penalty for convenience. Considering the FIGA training objective in Equation 2 our weight functions F(-) and F(-) in FIGA can be viewed as a simplified advantage function A() in Equation 3 to evaluate the importance of each token. Therefore, FIGA has a similar objective with RL but with a simplified roken-wise reward function. We do not use an extra learned critic model and remove the use of previous rollout model, which makes FIGA more efficient. In the later experiment section, we will verify the effectiveness of our method.
EXPERIMENT
4.1 EXPERIMENTAL. SETUP
4.1.1 BASELINE METHODS
In order to better evaluate the FIGA method, we choose several baselines for comparison: (1) SFT (Ouyang et al. 2022): it continues to fine-tume tune the initial model using pairs of data with sequence-to-sequeee
loss. (2) PPO (Ouyang et al.) b: it optimizes the initial model to achieve a higher reward score provided by the reward model. (3) CoH (Liu et al. 2023ap: annotates the dataset by prefixing "A helpful answer: " and "An unhelpful answer:
the responses of corre- sponding quality, employs SFT on it, and computes loss only for the specially masked tokens. (4) RRHF (Yuan et 2023): it applies SFT on the optimal responses and further optimizes the ranking loss among responses from multiple sources to encourage the model to achieve a greater log prob- ability for the response that ranks better. (5) DPO (Rafailov et al. 2023): it eliminates the need for explicit reward modeling and instead directly optimizes the policy model using comparison data.
4.1.2 IMPLEMENTATION DETAILS
Training Datasets For our SPA dataset mentioned in Section 3.1 we broadly select the follow-
ing datasets our initial instance pool: HH-RLHF (Bai al. 2022a). ShareGPT (ShareGPTT)
2023). Instruct GPT-J Pairwise (Dahoas) 2023). SHP (Ethayarajh et al.) (002)). and
OpenOrca (Lian et al., 2023). We employ Alpaca-7b Taori al. (2023) the rollout model generate responses Y and use gpt-3. 5-turbo to revise and obtain Y. The prompt used here can be found in Appendix [ As for the filtering process. we utilize www.tt//eewwddmmdeeee..
-v3--arge---22 (OpenAssistant 2023) as the reward model. According to reward score distribution 2 we empirically set the threshold values 7) = 1, 7/2 = 3, 7/a = 3.5, respectively. The statistics for reward scores and edit operations of the SPA dataset are presented in Table [ and the graphical illustration of reward scores is provided in Figure
2 We find that initial responses Y exhibit large distributional disparity compared with the reference responses Y, which may complicate the learning process for the model. In contrast, our modified responses not only align more closely with the original distribution but also enhance the quality, which simplifies the learning task for the rollout model. The completed SPA dataset consists of 17,333 instances, and more details and analysis can be found in Appendix D
Model Details (1) For SFT, we set the learning rate to le-5 and the batch size to 128. We conduct 5 epochs of training and choose the one with the highest reward score on the test set as the ultimate SFT model. (2) For PPO, we apply the OpenL (Open
2023) library and adhere to its hyper-parameter configurations. We use Alpaca-7b as the initial critic model and use the same reward model utilized in SPA construction. Given the modest gains observed in previous experiments when employing PPO-ptx on models with around 6B parameters (Ouyang et al. 2022), we refrain from introducing a pre-training mix as additional training objective. (3) For CoH, we annotate the SPA dataset with their method. Considering the smaller size of our dataset compared to theirs, set FCM (random masked token ratio to prevent overfitting) to O. Additionally, to ensure a fair comparison with PPO, we disable the pre-training dataset regularization. (4) For RRHF and DPO, follow the recommended hyper-paramest from the original papers. (5) For FIGA,

---

Published as a conference paper at ICLR 2024
Table The average reward of sponse data and the average number #ops of editing operations to them from the Y.
Deriity
0.2
0.1
Data
Y
Y
R
-1.07
3.94
1.78
0.0
-10.0
-7.5
-50
0.0
5.0
#ops
75.69
39.38
Reward Score
Figure 2: Reward score distributions.
the parameters a 1,3 y respectively. Besides, considering the instability when
training negative samples in practice (Bhardwaj & Porial 2023 Liu et al. 2023a), further select the bad tokens returned by Levenshtein distance in equation | by retaining only those with negative ligeliieddood less than 0.6.
4.1.3 EVALUATION TASKS
We evaluate the performances of different methods comprehensive benchmarks. We segment test set from the selected datasets and utilize the reward score evaluate how effectively the model has learned to align with human preferences. The resulting test set comprises total of 3,608 data entries. Additionally, we employ a broad array of out-of-distribution benchmarks to conduct more comprehensive evaluation of the model's capabilities. This includes assessing knowledge utiliza- tion (MMLU (Hendrycks et al. 2020)), human alignment (WinoGender (Rudinger et al.] 2018), CrowS-Pairs (Nangia et al. (2020), and TruthfulQA (Lin et al. 2021)), and open-ended generation (Vicuna (Chiang et al. 2023) and WizardLM (Xu et al. 2023)). The details of evaluation tasks can be found in Appendix
4.2 EXPERIMENTAL RESULTS
Table 2: Performance comparison of FIGA and other widely used alignment methods. Bold and underlined fonts indicate the best and the second-best score. denotes lower is better.
Methods
Reward
MMLU
TruthfulQA
CrowS-Pairs.
WinoGender
Vicuna
WizardLM
Average
Alpaca-7b
3.96
39.2
33.7
61.1
55.6
7.9
7.0
31.7
SFT
4,56
39.3
22.0
61.5
55.3
8.4
8.3
31.1
PPO (SPA)
4.06
39.6
30.1
61.3
56.2
7.6
7.4
31.5
PPO (85K)²
4.54
39.2
36.7
60.6
56,2
7.9
7.2
33.1
CoH
4.24
39.6
28.2
$9.6
52.1
8.3
8.1
32.7
RRHF
4.23
37.8
32,9
59.9
60.0
7.9
7.9
31.3
DPO
4.23
40.1
34.8
61.2
57.0
8.0
7.7
32.7
FIGA
4.62
40.8
42.0
61.2
59.6
8.6
8.3
34,9
As Table 2 FIGA surpasses all baselines, showing superior performance across benchmarks, even outperforming PPO using four times training data. This implies FIGA aligns more closely with human preferences and exhibits strong overall task-solving capabilities.
Moreover, assess the comparative advantages of each response, conduct comparison between the responses generated by FIGA and other baseline methods on the Vicuna and WizardLM bench- marks. The results are shown in Figure And also conduct human evaluation in Appendix for more fine-gratee fineigraiedd analysis.
To reflect the model's overall performance, we compute the average score. Specifically, multiply the reward score by and the score for CrowS-Pain Pairs calculated 100 minus the original score.
"Given that PPO does not utilize labels in the dataset and requires large amount of data to learn through trial and error, integrate additional open-source data with the SPA dataset to fully leverage the strengths of PPO. We obtain total of 84,908 entries, and the PPO trained with this dataset is referred PPO (85K).

---

Published as a conference paper at ICLR 2024
FIGA Wins
Tie
FIGA Loses
FIGA Wins
Tie
FIGA Loses
Alpaca 7B
52%
39%
9%
56%
32%
12%
PPO (SPA)
49%
42%
9%
PPO (SPA)
47%
39%
14%
PPO (85K)
49%
41%
10%
PPO (85K)
48%
39%
13%
DPO
46%
43%
11%
DPO
36%
49%
15%
RRHF
42%
50%
8%
RRHF
34%
48%
18%
CoH
30%
48%
22%
CoH
29%
49%
22%
SFT
27%
53%
20%
SFT
26%
49%
25%
ChatGPT
ChatGPT
16%
0%
25%
50%
75%
100%
0%
25%
50%
75%
100%
Figure
Win rate of FIGA vs other basclines on Vicuma (left) and WizardLM (right).
4.3 FURTHER ANALYSIS
4.3.1
PERFORMANCE COMPARISON W.R.T. SUBPAR ALIGNMENT DATASET
As mentioned in Section 3.1 the steps involved in constructing the SPA dataset include: (1) col- lecting existing datasets, including preference datasets and typical instruction datasets; (2) filtering data based on reward scores; and (3) revising the initial responses using LLM. To examine the effectiveness of each of them, we develop the following dataset variants:
Preference: we only use preference data to construct the initial instance pool D with 3,971 sam- ples.
Instruction: construct the initial instance pool D with typical instruction data that the reward model had not encountered during its training, totaling 3,971 instances.
W/o reward filtering: this variant excludes the step of data filtering according to reward scores. Wo revision: we do not utilize LLM to revise and instead directly employ the reference re- sponses.
Table Performance comparison of different instances pools.
Methods
Reward
MMLU
TruthfulQA
CrowS-Pairs.
WinoGender
Vicuna
WizardLM
Average
Preference
4.42
37.4
22.6
61.5 59.7
$7.1 57.5
7.4
6.6
30.5
Instruction
4.35
31.1
32.8
Table 4: Performance comparison different data annotations.
Methods
Reward
MMLU
TruthfulQA
CrowS-Pairs.
WinoGender
Vicuna
WizardLM
Average
FIGA
42.0
59.6
W/o reward filtering
4.41
38.0
28.8
61.1
58.5
8.3
8.0
32.1
W/o revision
4.39
37.5
26.7
62.1
55.6
8.2
7.7
31.1
From the results in Table and Table on typical instruction data that is new to the reward model, proving that its applicability is not restricted preference data. (2) Filtering based on reward scores crucial, resulting in a +0.21 reward score increase and +2.8 benchmark increase. This underscores the significance of training on queries where the model's original performance is subpar. (3) Addressing the distribution shift through revisions is important, as training with revisions yields +3.8 points on average.
that: (1) FIGA demonstrates strong performance
4.3.2 PERFORMANCE COMPARISON W.R.T. WEIGHTING FUNCTIONS
As mentioned in Section 3.2 F(-) and f(-) in Equation first make comparison between Ý and Y, and then assign distinct weights various tokens. Here, explore other weighting functions as how they acquire the tokens to be encouraged or discouraged, and study the influence of different hyper-parameters (a, B, and 7). More details hyper-parameters can be referred to in Appendix
8

---

Published as a conference paper at ICLR 2024
Variants of F(-): we set 3 to 0 and propose three different variants to explore alternative methods for identifying the tokens that should be encouraged.
Bag of words: sets F( t) only when y き Y: while the rest are set to O. ChatGPT (weighted): motivated by work (Lee al., 2023), employs ChatGPT to assess the impact of tokens on sentence quality. The specific prompt can be found in B The returned scores are adjusted to fall within the range of 0.7 to 1.3 and are set as F(e.. t). For words that ChatGPT doesn't It address, F(yr, t) 0.3.
ChatGPT (binary): it sets F( t) to I only when is returned by ChatGPT with non-zero score, while the rest are set to O.
Variants of f(:): for the tokens to be discouraged returned by f(-), we further filter bad tokens returned by Levenshtein distance and retain only those with negative log-likelihood below 0.6. To assess its effectivenesss, we design the variants including:
Inverted threshold: retains only the bad tokens returned by Levenshtein distance with negative log-likelihood 2 0.6.
-W/o further selection: it penalizes all the bad tokens returned by Levenshtein distance.
Variants of hyper-parameters: to explore the influence of ax 1,,8,7 y in Equation design:
- 3 = 0: it sets to with a = and & = 0.
- -y * 0: it sets y to 0.3 with a = 1 and 8 = 0.5.
- -R(): assigns Ry, Ry.0 to a, y respectively, where Ry and Ry are standardized
through the min-max method.
Table Performance comparison of different weighting functions.
Explorations
Methods
Reward
MMLLU
TruthfulQA
CrowS-Pairs)
WinuGender
Vicuma
WizardLM
Average
Ours
FIGA
4.62
40.8
42.0
61.2
59.6
866
8,3
34,9
29.3
$7.6
Eacouraged
ChatiPT (weighted)
39.8
21.7
60.0
57.9
ChaiGPT (hinsry)
4.32
39.0
24.4
59.9
59.0
7.8
7.6
31.6
Discouraged
Invened threshold Wio further selection
3.01
3.80
28.1
30.2
24.0
27.2
58.5
56.2
50.4
57.4
8.1
8.0
7.4
29.3
28.1
4.61
41.0
37.0
99.6
58.1
34.2
Hyper-porameter
7 0
4.54
41.2
32.2
60.1
56.0
8.4
82
33.0
Rg
4,54
39.7
37.8
62.9
57.1
8.2
8.2
33.4
The results in Jindictindicate that: (1) Levenshtein distance excels in extracting critical tokens, with over +1.5 and +2.6 average scores compared with the statistical method and ChatGPT annotation method. (2) It is necessary to further filter the bad tokens returned by Levenshtein distance, this leads to an average improvement of +6.8. (3) Remaining only the poor-quality tokens with negative log-likelihood < 0.6 is sensible choice, which aims to penalize tokens that the model relatively confident in generating, even though their actual quality is subpar. (4) Pumishing the undesirable actions beneficial, as it results in an average increase +0.7. (5) Focusing only on good and bad tokens is sufficient, since setting to non-zero value leads to decrease of 1.9, (6) The inferior performance of reward score weights can attributed to intrinsic inaccuracies of the reward scores, especially in out-of-distribution scenarios (Bai et al. 2022b).
CONCLUSION
In this paper, we have presented FIGA, a new approach that aligns language models with human preferences, by leveraging fine- gre-ggaieed quality signals to enhance the alignment quality during fine- tuning. In our approach, we firstly curate a high-quality alignment dataset that pairs initial re- sponses with revised responses on queries that a LLM cannot perform well. Furthermore, we have designed new learning objective that that can leverage the fine-grained quality signals by con- trasting initial with revised responses. Our approach inherits the merits of SFT (e.g., efficient and easy-to-impe -n-ttmmmmpemmen...... and meanwhile can better understand and leam what correct behaviors for alignment. FIGA shows superior performance on extensive tasks, with +3.2 points and +1 1.8 points against the initial snpervised-finetunel model and the strong PPO method.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was partially supported by National Natural Science Foundation of China under Grant No. 62222215, Beijing Natural Science Foundation under Grant No. L233008 and 4222027. Xin Zhao is the corresponding author.
REFERENCES
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0586. 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et Constitutional ai: Harm- lessness from ai feedback. arXiv preprint arXiv:2212.08072 2022b.
Rishabh Bhardwaj and Soujanya Poria. Red-teaming large language models using chain of utter- ances for aliint-aligmmmet. arXiv preprint arXiv:2308.0960 09662, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See htts::/vvicua, Imsys. org (accessed 14 April 2023), 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:220..02311, 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling ceastructionfinetutee language mod- els. arXiv preprint arXiv:2210.11411, 2022.
Dahoas. Dalloss/ynleeiicinntuucppppppppp http:/awwww......oooooooooooooaa
2023.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model aligmment. arXiv preprint arXiv:2304.06766 2023.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning. 2022.
Caglar Gulcchre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08994 088808880999,,8888 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.088 03000, 2020.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 25:22199-22211 2022.
10

---

Published as a conference paper at ICLR 2024
Ranjay Krishna, Donsuk Lee, Li Fei-Fei, and Michael S Bernstein. Socially situated artificial in- telligence enables learning from human interaction. Proceedings of the National Academy of Sciences, 2022.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267, 2023.
Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium"". Openorca: An open dataset of gpt augmented flan reasoning traces. htttp::/htttps
huggingface.. Corpceeoraoooooooaoa//aaaaaaaaaa /OpenOrca) 2023.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprini arXiv:2305.20050, 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958 2021.
Hao Liu, Carmelo Sferrazza, and Picter Abbeel. Chain of hindsight aligns language models with feedback. arXiv preprint arXiv:2302.02267 2023a.
Ruibo Liu, Chenyan Jia, Ge Zhang. Ziyu Zhuang, Tony Liu, and Soroush Vosoughi. Second thoughts are best: Learning to re-align with human values from text edits. Advances in Neural Information Processing Systems, 35:18 1-96, 2022.
Ruibo Liu, Ruixin Yang. Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M Dai, Diyi Yang, and Soroush Vosoughi. Training socially aligned language models simulated human society. arXiv preprint arXiv:2305. 2it.:2200..6660 2023b.
Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv preprint wwXiv:2309.066577 2023c.
Yixin Liu, Alexander R Fabbri, Pengfei Liu, Dragomir Radev, and Arman Cohan. On learning to summarize with large language models as references. arXiv preprint arXiv:2305. 14239, 2023d.
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Am- manabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591-27609, 2022.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02 02707, 2023.
Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R Bowman. Crows-pairs: A challenge dataset for measuring social biases masked language models. arXiv preprint 2x11220100000 2020.
OpenAssistant,
Opeaasssaanteeercccmmodddddeeeer https: https///huggingaace 2023.
OpeILLMAI. cc/000/hssisanneeeeeee//r//erre-
2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.1820 2023.
11

---

Published as a conference paper at ICLR 2024
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Chris- tian Bauckhage, Hannanch Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization. arXiv preprint 1rXiv:2210.01241, 2022.
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. arXiv preprint (00.::8804093000 2018.
John Schulman. Reinforcement learning from human feedback: Progress and challenges, 2023. URL ptt.:ttp:: /ttp::::/wwwww. youtube....b...........o.......m wtt.://be.oooooooooamama..eeeeee com/ com/wat.c wwt.hhhthccccccccc ? vv=hh Lw5Q_UFg
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 1707.06377 2017.
ShareGPT.
Sharegpt vicuna unfiltered.
https://hugee htt.:faccc.co/ataaatttttaa un anfileeredde 2023.
Ziang Song, Tianle Cai, Jason D Lee, and Weijie Su. Reward collapse in aligning large language models. arXiv preprint (rXxv:2305./7600 2023.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. arXiv preprint arXiv:2305.03047, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An cetrruction-oollowiii llama model. https://githuu: com/ tttttttttuuuu/ laah-llabssee sttsstanforrr. alpaca] 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint 2307::2307.09288, 2023b.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2021.
Sam Wiseman and Alexander M Rush. Sequence-to-sen pesence-to-sequeee to-ueuee learning as beam-search optimization. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1296-1306, 2016.
Yotam Wolf, Noam Wies, Yoav Levine, and Amnon Shashua. Fundamental limitations alignment in large language models. arXiv preprint wrXiv:2304110882 2023.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 222.:2234..224,,.. 2023.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXivv2304.05302. 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li. Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.0106 2022.
Tianjun Zhang, Fangchen Liu, Justin Wong, Pieter Abbeel, and Joseph E Gonzalez. The wisdom of hindsight makes language models better instruction followers. arXiv preprint arXiv:2302.05206 2023.
12

---

Published as a conference paper at ICLR 2024
Wen Zhang, Yang Feng, Fandong Meng, Di You, and Qun Liu. Bridging the gap between training and inference for neural machine translation. arXiv preprint arXiv:1906.02444. 2019.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yinggian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et survey of large language models. arXiv preprint arXiv:2303. 18223, 2023a.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Salch, and Peter Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. 10425, 2023b.
Lianmin Zheng. Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with int-bench and 2023.
chatbot arena. arXiv preprini arXiv:2306.05oc
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305. 2023.
A APPENDIX: DATA SOURCES
(1) HH-RLHF (Helpful and Harmless): It comprises two main categories of data: human prefer- ence data about helpfulness and harmlessness, and human-annotated red teaming dialogues. The first one is pivotal to train the reward model, and the second one gives insights into model red-teaming technique:
(2) ShareGPT: The dataset contains conversations through the API using process, Within each conversation, both user prompts and ChatGPT responses from OpenAl are presented
(3) Synthetic Instruct GPT-J Pairwise: Crafted for instruction-oriented tasks, it explores model- generated outputs when exposed to synthetic prompts
(4) Stanford SHP: It offers 385K human preferences across multiple disciplines. These preferences are designed to discern the relative helpfulness of responses. Contrary to the HH-RLHF dataset, all content in SHP is penned by humans, serving as valuable complement to other datasets
(5) OpenOrea: This dataset an extension of the FLAN Collection, including GPT-4 and GPT-3.5 model completions. Its primary application lies in training and evaluation in the realm of NLP. For our investigation, we've we've exclusively focused on the English instruction subse
B APPENDIX: PROMPTS USED FOR DATA AUGMENTATION
Details for revision Given a question, along with the poorer original model response and pre- ferred ground truth response, we instruct ChatGPT make minimal modifications to the original response, while ensuring that the output still remains closely aligned with the preferred response.
This process can be divided into two steps: first analyzing the reasons for the lower quality of the original response based on the comparison, and then, making revisions using the appropriate prompts based on these factors.
Among them, the quality of Response is inferior to that of Response Please compare them and choose one of the following four possible reasons for the area where Response 1 performed the worst: A. Needs more accurate content, B. Needs more comprehensive content or more details, C. Requires adjustments in structure, D. Other reasons (such containing harmful information or going off-topic). Do not just include analysis, but just return the choice.
Prompt to used analyze the reason: Question: Response I: Response 2:
unf infiltered
https:/
13

---

Published as a conference paper at ICLR 2024
The prompts used revise according to different reasons:
Prompt for reason A: Question: Response 1: Response 2: Please replace the content corresponding Response 1 with the accurate and high-quality essence from Response 2, and remain the original structure of Response
Ensure that the edit distance between the optimized Response and the Response I is as low as possible.
Please incorporate the compre- hensive topic or the details from Response into Response or necessary, replace any synony- mous content from Response with that from Response 2. You must remain the original structure of Response 1, ensure the edit distance between the optimized Response with the Response I is as low as possible, and not add new contents other than those contained in Response I and Response
Prompt for reason B: Question: Response 1: Response 2:
Response 2: The structure of Response well-organized, featuring elements including but not limited to: point-by-po by-point addressing. 2. providing an overview of the question before answering. Use the structure of Response 2 to rephrase Response 1. Ensure that the optimized Response should maintain a relatively low edit distance from the original Response 1.
Prompt for reason C: Question: Response 1:
Annotate the importance of each word Given question, along with the lower-quality original response from the initial model and a higher-quality ground truth response, we require ChatGPT to score each word based on comparison, terms of how much it improve the quality. Below is an example.
Below is an instruction that describes task, followed by an original response and belter response in terms of how well it aligns with human preferences, being helpful, harmless, and honest. Your task is to return a list containing tuples with words and corresponding scores, which are meant to measure the extent to which the words improve the quality of the original answer to the better answer. The scores are all integers, with being the lowest score and being the highest score. Instruction: Original Response: Better Response:
C APPENDIX: EVALUATION DETAILS
We utilize suite of benchmarks to evaluate the performance as mentioned in Section 4.1.3 Here, we provide details on how we use these benchmarks for evaluation.
The multitask multilingual language understanding (MMLU) benchmark consists of 57 subtasks. Our approach involves calculating the negative log-likelihood of the correct option across all tasks, ensuring it is the minimal among all options. Finally, calculate the accuracy of cor- rectly answered questions. The implementation details can be found at the di-evallation-hharee repository [
MMLU:
TruthfulQA: For the TruthfulQA benchmark, we select the multiple-choice task (MC1), which contains only correct option. Similar MMLU, we evaluate by determining if the negative log-likelihood of the correct option is the lowest among all options, and calculate the accuracy of correctly answered questions.
CrowS-Pairss This benchmark involves comparing two sentences: one exhibits a common form of bias, while the other one does not, differing only in a few words. We calculate the perplexity for each sentence and determine the bias rate by evaluating which sentence has a lower perplexity. A bias rate closer to 50% indicates less bias in the model.
Winogender: It assesses the model's accuracy in handling Winograd schema challenge tasks under three scenarios: male, female, and unspecified gender. It based on whether the negative log- likelihood of the correct sentence is lower. Our primary metric is the average score across these gender scenarios.
14

---

Published as a conference paper at ICLR 2024
Vicuna and WizardLM: These benchmarks instruct the model to generate responses to the given prompts. The responses are then rated by ChatGPT (Zheng et al. 2023) on a scale to 10, with the overall performance measured by the average score.
D APPENDIX: SPA DATASET DETAILS
We have conducted budget estimation based on the usage of the GPT-3.5-turbo API for our SPA dataset. The average imput token count per data entry in the SPA dataset is 952, and the average output token count is 143. Considering the pricing of API usage (i.e., $0.001 and $0.002 per 1K tokens for input and output respectively), the total estimated cost for the entire SPA dataset amounts to $21.45.
Additionally, we conduct experiments with various sizes of the SPA dataset investigate how much data is necessary to achieve reasonable performance gain. The results in Table 6 indicate that constructing a SPA dataset with 8000 samples and training based on the FIGA method can lead to substantial performance boost.
Table 6: Performance comparison of different data sizes.
#Data
Reward
MMLU
TruthfulQA
CrowS-Pairs,
WinoGender
Vicuna
WizardLM
Average
Alpaca-7b
3.96
39.2
33.7
61.1
55.6
7.9
7.0
31.7
4k
4.28
39,9
26.3
59.0
$5.7
8.6
31.8
8k
4.58
41.0
33.2
59.8
57.2
8.4
8.2
33.4
17k
4.62
40.8
42.0
61.2
59.6
8.6
8.3
34,9
E APPENDIX:
SETTINGS
Below are the results of our experiments conducted with various hyper- parameter configurations.
Table 7: Performance comparison of different hyper-parametta
Explorations
a/3/y
Reward
MMLU
TruthfulQA
CrowS-Pairs
WinoGeader
Vicuna
WizardLM
Average
1/0.50
4.62
40.8
42.0
59.6
8.3
34,9
R0,500
4.54
39.7
37.8
62.9
57.1
8.2
8.2
33.4
/,00
27.3
32.6
1/0.50
4.65
41.1
41.4
60.1
57.1
8.6
8.4
34.7
w.L. 3
1/0.25/0
4.59
38.8
40.3
61.5
60.6
8.3
7.9
34.3
1/0.2/0
4.64
40.1
38.3
61.7
59.9
84
8.3
34.2
1/0.5/0.3
4.54
41.2
32.2
60.1
56.0
8.4
8.2
33.0
IAV0.3
4.50
40.4
30.8
59.8
56.4
8.1
32.8
From the Table 7 it can be observed that: (i) should be set to 0, as non-zero y leads to decrease in results. (ii) should be set to as setting a the reward score results in decrease
in performance. (iii) The final results are insensitive to 8, indicating that as long as (3 is set to reasonable value, will not notably impact the overall performance.
F APPENDIX: HUMAN EVALUATION
In addition to all the automatic evaluation metrics, we further carry out human evaluation test how much the model align to human preference after FIGA training. To be specific, we explore the following aspects:
Firstly, we randomly select 20 instructions from the test set and choose the outputs before FIGA training and after FIGA training. We ask 3 participants assess whether these responses exhibit the four aforementioned types of errors. The results Table show that after FIGA training, errors of all types have significantly decreased.
Then, we evaluate the quality outputs through human evaluation. We randomly select 30 instruc- tions from Vicuna benchmark, along with corresponding responses from FIGA and other baseline
15

---

Published as a conference paper at ICLR 2024
Table Human evaluation of error types.
Error Type
Inaccuracy
Lack of Details
Structure
Others (off-topic or harmful)
before FIGA training
20% 5%
35% 10%
5%
0%
after FIGA training
0%
0%
methods (including ChatGPT). We invite three evaluators make pairwise comparisons between FIGA responses and those of each baseline method, aiming to identify the one with higher quality. Finally, we aggregate the assessments from the three participants and obtain the final evaluation results through the mode method. The results in Table 9 show that FIGA outperforms all baseline methods, indicating its ability to better align with human preferences.
Table
Human evaluation of overall quality.
FIGA against
Alpaca-7b
SFT
PPO
PPO (85k)
RRHF
CoH
DPO
ChatGPT
FIGA win rate
73%
57%
83%
73%
73%
40%
16


Reviews:
[{"review_id": "z9YYMGmw95", "reviewer": "Reviewer_J3kJ", "summary": "This paper mainly tackles the problem of insufficient supervision when aligning LLM's responses to human-preferred ones. The authors argue that the current paradigm of only giving the developed model (\"rollout model\") the quality label (e.g., good or bad) of the demonstration and instructing the rollout model to imitate human-preferred responses is insufficient, as the rollout model may not fully capture the correct alignment behaviors. Therefore, this training paradigm can make the rollout model suffer from the distribution shift between its own outputs and imitated outputs. \n\nThe author proposes to bridge the gap of this distribution mismatch by creating a surrogate target that is more similar (tries to maintain a small edit distance) to the model's original low-quality outputs, but is of better quality. The creation of such a target is mainly done by first using a pretrained reward model to filter out queries where the model gives low-quality output, and then instructing ChatGPT, the state-of-the-art LLM, to compare the developed model's original outputs and the human-preferred and do minimum edits to improve the original outputs. \n\nThe authors build a new dataset SPA based on previous human preference datasets by replacing the human demonstration with the surrogate target and training the developed model to edit its outputs to surrogate targets via a token-wise loss. Experiment results demonstrate the effectiveness of the proposed method over various tasks.", "questions": "It seems weird that using \"Average\" scores to compare the models trained under different methods -- the value magnitude of \"Vicuna\" and \"WizardLM\" (\"Reward\" scores are multiplied by 10 as the author noted in the footnote)  are not consistent with performance numbers in other benchmarks, and the test data numbers across different tasks are also not the same. What is the motivation for using this metric?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-motivated, makes a timely contribution to the distribution shift problem in the Human-LLM Alignment field, and contributes a useful LLM-in-the-loop method to create a higher-quality dataset better for model fine-tuning. \n\n2. The experiment results are comprehensive and solid enough to support the major claims of the paper.", "weaknesses": "1. **Narrowed Application Range**: It seems the proposed dataset creation method is strongly biased by the rollout model checkpoint and the reward model used for identifying low-quality outputs. As each rollout model and reward model combination may have different failure modes, it is not clear if the practitioners want to switch to other rollout models and other reward models, how much this dataset can still be helpful. It can be an unreasonably large consumption of computation power if we need to re-apply the proposed method for every rollout model and reward model checkpoint. This might be the problem for almost all human preference datasets as they also use rollout outputs from some model (e.g., Anthropic HH-RLHF uses Anthropic's own LLM outputs), but there seems to be more model-specific limitations for the dataset created by this method as the token-wise label used in the new dataset is also model-dependent. It would be better if the author could discuss about how much data is necessary to achieve reasonable performance gain, whether it can be applied to other LLMs (not only Alpaca-7B), and the budget decomposition. \n\n2. **Possible Unfair Comparison to Baseline**: The reviewer noticed that in Section 4.2 and Section 4.3.1, the dataset used for the proposed training method and baseline methods are not of the same size. This may or may not give an unfair advantage to the proposed method. It would be better if the author could verify the effectiveness of the proposed method by subsampling the created dataset to double-check the effect of the dataset size. \n\n3. **Unclear Alignment to Human Preference, and Insufficient Evaluation**:  The goal of Human-LLM Alignment is to make the model outputs more aligned with human values. It is okay to use a surrogate objective to better train the model for optimization purposes. However, it is important to ensure the training is still making progress towards the original goal. Therefore, there should be some evaluation on whether the model trained in the newly proposed way is getting closer to human preference.  A straightforward yet missing \"inherent\" evaluation is comparing the updated rollout model outputs to the original human-preferred ones -- how similar it is, and what is the proportion of samples getting more similar. Also, after training, whether the model truly learn to do \"desired behaviors\"? Has the model learned to avoid any of the aforementioned four types of mistakes? (In Section 3.1) The specific Reward Model can be biased and only reading \"Reward\" numbers at tables hardly gives readers confidence in the alignment performance in terms of human preference. This also applies to other benchmark numbers -- what the model is learning under the proposed method? The current version of the paper lacks analysis on this side. \n\n      As human preference is more complicated than the benchmarks listed in this paper, more careful investigation should be conducted on the user side to understand how such target-changing would influence user perception.  It has been widely reported that GPT-4-based Evaluation (e.g., Vicuna Benchmark) has various biases (e.g., positional bias, verbosity bias, and self-enhancement bias as reported in [1,2,3]) and cannot fully represent humans, so it would be helpful to do the human evaluation to examine the quality of the rollout model outputs after going through the proposed training method. \n\n4. **Unclear Experiment Detail**: Some hyperparameter setups in Section 4.3 seem arbitrary (e.g., in FIGA-contra, FIGA-reward and FIGA-fix). Why does the author set up the hyperparameter like that?\n\n\n\nReferences:\n\n[1] Zheng, Lianmin, et al. \"Judging LLM-as-a-judge with MT-Bench and Chatbot Arena.\" NeurIPS 2023. \n\n[2] Wang, Peiyi, et al. \"Large language models are not fair evaluators.\" arXiv preprint arXiv:2305.17926 (2023).\n\n[3] Pezeshkpour, Pouya, and Estevam Hruschka. \"Large language models sensitivity to the order of options in multiple-choice questions.\" arXiv preprint arXiv:2308.11483 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s3N820XSo1", "reviewer": "Reviewer_772X", "summary": "The paper proposes FIGA, an RL-less method that aligns LLMs with fine-grained token-level quality signals. A new curated dataset SubPar Alignment (SPA) is constructed as well, which contains pairs of initial/revised responses. FIGA is evaluated on a diverse set of benchmark tasks, outperforming baselines including SFT and PPO.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "- The paper is well written and is easy to follow.\n- The motivation of the paper is timely and important.\n- Many in-depth analysis on the constructed dataset and different experiment settings.", "weaknesses": "- The paper needs more comparison with works that do not leverage reward models (e.g. DPO [1]), to demonstrate the advantage over recent RLless methods\n- The paper needs to specify what each performance metric is for the benchmark datasets.\n- In Table 2, performance improvement is marginal compared to the best performing baseline method. Also, multiplying the Reward metric by 10 while leaving Vicuna, WizardLM at the original scale is misleading.\n- In Figure 3, the win rate of FIGA is below 50% most of the time, questioning the actual advantage of FIGA over baselines. I do not think that being a  “Tie” justifies the necessity of FIGA sufficiently.\n- There are quite a number of hyperparameters to tune. How sensitive is the method to alpha, beta and gamma?\n\n[1] Direct Preference Optimization: Your Language Model is Secretly a Reward Model (R.  Rafailov et al., 2023)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5CwLb6qOkv", "reviewer": "Reviewer_1jST", "summary": "The authors present a method FIGA as a new alignment method with more fine-grained signal compared to e.g., RLHF. For a given text output task, they obtain pairs of poor responses $\\hat{Y}$ and strong responses $Y$, and make local changes to $\\hat{Y}$ based on qualities of $Y$ to obtain an improved response $\\tilde{Y}$ that is still surface-level-similar to $\\hat{Y}$. Comparing $\\tilde{Y}$ to the original $\\hat{Y}$ then gives us fine-grained signal on particular token positions which can be used for more targeted SFT. They show improvement on a few different benchmarks.", "questions": "--on the topic of $\\beta$ being set to 0, i saw that even in your later ablation you only set $\\beta$ to 0.025. If you increase $\\beta$, do you get the issue with unlikelihood training [1] where (I believe) it's been shown that you unnecessarily penalize unrelated tokens that are just fluent in context?\n\n[1] Welleck, Sean, et al. \"Neural text generation with unlikelihood training.\" arXiv preprint arXiv:1908.04319 (2019).\n\n--similarly, in eqn 2, if e.g. hypothetically you make a change that touches several parts of the text in a consistent way (like say someone's name changes from Amy to Bob in 3 places), you might be penalizing the parts that come later even if they're simply being consistent with what was written elsewhere, which might throw off your optimization if you put nonzero weight on the second term? Not sure if this would help, but what might happen if you were to optimize by e.g., not using $\\hat{Y}$ and $\\tilde{Y}$ in their entirety, but doing more \"atomic\" changes like only changing one span at a time? Is it too difficult to align the spans? \n\n--thoughts on using this kind of method for tasks requiring even longer outputs? since i wonder if this fine-grained feedback is more valuable in longer-context settings, so i wonder if you would get larger performance gains compared to baselines. though those tasks might be more challenging to evaluate.\n\n--how consistent are the revisions from ChatGPT having smallish levenshtein distance between $\\hat{Y}$ and $\\tilde{Y}? any analysis (even qualitative) on how good ChatGPT's categorization of the 4 types of issues is, and how good are its revisions at addressing these? would be curious to see this analysis since it could point to potential future directions for improvement\n\n(note- i suggested a few different things here. to be clear, i definitely don't expect further experiments on all or even any of these.)", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "--interesting, novel method for generating pairs of responses with localized improvements for fine-grained feedback\n\n--i can definitely see the main intuition (distilling the qualities of a strong response into a more surface-level-similar response) as potentially inspiring future work", "weaknesses": "--how do you collect the human-preferred response $Y$? (i.e., the one that you feed into ChatGPT together with $\\hat{Y}$ to get $\\tilde{Y}$). based on Fig1, i assume this is not generated by the same model as used to get $\\hat{Y}$, but either provided with the dataset or generated by ChatGPT in a distillation-like setting? but since you have a threshold $\\eta_2$ for filtering the quality of $Y$ anyway, i wonder if you could generate $Y$ using your model as well-- I think this would really strengthen your contribution, demonstrating that the method works even without supervision / distillation from a stronger model; your method would then operate in a setup closer to e.g., RLAIF where you only need the initial prompts and principles, and generate all outputs/evaluations yourself without accessing stronger model. you'd only need ChatGPT for the part where you turn $Y$ and $\\hat{Y}$ into $\\tilde{Y}$, which I think isn't really distillation in the same sense.\n\n--seems like $\\beta$ is set to 0, so you built up this whole framework but then in practice turned off a good chunk of it (the part where you penalize the initial bad response)? see questions below", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jAugeKF8VH", "reviewer": "Reviewer_hMwD", "summary": "The paper proposes a novel alignment strategy as a replacement of the RL step in RLHF. This is motivated by the observed unstabilities in PPO, and the complexities involved in online optimization. The idea is to mitigate the limitations from behavior cloning by proposing expert annotations that are actually generated by GPT using the generations of the agent as a form of conditioning. Moreover, the training samples are selected thanks to an external reward model. They also include additional information during the supervised fne-tuning by over-weighting the key modifications/deletions. The proposed FIGA improves performances compared to baselines. They ablate the performances.", "questions": "- Rather than using Levenshtein distance, I am interested in knowing whether you could directly ask gpt-3.5-turbo to explicitly state which part of the sentence is correct/false.\n- Moreover, to further improve results, that would be interesting to give to gpt-3.5-turbo both the human-preferred generation along with the agent generation. Somehow, gpt-3.5-turbo could keep high rewards while reducing the #ops.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Human aligned AI is a critical topic, thus considering better alignment strategies is interesting, in particular if we could make the RLHF more stable.\n- Fine-grained reward is also a promising idea.\n- The idea of modifying the agent predictions with an external model (rather than taking directly the human-preferred generation) makes sense, and seems interesting to limit the drawbacks from behaviour cloning.\n- The procedure is siple, ad ths could be used in real-world applications.\n- The writing is very clear, and the writing is good.", "weaknesses": "- The proposed strategy actually requires both (1) a reward model and (2) a preference dataset. In contrast, most other strategies require only one of those. Actually, the empirical gains can come from the fact that you optimize those two guide the learning.\n- The approach requires to set many new different hyperparameters (6 I believe), reducing the generality of the strategy.\n- FIGA also requires an external model to be available, such as gpt-3.5-turbo. This is a huge limitation. Moreover, this should be ablated against other strategies that would also use gpt-3.5-turbo, such as distillation strategies.\n- More generally, the experiments lack many important baselines, in particular DPO [Rafailov2023]. I list other below, perhaps less critical and more recent, that adding them would be a big positive point.\n\n[Rafailov2023] Direct Preference Optimization: Your Language Model is Secretly a Reward Model \\\n[Gulcehre2023] Reinforced Self-Training (ReST) for Language Modeling \\\n[Zhao2023] SLiC-HF: Sequence Likelihood Calibration with Human Feedback \\\n[Liu2023b] statistical rejection sampling improves preference optimization \\\n[Peng2023] stabilizing rlhf through advantage model and selective rehearsal \\\n\n- Minors\n    * There is a typo in the legend of Figure 1, the main Figure of the paper.\n    * As illustrated in Algorithm 1, the proposed strategy remains essentially offline, as the SPA dataset is built once. One solution would be to build the dataset at each epoch, as done in [Gulcehre2023].\n    * The experiments are only made for Alpaca-7b, which may not be illustrative of larger models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)