The Trickle-down Impact of Reward Inconsistency on RLHF
Authors: Lingfeng Shen
Sihao Chen
Linfeng Song
Lifeng Jin
Baolin Peng
Haitao Mi
Daniel Khashabi
Dong Yu
Venue: ICLR 2024 poster (2024)
Keywords: Large language model
reward model
RLHF
consistency
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Standard practice within Reinforcement Learning from Human Feedback (RLHF) involves optimizing against a Reward Model (RM), which itself is trained to reflect human preferences for desirable generations. A notable subject that is understudied is the (in-)consistency of RMs --- whether they can recognize the semantic changes to different prompts and 
appropriately adapt their reward assignments

--- and their impact on the downstream RLHF model.

In this paper, we visit a series of research questions relevant to RM inconsistency:
(1) How can we measure the consistency of reward models? 
(2) How consistent are the existing RMs and how can we improve them? 
(3) In what ways does reward inconsistency influence the chatbots resulting from the RLHF model training?


We propose **Contrast Instruction** -- a benchmarking strategy for the consistency of RM.  
Each example in **Contrast Instruction** features a pair of lexically similar instructions with different ground truth responses. A consistent RM is expected to rank the corresponding instruction and response higher than other combinations. We observe that current RMs trained with the standard ranking objective fail miserably on \contrast{} compared to average humans. To show that RM consistency can be improved efficiently without using extra training budget, we propose two techniques **ConvexDA** and **RewardFusion**, which enhance reward consistency 
through extrapolation during the RM training and inference stage, respectively.
We show that RLHF models trained with a more consistent RM yield more useful responses, suggesting that reward inconsistency exhibits a trickle-down effect on the downstream RLHF process.

Published as a conference paper at ICLR 2024
THE TRICKLE-DOWN IMPACT OF REWARD (IN-)CONSISTENCY ON RLHF
L Lingfeng Shen*.
Sihao Chen
Linfeng Song
Lifeng Jin 2 Baolin Peng
Haitao Mi
Daniel Khashabi
Dong Yu"
Johns Hopkins University
University of Pennsylvania
Tencent Al Lab
ABSTRACT
Standard practice within Reinforcement Learning from Human Feedback (RLHF) involves optimizing against Reward Model (RM), which itself is trained to reflect human preferences for desirable generations. A notable subject that is understud- ied is the (in- Jinjsoonsistecy of RMs - whether they can recognize the semantic changes to different prompts and appropriately adapt their reward assignments and their impact on the downstream RLHF model.
In this paper, we visit a series of research questions relevant to RM inconsistency: (I) How can we measure the consistency of reward models? (2) How consistent are the existing RMs and how can we improve them? (3) In what ways does reward inconsistency influence the chatbots resulting from the RLHF model training?
We propose CONTRAST INSTRUCTIONS a benchmarking strategy for the con- sistency of RM. Each example in CONTRAST INSTRUCTIONS features pair of lexically similar instructions with different ground truth responses. A consistent RM is expected to rank the corresponding instruction and response higher than other combinations. We observe that current RMs trained with the standard rank- ing objective fail miserably on CONTRAST INSTRUCTIONS compared to average humans. To show that RM consistency can be improved efficiently without using extra training budget, we propose two techniques CONVEXDA and REWARDFU- SION, which enhance reward consistency through extrapolation during the RM training and inference stage, respectively. We show that RLHF models trained with more consistent RM yield more useful responses, suggesting that reward inconsistency exhibits a trickle-don effect on the downstream RLHF process.
INTRODUCTION
Recently, reinforcement learning from human feedback (RLHF) has emerged as popular technique to optimize and align a language model with human preferences (Ouyang et al.] 2022).. RLHF provides a natural solution for optimizing non-differretiable, scalar objectives for language models, and has been the centerpiece of recent state-of-the-art large language models (LLMs) (Lu et al.)
2022 Hejna III & Sadigh 2023
Go et al. 2023 Korbak et al. 2023 OpenAl 2023).
In RLHF. a reward model (RM) generates scalar rewards for model -ender-geeertted outputs as supervision during reinforcement learning. RMs are typically calibrated to proxy human preferences/rankings
of responses, in the context of an input instruction. Since policy gradient methods optimize based on this reward function, the reward function inevitably dictates the behavior of the resultant chatbot. As such, the properties of RMs and their impact on RLHF models have become points of interest for the research community (Gao al. 2022 Zhu et al. 2023 Dong et 2023).
In this work, we study the phenomenon of reward inconsistency in RMs, i.e., current RMs trained with the standard ranking objective on human preference data ($2) often fail to distinguish between more vs. less favorable responses with respect to real-world instructions. We observe that reward inconsistency has trickle-down effect on the RLHF process the more inconsistent the RM is, the more likely the resulting chatbot is to generate inaccurate or less useful responses.
Most of the work done while Lingfeng and Sihao were interss the Tencent AI L.ab.

---

Published as a conference paper at ICLR 2024
C A "consistent" reward model should assign higher rewards conesponding (instruction, resporsa) pairs than other distractors.
W We intraduce two approaches to enhance the consistency standard, preference
CONTAST INSTRUCTIONS
A pair similar instructions with different ground truth responses
Response Consistency
ComexD robuol patagg
RewurdFiiino
Standard
Instruction
Responses
Standderd
I: "What does RMM stand for computing?"
r: "Random Access Menory."
Consiston RM
L
RLHI with more consistent RM Better chatbot
Instruction Consistency
Standdadd RM
P": "What ROM stand for in computing?"
m "Read-Only Hemory." "
Sta Indand RM
Consistett RM
RM
Comsistent
We introduce CONTRAST INSTRUCTIONS as a way measure the consistency of reward model. Each example consists of two similar instructions with different ground truth responses. Given one of the instructions or responses, whether or not an RM could rank its corresponding response or instruction higher than the other distractor indicates the consistency of the RM.
Figure I:
To illustrate and quantify the degree of reward inconsistency in RMs, we propose CONTRAST IN- STRUCTIONS, a simple, intuitive benchmarking strategy that can be used with any instruction-tuning or human preference dataset in fully automated manner. CONTRAST INSTRUCTIONS involves pairs of similar instructions with different responses. As an example, consider two similar-looking prompts (I and I") one about "RAM" and "ROM" (Figure 1 left). Despite the similarity of these prompts, they warrant different responses. Our consistency metries measure whether a reward model can appropriately identify the correspondence between prompts and responses. Specifically, given a pair of such (instruction, response) examples, we consider an RM consistent if it assigns a higher score to the corresponding (instruction, response) compared to the other combinations, i.c. swapping instructions or responses between the two examples, as we show with 1 in Figure
We construct CONTRAST INSTRUCTIONS with four popular open-source human preference or instruction-tuning datasets. Surprisingly, we observe close to random-chance performance when evaluate standard RMs trained with ranking objectives (e.g. ILLMM-7BB CONTRAST IN- STRUCTIONS, while humans are able to rank the responses correctly Ra 80% of the cases. The performance gap indicates the inherent reward inconsistency from standard RM training and infer- ence.
To enhance the consistency of standard RM, we introduce two techniques ($5) CONVEXDA and REWARDFUSION (2) in Figure 13. The two methods can be incorporated during RM training and inference, respectively, without incurring extra computational costs during training. While our ex- perimental results indicate some improvements, the gap with respect to human performance remains large. Interestingly, our analysis ($ reveals that using a more consistent RM during RLHF training leads to more useful responses from the downstream RLHF model (3) in Figure 1). Our findings suggest the value of reward jinnsoonsi as an intrinsic evaluation metric for preference-bel RMs, potentially enabling easier access for future research on reward modeling and RLHF.
The main contributions of this paper are:
We introduce CONTRAST INSTRUCTIONS, intuitive yet scalable benchmarking strategy for evaluating RM consistency. We observe wide performance gap between standard, preference- based RM vs. human judgments, which suggests sizable room for improvements in standard reward modeling and evaluation strategy.
We show that reward consistency can be enhanced without extra training costs. We demonstrate this with two techniques CONVEXDA and REWARDFUSION, which can be applied during RM training and inference stages, respectively.
We empirically show that training RLHF models with more consistent RMs would result in the RLHF model generating more useful responses. We provide thorough analysis and examples, which help understand the advantage of more consistent RM over a less inconsistent one.
2

---

Published as a conference paper at ICLR 2024
2 PRELIMINARIES: REWARD MODELING FOR RLHF
Reward model. Following the conventional setup Ziegler et al., 2019). we define RM Rg as scalar function that allows us to train a generative model during RLHE. To supervise RM, we are given a dataset of human preferences Da. Each instance in this dataset (1,, r. is comprised of an instruction prompt 4, pair of responses r:" Ta where r; is preferred over r; by humans. On this labeled data, Rg is trained to assign higher scalar reward to human-preferree r,- over non- preferred r; in the context of I; This can be achieved by minimizing the ranking loss L, where is the sigmoid function and 1 r is the concatenation of 7, and r,-
( = -E (1.r) Y-D. [log (a (Ro(4 or)) - RR (4)r7)))
(1)
Reinforcement Learning. The last stage of RLHF is reinforcement learning. Specifically, a per- token KL penalty from the supervised fine-tuning (SFT) model at each token mitigate over- optimization of the reward model, and the value function is initialized from the RM. We maximize the following combined objective function (Ø) in RI training based on PPO algorithm (Schulman et al. 2017 Ouyang et al. 2022), RL training dataset D D mr. and pre-training dataset Dpre:
( B(r.r)Daag [Ro(I - Bog (TT (r 1)
SFT [D)] + D, [log (T RL (r))),
where Ro(l r) is the reward function, and RL is the learned RL policy parameterized by ø initialized from pretrained supervised trained model SFT. The first term encourage the policy = to generate responses that have higher reward scores. The second term represents per-token KL reward controlled by coefficient between RRL and π =S to mitigate over-optimization toward the reward model during RL training. The third optional term provides regularization by encouraging responses with high probability from the pretraining dataset with coefficient
CONTRAST INSTRUCTIONS: MEASURING REWARD (IN-)CONY
Conventionally in RLHF, reward models are trained explicitly to distinguish the more vs. less fa- vorable responses in the context of instruction (Eq. TT. One would expect RM to consistently predict higher reward scores toward the more favorable instruction instruction-response pairings. For instance, the example in Figure shows, ideal reward should assign higher score r appearing in response to LA. than /. In practice, however, RMs usually suffer from over-optimization towards the training distribution, leading to inconsistencies between RM predictions vs. human preferences at inference time (Gao et al. 2022).
To illustrate and measure reward inconsistency in RMs, we introduce CONTRAST INSTRUCTIONS.
A CONTRAST INSTRUCTIONS benchmark takes the form of DD = {(1,1 1 . .r ) (see Fig. 1.. Each test instance consists instruction-respone pairs (I r and (I ,)). To make the benchmark meaningfully challenging, sample a and 1 such that the two are lexically similar instructions with different semantics (details later in $3.1) r and y are the human- -peferreeeerred re- sponses A and 13 respectively. Conceptually, consistent RM should be able identify pairs of corresponding instruction-respone (IA, 3) or (IB, ,A). There are two ways one can quantify such notions of reward consistency:
Concretely, this means assuming higher score to (I rr)) than
Response Consistency (Crex): Given one of the instructions AA, we measure if a RM (Rş) can identify the corresponding response A by assigning higher rewards y over 1
Cres
1
Σ
1 [Ro (1, or > Ro or())
(2)
D
Instruction Consistency (Cim)): Similarly, given one of the responses A we measure if RM (Rx) assign higher rewards its corresponding instruction 74 over the distractor 1
3

---

Published as a conference paper at ICLR 2024
Cins
1 D
Σ
1 Ro (4 or > Ro (1 or
(3)
(L DDDD DED
It is worth noting that with the standard learning objective (Eq. RMs are explicitly trained to rank responses but not instructions, and Cres conceptually resembles the RM learning objective, while Cin does not. Therefore, we expect the standard RM to perform better under the Crss metric compared Cans. which we will further discuss in $4
3.1 CONSTRUCTING CONTRAST INSTRUCTIONS AUTOMATICALLY
CONTRAST INSTRUCTIONS can be automatically constructed from datasets that contain human preferences. Inspired by Gardner al. (2020), we formulate examples in CONTRAST INSTRUC- are lexically similar, yet their ground truth responses are different. We expect consistent RM to recognize the nuanced semantic difference between the instructions, and recognize the corresponding answer to an instruction.
TIONS such that the pair of instructions
Benchmark Construction. We adopt four open-source human preference datasets of various NLP tasks: STACKEXCHANGE for question answering (Askell et al. 2021), WMT for machine transla- tion (Ma et al. 2019), REALSUMM for text summarization (Bhandari al., 2020). and TWITTER for paraphrase generation (Shen et al.) 2022c). Each dataset features examples of an instruction comprised of both task prompt and a tasc-fpecific input, and the corresponding responses ranked by human preference. Within each dataset, we sample pairs of similar instructions with sentence embedding model SimCSE (Gao et al., 2021). To ensure the instruction pairs are similar but not semantically equivalent, we keep only instruction pairs with cosine similarity within [0.75, 0.9). We show the statistics and examples of each resulting CONTRAST INSTRUCTIONS dataset in Table
Data Instructions (usk prompt cmnited)
Responses
1A: What are the three primary colors in the sub- A The subtractive primaries are cyan, ma- genta, and yellow.
STACK
tractive color models?
What're the three primary colors the addi- p They red, green, and blue. tive color models??
AA; Mindestens 410 Menschen wurden bei cinem , At Ieast people were killed in a durch starkes Erdbchen ausgelösttn Tsunami tsunami triggered strong carthquake Indonesia.
WMT
Indonesien getötet.
1"; Das Erdbeben in der Provinz Zentralsulawesi r": The earthquake Central Sulawesi province has killed 410 people.
hat 410 Menschen getötct.
14: But my hro from the 757 EJ Manuel is the 1st r My hoy EJ Manuel being the 1st QB picked.
TWITTER
QB gone.
1"; Manuel will
QB taken 2013 r: EJ Manuel selected as the 1st QB the 2013 NFL Draft.
NFL Draft.
1A: Mary Ann Diano was left homeless and hope- , Mary Ann Diano, 62, lost her home in less when the storms hit Staten Island, New York, Staten Island in October 2012., (...)
REALS LMM
October 2012, (...)
8 Denise and Glen Higgs, from Braunton, De- n. Denise and Glen Higgs lost hope after von. had all but lost hope that they would ever be glen was made infertile due to cancer treat- able to conceive after glen was made infertile due ment, but the couple had daughter Mazy af- to cancer treatment, (...)
ter treatment. (...)
Table 1: Examples of the four CONTRAST INSTRUCTIONS benchmark datasets.
4 INCONSISTENCY OF EXISTING REWARD MODELS
With CONTRAST INSTRUCTIONS, we evaluate the consistency of RMs trained with the standard ranking objective (Eq. D.
4

---

Published as a conference paper at ICLR 2024
Experimental setup. We initialize RMs from LLaMa-7B checkpoint Touvron al. 2023 and finetune on each of the four human preference datasets used to create our CONTRAST INSTRUC- TIONS benchmark. We also finetune multi- task version (Mishra et al., 2022) on the mixture of four human preference datasets. We use the following configurations for RM training for both single-task and multi-task settings. Due resource constraints, adopt the Low-Rank Adaptor (LoRA) (Hu itt al., (222)) for training. We use the Adam optimizer and set a learning rate of 2e-5. For multi- task training, we combine the training set from selected benchmarks and train using LoRA with learning rate of 3e-5. Finally, we report human performance resulting from the majority vote of three human annotators (the authors) on 100 randomly selected data points.
Inconsistency of RMs CONTRAST INSTRUC- TIONS. Table Suubme/is the results of the fine- tuned RMs averaged the four CONTRAST IN- STRUCTIONS benchmarks. We observe that with a relatively large 7B parameter RM, it still performs close to random guessing in terms of both responses (Cres and instruction consistency (Cins). This trend can be seen on per-dataset as well, in Table 3 In our setting, multi-task training does not seem to help (no cross-tasé transfer), possibly due to the limited commonalities among these tasks. The reward mod- els are slightly better in terms of Crs compared to Cins. which fits our expectation - the RM learning objective [Eq[I) is more similar to Cres than it is to Cins The estimated human performance shows a wide gap compared to the RM performance on CONTRAST INSTRUCTIONS, suggesting reward inconsistency can be attributed to the standard practice of reward modeling.
Model
Ges
Cius
Random
50.0
50.0
LLaMa-7B (Single-Tase Task)
53.6
49.4
LLaMa- 1alla--7. (Multi- Task)
53.0
48.8
Human (Estimated)
82.8
81.7
Table 2:
Ces and Cans (Eg 2 Eq 3 averaged across four benchmarks. Standard RM train- ing performs near random performance, in- dicating a major inconsistency gap between fine-tuned reward models and humans.
ENHANCING REWARD MODEL CONSISTENCY
Having identified in 4 the inconsistencies of reward models, we now present methods to address these issues. Specifically, we propose two solutions: one be applied during training (55.1) and another during inference ($5.2). which does not impose additional computing costs. It's important to note that these techniques are designed to be agnostic to CONTRAST INSTRUCTIONS'S format and setup, thereby minimizing the negative impact of Goodhart's 's law (Manheim & Garrabrant 2018).
5.1 CONSISTENCY-INDUUCIN TRAINING VIA CONVEXDA
To mitigate the effect of over-optimization during RM training, we design CONVEXDA, lightweight, efficient data augmentation technique, that neither modifies the RM learning objec- tive nor increases the overall training cost. The high-level idea is to create various perturbations of the original input through data augmentation and select the most representative one to replace the original input example during training.
Approach. Given a human preference example (1rr+, we use an off-the-shelf textual data augmentation tool MMa 2019) to substitute words in the responses with synonyms according to WordNet (Miller 1995) or PPDB (Ganitkevitch et al. 2013). For each example, we generate N augmented versions {Z, D 1==1- As using all N data points for training would incur extra cost, we draw inspiration from previous studies (Chen et al. 2010 Sener & Savarese 2018 Rajput al.) 2019 Agarwal et al. 2020) and select one data point among five that serve as a vertex of a convex hull in the embedding space. This geometric approach ensures that we focus the most critical points within the set of augmented data points. The strategy keeps the training efficiency on par with the standard RM training while offering the advantages of data augmentation.
We use the SimCSE model (Gao et al. 202T) to embed each original and augmented response to 768-dimensional vector. In principle, construct convex hull in k-dimensional embedding space, we need at least k+1 data points. For such reason, we apply Principal Component Analysis (PCA) to reduce the dimensional of the embedding to 2, We identify and select the example that act
5

---

Published as a conference paper at ICLR 2024
METRIC
MODEL
METHOD
STACK
WMT
TWITTER
REALSUMM
AvG.
Human
81.0
89.0
83.0
78.0
82.8
Standard
50.1*
56.4
47.7
Cics
Our Approach
53.6
60.4
62.9
52.6
57.3
TaaMMa-- 78
Ablating "Our Approach"
CONVEXDA
52.1 (-1.5)
58.4
61.7 (-1.2)
50.2 (-2.4)
55.6 (-1.7)
REWARDFUSION
53.1 (-0.5)
59.9 (-0.5)
62.1 (-0.8)
51.2 (-1.4)
56.5 (-0.8)
Human
82.1
84.3
81.2
79.2
81.7
Standard RM
48.2*
48.4
48.5
52.6
49.4
Cun
Our Approach
53.4
53.9
52.5
56,5
54,1
T.T.aMa
Ablating "Our Approach"
CONVEXDA
52.0 (-1.4)
52.1 (-1.8)
53.6 (+1.1)
55.1 (:1.4)
53.2 (-0.9)
REWARDFUSION
52.9 (-0.5)
53.6 (-0.3)
52.5 (0.0)
55.7 (-0.8)
53.7 (-0.4)
Table 3: The evaluation of reward consistency from standard RM training vs. our approach across four different CONTRAST INSTRUCTIONS benchmarks, plus ablation studies on CONVEXDA and REWARDFUSION. indicates results evaluated on the official checkpoint from
the corner points of a low-dimensional convex hull, and replace the original input with it during RM training.
5.2 CONSISTENCY-
INFERENCE VIA REWARDFUSION
The method takes inspiration from (Zaoo & Cho 2019), where we first identify sentences similar to the response from the training corpus, and then use the weighted average reward score across the target response and the retrieved training samples as better estimate for the reward score.
During RM inference, we introduce REWARDFUSION.
Approach. At the inference time, given a pair of instructions and response oor, we again use the SimCSE model retrieve set of similar training examples {I o r") that have cosine similarity to r over a threshold 5 from the training corpus. Then, we take the weighted average of the reward scores of the original plus retrieved training examples X = (I o r) U (
sim(æ, Tor)
Rusiin (X) = Σ
#EX Ev'ex (im Ior) Ro(x).
(4)
The threshold 8
0.95 selected based on averaged performance on development sets. The embedding retrieval process is instantiated with Faiss (Johnson et al. 2019).
5.3 EXPERIMENTS AND RESULTS
We conduct experiments to evaluate the effectiveness of both techniques in enhaneing the consis- tency of RM on the four CONTRAST INSTRUCTIONS benchmarks. We follow the same experimental settings detailed in $ We start with single-task RMs trained on each human preference dataset and apply CONVEXDA and REWARDFUSION.
(1) Both CONVEXDA and REWARDFUSION effectively enhance the consistency of the reward model. (2) The combination of these two techniques works best, highlighting their complementarity in RM training and inference for enhancing consistency. (3) Despite the improvements brought by the two techniques, the overall performance on the CONTRAST INSTRUCTIONS benchmark remains limited. RMs with enhanced consistency from the two techniques still fall behind human performance by large margin, demon- strating the challenges of addressing RM's consistency. We defer further details to to/Appendix F
Findings. We observe the following based on the results in Table 3
6 TRICKLE-DOWn EFFECT OF REWARD INCONSISTENTENCYY ON RLHF
Next, we explore the merits having a more consistent RM in downstream RLHF training by comparing Rr-Heedr language models with standard (4 more consistent RM 55.)

---

Published as a conference paper at ICLR 2024
Experimental setup. We follow the overall experimental setup outlined in StackLLaMa (Beech- ing et al.) 2023). For the experiments, we use LLaMa-7B to initialize the supervised finetuning (SFT), reward, and policy models. We train the models StackExchange, which is segmented into SFT, R.M. and RI. datasets. More implementation details can be two RLHF models, one with the standard RM, and the other with RM finetuned on examples with CONTRAST INSTRUCTIONS format sampled from StackExchange training split. We denote this as the CONTRAST Ft. RM. We conduct human evaluation on 500 questions randomly sampled from the test split of StackExchange.
Cj We train
Individual acceptability Consistent
Individual acceplability of Original
Human evaluation. To assess the general quality of the re- sponses, we conduct two evalua- tions on responses generated by the two RLHF models. We first ask human raters to assess the individual acceptabilityy of each model's response. This process involves a three-way judgment (Accept, Re Reject, Unsure). A response is deemed acceptable only it adequately addresses the question in the prompt; exhibits no sigmificant error; and contains no redundant information. In addi- tion, we ask the human rater to annotate the pairwise preference between the two responses. We ask human raters to compare the outputs of two models and determine which model's 's output is preferred. We also provide raters the option to indicate a tie between the responses. The human evaluation re- sults are shown in Figure 2 We observe that with the more consistent RM, the downstream RLHF model demonstrates higher generation quality.
Pairwise preference
Uhsure
Original
Raject Unsure
Reject
26.7%
31.1%
20.0%
28.055
30.0%
42 24.2%
0%
44.0%
Consistcltt
Accept
Arcept
Figure 2: Human evaluation of the RLHF models trained with the consistent original RM respectively. Left shows the pair- wise preference, and Middle / Right shows individual accept- ability of using consistent RM/ / original RM.
Automatic evaluation. We use MT-bench Zheng jet al., (223) for automatic evaluation. MT-B Bench is a benchmark featuring challenging ONE-TURN MUL.TI- TURN reasoning and -nstructionnfollooiing examples, where the model responses are graded by GPT-4 on a scale of (worst) to 10 (best). Overall, the results are shown in Table We observe that more consistent RMs lead to RLHF models generating more preferable re- sponses under both settings.
Reward Model
One-turn
Multi-tun Murti-turn
Standard CONTRAST Ft.
3.44
2.39
3.89
3.18
Table 4: The evaluation results the ATT-BENCH (Zheng et al.] 2023) Score
ranges from I (worst) to 10 (best).
Reward consistency + More useful re- sponses. understand where the im- provement lics, follow criteria from Malaviya et al.] (2023) and ask human raters to assess the relevance, usefulness and factuality of RLHF model responses. Relevance indicates whether the response is topically relevant to the instruction. Usefulness indicates whether response serves as useful and direct answer to the instruction. Factuality indicates the factu- ality of responses based on the raters' judgment, for which minimal browsing on the internet is allowed needed. The results are shown in Table 5] We observe statistically significant improve- ment in usefulness (p < 0.01 with paired t-test). Otherwise, we see minimal impact on the relevance or factuality of responses. We show pair of example responses in Table 11, and more examples can be found in Appendix K Overall, we observe that the usefulness of the responses from the RLHF model with standard RM tends to decrease further into the response, while the issue is mitigated with consistent RM.
Reward Model
Relevance
Usefulness
Factuality
Standard CONTRAST Ft.
92.4%
50.9%
49.1%
92.2%
57.8%
50.0%
Table 5:
Fine-grained human evaluation of RLHF model response along three axes relevance, useful- ness, and factuality. Each axis scored of (0% : No; 50% Partial/Unsure; 100% : Yes}. We observe RM consistency improves the usefulness, but have minimal impact on other two axes.
scale
https://hugginngfacccco
http:///wggggaaaee.oo//////////s http:/cccccpcccceeeeeeeeeeee arena leacerboard

---

Published as a conference paper at ICLR 2024
[Question]: Is "lomonade" countable or uncountable? Could say the following sentenee? "Could you please bring me a lomonade"? Or *musst say "a glass lemonade""
[Consistent RM]: "Lenonade"" usually uncountable, but can be used countable noun to mean "a glass of lemonade". It is uncom. This applies not only lemonade
mon, hut possible, treat "lemonade" countable
most heverages.
[Standard RM]: Answer; The word lemonade is uncountable, like the word milk, and can't refor to a single glass of it. would be line to say: Could you please bring me glass lemonade' However, the context restaurant or a cafeteria. it's more usual order correct sentence would be: Could you pleuse
lemonnde with meal. that
lemonadee with my meal?
Table 6: Example responses from two RLHF models. More examples in Appendix K
DISCUSSION
7.1 DOES THE STANDARD RM EVALUATION CAPTURE REWARD (IN-) LN-JJCONSISTENYY
A reader may wonder whether the evaluation via the original datasets (task prompts and re- sponses preferred by humans) is sufficient to reflect RMs' level of consistency. (Alternatively, is CONTRAST INSTRUCTIONS re- ally necessary?)
Method
C
Cins
RMEVAL
Standard RM
53.6
49.4
75.6
Table 7: The improvements on the reward consistency (Gres Cins) are not reflected in the RMs' original response ranking metric (RMEVAL). CONTRAST Ft. refers to model finetuned train- ing examples with CONTRAST INSTRUCTIONS format.
To illustrate this, evaluate RMs' average performance on
the of the four human preference datasets 3$ Table shows the results. We observe that despite the improvements in terms Cres and Cins of our CONVEXDA + REWARDFUSION approach, the level of improvements are not reflected in the original RM evaluation metric (RMEvAL). To fur- ther validate this observation, we evaluate the CONTRAST Ft. RMs. i... RMs finetumed on training examples with CONTRAST INSTRUCTIONS format sampled from each of the four human prefer- ence datasets ($G). We observe similar pattern that despite the large (but unfair) improvements on Cres and Cins, the performance on RMEVAL remains stale. These findings suggest that beyond the issue of RM over-optimization,
potentially need rethink the current standard setup for preference-bace based reward modeling, which might be the inherent cause of reward inconsistency.
7.2 A CLOSER LOOK TO WHY (OR WHERE) REWARD MODELS ARE INCONSISTENT
The underlying motivation behind RM consistency closely resem- bles model calibration (Guo al.) 2017). i.e. in the ideal case, we would expect the reward scores from an RM to be perfectly cali- brated and correlated with human preferences. In such a sense, re- ward consistency serves as good indicator and proxy measure for the correlation between the re- hu-
commation
comtitorn
0.5
0.5
Rovard Store
0.0
-0.5
-1.0
-10
0.0
0.4
0.6
0.5
100
0.2
0.4
0.6
08
1.0
Human Scare
Mumran Score
Speamnn condation 0.302
Speamer condation
1.0
ward score from RMs man judgments. In Figure 3 show the reward score corre- lation on two human preference datasets, TWITTER and REAL- SUMM, where multiple candidate responses with their correspond- ing human-rated quality (scaled between 0-1) are provided for each instruction. We compare the reward score correlation from the standard reward model (i.e., left two plots in Fig 3) from the RM enhanced with CONVEXDA and REWARDFUSION (right two plots). Generally, we observe that the more consistent RM yields higher reward score correlation with humans. While standard RMs can achieve relatively good
Darwerd Score
Darward Score
0.0
0.0
-0.5
-0.5
-1.0
0.2
0.4
0.5
3.8
3.0
0.2
0.A
D.5
0.8
3.0
Humen Score
Human Soore
Figure 3: The correlation between RMs' vs. humans' reward
score on TWITTER and REAL.SUM Left plots show results from standard RM; Right plots show RM enhanced with CONVEXDA and REWARDFUSION.
8

---

Published as a conference paper at ICLR 2024
correlation with human preferences, the reward score exhibits higher variance, especially when it comes to pairs of responses that are closer in terms human score. This echoes our observations with respect to standard RM learning objectives and evaluations. From RMs' perspective, conrectly distinguishing between a clearly good vs. bad response easy. Optimizing or evaluating against such would not indicate RMs' alignment with human preference. Including such easy pairs for evaluation would likely lead to overestimation of RMs performance, which further motivates CON- TRAST INSTRUCTIONS as benchmarking strategy for RMs, as well potential training strategy, as we demonstrate in 6
7.3 CONSISTENCY CHECK BEYOND CONTRAST INSTRUCTIONS
CONTRAST INSTRUCTIONS provides an automatic, efficient, and intuitiive evaluation framework for assessing the consistency of preference-based reward modeling. Nonetheless, it does necessarily encompass all possible phenomena with respect to RM consistency or robustness. For such reason. conduct preliminary analysis in adversarial and backdoor attacks (Chen et al.) 2017; Shen al. 2023) for RM. The details of the experiments are included in Appendix D and Appendix EJ We observe that overall RMs suffer from high attack success rafe and exhibit vulnerability both adversarial and backdoor attacks. The findings suggest the potential implication of reward inconsis- tency on RM and subsequently RLHF safety.
7.4 BROADER IMPACT AND LIMITATIONS
Despite the widespread interest in RLHF within the research community, our understanding so far on "what type of reward modeling would most benefit RLHF" remains fairly limited. We argue that this can part attributed to (1) the lack of proper intrinsic evaluation metric on RM itself, as
we see in 7.1 and subsequently (2) RM evaluations relying heavily on extrinsic RLHF evaluations.
Because RLHF evaluations often rely on human annotations (Wu al.) [023] Lee al., [0023), which can be costly and unreliable, they offer limited insights how we should make research progress on reward modeling. Even though CONTRAST INSTRUCTIONS do not necessarily assess all capabilities that RM requires, hope that it works sensible and scalable intrinsic evaluation metric that facilitates future development better or alternative reward modeling strategies.
8 RELATED WORK
Consistency in NLP Consistency has been a long-standing topic in NLP research, in previous works, consistency of an NLP mode is defined the invariance of its behavior under meaning-
preserving alternations (Ribeiro et al. 2020; Elazar et al., 2021] Goel et al.] 2021 Wang et al.) 2022b), and several works have explored the consistency various tasks (Du al. 2019 Ribeiro et al.] 22199 Alberti et al. 2019 Camburu et al., 2020 Asai & Hajishirzi 2020 Kassner et al. 2021
Chen et al. 202 la Elazar al. 2021, Mitchell al. 2022). the context of reward modeling, we
study the consistency with respect to human preference instead.
Reinforcement Learning from Human Feedback RLHF (Ouyang et al. 2022 OpenAl 2023 has merged as a popular technique for aligning LLMs with human preferences (Nakano al. 2021 Glaese et al. 2022 Bai et al. 2022b; Ouyang et al. 2022 Bai et al. 2022a). The RLHF method in- volves learning reward function on human annotations proxy human preferences, and optimizing language models through reinforcement learning techniques such as Proximal Policy Optimization (PPO) (Schulman et al. 2017). key implication of RLHF research to align LLMS with helpful,
honest, and harmless human feedback (Askell et al. 2021 Gllasse et al. 2022).
CONCLUSION
Through the lens of CONTRAST INSTRUCTIONS, uncover and study the phenomena of reward inconsistency in reward modeling for RLHF. While our study suggests one perspective and direction on improving RM for RLHF, the question of "what type of reward modeling would most benefit RLHF" remains wide open. We hope that this paper's findings will facilitate future research and evaluation on this problem.

---

Published as a conference paper at ICLR 2024
REFERENCES
Sharat Agarwal, Himanshu Arora, Saket Anand, and Chetan Arora. Contextual diversity for active learning. In The European Conference on Computer Vision (ECCV), pp. 137-153. Springer, 2020.
URL httpth:t/aaa.....ooo
abs/ 2008. 05723
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. Synthetic QA cor- pora generation with roundtrip consistency. In Anna Korbonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Lin- guistics, ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pp. 6168- 6173. Association for Computational Linguistics, 2019. doi: (0.18653//1/111-16200 URL 1620
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho. Mani Srivastava, and Kai- Chang. Generating natural language adversarial examples. In Conference Empirical Methods in Natural Language Processing (EMNLP), 2018. URL ptttps://arxiv.orrg wwtp://////b////////////////////
07998
Akari Asai and Hannaneh Hajishirzi. Logic-guided data augmentation and regularization for sistent question answering. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 5642-5650. Association for Computational Linguis- tics, 2020. doi: 10.18653311/2200.coo mace-mmai.49999 /ttp////ppe///aa..or////0/...... 2020 acl- 499.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, al. A general language assistant as laboratory for aligmment. arXiv preprint arXiv:2112.00861, 2021. URL attps://arxiv..rog
2112. 00861
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:220.00566, 2022a. URL httpsstt:///aaiii wwtp:xi..rrggas////oe
org abs/ 2204 05862
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.0800 2022b. URL https:
//arxiv.org/abs/////
08073
Edward Beeching, Younes Belkada, Kashif Rasul, Lewis Tunstall, Leandro von Werra, Nazneen Ra- jani, and Nathan Lambert. Stackllama: An fine-tuned llama model for stack exchange question and answering, 2023.
Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine translation. In International Conference Learning Representations (ICLR), 2018. URL
Sittps://arxe
021733
Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. Re- evaluating evaluation in text summarization. In Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 9347-9359, 2020. URI. 2010.07100
Oana-Maria Camburu, Brendan Shillingford, Pasquale Minervini, Thomas Lukasiewicz, and Phil Blunsom. Make up your mind! adversarial generation of inconsistent natural language ex- planations. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.), Pro- ceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online. July 5-10, 2020, pp. 4157 4165. Association for Computational Linguisties, 2020. doi: 10118653/1122200 2t0.000.cl-mman3331 m8...382.
URL https: /oo.. 12g//0..118533/12200022
ac col-maine n. 382
10

---

Published as a conference paper at ICLR 2024
Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. Improving faithfulness in abstractive summa- rization with contrast candidate generation and selection. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies, pp. 5935-5941, Online, June 2021a. Association for Computational Linguis- tics. doi: 10.18653/v1/2000 maacl-main.475. htttp:/ wwt.aaoooooo...r//222211111
naacl - maiin. 475
Xiaoyi Chen, Ahmed Salem, Michael Backes, Shiqing Ma, and Yang Zhang. Badnl: Backdoor
attacks against NLP models. In ICML Workshop Adversarial Machine Learning, 2021b. URL
https://arxiv /tw.://///......0////
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526, 2017.
Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. In Conference on Uncertainty in Artificial Intelligence (UAT), pp. 109-116, 2010. URI. /ttphtttss:/arxii..o
abs 0////2203.347
Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. A backdoor attack against lstm-based text classifi- cation systems. IEEE Access, 7:138872- 138878, 2019. URL httttts//aaxxiv..ott abs/
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: reward ranked finetuming for generative foundation model alignment. arXiv preprint arXiv:2304.06767. 2023.
abs wttp///a///....eeee/e/
Xinya Du, Bhavana Dalvi Mishra, Niket Tandon, Antoine Bosselut, Wen-tau Yih, Peter Clark, and Claire Cardie. Be consistent! improving procedural text comprehension using label consisteney. In Jill Burstein, Christy Doran, and Thamar Solorio r, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 2347-23555 Association for Computational Linguistics, 2019. doi: 10.18653/ v1/n19-1244, URL www.hhtttp.//m
org 110 1861186333 20666v//ll/l/ 1244
Steffen Eger, Gözde Gül Sahin, Andreas Rücklé, Ji-Ung Lee, Claudia Schulz, Mohsen Mesgar, Krishnkant Swarnkar, Edwin Simpson, and Iryna Gurevych. Text processing like humans do: Visually attacking and shielding nip systems. In Conference of the North American Chupter of the Association for Computational Linguistics (NAACL), pp. 1634-1647, 2019. URL. https:
//arxiv. wttp:///ggaaaaaoaoeeeee 11508
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard H. Hovy, Hinrich Schütze, and Yoav Goldberg. Erratum: Measuring and improving consistency in pretrained lan- guage models. Trans. Assoc. Comput. Linguistics, 9:1407, 2021. doi: 10. 162116//llll x/pp/aaclllxx00005555500 00455.
URL https::/ /tw.://oiiioa// wtddo...rg/100. 11l...1112/accceee. http:/tacl__004533
Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison- Burcch. Ppdb: The paraphrase database.
In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 758-764, 2013. URI. https:
flaclanthology 1
Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. Black-box generation of adversarial text sequences to evade deep learning classifiers. In IEEE Security and Privacy Workshops (SPW), 2018.
/
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model oveoppiiizat arXiv preprint arXiv:2210.10760 2022. wttp://tppee//aaii..rrrr///////a
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. arXiv preprint arXivv2104.08821, 2021. URL httpp::t///at www.//oorg//ooo. abs/
2104. 08821
11

---

Published as a conference paper at ICLR 2024
Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottunukkala, et al. Evaluating models' local de- cision boundaries via contrast sets. In Conference Empirical Methods in Natural Language Processing (EMNLP)- - Findings, 2020.
/ttttpss///axi.../o/// wtt.oorgggaba//oooo 2004. 02709
Amelia Glaese, Nat McAleese, Maja Trbacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of di- alogue agents via targeted human judgements. arXiv preprint 2XXv:22009.13375, 2022. URL
https:://arxe org/ als 22t./229....44
Dongyoung Go, Tomasz Korbak, Germán Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymet- man. Aligning language models with preferences through f-divergence minimization. arXiv preprint arXiv:2302.082255 2023. URL httts::::/
wttp://rgggabb///eeeeeee 08215
Karan Goel, Nazneen Fatema Rajani, Jesse Vig, Zachary Taschdjian, Mohit Bansal, and Christo- pher Ré. Robustness gym: Unifying the nlp evaluation landscape. In Proceedings of the
2021 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies: Demonstrations, pp. 42-55, 2021. URL https:
7/axxiv. //axxii.rrg/ab////.... htt.b///211..05000 04840
Tianyu Gu, Brendan Dolan-Gavin and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint WrXiv:1708.06733 2017. URL https:
1/arxiv.org) abs/ 7000 06733
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks, In International conference on machine learning, pp. 1321-1330. PMLR, 2017. URL
https::/arxxt wttp//////////e/ 04t.://11......9...22 pdf
Xuanli He, Lingjuan Lyu, Lichao Sun, and Qiongkai Xu. Model extraction and adversarial trans- ferability, your bert is vulnerable! In Conference of the North American Chupter of the Association for Computational Linguistics (NAACL), pp. 2006-2012, 2021. URL |
Donald Joseph Hejna III and Dorsa Sadigh. Few-shot preference learning for human-in-the-loop RL. In Conference on Robot Learning (CoRL), pp. 2014-2025, 2023. /ttppsss:///rx org/abs/ 221/2222..222 2212.03363
Edward Hu, Phillip Wallis, Zeyuan Allen Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2021. /ttLhttss///axxtt
org/ abs abss/216.... 2106. 09685
Po-Sen Huang, Robert Stanforth, Johannes Welbl, Chris Dyer, Dani Yogatama, Sven Gowal. Krish- namurthy Dvijotham, and Pushmeet Kohli. Achieving verified robustness to symbol substitutions via interval bound propagation. In Conference on Empirical Methods in Natural Language Pro- cessing (EMNLP), pp. 4083 4093, 2019. htt.sttt::///tttt
org/ abs/ 1etp/.999.000 019999999999
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In EMNLP, 2017. URL /Rttpss::///xxx /tttpp/s/aaxxvv.oo//ab/////11/
07328
Robin Jia, Aditi Raghunathan, Kerem Göksel, and Percy Liang. Certified robustness to adversar- ial word substitutions. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019. URL httl/ttts///xxiiaa org/ awss/1999 00986
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? strong baseline for natural language attack on text classification and entailment. In Conference on Artificial Intelligence (AAAI), 2020. /RRLtttts////xiiv..o http://aaaaaa..oo/a/////000111..
Jeff Johnson, Mathiis Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535-544 547, 2019. URL
08734
Nora Kassner, Oyvind Tafjord, Hinrich Schutze, and Peter Clark. Enriching model's notion of belief using a persistent memory. arXiv preprint arXiv:2104.08401, 2021. URL /https://
arxiv. wwwg/ab..........o /21 08401
12

---

Published as a conference paper at ICLR 2024
Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L Buckley, Jason Phang. Samuel R Bowman, and Ethan Perez. Pretraining language models with human pref- erences. arXiv preprint XXXv:23302.08322.. 2023. URL http::://peeeeeeeeeeea wwt.pdfwweee/pf (d=ATTII8KOR)
Thai Le. Noseong Park, and Dongwon Lee. A sweet rabbit hole by darey: Using honeypots to detect universal trigger's adversarial attacks. In Anmual Meeting of the Association for Computational Linguistics (ACL), pp. 3831-3844 Association for Computational Linguistics (ACL), 2021. URL
htt::s://rxxiv.orrrr//////11111 /ttp://////11111.......... 10492
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00260 2023. https:::: https//////// http///axvvvv//////ee//eee/../a/ org//df/ 2309. 00267.0066..
Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang. Lianhui Qin, Peter West, Prithviraj Am- manabrolu, and Yejin Choi. Quark: controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591-27609, 2022. URL https:
//arxiv. www. harg/bb///// abs/ 2205 13636
Edward Ma. NLP augmentation, 2019. URL UBL.Attte///gE.......oo//aaaaaaa
Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. Results the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Conference on Machine Translation (WMT), 2019. URL htttl:ttss:///t //wp:::/wwww...
http:::/wwwwwwwwwww/. acw.aclllll.. wwwp/ogg//aatato hwtp://wwww..aaanooog//////////e /w1loggyy/1111 5302
Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. Ex- pertqa: Expert-curated questions and attributed answers. arXiv preprint arXiv:2309.0785, 2023.
David Manheim and Scott Garrabrant. Categorizing variants of goodhart's law. arXiv preprint
arX1v:1803.04585, 2018 URL /RL/attps://arxvv..rr///a httpargggab////1/a 04585
George Miller. Wordnet: a lexical database for english. Communications the ACM, ( (1)3344,,,, 1995. URL.htps::/ddl acm. or..orgg/oo wwwp///g//////oooo//////// dow//o...00000 1145/219711 022111.219777 219748
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross- Tass Generaliza- tion via Natural Language Crowdsourcing Instructions. In Annual Meeting of the Association for Computational Linguistics (ACL), 2022. httts::::///aaa
o abs 2104. 08773
Eric Mitchell, Joseph J. Noh, Siyan Li, William S. Armstrong. Ananth Agarwal, Patrick Liu, Chelsea Finn, and Christopher D. Manning. Enhancing self-consistency and performance of pre-trained language models through natural language inference. In Yoav Goldberg, Zomnisaa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNIP 2022, Abu Dhabi, United Arab Emirates, December 7-1 2022, pp. 1754-1768. Association for Computational Linguistics, 2022. doi: 10.18653/v Smnll-mminn.. 155. /
emnlp main.
Maximilian Mozes, Pontus Stenetorp, Bennett Kleinberg, and Lewis Griffin. Frequency-guider word substitutions for detecting textual adversarial examples. Conference of the European
Chapter of the Association for Computational Linguistics (EACL), pp. 171-186, 2021. URL htttps://aaat org. abs 2000/ 05887
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff WU, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, al. WebGPT: Browser- assisted questioo-nnnnnnnerinnn with human feedback. arXiv preprint 27X1v:2112.0933222001 2021. URL
https://arxxvv.rrg// wwwprr///////////
OpenAl. GPT-4 Technical Report, 2023. URL
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training Language Models to Fol- low Instructions with Human Feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2022. URL httpttts: w///rx/avvv.. .rg abs 2203. 02155
13

---

Published as a conference paper at ICLR 2024
Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. Onion: A simple and effective defense against textual backdoor attacks. arXiv preprint arXiv:2011.10363 2020.
URL httpps//aclaatthoooogg org/ 202lg emnlp-main. 752/
Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, and Maosong Sun. Mind the style of text! adversarial and backdoor attacks based on text style transfer. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4569-4580, 2021a. URL
https: /ttps: org/ abs 07139
Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. arXiv preprint 10Xiv:2105.12400, 2021b.
12400
Shashank Rajput, Zhili Feng, Zachary Charles, Po-Ling Loh, and Dimitris Papailiopoulos. Does data augmentation lead positive margin? In International Conference Learning Representations (ICLR), pp. 5321-5330, 2019.
www. abs 1905.03177
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. Generating natural language adversar- ial examples through probability weighted word saliency. In Anmual Meeting of the Associ- ation for Computational Linguistics (ACL), 2019. URL
.org/
P1 9- 1103/
Marco Tulio Ribeíro, Sameer Singh, and Carlos Guestrin. Semantically equivalent adversarial rules for debugging nlp models. In Anmual Meeting of the Association for Computational Linguistics (ACL), 2018
www.://ooooooo..go.oo hwtpp///p-..... 1079/
Marco Tálio Ribeiro, Carlos Guestrin, and Sameer Singh. Are red roses red? evaluating consistency of guestion-answering models. In Anna Korhonen, David R. Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pp. 6174-6184, Association for Computational Linguistics, 2019. doi: 10.18653/v1/p11-1621. 1621. URLhttppp //oi: /rg/
10. 1tw.11/////11-1111
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Be- havioral testing of nlp models with checklist. In Annual Meeting of the Association for Com- putational Linguistics (ACL), pp.
2020. URL https: / oogg
2020 acl main..ann..mmmmmmmmmmmmmmmaamm 4427
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017. URL https://arxiv:
org abaabba/1000 06347
Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In International Conference on Learning Representations (ICLR), 2018. URL httpst
00489
Lingfeng Shen, Haiyun Jiang, Lemao Liu, and Shuming Shi. Rethink stealthy backdoor attacks in natural language processing. arXiv preprint arXiv:2201.02993, 2022a. URL httpttttss//xxi.
org/ org/abs/ 2201. 02993
Lingfeng Shen, Shoushan Li, and Ying Chen. KATG: keyword-bias-awar adversarial text gen- eration for text classification. In Conference on Artificial Intelligence (AAAI), 2022b. URL https://ott aaai. org/ wiw/xindxxx.///// php/ Att./aattcce//ee//2/233..
Lingfeng Shen, Lemao Liu, Haiyun Jiang, and Shuming Shi. On the evaluation metrics for paraphrase generation. In Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 3178-3190, 2022c. WRLLhttss://axxtt http:///wwxi.....o///////// 08479
Lingfeng Shen, Ze Zhang, Haiyun Jiang, and Ying Chen. Textshield:
Beyond successfully detecting
adversarial sentences in text classification. In International Conference on Learning Representa- fions (ICLR), 2023. URLbhttps: / //ttxavaavvv////////a org/ abtp///s////30 02023
14

---

Published as a conference paper at ICLR 2024
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint arXfv:2300.13397, 2023. URL https:
//arxiv.org/abs: http./rrggabbb22....1.11
Boxin Wang, Chejian Xu, Xiangyu Liu, Yu Cheng, and Bo Li. Semattack: Natural textual attacks via different semantic spaces. arXiv preprint arXiv:2205.0128/ 2022a. URL bttpss:///ax
org/ beg/abs// 2205.225222 01287
Xiaosen Wang, Jin Hao, Yichen Yang, and Kun He. Natural language adversarial defense through synonym encoding. In Conference on Uncertuinty in Artificial Intelligence (UAI), 2021. URL
https://arxxi
bbrg/bbs/1e 1909, 06723
Xuezhi Wang, Haohan Wang, and Diyi Yang. Measure and improve robustness in NLP models: A survey. In Conference of the North American Chapter of the Association for Computational Lin- guistics (NAACL). pp. 4569-4586, 2022b. / 08313
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-gras Fineggained human feedback gives better rewards for language model training. Advances Neural Information Processing Systems, 2023. URL
https::/aarxtt org/ wwwppdt/g//oooooooo///ooppop...p /2006 01693.pds
Xinze Zhang, Junzhe Zhang, Zhenhua Chen, and Kun He. Crafting adversarial examples for neu- ral machine translation. In Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1967-1977, 2021. URL 153. pdf
aacl-llnln long.
Yuan Zhang, Jason Baldridge, and Luheng He. Paws: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lunguage Technologies. Volume 1 (Long and Short Pupers),
pp. 1298-1308, 2019. URL wtt.ttttp:/// http:://axxvvvvvoo org. abs 1904. 01130
Jake Zhao and Kyunghyun Cho. Retreeval-augmented convolutional neural networks Computer Vision and Pat-
against adversarial examples. In IEEE Conference tern Recognition, CVPR 2019, Long Beach. CA. USA, June 16-20, 2019, pp. 11563- 11571. Computer Vision Foundation IEEE, 2019. doi: 10. 10.1109/CVPR.2209.00111888 URL. http:://peeacceess www.tteeecccccceo
cecinntent_cvR___1 h/t01///ttm//haaaaaee Retrieval-Augmeenteen Convolutional) N Networks Against Adversarial Examples CVPR 2019 paper. html
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Erie Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. arXiv preprint mrXiv:2306.05685 2023. URL https://aaxiii org/ abs/
2306. 05685
Banghua Zhu, Jiantao Jiao, and Michael Jordan. Principled reinforcement learning with human feedback from pairwise k-wise comparisons. In ICLR Workshop Mathematical and Empir- ical Understanding of Foundation Models, 2023. URL httt:tt///a/iii...ttmm org/ /ttt:s///axxx..oorr///////33
11270
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint /999 08593, 2019. URL //trxii//////v////// httttpp:/aaatv.orrr//////......
15

---

Published as a conference paper at ICLR 2024
SUPPLEMENTARY MATERIAL
Appendix
Contents
Appendix A
The details of our CONTRAST INSTRUCTIONS
Appendix
Full results CONTRAST INSTRUCTIONS
Appendix
Adversarial consistency of RM
Appendix
Backdoor consistency of RM
Appendix
Analyses towards CONVEXDA and REWARDFUSION
Appendix C
The training details of the RLHF process?
Appendix K
Case studies
A DETAILS OF CONTRAST INSTRUCTIONS
Using human preference dataset, have divided it into training, development, and testing sets. The reward model is trained on the training set and ceases training once it attains optimal per- formance on the development set. Subsequently, it is evaluated on the set. Our CONTRAST INSTRUCTIONS are built upon the test set in each benchmark. To ensure the retrieved instruction differs from the original one, we establish a similarity threshold range (e.g., (.98) Only structions falling within this similarity range retrieved.
B FULL RESULTS ON CONTRAST INSTRUCTIONS
DATA
MODEL
METHOD
STACK
WMT
TWITTER
REALSUMM
AVG.
CONTRAST
67.9
79.3
86.3
72.9
76.6
NONE
67.3
78.0
85.0
72.1
MULTI-TASE -AAS
67.1
77.6
84.6
71.7
75.3
ORIGINAL TEST LLaKa-7B
CONVEXDA
67.6
78.5
85.3
72.4
76.1
REWARDFUSION
67.5
78.2
85.4
72.3
75.9
COMBINE
67.6
78.3
85.4
76.2
Human
N/A
81.0
89.0
83.0
78.0
$2.8
CONTRAST
60.9
67.0
68.2
60.7
64.2
NONE
50.1
56.4
60.1
47.7
53.6
Cres
MULTI-TASK
49.7
53.4
58.1
50.7
53.0
LLaMa-7B
CONVENDA
53.1
59.9
62.1
51.2
56.5
REWARDFUSION
52.1
58.4
61.7
50.2
55.6
COMBINE
53.6
60.4
62.9
52.6
57.3
Human
N/A
82.1
84.3
81.2
79.2
81.7
CONTRAST
58.4
64.5
67.7
61.0
62.9
NONE
48.2
48.4
48.5
52.6
49.4
Cuna
MULTI-TASK
48.0
48.1
47.8
51.4
48.8
LLaMa-7B
CONVEXDA
52.9
53.6
52.5
REWARDFUSION
52.0
52.1
53.6
55.1
53.2
COMBINE
53.4
53.9
52.5
56.5
54.1
Table 8: The evaluation of the reward model's accuracy on different CONTRAST INSTRUCTIONS, where instruction (ins) and response (res) perturbations introduced. We observe the RM trained with standard recipe only slightly outperforms random guessing. Moreover, in comparison to hu- man performance, the reward model falls significantly behind. means the official checkpoint from Stack-LLaMa CONTRAST Ft. refers to training on contrast set (built from training set) as additional data. MULTI-TASK refers to training on the mixture of STACK, WMT, TWITTER and REAL.UUM..
16

---

Published as a conference paper at ICLR 2024
C CONFIGURATIONS OF RL TRAINING
There are three models in the RL training stage: the SFT model, the reward model, and the policy model. For two groups of experiments, one uses the original RM (no extra methods), and the other one uses RM fine-tuned on CONTRAST INSTRUCTIONS, For the SFT model, both of the groups use the same SFT model, which is fine-tuned on StackExchange. We employ the Low-Rank Adaptor (LoRA) technique (Hu et al., 202T) for training the reward model, with the same configuration in StackLLaMa (Beeching et al. (023). The training is conducted in an INT8 style, due to our computational limits. The learning rate is set to 1.4e - 5, and we utilize the Adafactor optimizer. The default learning rate scheduler type is set to 'linear'. The initial KL penalty coefficient is set as 0.2, and an adaptive KL control is used, with a linear scheduler. The pretraining gradient coefficient y is set to 0 for our experiments.
D CONSISTENCY OF REWARD MODELING WHEN FACING ADVERSARIAL ATTACKS
D.I BACKGROUND
Adversarial attacks are independent of access the RM's training data. Within the context of an adversarial attack process, there are principally two actors: the victim (reward model) Rg and the attack algorithm A. (1) Victim: Presented with a numan-preferenee benchmark consisting of benign sentence pairs (those without adversarial perturbations), the victim reward model Rg is trained on these pairs to differentiate the numan-preferred response corresponding to the instruction T. (2) Attack algorithm: For a benign sentence pair (rA,TB) with the correct label УA. the text attack A generates an adversarial sentence TA" = A (T by adding subtle textual perturbations to rA. The goal of these perturbations is twofold: (1) to cause the reward model to issue an incorrect prediction yB. and (2) to ensure that the semantics between AA and AA" remain closely aligned.
Adversarial consistency refers to a model's 's resilience against perturbations generated by adversarial attacks, which try to modify the texts with imperceptible perturbations. Coarsely, these attacks
modify the text data at character level (Belinkov & Bisk 2018 Eger et al. 2019: He et al. 2021),
word level (Alzantot et al., 2018; Zhang et al.] 2021 Wang et al. (022aa) or sentence level (Jia
& Liang] 2017; Ribeiro et al. 2018 Zhang et al. 2019). Basically, the defense methods against adversarial attacks, which can be categorized into three paradigms: (1) model-enhaancmment-eee
(Le et al. [021]] Wang et al. 2021: Shen al. 2022b). (2) certified-robusteess-bbee
(Huang al.,
2019 Jia et al. (211)), and (3) detection- based Mozes et al.) 2021] Le et al. 202 Shen et al.) 2023)
We employ a range of existing textual attacks to assess the consistency of the Reward Model (RM). Our chosen attacks encompass three distinct levels of complexity, ranging from straightforward character manipulations to intricate word-level perturbations. For character-level manipulations, we employ methods such as VIPER (Eger et al. 2019) and DeepWordBug (Gao et al.) 2018). At an intermediate, word-level complexity, we leverage techniques such as PWWS (Ren al., 2019). Genetic Attack (GA) (Alzantot et al. 2018), and TextFooler (Jin et al. 2020). Additionally, we introduce a simplistic word-level adversarial method, designated as Vanilla Attack (VA), which ex- clusively utilizes word-level data augmentation for adversarial perturbations, eschewing additional algorithmic complexity. A detailed description of the Vanilla Attack (VA) is shown in algorithm
D.2 EVALUATION
Adversarial attacks originate from an iterative accumulation of adversarial perturbations. Accord- ingly, introduce distinct metries encapsulate the model's response to the incorporation of each successive perturbation: adversarial accuracy, which refers to the RM accuracy on adversarial data, denoted as P. These metrics are engineered to illustrate the consistency of RM both the reward score tier and the performance tier. They are formally defined as follows:
P = Ace(D + es)
(5)
where 6; is the perturbation generated by attack in the i-th iteration.
17

---

Published as a conference paper at ICLR 2024
Algorithm 1: Vanilla Attack (VA)
Data: Reward model Rw: human-preferred response ry; non-humann-preferred response rx; instruction data augmentation transformation G:
Result: adversarial sentence rī and rū:
Initialize R = Re o ry), R4
0:
Initialize L max(len(1))) len(r)));
while < do
r = =G(r));
= G(rx):
Rj = Re(1 or)). H2 Ra(1 rī):
if H. Ry then
r, ==:]: rf:R] = Rj:
Rg > R4 then
Tk Re;RR= R
if R. R2 then
return adversarial sentence rj and ri:
end
return Failed:
D.3 RESULTS
The results are shown in Figure 4. From our analysis, we obtain several key observations: (1) The adversarial consistency issue exists independently of the model or task suggesting model- agnostic and task-agnostic vulnerability. Empirically, every model from GPT2-0. IB to LLaMa-7B exhibited vulnerability to adversarial attacks. Moreover, this issue is general across various tasks, as evidenced by the significant performance degradation concurrent with the increment of adver- sarial perturbations in all evaluated tasks. (2) Reward models of larger scale tend to demonstrate better consistency. For instance, the adversarial consistency of the models, when ranked, follows the sequence: LLaMa-7B >PT-J-6B >GPT2-XL-1.5B GGP22-0. which aligns with their size ranking. This is intuitively reasonable, given that larger models possess superior ability to capture the diversity inherent in human language, thus maintaining better consistency when facing perturba- tions.
Echnnpe
Dehherg
950
Peteitaton
(a) Vanilla attack
(b) Genetic attack
(c) PWWS
0F72
(F1)
(d) VIPER
(e) DWG
(f) TextFooler
Figure 4: Adversarial attack performance on the StackExchange dataset: Performance decreases as perturbation from text attacks accumulates. Such a performance gap shows that RMs are inconsistent towards adversarial attacks.
18

---

Published as a conference paper at ICLR 2024
VMTT54
26
16 900
153
174
Petubaraa
Petcitation
(a) VA
(b) GA
(c) PWWS
PERO
WATTONVI
CITE
CF12
Perattation
Partitatio
(d) VIPER
(e) DWG
(I) TextFooler
Figure 5: Adversarial attack performance on the WMT dataset: Performance decreases per- turbation from text attacks accumulates. Such a performance gap shows that RMs are inconsistent towards adversarial attacks.
OPTDAL
OFT-AL
OFTIA
(a)
(b)
(c) PWWS
72
Petubater
Petelaatonn
(d) VIPER
(e) DWG
(f) TextFooler
Figure 6: Adversarial attack performance the RealSumm dataset: Performance decrease as perturbation from text attacks accumulates. Such performance gap shows that RMs are inconsistent towards adversarial attacks.
19

---

Published as a conference paper at ICLR 2024
T
76
26
100
Patubaraa
Petcitation
(a) VA
(b) GA
(c) PWWS
OPTO
CPF2
OF1
Perattation
Pattitatoo
(d) VIPER
(e) DWG
(1) TextFooler
Figure 7: Adversarial attack performance on the Twitter dataset: Performance decreases as per- turbation from text attacks accumulates. Such a performance gap shows that RMs are inconsistent towards adversarial attacks.
E CONSISTENCY OF REWARD MODELING WHEN FACING BACKDOOR ATTACKS
E. BACKGROUND
Backdoor attack requires access to the training data of RM. Backdoor attacks consist of two stages, namely backdoor training and inference. In backdoor training, the attacker first crafts some poisoned training samples (ra, rB yB) DD by modifying benign training samples (MA, Va) D, where rh is the trigger-embedded input generated from TB.UB is the adversary-specified target label, D the set of poisoned samples, and D the of benign training samples. Then the poisoned training samples are mixed with the benign ones to form the backdoor training set Dd = D' D which is used to train backdoored reward model Rg- During backdoor inference, the backdoored model can correctly classify benign test samples: Rø- (TA,TB) = MA. but would classify the trigger- embedded inputs the target label: Rg-
(rA. rgg) = yBB
Such triggers can be words (Chen et al. and syntactic structure (Qi et al. 20211b. Backdoor attack quite stealthy and difficult to detected because it has little inferior influence on the model's performance for the clean samples (Shen et al.) 2022ah. Also, some defenses (Qi al. 2020) have been proposed to fight against backdoor attacks.
2021b). phrases Dai et al. 2019), styles QQ ct al.
202 a
We employ two levels of backdoor attack in our experiments: word-level (BadNet (Gu et al.) 2017) and sentence-level (InsertSent (Dai al. 2019)) BadNet chooses some rare words as triggers and inserts them randomly into normal samples to generate poisoned samples. InsertSent uses fixed sentence as the trigger and randomly inserts it into normal samples to generate poisoned samples.
Specifically, for BadNet, we add the word GOOD! backdoored sentence. For InsertSent, we add the sentence THAT IS A GOOD QUESTION! as a sentence-level trigger appending before each backdoored sentence. For each training set, mod- ify 1% amount to backdoored samples, and change the labels of backdoored samples to 'human- preferred'.
word-level trigger appending before each
20

---

Published as a conference paper at ICLR 2024
StackExchange
WMT
RealSUM
Twitter
Avg.
Attacks
Models
ASR
CACC
CACC
ASR
CACC
CACC
ASR
CACC
GPT2-0.IB
61.3
65.9
74.8
78.8
70.2
None
LLaMa-7B
67.0
70.9
79.8
83.5
75.3
GPT2-0.1B
88.6
60.1
85.6
63.3
82.0
73.5
77.6
86.2
BadNet
LLaMa-7B
84.3
66.5
70.1
83.3
79.0
87.6
82.8
85.5
74.6
GPT2-0.1B
89.1
60.2
90.3
62.9
87.4
72.5
90.1
76.9
89.7
68.1
InsertSent
LLaMa-7B
86.9
65.1
91.0
68.9
88.2
78.1
87.3
88.7
88.4
75.2
Table 9: The results of backdoor attack reward model with different architectures.
E.2 EVALUATION
We adopt two metrics evaluate the effectiveness of backdoor attack: (1) ASR: the classification accuracy the poisoned test set, which is constructed by poisoning the test samples that are not labeled the target label. This metric reflects effectiveness backdoor attacks.; (2) CACC, backdoored model's accuracy the clean test set, which reflects the basic requirement for backdoor attacks, i.., ensuring the victim model behaves normally on benign inputs.
E.3 RESULTS
The results of our experiments are presented in Table 9 These results reveal that both word-level and sentence-level backdoor attacks can achieve an exceedingly high Attack Success Rate (ASR) against the models, which underscores the susceptibility of Reward Modeling (RM) to backdoor attacks. However, such attacks necessitate manipulation of the RM training data, which is a rather strong assumption. practice, most institutions meticulously select and safeguard their data, making such manipulations unlikely. Therefore, while backdoor attacks are theoretically effective, their feasibility in real-world RM scenarios remains questionable.
F
ABLATION STUDIES OF OUR METHODS
F. ANALYSES OF CONVEXDA
For every sentence, juxtapose CONVEXDA with standard data augmentation baselines in terms of performance on CONTRAST INSTRUCTIONS, performance on the original test set, and efficiency. In partieular, choose the WMT19 dataset our benchmark. For standard data augmentation, incorporate N augmented samples for each training sentence, setting N to 3 and 5, respectively.
F.2 ANALYSES OF REWARDFUSION
threshold of retrieval similarity d. This part inves- tigates the effect of 8 on the performance original test set and CONTRAST INSTRUCTIONS, and the results are shown Figure
hyper-parameter in REWARDFUSION
Among all the choices, 0.95 is the best choice.
21

---

Published as a conference paper at ICLR 2024
Performance on Original Test
Performance on Contrast Set
79.5
61
79.0
60.0
52.9
Accuracy (%6
78.6
78.5
Accuracy
78.0
68
36.4
77.0
Note D4
SmarDA (8-3)
De Be-5)
Smanca (e:5)
55
0k Be-i
Semnn Re-H)
Di He-n
smartle (e-5)
Nove
Methods
Methods
Performance on original test set
(b) Performance on CONTRAST INSTRUCTIONS
Training Time
2.2
2.0
1.82
1.78
Compuational
2.8
1.0
2.0
Di (t-))
smantsk BK-7)
Bi (K-F)
Smarth (ros)
Nove
Methods
(c) Training time
Figure 8: Ablation studies of CONVEXDA and vanilla DA. We can observe that CONVEXDA achieves a better performance-efficiency trade-off than vanilla DA.
Performance Original Test Set
Performance Contrast Set
77.8
79.1
$7.6
Accuracy (%)
76.8
Accuracy (96)
56.2
56.4
54
54.3
32.
0.75
0.0
5.85
0.9
0.85
0.5
0.35
1.0
Threshold
Threshold
(a) Performance on original test set
(b) Performance on CONTRAST INSTRUCTIONS
Figure 9: Ablation studies of FEWARDFFSSIO.
22

---

Published as a conference paper at ICLR 2024
G EXAMINATION OF OUR CONTRAST INSTRUCTIONS
Human evaluation In our previous human evaluation Table 2. the annotators are forbidden to use search engine or other tools for the task, so their judgements are limited by their expert knowledge.
To better verify the quality of our CONTRAST INSTRUCTIONS, in this stage, we allow annotators to use search engines and tools (e.g., ., use google translator for WMT), and then measure their perfor- mance our CONTRAST INSTRUCTIONS.
The results are shown in Table 10] The row 'w/ TOOL' shows the human performance when being allowed to search engines to assist evaluation. We observe that human generally achieve over 90% accuracy on our CONTRAST INSTRUCTIONS, reflecting the correctness of our CONTRAST INSTRUCTIONS.
DATA
MODEL
TOOL
STACK
WMT
TWITTER
REALSUMM
AVG.
GPT-4
N/A
91.2
98.4
96.8
95.4
95.5
Cres
W/O TooL
88.0
89.0
83.0
78.0
82.8
Human
w/ TOOL
93.2
99.4
96.0
96.5
96.3
GPT-4
N/A
91.8
98.8
97.2
96.2
96.0
Cins
W/O TOoL
882.1
84.3
81.2
79.2
81.7
Human
W/ TOOL
92.3
99.6
97.0
95.4
96.1
Table 10: The evaluation of the reward model's accuracy on different CONTRAST INSTRUCTIONS, where instruction (ins) and response (res) perturbations introduced.
Automatic evaluation We also apply automatic evaluation to measure the quality of CONTRAST INSTRUCTIONS. Specifically,
GPT-4, and prompt GPT-4 to decide which pair (instruction response) is more likely to be preferred by humans.
Concretely, we use the following prompt: Then, we measure the performance of GPT-4 on each
I will present you with two pairs of text. Each pair includes an instruction and a corresponding sponse. Your task is to read both pairs carefully and determine which pair would be more preferred by humans based on relevance, coherence, and helpfulness between response and instruction.
Table 11: Prompts used in evaluating CONTRAST INSTRUCTIONS using GPT-4.
version CONTRAST INSTRUCTIONS. The results are shown Table 10 from where we can see GPT-4 generally achieves 95% accuracy on our benchmarks, indicating that our CONTRAST INSTRUCTIONS quality guaranteed certain extent.
H RM CONSISTENCY ACROSS DIFFERENT MODEL SCALES
In this part, we conduct standard RM training on different scales, including LLaMa- - 7B, GBTJ-6B, GPTNeo 7B, and GPT2-1 5B.
I DIFFERENT PARAPHRASERS FOR CONVEXDA
In this part, we show the influences augmentators in CONVEXDA, including three different para-
phrasers. The results are shown in Table 13
23

---

Published as a conference paper at ICLR 2024
DATA
MODEL
METHOD
STACK
WMT
TWITTER
REALSUMM
AVG.
LLaMa-7B GPTJ-62
NONE
67.3
78.0
85.0
72.1
75.6
NONE
65.6
77.2
83.4
70.0
ORIGINAL TEST
GPTNeo-2.1
NONE
64.5
76.0
70.3
GPT2 1. 5B
NONE
63.0
74.8
81.4
70.2
72.35
NONE
50.1
56.4
60.1
47.7
53.6
CONVEXDA
52.1
58.4
61.7
50.2
55.6
M-aMa-7B
REWARDFUSION
52.1
58.4
61.7
50.2
55.6
COMBINE
53.6
60.4
62.9
52.6
57.35
NONE
50.2
50.6
52.7
51.75
CONVEXDA
51.9
52.4
55.7
53.9
53.48
GPTJ-62
REWARDFUSION
52.0
51.1
53.3
Cyp
COMBINE
52.3
52,9
55.9
54.5
NONE
52.0
51.6
52.7
52.7
52.25
CONVEXDA
52.8
53.0
53.9
53.8
53
GP GPTNeo-2,
REWARDFUSION
52.7
52.4
53.6
53.5
53.05
COMBINE
53.0
53.2
54.2
54.1
53.63
NONE
50.8
51.2
52.4
52.9
51.83
CONVEXDA
51.9
53.4
53.9
53.7
53.23
GPT2-1.55
REWARDFUSION
51.4
52.1
52.6
53.1
52.3
COMBINE
$2.0
53.4
54.1
54.0
53.38
NONE
48.2
48.4
48.5
52.6
49.4
CONVEXDA
52.9
53.6
52.5
55.7
53.68
LLaMa-7B
REWARDFUSION
$2.0
52.1
53.6
55.1
53.2
COMBINE
53.4
53.9
52.5
56.5
54.08
NONE
50.9
51.6
49.5
48.7
50.18
CONVEXDA
52.1
52.4
52.7
50.6
52.2
GPTJ-6B 6B
REWARDFUSION
51.5
52.0
51.3
49.2
51.0
Cins
COMBINE
52.4
52.6
52.7
51.0
52.18
NONE
50.9
51.6
48.5
49.7
50.18
CONYEXDA
52.1
52.5
51.7
51.5
51.95
GPTNeo-2. 7B
REWARDFUSION
51.5
52.0
50.9
50.8
51.3
COMBINE
52.4
52,9
52.0
51.8
52.28
NONE
49.9
50.6
51.5
52.7
51.18
CONVEXDA
52.2
51.9
52.7
54.0
52.7
GPT2 1.5B
REWARDFUSION
51.1
51.0
51.9
53.1
51.78
COMBINE
52.4
52.0
52.9
54.2
52.88
Table 12: The evaluation of the reward model's from different scales CONTRAST INSTRUC- TIONS, where instruction (ins) and response (res) perturbations are introduced.
DATA
MODEL
METHOD
STACK
WMT
TWITTER
REALSUMM
AvG.
NONE
50.1
56.4
60.1
47.7
53.6
CONVEXDA(TS)
52.1
58.4
61.7
50.2
55.6
Cys
LLaKa-7B
(FalexDA(Falcon)
51.9
58.2
61.5
50.4
55.5
CONVEXDA(Parru)
52.3
58.8
61.8
50.5
55.8
NONE
48.2
48.4
48.5
52.6
49.4
CONVEXDA(T5)
52.9
53.6
52.5
55.7
53.7
Cins
LLaMa-T7 7B
CoNvEXDA(Falcom)
52.3
53.4
52.7
56.1
53.9
CoNvEXDA(Purro))
53.1
53.8
52.6
55.8
53.8
Table 13: The evaluation of CONVEXDA based different paraphrasers, including T5, Parrot, and Falcon.
24

---

Published as a conference paper at ICLR 2024
Individual acceptability of Consistent RM
Individual acceptability of Original RM
Pairwise preference
Unsure
Original RM wins
Unsure
Reject
Unsure
Reject
25.0%
31.8%
21.4%
28.6%
27.1%
27.1%
43.2%
50.0%
45.8%
Consistert RM
Accept
Accept
Figure 10: Human evaluation of the RLHF models trained with the consistent original RM respectively. Left shows the pairwise preference, and Middle Right shows individual acceptability of using consistent RM/ original RM.
Relevance
Usefulness
Factuality
Cohen
0.94
0.88
0.79
Table 14: Cohen kappa correlation between two human evaluation results.
J EXTRA HUMAN EVALUATION RESULTS FOR TRICKLE-DOWN EFFECT OF RM CONSISTENCY
In this part, we ask three extra human raters (who own expertise in the computer sciece domain) to the individual acceptability of each model's response. The results are shown Figure 10 Moreover, we show the Cohen kappa correlation between this evaluation and previous evaluation in 6 Table 14
25

---

Published as a conference paper at ICLR 2024
K
CASE COMPARISONS BETWEEN RLHF MODELS GUIDED BY A CONSISTENT AND INCONSISTENT RM
Here we demonstrate few randomly sampled questions and compare the responses from RLHF models trained on the consistent vs. standard RM respectively.
[Question]: bought Dell Studio XPS 8100 desktop back in 2010, which had Windows installed and came with a partition for Dell Factory Restore. After having installed Windows 10, what happened to that partition? Did the installation get rid of it? If not and were to use it to do Dell Factory Restore, would it Windows 7? Sorry if this is duplicate of question somewhere, I didn't see one exactly like this asking about Windows 10 upgrade and Dell's Recovery partition.
[Consistent RM]: Comment: If you run Dell factory restore it will replace your current windows version with the one that is the recovery partition. If you want use the recovery partition for anything else you can but it will require you Answer: don't 't believe that the upgrade process to Windows 10 will affect the Recovery partition, can't 100% confirm that.
your recovery media and reinstall windows again.
It's best to just reformat the drive to the recommended specs (96GB 120GB for this machine, but this will depend on the model number) and start with fresh install of Windows 10, installing the drivers you need as required.
[Inconsistent RMJ: Answer: After upgrading to Windows 10, the factory restore partition will be present, and work as it would in Windows 7. This is how you able to do the upgrade and maintain the partition, because if it did not, then you would be stuck with an upgrade (which what the setup process actually tries do). have done several upgrades in past years, and have had no issues (though also have used the factory restore option about the same number of times).
Comment: I just bought new laptop and the Dell Refurbished restore partition for factory restore is missing. I'm planning on using it for clean Windows 10 install. But I cannot find it, even tell it is there. used the Dell Download Manager to install Windows 10 Home, but cannot find any partition.
Comment: there is no partition, then there will be no way to do a factory restore. you used the Dell Download Manager, you would also have to install an image from the CD for Windows 10 Home. If you don't have that CD, then you can't 't do it. Windows 10 Home (now Microsoft Basic) is only available as pre-installed OS, download. If not pre-installed, then it could not be recovery partition.
Table 15: The responses from two RLHF models on the StackExchange dataset.
[Question]: I am senior Siebel CRM developer having more than years of working experience. Now, am very keen and excited to learn Salesforce and get certified as soon as possible. Please guide me where to start from scratch
[Consistent RM]: Answer: This a great way learn [Beginner's Guide Become Salesforce Cortificd(httpss//llo..oo http:///////o/ooooooooooooniipie www.ingoooopciiiinnncoooo htw.omppetttiiissooooiiiddtoooot www.-hcoomm-slssssfs--eereiiiiee mentioned article will guide you to get started with the salesforce from scratch.
The above
[Inconsistent RM]: Answer: Salesforce is pretty broad topic, you're going need a good foun- dation programming (language, platform) and how web systems work. suggest you start with O' O'Reilly's Expert One-on-One: Salesforce.com Developer book. has the most complete ex- wwwpcooprssaessforr.cooooorooooo
I've seen. It's planation of the [Salesforce platform] the theory bit. [This Salesforce developer train- pretty exhaustive, you may want http://wwwggaaiiooos.oo/eeeeeeee
has some good videos, but it does cost money. don't know if you can find some videos free on YouTube or somewhere alse.Once you have a solid grasp of programming concepts (mostly Java/ISP/Ajax here) and how weh servers/weh applica- tions work. then you can move to Salesforce specific material. For examination, it looks like [this httptpp/wwwww.aalsforr....e/
http:///wwwottttiiai////////1/// is going be your best friend.
Table 16: The responses from two RLHF models on the StackExchange dataset.
26

---

Published as a conference paper at ICLR 2024
I've been wondering, there a nice way to make 'UIButtons' with multi-line "titleLabel" in Interface Builder? I've found none and puttin a separate 'UlLabel' on top of each button and tracking it's color on press and other events is not very straightforward.
[Question:]:
Maybe there are better alternatives done in code if there's 's no such way in IB?
[Consistent RMJ: Answer:
To allow multiple line you can use:
button.titlel.n www.tttteeeaelll.. Bitebe.iieeBreeodee = BeLLineerraMMdeWWooWWoe Lab.onnilleLbeel.eexAAAAiiie
UTTextAllinmennCCeneer
[button serTitle: @ Linel Line2forState: UIControlStateNormaal]
In **IOS 6'hh U and "UTTextAAlgameenCCeeee" are deprecated, so use: button. www.:ttteeeabelliiieee Biw.abb...iieBBeaMMoe = BySLineBreakByWong WordWrappingg:: button.titlel www.:iieeel..eleeeeeeeeeeee In interface builder select *UBButton* the right side Utilities pane under *Attributes Inspector*, you'l IIl an option for *Line Break* Choose *Word Wrap*
Unfortunately, there is no good way to do this in Interface Builder. You'll need use combination of 'IBOutlet' connections and some Objective-C in your code to accomplish what you Hent.Here is the general idea of what you'll want to do: begincodeimport "UIBut- @mplemennaation U viewDidLoad]; // Do any additional setup after loading the view, typically from nib. UlButton "button (UIButton "uutof, UlLabel *titleLabel (UILabel *Jsclf.iitleLabel: titleLabel. but- frame:;
[Inconsistent RM]: Answer:
(voidyviewDidLoad [super
options: @ NSFForggounnCCloorAtttttteeeeeee [NSAttributedString turrbuue&SrringFPonStrrrgbtttot hht.srrriigguuut...iiieel......e tet.elabelll..e
[UIColor lightGrayColorl, NSFFom-AttriuutNNamm: [UI- *)string (NDiitis ")options ArMutaale-AttibuuedSttti "attributedString [[NSMutableAtrribut edString alloc] initWithString:ssringg for (id option in options) [attributedString addAttribute:option [attributedString length])]: return attributedString end
Font systemFFontOOSizze:11 }:- (NSAutribaaeeSString
value:options[option]] aane:NSMMakeRaneeC endcodeTTher Then you can use it like thisthegincodel Aanngincooee//BButtotuuuuu Line mittllllllllee aahel "yourl.abor = (UTButtonMul- til-ineTiitlel Labll *)[[seff NewwWithTage Tag::] NiewWWth Tag: |:yourl Label. a text.text = @"Some multi-line class, and by searching the view with tag of '1' I know this solution not ideal, it's the best can come up with right now. Comment: Awesome! was looking for something like this! Thamen: What about a 'UIButton' with multiple 'labels' on the same 'frame" ?Comment: I've put the same code into custom 'UIButton' subclass and it works for me without problems.
text" redcodeddee "tag" parameter is the we're finding the
passed in to the MUlButtnnMlliiiineee/
Table 17: The responses from RLHF models the StackExchange dataset.
[Question:]: Is there anything preventing this from happening? Is it possible scenario? nsl. nsl.ed lsl..ecccasstds.no. net. 172,800 IN (Internet) A (IPV4 Host Address) 19. esl.eedgcasttds..og net. 172,800 IN (Internet) AAAA (IPV6 Host Address) 2606:2800:3::5 These two records have the same name so think it could be serving the wrong record.
[Consistent RM]: Answer: It sounds like you have EdgeCast configured to be your nameservers in your 'hosts' file, That is good idea. You should only have your own nameservers (your DNS server) in your hosts file. And, even if you do have EdgeCast as nameserver, the and AAAA records should resolve to different IP addresses, so something is definitely wrong.
[Inconsistent RM]: Answer: Yes, it could happen. The two records are the same name. If the two name servers also different nameservers, then what you have do add the ipvó address record to your zone file, then wait for and update the IPy4 name record to have the new IPv6 address. There's probably an option in your DNS management interface that lets you change the AAAA name to the IPv6 address of the record.
Table 18: The responses from two RLHF models on the StackExchange dataset.
27

---

Published as a conference paper at ICLR 2024
[Question:]: In the linear fractional transformation 7(x)) T(o))?
we define T(- = oc, then what is c
es4b
[Consistent RMJ: As long as od bc-0, it is easy to calculate that T(- -b For instance, 1t2
T(=) -I-2 has T(-2) 2-=====
Answer: We know T(}) = - (I) = +. then T( = = =o To = b it does not work. T(2) =
Then
a' m -a a' 0,c' 0, so 0 = 0, and T(T) =
[Inconsistent RMJ: I suppose the transformation should have well-defined limit, and it is not T(- Comment: The transformation not defined at _. And the answer is undefined. mndefined.Commentt is not the only such transformation; some of them have the limit. The answer depends the topology of the domain.
Answer: You right. The transformation T is defined
And the answer undefined. asked about the limit, not about the
Comment: My question not about why isn't defined
limits at -
Comment: Actually, you look the comment by Xi' an, the question is about whether you take the
limit at d think should be undefined, since the power series converges only on the positive half
of complex plane. Then limit + won't defined, since you would out of the range of
convergence of power series.
Comment: For the moment don't t think we can answer that question (I've put it on hold, I hope).
Answer: think you right. If you define =(=
the transformation should be well-defined.
Comment: is not, since the series doesn't I converge the negative half-platee
Comment: The other limit that you can take is limit when z - - where power series converges on the positive half-plane. But this doesn't give any answer, since the power series doesn't 't converge.
Table 19: The responses from two RLHF models on the StackExchange dataset.
28

---

Published as a conference paper at ICLR 2024
[Question:]: Is there anyway sync Outlook's calendar with Google's calendar? I've tried [following haasshhttp//suppor..oogg woom///llendrrbbuaaawwwryyy (995 before but doesn't work properly. I'm using Windows 7 and Outlook 2010. The thing is, it's really easy to check my Google Calendar on my phone since have an Android terminal, but on my computer I prefer just opening Outlook instead of opening web browser and logging into Gmail.
[Consistent RM]:
Answer:
You can install calendar syne tool. htmml) has calendar syne app (it's a which synes with Outlook
bit old). You can also try [Google
and other calendar clients.
Comment: Google Gears dead (http://google warth.blogssot.com We will continue to support gears through the Google Calendar API. Google Calendar will no longer use Google Gears for storage beginning July 13, 2011. When enable Google Calendar syne in your Google Account, you will be directed to set up your syne through the Google Calendar API.
1/05/farewell google-gears.hnneee
[Inconsistent RM]: Answer: I'm afraid this is not possible at the moment, but they are working on it. https://groups.. http://rrousggoggle.oooo hta.:eecooogrroggogggeeoooo potooee--cccttaaaaoolommmoaaaaaa http://mmsgbc1114761lllllll
Comment: The Google calendar now has import function that will import the calendar from your Out- look/Exchange account. Unfortunately. it seems that the import only does the schedule and not the actual contacts. So, you
see the contacts from Outlook the Google calendar.
[Link] (http://wwww httt:://wwwwhhwtooeee. howtogeek.com htmm////72//hww http///wwto--xortttteoo export-your- youroo--ou--oooll----o google-calendat calendar-events- events-intte into- microsoft outlook// You can syne your Google Calendar with your Microsoft Outlook in a few simple steps. It is great way to keep everything in one place and have all your appointments your Google Calendar, as well Microsoft Outlook. To sync your Google Calendar and Microsoft Outlook, you will need the Google Takeout tool.
Answer:
Answer: I've been using google calendar and IMAP for years now and never had any problems. I just use the standard IMAP address in Outlook and it works great.
Although there doesn't 't seem be solution for exporting contacts, using Google Takcout (link below) allows you export an ICS file that can be imported into Outlook. |Google Takcout [ (htpps//wwww.googlcccc sta.gll..c//aacout)
Answer: Doesn't it this already exist? http://www.outt www.wowwllooccaledaarroooo
Table 20: The responses from two RLHF models on the StackExchange dataset.
29


Reviews:
[{"review_id": "WREReQ7S8n", "reviewer": "Reviewer_5JFN", "summary": "The authors study reward models that are used for RLHF  to tune LLMs for desirable generations. Specifically the aspect of (in-)consistency is studied. The paper studies the following research questions:\n\n- How to measure in-consistency of reward models? : \n  - The authors introduce the Contrast Instructions benchmarking strategy to measure in-consistency.\n  - The benchmark consists of quadruplets consisting of lexically similar instructions but different responses. Two metrics are proposed: \n    - Response consistency: Can the RM assign a higher score to the correct response given the instruction \n    - Instruction consistency: Can the RM assign a higher score to the correct instruction given the response. \n  - The benchmarking is automatically constructed using existing open source human preference datasets. A sentence embedding model SimSE is used to find pairs of instructions that are lexically similar, but semantically different. \n  - This strategy has been explored with 4 popular open source human preference datasets. \n  - They find a huge performance gap on this benchmark between human judgements and reward models trained with the 7B LLaMa checkpoint\n- How to reduce the gap and improve consistency of reward models? : Two techniques ConvexDA and RewardFusion are introduced that can be incorporated into the training and inference stage of reward models at no additional computation cost. The authors show this helps improve consistency. \n  - ConvexDA: At training time, the data is augmented by substituting words in the responses with synonyms generated using WordNet \n  - RewardFusion:  At inference time, a weighted average reward score between similar training examples and given instruction response pair is used\n- How does reward inconsistency influence downstream performance of chatbots after RLHF? : The authors show that using a more consistent RM for RLHF can lead to more preferable downstream generations.", "questions": "- Could you please explain how the proposed approaches are designed to help address the inconsistency in the reward model?\n- Could you explain the reason behind the very low C_res score of the trained reward model on the same dataset, even though C_res resembles the RM training objective?\n- Could you explain what finetuning on contrast instruction format dataset involves?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 3, "strengths": "- The paper proposes a new way of evaluating reward models and focuses on the new important aspect of consistency of reward models. It also highlights how existing reward model evaluation methods do not capture this aspect. \n- The benchmark creation for evaluation is intuitive and automatic and can be easily applied to any existing instruction tuning dataset \n- The paper highlights the importance of using more consistent reward models for RLHF \n- The paper also propose simple methods that show slight improvements in performance on the proposed evaluation metric", "weaknesses": "- Contrast instructions benchmark: To sample lexically similar but semantically different pairs of instructions the authors only sample instruction pairs that lie within the similar range of 0.75 and 0.9. It would be useful to explain the sensitivity of this hyperparameter and study how well this prevents sampling semantically similar instructions. \n- When evaluating existing RM on this benchmark the reward models get a low C_res score of 53.6, even though C_res conceptually resembles the RM learning objective. This is surprising given the benchmark was constructed using the same datasets used for training. It would be useful to verify if appropriate hyper-parameter tuning was performed and if the model is able to overfit and get a high C_res score. \n- Limited models and scales: It would be useful to explore more pretrained models and perform a model and dataset scaling analysis to measure if and how the consistency of reward models varies across different models and scales and if similar observations are found. Would be useful to do a similar analysis when evaluating the impact of the proposed improvement methods ConvexDA and RewardFusion to see how well they perform with different models. \n- It would be useful to explain how the proposed data augmentation and inference time technique are designed to help with improving the performance in contrast instructions benchmark. This is especially important given the very small improvement on the Contrast Instructions benchmark with the proposed approaches. \n- The two methods ConvexDA and RewardFusion that are proposed are not used to build more consistent reward models that are used during the RLHF stage. Instead fine tuning on contrast instructions format is used. However, in appendix C, it is mentioned the “more consistent” reward model is equipped with ConvexDA. \n  - It would be useful to evaluate models that have been trained (during RLHF) using reward models that are equipped with the proposed strategies (ConvexDA and RewardFusion) to show the improvements brought by this approach\n  - Please clarify what “finetuning with contrast instructions format” consists of \n  - Please clarify if ConvexDA was used to train the reward models for this stage as mentioned in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yQUHud619k", "reviewer": "Reviewer_gxZU", "summary": "This paper studies the problem of Reward Model (RM) inconsistency and its impact on the downstream RLHF model. The paper introduces the \"Contrast Instruction\" benchmark to measure the ability of a reward model to consistently recognize semantic changes in prompts and adapt reward assignments accordingly. It then shows that current RMs trained with standard ranking objectives underperform compared to human judgment on this benchmark. To address this inconsistency issue, the paper proposes two innovative techniques: \"ConvexDA\" and \"RewardFusion,\" which leverage extrapolation during RM training and inference, respectively, to enhance RM consistency without requiring additional training resources. The authors demonstrate that these advancements lead to more useful responses from RLHF models trained with a more consistent RM, highlighting the crucial role of consistency in maximizing the effectiveness of the RLHF process.", "questions": "1. Please clarify how general the CONVEXDA and REWARDFUSION are in fixing inconsistency beyond lexically similar but different words.\n2. Please compare to methods that simply use LLMs for data augmentation.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper presents a novel way to investigate inconsistency in reward model via constructing benchmarks with inconsistent instruction and response pairs. The paper further proposes a fine-tuned reward model on top of the contrastive instruction-response pairs, which leads to improved RLHF performance. I think the investigation sheds insight in the RLHF research field and the proposed method is neat and effective.\n\n2. The authors have performed extensive empirical study of the inconsistency of RMs, which clearly show that the current RMs do suffer from inconsistencies. The paper also provides thorough empirical evidence that shows the RLHF model can improve a lot with a more consistent RM trained on the constructed contrastive pairs of instruction and response.", "weaknesses": "1. It's a little unclear why CONVEXDA is used as a way to robustify the RM. It seems that it might work because it specifically targets the proposed benchmark, which mainly tests inconsistent instruction-response pairs with lexically similar words, but I'm not sure if the method can work well in scenarios where inconsistent responses/instructions are beyond just lexically similar words with different meanings. Moreover, I wonder if it would work equally well to simply use a different LLM to generate those similar pairs for augmentation.\n\n2. I'm also less clear about why REWARDFUSION is needed and how it contributes to fixing inconsistency. Again, it seems to mainly target the type of inconsistency due to lexically similar words with different meanings, which may not be general enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wOdAhbhVf8", "reviewer": "Reviewer_iZzr", "summary": "The paper studies reward models in RLHF under the lens of consistency -- whether the RM can adapt its scores to semantic changes to different prompt response pairs. The authors create benchmarks to verify consistency of rewrd models called Contrast Instruction where the authors study the reward model score for lexically similar instructions with different responses. They claim that current RM with the standard loss functions suffer in these contrast instructions compared to human preferences. The authors provide two techniques -- one at train time and one at inference time that improve reward consistency through extrapolation across similar examples. They also claim that reward model consistency has a correlation with the usefulness of the RLHF responses.", "questions": "I have the following questions for the authors\n\n1. There were certain choices of methods made in terms of methods for response/instruction similarity and the dataset construction including the augmented data points for ConvexDA. Have other methods been evaluated and ablated against before choosing these methods ? I wanted to understand if we are doing the best we can in terms of construction of these datasets and methods.\n\n2. I know you have considered the classic RM loss, log (sigmoid (R_chosen - R_rejected)) ? Have you considered other loss functions such as the margin loss used in LLAMA 2 ? I am not specifically looking for one kind of loss, but just a choice of few other RM losses. Considering different losses can help us understand whether RM inconsistency arises from the dataset or the kind of losses used or a combination of both. This would be a good ablation to understand where RM developers should focus more of their efforts on.\n\n3. Lot of times, RM training can easily lead to overfitting to the training datasets and creating a generalization gap in the process. Can you kindly explain how you do model selection for RM training for the baseline and your variants ?\n\n4. I know that you choose only a single example in ConvexDA while you construct a few semantically similar examples. Have you evaluated what happens if you include all of them ? I know this is unfair to be compared to the baseline, but it would help us understand having how many semantically similar examples will be useful.\n\n5.In table 2, I was not sure what (estimated) human performance means. Can you kindly clarify ?\n\n6. I know evaluating with larger models would be resource intensive. But would it be possible to run the analysis with either a smaller or larger model than 7B to understand if and how reward model sizes influences reward consistency.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "The paper exposes an interesting and useful aspect of reward modeling in RLHF -- reward model consistency. Apart from standard RM eval, evaluating RMs for consistency is an important aspect that future works can take into consideration or the current eval sets be expanded to include this as a benchmark task. The authors clearly define what consistency means and setup constrast instructions dataset to evaluate the reward models. While the dataset needs to be vetted more carefully, having this dataset as part of standard RM evals could be a useful exercise for practitioners. The methods that the authors propose to improve consistency are simple to implement.", "weaknesses": "While the paper exposes an important aspect of reward models, one thing that could be improved about in  the paper is the limited nature of experiments and evaluation.\n\n*  I was not sure of whether the contrast instructions generated automatically do mean something different where one answer is strictly better than the other -- for instance . The authors say they restricted this based on a particular cosine distance range, but some more rigorous evaluation around this could have made the impact of the dataset a bit stronger. For instance, one can query a bigger open LLMs in order to validate that the contrast instructions are fit for purpose. The other option could have been to conduct human eval on a randomly selected subset of the contrast instructions dataset. \n\n*  The authors could have just chosen one single baseline LLAMA 7B and use that to validate their claims. The behaviour of LLAMA 7B could be ground in the kinds of datasets and training procedures that it was subject to. Having few other RM baselines trained with similar RM training loss could have made the claims even more stronger.\n\n *  The human evaluation was done on 100 randomly selected data points and no standard errors and variance numbers were reported around the results. Since some of the results are close to each other, having the error bands around the result should help clarify the significance of improvements.\n\nOne other concern about human evaluation that I have is about the possibilities of potential bias as the authors themselves serve as human annotators for human eval of the results. There are several important statements made in this paper based on human evaluation. I would have preferred at least a mixture of external annotators (who are not aware of the work or how the responses were generated) to de-bias the evaluations. \n\"Finally, we report human performance resulting from the majority vote of three human annotators (the authors) on 100 randomly selected data points.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7JROTldfix", "reviewer": "Reviewer_Kxr1", "summary": "The paper focuses on reward inconsistency while doing Reinforcement from Human Feedback studying the various reasons around such inconsistencies by developing relevant metrics to measure the same. They demonstrated the failure of the current reward models typically used in RLHF (with comparison to avg human)  on the CONTRAST INSTRUCTIONS introduced in the paper, which the authors attribute to reward inconsistency. Finally, the authors propose two strategies to mitigate such reward inconsistency with improved downstream performance.", "questions": "1. Why augmentation helps as a solution to mitigate the problem is not clear in the context of the Contrast Dataset. Will be helpful to have a more rigorous discussion on the same?\n2. ContrastDataset provides is challenging dataset, how are humans able to do good on it? Is it mainly hard for the LLMs since the representations are sub-optimal?\n3. A comparison with current SOTA RLHF or reward models on standard/open-sourced datasets will be helpful in understanding the crux of the problem.\n4. Recent work on Direct Pref Optimization (DPO) shows that it can learn with learning reward models, will such an issue happen there as well? \n5. Reward ensemble seems to work well in RLHF to mitigate over-optimization as also followed in [1] for Robotics. Will be interesting to see if that helps and have a discussion around the same.\n6. The notations used in the equation after 1 (missing no) are not very clear and it will be helpful if they can be updated as per standard RL notations of trajectory, state, etc.\n\n[1]. Thomas Coste, Usman Anwar, Robert Kirk, David Krueger \"Reward Model Ensembles Help Mitigate Overoptimization \" https://arxiv.org/abs/2310.02743", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The primary objective of the paper is to demonstrate the reward inconsistency with the standard RLHF training, as shown in Figure 1 where it shows for similar (but distinct) prompts, the rewards assigned are inconsistent with the current reward models. A primary reason attributed to the over-optimization is the fact that the current reward models are trained on datasets that don't represent close preferences in other words, the current reward models are not optimized for prompts where both the preferences are near-optimal and one is slightly better than the other. According to the authors, those are the places where the current reward models suffer and that's where CONTRAST INSTRUCTIONS helps in providing a meaningful evaluation and mitigation to such over-optimization.", "weaknesses": "1. The paper claims the issue in reward inconsistency is due to the reward over-optimization issue (citing Gao et, al). Still, the reason for such reward over-optimization and how that causes the inconsistencies in the particular Contrast Dataset is unclear.  More specifically the author claims that \"From the RMs’ perspective, correctly distinguishing between a clearly good vs. bad response is easy\". But the contrast dataset for example shown in Figure 1, they are very different questions although textually there are common words. Does that mean the current LLMs are not able to produce representations that can separate the two is not very clear and needs further clarification. For ex: \"A is a  good student\" and \"A is a bad student\", they have a lot of words in common but in representation space they should be extremely different and should be trivially separated if the representations are reasonable. Thus it's not very clear how Contrast Dataset is providing a challenging dataset to test RM model inconsistency.\n\n2. Another point is that why such reward inconsistencies are not observed in standard available RLHF datasets like Carper AI, hh, etc. are not made explicit. Does it mean that the majority of the datasets are easily separable and lacks samples towards the optima which is hard for human to segregate? A comparison with current RLHF methods is critical on standard datasets to understand the significance and mitigation of the problem properly.\n\n3. \"Surprisingly, we observe close to random-chance performance ..while humans are able to rank the responses correctly in ≈ 80% of the cases\" So, is it the case that humans are able to identify it but the reward models are not able to learn the same and failing is not very clear since its a supervised learning problem and can be shown to be strongly convex under certain settings. Hence, a clear description is missing on the same, and will be interesting to have a discussion with reference to the recent works showing convergence [1,2] and where this issue can arise in that context.\n\n[1]. Banghua Zhu, Jiantao Jiao, Michael I. Jordan \"Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons\"\n\n[2]. Souradip Chakraborty, Amrit Singh Bedi, Alec Koppel, Dinesh Manocha, Huazheng Wang, Mengdi Wang, Furong Huang \"PARL: A Unified Framework for Policy Alignment in Reinforcement Learning\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3r0WV196I3", "reviewer": "Reviewer_n2Hx", "summary": "This paper investigates an important yet often overlooked problem - the robustness of the reward model (RM). The authors propose a benchmarking technique called Contrast Instructions that gauges the reward consistency of an RM. The reward consistency is measured by consistency in preference ranking if given a pair of lexically similar instructions with different ground truth responses. Concretely, it is quantified by (1) response consistency, if the RM can identify the better response for a given instruction, and (2) instruction consistency, if the RM can identify the most fitting instruction for a given response. The benchmark dataset is constructed based on four open-source human preference datasets of various NLP tasks. The authors showcased that an RLHF model trained with a more reward-consistent RM outperforms an RLHF model trained with the original RM in human evaluations.", "questions": "1. How concerned are you about the risk of data leakage? What implications would arise if instances from the benchmark dataset were also present in LlaMa-7B's pre-training data or the dataset used to train the reward model? Have such overlaps impacted the assessment of reward consistency?\n2. I need more details on Section 4. For \"Single-Task\", what is the split between train and test?\n3. For Section 4, why would you fine-tune a benchmark dataset? And the fact that it performs so poorly after being trained and tested on the same dataset distribution is surprising. The consistency improvement seems marginal.\n4. What is RMEval in Table 7? Test set accuracy on binary classification task?\n\nOverall I am positive about this paper. This is an interesting piece of research. If the authors can properly address mines and other reviewer's concerns, I will agree to raise my review score.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-written and it is easy to follow.\n2. The idea is straightforward but the underlying research problem is significant and yet often overlooked.\n3. The trickle-down effect of reward consistency on RLHF training is an interesting observation, which intuitively makes sense.\n4. Table 7 is helpful in seeing that reward consistency and test set accuracy (if I understood correctly) do not necessarily correlate. This is similar in the sense that the (dis-)correlation between human score and FID is often discussed in generative models.", "weaknesses": "1. The way that the authors constructed the dataset, is filtering by the cosine similarity between SimCSE embeddings that are in the range of [0.75, 0.9]. This seems convenient but I wonder how reliable is this method. Have you done a manual inspection to measure the agreement rate between the method and human evaluators? Or maybe you could try using a model-based approach like prompting GPT-4?\n2. As I am more interested in the benchmark dataset itself, the evaluation for dataset validation seems limited. **The author should focus on providing a reliable benchmark as the major scientific contribution**; I believe the finetuning methods discussed in the manuscript are not as significant. Therefore, the authors should provide more evaluation results on more LLMs (open-source + closed-source) to validate your benchmark dataset. The evaluation results on popular models like GPT-4 would be very helpful. Although you can't get the weights, test by prompting would be sufficient. I am also curious to know the difference in consistency between the pre-trained model vs their SFT-ed version (i.e. llama2-7b vs llama2-7b-chat). If the compute resource allows, the parameter scaling on reward consistency could also be an interesting point for investigation.\n3. The benchmark dataset should be submitted along with the paper, as I believe this is the core contribution of the paper.\n4. The data variances in Figure 3 make the comparison in Sec. 7.2 rather inconclusive. However, I praised the authors' honesty in showing error bars.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)