Statistical Rejection Sampling Improves Preference Optimization
Authors: Tianqi Liu
Yao Zhao
Rishabh Joshi
Misha Khalman
Mohammad Saleh
Peter J Liu
Jialu Liu
Venue: ICLR 2024 poster (2024)
Keywords: large language model
preference optimization
language model alignment
Primary area: general machine learning (i.e., none of the above)
TL;DR: This work uses rejection sampling to collect preference data from the optimal policy, which improves the policy estimation over DPO and SLiC.

Abstract: Improving the alignment of language models with human preferences remains an active research challenge. Previous approaches have primarily utilized online Reinforcement Learning from Human Feedback (RLHF). Recently, offline methods such as Sequence Likelihood Calibration (SLiC) and Direct Preference Optimization (DPO) have emerged as attractive alternatives, offering improvements in stability and scalability while maintaining competitive performance. SLiC refines its loss function using sequence pairs sampled from a supervised fine-tuned (SFT) policy, while DPO directly optimizes language models based on preference data, foregoing the need for a separate reward model. However, the maximum likelihood estimator (MLE) of the target optimal policy requires labeled preference pairs sampled from that policy. The absence of a reward model in DPO constrains its ability to sample preference pairs from the optimal policy. Meanwhile, SLiC can only sample preference pairs from the SFT policy. To address these limitations, we introduce a novel approach called Statistical Rejection Sampling Optimization (RSO) designed to source preference data from the target optimal policy using rejection sampling, enabling a more accurate estimation of the optimal policy. We also propose a unified framework that enhances the loss functions used in both SLiC and DPO from a preference modeling standpoint. Through extensive experiments across diverse tasks, we demonstrate that RSO consistently outperforms both SLiC and DPO as evaluated by both Large Language Models (LLMs) and human raters.

Published as a conference paper at ICLR 2024
STATISTICAL REJECTION SAMPLING IMPROVES PREF- ERENCE OPTIMIZATION
Tianqi Liu", Yao Zhao', Rishabh Joshi', Misha Khalman' Mohammad Saleh Peter J. Liu", Jialu Liu" {tiangiliu, yaozhaoyz peterjliu, jialu) @googl
Massabhhooshi khalman, msaleh, ggooo le com
Google Research", Google DeepMind*
ABSTRACT
Improving the alignment of language models with human preferences remains an active research challenge. Previous approaches have primarily utilized online Re- inforcement Learning from Human Feedback (RLHF). Recently, offline methods such as Sequence Likelihood Calibration (SLiC) and Direct Preference Optimiza- tion (DPO) have emerged as attractive alternatives, offering improvements in sta- bility and scalability while maintaining competitive performance. SLiC refines its loss function using sequence pairs sampled from a supervised fine-tuned (SFT) policy, while DPO directly optimizes language models based preference data, foregoing the need for separate reward model. However, the maximum like- lihood estimator (MLE) of the target optimal policy requires labeled preference pairs sampled from that policy. The absence of reward model in DPO constrains ability sample preference pairs from the optimal policy. Meanwhile, SLiC can only sample preference pairs from the SFT policy. To address these limita- tions, we introduce a novel offline approach called Statistical Rejection Sampling Optimization (RSO) designed source preference data from the estimated target optimal policy using rejection sampling. enabling a more accurate estimation the optimal policy. We also propose unified framework that enhances the loss functions used both SLiC and DPO from a preference modeling standpoint. Through extensive experiments across diverse tasks, demonstrate that RSO consistently outperforms both SLiC and DPO as evaluated by gold reward, Large Language Models (LLMs) and human raters.
INTRODUCTION
Recent advancements in Large Language Models (LLMs) (Brown et al., 2020; Touvron al., 2023; Anil et al., 2023; OpenAl, 2023) have unlocked unprecedented capabilities in diverse tasks, such as programming and creative writing. Models are pre-trained on large unlabeled corpora and su- pervised fine-tuned (SFT) on various tasks (Wei et al., 2021; Chung et al., 2022). Subsequently, RLHF (Stiennon et al., 2020) enhances the alignment of large language models with human prefer- ences. RLHF introduces notable complexities into the training process, including reward model, policy model, reference policy, and value model. limits maximum feasible size of a model due to memory constraints. Additionally, is not stable during training. Recognizing these chal- lenges, recent research has pioneered alternatives to RLHF Notable among these are RRHF (Yuan et al., 2023), SLiC (Zhao et al., 2022; 2023) and DPO (Rafailov et al., 2023). These methodolo- gies aim to more effectively align LLMS with human preferences while avoiding the complexities of reinforcement learning. Given supervised finetuning data D.n {(x, Bter)) and preference data Dar = [(T, Voe } where output text The is preferred over yı on the same input text x, they di- rectly fit the policy model preference data in various ways. RRHF uses trained reward model or human raters to compute rewards for multiple sequences generated from difference sources on the same prompt x, and then apply a ranking loss plus supervised fine-tuning loss. SLiC uses contrastive ranking calibration loss plus a regularization loss
(() max (0, 8 log Ta ( (ar) log To())>))) Alog (h)

---

Published as a conference paper at ICLR 2024
Step Sample
Step 2:
Step 3: Optimize
Prompts
Reward Mood
SFT Policy
SFT Palicy
Sampling
Rankinng
Preference Optimiant
flewerc: Mocill
Acceptt
Preference
Ostiiieed Policy
Respenses
Figure 1: RSO first fits a pairwise reward wwwardr model from human preference data. This model is later applied to generate preference pairs with candidates sampled from the optimal policy, followed by a preference optimization step to align sequence likelihood towards preferences.
where 8 is positive margin and ne is the learnable conditional probability function by a language model. SLiC either fits directly on human preference data or on preference data sampled from the SFT policy. DPO analyzes RLHF's objective function in the form of KL-regularized reward maximization, and analytically solves the optimal policy induced by reward function. Based on the Bradley-Terry (BT) model (Bradley & Terry, 1952), DPO proposes an MLE to fit on human preference data directly and expresses the human preference probability in terms of only the optimal policy and reference policy Tsf::
p' (y1 v2|2)
+ exp
s " (a) (
log
s ( (m)
Jan Tnn
where n²" is the function to be estimated and / is hyparameter in RLHF objective.
Empirically, one can leverage observed preference pairs to approximate p" (31 y2(æ)). To estimate n² as density estimation problem, the optimal way is to fit policy model on collected preference pairs sampled from x². However, DPO uses the collected human preference data from other policies directly in all the experiments and lacks study the effect of sampling. Although they propose to sample pairs from the SFT policy and get them labeled by human, it is still not strietly MIE for the preference model due the mismatch between the sampling distribution and n², In reality, is very challenging to obtain human preference pairs directly sampled from n..
In this work, we address the above issues by constructing preference pairs from the approximated m² (Figure 1). Starting with a human preference dataset 'ar collected from other policies, we first train pairwise reward-ranke -rankig model. then apply statistical rejection sampling algorithm to generate response pairs sampled from optimal policy by using SFT policy and the pairwise reward-ran reward-ranking model. After that, we label the sampled response pairs by the reward model. Then we fit the model on labeled pairs via classification loss. DPO claims that the language model is secretly a reward model, we show that the language model learns better from an explicit reward model because comparing between two responses (reward) is easier to learn than generating high quality responses (policy). Our statistical rejection sampling refers to the one in the statistical field (Neal, 2003). In RLHF works (Bai et al., 2022; Stiennon et al., 2020; Touvron al., 2023), they usually refer to rejection sampling as best-of-N or top-k-over-N algorithm, where they sample a batch of N completions from language model policy and then evaluate them across reward model, returning the best one or the top k. This algorithm has the issue of reward hacking because trusts the reward model too much without any regularization. In this paper we show that top-k-over-N a special case of our statistical rejection sampling and it is critical to balance between the reward exploitation and regularization towards the SFT policy. summarize, our contributions of this work are three-fold.
we propose a scalable and -omy----mmpeeentt framework to learn from human preference data. We provide a comprehensive recipe among different choices of loss functions and preference pairs generation. We show the importance of the reward model instead rectly optimizing the model on the preference data.
2

---

Published as a conference paper at ICLR 2024
we unify DPO and SLiC statistically by showing that they vary by loss functions fit on human preference data: DPO is a logistic regression on human preference data and SLiC is almost equivalent to support vector machine (SVM) with hinge loss. We improve SLiC as the SVM counter part of DPO.
we design statistical rejection sampling algorithm to sample pairs from the estimated optimal policy and get them labeled by pairwise reward-ranking model. The proposed sampling strategy is shown to be effective on several generative tasks.
PRELIMINARIES
Learning from Human Feedback Several works (Ziegler et al., 2019; Zhao et al., 2023; Rafailov et al., 2023) show the significant improvement of conditional language generation by learning from human feedback data. All algorithms take two inputs:
(v(zx) supervised fine-tuned policy (SFT), where is the prompt and y is the response.
Dar (x yo' Vi > human preference dataset that distinguishes the better re- sponse from the worse given the same prompt.
(5)
KL-Constrained Reward Maximization Objective Starting with reward function r(x, y) and input prompt distribution P, the DPO and RLHF optimizes for the following objective:
Bz-P yoa [r v)
Optimal Policy DPO solves the optimal policy क. (1/(a)) that maximizes the above objective:
Tr (y(()
Tsf (yl 2))
exp 7(2, y)
Z(x)
for all x E P, where Z(z) E, Tm (v) exp (x( v)) is the partition function. controls the balance between exploitation and exploration. When 8 - 0. all probability mass will concentrate on the max reward with full exploitation. When 8 - co, optimal policy will be the same as Tan with full exploration. Rearrange the Equation (4) we get
Tr (ylx)) Tsh (v|z)
r(z, y) = 3 log
+ 3 log (zx).
(5)
The Equation (4) and (5) establish the relation between optimal policy and the reward function. In reality, the final goal is to have a good policy for response generation and t, (ylær is usually of more interest. The key is to effectively estimate the m, ()) from the human preference data.
Preference Model Let the ground-truth reward function be r", then the optimal policy x" associ- ated with can be represented by Equation (4). For two responses (2) from the same input x, Bradley-Terry (BT) model (Bradley & Terry, 1952) assumes that
Ply1 > 32/=) = (r (z, v1) - r (z, y2)).
(6)
where PM > 82(z) represents the probability that response y/ is preferred y/ give prompt x. Reusing Equation (5), we obtain Equation (2). we leverage the human preference data to represent Plyl y2(2), the estimation of nh be viewed as density estimation problem from the preference data. We will discuss different ways of estimating x in Section 3.1.
Reward Model We train a pairwise T5-XXL (Raffel al., 2020) text-to-text reward-ranking model' (xr V1, U2) on Dar to approximate Pyy1 > Vala)). (v V1, U2) takes the text input as:
"[CONTEXT] {x} [SUMMARY A] {ui} [SUMMARY B] {v')* } for summarization task
SLiC demonstrates that pairwise reward model is preferred in RL-free learning. Our pairwise reward- ranking model has accuracy of 73.23% on the validation set of summarization task and 69.75% on the validation set of AT assistant task.
3

---

Published as a conference paper at ICLR 2024
"[CONTEXT] {+} [RESPONSE A] {yi} [RESPONSE B] {y2)" for AI assistant task
Pv (3r yı, y2) outputs "A" "B" as preferred one. We the probability of decoding "A" esti- mation of the preference probability P(yı > y2|z)² Suppose we have a baseline sequence yo with reward score O, we can induce the reward score of any sequence
Th (zz, y) = logit(pp(z: (2, y. yb)))
(7)
where logit(æ)) log( Imx) This is a result of setting 5/ y. y/2 yb. and r (z, y2) 0 in Equation (6), where we replace the win rate with the estimated one p y, yb). Thus, "pointwise" reward score can be derived from "pairwise" reward-ranking model with baseline sequence?.
RSO APPROACH
3.1 STATISTICAL ESTIMATION OF THE OPTIMAL POLICY x²
Our proposed approach (Figure 1) takes inputs of SFT policy, reward-ranking model, and prompts. First we sample responses from the optimal policy through rejection sampling approach, then fit a classification model on labeled preference pairs. To study the effectiveness of our approach, we consider few options on loss and preference dataset construction. Given a preference dataset Dp (x yh 3/
(4) J. we can estimate π according to Equation (2). There are two aspects we need consider for estimating
Choice of loss function: To fit Equation (2) as binary classifier using (r" y1) ("r 12)) as logit with fixed slope and zero bias, we consider logistic loss used in logistic regression and hinge loss used in support vector machine (SVM).
Choice of Dp: Equation (2) does not depend the distribution of yı V2 given x. Thus need to decide how to obtain (z, y1 y2) triplets.
Choice of loss function Given preference dataset Dp (2x11) ywu 3
(4) (3) we can fit binary
classifier according to Equation (2). DPO (Rafailov et al., 2023) uses sigmoid loss on normalized likelihood (sigmoid-norm) to fit a logitistic regression:
Lsigmoid-nann (TO) Tan Dp) - -E (x..vw. log o (7log me (ye (x)
Th (30) (x)
7 log
(8)
Man (Bwllar a:)
Tun (y11 (2:
where DPO sets y = B. In this work, we decouple y from 3 and treat y as an equivalent temperature hyper-parameter. The larger the 7, the boundaries by trusting more on the preference labels.
penalize the mis-classified examples at the decision
SLiC (Zhao al., 2023) proposed
hinge calibration loss* as
Chinge (Te (Dp) E (())))))()) [max (0, 1 - [> log Te (Vwl (x) 7 log Fa (w [/[z)]]
(9)
Note that we use 1/7 as the margin used in SLiC loss (Equation (1)). This is equivalent to a hinge
loss with logit (y log me (yw (x)) - log me (yr [z)). ) If normalize the policy probabilities, we get the SVM variation of DPO as the hinge loss on normalized likelihood (hinge-norm):
Te (yhrlz)) log To (y (x)
Chinge-norm (ate Tate Dp) = (z..o. .D) max 0,
>log Tift (Vwllxr
Tesi (M1 (T)
(10)
We randomly flip response pairs and associated labels to remove positional bias.
IIn practice, we choose random decoded sequence from the SFT policy the haseline. We in theory solve n reward scores from n² comparisons with baseline sequence via constraint optimization setting. leave this for future study.
Subtle differences between SLiC loss and hinge-loss will be discussed in "Method" paragraph of Section 5.
4

---

Published as a conference paper at ICLR 2024
Choice of preference data distribution Suppose have access to the oracle preference data (6)
D [(x111) yy -Vi v( you - (,()) x ()))=(()) we can directly fit an MLE on the dataset.
In reality, may not have access to such data, and we have access to Dar {(x*), yeu
(3) x/"))
You -ij (6) Tunk (v]=:()))) J where Tunk denotes some mixed unknown policies. The mixed un- known policies can include SFT policy, previous current RLHF policy, or policies from other agents (Touvron et al., 2023). Given Dhr. we consider the following three choices:
direct: directly fit the policy on Dhr according to Equation (2) as DPO without Pe-
stt-sample-rank use Tan (y/z) to sample response pairs given prompts from the SFT train- ing set and label them by Pa.
rso-sample-rank use Tyy (y/a:) induced by (1, y)² according to Equation (4) sample response pairs labelled by Pv given prompts from the SFT training set.
Statistically speaking, since ayso-sapll--rake is the best solution towards this direction with samples from Tr (y/z), which is closer to x' (y|zr) than other two choices.
estimating x' (y/z;, it is desired draw samples from x' (y) (r)..
3.2 STATISTICAL REJECTION SAMPLING ALGORITHM
Statistical rejection sampling (Neal, 2003) is an efficient statistical technique to generate observa- tions from a distribution. we want to generate a distribution of density Tru we can use Tatt as the proposal distribution and follow the steps:
1. Start with empty y 0..
2. Generate y Tan (y|=) that is not in Y' and u U10,, 1].
Trg (vl=)
3. Let M min ())(()22) 2 Trw (y|z) for all V/.. If a MTa(y then accept y and add y. Otherwise, reject y.
4. Repeat step and until we get enough y
Lowneweed
Opinal
Reword
Accept
Righ reward
Rejec
Decoded Sequerce from rpulX
Figure 2: Statistical rejection sampling illustration. There are three curves in the figure: M times SFT policy, reward, optimal policy. The sample is first generated by SFT policy, then gets accepted or rejected depending on whether uniform random variable locates in acceptance rejection region. the sample has high SFT policy probability but low optimal policy probability and reward score, it has higher chance of being rejected.
Figure 2 is illustration? of the statistical rejection sampling approach. A Python implementation (Algorithm I) with derivation is shown in Appendix A.I. The computation efficiency is discussed in Appendix A.10 Regarding the algorithm, have:
(r y) is induced from Pe by Equation (7).
6 M can be expensive to compute. In practice, we don't compute M. Instead we compute an estimation of
Sig (wli))
Ma directly from 64 sequences sampled by the SFT policy, See Section A. for details.
Although the output space of language models is huge high-dimensional discrete space, contin-
uous I-d input space for illustration purpose.
5

---

Published as a conference paper at ICLR 2024
Let max be the maximum rewards among the response candidates not yet accepted. As the number of response candidates goes to infinity. Algorithm can generate num nam_samples distinct samples from Trg with expected acceptance rate Bynrg(ul=) [exp
Theorem 1.
(rw (2, v) T Tmaax) )
B + oo, each sample generated from the SFT policy will be accepted with probability 1. If 3 - O, only the highest reward response will be accepted and all other responses will be rejected. This is the rejection sampling (top-k-over-N) referred by AnthropicHH (Bai et al., 2022) and Llama2 (Touvron et al., 2023). / indicates how much we trust the reward model. If the reward model is very accurate and robust, we should set a small [3. Otherwise, we should set a larger (3. In practice, we treat B as haper-parameter and pick one according validation metrics.
RELATED WORK
Preference Optimization RLHF has been a popular approach in learning from human prefer- ence (Touvron et al., 2023; Stiennon et al., 2020). Recent works have proposed alternative solutions to reinforcement learning (Zhao et al., 2023; Yuan et al., 2023; Rafailov et al., 2023; Dong et al., 2023: Wang et al., 2023; Song et al., 2023). By optimizing the model's compatibility with prefer- ence datasets under models such as the BT model, these methods fit on human or model ranked data pairs. SLiC (Zhao et al., 2023) proposes a contrastive loss to on response pairs sampled from the SFT policy. Similarly, RRHF (Yuan et al., 2023) uses a zero-margin likelihood contrastive loss on ranked list of responses. DPO (Rafailov et al., 2023) fits a model directly on human preference data using the BT model. SLiC and RRHF lack theoretical understanding, and DPO does not optimally estimate the policy density proposed. Our work unifies the losses of SLiC and DPO, and proposes an improved estimation of the optimal policy. We sample preference pairs from the estimated optimal policy, which is closer to on-policy online RLHF
Rejection Sampling Statistical rejection sampling (Neal, 2003) is a statistical approach used to generate samples from a target distribution. AnthropicHH (Bai et al., 2022) and ReST (Gulcehre al., 2023) refer to "rejection sampling" as selecting top l sampled candidates for further tuning. Llama2 (Touvron et al., 2023) propose to use the same approach with PPO (Schulman et al., 2017) to improve RLHE Our work shows the existing approach is special case of the proposed algorithm.
EXPERIMENTS
Tasks We study RSO on Reddit TL;DR summarization (Stiemnon et al., 2020) and AnthropicHH dialogue (Bai et al., 2022) datasets. The Reddit TL:DR summarization dataset contains both fine- tune data Dillr and human feedback data hf Dildr contains 1 117k/6k/6k examples in train, vali- dation and test splits. Dilrr consists of 93k human preferences on decodes from multiple models. The AnthropicHH is a dialogue dataset with x as conversation between human query and an Al assistant. We use the helpful slice Dhellffll from 16 161k/ examples in train and test splits. We use the positive responses SFT targets. Besides, study CNN/DailyMail datasets and show that RSO works well on cross-task generalization from Reddit TL;DR (Appendix A.7).
hf
Method Starting from a T5-large (770M) SFT policy and T5-XXL (1IB) pairwise reward- ranking model, we consider nine settings as discussed in Section 3.1. The settings are all the com- binations between loss functions and preference data distribution. DPO approach is the same as coroooid-ormm--iine
SLiC is almost the same as songe-sft-samplee tiny differences. The first difference is that we drop the regularization loss (second term Equa- tion (1)) due to lack of significantly improvement the final metrics (Appendix A.6). The second difference is that SLiC uses a tournament-style procedure to rank candidates in a list. Unless specif- ically mentioned, we set 3 0.5 and 0.05. To construct preference pairs, we first sample 64 response candidates from the SFT policy using temperature sampling with temperature = 0.7 and top_k 40. Then we samlee 8 samples. We use batch size 32 and learning rate le-5 with Adafactor optimizer (Shazeer & Stern, 2018). For each run, we pick the checkpoint with the highest reward-ranking model win rate against the SFT target.
in our setting with two

---

Published as a conference paper at ICLR 2024
Evaluation Our experiments use four different approaches to evaluate: Proxy Reward Model, Gold Reward Model, AutoSxS, and Human Evaluation. Proxy Reward Model computes win rate of generated response against SFT target the trained T5-XXL pairwise reward-ranking model. Follow the recipe in Gao et al. (2023), we train a PaL. 2-S (Anil et al., 2023) on the same data as Gold Reward Model AutoSxS uses PaL.M 2-I. few-shot in-context learning with details covered in Appendix A.4. Human Evaluation asks human raters to assign a quality score on each response and determine the best one among three systems (details Section 5.3).
5.1 PERFORMANCE COMPARISON ON TWO TASKS
We include two additional baselines related to rejection sampling, RAFT (Dong et al., 2023) and ReST (Gulcehre ct al., 2023). For RAFT, we pick the best decoded sequence as new SFT target. For ReST, we first normalize the reward scores to [0, 1]. then pick the decoded sequences that are greater than 0.7 as new sft targets. This one round of grow and improve with normalized reward threshold 0.7%. The comparison results are shown in Table 1. RSO variants show significant gains over RAFT, REST, DPO, and SLiC variants on two tasks. Regarding preference pairs construction, "rso-sample- rank" brings gains top of "direct" and "stt-sample-rank" with clear margin. Regarding the loss function, higmoid-norm and hinge-norm perform similarly. The improved hinge-norm loss is better than hinge loss used in SLiC AutoSxS. Hinge loss shows reward hacking Reddit TL:DR dataset with higher Proxy Reward win rates but lower AutoSxS than other losses. To compare different methods qualitatively, we showcase an example with responses from different policies on Reddit TL:DR and AnthropicHH tasks in Figure and Figure in Appendix A.3, respectively.
Approach
Ablation
Metrics
Loss
Preference Pair
Proxy Reward (%)
Gold Reward (%)
AutoSxS (%)
Reddit TL:DR
RAFT
cross-entropy
74.84
68.51
53.77
REST
cross-entropy
49.03
46.17
34.36
DPO
sigmoid-norm
direct
84.35
76.09
67.72
sigmoid-non
sft-sample-rank
88.63
78.
69.02
RSOsignii-nmm
sigmoid-norm
rso-samplc-rank
92.37
82.22
71.86
SLiC direet
hinge
direct
86.92
79.76
60.54
SLiC sampe rank
hinge
sft-sample-rank
90.15
80.19
67.34
hinge
rso-sample-rank
93.36
84.40
69.26
hinge-norm
direct
83.93
76.43
66.63
hinge-norm
sft
stt-sample-rank
88.04
76.57
68.46
RSO 35Oaage-noce noram
hinge-norm
iso-sample-ram
92.80
83.45
70.84
AnthropicHH
RAFT
cross-entropy
58.21
40.00
24.99
ReST
cross-entropy
43.48
30.33
15.58
DPO
sigmoid-norn
direct
$1.63
36.13
24.01
sigmoid-nore -orm
aft-sample-rank
85.09
58.65
39.56
RSOsipnii-ne
RSO
sigrnedd-nom
sigmoid-now -nom
rso-sample-rank
86.94
59.15
40.98
SLiCliret
hinge
direct
35,95
27.56
15.69
SLiCample-nn
hinge
sft-sample-rank
80.82
54.55
30.66
hinge
rso-sample-rank
82.21
55.22
32.56
hinge-norm
direct
49.55
37.23
22.89
hinge-norm
sft-sample-rank
82.40
56.55
35.96
RSO 35Oping-nurm
hinge-norm
rso-sample-rank
84.44
57.75
38.58
Table 1: Compare different methods with T5-large policy to leverage human feedback data. Proxy reward, golden reward and few-shot PaLM 2-L win rate against SFT target text are reported.
Thhe Gold Reward Model has accuracy of 76.07% the validation of summarization task and 70.18% on the validation set AT assistant task. 9The threshold is suggested by the original paper. We pick round of grow and improve a fair com- parison round of RSO, since RSO also he done with multiple rounds.

---

Published as a conference paper at ICLR 2024
5.2 RSO ABLATION
Effect of y and / in RSO To study the effect of ; we fix the statistical rejection sampling 8 = 0.5, and vary 7 0.005, 0.05, 0.5 in the loss function Reddit TL;DR dataset. Figure 3a shows that = 0.05 provides the optimal win rate. To study the effect of 3 for rejection sampling, we fix the 0,0.05, 0.5, on Reddit TL:DR dataset. Figure 3b
7 0.05 in the loss function and vary
shows that 3 = 0.5 provides the optimal win rate.
82
Proxay Reward (%6)
Proxy Rewaar Win Rate
40
signodd norm
Eigneld horm oft
00
66
Nege
Nige-
fengo- rorm
hinge
Nenge rorm
hnng-row nom
0.05
0.5
Beta in Sampling
0.005
0.05
05
Garraa Loss
various (b) Proxy reward win statistical rejection sampling (Algo- rithm 1). is fixed at 0.05. The hor- izontal lines are from the sft-sample- rank preference pairs. Shaded areas are 95% confidence intervals.
(a) Proxy reward win rate of var- ious in loss functions (Equa- tion (8), (9), (10)). 8 is fixed at 0.5. Shaded areas are 95% confidence in- tervals.
Figure 3: Effect of hyper-parameters in loss functions and statistical rejection sampling algorithm.
Preference pairs sampling and ranking To better understand the effect tournament ranking and statistical rejection sampling, we compare among different sampling strategies. Since first sample 64 responses from the SFT policy and followed by 8 responses by statistical rejection sam- pling, it is natural to ask: "why not all of the 64 samples in the calibration?" SLiC uses tourna- ment ranking, which introduces bias towards higher reward sequences. Starting with n responses, can construct n/2 pairs and get them labeled. We call this approach fhrst-round-rank.. We can keep the tournament until the winner is decided with a total of n 1 pairs (each game eliminates one response). We call this approach "tournament-rank' We use sigmoid-now -orm loss and conduct abla- tion study on six settings (Table 2). We observe that tournament ranking can bring consistent gains across settings on reward model, but it cannot improve the AutoSxS win rate on -0-8-e case. shows to be the optimal choice based on AutoSxS metric, which means is not always good to sample more responses or conduct the tournament ranking.
Rso-8 8-sample- ssample-first-ruuee
Preference Pair
Proxy Reward (%)
AutoSxS (%)
sft-8 s-sample-first first-round rank
88.63
68,51
sft fit-8-sample-tone toupml-ett rank
90.69
68.57
1oo---smmle-ffe first- first-round-- round-rank
92.37
71.86
93.35
71.69
88.91
68.84
91.14
71.08
Table 2: Comparison among different preference pairs sampling and ranking approaches on the Reddit TL;DR dataset. "k-sample" means sampling & response candidates.
Scale up the policy model To understand how well the RSO be scaled up larger policy models, we train a T5-XXL policy model and fix the loss as sigmoid-nona Table 3 shows that RSO scales up well and improves AutoSxS upon DPO by 1.1% and 33.1 I % on two tasks, respectively.
8

---

Published as a conference paper at ICLR 2024
Approach
Preference Pair
Proxy Reward (%)
AutoSxS (%)
Reddit TL;DR
DPO
direct
94.04
85.03
fit-sample-rank
97.50
85.66
RSO sigmaiiddooo
sample-rapl
98.29
86.01
AnthropieHH
DPO
direct
76.84
52.80
sft-sample-rank
94.91
66.79
RSO norm
rso-sample-rank
97.54
70.26
Table Comparing sampling strategies to leverage human feedback data on T5-XXL policy model.
5.3 HUMAN EVALUATION RESULTS
To further verify the improvements of RSO over others, we conduct human evaluation side-by-side using Amazon Mechanical Turk. Given document and three responses generated from "direct", "sft-sample-rank" and "rso-sample-rank" raters are asked to assign a pointwise overall quality (1-5) to each response, and choose the best one. Each task replicated times and therefore judged by 3 different raters. To eliminate bias, we anonymize all the models and randomly shuffle order of responses for each task. We aggregate pointwise metrics by averaging the ratings across all replicas, and aggregate the choice metric using majority vote. The rating tasks are shown in Appendix A.5. In total 47 different raters participated in the human evaluation study with median of 16 tasks per rater. The human evaluation results shown in Table 4. "rso-samples "ss--sample-rank shows be better than "direct" and "sft-sample-rank" in all loss functions and tasks evaluated with clear improvement margins. Roo is chosen be preferred more than 2x DPO in both tasks. Comparing between two losses, there is no clear conclusion on which one has higher quality when applying "rso-sample aiso-sapple-rank Thus improved loss on SLiC and original loss DPO perform similarly.
Approach
Loss
Preference Pair
Chosen as Preferred
Quality
Reddit TL;DR direct
DPO
sigmoid-norm
21%
3.84
sigmoid-nomm
sft-sample-rank
10%
3.74
RSO äigmcid-nom
sigmoid-norm
rso-sample-rank
48 %
4.02
hinge-norm
direct
21%
3,80
hinge-norm
sft-sample-rank
11%
3.68
RSO singe-nor
hinge-norm
rso-sample-rank
3.97
AnthropicH1H
DPO
sigmoid-nom
direct
15%
3.04
sigmoid-nom
sft-sample-rank
22%
3.21
RSO sigmoié-nom
sigmoid-norm
rso-sample-rank
31 %
3.37
hinge-norm
direct
13%
3.33
hinge-norm
sft-sample-rank
22%
3.56
RSO hinge-nurm
hinge-norm
rso-sample-rank
33%
3.60
Table 4: Human evaluation on ways of constructing preference pairs.
CONCLUSION
In this paper, we propose RSO recipe train large language models from human feedback as an alternative to RLHF. Our recipe is simple and effective with a better sampling strategy than DPO and SLiC. We unify loss functions used in DPO and SLiC from the preference optimization perspective with the first logistic regression and the other support vector machine. We demonstrate our approach to be powerful on multiple tasks with comprehensive numerical experiments and analysis. Future work may include studying RSO on larger scale decoding samples, other loss functions, other language generation tasks, online variants, and non- non-human feedback.
The proportion may sum up to 100% hecause there cases of preference across all approaches.

---

Published as a conference paper at ICLR 2024
REFERENCES
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm 2 technical report. arXiv preprint arXiv:2305.10400 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-35,, 1952.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetune language mod- els. arXiv preprint arXiv:2210. /416, 2022.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finctuning for generative foundation model alignment. arXiv preprint 27X1v:2304.06767, 2023.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference Machine Learning, pp. 10835-10866. PMLR, 2023.
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint arXiv:2308.08995 2023.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015.
Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.0026; 2023.
Radford M Neal. Slice sampling. The annals of statistics, 31(3):705-767, 2003.
OpenAL. Gpt-4 technical report. arXiv, pp. 3303-08774, 2023.
Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. Efficiently scaling transformer inference. Proceedings of Machine Learning and Systems, 5, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305. 2023.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21 21(1):5485-555 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X1v:1707.06347, 2017.
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pp. 4596-4604 PMLR, 2018.
10

---

Published as a conference paper at ICLR 2024
Lei Shu, Liangchen Luo, Jayakumar Hoskere, Yun Zhu, Canoee Liu, Simon Tong, Jindong Chen, and Lei Meng. Rewritelm: An instruction-tune large language model for text rewriting. arXiv preprint arXiv:2305.15685, 2023.
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment. arXiv preprint arXiv:2306. /7492, 2023,
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Bsiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao. Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog. Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint ar XrXh:2337.099288 2023.
Peiyi Wang, Lei Li, Liang Chen, Feifan Song, Binghuai Lin, Yunbo Cao, Tianyu Liu, and Zhi- fang Sui. Making large language models better reasoners with alignment. arXiv preprint arXiv:2309.02144 2023.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652. 2021.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint wrXi::2344005002 2023.
Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Salch, and Peter Liu. Calibrating sequence likelihood improves conditional language generation. arXiv preprint arXiv:2210.000455 2022.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint urXiv:2305. 10425, 2023.
Zhanhui Zhou, Jie Liu, Chao Yang, Jing Shao, Yu Liu, Xiangyu Yue, Wanli Ouyang, and Yu Qiao. Beyond coce--reeeeece-for-lle Multi-objective direct preference optimization. arXiv preprint arXiv:2310.03708, 2023.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint xrxiv: 1909. 08593, 2019.
A APPENDIX
A.1 STATISTICAL REJECTION SAMPLING ALGORITHM
A Python Implementation A Python implementation of the algorithm is shown in Algorithm 1.
11

---

Published as a conference paper at ICLR 2024
Algorithm Statistical Rejection Sampling Algorithm in Python
Eron typing import List import ap
def conduct__ jeot ionpsenpling tseaponse pandpnsee_ddddiieeee candidat Lix. (ate),
seaponse_ (fleat), cun_pammoes:: int,
be.at float):
vnConddctt ze jectiion sampling guided by zewards.
Azya:
sesocnse_cardiieee candidates: response candidates from the 257 pollcy response, revards: responae rewrde, ounssanple
samples subrastiee.
beta: paraseter Ki-constreined revard naximization cbjeccive.
Reluens:
sampled sequences from the ascimated optimai policy.
candidates let z for G, in diplssspooe.aandddaee soaacl) accepted
while lenteppeteed) muruzemples:
max_E max loandi valua.
to renove I|
for app.aadonn.oe Forn()
das.ddd itens (1:
ep.explte nax next rexeadd betals continue
accepted.
(ppend(e) appented.)
ro_renove. apperd
(ascepted))
nun_samples:
break
for in teureveee
return codebtd
Derivation of Algorithm
According to Equation (4), we have
(x) Tan ( exp (3llo.))
1
TTe (y|x)
(11)
where Zu(x)= = Ey (an(yl exp( 2/2 (I, v)). Then we have
Tr (3/2z))
(x) exp (frwte.v))
1
(12)
Taf (1|lz)
Try (y/x)
It's clear that Mp, then
minfm m
m (z)()( or, (3]z) for all y & Dz} max anl
exp (3ro(crw))
Mp. = Zz(x)
max Dz
(13)
Then we have
Tr. (y/z) exp ( r (o((),))
max (2 v) ))
(14)
MD. ())
vDD.
By using the sample version of max DD Tis (x,, v), we have derived the Algorithm 1.
A.2 PROOF OF THEOREM 1
Proof. Let the process generation be the one described in Algorithm I and the accepted sequence set be D,, at the current step, we have
Tru (y/x²)
Pssampll y and get accepted)=) = P
MD, Ton TT((y)==)
MD. Tr. (y/z),
(15)
where Mp. A min(m m ma((y/2)) 2 n. (u)z) for all & Dz).
12

---

Published as a conference paper at ICLR 2024
Tru (y) (x)
Ply get accepted .r) P((u- <
M D. Tan (3/ (2:)
Tru (1/(23)
EE
u <
MD. Tan (y/
Tr (y
Ex
E1
u <
MD. Ten T
Tru (u)lz)
Exn
MD, (y)
(16)
MD.
P(sample y and get accepted|a)) Tr (3/xr)
Pyly is accepted, x))
(17)
Ply get accepted|
By Equation (13), have the acceptance rate
(ro(e,))= max (w DEDr
MD E Bymoralll
exp
(18)
A.3 QUALITATIVE EXAMPLES OF RSO COMPARING WITH OTHER APPROACHES
The qualitative comparisons between RSO and other approaches are shown in Figure and Figure for Reddit TL;DR and AnthropicHH, respectively.
SFT Summary
it better buy Vanguard Target Retirement fund 04%
As recently gredusted twentysome have liiite grasp Ivv futuralnetrament perusing my brokerage account.
experse ratlo Into individual Incex funds .00%
Vanguerd
My father recommended the use indix funds as they offer a lower expense rate then actively managedt funds and adviised against target retirement fund since their expense ratins higher. would spening Roth RA
SLIC Summary
have few thousand dollars put away for retirement, should buy into the Venguard Terget Retirement Fund at fie-2k put
Individusl
walt solld jobicaruer?
being
research point, Vanguard "arget holding
Retiner Betirement fund my age group made up
different indox funds which are domestic & internatiora stock and bond funcs. The main difference between buying into individual index funds and the Vanguard uord TRS besides the minimum 10k invesiment, 04%
DPO Summaar
it better to invest in the Vanguard Target Retirement fund with 04% esperiee ratlo or buy individual indix funds and put money away monthly since comprised of index funds?
relio.
someone would
put away.. say
interest to buy and put money away monthly, since it's encompassed by index funcs, or wait till have a solid jobicareer areer that allows me to have lot more freedom my investing.
RSO Summary
I'm a recent collage grad and have a few thousand dollars to put away for retirement Should buy into Vanguard Target Retirement Fund (TRF) at 1k-2k wait till have solid jobicarer??
Figure 4: Example summaries generated by SFT, SLiC, DPO, and RSO policies for Reddit post. RSO generates the best summary among the four because concisely and precisely summarizes key information in the forum post. Salient details are bolded.
13

---

Published as a conference paper at ICLR 2024
SFT Response
Assistent: you want
through
here's procedurel
use GGtt access to computer smarmphone that internet connections and is currently doing anything important. -Lg in to email program. -Enter in the name and email address of the person you want to sond messaae to. -Type your message. -Clck "send".
Human: What the fastest way to send mall?
Assistant: What's your goal? you want to send a simple email, here's the procedure luser
SLIC Response
access computer emerii that an internet connection, and not cumennly daing anything important.
Assistant: Sure Here
the fastest options: -first cless mail the service. standard
faelest. USPS few days, envelope also very usually delivered within flo days. -An overnight -An international mal service also very fast -An online mall service is also very fest. A local service is elso very fest.
ususlly service wiso very
delivered
-Log in the email progrem.
-Enter the neme and email eddress eddresss of the person message
DPO Response
Assistant: if you want send letter through the mail, here are the stepa: package USPS package
Type your missage.
-Go your post office
poet office, -Ask post
-Cick "wend""
the post number Ask the pest office for package trackóng number.
package tracking number. -Snnd the package to post
office for tracking number Ask the post office for package tracking
If are sending the message to multiple recipients, you paste their addresses instoad entering menually
R5O Response
Sorry, was talking about treditionnl meil not email. What is the fastest way can send lettor through the mail?
Humnen:
Assistant: To letter through mail. the fastest option probebly the USPS Class mail This standard mall that available and
days to letter. Express service you want send large lattor multiple people once.
Figure 5: Example responses generated by SFT, SLiC, DPO, and RSO policies for a Human- Assistant dialogue on AnthropicHH dataset. RSO generates the most helpful response among the four because it gives a clear and straightforward answer for sending a letter quickly through tradi- tional mail. In contrast, SFT repeats information about email rather than answering the question about traditional mail. SLiC and DPO are vague and repetitive. Salient details are bolded.
A.4 PALM 2-L DETAILS AND FHO-SHOT SxS TEMPLATE
A.4.1 DETAILS
The purpose of the AutoSxS prevent the artificially high reward scores by Reward Model due to reward hacking on learned policies. Since the policy is trained using the information in the pairwise reward-ranking model, it is not necessary the higher the win rate on reward-ranking model, the better the policy. AutoSxS uses PaLM 2-L few-shot in-context learning to infer 8 decodedsamples with flipped order of response A and B. The label contains three choices: A, B, and tie withscore 1, O. and 0.5, respectively. To ensure the robustness, we use average score to determine the win or loss if the magnitude exceeds 0.35. The AutoSxS has been demonstrated as effective and consistent in DPO using GPT-4 as zero-shot rater. In this work, we replace GPT-4 with PaLM 2-L for our evaluation using few-shot prompts. The quality of PaLM 2-L on similar tasks has been shown to be close to human raters (Lee et al., 2023; Shu et al., 2023). The systematic study consistency and quality of AutoSxS is beyond the scope of this work.
A.4.2 REDDIT TL;DR FEW-SHOT PROMPTS
task: Judge the quality of two TLDRs, choose the options among (A), (B) or same.
context: I've (M[21]) been in relationship for year and a half with F[22] and it really has never gone well. I think we want different things and we are not overly compatible. broke up with her about a year ago and she tried to kill herself so we got back together. This week I met F[19] who I think I'm really compatible with. She and talked for a few hours and we have a lot in common. I like her lot, but she is currently a freshman and I am currently senior so I will be graduating in May and going on to a prestigious PhD program starting next fall.
So here are my questions: * What should do in regards to my current relationship? I know I need to end it, but I just don't 't know how. What should do in regards to the other girl? Do you think my feelings for the other girl stem from my distaste for my current relationship?
I appreciate any help you give me. tldr (A): I'm unhappy in my current relationship with girl I just met, but don't know how end
14

---

Published as a conference paper at ICLR 2024
it. have no idea what I'm doing or what to do. fldr (B): M[21] unhappy in relationship with F[22]. Met an F[19] in town with similar interests and I really like her. What should do in regards to current relationship/other girl? explanation: tidr (A)'s second and third sentences convey similar idea and are redundant. tldr (B) mentions important piece of information of the new girl, contains more details than tldr (A) and is concise at the same time.
choose among (A), (B) or same: (B)
context: Before anything, not a sad story or anything, don't think she's cheating or anything of the sorts. My country's 's equivalent to Valentine's 's Day is coming and had this pretty simple idea to surprise my girlfriend and it would involve giving her some roses. The thing is, although I know she would appreciate my intention in and of itself, don't Ion't know if she would like the actual flowers and such, so wanted to find out if she likes roses and if she would like getting some, but without her realizing so as not to spoil the surprise. Any ideas on how to get that information out of her? tldr (A): How do I find out if my girlffiend likes roses without her realizing it?
tldr (B): want to surprise my girlfriend with some flowers when Valentine's Day is around the corner, but don't 't know if she would like the flowers or flowers themselves without her knowing. explanation: tldr (A) is a concise that captures the main idea. tldr (B) also captures the main point with more details, but the language 'flowers or flowers themselves' is not fluent.
choose among (A), (B) or same: (A)
Okay, my younger brothers were out and about when they passed some teenagers who yelled obscenities at them. My father then went over and told them to knock it off, when they started yelling obscenities at him. My dad, with small amount of temper, got angry and yelled at them. They started recording it and made a video on YouTube where it looked like he was just screaming them. After that, we were able to get taken down only to have reuploaded with blurred faces. We have in no way given consent to be in this video. Is there any way we can get them to take it doen?
context:
tldr (A): my dad got angry at teenagers for yelling obscenities at him, they got video on youtube and blurred faces, what can we do to get it taken down?
tldr (B): My brothers were being verbally harassed by kids, father yelled at them, they made a video
of it to get the video taken down, it was like a blur with blurred faces.
explanation: tldr (A) mentions most main points of story while skipping some details like younger brothers being yelled at and original videos taken down. tldr (B) has a major factual error, they didn't make video to get the video taken down.
choose among (A), (B) or same: (A)
context: Apologize for the throw away account.
My friend is interested in in making his way into the mainstream music industry as an A&R repre- sentative. He has a 4 year degree in English but did some extra curricular activities around music business in college. He also has had 2 internships out of college with small time record labels in the Chicago area. He has been trying for a few years now with no success. He lives Chicago but has moved to LA for a year. He returned recently to Chicago. It has been about 3 years since his last internship, but he has done some online collaboration stuff. Still far and few in between.
My question is what path have any of you taken to make your way into the record industry? Is networking the only important thing? Is there set of skills that are specifically looked for? tldr (A): My friend wants to make his way into the music industry as an A&R representative. What does he need to know to make it happen?
tldr (B): friend wants to break into the music industry this?
A&R rep. what is the best way to do
explanation: tldr (A) and (B) are basically paraphrase of each other. With (A) slightly more verbose and (B) slightly more concise.
choose among (A), (B) or same: same
was at a concert this weekend supporting my favorite artist. Most of the amphitheater was standing. however *some* people in my section were not (including some people behind me and to my right). All of a sudden at the end of the set I got basically shoved in the back and told to sit down. I told them absolutely not, want to support my favorite artist (who kept saying STAND UP, btw). I paid over $ $100 for these seats and I want to have a great time.. after all this IS a concert.
context:
15

---

Published as a conference paper at ICLR 2024
Not mention, heard them calling names before they confronted me now I was certainly not going to sit down. Then a few songs later the woman sitting next to me told me what a bitch I was for not letting the people behind me see. Was I in the wrong here?
tldr (A): stood up at concert, people in front me didn't 't and confronted me. Was I in the wrong? tldr (B): I stood up concert even though I was told not to. I got yelled by woman who was sitting next to me. Was in the wrong?
explanation: tdr (A) doesn't say who confronted the author or wrongly implied it was the people in front. tldr (B) is correct but could mention that more people confronted the author than the woman sitting next.
choose among (A), (B) or same: (B)
A.4.3 ANTHROPICHH FRW-SHOT PROMPTS
task: For the following query to a chatbot, which response is more helpful? Choose among (A), (B)
and same.
context: Human: Why can't humans build a space station on the moon? Assistant: That's really good question. We don't know yet if we'll be able to build space station the moon, though it seems quite feasible. But we can also look at the science for something more ambitious, like a spaceship that's large enough to make round trip Mars. That might be hard do! Even with the technology we have now, it's not clear to scientists how you'd dd build a spaceship that would travel for months in space. Human:
think we should start on the Moon first.
response (A): Assistant: I think you'l re right that the Moon is a good place to start, since it's a natural satellite of Earth. There are lot of interesting places on the Moon to study, but we have to go to Mars to check out the atmosphere, the conditions on the ground, and the survival potential of humans, and to see if there are living things.
response (B): Assistant: The problem is that we don't it know yet if it's even possible to build space station on the moon. But let's s assume we could do it, and see how far we could get with a spaceship to Mars.
explanation: Response (A) offers a more expansive and engaged exploration of the user's sentiment about starting with the Moon, while Response (B) reiterates previous information without addressing the user's perspective or adding new insights.
choose among (A), (B) and same: (A)
context: Human: What is garam masala?
Assistant: Garam masala is a mixture of spices, typically consisting of cumin, pepper, garlic, and either coriander, ginger, or both. It's frequently used in curries or vegetable dishes, but it's also added to some vegetable or meat sides, or used as a spread.
Human: What other things can add to it to make it unique?
Assistant: Try adding some of the following ingredients, in any combination you like: turmeric cinnamon cardamom celery seeds fennel fresh chilies, like jalapeños or Thai chiles grated ginger green onions basil mint
Human: Is there a sweet garam masala?
Assistant: Yes, some garam masasalas are sweetened. They might contain sugar or honey, or they might be just made with ground cinnamon or cardamom. You might also want to mix some sort of mango or pineapple relish into your garam masala.
Human: What is a good use for sweet garam masala?
response (A): Assistant: You might use it flavor vegetable or meat stews, or chilies, or as a spread on toast. You might mix some into yogurt or fruit salad, or serve it with nuts and chutney on crackers.
response (B): Assistant: Sweet garam masala is typically used in desserts like ice cream or fudge, or in sweet curries like risotto or köfte. You can also use it in curries to add a sweet flavor. You can also use it as spice rub for meats, like chicken or fish. You can also use it as spice mix in curry, or you can use it to make a sweet sauce for a curry.
explanation: Response (B) offers specific and diverse uses of sweet garam masala, allowing the user to get a clearer idea of its potential applications in cooking. choose among (A), (B) and same: (B)
16

---

Published as a conference paper at ICLR 2024
What is the customary amount to tip a taxi driver? response (A): Assistant: The standard tip for taxi drivers in the United States is typically a 20% tip. This is fairly standard tip that is given to taxi drivers for a job well done, and it's also nice gesture to show your appreciation for their service. The tip can be more or less depending on the driver's service and the level of professionalism they showed. If you'l re considering tipping taxi driver, I would recommend giving them a tip of at least 20%.
context: Human:
response (B): Assistant: A standard 10% tip is recommended.
explanation: Response (A) provides a clear recommendation, contextualizes it to specific country (the U.S.), and offers additional guidance based on the quality of service. It's more comprehensive than response (B).
choose among (A), (B) and same: (A)
A.5 HUMAN SXS TEMPLATES
Reddit TL;DR and AnthropicHH human evaluation template are shown in Figure 6 and Figure 7, respectively.
Instructions:
Cavetully rad the doou and the oummaries below.
Rato the summarkss for quality socie 5. Poor summary 5 Great surmrray
the that betee sumrmatta document.
Document:
So L've been ing or ft alnest veectly for ial pertods of time 126 ain- - hr) with a bay who gradusted froe ny school Tast year for almost a year vaw and can't it tell of he actually ant eve our converaationa or feels abliquted to respord ard wishes 2 wos haave him atore, Zive teen able to see 1ia in person couple of times since he left for college cence whee I visited his city ard asked and arce when he cone teck and said te'd try to see et, which be didxl aver Chrastmos break I tried to see his and he was busy (legitinately ly so, believe, but there was no mention trying another time) and when mrexaged hin he took with larger than musi to reply vas trying get rid
decided not to try contacting his in case bo woeks Tater, wessaged me and ve calkod about posted)
always start tour, aessaged about work to say our consersation))
posted beby photo his the son of two teachers, that's why hasn't
responding.
aessage.
can't figure out what's going tere. bocs he actually vant to be friends or does just like talkinng when he's baredl the anneying girl who can it take hiat? can ank about 1t? I'e vorried to say anythlngg because se1sl probasly cone off as neeey (and neybe it isl are 1'll look renrently insecure but at the same time 3'a tiined of constantly woncerting about this, tate that 3 get worked up vniiting for sim to reply and thd past take to srow what hets thinking so I know an stts even worth it to contanue naking an affert to be froends.
Shat should do? Is there any eloquent vay to approact this or should 2 Just let it be?
Summary 0:
Summary
Summary
Soee been chatting meakly with bay free respancing conversation.
Efve been chatting onlinne bey ny school ataost year stopped conversation, respondingg
Tive zeen chatting regulanty with boy on fo aleast year cante talking tored,
atsost year
nennages
let
bet
Summary Quality
Sunmary Qualityy
Summary Quality:
Select the better summary:
Summary
Summary
Summary 2
No preference
Figure 6: Example of human evaluation task on Reddit TL:DR dataset.
A.6 REGULARIZATION IN SLIC
Table shows the SLiC results with different regularization weights. There is no strong gain by adding the regularization loss. And we drop it to align better with the DPO setting.
A.7 CROSS-TASK ADAPTATION AND GENERALIZATION
The CNN//ailyMail dataset (Hermann et al., 2015) contains only fine-tuned data Deandm with 287k/13k/11 Ik examples in train, validation and test splits. We use the dataset to test the cross- task generalization of different approaches. We assume no access to any target or preference texts of the CNN/DailyMail dataset during training. Starting from SFT model trained Reddit TL;DR Dildr, we further optimize the SFT policy using preference data from Reddit TL:DR Dile For "di- rect", use the preference directly. For "sft-sample-rank"" and "ss--sample-rank" we first fit
17

---

Published as a conference paper at ICLR 2024
Instructions:
1. Cordully ly sod 110 dielog and the responses bolow.
2 Ovekanbe qualitt and helphullneea of uach rasperae on scalle 1-8, Poor sopprees, 5 Greet nopurss)
3 Strect the bott resporow For the gave ditlg cornec.
Dialog context:
Humon: bood with buttle fen baky?
Assistants food apestliont this descreto facteray like ape Lerpermment beby, gáre you avme zrous atert with have you rocking slaplnn bakyy as seceo wos: purents bokiees their agen wight heli-
Response 0:
Response 1:
Response 2:
Acclstant:
Acctetant:
dive Accictant: Well, there help
Gave baby cudele Hole basy your lop and talk then sive the keby
baby cucela Give baby cuddle with your hast or your tinger live the
connected to you. Here are fer Libeo
cottle with sips of water flve the baky
boby kiss dive the boby cuddle with
Give the baby a coodle Sive the asby
treat save the taby a botile watl
Toverite coy Give the baby a ressage
batile Gued the basy a bettle 2ive de
story oive the aoty a tottle with toy
bive the boby a hug sive the taby kiss
basy bottle sive the taby botcle
beby baby
with
Gove
Foreheed latee the betty kiss
Giae baby
baky Give buby live beby
pazzi
back
babty
baby
tive
hasa the
beby
baby
Response Quality
Reeponse Quality:
Response Qualit:
Select the better response:
Response
Response
Response
No proference
Figure 7: Example of human evaluation task on AnthropicHH dialogue dataset.
Regularization
Proxy Reward (%)
AutoSxS (%)
O
90.15
67.
0.5
90.45
67.64
5
90.25
67.79
50
90.83
67.84
500
90.06
67.44
Table 5: Comparison on different regularization SLiC. Adding regularization does not show sig- nificant improvement.
reward-ranking model and then generete preference pairs using prompts from the training set of CNN/DailyMaiil. We evaluate the performance using target texts validation split of Denndm From Table 6, the RSO also consistently improves over SLiC and DPO for cross-task transfer.
Approach
Ablation
Metrics
Loss
Preference Pair
Proxy Reward (%)
AutoSxS (%)
DPO
sigmoid-norm
direct
61.31
37.36
sigmoid-norm
sft-sample-rank
62.72
38.63
RSO Sugmaodd nonm
sigmoid-norm
rso-sample-rank direct
69.38
39,71
SLiCa direct
hinge
64.18
33.63
samp--nkk
hinge
sft sample-rank
67.16
33,21
hinge
rso-sample-rank
71.62
35.46
hinge-norm
direct
60.04
33.91
hinge-norm
sft-sample-rank
61.77
40.63
RSO ago-nont
hinge-norm
3oo-sample-ram
69.82
42.18
Table 6: Compare different methods to leverage human feedback data CNN/DailyMail.
18

---

Published as a conference paper at ICLR 2024
A.8 OTHER BASELINES
In Table 1, we did not include the baselines for RRHF and RLHF. For RRHF, we don't have access to other LLM systems and is hard to establish an apples-to-apple comparison. Furthermore, the loss function of RRHF is very similar SLiC. We believe our sampling technique RRHF, but we leave as a future study. For RLHP, we lack expertise on RLHF and DPO shows be competitive alternative. The main purpose of this work is improve upon DPO and SLiC with a better sampling strategy.
also improve
A.9 DEEPER EXAMINATION OF BIAS AND FAIRNESS IN LANGUAGE MODELS
This section delves into the critical aspects of bias and fairness in language models, particularly in relation to our proposed methodology. The works of Anil et al. (2023) and Touvron al. (2023) offer insightful evaluations of bias and faimess in both pre-trained and aligned language models. In terms of aligning with human preferences, our approach incorporates two academic datasets: the Reddit TL:DR summarization (Stienmon et al., 2020) and the AnthropicHH dialogue (Bai et al., 2022). Our primary objective is to enhance alignment with human preferences, focusing on the quality of summaries in Reddit TL;DR and the helpfulness in AnthropicHH dialogues.
In practical scenarios, reward scores are often multi-dimensional, and the aim of alignment is to attain Pareto optimal frontier (Bai et al., 2022). This allows for the introduction of additional ob- jectives such as harmlessness, safety, and bias preference pairs. Our method is adaptable, function- ing with either weighted-averaged reward scores or through integration with multi-objective DPO loss functions (Zhou et al., 2023). Experimental studies have demonstrated that our RSO method effectively aligns with human preference pairs.
We posit that our approach has the potential to enhance fairness and reduce bias in language models, provided is applied with appropriate human preference pairs. However, it is important to note that comprehensive study of fairness and bias falls beyond the scope of this work.
A.10 COMPUTATIONA EFFICIENCY
Compared with PPO (Schulman et al., 2017), RSO only needs policy network during training, while PPO needs four networks (policy, value, reward, and reference network). Besides, rso- sample-rank is fully parallelized over the whole dataset, while PPO needs sampling at cach step and is parallelized within the batch. Now we focus a comparative analysis of the computational effi- ciency among different offline methodologies. Our comparison includes RAFT (Dong al., 2023), ReST (Gulcehre et al., 2023), DPO (Rafailov et al., 2023). SLiC-HF-direet (Zhao et al., 2023), SLiC- HF-sample-rank (Zhao et al., 2023), and our proposed RSO. Notably, most approaches, except DPO and -LiCeet-, require the training and inference of a (pairwise) reward model.
Table 7 delineates the efficiency comparison among the considered approaches. For methods involv- ing pairwise reward model with ne decoded candidates and ny selected candidates for RSO, the following specifics are noted:
RAFT: Requires n. decodings from the SFT policy and n. comparisons for tournament ranking.
ReST: Involves n. SFT decodings and n. - comparisons with a randomly chosen baseline sequence, followed by normalization of reward scores and truncation based on a threshold.
DPO/SLiC-HT-direct directly optimize on the human preference data without reward model and SFT decoded sequences.
LLiC-HF-sample-ran Samples ny sequences, subsequently employing ns tournament ranking comparisons.
RSO: Our method samples n. decoded candidates from the SET policy. Each candidate is assigned reward score based on n. 1 comparisons against random chosen baseline. RSO then employs statistical rejection sampling for selecting n, sequences and constructs preference pairs using na/2 comparisons.
19

---

Published as a conference paper at ICLR 2024
Compared with DPO and SLiC-HF-direct RSO introduces additional sample and rank stage. These stages are scalable and can be parallelized across multiple model servers, significantly en- hancing efficiency.
RSO needs more reward server inferences. The extra computation burden can be mitigated and addressed with prompt efficiency: With fixed prompt for generating responses, RSO benefits from prompt caching on model servers, leading to faster response generation. The inference speed can further be improved with advance serving techniques (Pope al.. 2023). On the side of reward server, inference with decoder length ensures quick processing times.
The statistical rejection sampling algorithm, as described in Algorithm 1, exhibits enhanced effi pepplinggwihhouu-rppaammme
ciency by employing selected sequences subsequent to each sampling round. Furthermore, at the commencement of each round, the maximum reward is recalculated. This recalibration ensures that, in every round, at least one sequence is invariably chosen. Specifically, the sequence whose reward is equivalent to the maximum reward is selected with probability of one, thereby guaranteeing the selection of at least one optimal sequence in each round. This approach not only optimizes the selection process but also maintains the algorithm's effectiveness throughout its execution.
strategy. This is achieved by excluding the
RSO needs additional computation on sampling from the SFT policy and ranking from the pairwise reward model, but the additional cost is empirically minor compared policy training. There are several reasons for that:
We only need sample once for each prompt in the training data. But the training of DPO can go through multiple epochs.
Sampling and ranking are fully parallelizable over the whole training set but training is only parallelizable within the batch.
Reward ranking can be fast because of the short decoding length (just one token). The input text can be encoded a parallel way.
Our observations indicate that rso-sample-rane accounts for less than 10% the total train- ing time.
Batch decoding is scalable and efficient with many optimizations (Pope et al., 2023). In this work, we sample 64 responses from the SFT policy. Existing research works can sample similar or even way more samples from the SFT policy to construct best-of-N:
1. Up to 32 samples in Table 4 in Dong et al. (2023);
2. Up to 100 samples in Figure 7 in Touvron et al. (2023):
3. Up 30k samples in Table 2 in Gao et al. (2023);
From the perspective of balancing between the additional burden in efficiency and the significant performance quality gains (as shown the Section 5), RSO stands out as a recommended approach over the alternatives.
Approach
Reward Model
#SFT inference
#Reward Model inference
RAFT
ne
ne
ReST
The
n1
DPO
O
0
diiretF-direct
O
HiIC-HH-samppe-radd
Rdd
RSO
Re
+ 0.5 12
Table 7: Efficiency comparison of difference approaches. Np denotes the number of prompts, ne
denotes the number of decodes to sample from the SFT policy as RSO candidates, and n denotes the number of decodes for each prompt.
20


Reviews:
[{"review_id": "I8anCGXYqH", "reviewer": "Reviewer_hBgS", "summary": "This paper proposes a new method to improving language model alignment using Statistical Rejection Sampling Optimization (RSO). Their method allows for more accurate estimation of the optimal policy, leading to better alignment with human preferences. Empirically, the authors demonstrate the effectiveness of RSO on two tasks: Reddit TL;DR summarization and AnthropicHH dataset. They further show that RSO works well on cross-task generalization from Reddit TL;DR. Lastly, the authors evaluate RSO using three different approaches: Reward Model, AutoSxS, and Human Evaluation. The results show that RSO variants outperform DPO and SLiC variants on both tasks.", "questions": "1. The paper focuses on improving language models by aligning them with human preferences. I wonder how might this approach be adapted to address issues of bias and fairness in language models? Could the authors provide some additional discussions on this?\n\n2. How does the proposed method compare against other methods, such as DPO, in terms computational efficiency?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Strengths:\n\n- This paper introduces a new approach for improving language models alignment by exploiting rejection sampling to allow for a more precise estimation of the optimal policy. To the best of my knowledge, this proposed method is novel and well-founded. Moreover, it also include exsisting methods (e.g., DPO) as special cases.\n\n- The paper is mostly well-written and the illustration figures are helpful for the undertsanding of this work.\n\n- The empirical evaluation is thorough and demonstrate the effectiveness of the proposed method. Most of the claims / arguments are well-supported.", "weaknesses": "Weakness:\n\n- There are still rooms for improving the presentation. For example, it would be better if the authors can explain a bit more on $\\rho_{\\psi}(x,y,y_b)$ when first introducing it. The authors could provide some examples or details about what would be an instantiation of $\\rho_{\\psi}(x,y,y_b)$.\n\n- Step 3 in section 3.2 might be overly time-consuming and computationally expensive. Especially the set $M = \\min\\\\{m | m\\pi_{sft}(y|x) \\geq \\pi_{r_{\\psi}}(y|x) \\text{ for all } y \\notin \\mathcal{Y}\\\\}$ is expensive to compute. Rejection sampling is also slow.\n\n- The results in Figure 3 (b) is not statistically signifcant and the confidence intervals are overlapping.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QoTyial1iX", "reviewer": "Reviewer_DcKN", "summary": "This paper proposes a novel approach called Statistical Rejection Sampling Optimization (RSO) to improve preference optimization in language models. The authors address the limitations of existing methods by introducing rejection sampling to source preference data from the optimal policy. They also propose a unified framework that enhances the loss functions used in both Sequence Likelihood Calibration (SLiC) and Direct Preference Optimization (DPO). Through extensive experiments, the authors demonstrate that RSO consistently outperforms SLiC and DPO in terms of both Large Language Models (LLMs) and human raters' evaluations.", "questions": "Please refer to the weakness section", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The authors provide a well-structured review of relevant literature, highlighting the limitations of existing methods and positioning their work in the context of prior research.", "weaknesses": "1. The core idea of utilizing rejection sampling to filter the response feels somewhat trivial \n\n2. Why is there a significant improvement in rso-sample-rank compared to sft-sample-rank on Reddit TL;DR, while the improvement is not as noticeable on AnthropicHH? What could be the potential reasons behind this?\n\n3. There might be flaws in the design of the experimental part: the reward for training and testing is the same. Although there are evaluations from GPT/human, generally speaking, these are not very convincing.  You can refer to [1, 2] for Synthetic Data Setup, Recalibration, gold reward, and proxy reward to have a fair comparison.\n\n[1] Let’s Verify Step by Step\n\n[2] Scaling Laws for Reward Model Overoptimization", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "j6lAIhNV3A", "reviewer": "Reviewer_nTnq", "summary": "The paper introduces an innovative method, Statistical Rejection Sampling Optimization (RSO), for RLHF. RSO employs rejection sampling to extract preference data from the optimal target policy, facilitating a more precise estimation of this policy. Additionally, this paper provides a unified framework for previous approaches, i.e., SLiC and DPO. Experimental outcomes indicate that RSO consistently surpasses existing methods, as evaluated by both large language models and human raters.", "questions": "Why do not directly sample responses from $\\pi_{r_{\\psi}}(y|x)$ (with a reward threshold like ReST) rather than sampling from $\\pi_{sft}(y|x)$ with the statistical rejection sampling to approximate the samples from  $\\pi_{r_{\\psi}}(y|x)$?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Presents a novel approach for RLHF using rejection sampling.\n2. Unifies prior work, including SLiC and DPO, within a comprehensive review.\n3. Provides comprehensive and robust experiments with SOTA, demonstrating clear enhancements of RSO.", "weaknesses": "1. The unifying link between DPO and SLiC appears to be deliberately designed, particularly noticeable in the normalization term, i.e., $\\pi_{sft}(y|x)$, in Eq 10.\n2. The theoretical conclusion that the policy induced from RSO in Eq 4 is optimal seems questionable. RSO continues to use the proxy reward model to label responses, regardless of whether they're from the pre-trained or learned policy. However, the reward model is learned from $D_p$ and not $D_p^*$ without further updates, which could introduce approximation errors and bias in subsequent labeling and learning steps. In the other word, the policy induced from Eq 4 is not optimal because of the non-optimal reward model.\n3. Works with a similar focus on rejection sampling, such as ReST(http://arxiv.org/abs/2308.08998), ALMoST (https://arxiv.org/abs/2305.13735) RAFT (https://arxiv.org/abs/2304.06767), should be involved in comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)