Compositional Preference Models for Aligning LMs
Authors: Dongyoung Go
Tomasz Korbak
Germán Kruszewski
Jos Rozen
Marc Dymetman
Venue: ICLR 2024 poster (2024)
Keywords: language model alignment
preference model
Reinforcement Learning from Human Feedback (RLHF)
overoptimization
interpretability
scalable oversight
reward hacking
Primary area: generative models
TL;DR: We decompose the preference assessment into interpretable features using a prompted LM and aggregate their scores using a logistic regression classifier.

Abstract: As language models (LMs) become more capable, it is increasingly important to align them with human preferences. However, the dominant paradigm for training Preference Models (PMs) for that purpose suffers from fundamental limitations, such as lack of transparency and scalability, along with susceptibility to overfitting the preference dataset.
We propose Compositional Preference Models (CPMs), a novel PM framework that decomposes one global preference assessment into several interpretable features, obtains scalar scores for these features from a prompted LM, and aggregates these scores using a logistic regression classifier. Through these simple steps, CPMs allow to control which properties of the preference data are used to train the preference model and to build it based on features that are believed to underlie the human preference judgment.
Our experiments show that CPMs not only improve generalization and are more robust to overoptimization than standard PMs, but also that best-of-n samples obtained using CPMs tend to be preferred over samples obtained using conventional PMs.
Overall, our approach demonstrates the benefits of endowing PMs with priors about which features determine human preferences while relying on LM capabilities to extract those features in a scalable and robust way.

Published as a conference paper at ICLR 2024
COMPOSITIONAL PREFERENCE MODELS FOR ALIGNING LMs
Dongyoung Go
Tomasz Korbak University of Sussex tomasz. korbszzarraa com
Naver Corp
Yonsei University
dongyoung. beeayyonnggeeaaaerrrrp.ooo goenavercorp.com
Germán Kruszewski, Jos Rozen
Mare Dymetman Independent Researcher marc c dyme tman@gmail ccmm
Naver Labs Europe
(german. kruszewski jos. rozen) @naver com
ABSTRACT
As language models (LMs) become more capable, it is increasingly important to align them with human preferences. However, the dominant paradigm for training Preference Models (PMs) for that purpose suffers from fundamental limitations, such as lack of transparency and scalability, along with susceptibility overfitting the preference dataset. We propose Compositional Preference Models (CPMs), a novel PM framework that decomposes one global preference assessment into sev- eral interpretable features, obtains scalar scores for these features from a prompted LM, and aggregates these scores using a logistic regression classifier. Through these simple steps, CPMs allow to control which properties of the preference data are used to train the preference model and to build it based on features that are believed to underlie the human preference judgement. Our experiments show that CPMs not only improve generalization and are more robust to overoptimization than standard PMs. but also that best-of-n samples obtained using CPMs tend to be preferred over samples obtained using conventional PMs. Overall, our ap- proach demonstrates the benefits of endowing PMs with priors about which fea- tures determine human preferences while relying on LM capabilities to extract those features in scalable and robust way.
INTRODUCTION
As the capabilities of language models (LMs) continue to advance, there growing need for safe and interpretable models. The dom- inant approach to aligning LMs with human preferences, reinforcement learning from hu- man feedback (RLHF: Ouyang et al. 2022] Bai al. 2022a OpenAl 2023), consists training preference model (PM) to predict human preference judgments and then finetun- ing an LM to maximize the reward given by the PM. However, the current PM methodol- ogy exhibits certain limitations. First, it is sus- ceptible to overfitting the preference dataset. The PM can misrepresent human preferences by fitting to spurious correlations its train- ing data Gao et al. (2023). Heavily optimiz- ing an LM against PM incentivises the LM to exploit those flaws. This effect is known reward hacking or Goodhart's law (Goodhart)
Compositional preference model
How correct respense? (response)
8/18
How informative respense? (response)
3/18
Σ
1.2
How readable da this respense? (resperssa)
1/10
Standard preference
(response)
1.8
Figure 1: Compositional preference models score different features of LM responses separately and output a preference score as a linear combination of feature values.
1984). One way of addressing reward hacking

---

Published as a conference paper at ICLR 2024
is to impose certain inductive biases the PM or limiting its capacity. Second, PMs are often dif- ficult to interpret and to oversee They project preferences onto a single scalar feature, making it difficult know what factors are influencing their decisions. This is especially problematic for com- plex preferences, such as helpfulness or harmlessness, which often encompass a multidimensional combination of attributes (Bai et al. 2022a Glaese al., 2022] Touvron et al. 2023). Further, as LM capabilities improve, it will be increasingly harder for unassisted humans to provide feedback on LM's responses (Pandey al. 2022; Bowman al. 2022a). One way of addressing this problem is to use another LM to decompose those responses into simpler pieces that can be evaluated either by a human or an LMM
In this paper, we propose the Compositional Preference Model (CPM), novel framework for learn- ing a PM that robust to preference model overoptimization and allows for more transparent and interpretable supervision of complex behavior. A CPM decomposes one global preference assess- ment into series of simpler questions which correspond to inttrannttrrretable features. Then, a prompted LM (e.g. GPT-3.5) is asked to assign numerical value to each feature. Finally, the feature scores are combined into a scalar preference score using a trained logistic regression classifier.
CPMs have several advantages over standard PMs. First, they are more robust to overfitting and re- ward hacking. The pre-selected features on which CPMs operate provide a useful inductive bias that bootstraps learning human preferences. This, in turn, limits their vulnerability to reward hacking, as the parameter space of PM is spanned by features selected to be meaningful and robust. Second, CPMs allow for the modular and human-interpretable supervision of complex behavior. They ef- fectively decompose hard question (e.g. "is this text preferable?") into a series of easier questions (e.g. "is this text easy to read?", "is this text informative?")) that are easier evaluate for an LM and easier to inspect for a human overseer. This is a simple instance of a divide-and-congee approach (Cormen al., 2022). , which recursively breaks down a problem until it easily solvable and then combines the solutions (Irving et al.) 2018) Leike et al. 2018 Christiano et al. 2018).
supervision
In our experiments, we show that CPMs generalize better and that using them results in less pref- erence model overoptimization. Additionally, CPMs exhibit superior performance in capturing the underlying human preferences. In an auto-evaluation experiment with Claude (Anthropic) (2023) as an approximation of human evaluators (Chiang et al. [023 Mukherjje et al. [023 Liu et 2023 He et al., 2023), best-of-n samples obtained using CPMs are consistently preferred over samples obtained using conventional PMs]
Overall, the contributions of the paper include:
1. Introducing CPM, novel framework for learning PMs that is more robust to overoptimiza- tion and allows for more transparent supervision, by decomposing the preference problem into series of intuitive features linked to human preferences, and employing LLM as feature score extractor (Sec.
2. Investigating the performance of CPMs on a diverse array of dimensions, including model robustness (Sec. 4.2). generalization (Sec. 4.3 robustness to overoptimization (Sec. 4.4)
and effectiveness for preference alignment (Sec. 4.5).
3. Enabling an intuitive explanation of model optimization and generated responses (Sec. 4.6
2 BACKGROUND
Let us have dataset of comparisons D {xx vi V2 where is input query and y1 and ya are two possible responses to x, with y1 the preferred response. The dominant approach to aligning
language models, RLHF (Christiano et al. 2017 Ziegler et al., 2019 Ouyang et al., 2022 Bai
ct al. 2022a 2 involves training a parametrized PM R(y|z) = Re(y(=) by defining probability distribution
(y > /2(z) = o (Ro(y1 (x) (x2l=)) (1 + exp (x) (yy (y)
(1)
and estimating 9 by maximizing the likelihood of pa over D. Typically Rs is obtained by adding a
scalar head on top of a base language model and fine-tuning the resulting model. Since pa is invariant
to addition of a constant to Ro. it is standard to shift the scores such that E(r.)) [(y]=)) = 0.
"ooee accompanying the paper is available at httpgggiihiiuooooooogoooooooo
*CPMs also be used with other alignment training methods both during pretraining (Korbak et al. 2023
and finetuning (Rafailov al. 2023 Go et al. 2023).
2

---

Published as a conference paper at ICLR 2024
3 METHOD
The Compositional Preference Model (CPM) is a multi-step approach for decomposing preference learning into individual components. We first decompose preference judgements into set of C distinct features, each designed to evaluate a specific aspect of the response y (relative to context x). Then we use prompted LM to assign a pair (æ, y) scalar score for each individual feature c I,..., C. Finally, we employ a logistic regression classifier to combine these features into a global scalar score that best predicts the human preference judgements. This approach enables us to construct a coherent description of the characteristics that underlie these judgements.
3.1 FEATURE EXTRACTION USING A LANGUAGE MODEL
For each feature 6, we consider an individual preference model R. that maps an input query æ and response to scalar score. In order to do that, we associate each feature c with a specific prompt to and compute a score тę = Re(y/a x tc), where Rg can be a general LLM like GPT-3.5, prompted with a combination of te. Z, and y. These features designed to decompose the broad concept of preferability into a series of more straightforward and interpretable components D In general, the features should be "diverse" enough that they can the broad concept of preference, yet without too much "overlap" between them to decrease efficiency and interpretability. worthy that a feature can represent not only positive categories that are aligned with preferability (e.g. informativeness), but also categories that are assumed to be negatively correlated with it (e.g. biasedness). This procedure allows us to control which properties of the preference data are used to train the PM and to build it based on components that we believe to determine the human choices.
note-
3.2 COMBINING MULTIPLE FEATURES
The features assessed by the prompted LM serve as distinct modules, each which evaluates a different aspect. To combine the features into an interpretable single model, we employ logistic regression to classify the preferred response in a pairwise comparison dataset
Based on the dataset D {0r, vi. y J 1= obtain a feature matrix {z', r(yil="), r())=")
Here r xr) = (R1 (y xz ti), ..., (yly,, tc)) is a feature vector with decomposed feature scores. We standardize each feature score to have average 0 and variance within the train data. We then compute the pairwise difference of the feature vectors for each pair of responses, r(y) (x) - -r(j2 (z), and train logistic regression classifier with this difference to predict if yı is preferred, and 0 if y/2 is preferred. In other words, the distribution is formalized as:
ply1 > y/ (x) = ((A, (wn x)) r(va (x))) = (1 + exp((A, ((n|(=))))
(2)
where À (A : Ac:) is the vector of fitted coefficients. The coefficient A, indicates the impor- tance of the feature e for predicting human preference judgements. To obtain the preference score of single sample we simply compute (A, ryla=) 0) = (A, r(ylz)), where is the standardized average of the feature vector ryy|z) over the training data as explained above.
4 EXPERIMENTS
In this section, we empirically evaluate CPM on several aspects, including model robustness
(Sec. 4.2). generalization (Sec. 4.3) robustness overoptimization (Sec. 4.4), and effectiveness
for preference alignment (Sec. 4.5
We also provide an illustrative example of CPM interpretability
in Sec. 4.6
4.1 EXPERIMENTAL SETUP
Datasets We conduct experiments on two datasets, the HH-RLHF dataset (Bai et al. 2022a) and the SHP dataset (Ethayarajh et al. 2022). Both consist of pairs of responses based on helptulness.
3 See Sharma al. (023) and Hosking al. 2023 for further evidence that human preference judgements can accurately predicted from a linear combinations of such features.
"Expanding pairwise comparisons to rank data is possible, following the general approach of one-vs-one
Ouyang 2022).
3

---

Published as a conference paper at ICLR 2024
For each dataset, in order to establish consistent setting and control for the data size factor, sample 20K single-turn turl data points.
Features We use 13 features: helpfulness, specificity, intent, factuality, relevance, readability, enough-detail, biased, iww.-oosiiie---iiiidud-----eeeee
easy-to-understand,
repetitive, fail-too-consider-conttex and too-long, with pre-specified prompt templates (see App. C for the description of features and prompts). We use the same set of features for both datasets; prompt templates only differ in preamble that describes I as either a conversation with an AI assistant (HH-RLHFF or StackExchange question (SHP). We also use the length of y, which we find to be helpful on the SHP dataset.
Methods To find out the ability of an LM as feature extractor, we explore two LMs, GPT-3.5 18pt-3.5-tur 5-tu5-o--o0000 0301) and Flan-T5-XL (3B parameters) (Chung et al. 2022), using the same fea- tures and prompt templates. We refer to the CPM models based on these extractors as CPM-GPT-3.5 and CPM-Flan-T5, respectively. To select only the most important features, add regularization term in logistic regression and use hyperparameters selected with 5-fold cross-validation on the training dataset.
We then compare the conventional PM to these CPMs (trained respectively as described in Sec. and Sec. 3.2. For a fair comparison, we train the standard PM based on the same model that we use for the CPMs, but with an added linear head that outputs scalar preference score. We compare the performances of CPM-GPT-3.5 and CPM-Flan-TSS with this standard PM. Implementation details are provided in App-A
Best-of-n sampling (BoN) To assess the robustness of PMs to overfitting, we use Best-of-n (BoN) sampling (Gao et al. (023), simple yet effective method that has been shown to be competitive with more advanced techniques such as reinforcement learning (Hilton Gao 2022). BoN abstracts
away from RLHF design choices such the details of policy optimization and provides stable
proxy for RLHF performance (Nakano et al. 2021: Gao et al. 2023).
We generate responses using initial LM a(x) and evaluate the performance of the PMs on these responses. We consider the BoN distribution x ~ Po, where n candidates are sampled from and is the candidate maximizing the PM score. Following Gao et al. (2023), compare the robustness of two related PMs, PM, (2) and PM)(z), by measuring the gap between their average scores relative samples x from BoN(a, PM n), where typically (by construction) we have (x) > PMa(r). with the gap increasing with v
We generate up 25,600 BoN responses, with 256 responses for each of 100 prompts in held-out test set We use Flan-TT5Larg (780M parameters; Chung et al. 2022) as the initial LM to generate the responses. To ensure that the performance of different PMs can be compared on the same scale across different reward models, we normalize each PM score to have average 0 and variance 1 within the training data.
4.2 MODEL ROBUSTNESS
Model robustness refers to the sensitivity of predictive model to the selection of its training data (Hastie et al., 2009). Specifically, quantifies how much the model's predictions would change we were to train it on different subsets of the preference dataset. A model with low robustness will show poor generalization on unseen data.
To assess model robustness, we independently train two PMs for each PM method, PMA and PM B, on disjoint subsets of the training data, each of size 10K. We then conduct BoN experiment and check whether the scores of these two PMs diverge with increasing n. As explained above, we pick the response with highest PM, score among n samples and measure the gap between the scores of PMA and PMg that sample
'The PM used for the BoN distribution is determined by the experimental design (c.g. proxy PM in the overoptimization experiment).
Due to computational constraints, we only evaluate CPM-GPT-3.5 on BoN(n S 16).
We tested reversing the order for building BoN distribution, and the results remained unchanged. See Fig. in the Appendix.
4

---

Published as a conference paper at ICLR 2024
(a) HH-RLHF dataset
Stadard PM
CPN-PTTS
CFM-TaY-T
HR Alscedto WEPUO
12
MEAD VONM
644
43
23
06
16
133
226
16
35
64
323
256
(b) SHP dataset
Sunded PM
CINCPISS
Rt Alssed acevant
PM- ber oreman!
SE aroval
2.3
05
02
0.5
16 324
36
324
Figure 2: BoN comparison over two models fitted independently in same condition (left: Standard PM, middle: CPM-GPT-3.5, right: CPM-Flan-T5), PM A (blue line) is used for BoN selection.
Fig. shows that CPM is significantly more consistent between PM, and PMg than the standard PM method in terms of the score differences, even for BoN with size 256. The smooth scaling trend as function of n suggests that our findings will generalize larger n. This suggests that the small number of trainable coefficients (in this experiment 14 coefficients) makes the model robust to noise
in data sampling. Still. the features extracted by LM are informative enough build an effective cenference monee
preference model for alignment tuning, as we illustrate below.
4.3
COMPARISON WITH REFERENCE PMS
252
335
PM
0.5
659
Number
Number 20V
Figure 3:
Comparison between PM scores relative to the distributions BoN(a, PM n) (HH- RLHF dataset, left) and Bo PM n) (SHP-dataset, right).
To assess the generalizability of our CPMs, we compare them to two well-established reference PMs, PM and PM,2, both instances of DeBERTa (He et al. 2020), with PM finetuned on a large dataset including H and PM finetuned on large dataset including SHP (Sileo) 2023). These PMs, trained on larger and more diverse datasets, are shown to generalize better than PMs trained on a 10K dataset (see App. B). We select BoN responses with the reference PM and then examine how their scores diverge relative to the different PMs trained on a 10K dataset in Sec. We hypothesize that models that diverge less from such independently trained reference PMs will generalize better to unseen data. Fig. shows that all models scale monotonically with the reference PM, with the CPMs staying closer to it. This suggests that the extracted features are informative enough allow for learning more generalizable model of preference judgements.

---

Published as a conference paper at ICLR 2024
4.4 ROBUSTNESS TO OVEROPTIMEZATION
Bas Prcy Chfeaans Baay slaebaalin
14 24 Tpp FEB GP
Paay sinisalii
32
Namber of ropparics DUN
Numbor of rosporses 2012
Figure 4: Overoptimization experiment in BoN distribution BoN(a, PMProxy, n). Dashed line means proxy PM used for BoN selection, corresponding solid line means gold PM. (left: HH-RLHF dataset, right: SHP dataset)
Overoptimization is a type of misalignment that occurs when the preference model is overly opti- mized by exploiting flaws in the proxy objective (Amodei et al. 2016 Skalse et al., 2022). This can lead to the PM diverging from the true objective, which we want optimize in alignment tuning.
To investigate overoptimization, we follow Gao et al. (2023) and construct a synthetic dataset where the output of specific "gold" PM is assumed to be the ground truth for preferences. As gold PMs, we use reference PMs PM and PM, (described in Sec. 4.3). We then use the gold models to generate synthetic labels to train proxy PMs using each of the studied techniques. Depending on the
PM training method, overoptimizing the PM can cause it to diverge from the gold PM, which allows us to compare the robustness of different PM techniques.
Fig. shows that the gap between the gold PM and the proxy PM scores increases for each PM as the candidate size n increases. The distribution of the standard PM does not follow the gold PM distribution and has larger divergence as the candidate size increases. This illustrates that fitting a standard PM can lead to overoptimization, which is consistent with existing literature (Gao et al. 2023). On the other hand, the gap between the gold and proxy PM scores is smaller for CPMs, with the gold PM score beginning to diverge later than for standard PMs. This suggests that CPMs are more robust to overoptimization. The rank correlation of the PM scores with increasing n in Fig. which measures this quantitatively, is provided in Table 9 in the Appendix.
4.5 QUALITY EVALUATION
The ultimate goal of PMs is to help align LMs with human preferences. While in the previous section we compared PMs with a certain gold PM, in this section we will investigate whether LMs
aligned using CPMs preferred by humans over LMs aligned using standard PMs. Following
previous literature (Chiang et al. 2023 Mukherjee et af. 2023 Liu et al. 2023 simulate human evaluation using prompted LLM.
He et al. 2023 we
For each PM, we draw a response from BoN(a, PM, 16) by generating samples from a (namely Flan-T5) and selecting the best response based on the PM score. We then compare this response to vanilla Flan-T5, namely response randomly selected from the same set of candidates. We finally use the LLM choose which response is preferable. We refer to this metric as the "win rate". A good PM is expected to have high win rate against vanilla Flan-T55
Importantly, we use Claude (claude-2, Anthropic 2023), an LLM that was not used in feature extraction. Hence, we avoid potential subtle preference leaks from features extracted usig GPT-3.5. We use the prompt from (Chiang et al.] 2023 Mukherjee et al., 2023) to rate the quality of the response selected by each PM method"
70 (see Tab. 8 for the prompt used in evaluation). We perform one BoN trial with n = 16 for CPM-GPT-3.5 and 10 independent such trials for other PMs and report the average win rate.
Too prevent the known bias towards the first response Chiang al. scores with different orderings when making comparison.
2023 OpenAl 2023 we average the

---

Published as a conference paper at ICLR 2024
Win Rate
HH-RLHF
SHP
Tab. I shows evaluation results. Considering that both standard PM and CPM-Flan-TS use the same architecture and data, the higher win rate of CPM-Flan-TSS compared to standard PM suggests the advantage of decomposing prefer- ence into multiple features and using an LM as feature extractor, rather than directly using the PM based on fine-tuning the LM as in Eq. (T). CPM-GPT-3355 shows an even higher win rate, again indicating that using more powerful LM as feature extractor can further improve the per- formance of CPM.
CPM-GPT-3.5 CPM-Flan-TS
0.810
())
0.672 (.)
0.742 (0.034)
0.580 (0.045)
Standard PM
0.588 (0.030)
0.564 (0.037)
Table 1: Win rate over initial generation after
BoN sampling based on each PM. Except CPM- GPT-3.5, we independently conduct 10 rounds of BoN(n 16) samplings and report the average win rate along with standard error.
4.6 MODEL INTERPRETABILITY
CPMs. as linear models, have a high degree of interpretability Hastie et al. (2009). In this section, we provide few illustrative examples focussing on the dataset HH-RLHF.
The interpretability of our model is enhanced by the fact that the feature coefficients provide a direct indication of the factors that most influence the CPM's decisions. This information can help understand the CPM's internal workings. Tab. 2 shows the top 3 largest coefficients (see Tab. TO for full coefficients). Although the coefficients vary as they are extracted with different LMs,, their orders are generally consistent, except for few features. This observation provides some clues into how the CPM makes its decisions. In the current example, the CPM focuses on general helpfulness and also prefers responses that are detailed enough but also factually correct.
Coefficients
CPM-GPT-3.5
CPM-Flan-TS
Feature
Coefficient
Feature
Coefficient
helpfulness
0.246
fail-to-consider-conees
0.420
enough-detail
0.235
enough-detail
0.244
factuality 0.187
factuality
0.227
Table 2: Three largest CPM coefficients on HH-RLHF dataset.
LM-extracted features The features extracted by the LM enable intuitive explanation of gener- ated responses. This allows supervising complex behavior in a human- interppateerretablle way. Tab. 3 shows examples of these features, which can be used to identify which aspects of the response con- tribute most to the predicted preference judgement. By decomposing a hard preference ("This text is not preferable.") into series of easier features ("This text is generally unhelpful, as it is easy to read but has little detailed information"). it allows easier inspection for a human overseer.
5 ABLATION STUDY
In this ablation study, we evaluate the impact of two factors, data size and size of the LM for feature extraction on the HH-RLHF dataset.
Data scaling trend To investigate the effect of data size model robustness, we hold the model size constant (3B parameters) and vary the data size used to train the PMs. We independently train each PM method increase the data size from 100 to 10,000. Fig. 5 shows the results of the model robustness exper- iment. CPMs rapidly become consistent as the data size increases and achieve stable consistency between two PMs with data size of over 500. In contrast, standard PMs show poor consistency between models, especially when the data size is small. This suggests that CPMs are more robust than standard PMs and can produce reliable results even with a small amount of data.
disjoint subsets of the training data, described in Sec. 4.2 We gradually
Model scaling trend To investigate the effect of the size of the LM used for feature extraction, we gradually increase this size from Flan-T5 "small" (80M parameters) to "XI." (3B parameters) and
7

---

Published as a conference paper at ICLR 2024
Features
enough deall
to-comider
helpfelness
fecuiiity
Dialogue
Q: What things cun I create with java yrugrammring?
progrmmmig language designed create applications compuser.
2.117
R2: Robots
0.309
0.559
-0.075
-0.569
-0.239
Lucifer referred "moning
R1: Lucifer the Morning Staz because is brightest stur the sky.
1.209
0.880
1.001
0.845
0.232
angel
Q
art perhups. Can you tell me history about Mona the painting?
RI: The Mona Lisa famous puinting in the Lousre. Paris, France. attributed Leunardo Vinci.
1.909
0.880
1.901
1,690
2.588
82: was printed by Gina Lorenzo Sico the lite 88th century. The name the painting is inspired by painting deseribed
stocy fumous Italian Leonardo "portie" quality.
0.278
-0.239
Table 3: Examples of feature values of CPM-GPT-3.5, Each feature value provide information about which aspects the response are good or bad.
1.2
Data Size
ICE
Sizt
25
Data
3530
39300
2.0
I0e3
10093
1093 L0930
0.8
Staee
3.4
80
Numberot
respenees
Figure 5: BoN comparison of two models fitted independently with scaling data size in HH-RLHF dataset (left: CPM-GPT-3.5, middle: CPM--lan-TS, right: standard PM).
track two important metrics: model generalizability (described in Sec. 4.3 and win rate (described in Sec. 4.5) The training data size is fixed to 10K. As shown in Fig. 6 both model generalizability and win rate steadily improve with increasing LM size. This confirms that LM capability propagates to feature extraction, and that CPM can take advantage of it. This further means that CPMs can become even more useful as extractor LMs become more capable. The smooth and gradual increase of the win rate as function of LM size suggests that our findings generalize to the case of using even larger LMs for feature extraction.
2.35
Sizt
Mosel Sze
1.31
PV-Sore
0.35
-033
Numberef
Number Farmniters
Figure 6: Model size scaling experiment using Flan-T5. (left: comparison with the reference PM,
right: win rate over initial generation after BoN sampling based on each PM)
8

---

Published as a conference paper at ICLR 2024
6 RELATED WORK
Robustness of preference models PM overoptimization is an instance of reward hacking, a situ-
ation when policy exploits flaws in its reward function (Amodei et al. 2016 Skalse et al. 2022).
These flaws can come from errors of human evaluators (Pandey et al. (0022). the inherent difficulty
of learning preferences of irrational agents (Mindermann & Armstrong 2018 Shah al.
2019) or
the fragility of learned reward functions to adversarial attacks (McKinney et al. 2023). ). Gao et al.
(2023) studied the scaling properties of PM overoptimization and Casper al. (2023) discuss it in
broader context of open problems with RLHF. More generally, PMs can learn to be sensitive to spurious features associated with human feedback. This leads to failure modes such as sycophancy (a tendency to answer question with user's preferred answer, even if that answer is not correct; Cotra 2021 Perez et al. (2022) or social bias (due narrow demographics of feedback providers: [San-] turkar et al. 2023 Hartmann et al. 2023). . Despite its growing importance, the problem of learning robust PMs for aligning LMs is largely neglected. The present paper attempts to fill this gap.
Decomposing tasks for LMs. There are numerous examples of task decomposition increasing the accuracy or robustness of language models. Breaking down problems into steps (Wei et al. 2022 chain-of-thought:) into sequence of subproblems depending answers to previous subprob- lems (Zhou et al. 2023) are enormously beneficial for tasks involving reasoning. Others explored stronger separation: solving subproblems independently in different LM context windows. For instance. Creswell et al. (2022) alternate between selection and inference to generate a series of interpretable, casual reasoning steps. RRdhakrishhan al. (2023) found that solving subproblems in separate context windows improves faithfulness of reasoning. Reppert et al. (2023) build compo- sitional LM programs by applying decomposition iteratively, with a human in the loop, to facilitate science question answering. The present paper finds similar robustness benefits of decomposition for preference modeling.
Scalable oversight Scalable oversight is the problem of evaluating the behaviour of agents more capable than the evaluators (Bowman al. 2022b). On the one hand, LMs may soon grow capable of completing tasks for which humans will not be able to provide feedback. On the other, LMs might also be capable of reasoning about flaws in their evaluation procedures (Berglund et al., 2023 and exploiting them unbeknownst to overseers. Current proposals for solving scalable oversight focus on recursively relying on other LMS to assist human evaluators (Irving et al. 2018, Leike et al, 2018 [Chistiaano et al.) (018). RL from AI feedback (Bai et al.) (202bb) attempts to implement this idea by using carefully prompted LMs to generate training data for PMs. In contrast, we propose to rely on LMs during single inference step of a PM.
7 CONCLUSION
We introduce Compositional Preference Models (CPMs), simple and effective paradigm for train- ing robust and interpretable preference models. CPMs decompose global preference scores into interpretable features and rely language models (LMs) to extract those features. Despite their simplicity, CPMs are robust to different subsamplings of the dataset and to overoptimization, and they outperform conventional preference models at obtaining preferred best-of-n samples. We be- lieve that CPMs pave the way for combining human insights into preference judgements with the LM capabilities extract them. Given the recent advances in LM abilities, CPMs have the potential to being used for alignment and scalable oversight of models with superhuman capabilities. One limitation of our work is that instead of genuine human evaluation of the preferences, we use proxy LLM (Claude 2) for the evaluation. One research direction here could be to introduce task-oriented generation scenario (e.g. task accomplishment) where helpfulness could be evaluated easily and to understand how to inform the preference model with this scenario. Finally, another possible objective for future research would be to explore how to elicil decomposed features that can capture various kinds of complex preference judgements. A promising direction here would be to leverage LMs to not only score, but actually discover the component features that determine these judgements.
9

---

Published as a conference paper at ICLR 2024
REFERENCES
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Con- crete problems in Al safety, 2016. htt.l..pps///ax.v...r///////11. 06565 06565
Anthropic. Introducing Claude, 2023.
URL.
httpss//ww..ntthropicccco////
Introouuiigg-llaude:
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.0553 2022a.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Ols- son, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran- Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mer- cado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Con- erly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback. 2022b.
Lukas Berglund, Asa Cooper Stickland, Mikita Balesni, Max Kaufmann, Meg Tong, Tomasz Kor- bak, Daniel Kokotajlo, and Owain Evans. Taken out of context: On measuring situational aware- ness in 1lms, 2023.
Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamile Lukosuite, Amanda Askell, Andy Jones, Anna Chen, et al. Mcasuring progress on scalable over- sight for large language models. arXiv preprint arXiv:2211.035400 2022a.
Samuel Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilè Lukošiūté, Amanda Askell, Andy Jones, Anna Chen, et al. Measuring Progress Scalable Oversight for Large Language Models, November 2022b. /tRl//ttp////rxx.or///bbb arXiv:22 1.03540 [cs].
03540
Lars Buitinck, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, Andreas Mueller, Olivier Grisel, Vlad Niculae, Peter Prettenhofer, Alexandre Gramfort, Jaques Grobler, Robert Layton, Jake Van- derPlas, Arnaud Joly, Brian Holt, and Gaël Varoquaux, API design for machine learning software: experiences from the scikit-learn project. In ECML PKDD Workshop: Languages for Data Min- ing and Machine Learning, pp. 102-122, 2013.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaèl Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bryık, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. Open problems and funda- mental limitations of reinforeement learning from human feedback, 2023.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. URI. https:
//Imsyssorr//bllgg22223--00 30- vinva/aa/
Paul Christiano, Buck Shlegeris, and Dario Amodei. Supervising strong learners by amplifying weak experts, 2018
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- fems, 30, 2017.
10

---

Published as a conference paper at ICLR 2024
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu. Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery. Sharan Narang. Gaurav Mishra. Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling instruction- finetuned language models, 2022. URL htttt::////xii:i/ http::://arxii..rggbb//////2 wrgg/bs/////////o/ 11416
Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, and Clifford Stein. Introduction to algorithms. MIT press, 2022.
Ajeya Cotra.
Why ai alignment could be hard with modern deep learning. Blog post on Cold Takes, Sep 2021. URL https://www. http:/wwwwwwwwooooooooo.. takes. com/ Accessed [in-
why-ai hhwer:---oooo-----a-a-a-t-t-eeed
sert today's s date here].
Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inference: Exploiting large lan- guage models for interpretable logical reasoning, 2022.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5988-6008. PMLR, 17-23 Jul 2022.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Mari- beth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209. 14375, 2022.
Dongyoung Go, Tomasz Korbak, Germàn Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymet- man. Aligning language models with preferences through f-divergence minimization. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scar- lett (eds.), Proceedings of the 40th International Conference Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 11546-11583. PMLR, 23-29 Jul 2023. URL https://proceediing weepiinngss....eeeeee mtr..press/v22//goleee
html
Charles AE Goodhart. Problems of monetary management: the UK experience. Springer, 1984.
Jochen Hartmann, Jasper Schwenzow, and Maximilian Witte. The political ideology of conversa- tional ai: Converging evidence on chatgpt's pro-environmental, left-libertarian orientation. arXiv preprint arXhv:2301.01768. 2023.
Trevor Hastie, Robert Tibshirani, Jerome H Friedman, and Jerome H Friedman. The elements of statistical learning: data mining, inference, and prediction, volume Springer, 2009.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: secoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006 03654, 2020.
Xingwei He, Zhenghao Lin, Yeyun Gong, Alex Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, Weizhu Chen, al. Annolim: Making large language models be better crowd- sourced annotators. arXiv preprint MXXi:23333.1684. 2023.
Jacob Hilton and Leo Gao. Measuring goodhart's law, 2022. URL https://openai.com researcch/measuring-
Tom Hosking, Phil Blunsom, and Max Bartolo. Human feedback is not gold standard, 2023.
Geoffrey Irving, Paul Christiano, and Dario Amodei. Ai safety via debate, 2018. 88
11

---

Published as a conference paper at ICLR 2024
Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Vinayak Bhalerao, Christopher Buckley, Jason Phang. Samuel R. Bowman, and Ethan Perez. Pretraining language models with human prefer- ences. In Andreas Krause, Emma Brumskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learn- ing. volume 202 of Proceedings Machine Learning Research, pp. 17506-17536 PMLR, 23-29 Jul 2023, URL wttphhtss/ppocceeeiinnge
mlr. wtt.ppeee//2222korrbk22222 htmml
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: research direction, 2018.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu, Gpteval: Nlg evaluation using gpt-4 with better human aligmment. arXiv preprint arXiv:2303.16634 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 1.1.0100,, 2017.
Lev McKinney, Yawen Duan, David Krueger, and Adam Gleave. On the fragility of learned reward functions, 2023.
Soren Mindermann and Stuart Armstrong. Occam's razor insufficient to infer the preferences of irrational agents. In Proceedings of the 32nd International Conference on Neural Information Red Hook, NY, USA, 2018. Curran Associates
Processing Systems, NIPS'18 18, pp. Inc.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orea: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707. 2023.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint
2021.
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 17730-277447 2022.
Rahul Pandey, Hemant Purohit, Carlos Castillo, and Valerie L Shalin. Modeling and mitigating human annotation errors to design efficient stream processing systems with human in in-the-loop machine learning. International Journal of Htaman-Compunt Studies, 160:102772, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high perfomerfeee deep learning library. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Accé-Bucc, Emily B. Fox, and Roman Garnett (eds.), Proc. of NeurlPS, pp. 8024 8035, 2019. httptttpss//pcppppiisseeeeeeeeee neurips. http/ppper/2211///aaae F ott.://2000000010022227777000000
Ethan Perez, Sam Ringer, Kamilè Lukošiūté, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, al. Discovering language model behaviors with model-written evaluations, 2022.
Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Denison, Danny Hernandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilè Lukošiūté, et al. Question decomposition improves the faithfulness model- -enerrtee reasoning, 2023. URL https: hittps://arant //arxivv https://///xvvo.//abbb/// org/abs/ 2307.11768
12

---

Published as a conference paper at ICLR 2024
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Seirty-sevnnth Conference Neural Information Processing Systems, 2023. URL
https://openreviiee http://www.....eoooooooos/sssaa.
Justin Reppert, Ben Rachbach, Charlie George, Luke Stebbing Jungwon Byun, Maggie Appleton, and Andreas Stuhlmüller. Iterated decomposition: Improving science q&a by supervising reason- ing processes, 2023.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? arXiv preprint arXiv:2303. 17548, 2023.
Rohin Shah, Noah Gundotra, Pieter Abbeel, and Anca Dragan. On the feasibility of learning, rather than assuming, human biases for reward inference. In International Conference Machine Learning, pp. 5670-5679. PMLR, 2019.
Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bow- man, Newton Cheng, Esin Durmus, Zac Hatfield-Dodde Scott R. Johnston, Shauna Kravec, Tim- othy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy in language models, 2023.
Damien Silco. tasksource:
Structured dataset preprocessing annotations for frictionless extreme multi-task learning and evaluation. arXiv preprint arXiv:2301.0m 05544, 2023. URL https://
arxiv. arxiv.org/abs/2300 05948
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and character- izing reward gaming. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 9460-9471. Curran Associates, Inc., 2022. URL
nwwrips cc cc/papper_fiii fiaaper.filess/pper/2222. pdf
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://opeenreiiee http://pppmrrvvieee.efffrrrnnttt wwt.:ffmffmmm ViQQMMeBB
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proc. of EMNLP, pp. 38-45, Online, 2020. Association for Computa- tional Linguistics. doi: 10. 10.18653/1/2222..ee 10t.1/0000eemm/ddddmoo..... URL wwt.hhtsss//aaaaato.llloo .org///gyyogg/////o/oooo 2020.emnlp
emnlp- delll...eee66666 6
Denny Zhou, Nathanael Schärli. Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuur- mans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to- -ost prompting enables complex reasoning in large language models, 2023.
13

---

Published as a conference paper at ICLR 2024
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint ar arxi: (XX::199900899... 2019.
14

---

Published as a conference paper at ICLR 2024
A IMPLEMENTATION DETAILS
A.1 COMPOSITIONAL PREFERENCE MODEL
We used GPT-3.5 (gpt-3.5 3-pt--35-ttubbe 5-turro--330 and Flan-TS-XL (3B parameters) Chhng et al. 2022) as a feature extractor, using the same features and prompt templates in Tab. and Tab. 6 We excluded randommess from the generation process and selected the token with the highest likelihood.
For logistic regression classifier we used Scikit-learn (Buitinck et al. 2013). We set the choice of In and L2 regularization, weight of regularization, and solver of the logistic regression classifier as hyperparameters and selected hest hyperparameters based on 5-fold cross-validation in training dataset.
In the inference time, we made feature scores of the generated response using same LLM and tem- plates used in training phrase. The feature scores are aggregated with the trained logistic regression classifier as described in Sec. 3.2
A.2 STANDARD PREFERENCE MODEL
All standard PMs were implemented using PyTorch (Paszke et al., 2019) and HuggingFace Trans- formers Wolf
2020 We adopt the AdamW optimizer (Loshchilov & Hutter 2017) with 3 = (0.9, 0.98) and set the weight decay to 0.01. We conducted separate hyperparameter sweeps over learning rate and batch size for each dataset, using carly-stopping based on the evaluation set with 3 steps of patience. We used batch size of 32 and learning rate of 1e-5 for HH-RLHF dataset and 5e-5 for SHP dataset. We used cosine learning rate schedule with 100 linear warmup steps. We used Flan-T5-XL. (Chung et al., [2022] 3B parameters) for standard PMs, which is available on the Huggingface Model Hub under the model name of google/fle aeteefllannn-tlll Training was performed on Nvidia A100 GPU, with the longest run taking approximately 12 hours.
B CLAUDE EVALUATION OF THE REFERENCE PM
To evaluate the performance of reference PM in Sec in preference alignment, we follow the same quality evaluation framework as in Sec. 4.3 Specifically, we select the best sample among 16 responses generated by Flan-T5, based the reference PM score. We then compare this response to vanilla Flan-T5, a response randomly selected from the same set of candidates, as described in Sec. 4.5
Again, we use Claude to rate the quality of the response selected by reference PMs (see Tab. X for the prompt used evaluation). We perform 10 independent BoN trials with n 16 and report the average win rate. The results are shown in Tab. + The reference PM outperforms CPM-Flan-TT and standard PM. Interestingly, reference PM shows second best win rate in SHP dataset, highlighting the effectiveness in CPM-GPT-3.5 However, we note that this should be interpreted with caution as CPM-GPT-3.5 is not tested multiple experiments.
Win Rate
HH-RLHF
SHP
CPM-GPT-3.5
0.810
(-)
0.672 (.)
CPM-Flan- T5
0.742 (0.034)
0.580 (0.045)
Standard PM
0.588 (0.030)
0.564 (0.037)
reference PM
0.829 (0.050)
0.619 (0.060)
Table 4: Comparison win rate between CPM and reference PM externally fitted on multiple datasets. Reference PM shows high win rate in HH-RLHF dataset, and second best win rate in SHP dataset.
C
PROMPT USED
15

---

Published as a conference paper at ICLR 2024
You will be given a conversation between human and an AI assistant. Your job is to judge whether assistant's s reply @attribute_desc) Score that on a scale from to where means {attr_min) and 10 means (attr_max). Here's s the conversation:
Human:
(history)
Assistant:
(reply)
Answer by outputting a number from (and nothing else).
Answer:
Table 5: Prompt template used for HH-RLHF dataset to rate the response under given attribute.
You will be given Reddit post and reply. Your job is to judge whether the reply (attribute_desc), Score that on scale from to 10 where means (attr_min) and 10 means (attr_max).
POST:
{query)
Reply:
[reply)
Answer by outputting a number from (and nothing else).
Answer:
Table 6: Prompt template used for SHP dataset to rate the response under given attribute.
D ADDITIONAL MODEL INTERPRETABILITY
Feature distribution of policy Fig. shows how the feature values of the responses change in BoN(a, CPM, n) distribution with Flan-T5 Flan-T5-Large initial LM a, as a function of KL divergence KLBON. This simulates how the feature distribution changes as the initial policy is optimized with the current CPM and allows for inspecting which features drive reward maximization at differ- ent stages. The increase in readability is smaller than other features such as enough-detall and
This means that policy shifts towards generating responses with more detailed information, rather than focusing on generating readable responses. Additionally, af- ter a certain n, the slope of htl---t-cooiiie----ee at the end of optimization. This means that optimization pressure focuses not failing consider the context and avoiding unexpected lengthy responses. This decomposition makes it easier for human to understand why the LM generated such responses.
increases, and it achieves the highest score
E ADDITIONAL TABLES AND FIGURES
16

---

Published as a conference paper at ICLR 2024
Feature name
Attribute
Description
attribute_der desc altr_min allr max
is helpful for the original poster not helpful very helpful
helpfulness
attribute_dcse altr_min attribute_dese
is specific enough too vague very specific
specificity
understands the original poster's intent
intent
attr min
failure of understanding perfect understanding
attr max
attribute_dest _dec
is factually correct egregiously incorrect fully correct
factuality
attr_min
altr max
autribute_dcss altr_min
is casy to understand very difficult to understand very easy to understand
easy-to-understand
attr-mas
attribute_desc
is relevant to the original poster's question off-topic
relevance
attr_min
very relevant
attribute_dese attr_min
casy to read and too technical for original poster very difficult to read very casy to read
readability
attr_mis _ax
attribute desc
provides enough detail helpful too little detail
enough-detail
attr_min
attr_max
very detailed
attrihute desc attr_min
is biased or one-sided
biased
very biased not biased at all
attr-man max
attribute_ded deribuutedescc attr_min
fails to consider the original poster's 's cultural or individual preferences fails to consider the original poster's preferenees takes into account the original poster's preferences
fail fill-to-con to-consider-
individual preferences
attr_max
attribute desc
is repetitive very repetitive repetitive
repetetive
attr min attr_max
attrihute desc altr_min attr_max
fails to consider original poster's 's context fails to consider the original poster's 's context takes into account the original poster's context
fail-to-comsider-comeeet
autributedessc
is too long too long not too long
too-long
altr altr_min
attr max
Table 7: Features and descriptions used Tab, and
17

---

Published as a conference paper at ICLR 2024
Human:
You are a helpful and precise assistant for checking the quality of the answer. We would like to request your feedback on the performance of two AI assistants in response to the user question displayed below.
[Question]
{query)
[The Start of Assistant I's Answer]
{reply1}
[The Start of Assistant 2's Answer] (reply2}
Please rate the helpfulness, relevance, accuracy, level of details of their responses.
Each assistant receives an overall score on a scale of to 10, where higher score indicates better overall performance.
Please first output single 1ine containing only two values indicating the scores for Assistant and 2, respectively. The two scores are separated by a space. In the subsequent line, please provide a comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment.
Assistant:
Table 8: Prompt template to rate the writing quality of the candidate assistant model.
Feature Name
0.8
Feature Name
helpuirrsss
helpfuiness
readabiiity
readabilty
unough detail
enough detail
0.5
factuality
factuality
Festure Scone
Feature Score
0.4
0.3
0.0
0.0
11. distance between best-of-n policy initial policy
KL distance between best-afen policy Initial policy
Figure 7: Feature distribution of BoN experiment (left: CPM-GPT-3.5, right: CPM-Flan-T5). Note that the r-axes are different. Here the KL distance of the BoN distribution from the initial distribu- tion a(x) is computed as KLB log n-1 Nakano et 2021).
HH-RLHF
SHP
CPM-GPT-3.5
0.997
0.981
CPM-Flan-T5
0.926
0.928
Standard PM
0.665
0.057
Table 9: Rank correlation between gold PM scores and proxy PM scores in BoN experiment. For each PM technique used fit the proxy PM, we calculate and average PM scores over samples from BoN(a, PM Mprxxy, r1), and compute the rank correlation between the averaged gold and proxy PM scores over different n.
18

---

Published as a conference paper at ICLR 2024
aa HH-RLHF dataset
Alried
PML
L2
1.3
2.8
3.8
02
0.
33
0.6
Numberof
Mumber of in BrM
respenees
(b) SHP dataset
PH
194
12
2.6
1.0
0.4
32
02
48
254
Figure 8: BoN comparison over two models fitted independently in same condition (left: CPM- GPT-3.5, middle: CPM-Flan-TS, right: standard PM) The PM A with blue line indicates the PM used for selection in BoN
42
41
Figure 9: Feature distribution of BoN experiment (left: CPM-GPT-3.5, right: CPM-Flan-T55.. Note that the z-axes different. Here the KL distance of the BoN distribution from the initial distribu- tion a(x) is computed as KL log n-1 Nakano et al. 2021
19

---

Published as a conference paper at ICLR 2024
CPM-GPT-3.5
CPM-Flan-TS
Feature
Coefficient
Feature
Coefficient
helpfulness
0.246
fail-to-consider-oontett
0.420
enough-detail
0.235
enough-detail
0.244
factuality
0.187
factuality
0.227
readability
0.184
biased
0.178
token length
0.101
uusy-to-understand
0.124
specificity
0.094
specificity
0.106
biased
0.086
too-long
0.081
relevance
0.071
token lengh
0.075
casy-to-understand
0.069
helpfulness
0.037
mallto-consider-coneeee
0.043
intent
0.024
too-long
0.016
repetetive
0.015
fail-to-consider-
repetetive
0.014
-0,042
individual-preference
intent
-0.008
relevance
-0.056
fail-to-consider-
-0.056
readability
-0.120
individual-preferennes
Table 10: Fitted coefficient of CPM on HH-RLHF dataset.
Features
fail-to- consiier- context
enough-detail
helpfulness
readability
factuality
Dialogue
Q: why ramon laguarta bad ceo?
RI: a bad businessman
-0.890
-1.041
-0.569
-0.851
-0.239
R2: Manager
-0.890
-1.682
-1.063
-0.851
-0.710
Q: What a blizzard warning?
R1: Blizzard warnings issued the National
Weather Service to warn against snow, ice or strong winds.
1.909
0.880
2.395
1.690
3.059
R2: Blizzards can cause extremely dangerous driving conditions, with heavy snow, ice sleet.
1,559
0.88
1.901
1.408
2.117
Q: climate change raising temperatures above their normal levels?
RI: Climate change increases global temperature. R2: Earth is expected to experience warmer climate by the year 2050, the hottest on record, based on the International Solar System
1.559
0.559
1.901
1.690
-0.239
0.509
-0.721
0.419
0.278
-0.239
Q: I'd like to evaluate some art perhaps. Can you tell some history about Mona Lisa, the painting? R1: The Mona Lisa a I century Italian portrait
-0.89
-1.041
-0.569
-0.851
-0.710
Gian Lorenzo Bernini. R2: In 1853, the artist Leonardo da Vinci created the portrait of the Madre della Signore.
-0.540
-1.041
-0.075
-0.569
-0.239
Examples of feature values of CPM-GPT-3.5. Each feature value can provide information about which aspects of the response are good or bad.
Table 11:
20

---

Published as a conference paper at ICLR 2024
F ADDITIONAL RESULTS RELATED TO QUESTIONS FROM REVIEWERS
F.1 CORRELATIONS BETWEEN FEATURES
To investigate the diversity of features, we compute correlations between features in HH-RLHF dataset. The Figure 10 shows that the features have mostly positive correlations, as some of them addressing similar dimensions.
Feature correlation
helpfuiness
0.65
0.79
0.66
0.47
0.77
0.18
0.73
0.43
0.59
0.24
0.60
-0.17
specificity
0.65
0.59
0.65
0.59
0.66
0.26
0.81
0.45
0.47
0.45
0.60
0.04
0.6
factuality
0.66
0.65
0.70
0.62
0.71
0.33
0.65
0.58
0.61
0.45
0.69
0.06
casy understand
0.47
0.59
0.56
0.62
0.60
0.72
0.49
0.59
0.57
0.58
0.66
0.37
0.4
readabiitty
0.18
0.26
0.29
0.33
0.72
0.33
0.15
0.44
0.40
0.44
0.42
0.52
enough detail
0.73
0.81
0.64
0.65
0.49
0.67
0.15
0.42
0.49
0.34
0.58
.0.13
blased
0..
Individual- esefrrnccs
fall-to-consider-
0.59
0.47
0.68
0.61
0.57
0.66
0.40
0.49
0.61
0.41
0.74
0.08
repetetive
0:24
0.45
0.33
0.45
0.58
0.35
0.44
0.34
0.53
0.41
0.51
0.44
0.0
fallso-consider-contett
too-long
-0.17
0.04
-0.06
0.06
0.37
-0.01
0.52
-0.13
0.19
0.08
0.44
0.13
helpfuiness
specifictt
intent
factuaity
easydo--nderstandd
relevaniee
readability
biased
fall-to-consider erences
repetelive
Figure 10: Full matrix of feature correlations.
F.2 FEATURE SCALING TREND
To investigate the effect of the number of features, we gradually increase and check the win-rate of CPM-Flan-T5 with & features. For this, we order the features based on their importance in Ta- II, and then assess how the performance of the CPM - measured terms of 'win-rate' quality in Section 4.5 varies with & when we keep only the first & most important features. Note that regardless its coefficient rank, we put 'helpfulness' first in the ordered list, so that we can compare the case of "prompted PM with one holistic feature" and "compositional PM with k features".
The ordered feature list is: helpfulness, factuality, length, biased, easy-to-understann repetitive tive, fail-to-cons deneiiderinniii vidiii -praaalpppreeee -preferences,,...e relevance, readability. The win-rate averaged for trials described in Table 12
Fail-to-consider cows-ier-coooeeetettteeeee specificity, too-long, intent,
enough-detail,
The table suggests that the single holistic feature 'helpfulness' obtains reasonable win-rate (0.707) on its own but falls short of using the combination of all features (0.742). This suggests that
One reviewer made the interesting observation that win-rate of the prompted PM with one holistic feature "helpfulness" still comes out ahead that of standard PM (Table We hypothesize that the superior performance here of the holistic PM the standard PM is due the fact that preference dataset may not be large
21

---

Published as a conference paper at ICLR 2024
decomposing the features can have additional benefit for capturing the preference. Second, Table 12 shows that the performance CPM with k: = i4 worse than that CPM with k = 6 (0.754). This might be related to the overlap between features. However, the performance gap between A 14 and l = 6 small, as we employ a regularization term when fitting the logistic classifier.
Number of features k
Win Rate
k 1
0.707 (0.030)
k = 3
0.715 (0.024)
: 6
0.754 (0.038)
k 10
0.735 (0.037)
b = 14
0.742 (0.034)
Table 12:
Win rate of CPM-Flan-TT TS over initial generation after BoN sampling based on each PM with different number of features. We independently conduct 10 rounds of BoN(n 16) samplings and report the average win rate along with standard error.
F.3 EVALUATION WITH PARAPHRASED PROMPTS
To further investigate the impact various prompts and the robustness of the CPM's performance on prompts, employed GPT-3.5 to paraphrase each of the original descriptions in Table resulting
in Table
We evaluated the CPM's performance based this second table, using the 'win-rate' quality metric described in Section 4.5 The average win rate CPM-Flan-TS across five independent trials was 0.717 with standard error of 0.023, which is not statistically different from the original performance in Table 1 74.742 with a standard error of 0.034). This indicates that the CPM's performance shows some robustness relative to the specific prompt used.
enough for the standard PM to achieve robust performance, while the prompted PM utilizes the capabilities of generic I.M,, trained huge dataset.
22

---

Published as a conference paper at ICLR 2024
Feature name
Attribute
Description
attribute_dese attr_min
provides valuable assistance the original poster no assistance
helpfulness
altr-max
excellent assistance
attribute_dess
detailed and precise
specificity
attr_min attr max
overly vague
highly specific
attrihute_dest attr_min
accurately grasps original poster's intent
intent
misinterprets the original poster's intent
attr max
perfectly understands the original poster's intent
attribute_dese
is based on accurate and verifiable information blatantly incorrect entirely accurate
factuality
attr_min
altr-max
attribute_den desc attr_min
is clear and straightforward extremely difficult to understand exceptionally easy to understand
casy-to-understand
attribute_dese
directs addresses the original poster's query entirely irrelevant
relevance
attr_min
attr max
highly relevant
attribute _eecc attr_min
written in style appropriate for the original poster's level understanding extremely difficult read
readability
attr_max
exceptionally easy to read
attribute_den desc
provides a sufficient level of detail to be helpful insufficient detail
enough-detail
attr min attr_max
comprehensive level of detail
attribute desc
presents an objective and impartial perspective strong bias or one-sidedness completely unbiased
biased
attr_min
attr-max
attribute desc
fails consider the original poster's cultural individual preferences fails to consider the original poster's 's preferences carefully considers the original poster's preferences
fail to-consider-
attr altr_min
individual mndividual-peffernnees
max
attribute desc
avoids unnecessary repetition excessively repetitive not repetitive
repetetive
attr min
attr_max
attribute desc min
fails to consider the original poster's 's situation and background fails consider the original poster's context appropriately considers the original poster's context
fail-to-comsider-conteett
attr_mar max
attribute.dee desc
is concise and avoids unnecessary length excessively long appropriately concisc
too-long
altr_min
attr_mat
Table 13: Paraphrased features augmented from the original descriptions in Table used with the template in Table 5
Those features
23


Reviews:
[{"review_id": "noJCULf2IK", "reviewer": "Reviewer_Q5ds", "summary": "The paper presents a compositional preference model for LLM alignment. In contrast to standard monolithic preference models that assign a single scalar value to preference judgments. the model uses a number of features associated with individual scalar values (assigned by an LLM as an automatic evaluator) that are then linearly combined into an overall score.  The authors argue that this provides an inductive bias to the model that makes it more robust to overfitting and reward hacking and results in better generalization and human interpretability. The technique is evaluated with respect to consistency of responses for models trained on different subsets of the training data), comparison against reference PMs from the literature, robustness to overoptimization, and alignment of LLMs trained with the proposed model as opposed to a standard PM.", "questions": "1. It looks like the feature evaluator LLMs (Flat-T5 and GPT3.5) were used out of the box with prompting for assigning feature scores, without fine-tuning or in-context learning. I would have like to see the comparison against fine-tuned versions for each feature. \n2. How does the robustness of the CPM change with an increasingly larger list of features?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The core idea of the paper is simple yet powerful and addresses known weaknesses in traditional monolithic preference models; it should be of broad interest to the ICLR audience. The presentation is clear and -- with the exception of human alignment evaluation (see below) -- the evaluations are convincing.", "weaknesses": "For alignment with human preferences, another  LLM (Claude-2) was used rather than genuine human ratings. Although there is more effort associated with a human evaluation study,  and the literature you cite has shown some (imperfect) degree of correlation between human ratings and LLM scores, I really consider human evaluation a must here - otherwise, you are measuring alignment between different LLMs, which can simply result from similar training procedures or similar preference models used.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J32GiOgah8", "reviewer": "Reviewer_zwab", "summary": "The paper introduces compositional preference models (CPMs), a new method for aligning language models (LMs) with human preferences. CPMs break down preference scores into multiple features to improve robustness and generalization. This decomposition is accomplished by prompting an LM to assign a value to the answer based on a specific preference type. Experimental findings demonstrate that CPMs effectively mitigate overoptimization in preference modeling.", "questions": "1. How is the encoder for $x$ parameterized in logistic regression?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Modeling human preferences from different types of judgments is a promising research topic.\n2. Experimental results demonstrate that the suggested CPMs indeed improve both robustness and generation.\n3. The paper is generally easy to read.", "weaknesses": "1. Although CPMs offer a practical method for breaking down preferences by stimulating LMs, I consider it too simplistic and unrealistic to capture intricate human preferences. For instance, easy-to-understand answers and answers with enough details may contradict each other. I have reservations about whether logistic regressors can accurately represent this intricate pattern.\n2. In terms of the experimental setup, CPMs prompt much more than standard PM, which raises concerns about their comparability. I recommend that the author include another standard PM baseline that uses a similar prompt budget as CPMs. For instance, prompting the LM $n$ times (where $n$ represents the number of pre-defined preference types for CPMs) through sampling and selecting the final preference score via majority voting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sURqzy2O95", "reviewer": "Reviewer_pUSN", "summary": "This paper proposes Compositional Perference Models (CPMs), a new perferene model framework that decomposes one global perference assessment into several interpretable features, using a prompted LM to score these features, and finally aggregates these features together with their scores using a logistic regression classifier. Experiments show that CPMs improves generalization and robustness than standard PMs. The main contributions include: (1) new CPMs that allows more transparent supervision; and (2) better results at dimensions of model/overoptimization robustness, generalization, and perference alignment.", "questions": "1. in Table 5 and Table 6, scores from 1 to 10 are used, and did you try other ranges such as 1 to 5, and how did you decide to use a range of 1 to 10? Also, does different features require different scopes/ranges of scores? In addition, when changing from numbers to words (bad, good, better, best...), how shall the results change?\n2. any comparison of between supervised fine-tuning (SFT) and PEFT when using the CPMs? Or, any comparison of the usage of resources under different model alignment frameworks? So, (1) inference only stage controlling, (2) sft, (3) peft, any investigations on these directions of using CPMs?\n3. page 3, there are 13 features used, any detailed analysis of overlapping or diversities among these features?or when applying your method to other languages/tasks, how shall we reuse these features or how shall we design new features (any common rules?)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) new CPMs that allows more transparent supervision; \n(2) better results at dimensions of model/overoptimization robustness, generalization, and perference alignment.", "weaknesses": "1. prefer to see detailed investigations of applying CPMs to different stages of (1) inference only (2) sft, and (3) peft.\n2. not quite clear of the scalability of the usage of current 13 features to novel langauges/tasks, further investigations are preferred.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zD3fXvJx6X", "reviewer": "Reviewer_dDjG", "summary": "This paper introduces ‘compositional preference models’ (CPMs) that re meant to overcome issues of transparency and scalability found in regular preference models. CPMs decompose preferences into interpretable features, and subsequently aggregates them (with LR). Generally, results show improvement in generalization and avoidance of over-optimization, which are not themselves ’transparency and scalability’.", "questions": "- The definition of ‘model robustness’ in Sec 4.2 seems incomplete — surely a factor is the domain or scenario in which the model is to be deployed or evaluated, too?\n- Would it be possible to re-arrange the columns of Fig2a and Fig2b so Standard comes left-most (first)?\\\n- Would there be value in actually performing human evaluations, despite the findings of best-of-n sampling in related work?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Although a minor strength, the collection of papers in sections 1, 2, and 6 are sufficient, relatively recent, and relevant.\n- The compositional extension, though simple, is technically a novel contribution that appears to provide several benefits.\n- The setup for best-of-$n$ sampling seems fair. The full set of 25.6K responses, and code, would be appreciated.\n- the use of a separate LLM for evaluation is appreciated.", "weaknesses": "- Although somewhat minor, the use of logistic regression will naturally cause some confusion, especially to those who want an end-to-end trainable model for this task. Other models should have been attempted.\n- Section 3.1 should be more informative as to the nature of features _c_ and how their set is identified, selected, or defined. This should include both the specific list in Sec 4.1 as well as guidance for other tasks, in general.\n- Although another minor weakness, at least one other LLM  should have been used for extraction (e.g., one of the Llamas)\n- Very minor, but please correct the various issues with your references including capitalization and completeness (e.g., Amodei suffers from both — use brackets around {AI} and provide full paper details)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)