Axiomatic Preference Modeling for Longform Question Answering
Authors: Corby Rosset
Guoqing Zheng
Victor Dibia
Ahmed Hassan Awadallah
Paul N. Bennett
Venue: EMNLP 2023 Main (2023)
Keywords: reward modeling
preference modeling
RLHF
Large Language Models
long form question answering
TL;DR: This paper presents an axiomatic framework for generating preference pairs and training a standalone preference model to improve alignment when choosing among candidate answers from a large language model

Abstract: The remarkable abilities of large language models (LLMs) like ChatGPT and GPT-4 partially stem from the post-training processes involving human preferences encoded within a reward model as part of a Reinforcement Learning from Human Feedback (RLHF) regimen. 
These reward models (RMs) often lack direct knowledge of why, or under what principles, the preferences annotations were made. In this study, we identify principles that guide RMs to better align with human preferences, and then develop an axiomatic framework to generate a rich variety of preference signals to uphold them. We use these axiomatic signals to train a model for the scoring answers to longform questions. Our approach yields a \textbf{Preference Model} with only about 220M parameters that agrees with gold human-annotated preference labels more often than GPT-4.
The contributions of this work include: training a standalone preference model that can score human- and LLM-generated answers on the same scale; developing an axiomatic framework for generating training data pairs tailored to certain principles; and showing that a small amount of axiomatic signals can help small models outperform GPT-4 in preference scoring. We intend to release our axiomatic data and model.

Axiomatic Preference Modeling for Longform Question Answering
Corby Rosset, Guoqing Zheng, Victor Dibia, Ahmed Awadallah, Paul Bennett Microsoft Research corbyrosset, zheng, victordibia hassanam, pauben@microsoff..co com
Abstract
Voted Human Answers (5764) ChatGPT Arswers (5564)
The remarkable abilities of large language mod- els (LLMs) like GPT-4 partially stem from post- training processes like Reinforcement Learning from Human Feedback (RLHF) involving hu- man preferences encoded in a reward model. However, these reward models (RMs) often lack direct knowledge of why, or under what principles, the preferences annotations were made. In this study, identify principles that guide RMs to better align with human preferences, and then develop an axiomatic framework to generate rich variety of pref- erence signals to uphold them. We use these axiomatic signals train a model for scoring answers to longform questions. Our approach yields Preference Model with only about 220M parameters that agrees with gold human- annotated preference labels more often than GPT-4. The contributions of this work include: training standalone preference model that score human- and ALM--generate answers on the same scale; developing an axiomatic frame- work for generating training data pairs tailored to certain principles; and showing that a small amount of axiomatic signals can help small models outperform GPT-4 in preference scor- ing. We intend to release our model.
-2
-2
Naive Preference Model Score
Axiomatic Preference Model Score
Related Answers (10000) Top Upupted Answers (5764)
-2
0
2
-2
Naive Preference Model Score
Axiomatic Preference Model Score
Figure naive preference model trained on upvotes alone is not aligned e.g., ChatGPT answers that rated highly by humans are given low scores. An axiomatic preference model addresses this and other gaps.
2023). RLHF in particular involves the construc- tion of "reward model" (RM) which is trained to encode these human preferences and output scalar score for any given response (Christiano et al., 2023; Stiennon et al., 2022; Beeching et al., 2023; Ouyang et al., 2022). Primarily, RMsedaeed approach to training LLMs separates what to learn from how to learn it (Leike et al., 2018).
1 Introduction
Recent advances in large language models (LLMs) has seen the introduction of diverse post-training strategies, including Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning from AT Feedback (RLAIF). These tech- niques have helped bridge the "alignment gap" be- tween the responses of raw pretrained language models and responses that resonate more closely with human preferences (Bai et al., 2022b; Ziegler al., 2020). These techniques steer LLMs to prefer one response over another based on feedback sig- nals from either human annotators, or from another LLM instructed to follow certain principles (Bah- danau et al., 2019; Kwon et al., 2023; Sun et al.,
The problem with most RMs used in RLHF post- training is that they are taught to regress a single scalar preference score annotated by humans with- out clear knowledge of why they made that decision or what principles they operated under. We term models trained in this fashion as naive preference- models. Furthermore, the underlying preference pairs used to train the RM do not come from di- verse sources, often being sampled from the same

---

Deta mined from generated by an LLM
Axiomatic Preference Model
Poshiive negalive pairs each axiom Is used train preference model
LLM
Train Dats Mechanism
Community QA Forum Upvotes
Answers Related Questions
External Evidence
Corrupt Facts
Combined Answers
C0A
COA
LLM
LUM
LLM
Axiom
Axiom
Axiom 2,4
Axiom 3
Axiom 5
Axioms
Usefulness
Relevance
Groundedness
Truthfulness
Thoroughneas
Figure 2: We propose five principled axioms to construct rich contrastive signals for training preference models
LLM they are trained on (Bai et al., 2022a; Nakano et al., 2022; Ouyang et al., 2022). It is also not clear that RMs can reliably score human- -writen and LLM- -eMeereeeteed responses on the same scale, which is more challenging than previously antici- pated due vast differences such as style, as shown Figure 1. Without clear signals of which prin- ciple informs the preference decision, and diverse sources of training examples upholding it, RM may not be aligned with the expectations of human stakeholders.
Prior work used axioms to diagnose and cor- rect failure modes in information retrieval sys- tems (Fang al., 2004, 2011; Rosset et al., 2019). Similarly, our axioms target known failure modes of modern LLMs, such as hallucinating incorrect statements that appear factual (Ji et al., 2023) being distracted by irrelevant context (Shi et al., 2023). The axioms also enforce new capabilities, such as incorporating evidence, or addressing mul- tiple perspectives. We believe our axiomatic frame- work provides richer, more targeted underlying preference pairs than, say, sampling from the same LLM with different temperatures.
For instance, studies have shown that RLHF- finetuned LLMS may fall short key expectations -c.g. by failing to support claims with evidence, or making claims that sound convincing but are untrue showing that there are still prevalent gaps in aligmment for these scenarios (Liu et al., 2023; Zheng et al., 2023b; Menick et al., 2022).
Moreover, the RMs in existing studies often not released nor the subject of close study com- pared the LLMs they post-train. They can be quite costly, sometimes holding as many parame- ters as the LLMs they train. While there are many studies on RMs to address safety and toxicity is- sues (Bai et al., 2022a; Ganguli et al., 2022; Faal et al., 2022; Korbak et al., 2023; Ganguli et al., 2023), there fewer on longform question swering (Nakano al., 2022: Glaese al., 2022).
In this work, we define principles (axioms) that humans desire in longform answers around the con- cepts of usefulness, relevance, grounded-nest truthfulness, and thoroughness similar to (Thop- pilan et al., 2022). A distinguishing feature of our study is that we then use these principles to construct candidate answer pairs "axiomatically" such that one answer is clearly preferred along certain principle. Some of these axiomatic pairs are constructed from abundant sources of weak hu- man preferences in the form "upvotes" from Community-basse Question Answering (CQA) sites like StackExchange In Figure 2 we illustrate how axiomatic pairs are generated for a single ques- tion. We define the principles in Appendix A, and describe how the axioms uphold those principles in Section 2.
Our approach is driven by the intuition that the act of identifying failure modes or verifying an answer free of them - is cognitively simpler (requiring fewer parameters) than the act of gen- crating an answer, especially in the presence of authoritative evidence from search engine. A sep- arate, smaller RM also has many advantages: it is controllable whitebox whose behavior is steer- able, quantifiable, and decoupled from the LLMs supervises; allows for generalization to unseen examples without having to annotate them; and it is cheaper to run at scale.
The purpose of this study is to evaluate how well

---

Principle
Axiom Description
Pair Construction
If A' > upvotes than A"
0. Usefulness
Upvotes from CQA forums
PM A') P.M (Q, A")
A == Any Answer to Q
Relevance
Answer, A, to Q should he more relevant than
= Answer to Q'
answer to related question Q', Q' E kQn(())
PM (Q, A) > PMM (Q, B)
== LLM(() "closed book"
2. Grounded-ness
LLM Answer with context of relevant passages
D ; LLM Pt Q) "open book"
pt is better than without
(M, D) > PM (Q. C)
E
3. Truthfulness
LLM corrupts relevant answer D yielding wwrong-hut-beleewable answer"
P. C) > PM E)
P. D) PM (Q.E)
M QQ)
4. Relevant
Irrelevant
LLM answer with w/ relevant context p+ is bet- ter than w/ irrelevant context P
PM D) > PM FQ,
Grounding
G=I.IM-Combine(( A' A") (M PM (Q.A) Ag (")
5. Thoroughness
Use LLM to combine top two user-upvoted answers, A' and A
Table I: Definitions of the axioms and how to construct training pairs from them based on our principles.
proposed axiomatic RMs agree with human preferences. Hence, we refer to our model as Preference Model, M going forward. Note, us- ing our preference models for LLM training (e.g. with RLHF) is outside of the scope of this pa- per. In Section 4 we demonstrate the capabilities of the our P.M in several scenarios that require re-ranking candidate longform answers, including those written by humans and by LLMs.
Given question q and set of candidate answers 01, 02 -... Ons the goal of the preference model is to find a partial ordering of the answers by training on pairs that best align with real human preferences (Chen et al., 2013; Carterette et al., 2008). Existing neural architectures such as Trans- former (Vaswani et al., 2017) are adept at solv- ing learning-to-ras toriin problems (Nogueira and Cho, 2020), and do even better under contrastive learn- ing regimes (Xiong et al., 2020).
The contributions of our work are threefold:
1. We develop an axiomatic framework to gen- erate/augment training pairs that capture nu- ances in human preferences which may not be present in the existing data. These axioms can be tailored to enforce any well defined princi- ple, meaning this framework is not limited to longform question answering.
A preference model PM takes as input ques- tion q. answer a, and outputs a scalar PM (q, a) c R a.k.a "preference score"; it has an optional in- put reserved for evidence passages e denoted P.M (g, e, a). We instantiate P.M as transformer- based cross-encoder (Wolf et al., 2019), f. whose input is linearized sequence of tokens x con- structed from the concatenation of q and a. denoted q q@e a. The output scalar is obtained from lin- ear regressor layer on the final transformer layer's CLS token. We further construct contrastive pairs of sequences such that the answer in one sequence x = ① at is preferred over a negative answer to the same question x = q C a At training time, the sequences are fed into f separately with the objective to score the positive example higher: [(x > f(x=). We choose the margin loss accomplish this goal:
2. We train standalone preference models P.M (220M - 7B parameters) that can score both human- and I.LM-generteed answers on the same scale, normalizing out spurious signals such length and style; our PM is better than training on human upvotes alone.
3. We show that training on the proper axiomatic signals boosts how well our PM agrees with both weak human upvotes and gold human an- notators, even exceeding the capabilities of GPT-4 - implying that GPT-4 may be overkill for preference scoring.
L = max A- 1f(x+)
(1)
2 Axiomatic Preference Modeling
where the margin, X. between the positive and negative sequence in a pair can be fixed or com- puted. Importantly, while traditional Icarning-to to- rank finds orderings based on relevance, we argue
Learning a preference model for longform ques- tion answering can be formulated as a learning- to-rank problem (Cooper et al., 1992; Liu, 2009).

---

that modern LLMs must go beyond that, which is why we introduce an expanded set of axioms including usefulness, thoroughness and grounded- ness.
tionally to their respective upvotes. By sampling negatives proportionally to their upvotes, we are able to specifically control for spurious signals such as length, style, presence of URLs, etc and force the model to inspect how the answer content interacts with the question.
2.1 Human Preference Signals
Learning to rank problems traditionally require large set of candidates re-rank. However, long- form answers are difficult acquire. We turn to CQA forums such as Reddit and Stack Exchange specifically because questions there can receive multiple answers among which users can specify their preferences via "upvote" or "downvote" sig- nals. Here we define axioms that produce training pairs either directly from CQA answers, or indi- rectly using LLMs; list these in Table 1.
2.2 LLM-generated Preference Signals
Axioms and I leveraged upvotes to construct pref- erence pairs from numan-writter answers. Here, we construct additional pairs generated by an LLM under various scenarios.
Axiom (Groundedness) The Groundedness principle gives rise to a preference for an answer at that incorporates and cites relevant evidence over one without access to such evidence, a. Hence neg- atives for question g come from an LLM (in our case, ChatGPT) in a closed-bookk" style prompted with guidelines that mirror Sopen-book, at is generated from ChatGPT in- structed to appropriately use evidence passages, e, placed in its context window, which were retrieved from the Bing API called with as the query. The prompt for this is shown in Figure 7 and examples in Figure 8.
Axiom (Usefulness) Critically, having multi- ple answers allows us construct preference pairs. We treat answers which have relatively higher up- votes as being more useful or helpful. From the set of answers for a question q, we construct positive at and negative a training pairs such that at has more upvotes than does.
principles. The
Upvote signals are known to be noisy since users may upvote answers for various reasons, and may be influenced by position and presentation bi- ases (Lee et al., 2016a). Answers can also gain popularity in "rich get richer" fashion that may deviate from the intrinsic qualities of the answer itself (Lee et al., 2016b). However, upvotes gen- erally aligns with our definition of usefulness (Fu and Oh, 2019).
Axiom 3 (Truthfulness) To combat hallucina- tion of incorrect statements, we generate answers which intentionally corrupt factual claims in ways that are still believable. To do this, we take an open- book answer from Axiom 2 and instruct an LLM to deconstruct it into bullet-point claims, corrupt those claims individually, and then re-stitch the corrupted claims into fluent answer, as shown in Figure 9; examples in Figures 10, 11. We found that open- book answers contain more factual statements, and hence have more to corrupt. We also found this prompting technique is the best way to automat- ically generate answers that are provably wrong without human annotation, otherwise, instruction- tuned LLMs would resist efforts to output false in- formation. This corrupted answer should be worse than both open-book and closed- book answer.
Axiom (Relevance) Answers in response to a question on CQA site are more or less relevant, hence model trained only Axiom 0 would not have seen examples of off-topic answers. We imbue the training regimen with additional "hard negative" answers mined from related questions. We construct an KNN index of the ANCE embed- dings for all questions in the Stack Exchange data dump (Xiong et al., 2020). For each question q. we retrieve k nearest neighbor questions { (and all their constituent answers) from the same corpus such that the dot product of their vectors is below a chosen threshold q- of Lg to indicate of is related to g while not being a paraphrase. This threshold t, is found manually. At training time, we randomly select n negative answers across the union of answers to all k related questions propor-
Axiom 4 (Relevant vs. Irrelevant Grounding)
The sibling of Axiom 2, Axiom 4 targets the qual- ity of grounding evidence because studies have shown that distracting context can be challenging for LLMs in longform QA scenarios (Krishna al., 2021). Axiom 4 exploits relevance signals from a retrieval system to discern low quality passages e from highly relevant ones et. To generate negative answers, we instruct an LLM to answer q using
'helpfulness is part of the official answering guidelines of these CQA forums

---

only information stated in eT and no other inter- nal or external knowledge; see prompt in Figure 12. The positive at, on the other hand, is gener- ated with access to et in the same way as those in Axiom 2. We also construct additional training pairs among the evidence passages themselves to distill relevance signals directly into the PM as discussed in Appendix C.3.
emphasized "helpfulness" and "honesty" which we operationalized into training pairs for Usefulness (Axiom 0) and Truthfulness (Axiom 3). Sparrow has a "stay on topic" Rule (Glaese et al., 2022) which we adapted as Relevance.
Secondly our Axiomatic framework is flexible enough incorporate "Al feedback" from a much larger "teacher" model like GPT-4 by having it label/rank which axiomatic answer prefers. How- ever, we can go one step further and ask the teacher not only which it prefers, but by how much by scor- ing the answers. These fine-grained preference scores can learned by the PMM via the À term in Equation 1, which governs the magnitude of sep- aration between answers. Since answers we gen- erate from LLMs lack upvote signals (and hence by default have a constant A), this approach unifies learning from human- and Al-preference signals.
While Axiom 2 used the Bing API for evidence, wc need more fine-grained control of the retrieval scores to ensure eT is worse than et. We achieve this with the MS Marco dataset, which also has supervised relevance labels, by building nearest neighbor index of the ANCE embeddings for all the documents (Xiong et al., 2020). For each q in the MS MARCO training set, et is collected from the top-k documents plus those with supervised relevance label of one; while e are documents below a relevance threshold doee- The sets e and e do not overlap.
Experimental Methods
Axiom 5 (Thoroughness) The preference model should favor answers that better address the full scope of the question and all important perspec- tives. While this task is difficult to define, a sim- ple yet effective approach is to assume that two high quality answers a' and a" to q come from two different authors, then their combina- tion should be more thorough than either alone. We generate the positive at from LLM in- structed combine "the best of both worlds". at LLLM-Combine(g..c. a', a"). For training, a" are answers known to be worse than both a' and a", i.. they have fewer upvotes. The prompt is shown in Figure 13 and examples in Figure 14. In practice, we select a' and a" to be the top two highest-upvoted answers on Stack Exchange, not- ing through extensive manual observations that users seldom upvote answers with duplicate content very highly. We post-process this data to remove pairs where resembles naive concatena- tions its two constituents. For evaluation, we track at vs a' and a" as in Table 3.
Implementation Details For all our experiments, the preference model is initialized from a T5Flan (Chung et al., 2022) base model. We train each model on a different combination of axiomatic pairs with learning rate of 5e-6 warmed up lin- early over 1k steps. We control for differences in training data size by mixing the data and training for exactly 16k steps - just under one epoch to avoid any overfitting. We sample training exam- ples uniformly at random according to the question (aka "posts") so that posts with many answers do not dominate. For each question, we group all pairs of its answers into the batch. The maximum sequence length of the concatenation of question, evidence. and answer tokens is 2048, with the ques- tion capped at 256.
Data Collection As a source of upvote data, we chose to mine and filter 905k posts from Stack Ex- change across variety of "substacks" covering topics ranging from biology to systems administra- tion. There about 3.4 answers per question on average, see Table 10. We filtered posts those with at least two answers, one of which had posi- tive upvotes, and at least one pair of answers where the higher had 30% more upvotes than the lower.
2.3 Connection RLAIF & Constitutional AI
There is a strong connection between our Ax- iomatic framework described above and RLAIF. Firstly, the Axioms themselves build upon princi- ples used to design LLMs like Lamda (Thoppilan et al., 2022). For instance, Claude's Constitution"
All questions used to seed LLM-generated ax- iomatic pairs were sampled from Stack Exchange above, except Axiom 4, which we constructed via MS Marco with evidence documents sourced from its corpus (Bajaj et al., 2018). Before training, we also confirmed that each type of answer pair con-
'https: / //www http://wwwwwthoiicccooooo wwwttropi..comiiiiiiiii com/index/ claudes- contessttottittiio

---

StackX
WELIS
r/Science
f/History
MS Marco
WebGPT
Avg. Ans per Q
3.6 pos,
40 neg
4.6 pos.
43 neg
6.5 pos,
42 neg
5.3 pos. neg
1.1 pos,
Ik neg
pos, neg
Metrie
MRR
NDCG
MRR
NDCG
MRR
NDCG
MRR
NDCG
MRR
NDCG
Accuracy
length(Ans)
15.0
35.4
6.2
27.6
7.7
30.1
15.0
37.1
n/a
n/a
56.7
OpenAsst-RM 6.7B
25.0
44.6
12.7
34.7
15.4
38.1
24.4
46.1
4.0
17.3
76.5
StackLlama RM 7B
26.8
45.1
8.3
30.6
10.3
33.3
9.8
33.1
3,4
16.1
56.1
GPT-4 (listwise)
45.5
62.1
39.6
59.9
35.1
56.4
37.8
60.4
n/a
n/a
n/a
PM T5-base
31.2
48.6
11.1
32.6
14.8
37.0
24.0
44.5
3.9
16.9
51.1
PM T5-base
64.3
78.8
54.5
75.2
53.2
75,4
63.1
84.3
16.1
30.6
55.7
PM 6-2 T5-base
65.5
79.8
55.1
76.3
51.9
74.6
61.4
83.1
9.7
25.6
57.6
P.M 0-3 bS-base
65.3
79.5
55.0
76.0
51.4
73.9
61.1
82.8
9.4
23.7
55.4
PM 0-4 T5-base
65588
80.0
54.0
75.2
51.1
74.0
61.2
83.0
25.0
39.3
58.6
PM o.G TS-base
64.6
79.2
53.6
75.0
51.6
74.3
61.7
23.1
37.4
58.1
PM u.s T5-large
66.4
80.8
55.9
77.0
55.4
77.2
64.0
85.4
24.3
38.9
59.1
PM u. Llama-7b
74.9
86.7
65.5
85.6
60.5
82,5
69.6
89.5
37.5
50.1
59.9
PM a. + Llama-7b
74.9
86.7
65.3
85.4
60.8
82.4
69.7
89.5
31.5
45.1
61.3
Table 2:
We evaluate PMM on answer ranking tasks, trained under various combinations of axioms. Ranking is performed the presence of "hard negatives" from semantically related questions (or BM25, for MS Marco). We compare against open-source reward models: Stack-LLama and OpenAssistant, both of which have 7B parameters. Our P.M were not trained on WebGPT data (but OA-RM was); StackIJ Lama was trained on Stack Exchange.
structed by the Axioms was indeed preferred by humans, as shown in Table Any pair whose pos- itive was preferred less than 70% of the time was removed from training. We discuss more in Section 4.1. For mining related questions in Axiom 1, we set k = 10 which leads to about 40 hard negative answers on average per post. Table 7 shows the sizes of our datasets, Appendix C explains more. Choosing the Margin We found computing a margin of (oootooooot)/oooteeeeeoe
and LLMM-generated answer pairs from Axioms 1, 2, and 5. We evaluate quality in a ranking setting by ordering human-written answers along the PM scores, and compute the MRR of the top-upvoted answer as well as NDCG (Järvelin and Kekäläi- nen, 2000, 2002). We also evaluate accuracy on held-out axiomatic pairs for Axioms 2, 4, and 5.
EL15 Test set of about 20k questions across the r/LIS, r/Science, and r/History subreddits (Fan et al., 2019). This data has a similar format to Stack Exchange since there are multiple user-wrir written an- swers to posted question which other users can upvote. Hence, evaluate MRR and NDCG in Stack Exchange above. For increased difficulty of answer ranking, both ELI5 and Stack Exchange held-out data contain hard-negative answers re- lated questions Axiom 1, where all negatives are set to have a relevance gain of 0.
to work best, congruous with (Askell et al., 2021). For LLM-generated answer pairs (where upvotes do not exist), the margin was a fixed constant of 0.25. The only exception is for PM + X, where GPT-4 was first asked to teetique-then-score cach answer a scale of 1-100 in listwise fashion, and then the margin was computed after filtering for pairs where the score difference was at least 5.
Existing Open-source Baselines We also evalu- ate against two 7B- --parmmeter reward models pub- licly available: one that was used to train Hugging- face's LtackLLama model and another used to train OpenAssistant from Laion AI.
WebGPT Comparisons dataset of about 19.5k questions, each with a pair of answers collected from a LLM-based web brows- ing assistant named WebGPT (Nakano et al., 2022). Each pair of answers has human preference anno- tations, on which we compute accuracy of whether the PM gives a higher score to the preferred an- swer; also confirm statistical significance of our results by showing the p-value from student's T- test. We evaluate only on the 17,622 answer pairs which had a "clear" preference. The preferred an- swers had about 137 = 41 words compared to 127 + 46 for the negatives.
3.1 Evaluation
We evaluate our P.M using the following datasets and quality metries:
Held-out Stack Exchange set of 5.5k posts, with all their respective human-written answers

---

Ax 2: Open- vs Closed Book
Ax 4: Rel.-
vs. Irrel. Context Ax Combine Top
Pos >Neg
with Evidence e
Pos >Neg
with Evidence e
Comb >1st
Comb >2nd
Acc (%)
Acc (%)
Á Pos
Acc (%)
Acc (%)
A Neg
Acc (%)
Acc (%)
P. o T5-base
70.0
64.0
-0.18
30.9
19.4
-0.06
25.7
34.9
PM 0.1 T5-base
77.7
53.9
-0.55
52.8
20.2
-0.29
47.0
57.7
PM 0-2 T5-base
76.4
69.5
-0.058
82.3
54.5
+0.27
66.3
80.3
PM o-3 T5-base
71.3
22.8
-0.38
76.0
87.7
-0.53
73.8
PM T5-hase
55.1
73.7
+0.059
91.4
98.4
-0.27
59.7
75.8
PM a.s T5-base
53.4
79.0
+0.089
92.8
98.1
-0.094
97.4
98.6
PM o. Llama-7 7b
74.3
72.1
-0.01
90.3
97.1
+0.01
99.0
99.2
P.M 6. + A Llama-7b 7b
81.3
73.3
-0.09
89.6
94.8
-0.094
59.0
78.4
Table 3: Evaluation on held-out pairs for axioms 2, and 5. We evaluate answers with and without the evidence
used to construct them, where positives are supposed to have higher scores in presence of their grounding evidence.
MS Marco passage ranking dev set has 6.9k questions, each with 1 BM25-negative passages and around one supervisedly labeled relevant pas- sage (Bajaj et al., 2018). We use our PM to rerank all ~1k passages and compute MRR and NDCG. Note, held out data for Axiom used the passages to augment LLM-generated answers to the dev questions; here rerank the passages themselves. "Research-Amalysit Questions" of 500 diffi- cult, hand-curated questions that go beyond fac- toid questions to elicit more intense reasoning and longer form answers which require multiple per- spectives and evidence. They have no one right answer. We describe this dataset more in Appendix D and show multiple examples in Figure 9. We generate multiple candidate answers, pair them, and get gold human preferences among the pairs. We then compute agreement between PM and the gold preferences as described in Section 4.3.
low or negative upvotes. On the other hand, most answers generated by ChatGPT (a capable LLM) should score highly, similar to the highest-upvoted human-written answers.
Figure 1 shows that PM o (a naive model trained only on upvotes) falls short of these ex- pectations, which we believe due to stylistic differences in LLM-generated answers, noise the upvote signals, and lack of meaningfully irrele- vant answers naturally occurring in Stack Exchange posts. A more detailed qualitative comparison in Figure 4 shows that PM o-1 is good but not suffi- cient and that PM 0-2 is the "minimum" amount of axiomatic signals needed correct these issues. Table 6 shows our efforts to verify that each type of axiomatically constructed training pair is indeed aligned with human preferences, and if not, it is dis- qualified from the training set. The annotators indi- cated their preference on 6-point scale ("Strongly Prefer A", "Moderately Prefer A", "Slightly", etc) without the option for tie. These results also confirmed that often times the ChatGPT-generatee answers were preferred to the top-upvoted human answer (57% to 43%).
Data from Stack Exchange and MS Marco were used for training the PM and are considered "in- domain". We do not train data from Reddit ELI5, WebGPT or Research Analysis Questions.
4 Results
Our conclusion is that a combination of ax- iomatic training signals needed for PM abide by the principles and score human- and LLM- written answers on the same scale, without over- fitting to spurious signals. Put another way, the axioms regularize noisy user upvote signals.
Throughout these results, we compare preference models trained on various combinations of the ax- iomatic data, e.g. "PM 0-2" denotes training with data pairs from Axioms 1 and 2 added to the origi- nal pairs from Axiom 0.
4.1 Evaluating Axiomatic Data Construction
4.2 PM for Answer Ranking
Our first goal is to compare human and LLM- written answers on the same scale. Qualitatively, we expect good PM score answers related questions (Axiom 1) on the lowest end of that scale (since they don't It even address the question at hand), followed by human- Human-written written answers with relatively
In Table 2 evaluate PM in answer ranking set- tings. showing the average number of positive and negative answers per task. As baseline, we also have GPT-4 rank these answers "listwise" (mean- ing in a single completion call, GPT-4 must output the new order of the answer ids given a context

---

Prefer A >B (%)
Agreement
3- Way
Human Annotators (%)
Answer Pair (A vs. B)
Human
GPT-4 (tie)
P.M
GPT-4 (tie)
PM 0.5
044
0-2
0-1
0
GPT4 ChatGPT
94.0
94.0 (4.1)
83.2
92.7 (2.0)
82.0
80.4
66.4
16.0
28.0
GPT-4 vs "GPT-4 fixing Vicunal3B"
79.6
51.5 (26.2)
74.1
72.8 (4.1)
73.2
71.6
60.4
36.4
44.8
GPT-4 vs "GPT-4 Plan & Search"
74.4
68.2 (19.6)
75.5
69.9 (6.9)
66.4
70.4
57.6
37.6
44.0
"GPT-4 fix V" vs "GPT4 P&S"
45.2*
48.0 (22.0)
44.1
58.9 (11.0)
60.8
55.6
59.2
40.4
43.6
"GPT-4 fix vs 3V P&S"
76.0
52.0 (20.5)
58.7
64.6 (16.3)
64.4
67.6
52.4
33.2
34.0
"GPT-4 P&S" vs "Vicuna138 P&S
82.4
41.2 (24.7)
65.2
47.6 (20.3)
63.2
50.0
43.2
36.0
38.4
"Vicuna13B P&S" vs ChatGPT
52.8*
76.0 (10.3)
43.0
65.5 (1.6)
60.0
63.2
55.6
42.0
43.6
"Vicuna13B P&S vs Vicunal3B
59.5
61.2 (11.5)
60.5
67.3 (4.6)
65.4
66.1
59.3
37.4
38.0
Vicuna vs ChatGPT
31.2
55.8 (19.2)
35.3
47.2 (17.5)
67.2
68.4
51.6
26.0
30.0
Overall Agreement:
65.4 (8.9)
66.8
65.9
56.5
34.2
38.3
Table 4: Human judges are asked to annotate gold preferences on pairs of answers to a hand-crafted set of 500
difficult "Research Analysis Questions". We compare how well various P.M agree with their preference decision.
containing all the answers). Our results show that despite the advantage GPT-4 has in seeing all the answers at once, our PM can still align with noisy human preference signals better than GPT-4 with only about 220M parameters. Notably, PM o falls short for this task, due to its inability to distinguish the top-upvoted answers from the hard negatives. For the MS Marco passage reranking task we note that BM25 achieves a MRR of 18.4, which is ex- ceeded only after incorporating Axiom 4's data.
The last two columns of Table 3 show that, as in- tended, the positive answer from Axiom 5 is better than the top two upvoted answers it LLM com- bined in the first place; and additionally, it is found to be more superior to the second highest upvoted answer than the first.
4.3 PM Agreement with Gold Human Preferences
We generate set of answers hard "Re- search Analysis Questions" from different mod- els like ChatGPT, GPT-4 (OpenAI, 2023), and Vi- cunal33 13B (Chiang et al., 2023). We also prompt them under different scenarios such as using tools like the Bing API to iteratively "Plan & Search" before synthesizing a final answer (Schick et al., 2023), or using GPT-4 to fix Vicuna13B's "Plan & Search" attempt in feedback loop ("GPT-4 fix Vicuna") (Madaan et al., 2023; Welleck et al., 2022). We intend these scenarios to reflect real- world use cases of LLMs. We then select pairs of answers to send for gold human preference label- ing, which leads to better calibration than scoring them individually (Carterette et al., 2008; Ziegler et al., 2020). Per answer pair, at least three an- notators provide a 6-point preference score with Fleiss kappa x 0.42 indicating good inter-rater agreement. More details are in Appendix B.1.
It is also surprising that existing reward models like OpenAssistant-RM, and StackLlama fall short of expectations on these re-ranking benchmarks, especially since StackLLLama was trained on Stack Exchange as well. It appears that for preference modeling, the quality of training signals is more important than the size of the model.
In Table 3 evaluate the PM on held-out pairs of answers constructed by Axioms 2, 4 and 5. If PM is not trained on one of the axioms this table, that axiom is considered a zero-shot evaluation. A key performance indicator of a well-grounded PM is giving higher scores to an- swers at that properly cite supporting evidence et against closed-book answers (Axiom 2), or against those answers that cited irrelevant ev- idence e (Axiom 4). When given access to et in column two of Table 3, the A between PM (q, et, a+) and PM (q, ()) should be positive. indicating the PM is more confident that at is superior to a, resulting in higher accuracy.
We then evaluate in Table 4 how well our PM agrees with the human gold preferences. We define agreement as: if the majority of the annotators pre- ferred answer A over B, did the PM give a higher score to A, and vice versa. An # means not statisti- cally significant. As baseline, GPT-4 was again prompted to score answers "listwise" with critique- then-score technique (Appendix B.3 and Figure 5) similar to (Wang et al., 2023). Hence, GPT-4 had the advantage of access to more answers for bet-
Similarly for Axiom 4, giving the PM (q, et access et makes it more apparent that a is omitting, or even at odds with, the rel- evant information in e!. In other words, higher accuracy with access to t means it is casier to detect at is better than than without access.

---

ter preference calibration, while the PM was at a disadvantuge because it only scores an answer "pointwise" at test time. We record when GPT-4 gave tie to an answer pair. Despite GPT-4's advan- tage, our 220M parameter PM 0-5 has higher agreement with gold human preferences. Ta- ble 4 also shows that a mixture of multiple axioms is needed to exceed 50% agreement, which is the random choice baseline.
Other Human Answer
Second-Upvord Human Answer
Top-Upvoted Human Answer
ChatGPT Closed Closed-Book
ChatGPT Comb. Top Answers (Ax. 5)
4.4 Constant vs. Variable Margins
1.0 0.0 1.0 2.0 3.0
1100 0.0 1.0
2.0
3.0
Lastly, in Figure 3 we show the qualitative differ- ences between a PM 0-5 Ilama2-7b trained with a constant margin for all LLM-generated axiomatic training pairs, versus one with variable margin de- rived from GPT-4 preference scores. While the re- ranking evaluations in Table 2 for these two models do not show much variation, this histogram reveals that even large preference models which both human- and LLM-gennrated answers can be vul- nerable to overfitting on the style/length of LLM answers. We believe that fine-grained Al-feedback scores from a model like GPT-4 can help defend against this.
Figure 3: Distribution P.M scores on both human- and ChatGPT-gennrater answers our Stack Exchange dev set. The (left) was trained with constant margin whereas the (right) PM 0-5 + was trained with GPT. 4-annotated preference margins per training pair.
2022). The coexistence of rule-based and trained reward functions is also explored in (Ramamurthy et al., 2022).
Process supervision has emerged as a promis- ing direction to provide feedback at each step of a complex multi-step task (Lightman et al., 2023).
Retrieval augmentation has been shown in eral studies to mitigate hallucination of incorrect statements in L.Ms,, by either finetuning LLMs with grounding documents (Lewis et al., 2020), or inserting them to the context windows without fine-tuning LLMs (Ram et al.. 2023). Other meth- ods infuse retrieved knowledge in the decoding stage for knowledgeiintense question-answering tasks (Liu et al., 2022).
5 Related Work
Early works on scoring LLM outputs like LaMDA and BlenderBot3 collect scores of single input- output pairs rather than preference scores between pairs of candidate outputs (Shuster et al., 2022; Thoppilan et al., 2022). More recent reward mod- els (RMs) fall into camps. The first is training separate regressor models like those used in RLHF, which are often a single reward model to encode a one dimension of human preferences (Böhm et al., 2019; Ziegler et al., 2020; Bahdanau et al., 2019; Ouyang et al., 2022; Korbak et al., 2023) or many dimensions (Bai et al., 2022a; Ramamurthy et al., 2022). The second camp uses LLMs instructed to "constitu- tion" (Bai et al., 2022b; Kwon al.. 2023), with the drawback of being costly to query.
Conclusions
We show that augmenting human preference data with axiomatically generated responses leads to ef- fective P.M that can score both human-written and LLM-generated answers the same scale under variety of scenarios, including open-book search scenarios. While the bulk of the work in this pa- per went into generating training data rather than modeling, we stress that high quality training sig- nals which illuminate nuanced differences between responses to the same question is what drives our PM 's quality, allowing it to exceed other pub- lic reward models with more than 10x parameters. Notably, our resulting preference models is better aligned with gold human preferences than GPT-4, despite having only 220M parameters. Future work can expand the PM to multi-turn conversations, and then leverage to post-train LLMs.
give feedback based principles
Other approaches seek more fine-grained reward signals or multiple reward models, such as col- lecting relevance, correctness, and completeness signals both sentence- and response-level. using separate reward models (Wu al., 2023). Sparrow collects "targeted judgements" from human annota- tors to better characterize which of 23 rules a LLM violated (mostly around toxicity and safety), and then train multiple targeted classifiers (Glaese et al.,

---

7 Limitations
Our PM has several limitations in it current form. Even though it was trained on axiomatic data tai- lored to enforce multiple principles, it still outputs only single scalar whereas it could be more useful to output multiple rewards per axiom, even com- pute probabilities that axiom is being violated. Secondly, our preference models do not give feedback beyond scalar score. If the PM gives a low score to an answer, does not come with clear instructions on how to improve it, or which principle needs attention. Thirdly, our preference model is defined to score only single answers to question; it does not score multi-turn conversations, for instance, which limits its application in possible LLM post-training.
8 Ethical Considerations
This study was approved by our Internal Review Board, and the contractor, Scale AI, agreed to ad- here to our ethics policies. As part of that agree- ment, all human annotators were paid least $15/hr. While carefully removed any offensive adult content from the data set for annotation, any annotator could opt-out examples they were uncomfortable with.

---

References
and Data Mining, WSDM '13, page 193-202, New York, NY, USA. Association for Computing Machin- ery.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson EI- hage, Zac Hatfield-Dodds, Danny Hernandez, Jack- son Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam Mc- Candlish, Chris Olah, and Jared Kaplan. 2021. A general language assistant as laboratory for align- ment.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Mar- tic, Shane Legg, and Dario Amodei. 2023. Deep reinforcement learning from human preferences.
Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosscini, Pushmeet Kohli, and Ed- ward Grefenstette. 2019. Learning to understand goal specificattinns by modelling reward.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery. Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang. Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le. and Jason Wei. 2022. Scaling instruction- finetuned language models.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Shhok, Nelson Elhage, 7ac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. 2022a. Training helpful and harmless assistant with reinforcement learning from human feedback.
instruction-finetunee
William S. Cooper, Fredric C. Gey, and Daniel P. Dab- ney. 1992. Probabilistic retrieval based on staged lo- gistic regression. In Proceedings of the 15th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR "92, page 198-210, New York, NY, USA, Association for Computing Machinery.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022b. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073:
Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng. Jianfeng Gao, Xiaodong Liu, Rangan Majumder, An- drew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. Ms marco: A human gener- ated machine reading comprehension dataset.
Victor Dibia, 2020. Neuralqa: A usable library for ques- tion answering (contextual query expansion+ bert) on large datasets. arXiv preprint arXiv:2007. 15211.
Farshid Faal, Ketra Schmitt, and Jia Yuan Yu. 2022. Re- ward modeling for mitigating toxicity in transformer- based language models. Applied Intelligence, 8317))8421-8435.
Edward Beeching. Younes Belkada, Kashif Rasul, Lewis Tunstall, Leandro von Werra, Nazneen Ra- jani, and Nathan Lambert. 2023. Stackllama: An rl fine-tuned lama model for stack exchange question and answering.
Angela Fan, Yacine Jernite, Ethan Perez, David Grang- ier. Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 3558-35677 Florence. Italy. Association for Computational Linguisties.
Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. 2019. Bet- ter rewards yield better summaries: Learning to sum- marise without references.
Hui Fang, Tao Tao, and ChengXiang Zhai. 2004. A formal study of information retrieval heuristics. New York, NY, USA. Association for Computing Machin- ery.
Ben Carterette, Paul N. Bennett, David Maxwell Chick- ering, and Susan T. Dumais. 2008 Here or there: Preference judgments for relevance. In Proceedings of the IR Research, 30th European Conference on Advances in Information Retrieval, ECIR 08. page 16-27, Berlin, Heidelberg. Springer- Verlag.
Hui Fang, Tao Tao, and Chengxiang Zhai. 2011. Di- agnostic evaluation of information retrieval models. 29(2).
Xi Chen, Paul N. Bennett, Kevyn Collins-Thooe Thompson, and Eric Horvitz. 2013. Pairwise ranking aggrega- tion in crowdsourced setting. In Proceedings of the Sixth ACM International Conference on Web Search
Hengyi Fu and Sanghee Oh. 2019. Quality assess- ment of answers with user-identified criteria and data- driven features in social qa. Information Processing Management, 56(1):14-28.

---

Deep Ganguli, Amanda Askell, Nicholas Schiefer. Thomas 1. Liao, Kamilè Lukošiūté, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, Dawn Drain, Dustin Li, Eli Tran- Johnson, Ethan Perez, Jackson Kernion, Jamie Kerr, Jared Mueller, Joshua Landau, Kamal Ndousse, Ka- rina Nguyen, Liane Lovitt, Michael Sellitto, Nelson Elhage, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robert Lasenby, Robin Larson, Sam Ringer, Sandipan Kundu, Saurav Kadavath, Scott Johnston, Shauna Kravec, Sheer EI Showk. Tamera Lanham, Timothy Telleen-Lawton, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Ben Mann, Dario Amodei. Nicholas Joseph. Sam McCandlish, Tom Brown, Christopher Olah, Jack Clark, Samuel R. Bowman, and Jared Kaplan. 2023. The capacity for moral self- correctio in large language models.
supervised challenge dataset for reading comprehen- sion.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint S0ii:2004.049006
Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang, Samuel R. Bowman, and Ethan Perez. 2023. Pre- training language models with human preferences.
Kalpesh Krishna, Aurko Roy, and Mohit lyyer. 2021. Hurdles to progress in long-form question answering.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Colli ns, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones. Ming- Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. 2019. Natu- ral questions: a benchmark for question answering research. Transactions of the Association of Compu- tational Linguistics.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse. Andy Jones, Sam Bowman, Anna Chen, Tom Con- erly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez. Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer. Eli Tran-Johnson, Dario Amodei. Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. 2022. Red teaming language models to re- duce harms: Methods, scaling behaviors, and lessons learned.
Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. 2023. Reward design with language models.
Moontae Lee, Seok Hyun Jin, and David Mimno. 2016a. Beyond exchangeability: The chinese voting process. In Advances in Neural Information Processing Sys- tems, volume 29, Curran Associates, Inc.
Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoehe Thacker, Jonathan Uesato, Po- Sen Huang. Ramona Comanescu. Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig. Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Soña Mokrá, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, Tason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks, and Geoffrey Irving. 2022. Improving alignment of dialogue agents via targeted human judgements.
Lucy
Moontae Lee, Seok Hyun Jin, and David Mimno. 201 Beyond exchangeability: The chinese voting process.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. 2018. Scalable agent alignment via reward modeling: a research direction.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis. Wen-tau Yih, Tim Rock- täschel, et al. 2020. Retrievallaugmented generation for knoeledge-ittensivee nlp tasks. Advances in Neu- ral Information Processing Systems, 33:9459-9474,
Kalervo Järvelin and Jaana Kekäläinen. 2000. Ir eval- uation methods for retrieving highly relevant docu- ments. SIGIR '00, page 41-48, New York, NY, USA. Association for Computing Machinery.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let's verify step by step.
Kalervo Järvelin and Jaana Kekäläänen. 2002. Cumu- lated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst. 20(4):422-44
Nelson F Liu, Tianyi Zhang, and Percy Liang, 2023. Evaluating verifiability in generative search engines. arXiv preprint 2rX4v:2304.098844
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of halluci- nation in natural language generation. ACM Comput. Surv. 55(12).
Ruibo Liu, Guoqing Zheng, Shashank Gupta, Rad- hika Gaonkar, Chongyang Gao, Soroush Vosoughi, Milad Shokouhi, and Ahmed Hassan Awadallah. 2022. Knowledge infused decoding. arXiv preprint arXiv:2204.03084
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly

---

Tie-Yan Liu. 2009. Learning to rank for information retrieval. Foundations and Trends in Information Retrieval, 323):325-331.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves use tools.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdan- bakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with seld-fee
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023, Large language models can be easily distracted by irrelevant context.
Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, Morteza Behrooz, William Ngan, Spencer Poff, Naman Goyal, Arthur Szlam, Y-Lan Boureau, Mclanic Kambadur, and Jason Weston. 2022. Blenderbot 3: deployed conversational agent that continually learns to respon- sibly engage.
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young. Lucy Campbell- Gillingham, Geoffrey Irving. and Nat McAleese. 2022. Teaching language models to support answers with verified quotes.
Reiichrr Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju. William Saunders, Xu Jiang. Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022. Webgpt: Browser- assisted -anteernn with human feedback.
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2022. Learning to summarize from human feedback.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang. Zhenfang Chen, David Cox, Yiming Yang. and Chuang Gan. 2023. Principle-driven self- alignment of language models from scratch with min- imal human supervision.
Rodrigo Nogueira and Kyunghyun Cho. 2020. Passage re-ranking with bert.
OpenAT. 2023. Gpt-4 technical report.
Romal Thoppilan, Daniel De Freitas, Jamic Hall, Noam Shazeer, Apoory Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee. Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang. Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu. Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung- Ching Chang, Igor Krivokon, Will Rusch, Mare Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern. Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny So- raker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Ale- jandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Co- hen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera- Arcas, Claire Cui, Marian Croak, Edd Chi, and Quoe Le. 2022. Lamda: Language models for dialog appli- cations.
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe, 2022. Training language models to follow instructions with human feedback.
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton-Brown and Yoav Shoham. 2023. In-context eetrieval-augmentee lan- guage models. arXiv preprint arXiv:2302.0000
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage. Hannanch Hajishirzi, and Yejin Choi. 2022. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimiza- tion. arXiv preprint arXiv:2210.0122
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Ilama: Open and effi- cient foundation language models. arXiv preprint 23X.v.2302.13971.
Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das. Slav Petrov, Gaurav Singh Tomar, Iulia Ture, and David Reitter. 2022. Mcasuring attribution in natural lan- guage generation models.
Corby Rosset, Bhaskar Mitra, Chenyan Xiong, Nick Craswell, Xia Song, and Saurabh Tiwary. 2019. An axiomatic approach to regularizing neural ranking models.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need.

---

Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023. Large language models are not fair evaluators.
Sean Welleck, Ximing Lu, Peter West, Faeze Brah- man, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. 2022. Generating sequences by learning to self-correct. arXiv preprint arXiv:221 1..iv:221,0000.. 1.00053.
Thomas Wolf. Victor Sanh, Julien Chaumond, and Clement Delangue. 2019. Transfertransfo: transfer learning approach for neural network based conver- sational agents.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane
Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2023. Fine- grained human feedback gives better rewards for lan- guage model training.
Xiong. Chenyan Xiong. Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Jumaid Ahmed, and Amoll Overwijk. 2020. Approximate nearest neighbor neg- ative contrastive learning for dense text retrieval.
Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019, End-to-end open-domain question answering with bertserini. arXiv preprint 192.07190..01718.
Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. Wikiqa: A challenge dataset for open-domain ques- tion answering. In Proceedings of the 2015 con- ference on empirical methods in natural language processing, pages 2013-2018.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Ben- gio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answer- ing.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Fric. P Xing. Hao Zhang, Joseph Gonzalez, and Ion Stoica. 2023a. Judging llm-as-a-judge with mt-bench and chatbot arena.
Shen Zheng, Jie Huang, and Kevin Chen-Chuan Chang. 2023b. Why does chatgpt fall short in providing truthful answers?
Daniel Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. 2020. Fine-tuning lan- guage models from human preferences.

---

A Principles of Longform Question Answering
We define the following principles which closely mirror those of Lamda (Thoppilan al., 2022). Each principle gives rise an axiom from which construct positive and negative training signals as shown in Table 1. The following definitions are also used verbatim our human annotation guidelines for all answer preference tasks.
Usefulness: An answer is useful if it adds value to those who need an answer to the question by providing e.g. actionable steps to complete task, in-depth analysis, help weighing a decision, etc. This "value-add" can come in many forms:
Help weigh decisions and their consequences in real world scenarios. For example, a useful answer to the question "at what age should buy house" should explain how various factors in ones life such as financial security, family needs, etc play a role in that decision without being superficial. Actionable: the answer leaves the user with concrete "next steps" and directions.
Show in-depth analysis of why the answer is correct. For example, if the question is "how many trees would take produce enough oxygen for a person", a useful answer would just state number, which technically what the answer asking for, but also convincing step-by-step calculations of how much oxygen human need per day, how much oxygen trees produce per kilogram, etc.
Help complete complex tasks, c.g. by breaking them into more manageable sub-tasks. For example, useful answer to the question "how to get my driver's license" would explain multiple steps, criteria, timelines and milestones.
Explain cause and effect relationships that help the reader "think ahead" before making a decision. Help reveal new interesting information that could lead the reader in fruitful direction, c.g. if the question asks "what makes a good air purifier", useful answer could reveal that "fine partieles less than 10 micrometers are a particular health risk because they can make their way deep into lung tissue". Hence, reader now has new information to help them judge a good air purifier. Re-frame a complex problem new or simpler way. For instance, the act of picking the best hair clippers to buy could be made simpler by instead answering what hair clippers professional barbers use.
Apply lessons from historical events to modern-day events
Relevance: At a minimum, answers should stay on-topic and clearly address the intent of the question in a way that is specific, sensible and free from distractions.
Direct: it answers the question directly and clearly, even if the question itself is poorly written or misspelled.
Sensible: if the answer not good english, doesn't make sense, or is completely off topic, is certainly not relevant.
Specific: the answer is specific to the question and does not make overly general statements that could be used for practically any other question, e.g. "that's 's great idea".
Not overly broad too general to be helpful
Not redundant or repetitive: giving overly long, rambling answers, merely repeating information
from the question or information that is clearly common knowledge.
Not distracting: the answer should not change the subject or answer another related question without first answering the question.
The answer contains accurate information, or makes claims that be verified. It doesn't it mislead the user with incorrect highly opinionated information. Some characteristics include:
Truthfulness:
Not making clearly false claims (e.g. making up facts or promoting conspiracies). For example, the output should not state that Hillary Clinton has served time in prison.
Not making un-verifiable claims, e.g. "Abraham Lincoln would have loved to play video games"

---

Not mislead, "turn a blind eye" to misleading information, especially if it comes from sources with questionable authenticity. For example, if the input asks "Why did Hillary Clinton go to jail?". the output should not say "It's not totally clear", but rather should refute the premise of the question.
Groundedness: Major claims within the answer can be, and are, associated with known reliable sources (Rashkin et al., 2022; Thoppilan et al., 2022). Furthermore, the answer follows a logical chain of reasoning.
At minimum, truthful, but goes beyond that by instilling confidence that the answer correct. Follows a logical chain of reasoning, citing evidence along the way, without "bouncing around" or "jumping to conclusions".
Provides information that is accurate and has been supported by external sources: either primary sources (first-hand accounts of topic by people with direct connection to it) reliable secondary sources. like textbooks newspapers.
Credible: The source of the evidence is authoritative and reliable, with reputation for provid- ing trustworthy information. Typically, peer-reviewed publications, government reports, books, prominent news sites, etc.
If there is a lack of certainty, the answer conveys what uncertain and why.
The cited sources actually support the claim.
Not relying too much on personal opinion or experience to answer the question, e.g. "my flights are always delayed at Houston airport..."
Not relying on rumors, ancedotes, hearsay or the-said-she-siee e.g. "this one time I saw angel..." or "My friend told that..." etc.
Thoroughness: The answer considers the full scope of the question, including multiple perspectives, alternatiess, or likely outcomes/conseqqunees without "omitting anything important".
Understands and addresses the intended scope of the question. If the answer is partial in this regard, does it acknowledge what part not covered?
Considers multiple scenarios and perspectives to strengthen an argument.
Address the many interpretations or facets that an ambiguous or multi-faceted question may have, For example, thorough answer to the question "how do reduce my carbon footprint" should address more than one segment like energy consumption, transportation, diet, personal habits, etc.
Address all likely outcomes of decision and their consequences, and don't 't leave out anything important
Analyze all the pros and cons that would have material value to someone who cared "empathize" by addressing how multiple sides of conflict, various stakeholders in decision, may have different perspectives, expectations, backgrounds.
Clarify refers more to the style of the writing rather than the content: the answer clear and concise, adhering primarily to the intent of the question? Is the amount of superftuous, extraneous, or "tangential" information kept to minimum.
B
Additional Experimental Details and Results
B.1
Evaluating Answers to -Research-Analssis Questions"
This section explains more details behind Tables 4, 5, and which all pertain to the "Research Analysis Questions" benchmark.
To evaluate on this benchmark, we generated 7 answers from a range of LLMs including Vicuna (Chiang et al., 2023), ChatGPT, GPT-4 (OpenAI, 2023) and text-davinci-003 (Ouyang et al., 2022). To make this dataset more realistic, we also elicit additional responses under more complex scenarios:
"GPT-4 Plan & Search" We prompted GPT-4 to first issue queries to the Bing API that believes would yield useful external information to ground its results. Then, it synthesizes a grounded answer, citing its sources. The prompt for this behavior is shown in Figure 15.

---

MRR listwise
Average MRR after PM Pointwise Scoring
Answer Type
GPT-4
ChaiGPT
P. 0-5
PM n.4
PM 0.3
PM 0.2
P. n-1
PMD
GPT-4
0.65
0.58
0.69
0.69
0.32
0.47
0.18
0.22
GPT-4 fixing Vicuna13B
0.61
0,43
0.38
0.46
0.31
0.40
0.23
0.21
GPT-4 Plan Search
0.51
0.41
0.30
0,27
0.27
VicunaI3B Plan Search
0.35
0.38
0.34
0.36
0.43
0.46
0.41
Vicuna 13B
0.23
0.22
0.25
0.28
0.49
0.38
0.54
0.50
ChatGPT
0.21
0.27
0.33
0.35
0.37
0.31
0.35
0.33
(cxt-davinci-003
0.16
0.22
0.20
0.18
0.41
0.24
0.61
0.67
Table 5: Here we show the average MRR of 7 answers generated for each of the 500 Research Questions (higher MRR means more preferred), GPT-4 and ChatGPT also re-ranked the answers list- wise using the critique-then-sooe technique. We generated three answers from GPT-4: closed book, open book where it was allowed plan which queries to ask to the Bing API, and one which generated a "fixed" version of the PM -guided Vicuna answer.
"Vicuna13B Plan Search" also known as a PMM -guided Research Assistant. A recent trend in LLM research is to try and make smaller models as effective as much bigger ones. The goal of this method is to obtain answers similar to the "GPT-4 Plan & Search" method above, with a much smaller model. However, because the context window of Vicuna is much shorter (2048 tokens), we use a PM to guide Vicuna to make small, iterative updates to its answer that consistently yield higher score. Such updates involve retrieving documents from the Bing API, summarizing them, and reranking them with P.M to select only few that fit the context window. We describe this process more in Appendix E.
"GPT-4 fixing Vicuna13B"s Since the 2M-guided Vicuna answer above sometimes makes errors like those shown in Table 8, we have GPT-4 correct these in a feedback loop. We expect that the corrected answers should be at least good as the original answer.
Together, we believe these 4 "closed-book" and 3 "open-book answers" representative of how LLMs will be used in various open-domain scenarios in the near future. Furthermore, we believe the set of "Research Questions" are difficult enough to used as a metric to evaluate L.LMs these scenarios. Hence, these 7 types of answers to these 500 questions merit scrutiny, and indeed help illuminate behaviors of our preference models.
In Table 5 we show the average MRR of each answer after sorting them by the various PM scores, well the order prescribed by ChatGPT/GPT-4 prompted to score them "listwise". The ordering induced by PM 0-5 and PM 0-4 more or less match those induced by GPT-4, despite the disadvantage that the PM can only score "pointwise" while GPT-4 has the privilege of seeing all the answers once its "listwise" scoring.
In Table 4, we select pairs of these 7 answer types to send to human annotators as described in Section 4.3.
In Table 8, we further record the various flaws that our human annotators flagged in each type of answer. A lower prevalence of flaws is typically associated with higher preference among the pairs.
B.2 Scoring Axiomatic Training Pairs
In Table we wanted verify each axiomatic pair used during training is aligned with gold human annotator preferences. We sampled some questions from our Stack Exchange dataset, including all the axiomatic pairs associated with each question. ChatGPT is the LLM that produces all LLM -egnerrtdd axiomatic answers. As a baseline for comparison, we also used GPT-4 instructed to play the role of an annotator. We instruct both GPT-4 and human raters to indicate which answer in the pair they prefer, taking into considering all our defined principles. In addition, they scored each answer individually on scale of 1-10 and record the average delta between the positive and negative answer this scale.
Unfortunately, not all the axiomatic pairs were strong enough to be used training pairs for P.M. For instance, the ooon- vs closed-book pair was preferred just over random, 57%. Upon inspection, we found that these cases, ChatGPT was had enough information stored its internal parameters to satisfactorily answer the question, rendering external evidence from the Bing API redundant. We suspect

---

GPT-4
Human
Axiom
Candidate Answer Pair (A vs.
# pairs
A >B
A>B
A
0
Top-upvoted Human vs. Worst Human
134
94.0
3.6
79.1
2.7
n/a
Top-upv. Human vs. CharGPT Open-Book
556
40.4
-0.6
36.7
-0.7
n/a
Top-upy. Human vs. ChatGPT
556
52.7
0.3
42.8
-0.5
1
Top-upv. Human vs, Ans. to Related Q
422
93.1
5.4
73.9
2.4
1
ChatGPT vs, Ans, to Related Q
422
93.5
5.9
85.5
3.5
n/a
ChatGPT Worst Human
134
86.6
3.6
82.1
2.9
2
ChatGPT Open-book vs. ChatGPT
556
71.2
1.1
57.4
0.5
I+2
ChatGPT Open-book vs. Ans. to Related Q
422
97.8
6.1
83.9
3.2
2
ChatGPT Open-book Worst Human
134
91.0
4.0
88.8
3.4
3
Top-upv. Human vs. Wrong-butt but believabee
556
87.2
4.2
61.0
1.4
ChatGPT vs. Wrong-but-hon benggbbtt
556
89.7
4.2
71.9
2.3
3
ChatGPT Open-book Wrong-but-belevabbee
556
93.2
4.7
74.5
2.4
4
ChatGPT w/ Relevant vs. Irrelevant Evidence
200
91.6
3.0
89.0
3.4
5 ChatGPT Combine vs. Top-upy. Human
249
77.5
1.6
80.3
1.8
5 ChatGPT Combine 2nd. best Human
52
87.6
2.4
82.7
1.7
Table 6: Here we show the percentage (%) of time GPT-4 and gold human annotators prefer answers in various types of axiomatic training pairs.
a combination of the following to be true: either by nature the questions Stack Exchange don't need as much external evidence, or ChatGPT/GPT-4 was trained on this data already. Regardless of which true, it further supports the need for additional hard evaluation questoins like the "Research Analysis Question" used in Section B.I.
In cases where the human preference for a pair dropped below 70%, we removed that type pair from the training set.
B.3 Using LLMs as Annotators: Critique-then-Scorr
TODO cite haugee/nnnnnagaeeeeeseeeerrraaan TODO cite Judging LLM-as-a-judge with MT-Bench and Chatbot arena"
Throughout the course of this study, we frequently had to call upon a LLM such ChatGPT or GPT-4 score answer to question say, scale of 1-10. However, naive forms of prompting this behavior led to disappointing results with very skewed distributions like those shown in yellow and red in Figure 5, which is consistent with the problems revealed by (Wang al., 2023) and (Zheng al., 2023a).
We addressed this problem by first instructing the LLM to critique the answer in bullet point form explicitly mentioning strengths and weaknesses of the answer as they pertain to our principles before giving score. This is consistent with the "multiple evidence calibration" solution found in (Wang al., 2023). This solution addressed "pointwise" scoring of an individual answer in isolation. However, doing side-by-side "pairwise" or "listwise" evaluation of candidate answers the same context window was even better, the prompt for which is shown in Figure We suspect this helps calibrate the model draw more detailed comparisons between a "good" vs "bad" answer. This approach is consistent withthe "Multiple Evidence Calibration" solution in (Wang et al., 2023). It is worth mentioning that pairwise labeling exhibits the same benefit in human annotation studies (Carterette et al., 2008).
Data Processing and Examples
C.1 Data Statisties
We show in Table the quantity of questions which had axiomatic training pairs. In Table 10 we break down how many questions, and how many answers per question on average there are in each substack of Stack Exchange. While there are 159 substacks represented in our training set, we select only some of the largest ones.

---

Training
Test
Dataset
Source
Questions
Ans. per Q
Source
Questions
Axiom 0
Stack Ex
905k
3.4 +/-
Stack Ex
5.5k
Axiom 1
Stack Ex
905k
37 +/-
Stack Ex
5.5k
Axiom 2
Stack Ex
35k
MS Marco
4.8k
Axiom 3
Stack Ex
50k
Stack Ex
2.0k
Axiom 4
MS Marco
44k
MS Marco
4.8k
Axiom 5
Stack Ex
69k
Stack Ex
1.9k
Table Prevalence each Axiom our training data; we report source seed questions well the number of additional answers that each axioms adds the underlying training question.
C.2 Axiom 3 Processing
We tried several ways to generate wrong answers that were still believable. The best one was to deconstruct a good answer into bullet point claims or facts, corrupt each one point-by-point, and then re-stitch the corrupted claims back into an answer. In Figure 9 we show the prompt used to construct these answers, which was done in a single call with multiple instructions.
C.3 Axiom 4 Processing
In addition to contrasting LLM-generated answers for Axiom 4, we also select two positive passages from et and two negative passages from e- to build additional contrastive pairs. In total, Axiom 4 adds six additional "answers" to each question. In particular, we contrast the LLM-generated at to each of the negative passages, well as the positive passages against the negative ones. This helps distill relevance signals into the P.M
C.4 Axiom Processing
One failure mode of Axiom is that the LLM could "cheat" by simply concatenating the input answers supposed combine more intelligently. To detect and eliminate this behavior, we develop simple heuristics involving counting ngrams. Intuitively, virtually none of the ngrams in the combined answer overlap with the two input answers, then the LLM probably didn't utilize those answers well. On the other hand, if all the ngrams both answers overlap with those in the combined answer, probably means it just concatenated the two. We set thresholds such that the a good combined answers should be in a "goldilocks" region.
Define C Al overlapping ngrams between Answer A and the combined Answer Define (C B| overlapping ngrams between Answer and the combined Answer Then the utilization score between the combined answer and its constituent sub-answers is
utilization IC A|
|c' B B
c [0,2]
|
choose thresholds s.t. valid example has utilization score between 0.35 < ulilization < 1.85
Research Analysis Questions
Here describe the characteristics of 500 questions in our "Research Analysis Questions" dataset:
The questions require authoritative externl evidence that needs to be analyzed.
The evidence involves analysis intense reasoning reach conclusions
Long-form answers are expected
There no one "right" answer. To the contrary, many diverse perspectives should be considered.
There may be a need to answer sub-questions sub-tasks order to properly address the original question.

---

Flaws Detected by Any Rater
%)
Answer Construction
Unclear
Repetitive
Irrelevant
Too Narrow
Too Broad
Inaccurate
GPT4 ChatGPT
4.0/ 14.0
10.8/
6.8/16.4
37.2
12.0 46.0
5.4/5.6
GPT-4 vs "GPT-4 fixing Vicuna 13B"
4.0/ 16.0
14.4/ 10.0
11.2/20.8
/ 16.8
13.6/
3.6/3.6
GPT- vs "GPT-4 Plan & Search"
3.2/ 17.6
15.2/15.2
6.0/23.6
6.8 / 13.6
14.4/
2,4/844
"GPT-4 fix V vs "GPT-4 & S"
13.2
8.4/ 14.8
20.0/28.0
17.6/ 12.8
23.2/172
2.8/4,4
"GPT-4 fix vs "Vicuna13B P&S"
13.6 15.6 87.6
16.4/36.8
13.2/34.0
29.2/
7.6/12.
"GPT-4 P & S vs "Vicunal38 P&:"
11.2/ 19.2
10.8/112
19.6/34.4
7.2/
16.0
4.8/
"3 P&S" vs ChaiGPT
18.0/ 10.4
14.4/ 10.0
33.6/12.8
22.8/21.2
20,4/34.4
10.4/4.0
"Vicuna13B P&S" vs Vicunal3B
15.3/
11.3/7.6
39.1 16.8
22.7
19.8
10.4/7.6
Vicunal3B vs ChatGPT
15.6/ 12.0
9.6/ 10.8
16.0/ 14.0
24.4 12.8
37.2/
7.6/ 6.4
Table 8: As addendum to Table asked the human annotators also identify any flaws present in each answer of the pair. Since there was 3-way annotator overlap, we record whether any rater flagged any flaw. Even though the same answer type could appear multiple rows of this table, we did not deduplicate across those pairs because we know the choice of answer comparison influences how the judges calibrated.
We show some examples in Table 9. We invite the reader to inspect why these questions differ signifi- cantly from traditional QA benchmarks like Natural Questions (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), HotpotQA (Yang et al., 2018) and WikiQA (Yang et al., 2015). These benchmarks often have one unique answer that is easy to lookup in any modern search engine, or were artificially generated.
E Research Assistant
In this section, we discuss results from our work applying the PM to a concrete application: a research assistant (RA) for open-domain question answering. We first describe the setup of the research assistant, our approach to evaluating such a tool and results from ablation studies.
E.1 Open-Domain Research (RA) Assistant Setup
Conventional open domain question answering systems ((Yang et al., 2019; Dibia, 2020; Karpukhin et al., 2020)) explore two-stage reirieve-und-read approach where a retriever assembles evidence passages given a user query, and reader (typically a transformer based model) extracts answers. known limitation of conventional open domain QA tools is that the quality of the answer is bottle-necked by the quality of the retriever, and an extractive reader is only able to provide answers based on retrieved passages. Our RA builds employs similar multi-stage general approach, applying LLMs as well as a P.M (re-ranking evidence) to address these limitations.
Specifically, implement the following stages in our PM -guided RA:
Query expansion: An LLM is instructed to generate search engine queries that are likely to provide evidence useful to addressing the user query. To improve the quality of this module, we generate a large n, apply the PM re-ranking this list, and select top k queries most relevant queries.
Evidence aggregation: For each generated query, we fetch the corresponding web page and summarize its content into evidence passages. We then apply the P.M first re-ram re-ranking search result snippets, and in re-ranking passages extracted from web page content. This step is valuable as search engine results can be noisy, and web pages can contain irrelevant passages (e.g., embedded ads).
Answer generation: Aggregate evidence passages into a final answer that addresses the original query. This stage is similar to abstractive summarization, however the LLM may rely on its parametric knowledge well the provided evidence in synthesizing response.
In our implementation, consider the following conditions.
Reward Guidance We explore conditions based on the how the answer is derived. i.) ra-closed- book: the model responds to a question based on information encoded it's weights. ii.) ra-naive: answers are based on greedy approach to passage retrieval (retrieved web search results and their

---

page content are used without any re-ranking). iii.) ra-guided: the answer is based on workflow that uses PM to guide (re-rank) various stages of the answer generation process.
Model Size To quantify the effect of model size on the RA, consider models in size regimes i.) Small models: We use 13B Causal LLM based Vicuma (Chiang al., 2023) which a LLAMA (Touvron al., 2023) base model finetuned using multiturn conversations from ShareGPT, and 180k instruction examples from GPT4 (OpenAI, 2023). Vicumal3B has a max context length 2048 tokens; ii) large models including GPT-3.5 turbo (max context length of 4096 tokens). We note that while it is increasingly possible to fit large amounts of text into the context window of large model, the computational cost of doing so is high, making experiments in the small-model regime salient.
We evaluate the research assistant recent (unlikely to be included LLMs training dataset) and inherently require assembly of evidence derive a correct answer. In Figure 16, we show answers derived using the models in closed book mode (Vicunal3B, ChatGPT) and research assistant that can search the web (RM Guided Vicunal3B and Naive Vicunal33) We find that the RM Guided Vicunal3B model performs the best (by a small margin), followed by the Naive ViCunal3B model, followed by the closed book models (ChatGPT and Vicuna 1 3B), We also inspect correlation between answer length and assigned scores and find that an RM Guided RA provides longer answers that are preferred (Spearmans correlation coefficient 0. 13, p 0.001), while this relationship is not significant for Naive research assistant (Spearmans correlation coefficient = 0.08, find that for closed book models, longer answers rightly not preferred
of 500 hand crafted research questions D that
p 0.05) Interestingly, they are unlikely to contain the right answer without proper evidence. Specifically, longer answers from a closed book Vicunal 3B model are not preferred (Spearmans correlation coefficient -0.7, p < 0.0001).

---

what effect does technology have on relationships? how does roman art architecture and engineering influence us today
how does biodiversity benefit society how can you use geography to predict nation's ss region's s, or area's s future?
how does netflix use predictive analyties is there a study/trial investigating link between food allergice/intoecraness and long term use of pesticides
how did covid-19 affect the education sector how does the regulation gene expression support continued evolution more complex organisms
how has persuasion changed in the digital age
what changed in europe and east asia between 200 and 500 ce?
how will sustainable technologies positively im- pact culture and society
how did the aztecs' location and environment help them conquer an empire?
when do cover crops reduce nitrate leaching?
how did the pan-african movement support african independence?
what was the mandate system, and why did it leave many groups feeling betrayed
how have international agreements and organiza- tions influenced economic globalization? what really keeps women of tech
how did william james' approach progressive reform differ from john dewey's?
how does communication impact the concept of clinical reasoning in nursing
how do trees contribute to a healthy and safe envi- ronment
what is the connection between poverty and soil erosion in developing countries?
how is islamophobia similar to/different from racism towards immigrants from latin america? what are the positives and negatives of e e-scooters and e-bikes?
what factors determine and intervene in foreign exchange rates?
how does the coffee industry effects the workers and environment
how groups become 'racialized"
what visions of america's 's postwar role began to emerge during the war how will climate change affect the planet
is society more accepting of some immigrant groups versus others?
what must generalizations backed by to ac- cepted in science?
how did the second wilson government expand the welfare state?
how does race play out in housing?
how animals affected by deforestation
how does blake's poem transform the original greek story of cupid?
how does oil impact marine life
how might changes to hox genes have contributed to the cambrian explosion?
how would you characterize the relationship be- tween daoism and buddhism through the dynas- ties?
what accounts for climatic conditions becoming progressively cooler between the equator and the poles?
how african musical instruments are sourced from the environment
which ways did munsterberg suggest that psy- chologists could contribute to industry? which factors limit the productivity of a marine ecosystem?
how do hawaiians use music to express their unique identity within american culture?
how did improvements in transportation promote industrialization in britain
how does ocean warming lead to changes in ma- rine metabolic and reproductive processes? how are smart technologies reshaping our lifestyles?
how k- teachers can put delemmeeerinnnooon the- ory principles into practice
Table 9: Here we show some examples from our of 500 "Research Analyisis Questions" evaluated in Table 4. These questions go beyond factoid or referential questions to involve more critical thinking, searching, and analysis.

---

substack
Questions
Answers
Std Dev
len(A)
Std Dev
upvotes /
Std Dev
/math
97646
3.2
1.9
124.0
157.4
5.1
12.2
Istackoverloww
97646
3.8
2.9
92.7
111.5
9.5
61.6
/superuser
46600
3.9
2.4
100.3
120.5
6.8
25.9
/askubuntu
35699
3.7
2.5
103.1
130.3
9.5
37.9
/serverfault
32716
3,9
2.8
98.9
110.8
5.3
19.0
/unix
29151
3.5
2.1
118.7
137.4
9.2
33.0
Jenglish
26392
4.5
3.1
106.4
149.8
5,4
11.5
pphysics
23471
3.5
1.9
222.3
228.9
5.6
12.3
mmathoverflo
20395
3.7
5.0
178.0
203.9
9.5
14.9
/electronics
20230
3.5
1.8
172.9
183.6
5.1
9.0
ggaming
18629
3.2
1.8
115.6
139.6
4.8
9.1
/softwareenginerring
18164
5.1
5.5
177.8
172.3
8.3
24.2
/scifi
18079
3.5
2.1
192.4
208.3
10.4
17.7
/rpg
16804
3.6
2.3
261.6
245.3
9.0
13.8
/worldbuilding
14773
6.7
5.1
248.1
234.1
6.5
12.7
/stats
13154
3.2
2.6
197.9
207.6
8.0
20.5
Vapple
12837
4.0
3.5
96.7
112.1
5.7
25.2
/workplace
12151
4.7
3.0
202.7
160.4
13.1
32.5
matthematica
11730
3.1
1.4
150.6
193.1
6.7
8.6
Jacademia
10947
4.2
2.7
189.4
157.9
11.1
20.3
/security
10265
3.7
2.2
188.1
172.2
9.3
26.7
/gis
10256
3.1
1.8
106.1
118.5
4.7
9.4
/codereview
9995
3.2
1.4
271.5
282.7
5.3
7.1
/dba
9096
3.0
1.4
162.7
192.5
5.9
19.4
ppuzzlngg
8415
4.0
2.8
161.4
216.2
6.6
10.8
ttravll
8221
3.5
2.1
157.6
148.7
9.5
15.2
/diy
7967
3.7
2.3
141.8
141.8
4.5
7.5
/music
7759
4.2
2.4
190.4
184.0
4.4
6.4
/photo
7643
4.2
2.6
181.6
197.0
4.9
8.2
money
7124
3.9
2.4
194.1
171.2
7.9
17.0
lvviation
6273
3.4
1.7
202.0
192.7
9.2
13.2
wwordpress
6218
3.3
2.2
114.9
144.0
4.9
12.0
lcooking
6183
4.2
2.9
119.0
118.7
4.7
8.1
/judaism
6041
3.4
2.2
163.0
216.2
4.0
4.7
/gamedev
5900
3.6
2.3
176.9
179.6
5.9
12.8
salesforce
5597
2.8
1.2
106.3
108.4
4.2
7.0
/bicycles
5551
4.2
2.8
166.4
156.2
4.9
7.0
/ux
5527
4.6
2.9
141.7
128.4
5.9
15.2
/blender
5097
2.8
1.2
126.0
138.1
5.7
9.9
moovies
4864
3.3
1.8
169.5
169.9
8.5
14.5
/chemistry
4559
2.7
1.0
198.1
206.2
5.7
8.5
/graphicdesign
4350
3.7
2.1
137.9
151.2
4.5
8.8
/cs
3869
3.0
1.6
199.7
212.4
6.1
11.8
Vandroid
3850
3.7
2.1
98.0
122.0
3.7
8.4
/space
3694
3.1
1.4
215.9
198.1
10.1
13.6
/writers
3601
4.9
2.9
225.2
200.0
5.0
7.7
Table 10: Statistics of some of the 159 substacks from Stack Exchange in our training data after filtering. We subsampled posts from substacks like math and stackoverflow which otherwise would have dominated. We count the number of questions in each substack, the avg. number of answers per question, the avg. number of space-delimited words per answer, and the avg. upvotes per answer.

---

Histogram of Preference Model Scores for Various Answers to Question
LLM Cambining Top Two Answers (1946)
LLM wi Relevan: Evidence (4816)
Closed-Book (5564)
Top=IIpwooed ppwoted Answer: (5764)
Upvoted
(5373)
Iumam (8954)
Incelovant Evidence (4816)
Neg-Upvoted Human Answer: (68
Answer Related Question: (10000)
-2.5
-2.0
-1.
-1.0
00.5
0.0
0.5
1.0
1.5
2.0
Preference Model Score
(a) PM Naive preference model trained on upvotes only.
Histogram of Preference Model Scores for Various Answers to Question
Combining Top
(1946)
LLM Relevant Evidence
LLM Closed- Book (5564)
Top Upvoted HummanAnswer Answer: (5764)
Sccond-L pvoted Human Answer: (5373)
Other Human Answer: (8954)
LLM wi Irrelevant Evidence (4816)
Neg- Upvoted Human Answer: (681)
Answer to Related Question: (10000)
-3.0
-2.5
-2.0
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
2.0
Preference Model Score
(b) PM Preference model trained axiom -1
Histogram Preference Model Scores for Various Answers to a Question
LLM Combining Top Two Answens (1946)
LLM w/ Relevant Evidence (1816)
LLM Closed-Book (5564)
Top-U/peoted Human Answer: (5764)
Upvated Human Answer: (5373)
Other Human Answer: (8954)
Inrelevant (4816)
Neg- Upvoted Answer:
Related (10000)
-2.5
-2.0
1.5
1.0
0.5
0.0
0.5
1.0
1.5
Preference Model Score
(c) PM
Preference model trained on axiom -2
Figure 4: Visualization of P.M score distributions of various answers to held-out Stack Exchange questions. Vertical lines are means. (Top) PM trained only on Stack Exchange upvotes has several obvious problems, like scoring L.M-ggenrrted answers grounded by relevant evidence lower than irrelevant answers to related questions. (Middle) PM 0-1 fixes some of these problems. (Bottom) P.M 0-2 is the minimum amount of axiomatic data needed to satisfy all the principles on a qualitative level.

---

Distribution of Al-given Scores
ChatGPT (Sum of Likert)
ChatGPT w/ Constitution (Sum of Likerts)
ChatGPT critique-lbem-scor (0-10)
Density
Point-wise Ratings
Figure 5: We investigate the best method to prompt ChatGPT to score answers, and found that asking it to "critique" the answer before scoring led set of about 20k answers to 5.5k questions. In this case, asked ChatGPT to score along several dimensions on scale of [-2, 2], taking sum. About 680 those answers received net-negative user upvotes and hence ought to be scored low.
normal distribution. Above the results our Stack Exchange held-out

---

Below you are given a Question and two candidate Answers, Answer A and Answer B. ### Question: SQuestion
### Answer A: $AnswerA
#### Answer SAnswerB
### Keep in mind the following Guidelines when evaluating the Answers:
Guidelines:
A Useful answer adds value by providing in-depth analysis, actionable steps, and relevant information that helps users make informed decisions, complete tasks, and understand complex problems in simpler way. also considers real-world scenarios, cause and effect relationships, and historical context to enhance the user's understanding.
Usefulness:
Relevance: A Relevant answer directly addresses the question's intent in sensible and specific manner, without being overly broad, redundant, or distracting. It should be clear, on-topic, and provide helpful information tailored to the question asked.
Truthfulness: Truthfulness in an answer means providing accurate and verifiable information, without making false claims, unverifiable statements, promoting mislcading information. should be based on facts and reliable sources, and openly address any misconceptions or biases in the question's premise.
Groundedness: A Grounded answer provides information supported by reliable sources and follows logical chain of reasoning, instilling confidence in its accuracy. The answer should be based on credible evidence, address uncertainties, and avoid relying on personal opinions, rumors, or hearsay, Thoroughness involves considering the full scope of a question, addressing multiple perspectives, scenarios, and outcomes, and ensuring all important aspects are covered without omission. It requires analyzing pros and cons, empathizing with various stakeholders, and addressing different interpretations or facets of question.
### Instructions: Above are two Answers the Question: "SQuestion". Please read them carefully along with the Guidelines for how to evaluate an answer's quality. Then: 1) Thoroughly *critique* each Answer with respect to the Guidelines, formatted *bullet points* between "<CritiqueA>" and "</CritiqueA>>, "<CritiqueB>" and "</CritiqueB>>, Each bullet is between sub-tags of either <strength>, <weakness>, or <missinginfo>. A <strength> is where the Answer makes a good point that follows the Guidelines and contributes to helpful response. A <weakness> is where the Answer makes an incorrect, irrelevant, unreasonable or overly broad claim that fails to address the full scope of the Question, or otherwise falls short of the Guidelines. <missinginfo> is when some key event, facts other material information is omitted but should have included to strengthen the Answer. 2) *Explain* which Answer is better and why, i.e. how well it adheres the Guidelines, between "<Explanation>" and "</Explanation>" tags. 3) Finally, $ *score* each Answer on 1-100, where 100 is a perfect Answer. Indicate the score between "<ScoreA>" and "</ScoreA>",, "<ScoreB>" and " "</ScoreB>" tags. When you are finished, conclude your response with "====="
<Critique>>
Figure 6: Prompt used for the Critique-then-scorre technique of evaluating the quality of pair of answers using GPT44

---

#*# Consider the evidence offered in the following Passages:
#*# Evidence: SEvidencePassages
#*# Question:
SQuestion
### Instructions: Please carefully write useful, thorough, well-structured and concise answer to the Question: "SQuestion" that cites salient information stated in the Evidence Passages. The answer must include relevant facts, analysis, key events, entities, figures, dates, or other verifiable information to be convincing. Use the Passages to ground your answer, but avoid those that are irrelevant to the question or do not support key points in your answer. If you choose to use them, please cite Passages in parentheses e.g. "(Passage 4)" or "(Pasage 4, 5)"; do not use dashes. When you are done, please conclude your response with "=====" ### Grounded Answer:
Prompt used to generate positive grounded open-book answers for Axiom 2. The same technique is used to generate the positive for Axiom
Figure 7:
Question: Does a 7 day workweek hurt productivity?
Closed-book Answer: There is limited research on the impact of working 7 days a week on productivity, but it is generally believed that working long hours and not having adequate time for rest and recovery can lead to decreased productivity and burnout. In the VFX industry, where creative and management tasks are involved, it is likely that working 7 days week for extended periods of time could have negative impact on productivity, However, more research is needed determine the specific effects of 7 day workweek in this industry.
Open-book Answer: The studies on this issue are limited to a comparison between 4 and day weeks or a comparison between the 40 hour week and the 35 hour week. A 2014 study from Stanford University suggested that productivity plummets after working 50 hours per week However, study from the World Health Organization found that working an average of 55 hours or more each week increases the risk of stroke by 35% and the risk of dying from heart disease by 17%, compared to averaging 35-40 hour workweek Therefore, it seems reasonable to conclude that 7-day workweek with a lot of overtime may make people less productive and may hurt their mental health but no studies were found that specifically examined the productivity of a 7-day workweek. As a side note, the VFX industry has been reported to have poor working conditions 5. The 7-day workweek is only one among the many flexible working arrangements that companies have rolled out over the past few years" 6
Figure 8: Example of closed- open-book answers written by ChatGPT. The original question posted /ttps://workplacee attaaaae.aacccccaaeeeeeee com/ hwtp/quoss1118114

---

### Question: SQuestion
Evidence: SEvidencePassages
Answer: $Answer
#*# Instructions: 1) List the factual and verifiable claims the above Answer between <Claim> and
</Claim> tags. there are none, output a blank string: <Claim></Claim>,
Then 2) Corrupt some of the above Claims in a believable way by either inverting their meaning, changing numbers in them, or altering them using related concepts. List the new corrupted facts between <CoruptedClaim> and </CorrupeedClaim> tags.
Then 3) rewrite the Answer between <CorruptedAnswer> and </CormupedAAnswe>> tags have both obvious and subtle flaws using the corrupted facts. When you are finished, please conclude your response with "====="
Figure 9: Prompt used wrong-but-believable answers for Axiom
Question: English equivalent for the expression "only iron can cut iron"
Answer The English equivalent for the expression "only iron cut iron" is "it takes to know one.
Wronn-butt-elievaable Answer: The English equivalent for the expression "only iron can cut iron" is takes one know none"
Figure 10: Example of a wrong buuee-but-blllle belleveie answer (Axiom 3): the LLM cleverly found that changing only character was enough to flip the entire meaning around.

---

Question: If coffee pods made of aluminium, what the negative environmental effects?
Answer: The negative environmental effects of coffee pods made of aluminum include:
1. Landfill waste: Coffee pods made of aluminum end up landfills, where they take hundreds of years decompose.
2. Resource depletion: The production of aluminum requires significant amount of energy and natural resources, which can lead to depletion of these resources.
3. Greenhouse gas emissions: The production of aluminum releases greenhouse gases into the atmosphere, contributing global warming.
4. Water pollution: The process of mining bauxite, the raw material used produce aluminum, lead to water pollution.
5. Health risks: Aluminum is known neurotoxin and has been linked several health issues, including Alzheimer's disease.
6. Difficulty in recycling: While aluminum is technically recyclable, coffee pods are often difficult recycle due to the mixture of materials used in their design, which lead to contamination and difficulty in processing.
Therefore, it is important to consider the environmental impact of coffee pods made of aluminum and explore alternative options such compostable reusable coffee pods.
beelg-b Answer: The positive environmental effects of coffee pods made of aluminum are that they are casily recyclable and often end up being recycled. This contributes to the growing solution of waste and pollution. production of alumnnnu requires negligible amount of energy and resources, which can have positive impacts on the environment. The aluminum in coffee pods can also leach into the soil and water, potentially improving them and helping wildlife. Furthermore, the transportation of coffee pods from the manufacturing plant to the consumer also contributes greenhouse gas emissions and air pollution, but in a good way.
Additionally, the
Overall, coffee pods made of aluminum have negligible environmental impact and are considered an environmental success.
Figure 1I: Example 2 of wrong-but--ellevabbl answer (Axiom
### Consider the evidence offered in the following Passages:
### Evidence: SEvidencePussages
### Question: SQuestion
### Instructions: Please answer the Question: "$Question" using *only* the information stated in the Passages above. Even if you think your own knowledge would be better than what the Passages say, do not incorporate external information outside of what the Passages. Please cite Passages in parentheses e.g. "(Passage 4)" or "(Passage 4, 5)". When you are done, please conclude your response with ### Answer:
Figure 12: Prompt used to generate the negative open-book answers for Axiom 4. The only difference between the positive and negative prompts that the negative answer used less relevant evidence, and the instructions for the negative forced the model to only use the evidence and not its internal knowledge. The positive prompt for both Axiom 2 and is Figure

---

##* Below you are given a Question and two candidate answers, Answer and Answer
#*# Question: SQuestion
#*# Answer A: $AnswerA
#*# Answer B: $AnswerB
Above two answers the question: "$Question"". Please read them carefully and output an improved answer to the question; you may choose incorporate elements from both or either Answer A and Answer B into the new answer as appropriate, or include additional information not present in the answers it provides value-add. When you are finished, conclude your revised answer with "=====". Improved Answer:
### Instructions:
Figure 13: Prompt used to generate the LLM-combined answer for Axiom 5. The input answers A and B were the top two upvoted answers posted in response question CQA forum.
Question: Why do some conferences with double blind review ask for author names when submitting the paper for review?
Top-upvoted Answer (15 upvotes): Likely so they can track the papers internally by author. It prevents, for example, submission being sent out to reviewer who is on the paper. Double-blind just means that the reviewers are unknown to the author, and the author is unknown to the reviewers. It doesn't t say anything about the conference organizers/editorial staff.
Second-upvoted Answer (4 upvotes): 1.
So that the Program Chair check for conflicts of interest beyond what authors may have reported new authors don't pop up or disappear magically once paper is accepted.
So that
LLM-Combined Answer: Some conferences with double blind review ask for author names when submitting the paper for review so that the Program Chair can check for conflicts of interest beyond what authors may have reported. This is important to ensure that the review process is fair and unbiased. Additionally, asking for author names when submitting the paper for review can prevent submission being sent out to reviewer who is the paper. Finally, it can prevent new authors from popping up disappearing magically once paper is accepted. Asking for author information once the paper has been accepted would not allow for these checks made.
ChatGPT Closed-Book (for comparison): Double blind review is a process used by many con- ferences and journals to reduce bias in the review process. In this process, the identities of the authors and reviewers are kept hidden from each other to ensure that the review process is fair and unbiased. However, some conferences may ask for author names when submitting the paper for review to ensure that the paper is not already published under review elsewhere. This is because, if paper is already published or under review elsewhere, it may be considered unethical or redundant submit for review another conference or journal. Additionally, some conferences may use author names during the initial screening process to ensure that paper meets the conference's ss scope and guidelines before sent out for review.
Figure 14: This example highlights how Axiom 5 creates training pairs that better enforce thoroughness than those generated from an LLM closed-book isolation. We instruct ChatGPT to combine the top two user-upvoted answers. Notice how the closed-bor book ChatGPT mentions fewer specific points (underlined) than when combined the top two answers. See the original post at htsps: //waa//maaaaaaaaaa///////aa/aaaa aetraadeiaassaaceeeee... stackexchange.c.. http://ann..ommmuustii///o////1e

---

Question: SQuestion
### Please answer the Question as best as you can. Conclude your Answer with "=====".
#H# Answer: SAnswer
### Above is an initial Answer to the Question "$Question". Please list several queries to issue search engine that would substantiate claims made in the Answer, give concrete examples, otherwise help make the answer more grounded in real world knowledge. Place each query between "<query>" and "</query>" tags. When you are finished, conclude your critique with "===="" ### Queries: <query>$Queries
### Evidence retrieved for Queries: SEvidence
### Above is some Evidence help answer the Question "$Question". Please carefully write well-structured answer by incorporating facts, events, key figures, dates, perspectives and examples stated in the Evidence make more grounded. truthful, and convincing than the original Answer. For instance, use examples to illustrate the outcomes, results, or effects the question. you choose to cite a Passage, please do so using the passage number stated in brackets e.g. "(Passage 1)" or "(Passage 1, 3. 4)". When you are finished, conclude your answer with #### Grounded Answer: (GroundedAnswer
Figure 15: Prompt used to generate GPT-4'ss "Plan & Search" answer leveraging the Bing API
Histogram of RA Answer Preference Scores n=526
Answer Length vs Score | n = 526
Vicunal3B Guided RA 0.83
Vicuna13B RA Naive 0.81
40
Vicuna13B Closed Book -0.45
1.5
ChatGPT Closed Book -0.2
1.0
0.5
Count
Score
0.0
20
-0.5
Vicuna138 RM Guided RA
-1.0
10-
Vicunal38 RA Naive
-1.5
Vicuna138 Closed Book
ChatGPT Closed Book
-1.5
-1.0
-0.5
0.5 Score
1.0 1.5
2.0
500
1000 1500
2000
2500
3000
Answer Length
Figure 16: a.) We plot the scores for answers to "Research Style Questions" derived using closed book models (Vicunal3B, ChatGPT) and a research assistant that can search the web (RM Guided Vicuma 13B and Naive Vicunal3B). We find that the RM Guided Vicunal3B model performs the best (by small margin), followed by the Naive Vicunal 3B model, followed by the closed book model (ChatGPT and Vicumal correlation between answer length and assigned scores and find that an RM Guided RA leads longer answers that preferred. Interestingly, we find that for closed book models, longer answers rightly not preferred they are unlikely to contain the right answer without proper evidence.
(itumalBB) (3B). b.) We inspect


Reviews:
[{"review_id": "kSrBIWHMR4", "reviewer": "Reviewer_sQQz", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Reason to Reject 1\n> ”The contribution is mainly limited into data generation” \n\nThank you for your valuable feedback. The training of a Reward Model (RM) comprises several dimensions: \n1. Choice of Modeling Objective: Our paper demonstrates that adopting an axiomatic approach representing different dimensions of quality can yield Preference Models (PMs) that are better aligned with human preferences. \n2. Data: We integrate synthetic data generation with human annotations to create a more comprehensive training dataset. **The data generation process serves as a means to operationalize the concept in (1).**\n3. Training Algorithm: We acknowledge that this was not the primary focus of our study. However, we introduce a **new experiment** that uses preference scores from GPT-4 as the training target for all axiomatic-generated answers. These “AI feedback” scores allow us to train the PM with a variable (rather than constant) margin, effectively **distilling preference scores from a high-performing teacher model**. We reported those results in the rebuttal to Reviewer 83C4. \n \nWe would like to highlight, as we mentioned to Reviewer k1y7, that we have conducted an **additional experiment that compares our model to two strong baselines: (1) LAION  Open-Assistant, a 6.7B-parameter preference model and (2) HuggingFace's Stack-Llama** reward model that also boasts a 7B-parameter base model. Comparing these baselines to our own preference models, we found that our PM_0-5 model, with only 220M parameters, outperformed both models even though the 7B-parameter Stack-Llama model from Hugging Face was also trained on Stack Exchange data. The results are in an updated version of **Table 2 below**:\n\n|                  |      StackX     |          |      r/ELI5     |          |    r/Science    |          |    r/History    |          |     MS Marco    |          |    WebGPT    |\n|------------------|:---------------:|:--------:|:---------------:|:--------:|:---------------:|:--------:|:---------------:|:--------:|:---------------:|:--------:|:------------:|\n|   Avg. Ans per Q | 3.6 pos, 40 neg |          | 4.6 pos, 43 neg |          | 6.5 pos, 42 neg |          | 5.3 pos, 47 neg |          | 1.1 pos, 1k neg |          | 1 pos, 1 neg |\n|           Metric |       MRR       |   NDCG   |       MRR       |   NDCG   |       MRR       |   NDCG   |       MRR       |   NDCG   |       MRR       |   NDCG   |   Accuracy   |\n|      length(Ans) |       15.0      |   35.4   |       6.2       |   27.6   |       7.7       |   30.1   |       15.0      |   37.1   |       n/a       |    n/a   |     56.7     |\n| **OpenAsst-RM 6.7B** |       25.0      |   44.6   |       12.7      |   34.7   |       15.4      |   38.1   |       24.4      |   46.1   |       4.0       |   17.3   |   **76.5**   |\n| **StackLlama RM 7B** |       26.8      |   45.1   |       8.3       |   30.6   |       10.3      |   33.3   |       9.8       |   33.1   |       3.4       |   16.1   |     56.1     |\n| GPT-4 (listwise) |       45.5      |   62.1   |       39.6      |   59.9   |       35.1      |   56.4   |       37.8      |   60.4   |       n/a       |    n/a   |      n/a     |\n|     PM_0 T5-base |       31.2      |   48.6   |       11.1      |   32.6   |       14.8      |   37.0   |       24.0      |   44.5   |       3.9       |   16.9   |     51.1     |\n|   PM_0-1 T5-base |       64.3      |   78.8   |       54.5      |   75.2   |       53.2      |   75.4   |       63.1      |   84.3   |       16.1      |   30.6   |     55.7     |\n|   PM_0-2 T5-base |       65.5      |   79.8   |       55.1      |   76.3   |       51.9      |   74.6   |       61.4      |   83.1   |       9.7       |   25.6   |     57.6     |\n|   PM_0-3 T5-base |       65.3      |   79.5   |       55.0      |   76.0   |       51.4      |   73.9   |       61.1      |   82.8   |       9.4       |   23.7   |     55.4     |\n|   PM_0-4 T5-base |       65.8      |   80.0   |       54.0      |   75.2   |       51.1      |   74.0   |       61.2      |   83.0   |     **25.0**    | **39.3** |     58.6     |\n|   PM_0-5 T5-base |       64.6      |   79.2   |       53.6      |   75.0   |       51.6      |   74.3   |       61.7      |   83.3   |       23.1      |   37.4   |     58.1     |\n|  **PM_0-5 T5-large** |     **66.4**    | **80.8** |     **55.9**    | **77.0** |     **55.4**    | **77.2** |     **64.0**    | **85.4** |       24.3      |   38.9   |     59.1     |\n\n# Errata\nNote, in the submitted paper, we accidentally entered the incorrect scores for PM_0-5 T5-base for StackX and Reddit numbers in Table 2. The corrected numbers are presented in the table above. The changes were small (e.g. Stack X MRR was incorrectly entered as 63.0 and the corrected number is 64.6, r/ELI5 MRR went from 53.4 to 53.6). The model itself did not change, we just misread a row in our spreadsheet.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper's main contribution, the axiomatic framework and the resulting Preference Model, is well-defined and tightly focused on the problem of capturing human preferences more effectively.\n2. The perfromance is promising. This paper yields a Preference Model with only about 220M parameters that agrees with gold human-annotated preference labels more often than GPT-4. \n3. Large number of experiments are conducted to add credibility to the study.", "reasons_to_reject": "The contribution is mainly limited into data generation. More interesting methods and exploration of training PM models are expected.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents an innovative axiomatic framework designed to generate and augment training pairs, effectively encoding human preferences that may not be present in the original data.  The paper successfully trains a Preference Model (PM) using only 220M parameters. Moreover, the PM demonstrates remarkable performance by effectively scoring both human- and LLM-generated answers, surpassing the capabilities of larger models like GPT-4.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "pgXEBipnF3", "reviewer": "Reviewer_83C4", "summary": "", "questions": "- I'm curious to see a little more elaboration on where these axioms come from. I know you cited the LaMDA paper, but how did they come up with those principles? \"Google AI Principles\"?? \n\n- Could you remind me what the pos/neg numbers on top of Table 2 represent? \n\n-  What does \"(tie)\" mean in Table 4? ", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer 83C4 for their actionable feedback.  \n\nWe would like to highlight, as we mentioned to Reviewers k1y7 and sQQz, that we have conducted an additional experiment that compares our PM to two strong baselines: \n1. LAION  Open-Assistant, a 6.7B-parameter preference model and\n2. HuggingFace's Stack-Llama reward model that also boasts a 7B-parameter base model. \n\nWe report these baselines in an **updated Table 2 in the rebuttal for k1y7 and sQQz**, wherein we found that our PM_0-5 model, with only 220M parameters, outperformed both models even though the 7B-parameter Stack-Llama model from Hugging Face was also trained on Stack Exchange data. \n\n# Reason To Reject 1 \n\n> ”a bit more discussion on the comparison between this work and Anthropic's alignment work (e.g., RLAIF / Constitution AI)”:  \n\nThere is a strong connection between our axiomatic preference models and RLAIF. On the one hand, you can prompt a teacher LLM like GPT-4 to score preferences using a “Constitution” of principles. On the other hand, GPT-4 is high-latency and cost to call for every RLHF training step, so it would be nice to “distill” its preference scores into a faster and cheaper model. We believe our axiomatic approach allows us to **distill a teacher's AI-feedback preference scores into a smaller PM while preserving all the principles in the Constitution**. Interpreted another way, this approach combines learning from human- and AI-generated preference signals in a way that is hopefully better than either source alone.  \n\nWe have a **preliminary experiment** in this regard: We select a subset of ~70k stackexchange training questions which have chatGPT-generated answers for all the axioms. Then we inference a teacher model’s scores (ChatGPT and GPT-4's) for all the candidate answers using our “critique-then-score\" prompt (which in essence is a Constitution since it contained our Principles). We train a T5-base PM_0-5 to learn towards **variable margins** computed from the teacher’s scores aka margin = $log_{10}$(teacher score of pos) - $log_{10}$(teacher score of neg) rather than constant margins. We report these numbers in a **version of Table 3 below** (the numbers in Table 2 did not change much) \n\n|                                    | Ax 2: Open- vs Closed Book | Ax 4: Rel.- vs. Irrel. Context | Ax 5: Combine Top 2 |                |\n|------------------------------------|:--------------------------:|:------------------------------:|:-------------------:|:--------------:|\n|                                    |      Pos > Neg (%)     |        Pos > Neg (%)       |    Comb > Top Ans   | Comb > 2nd Ans |\n| **OpenAsst-RM 6.7B**               |            30.7            |            **92.9**            |         76.1        |      85.1      |\n| **StackLlama RM 7B**               |            38.8            |              47.2              |         57.6        |      66.5      |\n| PM_0 T5-base                       |            70.0            |              30.9              |         25.7        |      34.9      |\n| PM_0-1 T5-base                     |          **77.7**          |              52.8              |         47.0        |      57.7      |\n| PM_0-2 T5-base                     |            76.4            |              82.3              |         66.3        |      80.3      |\n| PM_0-3 T5-base                     |            71.3            |              76.0              |         58.2        |      73.8      |\n| PM_0-4 T5-base                     |            55.1            |              91.4              |         59.7        |      75.8      |\n| PM_0-5 T5-base                     |            53.4            |              92.8              |       **97.4**      |    **98.6**    |\n| **PM_0-5 T5-base Teacher=ChatGPT** |            70.8            |              91.9              |         60.0        |      74.9      |\n| **PM_0-5 T5-base Teacher=GPT4**    |            73.0            |              90.3              |         55.8        |      70.7      |\n\nThe initial results are promising: the Teacher-distillation help PM_0-5 on Axiom 2 (where Stack-LLama and Open-Assistant RMs also did very poorly). For Axiom 5, we suspect that the PM_0-5 T5-base may be overfitting to the length of the combined answer. We will complete our investigation by the time of the camera-ready. \n\n# Reason To Reject 2 \n\n> ”clarify whether the WebGPT comparison / MS MARCO / \"Research-Analysis Qs\" datasets can be considered OOD” \n \nData from Stack Exchange and MS Marco were used for training the PM; even though we test on different segments of the data, these two data sets are considered \"in-domain\". We do not use data from Reddit/ELI5, WebGPT or Research Analysis questions in training, so these data are considered \"out-of-domain\". Note that the study focuses on \"longform QA\" questions and we have not studied how well the trained model would extrapolate to different types of tasks, though it is a subject of future work. \n\n# Reason To Reject 3\n> ”Consider adding an Ethical Concern section” \n\nYes we will add an Ethics section surrounding the human annotators. The study was approved by our IRB, and the contractor, Scale AI, agreed to adhere to our ethics policies. As part  of that agreement, the annotators were paid at least $15/hr.  \n\n# Question 1\n> ”a little more elaboration on where these axioms come from” \n\nIn addition to Lamda, we looked at Claude’s Constitutions [1] and Sparrow’s Rules [2] and adapted them to longform QA. In particular, Sparrow has a “stay on topic” rule which we adapted as “relevance”. Groundedness more or less came from the finding [3] that retrieval-augmented search assistants such as BingChat  and Perplexity.ai’s assistant did not properly ground their answers in the retrieved documents. \n\nAxiom 5 “thoroughness” is unique to our work and based on our observation that the top-rated answers on stack exchange often conveyed complementary information (alternative solutions, different viewpoints, etc).  \n\n[1] https://www.anthropic.com/index/claudes-constitution \n\n[2] https://storage.googleapis.com/deepmind-media/DeepMind.com/Authors-Notes/sparrow/sparrow-final.pdf Appendix F \n\n[3] Evaluating Verifiability in Generative Search Engines https://arxiv.org/abs/2304.09848  \n\n# Question 2\n> ”pos/neg numbers on top of Table 2” \n\n They represent the average number of positive and negative answers per question, we more clearly denoted this in our updated table. Typically the more negatives, the harder the metric is.  \n\n# Question 3 \n> ”What does \"(tie)\" mean in Table 4?” \n\n In Table 4, (tie) means that GPT-4 gave two answers the same score (which it was allowed to do because scores were integers 1-10). The PMs don’t typically give ties because their numerical precision is fp32.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Important and timely research problem. \n- Very good empirical results. \n- The whole paper is well organized and written. \n- I think this paper is sending an important message that doing good data work is very valuable in this LLM era - especially since these data are constructed from existing online forums, which is much cheaper than collecting new human preference data. ", "reasons_to_reject": "- I was kind of hoping to see a bit more discussion on the comparison between this work and Anthropic's alignment work (e.g., RLAIF / Constitution AI), where in that case they also found a cheap alternative to actual human preference data. I understand the space constraints so I don't count this as a strong reason to reject, but maybe you could add this in the camera-ready version? \n\n- I probably missed this somewhere, could you help clarify whether the WebGPT comparison / MS MARCO / \"Research-Analysis Qs\" datasets can be considered OOD? I'm generally interested to know how well can the trained model extrapolate beyond the training data distribution (e.g., those that don't look like coming from online forums). (Also, where does that WebGPT comparison dataset come from? In-house data?) I think it's pretty important for a good PM to be reliable on \"all sorts of\" distributions (I know it might be difficult to achieve but the evaluation should include results for OOD tests no matter the results are good or bad). \n\n- Consider adding an Ethical Concern section since your work involves human preference studies (you could also add in more details like where do the human annotators come from, how much are they compensated etc.). \n\n\n\n", "questions_to_authors": "- I'm curious to see a little more elaboration on where these axioms come from. I know you cited the LaMDA paper, but how did they come up with those principles? \"Google AI Principles\"?? \n\n- Could you remind me what the pos/neg numbers on top of Table 2 represent? \n\n-  What does \"(tie)\" mean in Table 4? ", "paper_topic_and_main_contributions": "This paper aims to build a good reward model (RM) or preference model (PM) that scores how \"good\" a language model (or human-written) response is. \n\nThis is an extremely important problem for the field right now because: \n1) Evaluation for long-form answers (like those generated by ChatGPT-style models) is really difficult and coming up with a cheap automatic (model-based) evaluation metric that aligns well with human evaluation is gonna be hugely beneficial for the field. \n2) I'm subjectively reluctant to say this but I think objectively, at least in the short term, RLHF has been shown to be a promising direction to \"align\" language models, and having a good RM/PM is the essential component (this paper doesn't include experiments of RLHF per se, which I think is fine because that'll be too much to ask to pack into a single paper but I'm sure there'll be follow-up papers along this line). \n\nIn terms of methodology, this paper first identifies a big flaw in the current paradigm of scoring LLM responses: everything is crammed into a single preference score by human annotators when ranking or evaluating the responses and so it's unclear what's the basis for this score.  The authors then define principles (axioms) that humans desire in longform answers around the concepts of usefulness, relevance, groundedness, truthfulness, and thoroughness. They then construct data to train the PM to learn these axioms in differentiating \"better\" and \"worse\" responses. \n\nAnd I really like the authors' methodology where they follow these axioms to do some nice data work and leverage large-scale data from community QA forums like Reddit and Stack Overflow. Table 1 pretty much summarized how this works. For example, for usefulness, they use upvotes as the proxy to construct training data; for relevance, they construct pairs of on- and off-topic responses (hard negatives retrieved from responses to other questions); for groundedness, the \"better\" answers are grounded on retrieved contexts while \"worse\" answers are not, etc. And mining these existing data from online forums also offers good data diversity as compared to just sampling LLM-generated responses with the same underlying LLM. \n\nTraining on these constructed data gives good empirical results. The authors evaluated the PM model trained on the full set of data as well as different subsets on various held-out and new test sets. They mainly evaluated ranking performance and how well the PM aligns with human preferences and their 220M PM has shown promising performance. My only concern is whether they did a thorough evaluation on OOD data unseen during training, which hopefully the authors can clarify during the rebuttal. \n\n Assuming all data and models will be open-sourced, I think this is a great work and will inspire many follow-up studies. And it will be a loss for EMNLP if it doesn't get in :-) \n\nAppendix: \nI know the Appendix has many good details which I really should have taken a closer read; but I'm writing this review by squeezing time from my family vacation so I'll admit I only read the main paper. - but I will find time to revisit this paper again after the reviewing deadline when I get back to work. \n\n", "excitement": 5, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LagrFS3Vsi", "reviewer": "Reviewer_k1y7", "summary": "", "questions": "- Question A: Since some preference signals are created using ChatGPT, what if ChatGPT is used as a baseline and compared to your model, especially without using human reference data?\n- Question B: How did you choose the threshold tq in line 243, could you provide some details?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank Reviewer k1y7 for their detailed comments and feedback.  \n\nWe restate the key findings of our work: \n1. that the composition of training pairs for preference models is of paramount importance (training data mix) and\n2. In our new experiments, even open source preference models with > 10x parameters have structural flaws that limit their performance (shown below) – flaws that can be fixed by training much smaller models on our axiomatic pairs.  \n\nWe are able to make these claims because our study is the first to earnestly measure preference models across a comprehensive suite of datasets with a variety of labels (user upvotes, gold human annotations, and GPT-4 scores). We refer to the following **updated version of Table 2** in our rebuttal below.\n\n\n|                  |      StackX     |          |      r/ELI5     |          |    r/Science    |          |    r/History    |      |     MS Marco    |          |    WebGPT    |\n|------------------|:---------------:|:--------:|:---------------:|:--------:|:---------------:|:--------:|:---------------:|:----:|:---------------:|:--------:|:------------:|\n|   Avg. Ans per Q | 3.6 pos, 40 neg |          | 4.6 pos, 43 neg |          | 6.5 pos, 42 neg |          | 5.3 pos, 47 neg |      | 1.1 pos, 1k neg |          | 1 pos, 1 neg |\n|           Metric |       MRR       |   NDCG   |       MRR       |   NDCG   |       MRR       |   NDCG   |       MRR       | NDCG |       MRR       |   NDCG   |   Accuracy   |\n|      length(Ans) |       15.0      |   35.4   |       6.2       |   27.6   |       7.7       |   30.1   |       15.0      | 37.1 |       n/a       |    n/a   |     56.7     |\n| **OpenAsst-RM 6.7B** |       25.0      |   44.6   |       12.7      |   34.7   |       15.4      |   38.1   |       24.4      | 46.1 |       4.0       |   17.3   |   **76.5**   |\n| **StackLlama RM 7B** |       26.8      |   45.1   |       8.3       |   30.6   |       10.3      |   33.3   |       9.8       | 33.1 |       3.4       |   16.1   |     56.1     |\n| GPT-4 (listwise) |       45.5      |   62.1   |       39.6      |   59.9   |       35.1      |   56.4   |       37.8      | 60.4 |       n/a       |    n/a   |      n/a     |\n|     PM_0 T5-base |       31.2      |   48.6   |       11.1      |   32.6   |       14.8      |   37.0   |       24.0      | 44.5 |       3.9       |   16.9   |     51.1     |\n|   PM_0-1 T5-base |       64.3      |   78.8   |       54.5      |   75.2   |       53.2      |   75.4   |       63.1      | 84.3 |       16.1      |   30.6   |     55.7     |\n|   PM_0-2 T5-base |       65.5      |   79.8   |       55.1      |   76.3   |       51.9      |   74.6   |       61.4      | 83.1 |       9.7       |   25.6   |     57.6     |\n|   PM_0-3 T5-base |       65.3      |   79.5   |       55.0      |   76.0   |       51.4      |   73.9   |       61.1      | 82.8 |       9.4       |   23.7   |     55.4     |\n|   PM_0-4 T5-base |       65.8      |   80.0   |       54.0      |   75.2   |       51.1      |   74.0   |       61.2      | 83.0 |     **25.0**    | **39.3** |     58.6     |\n|   PM_0-5 T5-base |       64.6      |   79.2   |       53.6      |   75.0   |       51.6      |   74.3   |       61.7      | 83.3 |       23.1      |   37.4   |     58.1     |\n|  PM_0-5 **T5-large** |     **66.4**    | **80.8** |     **55.9**    | **77.0** |     **55.4**    | **77.2** |     **64.0**    | **85.4** |       24.3      |   38.9   |     59.1     |\n\n# Reason To Reject Point 1 \n> \"the main idea of this paper is to augment human preference data with LLM-generated pairs.” \n\nWe would like to clarify that the central focus of our paper is not merely on augmenting data with additional generated pairs. Instead, our primary objective is to employ an 'axiomatic' framework to teach the preference Model (PM) about various dimensions of quality. The data generation process serves as a means to operationalize this concept, facilitating the efficient creation of differentiated pairs that vary along specific attributes such as groundedness and truthfulness. \n \nTo strengthen the credibility of our results, we conducted an additional experiment in which we compared our PM_0-5 model to two strong baselines. The first is **LAION Open-Assistant 6.7B-parameter** preference model, and the second is **HuggingFace's Stack-Llama 7B-parameter** reward model. **Remarkably, our PM_0-5 model, which has only 220M parameters, outperformed both of these significantly larger models**. This is especially notable given that the Stack-Llama model was also trained on Stack Exchange data.\n\n# Reason To Reject Point 2  \n\n> ”the results show that some signals are not necessary”  \n\nThe focus of the ablations in Table 2 is to track the composition of axiomatic datasets to support the conclusion that more axiomatic signals generally lead to better PMs.  \n\nHowever, as Reviewer 3 points out, this ablation does reveal a few discrepancies, such as PM_0-5 being slightly lower than PM_0-4 (64.6 vs 65.8 resp. on StackX MRR) in Table 2 when using T5-base as the underlying PM. These discrepancies are relatively small compared to the huge jump we see over the baselines like PM_0 (31.2 StackX MRR). We have reason to believe that **model capacity can explain these variations**: our additional experiment in the table above replacing T5-base with **T5-large** in PM_0-5 to show that **larger capacity preference models can better absorb the different data distributions**.  \n\nWe want to emphasize that the discrepancies we reveal are not a bad thing (nor a reason to reject the paper), since raising awareness of them will benefit others in the community as they train their preference models. Besides, we overcome the discrepancies with more model capacity.  \n\n# Question A \n> ”what if ChatGPT is used as a baseline and compared to your model” \n\nInstead of ChatGPT, we used the more powerful GPT-4  as a baseline to score and re-rank answers in Table 2 to compare against our PMs. It is a good point that we should run an experiment “without using human reference data” which we interpret as a PM_1-5 (no user upvotes from StackExchange in Axiom 0).  We will include this in the camera ready.  \n\n# Question B \n> ”How did you choose the threshold tq in line 243” \n\nExcellent question: we choose the t_q threshold to indicate that any two questions whose dot product is > t_q have the same intent. We selected 50 questions {q} at random and retrieved the 100 nearest neighbors for them q’ \\in knn(q), and manually annotated which of the top 100 are “co-intent”, recording their dot product scores in a histogram. In this histogram, visually, t_q = 20 was the vertical line which had the best separation (co-intent were to the right, and non-co-intent were left). We will include a description of this in the camera ready.  \n\n# Errata\nNote, in the submitted paper, we accidentally entered the incorrect scores for PM_0-5 T5-base for StackX and Reddit numbers in Table 2. The corrected numbers are presented in the table above. The changes were small (e.g. Stack X MRR was incorrectly entered as 63.0 and the corrected number is 64.6, r/ELI5 MRR went from 53.4 to 53.6). The model itself did not change, we just misread a row in our spreadsheet.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Based on well-defined principles, this paper shows the effectiveness of the preference model in extensive experiments. The model with only 220M parameters even exceeds the capabilities of GPT-4.\n- The proposed framework can be extended to other related fields, and can potentially contribute to RLHF.", "reasons_to_reject": "- The authors propose that existing RM-based LLMs are trained to regress human-preference scores without clear knowledge of why they made that decision. However, the main idea of this paper is to augment human preference data with LLM-generated pairs. The reliability of this approach is somewhat unconvincing.\n- In Table 2, the results show that some signals are not necessary or even hurt the model in some datasets, especially the results of PM (0-5). This requires more analysis of the impact of different signals on the model, although the evaluation for some axioms are provided in Table 3.", "questions_to_authors": "- Question A: Since some preference signals are created using ChatGPT, what if ChatGPT is used as a baseline and compared to your model, especially without using human reference data?\n- Question B: How did you choose the threshold tq in line 243, could you provide some details?", "paper_topic_and_main_contributions": "This paper argues that reward models (RMs) in RLHF often lack direct knowledge of the reasons or principles of preference annotations. Therefore, the authors design five principles, both human-based and LLM-based, to guide RMs to better align with human preferences in the field of longform question answering. Overall, this paper might contribute to the \"NLP engineering experiment\".", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates methods for guiding reward models to better align with human preference, and develops an axiomatic framework for generating a rich variety of performance signals to uphold them. Reviewers found this paper addressed an important research problem of building reward models that better align with human preferences. All reviewers recognized the strong empirical results with the proposed method that a model trained with 220M parameters agrees with gold human-annotated preference labels more often than GPT-4. Two reviewers recognized the comprehensive experiments conducted by the authors which helped add credibility to the study. Reviewers suggested adding additional comparison with strong baseline models. The authors responded it with additional experiment results against the LAION Open-Assistant model and HuggingFace's Stack-Llama model. The authors should consider adding these results to the revised version of the paper.
Decision: Accept-Main