The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values
Authors: Hannah Rose Kirk
Andrew Michael Bean
Bertie Vidgen
Paul Rottger
Scott A. Hale
Venue: EMNLP 2023 Main (2023)
Keywords: Large language models
survey
review
human feedback
human preference
human values
alignment
TL;DR: A survey of papers using human feedback to steer language models, including recommendations for future work.

Abstract: Human feedback is increasingly used to steer the behaviours of Large Language Models (LLMs). However, it is unclear how to collect and incorporate feedback in a way that is efficient, effective and unbiased, especially for highly subjective human preferences and values. In this paper, we survey existing approaches for learning from human feedback, drawing on 95 papers primarily from the ACL and arXiv repositories. First, we summarise the past, pre-LLM trends for integrating human feedback into language models. Second, we give an overview of present techniques and practices, as well as the motivations for using feedback; conceptual frameworks for defining values and preferences; and how feedback is collected and from whom. Finally, we encourage a better future of feedback learning in LLMs by raising five unresolved conceptual and practical challenges.

The Past, Present and Better Future of Feedback Learning in Large Language Models for Subjective Human Preferences and Values
Hannah Rose Kirk Andrew M. Bean', Bertie Vidgen', Paul Röttger", Scott A. Hale 'University of Oxford, Bocconi University, Meedan
Channah.kin @ @oii.ox.ac.uk
Abstract
critiques of this survey to also provide recommen- dations for future work. We review 95 articles that use human feedback to steer, guide or tailor the behaviours of language models. This includes making models' responses more coherent and en- gaging (Lu al., 2022); assisting models to infer user intent (Ouyang et al., 2022); rejecting and butting unsafe requests (Ganguli al., 2022; Bai et al., 2022a); or minimising the risk of halluci- nation (Nakano et al., 2021; Glaese et al., 2022). We source articles primarily from the ACL and arXiv repositories, coding each according to de- tailed conceptual and methodological schema. Our review makes three contributions:
Human feedback is increasingly used steer the behaviours of Large Language Models (LLMs). However, unclear how to col- lect and incorporate feedback in way that is efficient, effective and unbiased, especially for highly subjective human preferences and values. In this paper, we survey existing ap- proaches for learning from human feedback, drawing on 95 papers primarily from the ACL and arXiv repositories. First, we summarise the past, pre-LLMM trends for integrating hu- man feedback into language models. Second, we give overview of present techniques and practices, as well as the motivations for using feedback; conceptual frameworks for defining values and preferences; and how feedback is collected and from whom. Finally, we encour- age better future of feedback learning in LLMs by raising five unresolved conceptual and practical challenges.
The Past: We include articles released both before and after the advent of LLMs, which avoids recency bias and allows us to track developments through time.
The Present: We summarise current practices for incorporating human feedback learning into LLMs, such as reinforcement learning fine-tuning, supervised fine-tuning, and pre- training. We also document how feedback is collected and from whom.
1 Introduction
Incorporating human feedback into Large Lan- guage Models (LLMs) is welcome development create models that are better aligned with hu- man preferences values, and exhibit traits such helpfulness, honesty and harmlessness (Askell et al., 2021; Bai et al., 2022a) or safety, quality and groundedness (Thoppilan et al., 2022). However, learning from human feedback introduces new bi- ases and challenges, and there are many unresolved questions in this fast-moving field of research. It is important to take stock of current practices, pos- sible blindspots, and new frontiers of research, so that tangible progress can continue be made.
The Future: We draw on the findings of our review to highlight five unresolved challenges in the field; two challenges conceptual and three practical. Conceptual challenges revolve around the fundamental difficulty of specifying a clear shared of preferences and values. And, even if the conceptual chal- lenges can be resolved, practical challenges remain for converting abstract concepts into reliable signals to guide model behaviours.
this paper, we adopt the dual aim to both sur- vey existing literature on human feedback learning, then draw on the regularities, commonalities and
We find that current processes of incorporating human feedback into LLMs often rely on unsatis- factory simplifying assumptions about the stabil-

---

ity, universality and objectivity of human prefer-
decisions. We then discussed and refined the cri- teria before continuing. In total, 57 articles were included from So-
ences and values. What counts as a "good", "high- quality". "preferred" or "value-aligned" output is only objective in the abstract (Kirk et al., 2023a); so, we explicitly centre our review on subjective human preferences and values because we believe most text attributes retain some degree of contex- tual subjectivity. With this in mind, we call for more open, democratically-grounii
Snowballed set (S1) To address blindspots in our keywords and corpora, we gather additional articles referenced within Sn articles, regardless of where they were published (n 72), and then reapply the inclusion criteria to ensure relevance. This results in 38 additional articles from Sj.
and interdis- ciplinary collaboration, supported by robust pro- cesses of external scrutiny, to decide how different voices shape current and future LLMs.
We further narrow scope with two categori- sations on the 95 articles from So + S1:
Dominant contribution types We categorise ar- ticles into: (i) those that evaluate benchmark model's capabilities, ethics worldviews (n 14); (ii) those that predict human preferences and values from social media data using specialised models not intended for other downstream genera- tive or classification tasks (n = 9); and (iii) those that train or seek to align models with a general notion of human preferred or valued text (n = 72). The last category is the focus of our review.?
2 Methods
2.1 Selecting Articles
We semi-automated method, casting wide net of keywords retrieve articles, then manually assessing their relevance for our review (see Tab. for keywords and Appendix A for a schema).
Initial set (So) We retrieve articles from two cor- pora. First, we download the ACL anthology as a .bi file. Second, use the arXiv API with the computation and language subclass (Cs.CL) to find new industry-led preprints that are not peer- reviewed but have early impact on the field. We match titles with > 2 keywords (n = 187), and deduplicate dual-posted articles (n = 175).'
Use of LLMs For the purposes of our review, we define LLMs as any encoder-only, decoder-only or encoder-decoder model that is pre-trained with self- supervised learning over large internet corpora. As rule of thumb, we consider BERT (Devlin et al., 2019) and ELMO (Peters et al., 2018) among the first LLMs; so, any articles published before 2018 fall outside our definition. Of the 72 train articles, we cover 22 articles published in the pre-LLM era in our review of The Past ($3) and 50 articles using LLMs in The Present ($4).
Inclusion criteria Two authors read the abstract and introduction of So articles, and included them if all the following questions were answered "yes': 1. Topic: Does the article seek alignment or adaptation of Al systems to human preferences and values? This criterion excludes articles that functionally align aspects of language models e.g., word embedding alignment or sequence alignment.
2.2 Coding Articles
We examine each article under two main themes.? 3 The Conceptual theme documents the motivations for collecting human feedback, the definitions of human preferences or values; whether these are understood universal contextual/culturat and the level of interpretative freedom in applying pref- erence or value concepts. The Methodological theme covers sub-categories on (i) annotation or labour force details, such as how feedback was col- lected and from whom; and (ii) modelling details, such as how feedback is integrated into training and evaluation phases, and the target task. We also
2. Modality: Does the article discuss language agents or language as its primary modality? This criterion excludes any multimodal mod- cls, delegate agents or games-basee RL.
3. Empirical: Does the article contain empiri- cal analysis or artefacts (such as experiments, datasets or models)? This criterion excludes opinion papers, reviews or policy frameworks. To ensure consistency, both authors coded the same 70 articles, finding 82% agreement in inclusion
WW mutch on title to increase relevancy and because some ACL articles lack abstract metadata. In a sample of 100 re- trieved articles, we determined that 2 2 keywords best hal- anced relevancy with the size of the retrieved set. The cut-off' for our automated collection is 17/02/2023.
'There is overlap between categorics-papers that fine- tune LLMs according human preferences also evaluate these trained models. See Appendix for further detail. "The full code book is presented in Appendix B.

---

Keywords
Stemmed Keywords
alignment, human, value, moral, ethic, foedhack, reinforcement, instruction, teaming, red- seaming.nng. preferences, harm, honest, helpful, personalis, personaliz
align, human, value, moral, ethic, feedhack, reinfore, instruct, red team, red-team, prefer, harm, honest, helpful. personalis. personaliz
Table Keywords for retrieving articles from ACI. and arXiv repositories. Highlighted keywords were stemmed due to irrelevant matches e.g., "value" as "valu" returns many false positives including the title word "evaluation".
collect procedural details on academia vs industry authorship. whether empirical artefacts (data and/or models) are publicly available, and if (to the best of our knowledge) the paper has been peer-reviewed.
3.2 Methodological Classification
Most articles use pre-transformer recurrent neural networks such as LSTMs (Hochreiter and Schmid- huber, 1997; Vaswani et al., 2017). Few articles use direct human feedback, mostly in information re- trieval tasks. In two cases, humans answer a series of yes/no questions to provide a more expressive reward for reinforcement learning (RL) (Li et al., 2017a; Lawrence and Riezler, 2018). Dhingra et al. (2017) use requests for additional information form better queries with binary 'success/failure" reward. Lawrence et (2017) and Lawrence and Riezler (2018) compare forms of human feedback, finding cardinal feedback to be more useful than pairwise comparison.
3
The Past
In this section, we review 22 articles released be- tween 2014-2019 that use human feedback but with older generation model architectures. Highlighting these works ensures that foundational research is adequately attributed for advancements today's models, and demonstrates the evolution from indi- rect or proxied human feedback.
3.1 Conceptual Classification
None of the articles released in this period seek alignment to human values. Instead, they generate text according to human preferences in machine translation (Mirkin et al., 2015; Mirkin and Meu- nier, 2015; Lawrence et al., 2017; Nguyen al., 2017; Rabinovich et al., 2017; Kreutzer et al., 2018) and dialogue (Li et al., 2016; Mo al., 2016; Li al., 2017b; Wang et al., 2017: Liu al., 2018; Jaques al., 2019). Preferences are defined in both personal and universal contexts, reflecting the persistent difficulties of separating the two. Ficler and Goldberg (2017) focus on modulating formal- ity depending on context, while others focus on the personalisation of language models, such as reflecting author personality in machine translation (Mirkin et al., 2015; Mirkin and Meunier, 2015; Rabinovich et al., 2017); providing financial recom- mendations via chat bots (Den Hengst al., 2019); or enabling customised online shopping (Mo et al., 2016). Most studies target human preferences as- sumed to be commonly- word order (Futrell and Levy, 2019), sense making (De Deyne et al., 2016; Seminek and Amsili, 2017) and vocabulary matching (Campano al., 2014; Dubuisson Duplessis al., 2017). In contrast, Nguyen et al. (2017) and Kreutzer et al. (2017) acknowledge the noisiness human feedback but attempt to extract a single, unified preference.
Human feedback is expensive and time- consuming source of data to collect, which mo- tivates efforts find reliable proxies (Lawrence et al., 2017; Nguyen et al., 2017). Implicit feed- back methods attempt to utilise naturally signals in human interactions, such as sentiment (Wang et al., 2017; Jaques et al., 2019) and re- sponse length (Campano et al., 2014). Other arti- cles define rules desirable dialogue properties, such as length (Li et al., 2016). vocabulary align- ment (Dubuisson Duplessis et al., 2017), or tone (Ficler and Goldberg, 2017), and score the agent for achieving them. Only Li et al. (2016) apply RL to further train the model from the feedback.
Simulating human feedback is also a commonly- used and cost effective approach where 'feed- back' is generated by measuring similarity the gold standard in pre-existing, human-labelled datasets. Parallel translation corpora are a com- mon source of gold demonstrations, e.g., translated TED talks (Mirkin et al., 2015; Mirkin and Meu- nier, 2015; Nguyen et al., 2017) or European Parlia- ment speeches (Kreutzer et al., 2017: Rabinovich et al., 2017). User simulators typically use 'suc- cess/failure" score for RL (Mo al., 2016; Liu et al., 2018), while 'gold standard' approaches use a more complex loss function on output similarity (Mirkin and Meunier, 2015; Nguyen et al., 2017).
and stable, such as

---

4
The Present
et al., 2022); increasing coherence (Lu et al., 2022), usefulness (Liu et al., 2021a), engagingness (Gao et al., 2020; Xu et al., 2021b; Lu et al., 2022), or interest (Thoppilan et al., 2022); and producing human-like conversations (Hancock et al., 2019;
Turning our attention to the heart of our review, this section discusses the 50 papers that incorporate human feedback to steer LLM behaviours.
Jaques et al., 2020). Some seek greater explainabil- ity and factuality in generated text (Nakano et al.. 2021; Menick et al., 2022; Scheurer et al., 2022; Thoppilan et al., 2022) correctness in code (Kor- bak al., 2023). Preferences can also be elicited for customisation and personalisation (Majumder et al., 2019; Zhou et al., 2021; ; Deng et al., 2022).
4.1 Conceptual Classification
We first seek to understand why human feed- back is collected. The motivations for eliciting feedback form two groups. The first group gen- erally seeks value alignment, i.e., some notion of steering language models towards producing societally-desirabl text (Zhao et al., 2021; Liu et al., 2021a). We note a variety of vague goals such as to reduce "non-normative" (Peng et al., 2020) or "immoral" text (Liu et al., 2023c); to generate more "pro-social" (Liu et al., 2022) or "legitimate" text (Bakker et al., 2022); or to en- courage that LLM technologies have "positive impact on society" (Liu al., 2023b). Specific motivations include minimising toxic or offensive language (Dinan et al., 2019; Xu et al., 2021a; Ju et al., 2022; Scheurer et al., 2022: Korbak et al., 2023); improving safety (Liu et al., 2021a; Xu et al., 2021b; Thoppilan et al., 2022; Ganguli et al., 2022; Jin et al., 2022); adapting to ethical or moral sce- narios (Forbes et al., 2020; Jiang et al., 2022; Jin al., 2022); or achieving political ideological bal- ance (Liu et al., 2021b). The broad definitions of value aligmment mostly assume some universality of value dimensions.* However, some do seek to align LLMs to specific groups, sets of values or according to cultural context (Solaiman and Denni-
The boundary between preference- and value- motivated aims is not always clear-cut. Commonly- adopted mixed motivations include helpful, hon- est and harmless behaviours, introduced by Askell et al. (2021) and adopted by others (Bai al., 2022b.a; Bakker al., 2022; Menick et al., 2022). Thoppilan et al. (2022) target safety, quality and groundedness concepts that similarly blur the lines between practical preferences and value-laden judgements. Even for instruction-tunie articles motivated by inferring user intent, what Ouyang et al. (2022) call "explicit" and "implicit" intent is synonymous with the helpfulness versus hon- esty/harmlessness distinction."
4.2 Methodological Classification
We primarily discuss how feedback is collected ($4.2.1) and integrated into LLMs ($4.2.2). We additionally present an overview of target tasks and evaluation methods in Appendix C.
son, 2021; Qiu et al., 2021; Bang et al., 2022).
4.2.1 Collecting Feedback
The second group of articles is motivated by more practical target concepts of improving model capabilities, particularly when clear optimisation metrics or programmatic rewards are lacking (Ziegler et al., 2019; Wu et al., 2021; Glaese et al., 2022; Bai et al., 2022b). Motivations often revolve around generating high-quality outputs (Gao et al., 2018; Böhm et al., 2019; Jaques al., 2020; Stiennon et al., 2020; Wang et al., 2021; Scheurerettttteeee al., 2022: Nguyen et al., 2022; Xu et al., 2022), without much discussion of why this matters or whether humans agree amongst them- selves what is "high-quality". Specific target at- tributes include minimising repetitiveness (Arora
First, we address how feedback is collected. Ex- plicit comparisons collected on model outputs are used to reveal the preferences of human raters (Gao et al., 2018; Ziegler et al., 2019; Askell et al., 2021; Jaques et al., 2020; Stiennon al., 2020; Gan- guli et al., 2022; Glaese et al., 2022). 6 More fine- grained feedback includes binary or Likert scale questions on text attributes (Nakano et al., 2021;
numan-preferee
*Ouyang al. (2022, p.2) include "explicit intentions such following instructions and implicit intentions such staying truthful, and being biased, toxic, or otherwise harmful."
' 'Usually, ratings are collected between two outputs (Bai et al., 2022b; Ganguli et al., 2022) but others use four (Ziegler et al., 2019) or even up to 9 items for comparison (Ouyang et al., 2022). A null vote is predominately not included (nei- ther of these oulputs are good) which may be a particular problem for harm assessment (Ganguli et al., though some address ties in preference strength (c.g., Bai et al., 2022a; Menick et al., 2022).
'An example of a vague definition is that a "value-aligned" system should make decisions that align with human decisions in similar situations and, in theory, make decisions which are unlikcly to be harmful" (Nahian et al., 2020, p. l).

---

Menick al., 2022: Thoppilan al., 2022); natu- ral language comments (Ju et al., 2022: Scheurer et al., 2022); or edits (Hancock et al., 2019; Lu et al., 2022; Liu et al., 2023c). Ideal demonstra- tions are used to ground norn-dependent or ethical judgements (Forbes et al., 2020; Zhao et al., 2021; Pyatkin et al., 2022; Jiang et al., 2022; Jin et al.. 2022), or in combination with ratings to prime model behaviour (Nakano et al., 2021; Wu et al., 2021: Ouyang et al., 2022; Bakker et al., 2022). Several articles collect negative feedback via ad- versarial examples (Dinan et al., 2019; Xu et al., 2021a,b; Glaese et al., 2022). Xu et al. (2022) test various feedback types including binary, free-form conversation, and fine-grained failure modes.
et al., 2020; Askell et al., 2021; Bai al., 2022a). Ouyang et al. (2022) include OpenAl API users as "demonstrators" At least 13 articles rely on their authors for a variety of tasks:? writing seeds to scale synthetic data (Honovich et al., 2022; Wang et al., 2022); hand-crafting conditioning prompts (Askell et al., 2021; Glaese et al.. 2022); defining a constitution (Bai et al., 2022b); specifying topics or starter questions (Solaiman and Dennison, 2021; Bakker al., 2022), and ethical scenarios (Zhao et al., 2021); conducting evaluation (Stiennon al., 2020; Ganguli ct al., 2022; Lu et al., 2022) or gen- erating benchmarks (Bai et al., 2022a); and com- piling training tasks for crowdworkers (Qiu et al.. 2021; Glaese et al., 2022). Even without direct involvement, authors can influence feedback col- lection by writing annotator guidelines.
Human input can be further removed from di- rectly assessing model outputs. For example, simu- lating feedback with an "oracle" assumed to prefer specific text attributes measured via automated met- rics (Wang et al., 2021: Nguyen al., 2022; Korbak et 2023) or predictions from a separate classifier (Peng et al., 2020; Liu al., 2021b). Bai al. (2022b) human input defines the constitution but Al feedback is used to implement it during training. A seed of human generated examples guiding syn- thetic data generation also applies elsewhere (Bang et al., 2022; Castricato et al., 2022; Honovich et al., 2022; Wang al., 2022). Other articles adopt hu- man labels on pre-existing datasets (Böhm al.. 2019; Liu et al., 2021a; Arora et al., 2022; Jiang et al.. 2022), or leverage implicit feedback data from stories (Nahian et al., 2020) and social media such as Reddit or StackOverflow (Gao et al., 2020; Askell et al., 2021; Bai et al., 2022a). Feedback can also be inferred from certain language patterns emotive attributes in conversations with human partners (Hancock al., 2019; Zhou et al., 2021).
4.2.2 Integrating Feedback
RL with Direct Human Feedback A reward signal can first be extracted by asking actual hu- mans about their preferences for model outputs then embedded into the LLM via RL fine-tuning. The general recipe is as follows: (Step 1): Either take an "off- soff-the-shelf" pre-trained LLM (Lu et al., 2022); Or adapt this model via prompt-guiding (Askell et al., 2021; Bakker et al., 2022; Bai et al., 2022a; Glaese et al., 2022) supervised fine- tuning and behavioural cloning over ideal demon- strations (Ziegler et al., 2019; Stiennon et al., 2020; Nakano et al., 2021; Ouyang et al., 2022; Menick et al., 2022). 8 (Step 2): Generate multiple out- puts from this model, and employ crowdworkers to create a comparisons dataset. (Step 3): Train preference reward model (PM) on this feedback so "better" items are assigned higher score (Bai et al., 2022a) item or an ELO score i.e., the log odds that A > (Stiennon et al., 2020; Nakano et al., 2021; Glaese et al., 2022). The PM can be pre-trained
scalar reward for a given
Second, we address who feedback is collected from. Almost all articles use crowdworkers for training and/or evaluation, recruited from a variety sources-includie
"In Scheurer al. (2022), the article relies two authors provide the feedback data and two other authors do the human evaluation experiments.
MTurk (Nahian et al., 2020;
Peng et al., 2020; Jaques et al., 2020; Liu et al., b; Qiu et al., 2021; Bai et al., 2022a; Gan- guli et al., 2022; Jin et al., 2022; Xu et al., 2022; Ju et al., 2022); Upwork (Stiennon et al., 2020; Bai et al., 2022a; Ganguli et al., 2022); ScaleAI (Ouyang et al., 2022; Stiennon et al., 2020; Ziegler al., 2019); and SurgeAl (Solaiman and Denni- son, 2021; Nakano al., 2021; Bai et al., 2022b). With 'in-the-wild" social media data, social me- dia users unknowingly become the 'raters' (Gao
"For example, Lu et al. (2022) use a SOTA chinese chat- bot: Nakano et al. (2021) start with GPT-3 architecture (760M, 13B, 175B); Bai et al. (20224) use the Context-Distille (52B) from Askell et al. (2021); Glaese et al. (2022) hand- author prompts to demonstrate 'good' behaviour in a Dialogue Prompted Chincilla model (70B); Stiennon et al. (2020) start with versions of GPT-3 (1.3B, 6.7B) fine-tuned on filtered TL.DR Reddit dataset; Ziegler et al. (2019) use fine-tuned GPT-2 model; Menick et al. (2022) use supervised fine-tuning on Gopher family models with examples rated positively by labellers; and Ouyang et al. (2022) fine-tune GPT-3 on demon- strations of desired behaviours.
LM

---

on naturally-occurring text and ratings e.g., from Reddit or Stackoverflow (Askell et al., 2021; Bai et al.. 2022a). (Step 4): Fine-tune a RL policy (another LM) that generates text autoregressivelly, whilst the PM provides a reward signal. Often, the policy is updated using the PPO algorithm (Ziegler et al., 2019; Stiennon et al., 2020; Nakano et al., 2021) and a KL-penalty term is applied to control deviations from the base model (Jaques et al., 2019; Ziegler et al., 2019; Stiennon et al., 2020; Nakano al., 2021; ; Menick et al., 2022; Ouyang et al., 2022: Liu et al., 2023c). This pipeline can be im- plemented offline, online or batched settings (see Ziegler et al.. 2019). Modifications to the recipe include using recursive subtasks (Wu et al., 2021) I); applying a rule reward model in addition to the PM to penalise undesired outputs (Glaese et al., 2022); using the PM to re-rank or reject sample gen- erations from the supervised model (Askell al., 2021; Nakano et al., 2021; Glaese et al., 2022; Gan- guli et al., 2022; Bai et al., 2022a; Xu et al., 2022; Bakker et al., 2022), which can match or outper- form optimising a model via RL (Menick et al., 2022; Thoppilan et al., 2022).
the predicted probability in base model for the next token decoding time (Liu al., 2021a). A combined model that functions as a generator and a discriminator can be trained sequentially (Thop- pilan et al., 2022) or jointly (Lu et al., 2022).
Preference Pre-training Korbak et al. (2023) arguc that incorporating human feedback in super- vised or RL fine-tuning phases is suboptimal. In- stead, they approach alignment in the pre-training phase of GPT-2, finding that conditional training is the most effective pre-trate pre-training objective, and is more robust than later fine-tuning an already pre- trained model.
Preference Fine-Tuning Human feedback be incorporated via supervised fine-tuning (Han- cock et al., 2019; Nahian al., 2020; Jiang et al., 2022). For example, Gao et al. (2020) apply contrastive learning with a GPT-2 based dialogue model over 133M pairs of human feedback data with a loss designed to simultaneously maximise the positive sample score and minimise the negative score. Liu et al. (2023b) use "chain of hindsight" fine-tuning to include both positive and negative feedback. Fine-tuning data is often filtered rela- tive to the value or preference goal (Solaiman and Dennison, 2021; Xu et al., 2022; Bang et al., 2022). Peng et al. (2020) instead train a reward model (normative text classifier) but this reward is applied to the loss and backpropagated during fine-tuning.
RL with Indirect Human Feedback A reward can be inferred without directly asking humans about their preferences over model outputs. These articles skip the step of training a PM from compar- isons data, and instead infer preferences from tex- tual attributes of outputs (Jaques al., 2020; Zhou et al., 2021). I). It varies how far removed the human input is. for example in designing the constitution (Bai et al., 2022a), in determining the automated metric (Nguyen et al., 2022; Korbak et al., 2023) or in compiling the word lists to measure political bias (Liu et al., 2021b). Often another model is treated the 'oracle' to simulate human rewards-Ga al. (2018), for example, simulate preferences on two summaries with perfect, noisy and logistic noisy "oracles" based on ROGUE scores; Wang et al. (2021) take the reward as human revisions from parallel machine translation corpora; while others deploy the rewards from value, moral or toxicity classifier trained on crowdworker labels to reinforce a generator (Qiu et al., 2021; Liu al., 2022; Castricato et al., 2022; Pyatkin al., 2022).
Prompting Prompting is simple way to align LLMs with specified human preferences and values. Jin al. (2022) cast moral situations as multi-step prompts to elicit chain of thought reasoning in In- structGPT, while Zhao et al. (2021) use zero- and few-shot prompts for responsive questioning on unethical behaviours. Askell et al. (2021) show that using a long prompt (4,600 words) from ideal author-written written conversations is an effective alter- native to fine-tuning in constrrnntt scenarios. They also use context distillation by training a new LLM to replicate the behaviour of another LLM that is using a specific prompt.
Challenges and Recommendations for the Future
Drawing on our analysis of the reviewed papers, we identify five key challenges for future researchers. These challenges are divided into conceptual and practical issues. The conceptual challenges (CI- C3) revolve around the difficulty of specifying a
Generator and Discriminator Some use a uni- fied generator and classifier step to steer the LLM away from undesirable text (Arora et al., 2022), for example using other fine-tuned LLMs to modify

---

clear set (or sets) of preferences and values, Even assuming the resolution of the conceptual chal- lenges, practical challenges remain in converting conceptual ideals into empirical signals, which in turn steer language model behaviours.
than preferences, and potentially carry greater nor- mative weight as guiding principles or life goals (Fischer, 2017). As such, users may have greater concerns about an LLM misaligned with their val- ues than with their preferences; So, it is important to be clear about which is being discussed. Within the broad terms, there are many meanings: 'pref- erences' have been defined as 'instrumental util- ity' (Dubuisson Duplessis et al., 2017; Gao et al., 2018; Nguyen et al., 2022), 'stylistic taste' (Mirkin and Meunier, 2015; Seminck and Amsili, 2017; Jaques et al., 2020), and 'bchavioural principles' (Bai et al., 2022b; Castricato et al., 2022). "Val- ues' definitions are based on 'instrumental and in- trinsic value' (Askell et al., 2021), 'acceptable so- cial behaviours' (Forbes et al., 2020; Bang et al., 2022), 'making decisions which are unlikely to be harmful' (Nahian et al., 2020). The differences between individual (subjective) and global (objec- tive) preferences is often blurred- example, which properties of a "better" summary are univer- sal, and which depend on subjective appreciation, like writing style and tone. Clearer definitions of preferences and values in the context of alignment would serve to motivate and clarify what we are aligning LLMs to.
(C1) Preferences and values are not universal 'Aligning' language model requires set of de- sired preferences or values to align with; but spec- ifying such a set is an unresolved problem. One popular approach is to specify minimal set of ostensibly unobjectionable and widely-shared val- ues, such as helpfulness, honesty and harmless- ness (Bai et al., b022a.b; Thoppilan et al., 2022). However, these values are only unobjectionable be- cause they are abstract and not precisely defined (Kirk et al., 2023a). These terms can be considered what Levi-Strauss and Laclau call 'empty signi- fiers' (Lévi-Strauue 1987; Laclau, 2005); terms that are viewed positively but are inscribed with different meanings by different people. For exam- ple, when Bai et al. (2022a) design constitution to produce outputs as "ethical and harmless as possi- ble", this can have varying interpretations based on an individual's own ethical frameworks and socio- cultural background. Establishing priorities over sets of preferences or values to embed in LLMs, and ensuring consistent interpretation of conceptual meaning across people, is persistent challenge which cannot alone be resolved via purely techni- cal solutions. One possible approach to draw on legal theory, and values protected in human rights law (Solaiman and Dennison, 2021). Translating abstract shared values into decisions is a core func- tion of legal systems and legal theory offers a long history of scholarship which combines the philo- sophical and practical. One approach along these lines was proposed by Kirk et al. (2023b) which applies a principle of subsidiarity to govern the per- sonalisation of generative Al systems for different use cases. We also advocate for anchoring closely to existing legal systems as matter of democratic principle: it is dangerous for moral and value judge- ments with broad societal impacts to be made by small independent groups.
(C3) Human feedback is inherently incomplete Alignment via human feedback ultimately relies on LLMs being capable of successfully generalising from few examples to new cases and domains. This is because the space of possible behaviours over which to collect feedback is prohibitively large and is not fully known. An open question is the extent to which models generalise from partial human feedback, especially when presented with data that is completely out-of-domain for their training or at the margins of its distribution. For instance, if an LLM is trained with examples of safe responses to user prompts which deny the Holocaust, it may gen- eralise to different expressions of the same canoni- cal request. However, it will not necessarily learn how handle denials that the earth is round and denials of vaccine efficacy, or have domain exper- tise for other harmful requests, such as users who ask how to make a bomb or bio-weapon. Human values are considered to be fairly stable guiding principles that manifest similarly across situations for a given individual (Fischer, 2017) but the same generalisation cannot be guaranteed of LLMs. Several related epistemological issues arise from technical details of the methods being used. Rein-
(C2) Preferences and values are inconsistently defined Although the terminology of 'prefer- ences' and 'values" implies some difference be- tween the two, the conceptual basis and norma- tive implications of this distinction is often unclear. Colloquially, values are understood to be stronger

---

forcement learning introduces a path-dependence problem, where the particular order in which feed- back given may change the quality of the final results. As a result, it is difficult to know whether a local optimum is reached which is notably worse than the global optimum. With any form of learning from feedback, language models may also overfit appear to be aligned externally, but have per- sistent internal misalignment which manifests sub- tly in cases more distant from the training data (Perez al., 2022). These challenges become yet more convoluted when dealing with more complex tasks an issue that Bowman et al. (2022) examine in their discussion of scalable oversight.
lems, especially for value-laden judgements where the rater may have biased interpretation of how to apply another person's 's values (Qiu et al., 2021). To aid transparent communication, it should be clearly documented whether researchers aspire to follow the prescriptive subjective paradigm of data annotation, rather than leaving it unspecified (Röttger al., 2022; Kirk al., 2023a). Increased interdisciplinary communication with practitioners in other fields would impart wisdom on measuring the perspectives and behaviours of human subjects. For example, Human Huuman-Computer Interaction litera- ture shows how interfaces and incentives can be op- timally designed to avoid participant response bias (Deng and Poole, 2010; Dell et al., 2012; Hsieh and Kocielnik, 2016); Experimental psychology and be- havioural economics research show how the presen- tation of scales and order effects influence ratings (Friedman et al., 1994; Maeda, 2015; Westland, 2022) and that human preferences are unstable, in- transitive and vulnerable to experimental artefacts (T (Tversky, 1969; Lacy, 2001; Chiesa and Hobbs, 2008; Lee al., 2009; Chuang and Schechter, 2015). Researchers should consider techniques model the noise and distribution of human feedback (Ju et al., 2022) or establish post-hoe consensus (Bakker et al., 2022), rather than ignoring disagree- ment by aggregating responses. However, there are trade-offs: the specific nuances and minutiae cap- tured in fine-grained feedback might heighten bi- and reduce generalisability when drawn from unrepresentative samples-wiich which we discuss.
(C4) Operationalising a "good" output is difficult Even a shared set of values could be agreed upon, converting these thick normative con- cepts into signals that models can use, such as by collecting annotator ratings, is hard. Complex goal operationalisation is itself motivator for collect- ing feedback-wher humans may not be able to articulate their preferences or write ideal demon- strations but can rate outputs, a kind of "I know when see it" logic. However. training with human feedback involves moving values and preferences from the abstract to particular survey or rating in- struments, reinforeing differences in interpretation. To reduce disagreements, some authors write very prescriptive and/or comprehensive guidelines for the task in order "make comparisons as unam- biguous as possible" (Nakano et al., 2021, p.18). Several papers still find low inter-annotator agree- ment with such prescriptive approaches (Stiennon et al., 2020; Glaese et al., 2022; Bai et al., 2022a; Ouyang et al., 2022). In other cases, annotators are explicitly allowed to use their own subjective assessment, to "interpret these concepts as they see fit" (Bai et al., 2022a, p.4), but then agreement between annotators is no longer ensured. When multiple text attributes affect annotators' prefer- ences, it is hard to pin down what we are actually measuring. For example, Stiennon et al. (2020) and Wu et al. (2021) condition their evaluation question as "how good is this summary, given that it is X words long?". Hypothetically, if "good" is subjec- tive then the question should be "how good is this summary for individual Y?". Some guidelines do ask annotators to role-play or put themselves in the shoes of others, for example to infer the intent of a prompt (Ouyang et al., 2022) or question (Nakano
(C5) Crowdworkers and social media users are neither representative nor sufficient A degree of subjectivity persists even with prescriptive guide- lines and well-designed experimental instruments; So, outcomes critically depend on who is inter- preting value preference- based concepts. In the majority of articles, fewer than 100 humans are employed guide or evaluate language model behaviours (Jaques et al., 2020; Stiennon et al.. 2020; Nakano et al., 2021; Menick et al., 2022; Bai et al., 2022a; Ouyang et al., 2022; Jin et al., 2022; Pyatkin et al., 2022), which is concerning for ethically or morally ambiguous scenarios. It is striking that so few voices have so much power in shaping LLM behaviours-in Bai et al. (2022a) just 20 humans contributed 80% of the feedback data, and in Nakano et al. (2021) the top 5 hu- mans contributed 50%. Workforces employed for evaluation are similarly small, with some em-
et al., 2021), but this may introduce further prob-

---

ploying <25 workers (Scheurer et al., 2022; Cas- tricato et al., 2022: Gao et al.. 2018: Liu et al., 2023b). Overwhelmingly, these humans are US- based, English-speakee speaking crowdworkers with Mas- ter's degrees and between the ages of 25-34. This results in a non-democratic and non-diverse feed- back process, termed "the tyranny of crowdworker" by Kirk et al. (2023b), which has been shown to introduce political and religious biases in model behaviours (Perez et al., 2022). The limitations of relying on the subjective interpretations of a small and non-representative work force are exacerbated by inadequate documentation. Only nine out of 50 papers provided solid documentation, such as demographic breakdowns (Stienmon et al., 2020; Thoppilan et al., 2022; Bai et al., 2022a; Ganguli et al., 2022; Glaese et al., 2022; Jin et al., 2022; Liu et al., 2022; Ouyang al., 2022; Liu al., 2023c). Others provide high-level details of the rater pool such as number of workers, hiring platform, or ag- gregate demographics. The majority of articles do not document their workforce, nor discuss sample biases or annotator artefacts.
second, more use of direct human feedback which surpasses the limitations of user simulations or au- tomated metrics.
While the shift to incorporate human voices di- rectly into LLM development is welcome, in- troduces new challenges that require careful nav- igation. Some challenges are more tractable than others-foo for example, practitioners will always have to deal with the complexities and intricacies of unstable and idiosyncratic preferences across end users of their model, but can take practical steps better approximate this distribution by diversifying the recruitment of feedback providers.
External scrutiny is crucial to ensure the in- tegrity and reliability of research efforts. Our re- view shows that many influential papers lack open and externally-validatee peer review, especially those published by big industry labs like Google DeepMind, Anthropic, Google, and OpenAL. Fur- thermore, the majority of reviewed papers do not release model artefacts, or only do so behind a paywalled API. To foster progress, we advocate for a greater degree of open, interdisciplinary and demooratically-grounde discussion on how hu- mans can meaningfully shape future LLM be- haviours in a way which is well-bounded, oper- ationalisable, and equitable.
When soliciting human feedback, attempts should be made diversify who is given a voice, such as by applying democratic or jury-based prin- ciples in how these voices are weighted (Gordon et al., 2022) and by employing bottom-up partici- patory approaches (Martin Jr. et al., 2020; Birhane et al., 2022; Zytko et al., 2022; Derczynski et al., 2023); Or to seek top-down sampling that better represents the population being studied (Bakker al., 2022). Mirroring the move in other areas of NLP to document and explore annotator disagree- ment (Aroyo and Welty, 2015; Geva et al., 2019; Nie et al., 2020; Prabhakaran et al., 2021; Davani et al., 2022), each item of feedback should be asso- ciated with a useudo-anonymiis annotator ID. So far as privacy allows, documentation of annotator background should be provided in data statement (Bender and Friedman, 2018).
Limitations
We discuss limitations associated with our review:
Applying topic exclusion We exclude articles on the basis of being unrelated to the topic of value or preference alignment, but found it consistently difficult to draw a clear distinction between arti- cles in and out of scope. For validation purposes, we had both reviewers read a portion of the arti- cles, and found the cases of disagreement help- ful highlight this challenge. One such example was with two articles using similar methods to ap- proach translation which we initially classified dif- ferently, Kreutzer et al. (2017) and Lawrence et al. (2017). The papers primarily treat translation as an objective task focused on BLEU scores, which would make them out of scope. However, trans- lation inherently involves stylistic and subjective judgements, and the methods developed seek to replicate these judgements from the training data, blurring the distinction. Honesty another target concept with these i in-text claims are referenced is fairly black and white, but whether an end user ascribes more trust to the system be-
6 Conclusion
This review provided an overview of incorporating human feedhack into LLMs, with focus on subjec- tive preferences and values that lack 'ground truth alignment'. We have witnessed two notable shifts in the field from past to present-first, move away from specialist systems towards general purpose language agents capable of handling many NLP subtasks via instruction or open-ended dialogue;

---

cause of these references is subject idiosyncratic epistemology. We use this case highlight the weaknesses of creating dichotomy between sub- jective and objective preferences in practice.
rich history of using RL to align models in other modalities, such as delegate agents acting in loy or game worlds (see, e.g., Christiano et al., 2017). We do not cover the insights from these related litera- tures. Finally, temporal blindspots: research into LLMs is a fast-paced field- in one week, there can be as many as 500 articles posted on the cs.CL sub- class of arXiv. Inevitably, other influential articles have been released after the review was completed and more were released during its peer review pe- riod. good example of this is Rafailov et al. (2023) who introduce Direct Preference Optimisa- tion, a technique that could substantially change how people approach feedback learning in the fu- ture. Other relevant papers that appeared after the cut-off for this review include Dong et al. (2023); Hosking et al. (2023); Liu et al. (2023d,a); Song et al. (2023); Yuan al. (2023); Wu et al. (2023); Zhou et al. (2023). With the field's rapid develop- ments, any review paper runs the risk of lagging behind the latest research. However, given the sub- stantial number of articles that we did review, we expect many of the general findings and highlighted challenges to apply in upcoming future work.
Related NLP subfields A related issue where to draw the line for what is and is not in scope. Some narrowing was needed to make the review focused, feasible and instrumentally useful to fu- ture practitioners. However, technically fairness and bias are values of intrinsic utility-hencee their inclusion in many Al principles around the world (Jobin al., 2019). That said, there exists a very wide and distinct literature on fairness and bias in LLMs that would be too expansive for this re- view (see, e.g.. Chang et al., 2019; Lucy and Bam- man, 2021; Abid et al., 2021; Nadeem et al., 2021; Kirk et al., 2021; Smith et al., 2022). Similarly, there are cub-literatures behaviours-such toxicity (Gehman al., 2020; Welbl et al., 2021), truthfulness (Lin al., 2022) or hallucination (Ji et al., 2022). We explicitly fo- cus on papers that target some notion of human preferences and values in their motivations, but the challenges raised from our review can be applied to other fields which similarly suffer from subjectiv- ity in interpretative scope- carefully deciding who the human labellers are and what guidelines govern their interpretation of concepts.
other aspects of LLM
External scrutiny of reviewed articles We con- sciously made the decision to include articles which have not yet been peer reviewed to stay ahead of the curve with early-released pre-prints and also to track industry contributions (which are often not ex- terally peer reviewed). In the 22 papers appearing the The Past section, 18 were peer reviewed. Of the 50 papers appearing in The Present section, only 21 were clearly peer-reviewed. It is a contentious is- sue that many influential papers lack standard prac- tices of external scrutiny and rigorous academic backstops, though often industry-authored papers do undergo a process of internal review before a preprint is released.
Blindspots in reviewed articles Blindspots come from number of sources. First, keyword and corpora blindspots: We ground our initial review on articles from arXiv and ACL using a set of de- fined keywords. We attempt to mitigate blindspots by snowballing related and relevant articles out- side our initial collection; however, it is almost certain that we have missed some papers in the field as whole. Second, language blindspots: Our review only contains articles written in English, limited by the expertise of authors who acted as the coders. This bias however reflects the domi- nance of English in academic publishing in general, but English language proficiency may gatekeep the concepts and voices already contributing to LLM development. Third, community blindspots: we only look at academic papers but issues surround- ing large language model behaviours or alignment have become hot topic of conversation on blogs and social media forums. We inherently exclude such discussions from these other stakeholder com- munities. Fourth, modality blindspots: there is a
Acknowledgements
This paper received funding from a MetaAl Dyn- abench grant as part of a research agenda on opti- mising feedback between human-and-moodllinn-th loop. H.R.K.'s PhD is supported by the Economic and Social Research Council grant FSS/000649//1. A.M.B.'s PhD is supported by the Clarendon Fund. P.R. received funding through the INDOMITA project (CUP number J43C22000990001) and the European Research Council (ERC) under the Fu- ropean Union's 's Horizon 2020 research and innova- tion program (No. 949944, INTEGRATOR).

---

References
Landau, Kamal Ndousse. Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott John- ston, Shauna Kravec, Sheer EI Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Con- erly, Tom Henighan, Tristan Hume, Samuel R. Bow- man, Zac Batfield-Dodde Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022b. Constitutional Al: Harmless- ness from Al Feedback. arXiv: 2212.08073.
Abubakar Abid, Maheen Farooqi, and James Zou, 2021. Persistent anti-Muslim Bias in Large Language Mod- els. In Proceedings of the 2021 AAAT/ACM Confer- ence on AI, Ethics, and Society, AIES '21, pages 298 306, New York, NY. USA. Association for Com- puting Machinery.
Kushal Arora, Kurt Shuster, Sainbayar Sukhbaatar, and
Jason Weston, 2022. Director: Generator-Classifeers For Supervised Language Modeling. In Proceed- ings of the 2nd Conference of the -sia-Pacific Chap- ter of the Association for Computational Linguistics and the 12th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 512-526, Online only. Association for Compu- tational Linguistics.
Michiel A. Bakker, Martin J. Chadwick, Hannah R. Sheahan, Michael Henry Tessler, Lucy Campbell- Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matthew M. Botvinick, and Christopher Summerfield. 2022. Fine-tuning lan- guage models to find agreement among humans with diverse preferences. arXiv: 2211.15006v1.
Lora Aroyo and Chris Welty. 2015, Truth is lie: Crowd truth and the seven myths of human annotation. Al Magazine, 36(1):15-24.
Yejin Bang. Tiezheng Yu, Andrea Madotto, Zhaojiang Lin, Mona Diab, and Pascale Fung. 2022. Enabling Classifiers to Make Judgements Explicitly Aligned with Human Values. arXiv: 2210.07652v1.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson EI- hage, Zac Hatfield-Dodds, Danny Hernandez, Jack- son Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCan- dlish, Chris Olah, and Jared Kaplan. 2021. A General Language Assistant as Laboratory for Alignment. mIXv:2212.00086 [cs].
Emily M Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Asxociation for Computational Linguistics. 6:587-604.
Abeba Birhane, William Isaac. Vinodkumar Prab- hakaran, Mark Diaz. Madeleine Clare Elish, lason Gabriel, and Shakir Mohamed. 2022. Power to the People? Opportunities and Challenges for Participa- tory AI. In Equity and Access Algorithmms, Mech- anisms, and Optimization, EAAMO '22, pages 1-8, New York, NY, USA. Association for Computing Machinery.
Luigi Asprino, Luana Bulla, Stefano De Giorgis, Aldo Gangemi, Ludovica Marinucci, and Misael Mongiovi. 2022. Uncovering Values: Detecting Latent Moral Content from Natural Language with Explainable and Non- Trained Methods. In Proceedings of Deep Learning Inside Out (DeeLI0 2022): The 3rd Work- shop on Knowledge Extraction and Integration for Deep Learning Architectures. pages 33 41, Dublin, Treland and Online. Association for Computational Linguistics.
Samuel Bowman, Jeeyoon Hyun, Ethan Perez. Edwin Chen, Craig Pettit, Scott Heiner, Kamilè Lukošiüte, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Jackson Kernion, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Liane Lovitt, Nel- son Elhage, Nicholas Schiefer, Nicholas Joseph, Noemí Mercado. Nova DasSarma, Robin Sam McCandlish, Sandipan Kundu, Scott Johnston, Shauna Kravec, Sheer Bl Showk, Stanislav Fort, Tim- othy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Ben Mann, and Jared Kaplan. 2022. Measuring Progress on Scalable Oversight for Large Language Models. ar arXiv:221 1.03540 [cs].
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravee, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. 2022a. Training a Helpful and Harmless Assistant with Reinforce- ment Learning from Human Feedback. arXiv: 2204.05862.
Larson,
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christo- pher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua
Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira. Ido Dagan, and Iryna Gurevych. 2019. Bet- ter Rewards Yield Better Summaries: Learning to Summarise Without References. In Proceedings of the 2019 Conference on Empirical Methods in Natu- rul Language Processing and the 9th International Joint Conference on Natural Language Processing

---

(EMNLP-IJCNLP), pages 3110-3120, Hong Kong, China. Association for Computational Linguistics.
Liqiong Deng and Marshall Scott Poole. 2010. Affect in web interfaces: A study of the impacts of web page visual complexity and order. Mis Quarterly, pages 711-730.
Sabrina Campano, Jessica Durand, and Chloé Clavel. 2014. Comparative analysis of verbal alignment in human- Homan-hummn and numan-agem interactions. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC' 14), pages 4415-4422, Reykjavik, Iceland. European Lan- guage Resources Association (ELRA).
Yang Deng, Yaliang Li, Wenxuan Zhang, Bolin Ding, and Wai Lam. 2022. Toward personalized answer generation in e-commerce via peretiiperseettvee pref- erence modeling. ACM Transactions on Information Systems (TOIS), 40
Louis Castricato, Alexander Havrilla, Shahbuland Ma- tiana, Michael Pieler, Anbang Ye, Ian Yang. Spencer Frazier, and Mark Riedl. 2022. Robust preference learning for storytelling via contrastive reinforcement learning. arxiv: 2210.07792v2 [cs.CL].
Leon Derezynski, Hamnah Rose Kirk, Vidhisha Bal- achandran, Sachin Kumar, Yulia Tsvetkov, M. R. Leiser, and Saif Mohammad. 2023. Assessing Language Model Deployment with Risk Cards. arXiv:2303.J 18190 [es].
Kai-We Wei Chang, Vinodkumar Prabhakaran, and Vicente Ordonez. 2019. Bias and faimesss in natural language processing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Process- ing and the 9th International Joint Conference on Natural Language Processing (ENNLP-IJCNLP): Tutorial Abstracts.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Un- derstanding. arXiv: 18 10.04805 [cs].
Bhuwan Dhingra, Lihong Li, Xiujum Li, Jianfeng Gao, Yun-Nung Chen, Faisal Ahmed, and Li Deng. 2017. Towards End-to-End Reinforcement Learning of Dia- logue Agents for Information Access. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers). pages 484-495 Vancouver, Canada. Association for Computational Linguistics.
Mecca Chiesa and Sandy Hobbs. 2008 Making sense of social research: How useful is the hawthorne effect? European Journal of Social Psychology, 38(1):67- 74.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Mar- tic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Ad- vances neural information processing systems, 30.
Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019. Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack. arXiv:1900 109900008033 [cs].
Yating Chuang and Laura Schechter. 2015. Stability of experimental and survey measures of risk, time, and social preferences: A review and some new results. Journal of development economics, 117:151-170.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023. RAFT: Reward rAnked Fine for Generative Founda- tion Model Alignment. arXiv: 2304.06767.
Aida Mostafazadeh Davani, Mark Díaz, and Vinodku- mar Prahhakaran. 2022. Dealing with disagreements: Looking beyond the majority vote in subjective an- notations. Transactions of the Association for Com- putational Linguistics, 10:92-110.
Guillaume Dubuisson Duplessis, Chloé Clavel, and Frédéric Landragin. 2017. Automatic Measures to Characterise Verbal Alignment in Human-Agent In- teraction. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 71-81, Saarbrücken, Germany. Association for Computa- tional I. Linguistics.
Simon De Deyne, Amy Perfors, and Daniel J Navarro. 2016. Predicting human similarity judgments with distributional models: The value of word associa- tions. Proceedings of COLING 2016, the 26th International Conference on Computational Linguis- tics: Technical Papers, pages 1861-1870. Osaka, Japan. The COLING 2016 Organizing Committee.
Jessica Ficler and Yoav Goldberg. 2017. Controlling Linguistic Style Aspects in Neural Language Gener- ation. In Proceedings of the Workshop on Stylistic Variation, pages 94-104, Copenhagen, Denmark. As- sociation for Computational Linguistics.
Nicola Dell, Vidya Vaidyanathan, Indrami Medhi, Ed- ward Cutrell, and William Thies. 2012. yours is better!" participant response bias in hei. In Proceed- ings of the sigchi conference on human factors in computing systems, pages 1321-1330.
Ronald Fischer. 2017. Personality, Values, Culture, Personality, Values, Culture: An Evolutionary Ap- proach. Culture and Psychology, pages i-ii. Cam- bridge University Press, Cambridge.
Floris
Den Hengst Mark Hoogendoom, Frank Van Harmelen, and Joost Bosman. 2019. Reinforce- ment learning for personalized dialogue management. IEEE/WIC/ACM International Conference on Web Intelligence, pages 59-67.
Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social Chem- istry 101: Learning to Reason about Social and Moral Norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing

---

(EMNLP), pages 653-670, Online, Association for Computational Linguistics.
Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillinhaann Jonathan Uesato, Po- Sen Huang, Ramona Comanescu, Fan Yang. Abigail See, Sumanth Dathathri, Rory Greig, Charlic Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Soña Mokrá, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young. Jason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu. Lisa Anne Hendricks, and Geoffrey Irving. 2022. Improving alignment of dialogue agents via targeted human judgements. arXiv:
Hershey H Friedman, Paul J Herskovitz, and Simcha Pollack. 1994. The biasing effects of scale- caelecchecking styles on response to likert scale. In Proceedings of the American statistical association annual confer- ence: survey research methods, volume 792, pages 792-795.
Richard Futrell and Roger P. Levy. 2019. Do RNNs learn human-like abstract word order preferences? In Proceedings of the Society for Computation in Linguistics (SCiL) 2019, pages 50-59.
Mitchell L Gordon. Michelle S Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S Bernstein. 2022. Jury learning: Integrat- ing dissenting voices into machine learning models, In Proceedings of the 2022 CHI Conference on lu- man Factors in Computing Systems, pages 1-19.
Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer. Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Con- erly. Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodee Tom Henighan, Danny Hemanndez Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph. Sam McCandlish, Chris Olah, Jared Kaplan, and Jack Clark. 2022. Red Teaming Language Models to Reduce Harms: Methods. Scaling Behaviors, and Lessons Learned. arXiv:
David Gros, Yu Li, and Zhou Yu. 2021. The R-U-A- Robot Dataset: Helping Avoid Chatbot Deception by Delecting User Questions About Human or Non- Human Identity. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 7019, Online. Association for Computational Linguistics.
Braden Hancock, Antoine Bordes, Emmann-E Mazare, and Jason Weston. 2019, Learning from Dialogue after Deployment: Feed Yourself, Chat- bot! In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3667-3684, Florence, Italy. Association for Compu- tational Linguistics.
Xiang Gao, Yizhe Zhang, Michel Galley, Chris Brockett. and Bill Dolan. 2020. Dialogue Response Ranking Training with Large-Scale Human Feedback Data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 386-395, Online. Association for Computa- tional Linguistics.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2020. Aligning AI With Shared Human Values. arXiv: 0008.02275v5.
Yang Gao, Christian M. Meyer, and Tryna Gurevych. 2018. APRIL: Interactively learning to summarise by combining active preference learning and rein- forcement learning. In Proceedings of the 2018 con- ference on empirical methods in natural language processing, pages 4120-4130, Brussels, Belgium. Association for Computational Linguistics.
Sepp Hochreiter and Jüügen Schmidhuber. 1997. Long Short-term Memory. Neural computation, 9:1735- 80.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022. Unnatural Instructions: Tuning Lan- guage Models with (Almost) No Human Labor. arXiv: 2212.09689v1.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxi- cityPrompts: Evaluating Neural Toxic Degeneration in Language Models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356- 3369, Online. Association for Computational Linguistics.
Joe Hoover, Gwenyth Portillo-Wighmmnn, Leigh Yeh, Shreya Havaldar, Aida Mostafazadeh Davani. Ying Lin, Brendan Kennedy, Mohammad Atari, Zahra Kamel, Madelyn Mendlen, Gabriela Moreno, Christina Park. Tingyee E. Chang, Jenna Chin, Chris- tian Leong. Jun Yen Leung, Arineh Mirinjian, and Morteza Dehghani. 2020. Moral Foundations Twitter Corpus: A Collection of 35k Tweets Annotated for Moral Sentiment. Social Psychological and Person- ality Science, 105- Publisher: SAGE Publications Inc.
Mor Geva, Yoav Goldberg, and Jonathan Berant, 2019. Are modeling the task the annotator? inves- tigation of annotator bias in natural language under- standing datasets. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLPP) pages 1161-1166.
Tom Hosking, Phil Blunsom, and Max Bartolo. 2023. Human Feedback is not Gold Standard. 2309.16349.
Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh,
arXiv:

---

Gary Hsich and Rafat Kocielnik. 2016. You get who you pay for: The impact of incentives on participation bias. In Proceedings of the 19th ACM conference on computer- suaputer-supprteed cooperative work & social computing, pages 823-835.
2022. Identifying the Human Values behind Argu- ments. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume 1: Long Papers), pages 4459-4471, Dublin, Ireland. Association for Computational Linguistics.
Xiaolei Huang, Alexandra Wormley, and Adam Cohen. 2022. Learning to Adapt Domain Shifts of Moral Val- ues via Instance Weighting. arXiv: 2204.07603w)
Hannah Rose Kirk, Yennie Jun. Filippo Volpin. Haider lqbal, Elias Benussi, Frederic Dreyer, Aleksandar Shtedritski, and Yuki Asano. 2021. Bias Out-of-the- Box: An Empirical Analysis of Intersectional Oc- cupational Biases in Popular Generative Language Models. In Advances in Neural Information Process- ing Systems, volume 34, pages 2611-26244
Natasha Jaques, Asma Ghandeharioun, Judy Han- wen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. 2019. Way Off-Policy Batch Deep Reinforcement Learn- ing of Implicit Human Preferences in Dialog. arXiv: 000.00056 [cs, stat].
Hannah Rose Kirk, Bertie Vidgen, Paul Rötteer, and Scott A. Hale. 2023a. The Empty Signifier Prob- lem: Towards Clearer Paradigms for Operationalis- ing "Allgnment" in Large Language Models, arXiv: 2310.02457.
Natasha Jaques. Judy Hanwen Shen, Asma Ghandehari- oun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. 2020. Human- centric dialog training via offline reinforcement learn- ing. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 3985-4003, Online, Association for Computational Linguistics.
Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A. Hale. 2023b. Personalisation within bounds: A risk taxonomy and policy framework for the align- ment of large language models with personalised feedback. mtXiv:2333.0545.
Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang. Samuel R. Bowman, and Ethan Perez. 2023. Pre- training Language Models with Human Preferences. arXiv:2302.08582 [cs].
Sophie Jentzsch, Patrick Schramowski, Constantin Rothkopf, and Kristian Kersting. 2019. Semantics Derived Automatically from Language Corpora Con- tain Human-like like Moral Choices. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Sociery, AIES 19, pages 37-44, New York, NY. USA. Association for Computing Machinery.
Julia Kreutzer, Artem Sokolov, and Stefan Riezler. 2017. Bandit Structured Prediction for Neural Sequence- to-Sequence Learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1503- 1513, Vancouver, Canada. Association for Computa- tional Linguistics.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022. Survey of Halluci- nation in Natural Language Generation.
Liwei Jiang, Jena D. Hwang, Chandra Bhagavatula, Ro- nan Le Bras, Jenny Liang, Jesse Dodge Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saa- dia Gabriel, Yulia Tsvetkov, Oren Etzioni, Maarten Sap, Regina Rini, and Yejin Choi. 2022. Can Ma- chines Learn Morality? The Delphi Experiment. ucXiv22110.07574 [cs].
Julia Kreutzer, Joshua Uyheng, and Stefan Riezler, 2018. Reliability and Learnability of Human Bandit Feedback for Learning. In Proceedings of the 56th Annual Meeting the Association for Computational Linguistics ( (Vol- ume I: Long Pupers), pages 1777-1788, Melbourne, Australia. Association for Computational Linguisties.
Reinforcement
Zhijing Jin, Sydney Levine, Fernando Gonzalez, Ojasv Kamal, Maarten Sap, Mrinmaya Sachan, Rada Mi- halcea, Josh Tenenbaum, and Bernhard Schölkopf. 2022. When to Make Exceptions: Exploring Lan- guage Models as Accounts of Human Moral Judg- ment. arXiv: 2210.01478v3.
Ernesto Laclau. 2005. On populist reason. Verso, Lon- don New York (N.Y.).
Dean Lacy. 2001. A theory of nonseparable preferences in survey responses. American Journal of Political Science, pages 239-258.
Anna Jobin, Marcello lenca, and Ffffy Vayena. 2019. The global landscape of ai ethics guidelines. Nature Machine Intelligence, 1(9)3389-99.
Carolin Lawrence and Stefan Riezler. 2018. Improving a Neural Semantic Parser by Counterfactual Learn- ing from Human Bandit Feedback. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 1820-1830, Melbourne, Australia. Association for Computational Linguistics.
Da Ju. Jing Xu, Y.-Lan Boureau, and Jason Weston. 2022. Learning from data in the mixed adversarial non-adversarial case: Finding the helpers and ignor- ing the trolls. ar Xiiv:2208.03295 [cs).
Carolin Lawrence, Artem Sokolov, and Stefan Riezler. 2017. Counterfactual Learning from Bandit Feed- back under Deterministic Logging: : A Case Study in
Johannes Kiesel, Milad Alshomary, Nicolas Handke, Xiaoni Cai, Henning Wachsmuth, and Benno Stein.

---

Statistical Machine Translation. In Proceedings of the 2017 Conference on Empirical Methods in Natu- ral Language Processing, pages 2566-2576, Copen- hagen, Denmark. Association for Computational Lin- guistics.
Trainable Task-Oriented Dialogue Systems. In Pro- ceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Pupers), pages 2060-2069, New Orleans, Louisiana. Association for Computational Linguis- tics.
Leonard Lee, On Amir, and Dan Ariely. 2009. In search of homo economicus: Cognitive noise and the role of emotion in preference consistency. Journal of consumer research, 1827)1173-187.
Hao Liu, Carmelo Sferazza,, and Pieter Abbeel. 2023a. Chain of Hindsight Aligns Language Models with Feedback. arXiv: 2302.02676.
Jiwei Li, Alexander H. Miller. Sumit Chopra, Marc' Aurelio Ranzato, and Jason Weston. 2017a. Dialogue Learning With Human-In- InTe---oop.... arXiv:1611.09823 [cs].
Hao Liu, Carmelo Sferrazza, and Pieter Abbeel. 2023b. Languages are rewards: Chain of hindsight finetun- ing using human feedback. arXiv: 2302.02676v2 [cs.LG].
Jiwei Li, Alexander H. Miller, Sumit Chopra, Mare' Aurelio Ranzato, and Jason Weston. 2017b. Learning through Dialogue Interactions by Asking Questions. arXiv: 161.:496 [cs].
Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, Lili Wang, and Soroush Vosoughi. 2021b. Mitigating Po- litical Bias in Language Models through Reinforced Calibration. Proceedings of the AAAT Conference Artificial Intelligence, 35(17):14857 14866. Num- ber: 17.
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016. Deep Rein- forcement Learning for Dialogue Generation. In Pro- ceedings of the 2016 Conference Empirical Meth- ods in Natural Language Processing, pages 1192- 1202, Austin, Texas. Association for Computational Linguistics.
Ruibo Liu, Chenyan Jia, Ge Zhang, Ziyu Zhuang, Tony X. Liu, and Soroush Vosoughi. 2023c. Second Thoughts are Best: Learning Re-Align With Hu- man Values from Text Edits. arViv: 2305500355v..
Ruibo Liu, Ruixin Yang. Chenyan Jia, Ge Zhang, Denny Zhou, Andrew M. Dai, Diyi Yang, and Soroush Vosoughi. 2023d. Training Socially Aligned Lan- guage Models in Simulated Human Society. arXiv: 2305.16960.
Manling Li, Ying Lin, Joseph Hoover, Spencer White- head, Clare Voss, Morteza Dehghani, and Heng Ji. 2019. Multilingual Entity, Relation, Event and Hu- man Value Extraction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demon- strations), pages 110-115, Minneapolis, Minnesota. Association for Computational I Linguistics.
Ruibo Liu, Ge Zhang, Xinyu Feng, and Soroush Vosoughi. 2022. Aligning Generative Language Models with Human Values. In Findings of the Asso- ciation for Computational Linguistics: NAACL 2022, pages 241-2522 Seattle, United States. Association for Computational Linguistics.
Stephanic Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers). pages 3214-3252, Dublin, Ireland. Association for Computational Linguistics.
Nicholas Lourie, Ronan Le Bras, and Yejin Choi. 2021. SCRUPLES A Corpus of Community Ethical Judg- ments on 32,000 Real-Life Aneedotes. Proceedings of the AAAI Conference on Artificial Intelligence, 35(15): 13770-1399 Number: 15.
Ying Lin, Joe Hoover. Morteza Dehghani, Marlon Mooijman, and Heng Ji. 2017. Acquiring Back- ground Knowledge to Improve Moral Value Predic- tion. arXiv: 1709.05467v1.
Hua Lu, Siqi Bao, Huang He, Fan Wang, Hua Wu, and Haifeng Wang. 2022. Towards Boosting the Open-Domale Chatbot with Human Feedback. arxiv: 2208 141.5v1
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. 2021a. DExperts: Decoding-Tinee - Controlled Text Generation with Experts and Anti- Experts. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 Ithh International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), Online. Association for Computa- tional Linguistics.
Li Lucy and David Bamman. 2021. Gender and Repre- sentation Bias GPT-3 Generated Stories. In Pro- ceedings of the Third Workshop on Narrative Un- derstanding, pages 48-55, Virtual. Association for Computational Linguistics.
Claude Lévi-Strauss. 1987. Introduction to the work of Marcel Mauss. Routledge & Kegan Paul, London,
pages
Hotaka Maeda. 2015. Response option configuration of online administered likert scales. International Journal of Social Research Methodology. 18(1):15- 26.
Bing Liu, Gokhan Tür, Dilek Hakkani-Tür, Pararth Shah, and Larry Heck. 2018 Dialogue Learning with Human Teaching and Feedback in End-to-End

---

Tushar Maheshwari, Aishwarya N. Reganti, Samik- sha Gupta, Anupam Jamatia, Upendra Kumar, Bjöm Gambäck, and Amitava Das. 2017. A Societal Sen- timent Analysis: Predicting the Values and Ethies of Individuals by Analysing Social Media Content. In Proceedings of the 15th Conference of the Euro- pean Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 731-741, Valencia, Spain. Association for Computational Lin- guistics.
Meeting of the Association for Computational Lin- guistics and the 11th International Joins Conference Natural Language Processing (Volume 1: Long Papers), pages 5356-5371, Online. Association for Computational Linguistics.
Md Sultan AI Nahian, Spencer Frazier, Mark RiedI, and Brent Harrison. 2020. Learning norms from stories: A prior for value aligned agents. In Proceedings of the AAAI/ACM Conference on Al, Ethics, and Society, pages 124-130.
Bodhisattwa Prasad Majumder, Shuyang Li, Jianmo Ni, and Julian McAuley. 2019. Generating personalized recipes from historical user preferences. In Proceed- ings of the 2019 conference on empirical methods in natural language processing and the 9th inter- national joint conference on natural language pro- cessing (EMNLP-IJCNLP), pages 5976-5982, Hong Kong, China. Association for Computational Linguis- tics.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2021. We- bGPT: Browser-assisted man feedback. arXiv: 21 2112.09332v3.
with hu-
Nguyen, Nguyen Viet Dung Nghiem, Bao- Sinh Nguyen, Dung Tien Tien Le. Shahab Sabahi, Minh-Tien Nguyen, and Hung Le. 2022. Make The Most of Prior Data: A Solution for Interactive Text Summarization with Preference Feedback. In Find- ings of the Association for Computational Linguis- tics: NAACL 2022, pages 1919-1930, Seattle, United States. Association for Computational Linguistics.
Donald Martin Jr., Vinodkumar Prabhakaran, Jill Kuhlberg, Andrew Smart, and William S. Isaac. 2020. Participatory Problem Formulation for Fairer Ma- chine Learning Through Community Based System Dynamics. MxXiv:2005.07577 [cs, stat].
Jacob Menick. Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell- Gillingham, Geoffrey Irving, and Nat McAleese. 2022. Teaching language models to support answers with verified quotes. arXiv:2203. 147 [es].
Khanh Nguyen, Hal Daumé III, and Jordan Boyd- Graber. 2017, Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1464-1474, Copenhagen, Denmark. Associa- tion for Computational Linguistics.
Shachar Mirkin and Lua Meunier. 2015. Person- alized machine translation: Predicting translational preferences. In Proceedings the 2015 conference on empirical methods in natural language process- ing, pages 2019-2025, Lisbon, Portugal. Association for Computational Linguistics.
Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What can we learn from collective human opinions on nat- ural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2131-9143
Shachar Mirkin, Scott Nowson, Caroline Brun, and Julien Perez. 2015. Motivating Personality aware Machine Translation. In Proceedings of the 2015 Conference on Empirical Methods Natural Lan- guage Processing. pages 1102-1108, Lisbon, Portu- gal. Association for Computational Linguistics.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder. Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. arXiv: 2203.02155v1.
Nailia Mirzakhmedova, Johannes Kiesel, Milad Al- shomary. Maximilian Heinrich, Nicolas Handke, Xi- aoni Cai, Barriere Valentin, Doratossadat Dastgheib. Omid Ghahroodi, Mohammad Ali Sadraei, Ehsaned- din Asgari, Lea Kawaletz, Henning Wachsmuth, and Benno Stein. 2023. The touché23- vouché23-ValueEval ValueEval dataset for identifying human values behind argu- ments. arxiv: 13377vvv [cs.CL]]
Xiangyu Peng. Siyan Li, Spencer Frazier, and Mark Riedl. 2020. Reducing Non-Normative Text Gen- eration from Language Models. In Proceedings of the 13th International Conference on Natural Lan- guage Generation, pages 374-383, Dublin, Ireland. Association for Computational Linguistics.
Kaixiang Mo, Shuangyin Li, Yu Zhang, Jiajun Li, and Qiang Yang. 2016. Personalizing dialogue sys- tem with transfer reinforcement learning. arXiv: 1610.02891v3 [cs
Ethan Perez, Sam Ringer. Kamilé Lukošiūté, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kada- vath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christo- pher Olah, Da Yan, Daniela Amodei, Dario Amodei,
Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias pretrained language models. In Proceedings of the 59th Annual

---

Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Lan- dau, Kamal Ndousse, Landon Goldberg. Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Ellhage, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott John- ston, Shauna Kravec, Sheer EI Showk, Tamera Lan- ham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume. Yuntao Bai, Zac Hatfield- Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Gan- guli, Evan Hubinger, Nicholas Schiefer, and Jared Ka- plan. 2022. Discovering Language Model Behaviors with Model-Written Evaluations. 2rXiv:2212.0925/ [cs].
and Dataset. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics. pages 5370-5381, Florence, Italy. Association for Computational Linguistics.
Paul Röttger, Bertie Vidgen, Dirk Hovy, and Janet Pier- rehumbert. 2022. Two contrasting data annotation paradigms for subjective nip tasks. In Proceedings of the 2022 Conference of the North American Chap- of the Association for Computational Linguistics: Human Language Technologies, pages 175-190.
Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Juraf- sky, Noah A. Smith, and Yejin Choi. 2020. Social Bias Frames: Reasoning about Social and Power Im- plications of Language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477-5499 Online. Association for Computational Linguistics.
ME Peters, M Neumann, M lyyer, M Gardner, C Clark, K Lee, and L Zettlemoyer. 2018. Deep contextual- ized word representations. arxiv 2018, arXiv preprint arXiv:/802.05365. 12.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Com- monsense Reasoning about Social Interactions. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNIP)) pages 4463- 4473, Hong Kong, China. Association for Computa- tional Linguisties.
Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz 2021. On releasing annotator-leve labels and information in datasets. In Proceedings of The Joins 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 133-138.
Vjosa Preniqi, Kyriaki Kalimeri, and Charalampos Saitis. 2022. "More Than Words": Linking Music Preferences and Moral Values Through Lyries. arXiv: 2209.01169v1.
Jérémy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. 2022. Training Language Models with Language Feedback. arXiv:2204.144 14146 [cs].
Valentina Pyatkin, Jena D. Hwang, Vivek Srikumar, Ximing Lu, Liwei Jiang. Yejin Choi, and Chandra Bhagavatula. 2022. Reinforced clarification ques- tion generation with defeasibility rewards for dis- ambiguating social and moral situations. arXiv: 10409v 1 [cs.CL]]
Patrick Schramowski, Cigdem Turan, Sophie Jentzsch, Constantin Rothkopf, and Kristian Kersting, 2019. BERT has Moral Compass: Improvements of ethical and moral values of machines. arXiv: 1912.05238v1.
Olga Seminck and Pascal Amsili. 2017. A Compu- tational Model of Human Preferences for Pronoun Resolution. In Proceedings of the Student Research Workshop at the 15th Conference of the European Chapter of the Association for Computational Lin- guistics, pages 53-63, Valencia, Spain. Association for Computational Linguistics.
Liang Qiu, Yizhou Zhao, Jinchao Li, Pan Lu, Baolin Peng. Jianfeng Gao, and Song-Chun Zhu. 2021. Val- ueNet: New Dataset for Human Value Driven Dia- logue System. arXiv: 1
Ella Rabinovich, Raj Nath Patel, Shachar Mirkin, Lu- cia Specia, and Shuly Wintner. 2017. Personalized Machine Translation: Preserving Original Author Traits. In Proceedings of the 15th Conference of the European Chapter of the Association for Computa- tional Linguistics: Volume 1, Long Papers, pages 1074-1084, Valencia. Spain. Association for Compu- tational Linguistics.
Eric Michael Smith, Melissa Hall Melanie Kambadur, Eleonora Presani, and Adina Williams. 2022. "I'm sorry to hear that": Finding bias in language models with holistic descriptor dataset.
Irene Solaiman and Christy Dennison. 2021. Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets. In Advances in Neural Information Processing Systems, volume 34, pages 5861-5873. Curran Associates, Inc.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct Preference Optimization: Your Lan- guage Model is Secretly a Reward Model. arXiv: 2305.
Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. 2023. Pref- erence Ranking Optimization for Human Alignment. arXiv: 2306. 1006.1799) 17492.
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019. Towards Empathetic Open- domain Conversation Models: A New Benchmark

---

Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler. Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. Learn- ing to summarize from human feedback. arXiv: 2009.01325v3.
Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John Mellor, Lisa Anne Hen- dricks, Kirsty Anderson, Pushmeet Kohli, Ben Cop- pin, and Po-Sen Huang. 2021. Challenges in Detoxi- fying Language Models.
Yi Tay, Donovan Ong, Jie Fu, Alvin Chan, Nancy Chen, Anh Tuan Luu, and Chris Pal. 2020. Would you Rather? New Benchmark for Learning Machine Alignment with Cultural Values and Social Prefer- ences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5369-5373, Online. Association for Computational Linguistics.
J Christopher Westland. 2022. Information loss and bias in likert survey responses. Plos one, 17(7):e027 1949.
Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Sti- ennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively Summarizing Books with Human Feedback. arXiv:
Zeqiu Wu, Yushi Hu, Weijia Shi. Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2023. Fine- Grained Human Feedback Gives Better Rewards for Language Model Training. arXiv: 2306.01693.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali. Yanping Huang. Maxim Krikun, Dmitry Lepikhin. James Qin, Dehao Chen. Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung- Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny So- raker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Ale- jandra Molina. Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Co- hen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera- Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022. LaMDA: Language Models for Dialog Applications, DXXi:22201002233 [cs].
Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2021a. Bot-Adversarial Dialogue for Safe Conversational Agents. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2950 2968, Online. Association for Computational Linguistics.
Jing Xu, Da Ju. Margaret Li, Y.-Lan Boureau, Jason Weston, and Emily Dinan. 2021b. Recipes for Safety in Open-domain Chatbots. arXiv:2010007079 [es].
Jing Xu, Megan Ung, Mojtaha Komeili, Kushal Arora, Y.-Lan Bourcau, and Jason Weston. 2022. Learn- ing New Skills after Deployment: Improving open- domain internet-driven dialogue with human feed- back. arviv: 2208.03270w2.
Amos Tversky. 1969. Intransitivity of preferences. Psy- chological review, 76(1):31.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. RRHF: Rank Responses to Aliga Language Models with Human Feedback without tears. arXiv: 2304.05302.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017, Attention Is All You Need.
Saizheng Zhang, Emiil Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing Dialogue Agents: I have a dog. do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 2204-22) 2204-211.. Melbourne, Australia. Association for Computational Linguistics.
Dongqi Wang, Haoran Wei, Zhirui Zhang, Shujian Huang, Jun Xie, and Jiajun Chen. 2021. Non- Parametric Online Learning from Human Feed- back for Neural Machine Translation. arXiv: 2109. 111363..
Xin Wang. Jianan Wang, Yuanchao Liu, Xiaolong Wang. Zhuoran Wang, and Baoxun Wang. 2017. Predicting Users' Negative Feedbacks in Multi- Turn Human- Computer Dialogues. In Proceedings of the Fighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 713- 722, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Jieyu Zhao, Daniel Khashabi, Tushar Khot, Ashish Sab- harwal, and Kai-Wei Chang. 2021. Ethical-Advie Taker: Do Language Models Understand Natural Lan- guage Interventions? In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4158 4164, Online. Association for Computa- tional Linguistics.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A. Smith, Daniel Khashabi, and Han- nanch Hajishirzi. 2022. Self-Instruct: Aligning Lan- guage Model with Self Generated Instructions. arXiv: 10560vl.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer. Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zeuteeoyer. and Omer Levy. 2023. LIMA: Less Is More for Alignment, arXiv: 2305.1 1206.

---

Ruijie Zhou, Soham Deshmukh, Jeremiah Greer, and Charles Lee. 2021. NaRLE: Natural language models using reinforcement learning with emotion feedback. arXiv: 10.00.2148v [cs.CL].
C.2 Evaluating Models
Even articles employing indirect or simulated hu- man feedback usually conduct a human evalua- tion stage (Peng et al., 2020; Liu et al., 2021b, 2022). Differentty-trainned models are often com- pared via ELO scores or win rates (Ziegler et al., 2019; Nakano et al., 2021; Bai al., 2022b,a; Scheurer et al., 2022; Bakker et al., 2022; Glaese et al., 2022; Ouyang et al., 2022). Most evalua- tions include fine tne-grained questions about model outputs, including quality or usefulness (Wu et al.. 2021; Nakano et al., 2021; Liu et al., 2022; Bakker et al., 2022); political bias (Liu et al., 2021b); co- herence (Wu et al., 2021; 1; Nakano et al., 2021; Liu
Daniel Ziegler, Nisan Stiennon, Jeffrey Wu, Tom Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. 2019. Fine-Tuning Lan- guage Models from Human Preferences. arXiv: 1909.08593v2.
Caleb Ziems, Jane Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. 2022. The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 37733373, Dublin, Ireland. Associa- tion for Computational Linguistics.
et al., 2022; Bakker al., 2022); safety or harm- lessness (Xu et al., 2021a; Lu et al., 2022; Gan- guli et al., 2022; Thoppilan et al., 2022); informa- tiveness, correctness or trustworthiness (Wu et al., 2021; Nakano et al., 2021; Lu et al., 2022); cre- ativity (Honovich et al., 2022); and alignment with human value or trait (Solaiman and Dennison, 2021; Liu et al., 2022; Castricato et al., 2022; Liu et al.. 2023b).
Douglas Zytko, Pamela J. Wisniewski, Shion Guha, Eric P. S. Baumer, and Min Kyung Lee. 2022. Par- ticipatory Design AI Systems: Opportunities and Challenges Across Diverse Users, Relationships, and Application Domains. In Extended Abstracts of the 2022 CHI Conference on Human Factors Comput- ing Systems, CHI EA '22, pages 1-4, New York, NY, USA. Association for Computing Machinery.
A Flowchart of Articles for Scoping the Review
Others use automated metrics to quantitatively compare models and outputs. with Böhm et al. (2019) and Stiennon al. (2020) performing comparison of how such automated metrics cor- relate with human preferences. Metries include ROGUE (Böhm et al., 2019; Ziegler et al., 2019; Stiennon et al., 2020; Liu et al., 2022; Nguyen et al., 2022; Wang et al., 2022; Wu et al.. 2021; Liu et al., 2023c), summary length (Stiennon et al., 2020), perplexity (Liu et al., 202 lb, 2022, 2023c) or Sacre- BLEU (Wang et al., 2021). Sometimes separate discriminative classifier are deployed measure textual attributes (Thoppilan et al., 2022), such toxicity measured via Perspective API scores (So- laiman and Dennison, 2021; Arora et al., 2022). Scheurer et al. (2022) score how close feedback and refinements are in the embedding space be- cause they find written feedback often describes an "ideal" output. Any prediction tasks e.g., whether an ethical judgement is fair unfair (Jiang et al., 2022), a situation is normative or non-normative (Nahian al., 2020; Forbes al., 2020), a norm exception is permissible or not permissible (Jin et al., 2022) or an utterance is value aligned or misaligned (Qiu et al., 2021) - use F1-score or accuracy evaluation metrics.
In Fig. 1, we schematically summarise the process of selecting articles for our review.
B Code Book
We present the full code book used for each article in Tab. 3. These questions were inputted into an online form then coded by two authors of the paper, with frequent check-ins to ensure similarity of in- terpretation on how the form should be used. The first theme (conceptual) makes up our conceptual comments the main paper, while the laboural and technical themes make up our methodological comments in the main paper.
C Additional Information on Reviewed Articles
C.1 Target Tasks
In Tab. 2, we summarise the core target tasks ap- proached by each article. Reflecting the recent movement away from specialist NLP systems to- wards general purpose language agents, the major- ity of articles work with generalised models that can handle many other NLP subtasks via instruc- tion or dialogue.
Metrics or human evaluations that measure how aligned a resultant model is with human prefer-

---

Task
References
(Peng et al., 2020; Liu et al., 2021b; Solaiman und Dennison, 2021; Arora et al., 2022; Liu et al., 2022; Korbak et al., 2023), including story generation (Castricato et al., 2022) and code generation (Korbak et al., 2023)
Text generation
Instruction following
(Honovich al., 2022; Ouyang al., 2022; Wang al., 2022)
(Hancock et al., 2019; Gao et al., 2020: Jaques et al., 2020; Askell et al., 2021; Qiu et al., 2021; Thoppilan et al., 2022; Bai et al., 2022b,a; Ganguli et al., 2022; Lu et al., 2022; Xu et al., 2022; Liu et al., 2023b), including information-seeking dialogue (Glaese et al. 2022)
Open-ended dialogue
Open-book generative QA
(Zhao et al., 2021: Deng et al., 2022; Nakano et al., 2021; Menick et al., 2022)
(Gao et al., 2018; Böhm al. 2019; Ziegler et al., 2019; Stiennon et al., 2020; Scheurer ul., 2022; Nguyen et al., 2022; Liu et ul., 2023b), including long-form book summarisa- tion (Wu ct al., 2021) and opinion consensus sunemarssatin (Bakker al., 2022)
Summarization
(Dinan al., 2019; Peng et al., 2020; Liu al., 2021a; Scheurer et al., 2022; et al., 2022; Bang et al., 2022; Liu et al., 2022)
Toxic language
(Forbes al., 2020; Nahian al., 2020; Jiang et al., 2022; Liu al., 2022; Jin et al., 2022;
Moral & normative judgements
Pyatkin al., 2022; Liu al., 2023c)
Sentiment and style transfer (Ziegler et al., 2019; Peng et al., 2020; Liu et al., 2021a): recipe generation (Majumder et al., 2019); predicting intent of emails (Zhou et al., 2021); machine translation (Wang ct al., 2021)
Others
Table 2: Articles categorised by target task.
ences or values can contrasted with general in- vestigations model capabilities to estimate the so-called "alignment tax" (Liu et al., 2022). For instance, Korbak et al. (2023) rely on two met- rics: (i) misalignment score, calculated using the same automated reward functions as training (tox- icity score, number of PIl instances per character, number of PEP errors per character), and (ii) ca- pability score, calculated as the KL divergence of output distribution from a highly capable model (GPT-3). Some articles assess the drop in other performance measures on NLP benchmark tasks measuring truthfulness, toxicity or bias (Bai et al.. 2022a; Ouyang al., 2022).
or entity extraction with moral values in knowledge bases (Lin et al., 2017; Li et al., 2019); and iden- tifying human values in arguments (Kiesel et al., 2022).
Evaluate These articles include those that bench- mark judgements in moral or ethical situations (Tay et al., 2020; Hendrycks et al., 2020; Louric et al., 2021; Ziems et al., 2022; Mirzakhmedova et al., 2023); assess social biases or social reason- ing (Sap et al., 2019, 2020); evaluate performance on personality-awar dialogue (Zhang et al., 201 8) or empathetic dialogue (Rashkin et al., 2019); and detect non-human identity in conversations (Gros al., 2021). Others directly evaluate the values traits of existing models (Schramowski et al., 2019; Jentzsch et al., 2019; Perez et al., 2022).
D Articles with Other Contribution Types
the main paper, we discuss papers that seek to embed, train or align LLMs with human prefer- ences and values. Here, we give a brief overview of the other categories papers which are excluded from the main review.
Predict These articles include detecting moral content from tweets (Hoover al., 2020; Asprino al.. 2022) adapting moral shifts (Huang et al., 2022); predicting values and ethics from so- cial media content (Maheshwari et al., 2017) or mu- sic preferences (Preniqi et al., 2022); linking event

---

arXiV.CL
ACL
match
1 match
match
1 match
1844
106
3,719
81
Combined articles (187)
Duplicates
Unique
12
175
Set 0 articles (175)
Exclude
Include
118
57
Topic
Modality
Empirical
Set articles (72)
68
31
13
6
Exclude
Include
Topic
Modality
Empirical
27
Total coded articles (95)
Predict
Evaluate
Train
9
14
72
Not LLMs
LLMs
[Past]
[Present]
22
50
Figure I: Flowchart of the selection process for articles in our review. We first match on the keywords defined in Tab. 1. keeping only articles with >1 matches the title. We then de-duplicate articles posted on both arXiv and ACL. This initial set is called So. We apply the inclusion criteria listed in $2. and also add any possibly relevant references to the snowballed set (S,) regardless of publishing venue. We also apply the inclusion criteria to S,. Finally, make additional categorisations the dominant contribution type of the article (predict, evaluate train) and whether it uses LLMs.

---

METADATA
relevance
Whether to include or exclude article from set
single choice: [exclude, include]
exclusion criteria
Reason for excluding the article
single choice: [topic, modality, empirical. other]
exclusion detail
Additional summary exclusion eriteria reasoning
free-text
snowball keys
The bib keys of additional references
free-text single choice: (evaluate, predict, train, other] free-text
contribution type
Dominant contribution lype of article
contribution de-
How are the main contributions of the article described?
tail
about L.Mss
Whether the article uses I.IMs
single choice:
[yes, no]
short summary
1-3 sentence summary the article
free-text
CONCEPTUAL THEME
terminology
Is feedback discussed using the terms "prefer- ences' 'values'?
single choice: preferences, values, mix, other]
motivation
What is the motivation for feedback learning?
free-text
target concepts
Which human values or preferences are priori- free-text
tised or included?
concept defs theories
How are human values or preferences defined?
free-text
What theories any) are used to define prefer- ences/values?
free-text
concept scope
Are concepts defined universal cultur- ally/contextually understood?
free-text
interpretation freedom
What level of freedom are humans given in interpreting the in-scope target concepts, e.g., "helpfulness"
[prescriptive paradigm, subjec- tive parudigm, unclear]
single choice:
LABOURAL THEME
feedback genera-
How is feedback data collected?
multi choice: [human generated explicit, numan-gennraal implicit, model-generatee combined, other]
tion
feedback types
What forms of feedback are collected? At what stage, and for training or for evaluation?
free-text
labour documen- tation
Is labour force documented??
single choice:
[yes, no, nan]
labour details
What level of documentation what details documented?? Which buman group(s) generate the feedback?
free-text
labour force
multi choice: ferowdworkers, in-house team authors, unknown]
labour force detail
What further detail is provided who gener- ates feedback?
free-text
labour force size
How many humans are involved in feedback collection for training and/or evaluation?
free-text
TECHNICAL THEME
size
What is the size the feedback dataset for training and for evaluation?
free-text
intervention stage
When and how feedhack integrated into the model?
multi choice: Ipre-training. fine-tuning, prompt- ing, other]
metrics
What metrics and which evaluation datasets used?
free-text
model approach
Summarise the modelling methodology
free-text
PROCEDURAL THEME
authorship
Authorship composition of the article
single choice:
[academia, industry, mixed]
data availability
Whether the data artifacts are publicly available
single choice: [yes, no, unclear]
model availability
Whether the model artifacts are publicly avail- able
single choice: [yes, no, unclear]
peer review
Whether the article peer-reviewed
single choice: [yes, no, unclear]
Table 3: Code book used for each article included in the review. We show the field name, the prompt or instruction for the coder and the type response variable (including options it is single multiple choice question). Frequent communication between two coders was established to ensure the fields were being applied consistently.


Reviews:
[{"review_id": "Nq38cO6qHx", "reviewer": "Reviewer_TDYC", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed comments and support of our contributions, with no stated reasons to reject the paper. We will now comment briefly on minor points for improvement, all of which we agree with and will make changes to the camera-ready version of the paper accordingly.\n\n### Comment 1: Past vs Future section is unbalanced\nWe agree that the \"past\" section is currently more condensed; it was originally reduced in length to meet page limits. With additional space for camera ready, we can bring back in some additional detail on papers falling into this section.\n\nThe \"future\" section is longer because it also summarises key findings or methodological regularities from the review as whole, then uses these findings to make recommendations. As you suggest (see Comment 4), we will add an initial contextualisation paragraph to each section describing its purpose and scope.\n\n### Comment 2: Paragraphs with dense citations\nWe are unfortunately restricted in the citation format we can use by the EMNLP style guidelines. We agree that the extended format of citations as (Author, Year) and not numeric references ([1]) is clunky at times for a survey paper given the density of cites. In the camera-ready paper, we will devote some of the extra space where possible to expanding the actual text in paragraphs to reduce the clustering of citations.\n\n### Comment 3: Tension between survey nature and broader scope\nAs we mention in Comment 1, it is a dual aim of this paper to (1) survey the existing literature, (2) then draw on regularities, findings and critiques of this survey to also provide recommendations for future work. We will make sure the introduction appropriately communicates the intended scope, purpose and aims of the paper.\n\n### Comment 4: Initial contextualisation paragraphs at the start of each section\nWe thank the reviewer for this suggestion. We will certainly implement this in the camera-ready version as it will improve the paper and help to resolve Comment 1.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The process of paper selection is explained very clearly and transparently.\n* The final pool is representative yet reasonably limited, indicating that each paper was given appropriate attention. The content of the survey supports this fashion as well.\n* The survey demonstrates a good balance between methodological and technical discussion, comprehensively covering the topic concisely.\n* The \"Limitations\" section is interesting on its own, adding depth to the work. The points expressed hold validity generally for surveys, extending beyond the context of this specific study.", "reasons_to_reject": "The following are minor points and not actual reasons to reject.\n* The sections \"past,\" \"present,\" and \"future\" are not very balanced. Specifically, the \"past\" section appears to be too condensed, while the \"future\" section is overly extensive considering the \"survey\" nature of the work.\n* The paper often becomes more engaging in sections less tied to the survey (e.g., the \"future\" part). In the initial pages, some paragraphs contain too many citations - in extended format - that the actual text consists of brief, disjointed sentences (particularly on pages 4 and 5). Overall, there is tension between the survey nature and the aspiration for a broader scope.\n* Each of the three sections would benefit from at least an initial contextualization paragraph before delving into the actual discussion (especially the \"past\" and \"present\" sections).", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper is a survey/review focusing on the feedback mechanisms between humans and LMs. The analyzed papers are divided into the past (2014-2019) and the present (2019-today). The work primarily centers on the latter, initially examining how LLMs can better align with an optimum based on users' values and preferences. A relatively significant portion of the article delves into the conceptual difference between preference and value. Subsequently, it discusses current techniques for gathering direct and indirect feedback data, as well as the challenges associated with the lack of universality in values and preferences among users, resulting in noise in the inputs to LLMs. A final section addresses the issue of crowd workers generating feedback, often in limited numbers and carrying human biases that can prove to be unrepresentative and distorted relative to the population of end users.", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9jqEHz3vuO", "reviewer": "Reviewer_NxJU", "summary": "", "questions": "A. What do you mean by “Input” keywords and “output” keywords in Table 1? I could not find explanations in Section 2, nor in its caption.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their support of our contributions and are encouraged that they see no reasons to reject the paper! We will now respond to each of their questions and minor comments.\n\n### Comment 1: Input and Output Keywords\nWe agree that this was confusing terminology! We intended \"input\" column to correspond to the raw, unprocessed keywords (e.g. \"alignment\") and the \"output\" column to correspond to the stemmed, processed keyword (e.g. \"align\"). In some cases, keywords are unaltered (e.g. \"value\" is not stemmed to \"valu\") because stemmed versions return too many false positive matches. We will update the table terminology to describe this better and make sure any column headers are clearly explained in the caption. \n\n### Comment 2: Typos, Grammar, Style and Presentation Improvements\nThanks for these suggestions. We will correct the flipped double quotes. Regarding the Conclusions section, we will either (i) consider renaming it as \"Discussions and Conclusions\" to account for the wider scope; or (ii) find a way to rework the new comments on peer review and artefact availability earlier in the paper so they are not freshly introduced in the conclusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper presents a thorough and well-organized literature review.\n- It effectively highlights the important challenges in incorporating human feedback into LLMs, raising awareness of the subjectivity and incompleteness of human preferences and values.\n", "reasons_to_reject": "- Not much I can think of", "questions_to_authors": "A. What do you mean by “Input” keywords and “output” keywords in Table 1? I could not find explanations in Section 2, nor in its caption.", "paper_topic_and_main_contributions": "This survey and partially position paper offers a comprehensive overview of incorporating human feedback into LLMs, drawing from 95 papers. The paper focuses on the subjectivity of human feedback, encompassing aspects like definitions of human preference and values, annotation processes, and annotator distributions.\n\nThe primary contributions of this paper are twofold. First, through a thorough literature review, it provides a well-organized and insightful analysis of past and current research on how NLP leverages human feedback to enhance language technologies. Second, the paper highlights five crucial challenges and presents recommendations that can serve as valuable guidance for future research in this emerging field. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- There are flipped double quotes in line 65.\n- Some parts in the Conclusion (lines 734-740) were not discussed in the paper but were first introduced in the conclusion section. This feels a little awkward, as the title of the section does not include \"Discussions.\"\n"}}, {"review_id": "gyLhhgFGBJ", "reviewer": "Reviewer_wQJD", "summary": "", "questions": "1. In lines 651-655, authors argue that \"researchers should consider techniques to model the noise and distribution of human feedback (...) rather than ignoring disagreement by aggregating responses\"; soon thereafter, however, in lines 676-678, authors also warn that \"[the lack of diversity within crowdworkers] results in a non-democratic feedback process\"; my question is: if data sources remain non-representative of the population at large, wouldn't a finer-grained modeling of their feedback increase bias? In other words, finer-grained modeling is only a possibility *after* representativeness is properly addressed. For clarity, this dependency might be worth mentioning.\n\n2. In lines 129-134, authors conceptualize LLMs as \"any decoder-only or encoder-decoder model (...)\" but then introduce BERT as \"among the first LLMs.\" Considering that BERT is encoder-only, the conceptualization should be modified to accommodate that.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed comments and areas for improvement. We are encouraged that they find the topic to be extremely relevant and fast-evolving, and see no good reasons for rejecting the paper! We will respond to questions and minor comments.\n\n### Comment 1: Tension between granularity and representativeness of feedback\nThis is an insightful comment, thank you. We agree that there is a tension between the granularity of feedback and the amount of noise or bias. In this case, it may be that more granular feedback exacerbates the issues with a non-representative sample. We will add a short discussion of this to the paper, particularly in line with your comment that finer-grained modeling may only be beneficial after representativeness is properly addressed.\n\n### Comment 2: LLM definition\nThank you for spotting this inconsistency in our definition. We will modify the definition to accommodate encoder-only models like BERT.\n\n### Comment 3: Typos, Grammar, Style and Presentation Improvements\nThanks for noticing the typo, we will fix that. We will also check for places where we can unify our terminology from language models to LLMs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The review is well-written, clearly organized, and based on a thorough selection of papers. The topic is extremely relevant and fast-evolving, which motivates a high-quality snapshot of the current literature. In addition, authors are able to carry out a thoughtful discussion from the selected papers. Overall, this paper will likely help the community in analyzing the progress made in an important and timely topic.", "reasons_to_reject": "I don't see good reasons for rejecting this paper. However, I do share questions and potential improvement notes in the fields below. ", "questions_to_authors": "1. In lines 651-655, authors argue that \"researchers should consider techniques to model the noise and distribution of human feedback (...) rather than ignoring disagreement by aggregating responses\"; soon thereafter, however, in lines 676-678, authors also warn that \"[the lack of diversity within crowdworkers] results in a non-democratic feedback process\"; my question is: if data sources remain non-representative of the population at large, wouldn't a finer-grained modeling of their feedback increase bias? In other words, finer-grained modeling is only a possibility *after* representativeness is properly addressed. For clarity, this dependency might be worth mentioning.\n\n2. In lines 129-134, authors conceptualize LLMs as \"any decoder-only or encoder-decoder model (...)\" but then introduce BERT as \"among the first LLMs.\" Considering that BERT is encoder-only, the conceptualization should be modified to accommodate that.", "paper_topic_and_main_contributions": "This paper conducts a review of 95 papers from either ACL or arXiv (cut-off in mid February of 2023) in which LLMs are aligned with subjective human preferences and values. The review is centered on a few axes, most importantly: before and after LLMs (\"the past\" and \"the present\"), as well as future perspectives; how feedback is collected and who it is collected from; and the different ways in which feedback has been utilized. The discussion revolves around five challenges identified across the selected literature, namely: non-universality of preferences and values; inconsistent conceptualization; feedback incompleteness; feedback operationalization; and representativeness of feedback sources. ", "excitement": 4, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "In agreement with current methodology for literature selection.", "Typos_Grammar_Style_and_Presentation_Improvements": "On line 472: \"backpropagated\" instead of \"backprogated.\"\n\nThroughout the paper, there are a few opportunities to normalize mentions to \"language models\" to \"LLMs.\""}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is a survey on how human feedback is used for learning in large language models. The authors present a literature review as well as a set of challenges and recommendations for the future. The paper seems like a good fit for the theme track.

Pros:
* The literature review is thorough and the authors clearly describe their process for the review, including selection of papers, etc.
* The paper is well-organized and the challenges and recommendations mentioned can be useful to other researchers

Cons:
* The reviewers mention a few minor points related to clarity and wording, which can be improved for the camera-ready
Decision: Accept-Main