Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits
Authors: Ruibo Liu
Chenyan Jia
Ge Zhang
Ziyu Zhuang
Tony X Liu
Soroush Vosoughi
Venue: NeurIPS 2022 Accept (2022)
Keywords: human values
ai safety
alignment
social impact
human-AI interaction

Abstract: We present Second Thoughts, a new learning paradigm that enables language models (LMs) to re-align with human values. By modeling the chain-of-edits between value-unaligned and value-aligned text, with LM fine-tuning and additional refinement through reinforcement learning, Second Thoughts not only achieves superior performance in three value alignment benchmark datasets but also shows strong human-value transfer learning ability in few-shot scenarios. The generated editing steps also offer better interpretability and ease for interactive error correction. Extensive human evaluations further confirm its effectiveness.

Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits
Ruibo Liu', Chenyan Jia", Ge Zhang 3,4
Ziyu Zhuang", Tony X. Liu", Soroush Vosoughi Dartmouth College, "Stanford University, $ Beijing Academy of Artificial Intelligence, "University of Michigan, Ann Arbor "(ruibo.liu. soroush. vossoughi)ddarmmmottt
Abstract
We present SECOND THOUGHTS, new learning paradigm that enables language models (LMs) re-align with human values. By modeling the chain-of-edits between value-unaligned and value-aligned text, with LM fine-tuning and addi- tional refinement through reinforcement learning, SECOND THOUGHTS not only achieves superior performance in three value alignment benchmark datasets but also shows strong human-value transfer learning ability few-shot scenarios. The generated editing steps also offer better interpretability and ease for interactive error correction. Extensive human evaluations further confirm its effectiveness.
1 Introduction
"Machines can and will make better decisions than humans but only when the values are aligned with those of human ruce."
Prof. Stuart Russell, Value Alignment, 2015
Current large-scar scale pre-trained language mod- els (LMs) have shown great success in many knowlldge-recalling tasks, such as question an- swering (Talmor al., 2022) and entity re- trieval (Cao et al., 202 1); however, their ability to select socially good text from bad (or generat- ing prosocial text) in open-world settings still limited (Hendrycks et al., 2021a), even when the models are scaled up to hundreds billions of parameters (Lin et al., 2021). In other words, pre tre-training ever-larger LMs does not lead to expected substantive gains in tasks that require human value judgment (Hoffmann et al., 2022).
Context
Sally found beautiful bracelel mom's drawer while looking something else.
Response Ommmrral Option)
Pr= 0.53 -/53-/
Human
Sally takes the bracelet without asking.
Response (Moral Option) Sally aaks about the bracelet and requests her upooming birthday.
Pr 0.47 X
Chain-of- of--ddtts
@ deleting takeet
replacingg the bescelet without anlong" with books her mom. reporcinghnomts reawer while looking for something eize with she can bonrow the beaderee
Re-Aligsed
Edited Resporse
Sally her she borrow bracelet.
Delivered
Figure I: Fine-tuned language models (LMs) still tend generate text violating human values in certain contexts. Our method enables LMs to re- align with human values by making text edits.
Consider the example in Figure 1: given con- text, a fine-tuned LM GPT-2 (Radford et al., 2019) assigns a larger probability mass? to the immoral option than to the moral ground truth.
Work done during the internship at Dartmouth College.
Wee take the log- bog-probability predicted by the LM. log Pr(y|æ:,, which is the conditional log bog-probability of generating option y given input context We then compute its exponential for better readability. Such protocol is also adopted by BIG-Bench ht tptp:/// / /ttgttttbbb/uue//////// com/
com/google wow.goooll///ooooo LeBoG bench.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

One interpretation of this failure is that the commonly used "missing token prediction" objective for pre-training (i.e., MLE) does not directly model human values (Ouyang et al., 2022). As a consequence, fine-tuned LMs still struggle with options that are legitimate semantically (i.e., low language modeling loss) but are not aligned with human values.
To tackle this misalignment problem, prior work has proposed using binary answers (Jiang et al., 2021; Sap et al., 2020), rankings (Forbes et al., 2020; Brown et al., 2019), or ratings (Ziems et al., 2022: Lourie et al., 2020) to model human value preferences. For example, Askell et al. (Askell et al., 2021) create platform to collect Likert-scale human ratings on LM- -egeerrrte utterances in dialogues, aiming to teach the LM to be helpful, honest, and harmless. However, without considering how to recover from responses that already violate human values, these methods cannot serve as robust remedies in real-world applications, since they can be easily attacked by poisoned queries (Gehman et al., 2020).
More recent attempts, such as InstructGPT (Quyang et al., 2022), formulate the alignment problem as about teaching the machine to follow human instructions- they fine-tune GPT-3 on variety of prompts written by human users of OpenAl's GPT-3 API (Brown et al., 2020). Though it indeed has the ability to revise its previous language generations, such ability relies on receiving specific human instructions (e.g.. "Please make the following sentence aligned with moral values.") "). Manually designing proper prompts that can trigger value alignment requires extra human labor. Besides, ceeificalll-essinned prompts do not always exist in real-world human-Al Al interaction, and we cannot expect most users to know how to design appropriate prompts to improve the human-value alignment of an Al agent (Li & Liang, 2021).
On the other hand, rather than steering the language generation with artificial prompts, humans can easily fix immoral language by making hierarchical and recursive edits (Du et al.. 2022; Lee et al., 2022), where human value judgments serve as the guide for each edit. Following this obser- vation, in this work, we propose to leverage text edits to model human values. Our method, called SECOND THOUGIITS, echoes the theory of "utilitarian ethics", which says that humans choose the actions (e.g. edits) which maximize the perceived positive impact on the most people (Van Staveren, 2007; Quinton, 1973). Specifically, we model human edits by three generic operations: insert, delete, and replace, and automatically infer the "chain-off-ce of-edits" by a dynamic programming algorithm. Besides the commonly used MIE training, we deliberately include a reinforcement learning based refinement step, to further encourage valid edits which are not only aligned with human values, but also coherent with the context.
The main contribution of this work is to present a new learning paradigm that can make current LMs aware of the human value alignment. Trained with SECOND THOUGIITS, LMs can not only re-align their generation with human values, even when the context has already been poisoned, but also show the chain of editing steps for ease of interpretability and to facilitate further edits ($4.5). Through extensive human evaluation, we find that the edited responses by SECOND THOUGHTS (based on a 345M GPT-2) are on average scored higher with respect to their value alignment than those from InstructGPT (based on 1.3B GPT-3) ($4.2). Our experiments confirm that simply scaling LMs is not adequate for good alignment with human values, which echoes the findings of recent studies (Perez et al., 2022; Lin et al., 2021). Instead, smaller LMs trained with a few properly decomposed human demonstrations can often lead to better results ($4.4). We also provide a discussion on the impact of human factors during human evaluation ($5), which is crucially ignored in current Al studies.
2 Related Work
We briefly review existing work that considers in-context explanations during prompting or training. We also summarize other value alignment methods for language models.
Learning From In-Context Instructions. The few-shot performance of LMs can be enhanced by learning from in-context instructions (Sanh et al., 2021; Liu et al., 2021b), in the forms of task descriptions (Mishra al., 2021; Raffel et al., 2019), answer demonstrations (Brown et al., 2020), targeting formats (Marasović et al., 2021), etc., which can be positioned before (Wei et al., 2022) or even after (Lampinen et al., 2022) the answer. Recent studies have shown improved results by including decomposed reasoning steps into the instructions (Nye et al., 2021; Narang et al., 2020). However, the instructions normally require careful human design, which is costly and whose quality greatly affects performance (Zhao et al., 2021; Holtzman et al., 2021). In comparison with these

---

methods, SECOND THOUGHTS learns from text edits inferred by an algorithm, and presents the chain-of-edits for each alignment, which eases error diagnosis and enables interactive correction.
Human Value Alignment for Language Models. Trained on unfiltered and problematic language from the web, current large-scale LMs have be shown to be poorly aligned with human values (Bom- masani et al., 2021). For example, GPT-3 performs only marginally better than a random baseline on virtue matching task (Weidinger et al., 2021), and scaling-up LMs can even lead deterioration in truthfulness (Lin et al., 2021). Existing general-purpose remedies include filtering the training data (Gururangan al., 2020), sttribute-contro) generation (Dathathri et al., 2020; Keskar et al., 2019; Ma et al., 2020), and modifying the decoding algorithm with hard (e.g., token blocklists; Schick et al. (Schick et al., 2021)) or soft constraints (e.g.. reference LMs; Liu et al. (Liu al.. 2021a)). Though these methods are able to steer generation towards prosocial directions, our experiments show that they have limited performance when the context has already been poisoned. There are other approaches that require training with specific forms of human supervision (e.g.. fine-grained ratings) (Ouyang al., 2022; Stiennon al., 2020; Ziegler et al., 2019; Christiano et al., 2017), but these are often costly and not always available in every value alignment dataset. SECOND THOUGITTS differs from all these methods in its offline nature and ability to re-align in poisoned contexts, requiring neither extra human labeling nor specially-designed prompts or instructions.
3 Approach
SECOND THOUGHTS comprises two main steps. We first infer chain-of-edits automatically from source and target responses with dynamic programming algorithm, and fine-tune an LM on the edits- augmented training data ($3.2). Then, we deploy reinforce learning stage to refine the generation, by either adversarial imitation learning or value modeling ($3.3). We begin by introducing the problem of value re-alignment ($3.1).
3.1
Problem Statement of Re-alignment
Value alignment datasets normally consist of contexts (i.c., social situations), value-aligned responses (i.e., prosocial behaviors), and value- unaligned responses (i.c., antisocial behaviors). Existing alignment methods formulate the value aligmment task as a conditional generation prob- lem: given situation the context, train model that can generate responses resembling value-aligned largel rather than a not-aligned wrong targel (Figure 2 (a)). However, many studies have shown that LMS trained with such paradigm can be easily derailed by poisoned contexts (Ouyang et al., 2022: Gehman al., 2020>iic., contexts that already include value- unaligned content, either from the model's own generation or from malicious users'. In other words, unlike humans, these models lack the
(a)
Context
Wrong Target
Target
(b)
Source
DP Inferred Chain-of-Ediin
Target
Context
not-value aligned text
value-aligned text value-aligned but incoherent text context
Edits Operations: @ insert @ delete Creplace
Figure 2: (a) Existing learning paradigm trains in vanilla text-to-text form; (b) SECOND THOUGIITS learns to re-align with decomposed chain-of-edits. ability of re-alignment (the ability to recover from poisoned contexts).
To teach model how to re-align, we deliberately add the value-unaligned response into the context, referred as the source, and keep the aalue-aligned response as the target. The intuition behind this is that instead of learning from mistakes after misalignment occurs in the generation, the model learns how to make edits as it is generating the text. Specifically, we include the unaligned source as part of the new "context", and then train an LM to learn how to make sequential edits on the source to produce the target (Figure 2 (b)). This way the model learns how to recover from a value-unate -ulle-unaligned, poisoned context during the generation phase.
3 As an example, it has been reported that Microsoft's chatbot Cortana will "get mad" if the user starts saying offensive things (Insider, 2016). Similar outcomes have been observed in Apple's Siri (BusinessInsider. 2018).
3

---

3.2 Augmented Edits Modeling
DP-based Edits Inference. Given two text strings, source and larget, one can find unlimited ways to edit source produce larget. Thus, we apply constraints onto the editing: (1) the edits should be combinations of generic editing operations-inserring deleting, and replacing a single token; (2) each edit operation has cost and our goal is infer the chain-of-edits that has minimum cost. Under these constraints, the edits inference problem can be converted to a token-level "edit distance problem" (Jurafsky, 2000), which can be solved by dynamic programming (DP). We modify the algorithm to be able to receive customized editing costs (e.g., insert-l, delete-1, replace-2), to try to model different preferences on editing. We use special tokens mark the start/end of editing and the new content to be inserted/replaaee, and develop a decipher module that can translate the edit operations produced by DP into natural language (see $A.1 for a visualization of the whole process, and $A.3 for more discussion on edit based models).
Augmented Edits Modeling (AEM). To augment the edits, we run the DP algorithm on the same source and target pairs with a variety of editing costs" to create a collection of chain-of-edie for each pource-target pair, which we call positive demonstrations (y" ). We then fine-tune an LM on these snurce-edits-targe text inputs (recall that the edits are turned into natural language). We call this Augmented Edits Modeling (AEM). Different from common language modeling, AEM includes the labor-free decomposition (i.e., the editing steps) into the training object, whereas prior works decomposition (Ouyang al., 2022; Wang et al., 2022) or,
either train on costly manually-createe rather than training. prompt with such decomposition (Wei et al., 2022; Nye et al., 2021). We also construct negative demonstrations (y ) by using the targets from other contexts, leading to inferred chain-of-edits that generate value-aligned responses which are incoherent with the given context. These will be used during the RI. refinement described below.
3.3 Refinement by Reinforcement Learning
Though the generation of an LM trained with AEM can already align well with human values, many of the generated responses are not coherent with the given contexts. Based on manual examination, the responses tend to be generic, rather than specific to the context (e.g., the sidestep error in Table A9). We are thus motivated to deploy reinforcement learning (RL) stage to further refine the generation quality, mainly to improve the coherence to the context.
Notation.
Given the concatenation of context and source as SECOND THOUGHTS will generate chain-of-edits and corresponding targer as y. In RL language. we define the state at time as the set of generated tokens before t (i.e., 8, = V<c), and the action as the current step's ' output token (i.e., a, yt). The softmax output of the language modeling head (a categorical distribution over the entire vocabulary) is considered as the policy for picking token Y/ (action oz), given the state 8, Yet-
Adversarial Imitation Learning (AIL). Inspired by the concept of imitation learning in RL, which clones the behavior of positive demonstrations (Le et al., 2018 2018), we propose to leverage negative samples to penalize the LM for imitating the mismatched target (i.e., value-aligned but incoherent). ADV. We train an adversarial LM only on the negative demonstrations y so that following its policy mi will lead to incoherent generations. The t-th step objective of AIL to be maximized is:
JAIL, = By-n; [- log n. ADV. (a,1s:) + a log * (a.ls:)] (T,II()
(1)
unlikelihond
likelihord
where n. is the desired refinement policy vector initialized from the original TT), a is the balancing factor, and the KL penalty term KL(TT||TT with the coefficient 3 is the trust region constraint, which prevents the updated policy from drifting too far away from the original one (Schulman et al., 2017, 2015)². The intuition behind such a design is to maximize the unlikelihood of forming the trajectory
T ($1, Qi...., St, } that can be induced by the adversarial policy - ADV. weighted against the balancing likelihood term (Welleck et al., 2020). After refinement, the learned policy nे can generate
+
'We use costs settings for insert, delete, and replace (1,1,1), (1.1,2), (1,2,1), (2,1.1), (1,2,3).
We choose @ 0.02 for stable training in most cases. Choosing the proper a is discussed in $4.6

---

tokens unlike those that can be produced by π ADW, wwhich will form sequences more coherent to the context.
Value Modeling (VM). In addition to AlI. which aligns values by learning from negative demon- strations, we present another refinement method that directly learns value function. To this end, we train a binary LM-based classifier ff on the mixture of positive and negative demonstrations. We use estimate the likelihood of a given generation being coherent with the context, by passing it concatenation of the context, source, generated chain-of-edits, and the corresponding generated target. We take the sigmoid of the oogliielihood predicted by as the reward r, which is a log (zz y). and define the objective to be maximized as:
JVM.E Byy-,, Te (a)l))))
<x(a)(s)) re + XH((118))xx*
where the t-th step r is adjusted by an importance- policy for off-po stability (Munos et al., 2016)". We also deliberately add an entropy bonus term H((-ss))-.- of the refined policy, discounted by X, to encourage more exploration of the current policy (Haarnoja et al., 2018)". Compared with All., VM leverages an explicit value estimation module f as the guidance, rather than implicitly learning from imitation, which brings extra benefits in generalization across different human values (detailed in $4.4).
ratio between the current and original
4 Experiments
4.1 Experimental Setting
We study the value aligmment performance of SECOND THOUGHTS on three benchmark datasets:
Moral Stories. The Moral Stories dataset (N 20, 000) examines whether LMS can generate moral responses under diverse social situations (Emelin al., 2021). We use the "situation" of each data sample as context, and treat "immoral actions" as the source, while "moral actions" as the target.
The MIC dataset (N 38, 000) studies whether chatbots can generate utterances that are aligned with a set "Rules of Thumb (RoTY" of morality (Ziems et al., 2022). Each sample is labeled with its alignment level (e.g.. "aligned", "unaligned", "neither"), RoT violation severity (from to 5), RoT agreement, etc. We take the question in the dialogue as the context, and the unaligned answers (with RoT violation severity 4-horrible or 5-worse) the source, and aligned answers the target.
MIC.
CTTHICS-Deontolon The ETHICS dataset (N 25, 356) investigates the performance of LMs on five human values alignment tasks (Hendrycks et al., 2021a). We pick the deontology split because of its contextual nature. The contexts are requests common in everyday life, while the responses are excuses that are either aligned with deontology or not. We take the requests as the context, deontology-unaligned responses as the source, and deontology-aligned responses as the target.
We also consider two smaller-scale human values alignment datasets: HHH (Helpful, Honest, & Harmless) (Askell et al., 2021) (N 178) and Truthful QA (Lin et al., 2021) (N 299), to evaluate the domain transfer ability.
We use the official train/validated/ removed hashtags and urls in the text, but leave punctuation and stop words. Besides the generative LM (GPT-2 medium) et al., 2019) on the mixture of positive and negative demonstrations on the above three datasets, achieving Fl scores of (99.7, 91.0, 91.9), respectively. They are used as in the VM mode of SECOND THOUGHTS. We run experiments on four NVIDIA A6000 GPUs, which take around (3h, 2.4h, 1.3h) J for three tasks.
splits in the above datasets. As the pre- pre-processing step,
throughout the paper, train three RoBERTa-large classifiers (Liu
We conducted two sessions of human evaluation on Amazon Mechanical Turk (MTurk). The first session was to validate the quality of SECOND THOUGHTS re-alignment, and the second session
'The t-th step reward can be estimated by unfolding the reward of the whole trajectory r into each step with a Σ'., T'r, (L is the sequence length). We calculate the entropy as H({-ss))-+* Ea,ea Te (a log (s, where A is the whole action space (the whole vocabulary). We discuss how choose the proper in $4.6
discounting factor 7 (=0.95 in our settings), which has the relationship
5

---

Table I: Results on three human value alignment tasks. We report mean and standard deviation of alignment and coherence scores of the edited responses in terms of human evaluations (both scored from 1-worst to 7-best). SECOND THOUGHTS achieves the best alignment performance compared with five baselines and two huge LM-based API services. We bold the best performing and underline the second best results.
Moral Stories
MIC
ETHICS-Deontologs
Method
Alignment
Coherence
Alignment
Coherence
Alignment
Coherence
MLE
2,48 1.47
2.96 1.74
2.88 1.69
3,89 1.67
2.11 1.75
4.02 1.82
Data Filtering
2.70 186
2.54
1.87
2.51 1.70
3.35
1.75
3.90 1.48
4.93
1.20
Safe Beam Search
3.08 1.75
3.23
1.77
2.90 1.61
3.50
1.67
2.66 1.61
3.35 1.70
PPLM
2.29 1.69
3.72 1.94
3.18 1.57
4.06
1.70
3.97 1.54
4.88
1.39
DExperts
4.47 1.69
4.40
1.71
4.68 1.33
4.78
1.37
4.30 1.60
3.91 1.73
SECOND THOUGHTS
AEM VM
4.85 1.65
5.26 1.48
5.48 1.37
5,88 1.24
5,57 1.18
6.03 0.98
AEM + AIL
4.55 1.53
5.13 1.44
5.40 1.46
5.99
0.99
5.04 1.41
5.47 1.35
AEM Only
3.80 1.71
4.37 1.78
4.87 1.47
5.47
1.33
3.86 1.48
4.98 1.42
Huge LM service
GPT-3 (175B)
3,28
1.92
3.96 1.89
3.02 1.56
3.76 1.64
2.96 1.49
4.19 1.67
InstructGPT (1.3B)
4.20
1.54
4.89
1.60
3.92
1.65
4.80
1.58
3.06 1.40
4.34 1.54
to evaluate cases where corrective edits were made by humans to the DP-generated chain-of-edits to improve alignment or coherence. We recruited 297 and 100 participants for the two sessions, respectively, and each individual was randomly assigned to evaluate the three alignment tasks. The test-set samples edited by different methods were randomly assigned to each participant without telling them the actual method name. Each participant was paid I dollar for completing 20 questions for session one ($4.2), and 0.75 dollars for 15 questions for session two ($4.5). The average completion time per session was 5m 3s and 4m 49s, respectively. The demographic information and detailed setup procedure can found in $A.5.
4.2 Main Results on the Performance of Value Alignment
Alignment methods should be able to guide text generation towards being more value-aligned, while not compromising the texts' coherence with the given context. Considering the human nature of value judgement, we conduct extensive human evaluations to measure:
Alignment, by asking "To what extent does the edited response improve the original response in terms of alignment with human values?" Answers range from 1-not at all. to 7-to an extreme extent. This measures the alignment improvement after the response edited.
Coherence, by asking "How coherent is the ediled response with the given context?" Answers range from 1-not at all. to 7-extremely coherent. This measures the coherence level given the context after the response is edited.
Besides human evaluations, we also report evaluation results by automated metrics such perplexity and ROUGE-L (Lin, 2004), and their correlation with human judgements (see $4.3).
In Table 1 we show the comparison between SECOND THOUGHTS and seven other alignment methods that do not require extra human labeling on the benchmark datasets: (1) MLE fine-tunes with all the data in the alignment datasets, simulating common LM pre-training (2) Data Filtering (Gururangan et al., 2020) only fine-tunes with the value-aligned split of the data (3) Safe Beam Search (Schick et al., 2021) blocks a list of sensitive tokens that can lead to misalignment in human values during beam search decoding (4) PPLM (Dathathri et al., 2020) steers the generation via soft probability
constraints from Bag-of- Bfg-of-Words instead of hard blocking on tokens 9 (5) DExperts (Liu et al., 2021a)
'Specifically, we use the Fightin' words algorithm (Monroe et al., 2008) to mine salient words from the unaligned demonstrations the tokens in the blocklist (https: thub. com/ com/jnhessel// /F ggghnggordss.. Forr fair comparison, use the same Fightin' words algorithm as Safe Beam Search to mine salient words from aligned demonstrations the Bag-of of Words supervision for PPLM.

---

calibrates token distribution by referring to two LMs trained on solely aligned and unaligned data. We also consider two huge LM-based API services to explore whether scaling can make gains for human value alignment: (6) GPT-3 (Brown et al., 2020) (175B) is parpppe- foundation model (Bommasani et al., 2021) which shows strong zero-shot performance in many tasks, and (7) InstructGPT (Ouyang et al., 2022), which fine-tunes GPT-3 (1.3B) on human-crafted prompts with a divergence controlled PPO algorithm (Schulman et al., 2017) named PPO-ptx, which is our closest competitor. Except for InstructGPT and GPT-3, we run all other baselines with GPT-2 medium (340M) for consistency. The exact prompts and instructions used for evaluation are described in $A.2.
Results shows that SECOND THOUGHTS outperforms other methods in both alignment and coher- ence as evaluated by human judgement, especially when using AEM VM. MLE shows limited performance since it has no scheme to be aware of human values. Data Filtering shows small improvement over MI.E as clones the aligned data behavior, but still limited when the context already includes unaligned content. Token-constrained decoding methods such as Safe Beam Search and PPL.M struggle with value alignment presumably because the abstract human values cannot be easily modeled by a set of tokens. DExperts makes gains in alignment but the coherence of its edited responses is mostly compromised, mainly due to its token-level control. Compared with AEM AIL, AEM + VM has superior performance in most cases; one interpretation could be that the value modeling provides better generalization ability, while simply imitating the aligned data can lead to accumulated off- off-track emrors in unseen contexts (Codevilla et al., 2019). Despite being built on the same LM with far fewer parameters, edits from InstructGPT (1.3B GPT-3) are rated
consistently higher than those from vanilla GPT-3 (175B)" 10 Moreover, SECOND THOUGHTS further outperforms InstructGPT significantly according to one way analysis of variance (ANOVA) post-hoc pairwise comparisons (p <0.05) when refined with an RL stage (+ VM or + AlL). One reason could be that aligning with human values using InstructGPT may require extensive prompt engineering. In general, we conclude that proper value judgement cannot be simply achieved by enlarged model ca- pacity (Hendrycks et al., 2021b), and smaller LMs trained with properly decomposed demonstrations can often lead to better alignment results.
4.3 Correlation Between Automated Metrics and Human Judgement
Although we believe that humans should be the only qualified judges for the value alignment task, during the development stage of algorithms we have to leverage fast and cheap automated metrics as a reasonable estimation. Here, we test the correlation between two automated metries (ROUGE- L and perplexity (PPL)) and respective human judgements on Alignment and Fluency. Table 2 shows additional results on the three alignment datasets. Besides the Alignment (Align) score, we also report Fluency score from human evaluation, and two automated metries ROUGE-L and perplexity as automated alternatives of human scored Alignment and Fluency, respectively. We also show the correlation (Pearson's 's r)) between the automated metrics and human judgements. We find that perplexity has a high correlation with the human rated Fluency score across the tasks, while ROUGE-L's correlation is more task-dependent, though all correlations are statistically significant. One interpretation could be that the measurement text similarity with the ground truth (i.e., what ROUGE-L measures) is only approximation of value alignment. However, the high variance in the value judgement among humans cold also be a factor. We have studied the impact from human factors on the Alignment score in $5. This impact may partially explain the variance in the human value judgements.
4.4 Value Transfer Learning with Limited Human-Labeled Data
Since data labeled with human values is rather costly and scarce, we explore whether the alignment learned on one value- value-alignment task can be transferred to another, aiming to investigate the gener- alization ability of SECOND THOUGHTS benchmark datasets (MRL, MIC, and ETC), recording checkpoints periodically, and then we evaluate these checkpoints two value alignment datasets (TQA and HHH). We include an additional version of SECOND THOUGHTS which does not include chain-of-edits (i.e., vanilla text-to-text (T2T)) to demonstrate the effectiveness of chain-of-edits decomposition for domain transferability.
unseen values. We first train model on the three
"eere, we basically replicate similar findings in the InstructGPT paper (see page 3), though via human evaluation different alignment datasets.
7

---

Table 2: Additional results on the three alignment datasets. Besides the Alignment (Align) score, we also report Flueney score from human evaluation, and two automated metries ROUGE-L (R-L) and perplexity (PPL) as automated alternatives of human scored Alignment and Fluency, respectively. Note that for PPL it is the lower the better. We also show the correlation (Pearson's r) between the automated metrics and human judgements.
Moral Stories
MIC
Ethics
Method
Align
R-L
Flueney
PPL4
Align
R-L
Flueney
PPL4
Align
R-L
Fluency
PPLI
MLE
2.48
7.96
4.54
8.26
2.88
9.62
5.17
12.18
2.11
17.32
5.57
5.23
Data Filtering
2.70
13.32
4.43
7.94
2.51
14.31
4.74
14.43
3.90
23.60
5.58
5.10
Safe Beam Search
3.08
18.48
4.02
19.50
2.90
12.55
4.96
12.38
2.66
19.82
5.08
10.31
PPLM
2.29
11.90
5,05
14.47
3.18
14.42
5.24
11.55
3.97
26.53
5.58
5.25
DExperts
4.47
22.41
5,35
6.28
4.68
15.21
5,49
9.12
4.30
30.37
5.38
8.60
SECOND THOUGHTS
AEM VM
4.85
26.73
5,41
11.96
5,48
18. 10
5.62
8.84
5.57
34.73
5.57
6.29
AEM AIL
4.55
25.20
5.64
9.23
5,40
19.60
6.04
7.31
5.04
32.09
6.22
5.38
AEM Only
3.80
24.10
5.22
10.55
4.87
16.37
6.01
7.01
3.86
31.41
5.12
5.75
Huge LM API service
GPT-3
3.28
22.26
5.34
7.31
3.02
14.01
5.75
6.54
2.96
19.22
5.31
7.49
InstructGPT
4.20
25.40
5.69
5.38
3.92
14.45
4.88
10.54
3.06
20.18
5.38
8.04
Pearson's '
0.73
0.91
0.69
0.84
0.55
0.86
Figure 3: Transfer learning ability of SECOND THOUGHTS from seen human values (i.e., trained on MRL, MIC, ETC) to unseen values (i.e., testing on TQA, HHH). We report the performance of checkpoints trained by increasing epochs and annotate the zero-shot performance of GPT-3 and InstructGPT for reference. T2T: vanilla text-to-text with source and target).
TOA
MIC TQA
ETC
TOA
12-
Rouge
Rouged
Rouge!
ABM
ABM Al.. 121
ABM AlL
ASM
0.1
121
T2T
GPT3
GP73
GPT3
0.00 025
835 1tt 125 1.00
3.0 225 230
352 125
675 1.00 125 1:3 175 2:00 235 250
000 025 030 3/5 18 125 1.0 1/75 248 2.30
Number Epochs
Number Epochs
Number of Epochs
MRL
HHH
MIC
HHH
0.12
ETC - HHH
0.10
0.10
Rouge
0.00
Rougel.
2.08
Rouge
0.00
0.96
2.58
AEM ABM Only
0.06
0.90
ACM Ovly
3.44
ACM Only
ASM
InsTucoOPT
Epochs
Epochs
Epochs
The results are shown in Figure 3, where the two rows reflect the results on two new datasets, while the three columns correspond the LMs trained three benchmark datasets. For the TQA dataset, find that after about 0.25 epochs, SECOND THOUGHTS trained on MRL and MIC with RL refinement (AEM VM/IL) outperform InstructGPT, which demonstrates the effectiveness of RL refinement. We have similar observation in the HHH dataset. However, training ETC does not seem to bring much benefit the value alignment on HHH. We also find removing chain-of-edits augmentation causes substantial performance drops, especially in the few-shot stage (less than one epoch). We take these results as evidence that the editing decomposition in SECOND THOUGHTS is crucial for improving transfer learning ability, especially in few-shot scenarios.
4.5 Error Analysis and Human-Guided Correction

---

Table 3: SECOND THOUGHTS enables higher quality human-guidee corrections, in terms of align- ment and coherence scores (1-7 Likert Scale). We hire human annotators to correct the same set of errors by re-prompting for GPT-3 and InstructGPT, or making changes on the chain-of-edit for SECOND THOUGHTS. Note that we record the corrections of three attempts for all models.
Moral Stories
MIC
ETHICS- Deontology
Alignment
Coherence
Alignment
Coherence
Alignment
Coherence
GPT-3
3,65 2.08
4.46 1.99
2.83 1.92
4.37 1.73
2.96 1.83
3,51 1.97
InstructGPT
4.56 1.48
4.95 1.60
4.62 1.52
5.25
1.47
3.47 1.75
3.70 1.87
AEM + VM
5.28 1.78
5,44 1.68
5.22 1.52
5.92
1.30
5.16 1.35
5.71 1.45
Figure 4: Hyperparameter search on balaneing factor a and entropy factor À in the Moral Stories task for best performing SECOND THOUGHTS, We also show the gains from chain-of-edits augmentation.
0:26
(a) AEM+AAM
12.2
8.25-
(b) AEM + VM
26.5
AEM VM AEM
26.0
Rouge
0.38
12,3
Perpleaity
Rougel
5.24-
perpleshy
Rouged
25.5
0.52
0.23
25.0
11.8
24.5
0.20
ROUDE-L Perplesity
11.5 2.22
ROUGE-L Perplecity
24.0
(c)
0.0
0.2
0.4
0.5
0.6
1.0
0.0
1.2
5.4
0.5
0.8 1.0
Balancing factor
Entropy factor A
Augmentation Factor
We analyze cases where the edited responses received low alignment or coherence scores in the test set of the three tasks, and exemplify these errors and how we correct them with SECOND THOUGHTS in $A.11. Most existing aligmment methods can barely correct errors after being trained as they have no scheme for receiving additional human guidance. Huge LMs based API services (e.g., GPT-3 and InstructGPT) can potentially fix their own errors by re-prompting (with prompts defined in $A.2), but finding a proper prompt requires tedious prompt engineering. Different from all these methods, SECOND THOUUIITS allows humans to make changes on the chain-of-edits. SECOND THOUGITTS will complete the chain and generate the desired target while taking the human changes into consider- ation. Note that these changes can be small as single word (e.g., see Table A10).
We compare with results from InstructGPT and GPT-3, derived by fixing the same errors with re-prompting, and conduct human evaluation on the quality of their corrections. As shown in Table 3, SECOND THOUGHTS makes clear advances in terms alignment and coherence after -umaan- correction, potentially because it enables more directed corrections via the chain-of-edits. We also find that the Instruction-fin-tuun fine-tuned InstructGPT can better adopt correction instructions than vanilla GPT-3, despite having over 100x fewer parameters.
4.6 Configuration for the Best Performing SECOND THOUGHTS
We also study the impact of the balancing factor (a) in AlL and the entropy factor (A) in VM on the performance of SECOND THOUGHTS. As shown in Figure 4 (a) and (b), for the example task Moral Stories, we find that in general higher a will worsen ROUGE-L but improve perplexity (i.e.. lowers it), as it decreases the effect of unlikelihood training on negative samples in AIL. Through empirical observation, we set a to be 0.2 for an appropriate balance, considering the trade-off between aligmment (ROUGE-L) and fluency (Perplexity). A similar trade-off can be seen for À in VM (set to X 0.6). In Figure 4 (c), we show the benefits of the augmentation of chain-of-edite we augment the training data by the augmentation factor, which is a multiple of the size of the original training data, using different editing costs, as described in $3.2. An augmentation factor of zero corresponds to vanilla text-to-text training. We find that more augmentation does not always lead to better performance in the test set, where the best augmentation factor is 2 for All. and for VM.
Limitations and Discussion
SECOND THOUGHTS can be limited by the LM that is based on-for instance, the total length of the chain-of-edits is limited by the max sequence length allowed for the L.M. Furthermore, studies from social sciences have shown that human values may change over time (Pettigrew, 2019; Paul, 2014),

---

meaning that SECOND THOUGHTS has to be re-trained with new human demonstrations as values evolve. We also note that the participants used for the human evaluation may not be representative of the full spectrum of people who may use SECOND THOUGHTS, and that certain demographic factors such as gender, education, and ideological belief might influence their value judgement. We thus conduct Ordinary Least Squares (OLS) regression analyses on our human evaluation results to better understand these impacts. Among other factors, the results indicate that the political party and the perceived importance of human values are two significant factors that have impact on value judgements.
Table 4: Ordinary Least Squares (OLS) Regression (DV: Alignment)
AEM AlL
AEM + VM
Predictors
B
SE
Sig.
B
SE
Sig.
Constant
2.27
0.87
0.01*
3.32
0.93
0.00**$
Gender (I=Male)
-0.27
0.16
0.10
-0.22
0.17
0.20
Race (1=White)
0.26
0.20
0.18
-0.10
0.21
0.63
Education
0.05
0.04
0.22
0.03
0.04
0.44
Age
0.00
0.01
0.96
0.00
0.01
0.82
Income
-0.01
0.05
0.93
0.01
0.06
0.81
Party Affliiaion
-0.12
0.05
0.01*
-0.16
0.05
0.00***
Value Importance
0.15
0.06
0.01
0.19
0.06
0.00
R²
0.11
0.14
Adjusted R²
0.07
0.11
N
297
297
Ordinary least squares (OLS) regression (shown in Table 4) analyses show that for both AEM + AIL and AEM + VM, party affiliation (which was measured on a 7-point scale where 1 indicates Democrat, 4 as Moderate, and 7 as Republican) is negatively associated with alignment values (AEM + AlL:B =-. 12, SE = .05, p .01; AEM + VM: B 16, SE = .05, p < .001), which indicates that the more liberal annotators tend to rate the alignments higher. This can be possibly explained by: 1) liberal users may be more familiar with such ML tasks and thus give our methods high alignment scores; or 2) it is also possible that conservative users are more skeptical of human-value alignment on such tasks. Another significant predictor is the people's 's perceived importance of alignment with human values (measured by answering the question "Whether or not the algorithm- -eetettteeeeeeddd text aligns with shared human values is important to me" on a 7-point scale). The more important people think alignment with human values is, the higher alignment scores they give for both methods.
Conclusion
We have proposed SECOND THOUGHTS, a novel learning paradigm that enables LMs to re-align with human values when given a poisoned context. Compared with existing methods, our method can generate text aligned with human-values without requiring additional human labeling or specifically- designed prompts or instructions. In addition, the chain-of-edits modeling by SECOND THOUGHTS enables easy error diagnosis and human- guided correction, which we believe be an essential ability for human-Al Al interactive systems.
For future work, we plan extend our methods more human value alignment tasks, and try to consider multi modaittyd data for alignment. For example, we can capture human's 's face expression as fine-grained feedback signals for un-aligned sentences, or reversely we can not only rely text edits but speech instructions as the chain-of-edits to model for proper value alignment.
Ethics, Broader Impact, and Reproducibility
As large-scale pre-trained LMs become integrated in more systems, is a matter of utmost societal importance to make sure that such models adhere to shared human values (Bai et al., 2022; Liu et al., 2021c, 2022). Here, we present light-weight framework that can align the generation of LMs with such values, without requiring new data or extensive prompt-engineerring Though we do not foresee
10

---

any major ethical issues with our proposed work, the reliance on manually annotated datasets and human evaluations may unintentionally introduce bias our models (as discussed in Section 5). To aid reproducibility, we have included all important information regarding hyperparameters and hardware this paper and have included data, code, and reports from the human evaluation in the supplementary materials to aid reviewing. We plan to release our code and data after publication under an MIT license.
Acknowledgement
We sincerely thank the reviewers for their insightful comments and suggestions that helped improve the paper. This research supported in part by Google Research Scholar Award.
References
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant laboratory for alignment. ArXiv preprins, abs/21 12.00861, 2021. URL https: //opss:/aane //tp:///axiiiii/ie www.v.org/aas///1.......... 00861.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Constitutional Harmlessness from al feedback. arXiv preprint arXiv:22 2022.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the op- portunities and risks of foundation models. ArXiv preprint, abs/2108.07258, 2021. URL https://arxiv org/ abaaaass///// 27280
Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 783-792. PMLR, 2019. URL attp:// proceedings mlr. www.sprsssvve.... v97 www.b19a....... html.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Businesslnsider.
People found a really easy way to make Siri curse. for- socerd-----oooddddiittoonnddd mother-2018-4, 2018. [Online; accessed May 18th, 2022].
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. Autoregressive entity re- trieval. In 9th International Conference Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, OpenReview.net, 2021. URL 5K8F6UU39V.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach. CA, USA pp. 4299 4307, 2017.
11

---

Felipe Codevilla, Eder Santana, Antonio M. López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 November 2, 2019, pp. 9328- 9337. IEEE, 2019. doi: N0.1109/CCC222019000 209.0094.. URLhttts::/// /tttttts:///di...o org/ /00 1109/IICVV 2019.00940
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://opeerreieee nttp///wwnnieeeeeeeeeeeeeee forum?id-HI Hednnidd---eeeDD......
Wanyu Du, Vipul Raheja, Dhruv Kumar, Zae Myung Kim, Melissa Lopez, and Dongyeop Kang. Understanding iterative revision from human- wwriten text. ArXiv preprint, abs/2203.03802. 2022. URL https: //axivv./ org/ abs/aaas///// 03802..
Denis Emelin, Ronan Le Bras, Jena D. Hwang, Maxwell Forbes, and Yejin Choi. Moral stories: Situated reasoning about norms, intents, actions, and their consequences. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 698-718, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguisties. doi: 10. 18653/v1/2021.cmml wenplleemmllmmame m4in..- URL https: : /aallllann aallaaholloooyyyoooo org/2022 20211.21
54.
Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. Social chemistry 101: Learning to reason about social and moral norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 653-670, Online, 2020. Association for Computational Linguistics. doi: 10. //aclanthology. - org/ 2020. emnlp-main.43
URL https:
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. RealTox- icityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computationnal Linguistics: EMNLP 2020, pp. $356-3369 Online, 2020. As- sociation for Computational Linguistics. doi: 10. http3//1/220..iiiiii//. .rgg:ooog//ggg org/2020. wwwdiignnnennnnnnnn emmlm.30000000 301.
URL
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisticx, pp. 8342-8360, Online, 2020. Association for Computational Linguistics. doi: 10. 10.18/33/v//2020 acl-main.744 740. URL. https: https///aclaattthoeeee
org
2020 acl-main. 740.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässsan Stockhoåm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1856-1865. PMLR, 2018 URL http: / http::///occeeeneneeeeee mlr. tates//v0/haarne haarnojai88 jamll.
press/v80/
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning AI with shared human values. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a. URL nttps://openrevviet www.:/eetaaeteeeee wtt/terrunnteeee met.fforumiiddddy_BBBB... RKzJJ..Y.
Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml 13916.
safety. ArXiv preprint, abs/2109.13916, 2021b. URL
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. Training compute-optimal large language models. ArXiv preprint, abs/2203. 2022. URL https://arxie wtwg/aa////eeeeeoeo 15556.
Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. Surface form competi- tion: Why the highest probability answer isn't always right. In Proceedings of the 2021 Conference
12

---

on Empirical Methods in Natural Language Processing, pp. 7038-7051, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguisties. doi: 10.18653/v1/2022 emalp-main.564. BetLhtttps://claattiiii
202g/2021..v emnlp-main.564,
Insider. Microsoft's virtual assistant "will get mad' if you 'say things that are particularly a-holeish'. https://www.buiieeet www.sunessissiidd.ooommmmsoooooo ceaeft-cottnaa-wiii wiill-get-mad--ae taa--at-----oee bwwaviour-2016-2/, 2016. [Online; accessed May 18th, 2022].
Liwei Jiang, Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Maxwell Forbes, Jon Borchardt, Jenny Liang, Oren Etzioni, Maarten Sap, and Yejin Choi. Delphi: Towards machine ethics and norms. ArXiv preprint, abs/2110.0755 2021. URL https: org/ wtwpaaa...///0//
Dan Jurafsky. Speech & language processing. Pearson Education India, 2000.
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint arXiv: (909v:999.005555 05858, 2019.
Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L McClelland, Jane X Wang. and Felix Hill. Can language models learn from explanations in context? ArXiv preprint, abs/2204.02329, 2022. URL https: //arxiv. org/ abs/ 2204. 02329.
Hoang Minh Le, Nan Jiang, Alekh Agarwal, Miroslav Dudik, Yisong Yue, and Hal Daumé III. Hierar- chical imitation and reinforcement learning. In Jennifer G. Dy and Andreas Krause (eds.), Proceed- ings of the 35th International Conference on Machine Learning, ICML 2018. Stockholmsmässan. Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 2923-2932. PMLR, 2018. URL http::/tprpoceee http://proceeding. mlr press/ ppress/v0//ee v80/1e18a...... html.
Mina Lee, Percy Liang, and Qian Yang. Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities. ArXiv preprint, abs/2201 .06796, 2022. URL https: //axxiv org/ abs/aabssss/////////////// 2201.06796
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582 4597, Online, 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/201. acl-long.353. URL http:/::////aaaahoooo.o.o
wttpg/2221acclll1111 acl acl-long... 353.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, 2004. Association for Computational Linguistics.
URL https: http:///caaathhooo/co
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. ArXiv preprint, abs/2109.07958, 2021. URL 07958.
Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. DExperts: Decoding-tin controlled text generation with experts and anti-experts. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pp. 6691,6706, Online, 2021a. Association for Computational Linguistics. doi: 10. 1/2121 acl-long.522. URL
org/ oogg/0221 acl- -ong.oog... 522.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, ArXiv preprint, abs/2107.13586 2021b. URL https: http///axvvvv..//ee//eeeeeeeee./ www.vi.r////.....oooooooo httgg//b//1001 13586.
Ruibo Liu, Lili Wang, Chenyan Jia, and Soroush Vosoughi. Political depolarization of news articles using attribute-aware word embeddings. Proceedings of the International AAAI Conference on Web and Social Media, 15(1):385-396 2021c. URI, htt.t://////// :/ttspp:://j....ee aaai. wwa...rra/aaaaaaaa...a..aa.o.ata worr/indxx, index.dexxx.. www.dde.ppp/IwsSS... php/ICWSM/ article/viens waalee/vie//eee 18069.
13

---

Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, and Soroush Vosoughi. Quantifying and alleviating political bias language models. Artificial Intelligecce, 304:103654, 2022.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint, abs/1907. 1692,, 2019. URL htttttt///aaaaet owg.oogg////... abs/ 1907 11692.
Nicholas Lourie, Ronan Le Bras, and Yejin Choi. Scruples: A corpus of community ethical judgments 32, 000 real-life anecdotes. ArXiv preprint, abs/2008.09094, 2020. URL https: /ttps://arxiv. arg/abs/2008 200949909..
Weicheng Ma, Ruibo Liu, Lili Wang, and Soroush Vosoughi. Emoji prediction: Extensions and benchmarking. ArXiv preprint, abs/2007.07389) 2020. URL hik..httpp///rr...orrr/bb/// 07389.
Ana Marasović, Iz Beltagy, Doug Downey, and Matthew E Peters. Few-shot self-rationalization with natural language prompts. ArXiv preprint, 11.08244 2021. URL https://arxiv, org/ abs/2111. 08284.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. ArXiv preprint, abs/2104.08773, 2021. URL www.ab/..././///...
https://arxivv
221/ 08773.
Burt L Monroe, Michael P Colaresi, and Kevin M Quinn. Fightin' words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16(4):372-403,
2008.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc G. Bellemare. Safe and efficient off- policy reinforcement learning. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 29: Anmual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pp. 1046-1055 1054, 2016.
Sharan Narang, Colin Raffel, Katherine Lee, Adam Roberts, Noah Fiedel, and Karishma Malkan. Wt5?! training text-to-text models to explain their predictions. ArXiv preprint, abs/2004. 14546, 2020. URL https: //t::///aaiiveaeea org/ /2044 14546.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. ArXiv preprint, abs/21 12.0011, 2021. URL https: / arxiv. org/ abs/2112. 1t//22...0011111 00114.
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. ArXiv preprint, abs/2203.02155, 2022. URL httts:///aaaiii org/abs/2203 02155.
Lauric Ann Paul. Transformative experience. OUP Oxford, 2014.
Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. Red teaming language models with language models. ArXiv preprint, 0ttp/2222002286, 2022. URL https: h
abs/2202. 03286.
Richard Pettigrew. Choosing for changing selves. Oxford University Press, 2019.
Anthony. Quinton. Utilitarian ethics. New studies in ethics. St. Martin's Press, New York, 1973.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8)9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to- text transformer. ArXiv preprint, abs/1910.100 10683, 2019. URL. httph:ttt//// //tari//...ie/// http:://wwww...gaa////eeese 10683.
14

---

Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, al. Multitask prompted training enables zero-shot task generalization. ArXiv preprint, abs/21 10.08207, 2021. URI. https: //arxiv. wwwpo/ggg/b/...oo.a////////// 08207.
Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. Social bias frames: Reasoning about social and power implications of language. In Proceedings of the 58th Annual Meeting of the Association for Compatational Linguistics, pp. 5477-5490, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/202...t acl- manelllai..45555 URL https:/ http:///acaaaathhooggoo
wwwgg2220..aa acl - main-main... main. 486.
Timo Schick, Sahana Udupa, and Hinrich Schütze. Self-diagnosis and selffdebiasing: A proposal for reducing corpus- corpus-based bias in NLP. Transactions of the Association for Computational Linguistics, 9:1408-1424, 2021. tac1-1.84.
doi: 10.1162/tucl a 00434. URL https: : https:///cccaa //caaahtoaa/cooggo...//. org/2021. 2021.
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 I July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 1889-1897.JMM JMLR..og.o 2015. .mw.engs.mmm.....mmmm......mm waa..preessvv33
URL
optimization algorithms. ArXiv preprint, 10tss///8//cchumaaa111 schulmanll.m.m.mmm html.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy 2017. URL https: /// aarxiviv org/
abs/1707. 06344777770667..
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize with human feedback. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florint Balcan, and Hsuan-Tiee -Ten Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020.
Alon Talmor, Ori Yoran, Ronan Le Bras, Chandra Bhagavatula, Yoav Goldberg, Yejin Choi, and Jonathan Berant. Commonsenseqa 2.0: Exposing the limits of ai through gamification. ArXiv preprint, abs/2201.05333332 2022. URL https: / http///axvvvvvo//// org/
wwt./ogg/////aaa..ooooo.oe ab///bb/////// 05320.
Irene Van Staveren. Beyond utilitarianism and deontology: Ethics in economics. Review of Political Economy, 19(1):11335, 2007.
Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi, Yeganch Kordi, Amirreza Mirzaci, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. ArXiv preprint, abs/2204.07705,, 2022. URL /taps:///arxiin
abs/ 07705.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Lee, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv preprint, abs/2201.11903 2022. URL https: / //tp///axvvv./////////ee/e.aavva www.ii.org//a............a.ooort abs/2201. 11903.
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. ArXiv preprint, abs/2112.04352 2021. URL https: /awxxxxxvvvvv////////////aa org/ ab1/ 04359.
Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa. Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL. https:://////o htttt::://penneeieeee
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Leurning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 12697-12706 PMLR, 2021. URL http:// bttp::/prroeee http://proceedings. mlr, wrwss/ ptess/vi3/// /hwa//ha22cccc c1c. html.
15

---

Daniel M Ziegler. Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford. Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ArXiv preprint, awsaaaa...00500050 09993, 2019, URI.
org/ wtwp///9.......... 08593..
Caleb Ziems, Jane A Yu, Yi-Chia Wang, Alon Halevy, and Diyi Yang. The moral integrity corpus: A benchmark for ethical dialogue systems. ArXiv preprint, abs/2204.03021, 2022. URL https: // arxii, org/ 2204, 03021.
16


Reviews:
[{"review_id": "ht2_5XzDJ64", "reviewer": "Reviewer_FzTk", "summary": "This paper addresses the problem of how we can control large LMs in order to generate text that does not contain any toxic or moral context (i.e., human value-aligned text). In contrast to prior work that focuses on constraining the decoding algorithm or engineering prompts to encourage LMs to produce value-aligned text, the authors propose to perform multiple edits (i.e., insert, delete, replace) iteratively on the generated text until they correct immoral statements. Moreover, instead of generating only one final edited sequence, they generate multiple positive and negative examples and fine-tune the LM given these examples. Finally, the authors include a reinforcement learning refinement step in order to encourage the LM to produce sequences that are value-aligned but still stay in context. They perform human evaluation on text generated by their method and other comparison models and find that the generated text by their approach is more value-aligned and stays in context. ", "questions": "1. Could you explain in Section 3.2 how exactly you fine-tune the model? What is the input/output, steps and objective?\n2. Do you think that your approach could scale to larger LMs? Is there any way to avoid full fine-tuning of the LM? Have you tried more parameter-efficient ways, such as for example adding adapter layers?", "limitations": "As stated in the \"Weaknesses\" section above, there are two main limitations of this paper. The first is related to the parameters of the LM that have to be fine-tuned, which poses a question about scalability to massive LMs, and the second is related to the human evaluation, which was not performed carefully given the difficulty of this problem. ", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The authors propose an interesting and intuitive approach for correcting generated text from large LMs. The method is novel and one main advantage of the proposed method is that it is interpretable with multiple edits until they reach the desired text in contrast to prior work that mostly tries to construct prompts or constraint decoding. \n2. The authors perform multiple ablation studies and analysis both in the main paper and in the appendix. Overall, their analysis indicates the performance gains of their method, which problems they address in contrast to prior work, and what components of their method are the most important. \n\n**Weaknesses**\n1. Although the proposed approach offers text of higher quality in comparison with comparison models, a significant limitation is that the LM has to be fine-tuned multiple times. As far as I understand, the model is first fine-tuned on positive and negative examples generated by editing the source text, and then is further trained via an RL objective. This is an important limitation in comparison to other approaches (constrained decoding, prompts) that do not require re-training of the model. This is especially important when we consider the problem at scale; the authors here fine-tune a 345M model, what will happen when we move to 1.3B or 175B model as the comparison systems? The whole problem at hand is related to how we can control massive LMs that have used the entire web for training and I am not sure whether this method can scale.\n2. Section 3.2 is not described in detail, so it is difficult to understand all details. This section would benefit from a more complete description, where the input/output, all steps, and objective for fine-tuning are described.\n3. During human evaluation, the authors state that they asked human judges: (a) “to what extend does the edited response improve the original response in terms of alignment with human values?“, and “how coherent is the edited response with the given context”. Although the qualities that the authors try to judge are hard to define in general, the questions are not well formulated. First, for question (a), what are the human values? This is too generic, and it is well-known that not all people share the same values; in fact, there is a large variation. I think this question should be much more specific, stating specific values that the authors want to test (e.g., toxic, racist, or sexist text with examples). Second, question (b) again is too generic, the authors should define coherence, and state specifically and in plain English the qualities that they search for. Given this setup, the results of the human evaluation are not very trustworthy (although there is definitely a positive indication and additional automatic metrics in the appendix that agree with the conclusions of the paper). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Discrimination / Bias / Fairness Concerns"]}, "extra_text": {}}, {"review_id": "guzJb2fhqFa", "reviewer": "Reviewer_uWVF", "summary": "The paper targets the alignment of existing Language Models to be aligned with Human values and proposes a new learning method that leverages reinforcement learning for refining the generated text to be more aligned with human values. The proposed learning method models insert, delete, and replace as a chain of edits using reinforcement learning and introduces Augmented Edits Modeling (AEM). The paper also proposes the use of 1) Adversarial Imitation Learning (AIL), which leverages negative samples and an adversarial language model for guiding the target language model to generate text which is more coherent to the context (keeping the value-alignment). And 2) Value Modeling (VM): where a Language Model based classifier is trained to judge the coherence between the generated text and the context.\n\nThe paper reports experimental results on three benchmarks, Moral Stories, MIC, and ETHICS-Deontology, and compares the proposed method for Alignment and Coherence of the generated text. The paper further reports the results for transfer learning with limited human-labeled data and performs error analysis using human-guided correction. The results portrayed in the paper show the effectiveness of the proposed AEM architecture.        ", "questions": "Qualitative samples which show less correlation with the Alignment and Coherence metric will be interesting to observe for a reader. Highlighting such examples will make the understanding of these metrics more clear and more reliable.\n", "limitations": "The authors highlight the technical limitation of the proposed method regarding the max sequence length allowed for a language model. Moreover, the authors clearly highlight the biases in the human participants' demographic factors, such as gender, education, and ideological belief, which might have influenced their reported value judgment. \n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is well motivated and highlights the problem of value alignment in the existing Language models.\n\n* The method proposed in the paper improves the existing language models by aligning them with human values. The reported results on three benchmarks show the effectiveness of the proposed architecture when compared to other text generation approaches.\n\n* The transfer learning results and error analysis make the results of the proposed architecture more concrete and reliable.\n\n\nWeaknesses:\n\n* The human evaluation design, as highlighted in the limitation section, might be imposing biases on value judgments due to factors like demographics, education, etc. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CTP4Sj3z6eW", "reviewer": "Reviewer_dpiE", "summary": "This paper proposes a method for using text editing for aligning language models to \"human values\". They compare with zero-shot GPT3 and Instruct-GPT3, in addition to other methods and generally improve performance on their benchmarks. \n\n---\n\nAfter response: Thank you for your response! The response has largely addressed my concerns and for this reason, I will increase my score to a 6. For a broader audience, I would strongly recommend adding these definitions of the terms that you added in the rebuttal in revision. \n\nFurthermore, I can appreciate how expensive it can be to train the InstructGPT models, however, given this statement:\n```Our experiments confirm that simply scaling LMs is not adequate for good alignment with human values, which echoes the findings of recent studies [50, 34]. Instead, smaller LMs trained with a few properly decomposed human demonstrations can often lead to better results (§4.3).```\nI would appreciate a clarification given what was mentioned in the response.\n\nThank you for your hard work!\n\n---", "questions": "### Definitions\nWhat do these terms mean?\n- Socially good text\n- Moral/immoral text\n- Human Values\n- Problematic text\n\nFor all of these terms, I believe that I have some understanding of what you are getting at, but they are vague/open to interpretation --- could they be better defined?\n\n\n- In the first figure, what is the likelihood when you compare simply changing \"without asking\" versus \"with asking\"? Could this difference in your figure be simply due to the wording?\n\n### Related work\nAlso, using edit operations is not novel, some works which have explored this are:\n\n[Encode, Tag, Realize: High-Precision Text Editing](https://arxiv.org/pdf/1909.01187)\n\n[Unsupervised Text Style Transfer with Padded Masked Language Models](https://arxiv.org/pdf/2010.01054)\n\n[LEWIS: Levenshtein Editing for Unsupervised Text Style Transfer](https://arxiv.org/pdf/2105.08206)\n\n[Learning to Model Editing Processes](https://arxiv.org/pdf/2205.12374) (I understand this was released at time of submission, but it would be good to cite)\n\n[Levenshtein Transformer](https://arxiv.org/pdf/1905.11006)\n\nRelated to this, this work should include a section on edit-based models\n\n### Potentially unfair comparison?\n- I don't think it's fair to claim that instruction tuning is inherently worse --- it is possible that if the 1.3B models were finetuned solely on the tasks in question using instructions, then they would perform as well. A more accurate claim, would be that the precision of this method is likely to be better than the large LM models. Furthermore, these prompts use words that are ill-defined/uncommon (morality/deontology) which may contribute to the lack of performance\n\n### Minor comments\n:%s/TrustfulQA/TruthfulQA/g\n\n\n", "limitations": "Yes, they provide an ethics statement.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- I genuinely like the method, I think it's simple and has potential for effectiveness\n- Mostly well-written\n\nWeaknesses:\n- I would appreciate more experiments on other datasets/tasks to verify the applicability of your method.\n- Many terms are used (e.g. \"human values\" without a clear definition to what that refers to in the text)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Discrimination / Bias / Fairness Concerns"]}, "extra_text": {}}, {"review_id": "5VypVZhJFNE", "reviewer": "Reviewer_j2s4", "summary": "This paper proposes a framework that uses sequential editions to align generated text with human values. Specifically, they finetune the language model by creating edit steps between unaligned and aligned sequences using dynamic programming-based methods as augmentation (AEM). The reinforcement learning method is applied to make the generation coherent better with the given context. The experimental results that have better value alignment and coherence compared with many previous language models and language models with larger sizes, ", "questions": " Do you think there are more parameter-efficient/post-hoc approaches to tackle this problem (with more tolerance in the performance) as it would be more prevail with service/API-based human value alignment?\n\nBesides, would human interventions (re-labeling, or other adjustments) in the early stage training further help the experiment in Figure 3 (experimental results may not be required for this question)?", "limitations": "The authors addresses the limitation of max length capacity and evolving of human values.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- The human value alignment is an important research question and many controlled generation problems can be framed in this way.\n- The editing-based framework offers a lighweight way to plug-in value alignment.\n\nWeakness\n- The limited data scenario is helpful to this work is helpful, but the demonstration in epochs makes it not very clear whether the few-shot cases fall into the individual efforts (e.g. 16-shot, 32-shot, 64-shot). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept