Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a method for using text editing for aligning language models to "human values". They compare with zero-shot GPT3 and Instruct-GPT3, in addition to other methods and generally improve performance on their benchmarks. 

---

After response: Thank you for your response! The response has largely addressed my concerns and for this reason, I will increase my score to a 6. For a broader audience, I would strongly recommend adding these definitions of the terms that you added in the rebuttal in revision. 

Furthermore, I can appreciate how expensive it can be to train the InstructGPT models, however, given this statement:
```Our experiments confirm that simply scaling LMs is not adequate for good alignment with human values, which echoes the findings of recent studies [50, 34]. Instead, smaller LMs trained with a few properly decomposed human demonstrations can often lead to better results (§4.3).```
I would appreciate a clarification given what was mentioned in the response.

Thank you for your hard work!

---
strengths_and_weaknesses: Strengths:
- I genuinely like the method, I think it's simple and has potential for effectiveness
- Mostly well-written

Weaknesses:
- I would appreciate more experiments on other datasets/tasks to verify the applicability of your method.
- Many terms are used (e.g. "human values" without a clear definition to what that refers to in the text)
questions: ### Definitions
What do these terms mean?
- Socially good text
- Moral/immoral text
- Human Values
- Problematic text

For all of these terms, I believe that I have some understanding of what you are getting at, but they are vague/open to interpretation --- could they be better defined?


- In the first figure, what is the likelihood when you compare simply changing "without asking" versus "with asking"? Could this difference in your figure be simply due to the wording?

### Related work
Also, using edit operations is not novel, some works which have explored this are:

[Encode, Tag, Realize: High-Precision Text Editing](https://arxiv.org/pdf/1909.01187)

[Unsupervised Text Style Transfer with Padded Masked Language Models](https://arxiv.org/pdf/2010.01054)

[LEWIS: Levenshtein Editing for Unsupervised Text Style Transfer](https://arxiv.org/pdf/2105.08206)

[Learning to Model Editing Processes](https://arxiv.org/pdf/2205.12374) (I understand this was released at time of submission, but it would be good to cite)

[Levenshtein Transformer](https://arxiv.org/pdf/1905.11006)

Related to this, this work should include a section on edit-based models

### Potentially unfair comparison?
- I don't think it's fair to claim that instruction tuning is inherently worse --- it is possible that if the 1.3B models were finetuned solely on the tasks in question using instructions, then they would perform as well. A more accurate claim, would be that the precision of this method is likely to be better than the large LM models. Furthermore, these prompts use words that are ill-defined/uncommon (morality/deontology) which may contribute to the lack of performance

### Minor comments
:%s/TrustfulQA/TruthfulQA/g



limitations: Yes, they provide an ethics statement.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper targets the alignment of existing Language Models to be aligned with Human values and proposes a new learning method that leverages reinforcement learning for refining the generated text to be more aligned with human values. The proposed learning method models insert, delete, and replace as a chain of edits using reinforcement learning and introduces Augmented Edits Modeling (AEM). The paper also proposes the use of 1) Adversarial Imitation Learning (AIL), which leverages negative samples and an adversarial language model for guiding the target language model to generate text which is more coherent to the context (keeping the value-alignment). And 2) Value Modeling (VM): where a Language Model based classifier is trained to judge the coherence between the generated text and the context.

The paper reports experimental results on three benchmarks, Moral Stories, MIC, and ETHICS-Deontology, and compares the proposed method for Alignment and Coherence of the generated text. The paper further reports the results for transfer learning with limited human-labeled data and performs error analysis using human-guided correction. The results portrayed in the paper show the effectiveness of the proposed AEM architecture.        
strengths_and_weaknesses: Strengths:
* The paper is well motivated and highlights the problem of value alignment in the existing Language models.

* The method proposed in the paper improves the existing language models by aligning them with human values. The reported results on three benchmarks show the effectiveness of the proposed architecture when compared to other text generation approaches.

* The transfer learning results and error analysis make the results of the proposed architecture more concrete and reliable.


Weaknesses:

* The human evaluation design, as highlighted in the limitation section, might be imposing biases on value judgments due to factors like demographics, education, etc. 

questions: Qualitative samples which show less correlation with the Alignment and Coherence metric will be interesting to observe for a reader. Highlighting such examples will make the understanding of these metrics more clear and more reliable.

limitations: The authors highlight the technical limitation of the proposed method regarding the max sequence length allowed for a language model. Moreover, the authors clearly highlight the biases in the human participants' demographic factors, such as gender, education, and ideological belief, which might have influenced their reported value judgment.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a framework that uses sequential editions to align generated text with human values. Specifically, they finetune the language model by creating edit steps between unaligned and aligned sequences using dynamic programming-based methods as augmentation (AEM). The reinforcement learning method is applied to make the generation coherent better with the given context. The experimental results that have better value alignment and coherence compared with many previous language models and language models with larger sizes, 
strengths_and_weaknesses: Strength
- The human value alignment is an important research question and many controlled generation problems can be framed in this way.
- The editing-based framework offers a lighweight way to plug-in value alignment.

Weakness
- The limited data scenario is helpful to this work is helpful, but the demonstration in epochs makes it not very clear whether the few-shot cases fall into the individual efforts (e.g. 16-shot, 32-shot, 64-shot). 
questions:  Do you think there are more parameter-efficient/post-hoc approaches to tackle this problem (with more tolerance in the performance) as it would be more prevail with service/API-based human value alignment?

Besides, would human interventions (re-labeling, or other adjustments) in the early stage training further help the experiment in Figure 3 (experimental results may not be required for this question)?
limitations: The authors addresses the limitation of max length capacity and evolving of human values.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper addresses the problem of how we can control large LMs in order to generate text that does not contain any toxic or moral context (i.e., human value-aligned text). In contrast to prior work that focuses on constraining the decoding algorithm or engineering prompts to encourage LMs to produce value-aligned text, the authors propose to perform multiple edits (i.e., insert, delete, replace) iteratively on the generated text until they correct immoral statements. Moreover, instead of generating only one final edited sequence, they generate multiple positive and negative examples and fine-tune the LM given these examples. Finally, the authors include a reinforcement learning refinement step in order to encourage the LM to produce sequences that are value-aligned but still stay in context. They perform human evaluation on text generated by their method and other comparison models and find that the generated text by their approach is more value-aligned and stays in context. 
strengths_and_weaknesses: **Strengths**
1. The authors propose an interesting and intuitive approach for correcting generated text from large LMs. The method is novel and one main advantage of the proposed method is that it is interpretable with multiple edits until they reach the desired text in contrast to prior work that mostly tries to construct prompts or constraint decoding. 
2. The authors perform multiple ablation studies and analysis both in the main paper and in the appendix. Overall, their analysis indicates the performance gains of their method, which problems they address in contrast to prior work, and what components of their method are the most important. 

**Weaknesses**
1. Although the proposed approach offers text of higher quality in comparison with comparison models, a significant limitation is that the LM has to be fine-tuned multiple times. As far as I understand, the model is first fine-tuned on positive and negative examples generated by editing the source text, and then is further trained via an RL objective. This is an important limitation in comparison to other approaches (constrained decoding, prompts) that do not require re-training of the model. This is especially important when we consider the problem at scale; the authors here fine-tune a 345M model, what will happen when we move to 1.3B or 175B model as the comparison systems? The whole problem at hand is related to how we can control massive LMs that have used the entire web for training and I am not sure whether this method can scale.
2. Section 3.2 is not described in detail, so it is difficult to understand all details. This section would benefit from a more complete description, where the input/output, all steps, and objective for fine-tuning are described.
3. During human evaluation, the authors state that they asked human judges: (a) “to what extend does the edited response improve the original response in terms of alignment with human values?“, and “how coherent is the edited response with the given context”. Although the qualities that the authors try to judge are hard to define in general, the questions are not well formulated. First, for question (a), what are the human values? This is too generic, and it is well-known that not all people share the same values; in fact, there is a large variation. I think this question should be much more specific, stating specific values that the authors want to test (e.g., toxic, racist, or sexist text with examples). Second, question (b) again is too generic, the authors should define coherence, and state specifically and in plain English the qualities that they search for. Given this setup, the results of the human evaluation are not very trustworthy (although there is definitely a positive indication and additional automatic metrics in the appendix that agree with the conclusions of the paper). 
questions: 1. Could you explain in Section 3.2 how exactly you fine-tune the model? What is the input/output, steps and objective?
2. Do you think that your approach could scale to larger LMs? Is there any way to avoid full fine-tuning of the LM? Have you tried more parameter-efficient ways, such as for example adding adapter layers?
limitations: As stated in the "Weaknesses" section above, there are two main limitations of this paper. The first is related to the parameters of the LM that have to be fine-tuned, which poses a question about scalability to massive LMs, and the second is related to the human evaluation, which was not performed carefully given the difficulty of this problem.
review_title: null

Meta-review: metareview: This paper received reviews and ratings that are leaning positively, the reviewer discussion and ethics review highlighted weaknesses that make this paper quite borderline.

Strengths:
1. The goal and task of the paper are quite well motivated, and they are geared towards positive societal impact: refining generated text to be more aligned with “human values” (e.g., gearing text towards “moral actions”). 
2. The text editing approach of the paper using adversarial imitation learning is both novel and intuitive according to the reviewer.
3. The authors give a convincing justification for their text-edit paradigm, as prior attribute-control generation methods (e.g., PPLM) tend to struggle when the context is polluted. Experimental results appear to support their claim.
4. The chain-of-edit paradigm of the paper (showing, e.g., how the text morphs into a more moral one) eases error diagnosis and enables interactive correction.

Weaknesses:
1. The questions asked to AMT workers in the human evaluation do not seem to be well formulated: (i.e., “To what extent does the edited response improve the original response in terms of alignment with human values?”).  First, the term “human value” is very generic, even the specific human value gets defined later (e.g., “deontology”). The AMT question form reads more like one that would be given to an expert rater, and would need to be either written in plain English or given to trained judges. Second, the judges need to identify *improvement* in alignment with human values could be a source of confusion, as the revised response could align well but no better than the original response (in which case the improvement is non-existent).
2. The ethics reviewers made comments that have bearing on the technical merits of the paper, as they both pointed out that the authors’ modeling assumption that human values are static and consistent across contexts is probably too simplistic. As ethics reviewer Udnx, “views and norms of the world are varied", and may depend on complex contexts (e.g., one may need a lot of background information about a given situation to know which action or statement is more moral). What seems concerning in the paper is that the “context” seems to be always reduced to one sentence, and it seems doubtful this provides enough information to make value judgments in many real-world situations.

In sum, the paper makes valuable contributions, but there may be biases in the results (Weakness 1) and the practical utility may be somewhat limited (Weakness 2). We would recommend that the authors address the AMT-related concern and discuss more extensively their human-value assumptions considering the ethics reviews.

Regarding ethical concerns: we also highly recommend that the authors follow the suggestions proposed by the ethics reviewers, e.g., include more information about the representativeness of the participants.
confidence: Less certain
Decision: Accept