Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper studies the feasibility of using a small, carefully-constructed corpus to guide large language modeling toward particular values embodied by the corpus. In particular, the authors perform a case study of whether such a process can make GPT-3 more sensitive, according to a Western, US American lens.

The authors describe an iterative procedure for collecting a dataset that reflects the targeted values and evaluating whether the language model has improved, according to the targeted standards. They apply their procedure to several topics and language models of varying sizes, showing that the process is generally successful for steering language models toward the desired values.

main_review: Originality: This paper is quite original, and is the first of my knowledge to contribute such an analysis for language models of the scale they study.

Clarity: I thought this paper was clear. I appreciated the authors’ efforts toward contextualizing the lenses / values that they are coming from, and recognizing that different people and different cultures have different lenses. The description of the data collection process and evaluation was also pretty easy to follow.

Quality / Significance: I thought that this paper was reasonably thorough in controlling for potential confounds. I wish there was a bit more discussion about why high-quality data in general (the “control” text) decreases language model toxicity---do the authors think that this is specific to the particular values they target? I also appreciated the level of detail to which the authors described the data collection and evaluation process. I wish there was more space in the main paper to highlight results across different topics, but the paper is quite content-full as is.

There are clearly many open questions left: how well does this method work for values that aren’t as widely held? Perhaps this method only works well if the language model has seen text that is broadly representative of the target values during pretraining. How can we have more fine-grained control over what prejudices are replaced with (e.g., the top descriptive word for “Muslim” switching from a bias toward Islamism to a bias toward heterosexuality).

Despite these questions, I think that this study will be of interest to the broader community, and that the work is significant.

limitations_and_societal_impact: Yes, I found the discussion to be satisfactory, and this paper in particular is one that requires such a discussion.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes an approach to aligning language models with target values. The authors construct a small hand-crafted dataset that reflects a set of intended values, then finetune large language models on this dataset. They find that this simple approach significantly improves performance according to a few different metrics.
main_review: This paper addresses an important problem (controlling the outputs of language models, especially in ways that are more aligned with human values). It proposes a simple method for addressing this problem that indeed seems to improve performance significantly. The experiments generally seem convincing, and the improvement seems reasonably large. 

The approach taken by the paper is *extremely* simple and essentially amounts to finetuning (albeit on a dataset constructed in a particular careful way). I think some would view this as a drawback, and it does admittedly makes the paper feel a little on the thin side in terms of originality, but for the most part I think this is a positive thing. Indeed, one of the main takeaways is that one can do a good job using this method without much effort by using an extremely small number of examples (in the appendix they claim that performance starts to converge at around only 60 samples, at least for the 175B model). 

The paper itself is also well-written, with a very detailed appendix. Overall I think it's relatively straightforward, but well-executed, with a couple arguably surprising results, and makes meaningful progress in both addressing and understanding a very important problem. 

Other feedback:
- I think it's an important observation that the approach becomes more effective with model size, but to the extent that larger models are more sample efficient this isn't too surprising. I'd like to know how much of this is due to larger models converging to a higher level of performance vs merely being more sample efficient than smaller models. To be fair, they do say on the last page that future work should assess scaling laws for this problem, which is basically what I'm advocating here. But it seems especially important in this context because it affects how scalable this approach will be moving forward. I'd be less excited about it if peak performance is pretty much the same but converges at 6 samples instead of 60 samples with GPT-4 or whatever.
- I'm also curious to know how finetuning on this hand-crafted dataset affects performance out-of-distribution. For example, if you have a rather different input format (such as using the 175B model as a chat bot having a long back-and-forth conversation with a human), does it still tend to output more aligned outputs?

I think it would be quite a bit stronger with these sorts of additional experiments, which is why I'm only giving it a 7, but I think it still probably deserves to be accepted.

Minor comments;
- ALMBFVD is a very cumbersome name, so I would seriously consider changing it. Even just FVD would be much better in my opinion.
- Many of the figures could be much nicer (partly aesthetically, but especially in terms of ease of understanding what's going on).
limitations_and_societal_impact: This paper is largely focused on addressing a problem that is societally important. I think it mostly does a good job of this. One point I would probably want it to emphasize somewhat more is that the target values to finetune on are not only subject to some debate, but may also vary from application to application even if there is consensus (e.g. because it may depend on whether a model is making descriptive claims or normative claims). But especially given that they're mainly proposing a general framework/approach rather than, say, a particular dataset to finetune on, this is a pretty minor point.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: Language models exhibit harmful biases. The authors try to address the toxic behaviors of the GPT-3 model as follows. First, they construct a set of examples that correspond to a large set of desired “values” (e.g., gender equality, realistic beauty standards). The examples are prompt-continuation pairs. Then, they finetune the language model to generate the continuation (which has the desired values) given the prompt. 

The authors find that using a very small set of examples, the post-finetuning models would generate continuations that are less toxic, using both quantitative (human-eval scores and automatic toxicity scores) and qualitative evaluation methods that analyze word associations given a specific gender, religion, or race. 

The paper targets U.S. values only which are not necessarily the universal values; however, the authors acknowledge and point this issue out, in the introduction and the discussion sections. 

main_review: Issue on formatting: The NeurIPS website says “you must format your submission using the NeurIPS 2021 LaTeX style file which includes a preprint option for non-anonymous preprints posted online” but the font of the submitted article does not follow the provided tex file.

(+) The sensitive topics are chosen according to official sources like the government and the United Nations guidelines. They cover a wide range of categories. 

(+) Significant testing of the quantitative results (when comparing the post-finetuning model and the pre-finetuning model) is comprehensive. 

Lots of critical information, which is required to judge the quality of the paper, is hidden in the 30-page-long appendix; e.g., the methodology for choosing the number of prompts, how specifically the authors wrote the prompts. However, I appreciate the detailed explanations of the prompts in the appendix. 

(-) Hyperparameter tuning seems arbitrary (e.g., “training loss weight was 0.1 for the prompt and 1.0 for the completion”, “using learning rate 30x less than the default learning rate” in line 1325). Training details should be expanded. Important parts can be moved to the main text. It’s unclear how long the finetuning lasts. What are the criteria for “convergence”?

(-) Currently the train/dev/test prompts are similar in terms of style. The models can generate better continuations when using top-p sampling (p=0.8) & when there is a prompt that follows a specific format (e.g., one sentence long, usually under 15 words, ending in question marks, etc.). Given the evaluation strategy, it’s unclear whether the improvement in values can transfer to downstream applications.

(-) It’s unclear whether the post-finetuning models still do as well as the pre-finetuning models on GLUE/SuperGLUE classic benchmark tasks.

(-) Missing citations: Nangia et al. (EMNLP 2020), https://arxiv.org/pdf/2010.00133.pdf, which deals with the discussion on categories of biases as well as the evaluation of LM bias. A comprehensive discussion on previous LM-bias-related works should be included; e.g., http://web.cs.ucla.edu/~kwchang/publications_area/.

I would hope for more information related to human factors. 
- Who are the writers? How are the writers trained? 
- How do the authors validate the continuations? 
- Who are the human annotators? How are they trained? 
- How do the authors validate the annotation results? 

Minor:
- Footnote 9: "top-p controls diversity via nucleus sampling" but top-p sampling and nucleus sampling are the same thing.
- A space before bracket citations?
limitations_and_societal_impact: The project is done using American English, and the project caters to a specific set of U.S. values. However, the authors acknowledge the issue and claim that a similar procedure to fix a biased language model can be done in the context of other cultures as well. 

IF the post-finetuning models generalize to downstream tasks (i.e., the tasks still achieve the same performance; there is less bias in downstream tasks), then this work would be encouraging for the NLP/ethics community because it claims that using a small set of expert-written examples could greatly debias a language model.
review_title: null

Meta-review: This paper demonstrates that large language models (i.e., GPT-3) can be relatively cheaply fine-tuned so as to generate text continuations consistent with some specific values/toxicity-related goal. Reviewers had some concerns about how broadly these results would apply, but nonetheless all found the results to be significant and timely.
Decision: Accept (Spotlight)