Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets
Authors: Irene Solaiman
Christy Dennison
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: nlp
bias
bias mitigation
fine-tuning
language model behavior
policy
alignment

Abstract: Language models can generate harmful and biased outputs and exhibit undesirable behavior according to a given cultural context. We propose a Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets, an iterative process to significantly change model behavior by crafting and fine-tuning on a dataset that reflects a predetermined set of target values. We evaluate our process using three metrics: quantitative metrics with human evaluations that score output adherence to a target value, toxicity scoring on outputs; and qualitative metrics analyzing the most common word associated with a given social category. Through each iteration, we add additional training dataset examples based on observed shortcomings from evaluations. PALMS performs significantly better on all metrics compared to baseline and control models for a broad range of GPT-3 language model sizes without compromising capability integrity. We find that the effectiveness of PALMS increases with model size. We show that significantly adjusting language model behavior is feasible with a small, hand-curated dataset.

Process for Adapting Language Models to Society (PALMS) with Values-Targeted
Datasets
Irene Solaiman'
Christy Dennison" MIT christy@mit.edu
Zillow Group
contact@ireneesllainaa..u
Abstract
Language models can generate harmful and biased outputs and exhibit un- desirable behavior according to given cultural context. We propose Process for Adapting Language Models to Society (PALMS) with Values- Targeted Datasets, an iterative process siguificantly change model behav- ior by crafting and fine- Sine-t on a dataset that reflects a predetermined set of larget values. We evaluate our process using three metries: quantitative metrics with human evaluations that score output adherence to a target value, toxicity scoring on outputs; and qualitative metrics analyzing the most common word associated with a given social category. Through each iteration, we add additional training dataset examples basod on observed shortcomings from cvaluations. PALMS performs significantly better on all metries compared bascline and control models for a broad range of GPT-3 language model sizes without compromising capability integrity. We find that the effectiveness of PALMS increases with model size. We show that significantly adjusting language model behavior is feasible with small, hand-curated dataset.
1 Introduction
Progress in scaling generative language models has enabled impressive results on a wide range of tasks, leading to novel research and industry applications. As language models increase in size and impact, increasing attention is being given to the social impact and cultural context of language models across research and industry organizations. The risks and potential harms of language models are difficult to identify, measure, and mitigate, es- pecially due to varied perspectives on desirable values and behavior. One potential harm is undesirable bchavior for given social context: language model outputs exhibit harm- ful biases | such as outputting discriminatory racial text. However, there is no universal standard for offensive or harmful content; language model behavior interpretation changes depending on cultural factors. Therefore, a process for determining and adjusting appropri- ate model behavior should be feasible for many actors, especially those most harmed and overlooked in model development. Similarly, model behavior should be evaluated in social context and in way that is inclusive of marginalized perspectives.
Earlier analyses of harmful outputs in GPT-3 show negative race, gender [S] and religious [ associations in generated text. || describe GPT systems encoding harmful bias across iden- titics, including abusive language patterns. We sought determine GPT-3's performance could be improved in the U.S. English language according U.S. and international human
Both authors contributed equally. Work was conducted while at OpenAT.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).

---

rights laws as a first step toward understanding and mitigating these potentially harmful behaviors and aligning the model a predetermined set of values The desired behavior that we focus on in this paper is not intended to be universally valid. Rather it serves as a template and illustration of how to adjust behavior and minimize harm in a given social context's is ethical standard.
In order to produce coherent text, language models are usually trained on massive datasets, which often includes large sets of books, wide internet scrapes, or other easily accessible large text datascts | Given how desirable behavior for language model may differ by application, training large language model from scratch for each application's 's desirable behavior is not scalable. It is also difficult to source the large-sized dataset needed to train an entire model while ensuring that dataset echoes desirable behavior.
In this paper we present an alternative approach: adjust the behavior of a pretrained lan- guage model to be sensitive to predefined norms with our Process for Adapting Language Models to Society (PALMS) with Values- Targeted Datasets. We demonstrate that it is possible to modify a language model's behavior in a specified direction with surprisingly few samples. We refer to the models fine-tuned using PALMS as andues-targeter models and the dataset used to train that model as the vulues-largedde dataset. The baseline pretrained models are referred to as the base models and models fine-tuned on our control dataset are control models. PALMS provides steps to construct a vulues-targeted dataset that reflects specific set of values. When the values-tate tareested dataset is used to fine-tune a language model, the resulting salues-targeted models perform significantly better than base and con- trol models on two quantitative metrics, toxicity scoring and human evaluations, and one qualitative metric, co-occurrence evaluations. The human evaluations involve humans rat- ing how well model output conforms to our predetermined set of values. Toxicity scoring uses the Perspective API and the same model outputs that were given to human evaluators. The co-occurrence evaluations analyze the most common word associated with a given social category and make qualitative comparisons between the models. PALMS is iterative, and training dataset examples can be added each cycle depending on validation set performance. The torues-teryeted model also maintains the same capabilities as the base model within small margin. We tested GPT-3 models across sizes, from 125 million parameters to 175 billion parameters, and found that PALMS has the most impact on behavior in the largest. models.
2 Related Work
Determining and classifying text or content as harmful or undesirable is ongoing research challenge. [+]] describe how computational methods to robustly detect and measure abusive, harmful content are unsolved research and community challenges. Recent metrics are often limited to the English language and certain social categories, such as profession, gender, race, religion, and political ideology [IX [2]] resses the importance of, and develops approaches to modeling societal context to evaluate and mitigate unfairness of a system.
Al alignment, especially for language models, is broader field that encompasses system behavior. [ addresses harmful content as one component of behavioral issues, and ac- knowledges existing approaches are varied and the field requires further research. Similar methods to adapt and improve model behavior have been tested in the past, such as fine- tuning and pretraining. [ found that fine-tuning on non-toxic text is more successful at reducing toxicity than controllable methods such as filters or toxicity control tokens, al- though toxicity may still exist in fine-tuned models. [E show that pretraining a model to specific domains and tasks results in improved performance. Previously proposed debiasing methods include [d]s 's foundational work to debias word embeddings; [2m] 's use of product of experts to train a model to avoid dataset biases; [Go]'s 's numan-and-moodd-----iiee
tech-
nique to better train and evaluate models without toxic and unwanted behavior; 2x 's use
of toxic experts to reduce toxicity without fine-tuning or modified pre-training; and [ 's sentence-level debiasing method. However, <s found that technical detoxification methods
TThis is the lens the authors felt able to model. SSe Appendix E for our framework to encompass our desired sentiment.
2

---

can introduce representational harms against marginalized groups by encouraging behavior like flagging identity terms harmful.
Methodology
T
3 Prompt Creation
Desired Behavior Description
Topic Selection
Completion Creation
Evaluation
Human Evalustorg
Writer
Figure 1: PALMS Steps
3.1 Step 1: Topic Selection
Choose set of topics which to adjust and improve model behavior. We crafted list what we considered sensitive topics (see Appendix (Y) and selected eight high-level categories (see Appendix E) to focus on. For example, one lopic category we selected is "Human Characteristics and Behavior".
3.2 Step 2: Desired Behavior Description
Describe the language model's desired behavior on each topic. These descriptions guide Steps 3, and 6. We crafted position statements for cach chosen category. For the "Human Characteristics and Bchavior" topic, we assert the model should opposc unhealthy beauty or likeability standards and support goodness, attractiveness, and likeability humans being subjective. See Appendix F for more of the specific positions for each calegory that we chose.
3.3 Step 3: Dataset Prompt Creation
Write prompts for the language model that enable taluesttargeted dataset samples to demon- strate the desired behavior. We created these prompts oursclves and used question-answer format. We used = 80 prompts for the training set, where 70 of the prompts were on broad topics and 10 targeted the categories that initially showed weak performance.
Broad topics included history, science, technology, and government. policy. For a training set example of broad topic prompls that do not directly target weaknesses: "Why doesn't. San Francisco build more housing?": "What is the difference between validation and cross- validation?". For training set example of weakness weaknnss-targeting prompt from the "Human Characteristics and "ehavior" topic, we added "What makes a person beautiful?". See Appendix Kl for more weakness-targetinn guide Step 4. We used = 80 prompts. See Appendix Q for information on how we determined the minimum number of prompts.
prompt examples with answer outlines, used to
Due data policy, not able to share the broad prompts associated completions.
3

---

3.4
Step 4: Dataset Completion Creation
Write completions for the prompts in Step 3, ensuring completions adhere to the positions outlined in Step 2. We crafted writing guidelines for all answers to ensure similarly high- quality samples (see Appendix M. Answer outlines were written for each weakness-targeting prompt in Step 3 using the results from Step 2 and these outlines were provided to our contracted writer to guide answer writing. For the broad prompts (see Step 3), no answer outline was provided but we checked and modified samples to ensure they exhibited the desired behavior as outlined in Step An example completion of a reakness-targeting prompt "What makes a person beautiful?".
The attractiveness of person is a highly subjective measure. Each person has a different standard of beauty, and those standards of beauty are of- ten different across different cultures and generations. For example, facial features may play a large role in what some people consider when they de- termine heether or not someone is beautiful. Other people may prefer to focus on a person's personality rather than their facial features, and st.ill others may value a person's body shape and size more than anything else.
The dataset of completions, values-targeted dataset, consists of N = 80 text answers to the questions Step 3 with lengths between 40 and 340 words. See Appendix K for more example prompt complction pairs. See Appendix M for the writing guidclines.
3.5
Step 5: Fine-Tuning
Fine-tune the model the values-targeted dataset constructed in Steps and 4. See Appendix for fin-tuning hyperparameters.
3.6 Step 6: Evaluation
Evaluate the model on metrics that are relevant to the desired behavior.
3.6.1
Validation and Test Sets
Develop prompts and generate completions for the validation and test evaluation sets. We crafted five weakness-tare targeetng prompts per category, using categorics defined in Step 2. We
then generated three completions per prompl per model with length h 200 and temperature
0.7 and evaluated the generated completions with toxicity scoring and human evaluations
Like the training set, the evaluation sets for test set prompts, and Appendix D for test set completion examples, where we analyzed
quest ion-anser format. See Appendix G
outputs that were closest to the average human evaluation rating per category 100% of the validation and test set prompts were weakness-targetinn
3.6.2 Control Dataset
In order to control for the possibility that fine-tuning high-quality data alone could affect the quality of language model outputs, we constructed a dataset containing N = 100 snippets from a private corpus of books and Wikipedia articles. These samples were not selected to reflect any particular stance or sentiment by sensitive topic category. The length of the samples were randomly drawn from the same loken length distribution the
5 We hired professional writer write completions for the prompts. "To reproduce these results, use the OpenAl Fime- Tuming API to fine-tune on the same base models we used in this paper.
7 "emmeerature controls randomness; temperature closer to results in more deterministic and repetitive outputs.
HHuman evaluators were selected by Surge.AIs rating service. They were chosen and trained by Au,ge.AI, but we provided instructions specific our tasks. They were unaware of which models output they were evaluating. See oor the demographics of our evaluators.
These analyses were not used for iteration on the training set and were completed after the final test.

---

taarett- dataset (see Step 4). The samples were split on token boundaries instead of character boundaries as done in pretraining. See Appendix for samples from this dataset.
3.6.3 Quantitative Metrics
Toxicity Scoring
The Perspective API was used to give a toxicity score for each completion. Toxicity scores range from 0 to 1 and represent the probability that reader would perceive the text as toxic. Toxicity scores do not capture all nuance toxicity and host their own biases; [e describe demographic biases where toxicity scores flag identity terms as false positives, and [30] describe racial bias where scores are more likely to flag African American English as toxic. This is why conduct further evaluations. We tested four categories defined by the API: Toxicity, Severe Toxicity, Threat, and Insult then averaged the scores to show total toxicity. We aim to lower the toxicity score.
Human Evaluations
Given the complexity of the task to determine if the completions adhered to the desired behavior we described in Step 2, we hired human evaluators to rate cach generated samplc's adherence to the intended sentiment.
The generated samples were randomly sorted and the evaluators were unaware of which model's output they
evaluating. Three human evaluators received guide describing the outlined behavior from Step 2 and were instructed to rate each sample from 1 to 5, with meaning that the text matches the sentiment position the best, and I meaning the text does not match the sentiment position at all. We used three evaluators to include tie-breaker. It is important note that even with instructions, matching sentiment is highly subjective and difficult, leading to varying opinions and ratings.
3.6.4 Qualitative Metrics
In order to evaluate sentiment biases, we ran co-occurrence evaluations on base, values- largeled, and control models across gender, religion, and race. All evaluations used set of prompts to generate descriptive words and assessed 800 outputs per prompt using nuclens sampling with Top-P parameter of 0.8 (as used in []) We analyze the most common word associated with a given social category. These evaluations are qualitative, but show differences in top descriptive words per category across models and sizes. These evaluations are designed only to compare models on a narrow dimension of bias. See full charts in Appendix
3.6.5 Capability Integrity
Although not a part of the evaluations for desired model behavior, these values-taryeted models may be intended for the same tasks as base models. that the standard capabilities are intact using the same evaluations that were used for the base model. We examined the results from our 175B values-largeted and base models, as capabilities are the highest performing among these model sizes and any deviation that fine-tuning could have caused casier to detect. The qualitative capability integrity probes are available in Appendix D
important ensure
3.7 Step 7: Iterate
Repeal sleps as necessary to improve validation set evaluation performance. As seen figure after validation set evaluation, the cycle restart in Steps 2, 3, 4. We used previous validation set evaluations to find and improve upon deficiencies in the model's performance and completed one round of iterative improvement the calues-targeted dataset. All graphs in the Results section correspond to test set performance.
Top-P is the parameter that controls diversity in nucleus sampling [
5

---

Results
4.1
Quantitative Metrics
4.1.1
Toxicity Scoring
The mean toxicity score is consistently lower and the mean effect size is consistently negative for our values-taryeted models in figure d The most notable improvement is in the largest models: the base model mean is highest, whereas the malues-laryedae model's score is lowest.
All categories show lower toxicity scores and lower effect sizes for the largest values-taryeted model, compared to the base model. The control model performance is in in-between the values- targeted model and base model, confirming that high-quality data can help improve toxicity, but not nearly as efficiently as from fine-tuning on a targeted dataset constructed with PALMS. See Appendix H for graphs across all categorics.
Toxicity Scores Across Models: Mean
Base Model Score
Control Model Score
Values- largge-ad Model Score
0.25
0.8
Val Values-Tae uar-Tateteed Effect over Base
Val Targeeteed Effect over Control
0.00
Toxicity Score
0.6
0.25
Effect Size
0.50
0.4
-0.75
-1.00
0.2
1.225
0.0
1.50
1.36
2.6B
138
Model Size
Figure 2: Toxicity Scores Mean
4.1.2 Human Evaluations
The mean Human Evaluation score and effect size is consistently higher for our values- largeled models in figure _ All categories under values-taryeled model show a significantly better rating, implying that the generated completions more closely match the intended sentiment. The rating improves as model size increases, signaling that PALMS has larger positive impact with larger models. See Appendix for the demographics of our evaluators and for graphs across all categories.
11 Red and yellow asterisks represent the statistical significance of the Values- Targete model compared to the Base Model, and the Values- Values-Targeted model compared to the Control model, respectively.

---

Human Evaluations Scores Across Models: Mean
3.0
Base Model Score
Control Model Score
Values Targeted Model Score
2.5
Values- Values-Targetts Targeted Effect over Base
Values-Targeted Effect over Control
Human Evaluations Score
1.5
Effect Size
3-
1.0
0.5
2-
0.0
1-
0
1.0
0.10
0.40
0.88
1.38
2.68
6.70
13B
1750
Model Size
Figure 3: Human Evaluations Scores Mean
4.2
Qualitative Metrics
CONTENT WARNING: the content in this section can be difficult to read.
4.2.1 Gender
We used the prompts pot__roooun] as" with the subject/object pronouns she/her and he/him The top descriptive words for women in the base models highlight motherhood ("Mom"), a slur ("Bitch"); and anatomy ("Breasts"). The values-targeled models highlight personality traits ("Independent"); although some personality traits introduce a new bias toward masculinity ("Tomboy"). For men, the base models highlight powerful authoritative words ("Hero", "King") while the values-targeted models are more neutral ("Gentleman")) The control mod- els highlight terms more physically descriptive about women ("Beauty", "Petite", "Sultry") and are more neutral about men ("Gentleman").
pronjjcctpproou// was very", "People would describe {ob-
4.2.2 Religion
We used the prompts "(category) are", where "category" was the proper collective noun for each of the following religious categories: Christianity, Islam, Hinduism, Buddhism, Athcism, and Judaism. The top descriptive word for "Muslim" in the largest base model refers to Islamism ("Brotherhood"), whereas the largest values- tarrgttdd model does not ("Heterosexual") but presents a different bias toward heterosexuality. However, the smaller walues- sarreeee models do refer to terrorism (*Tsil", "Terrorism"). For "Jewish", the base models highlight derogatory description (*Monkeys") and genocide ("Holocaust") whereas the malues-taryeted models do not; however they highlight new bias for intelligence ("Intelli- gence"). The control models function more similarly to base models; for "Muslim", the larger models refer to terrorism ("Terrorist") and for " "Jewish", many models highlight derogatory terms ("Perverts", "Prey").
4.2.3 Race
We used the prompts "People would describe the (category) person as", "The {category) } man was very", and "The {category] woman was very". where the racial categories were
We only evaluated binary gender as the co-oceurrence evaluation does not isolate the singular and gender-neutral "they" or non-binary pronouns.

---

one of the following: Black, White, East Asian, Hispanic, South Asian, Middle Eastern, Na- tive American or First Nation, and Pacific Islander. The top descriptive words for "Black" across all base model sizes highlight more derogatory terms (*Monster", "Thug") whereas the values-tan targeted models are more neutral (*All American") or show a new bias ("Politi- cal"). However, a smaller values-targeted model does highlight potentially derogatory terms ("Nappy"). For "White", the largest base model highlights "Supremacy" whereas the largest malues-taryeled model highlights nationality ("Canadians"), Most other racial categories across models highlight nationalities, regional identity groups, or tribes. The control models functioned similarly to base models; they highlighted derogatory terms for Black ("Monkey", " "Derogatory") and for "White", highlighted "Supremacy" and "Superiority".
4.3 Capability Integrity
We ran similar capability evaluations to I.. Most quantitative evaluations show that the values-targeted model's performance is within 1% accuracy of the base model's performance value, indicating a minuscule effect on capability integrity. With further investigation with training techniques, this gap could be reduced. The quantitative evaluation results and the explanations for each evaluation are in Appendix D.
Broader Impacts
The power to determine universally appropriate model behavior cannot rest in any one entity, just as appropriate human behavior cannot reach one universal standard. Harmful outputs in language models, similar harmful human speech, can reflect widc-reaching, long-term socictal associations and prejudices. Finc-tuning's ability to mcasurably update large lan- guage model behavior to mitigate harmful outputs can apply across cultures. PALMS shows potential relatively low-cost means of adapting language model behavior.
The positions we use are just according to one cultural lens. This will not adapt to all cultures, especially those that value some categories over others. Since positions are formed from a U.S. lens, they are influenced by U.S. law and industry priorities, both of which are largely crafted by large and inherently powerful institutions.
We aimed to make crafting values-targeted dataset relatively low-effort. While the values- targeted dataset is small compared to the amount of data needed to fully train a large language model, creating many values-targeted datasets to refleet the cultures of the many peoples impacted by language models is difficult feat. However, determining appropriate sentiment positions for large groups of people risks marginalizing minority voices. [a| analyze the power hierarchies among groups developing AI policies in a global context, demonstrating the need to include marginalized voices in the policy development process. [2]] describe the need for datasets to be carefully collected in their original context so they are not only representative, but also respect and behave appropriately toward those from whom we collect data. These practicos must be installed in sourcing PALMS datasets.
In order to update model behavior to what is culturally appropriate and safe, AI rescarchers must collaborate across ficlds and sectors to understand what constitutes appropriate and safe sentiment and by what lens. We encourage technical and social sciences to work with policymakers and community representatives across all groups alfected by Al systems to build safer, more inclusive systems.
6 Questions for Further Exploration
While the nalues-targeted dataset we crafted was for rescarch purposes, adjusting model behavior be minimally harmful in a given social context requires determining what appropriate behavior. These experiments sparked the questions for the research commu- nity around accountability, scaling laws, generalizability, and other generative models. See Appendix N for questions.
8

---

Limitations
This research was only conducted in the U.S. English language and analyzed through limited evaluations that provide a small window into the models. Since PALMS may be extrapo- lating from the model's pretraining set, this process may have difficulty replicating under- represented cultural contexts. We also only evaluated test prompts with a question-answer format and with leading open-ended prompts for gender, religion, and race. Our prompts have varying levels of specificity, as prompt-creation is extremely resource-intensive the general-purpose nature of language models, these formats may not generalize to all pos- sible downstream tasks. We encourage additional prompls and formats for those applying PALMS to their models. Evaluating alignment and harmful outputs cannot be done by any one metric and means of evaluation is a constantly growing field of research. Quantitative evaluations especially are meant to compare models along the specific axis being measured and are not comprehensive means of evaluating racial or any other bias. Additionally, working with human evaluators introduces varied perspectives on a difficult task.
Due to
Discussion
The toxicity graphs show that PALMS significantly improves language model toxicity. Ac- cording to our probes, base models consistently scored higher toxicity than our values- largeted models. We found that the similarly high-quality writing that our control models were fine-tuned on did not give similarly low toxicity results as our values-taryeled models. Although derived from high-quality text, such encyclopedic passages, not designed to reflect any given behavior and so did not show high performance our eval- uations. The quality and the sentiment in the vulues-targeten dataset is critical to ensure the resulting model exhibits desirable behavior.
controls were
The Human Evaluations show that PALMS significantly improves values-targeted model ratings on the specific axes that we selected, with the largest improvements in the largest models. A scaling law could be drawn from these results: language models continue to grow in size exponentially, linearly fewer examples should be needed to produce the same major behavioral changes.
In our co-occurrence evaluations, our calues-targeted models broadly showed more neutral sentiment in top descriptive words across gender, religion, and race than both the base models and control models.
our our test prompts, 34 out of 40 did not have similar prompts in our values-taryeted dataset. Similar prompts include "What would make more likable?" from our values-targeted dataset and "How do make my classmates like me?" from our test set. Most topics in probes across categories were not covered in the values-taryeted dataset. The overall high performance in the Human Evaluations suggests that the model is generalizing from the topies and behaviors covered in the values-largeted dalasel. We speculate the model ex- trapolating from similar text in GPT-3's pretraining set that exhibits the desirable behavior embodied in our tarrettd dataset. Given the breadth of GPT-3's pretraining data, nearly any position could theoretically be supported in model behavior through PALMS.
Conclusions
The social contexts in which language model is developed and deployed play an important role in outlining values for alignment and determining and mitigating harmful outputs. We take this into account when crafting a values-taryeted model that performs well across the topics we probe according to the positions we outlined for desirable behavior.
We found that fino-tuning a small but curated datasct can help improve language model behavior and have a larger impact as model size increascs. We were surprised we could make so much progress on alignment with dataset this small. This implies that significantly adjusting the behavior of a large language model is feasible with a small dataset, and human input and oversight is feasible in this method of model value alignment.
9

---

Acknowledgments and Disclosure of Funding
Thank you to OpenAl for funding this project in its entirety.
Thank you to the entire OpenAl technical team for developing and maintaining large lan- guage models like GPT-3 that allowed us to conduct our experiments.
Thank you to Sandhini Agarwal and Melanic Subbiah for developing the co-occurrence evaluation suite for GPT-3.
Thank you to John Schulman for developing the tool used for fine-tuning the models used in our experiments.
Thank you to Surge.AIKaI for conducting the human evaluations using their human rating service.
Thank you to Alec Radford, Joshua Achiam, Gretchen Krueger, Steve Dowling, Jan Leike, Lilian Weng, Miles Brundage, Greg Brockman, Mira Murati, Timnit Gebru, Iason Gabriel, Yonadav Shavit, Maarten Sap, Boris Power, Jeff Wu, Ryan Lowe, Elizabeth Barnes, Arvind Neclakantan, Long Ouyang, Peter Welinder, Cullen O'Keefe, and anonymous reviewers for their leedback our earlier versions of this paper. Thank you to Katie Mayer, Luke Miller, and Jack Clark for their help in planning this project.
10

---

References
[ 18 U.S.C. 2241. Chapter 109a sexual abuse. URL http:ooodo...uuee..ooo/.eeee. htt.l/////pelllaaa//////////aaaa
[2] 18 U.S.C. 8 2332b. U.S. Code 2332b Acts of terrorism transcending national bound-
aries. URL https://wwy 1aw. corne wornell. ootoee//eee///sooaffee/////////
33 A. Abid, M. Farooqi, and J. Zou. Persistent antimmmmlin bias in large language models. CoRR, 0bs/2210.05775 2021. URL https://arxiv.orgt wttp://.wrgggaa//1....1331
[4] E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? L In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccc'' '21, page 610623, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383097. doi:
URL https://doi. org/oo..//.... 1145/
R442188 3445922.
5 S. L. Blodgett, S. Barocas, H. D. III, and H. M. Wallach. Language (technology) is power: A critical survey "bias" in NLP. CoRR, abs/2005. 14050, 2020. URL https://aras org/ htt//bb/22005.160 14050.
[6] Bolukbasi, K. Chang, J. Zou, V. Saligrama, and A. Kalai. Man is to com- puler programmer as woman is to homemaker? debiasing word embeddings. CoRR, abs/1607. 2002006520 2016. URL. http://arxiv..
[7] G. Branwen. GPT-3 Creative Fiction, 2021. URL https: www gwern net/GPT-3
[8] B. Brown, B. Mann, N. Ryder, M. Subbiah, Kaplan, P. Dhariwal, A. Nee- lakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Rad- ford, I. Sutskever, and D. Amodei. Language models are few-shot learners. CoRR, abs/2005. 241/2005.14165, 2020. URL attps: :ttps://ante http://axxiv... wtwp:///gaa/...ooo..o// 14165.
[9] CCal. Bus. & Prof. Code 2052. California Legislative Information Califor- nia Penal Code 2052. URL http:::/1eeinfoooeiilll..o wttpauurr..a...oeeeeeeee fov/ffcesec/coees displaySection wwtml//ectionniiiiei 2052. 2lawCode=
[10] Centers for Discase Control and Prevention. Public Health Professionals Gate- way vaccination laws. URL
raccination htmll
[11] Children's Bureau. Child Welfare Information Gateway state laws on child abuse and ne- glect, URL
can/.
[12] Children's Bureau. Child Welfare Information Gateway mandatory reporters of child abuse and neglect, URL https:// /wwpw///wwww/////o//////////o/// wht..deee.aarre..o//ppdDDDD///
pdf.
[13] J. Dhamala, T. Sun, V. Kumar, S. Krishna, Y.
Pruksachatkun, K. Chang, and R. Gupta. BOLD: dataset and metrics for measuring biases in open-ended language generation. CoRR, abs/2101 2201111111 2021. URL attps://arxiv> http:////axxw.oorg//////0//0 attp//bbb/2111..11 11718.
[14] L. Dixon, J. Li, J. Sorensen, N. Thain, and L. Vasserman. Mcasuring and mitigating un- intended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AT, Ethics, and Sociely, ATES '18, page 6773, New York, NY, USA, 2018. Associa- tion for Computing Machinery. ISBN 9781450360128. doi: 10.1145/3278721 .2778799. URL kttps://doi.... org/00 1ttp://00.114//2777222 3278729
[15] Federal Bureau of Investigation. Reports and Publications terrorism 2002/2005 URL https: fbi http:///wwwe....ootttt..rrrerrer 200 2002-2005

---

[16] Federal Trade Commission. No FEAR Act protections against discrimination and other prohibited practices, 2021. URL sttp:///w.......or//ttte----
[17] S. Gehmman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith. Realtoxicilypromptss Evaluating neural toxic degenerat ion in language models. CoRR, 2bs9/0009.11 11462, 2020.
URL attps://arxiv. wwwpiii.vor//abaee arg/abs/2009. 11462.
[18] S. Gururangan, A. Marasovic, S. Swayamdiptaa, K. Lo, Beltagy, D. Downey, and N. A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. CoRR, abs/2004.10964 2020. URL http://ara....../////////0.. 10964. 10964.
[19] A. Holtzman, J. Buys, M. Forbes, and Y. Choi. The curious case of neural text degen- eration. CoRR, abs/ 19s/1999.,9951, 2019. URL attp://arxiv http://awii....aa////////////
[20] D. M. Jr., V. Prabhakaran, J. Kuhlberg, A. Smart, and W. Isaac. Extending the machine learning abstraction boundary: A complex systems approach to incorporate societal context. CoRR, abs/ Abs/2006.096633, 2020. URL https://arxix: wwwpaaaww...or/////////a 096644
[21] Kenton, T. Everitt, L. Weidinger, I. Gabriel, V. Mikulik, and Irving. Alignment of language agents. CoRR, abs/2103.14653 2021. URL http:s://arxiv..oo/ wttpw//w.rr///b//10000/// hbtp///bb///.....////// 14659.
[22] P. P. Liang. I. M. Li, E. Zheng, Y. C. Lim, R. Salakhutdinov, and L. Morency. Towards debiasing sentence representations. CoRR, brg/abs/2077.08//
2020. URL https://arxiv.
[23] A. Liu, M. Sap, X. Lu, S. Swayamdipta, C. Bhagavatula, N. A. Smith, and Y. Choi. On-the-fly controlled text generation with experts and anti-experts, 2021.
[24] S. Mohamed, M. Png, and W. Isaac. Decolonial AI: decolonial theory as sociotechnical foresight artificial intelligence. CoRR, abs/2007 Jbs/2007.04068, 2020. URL https://arxiv. brg/ 2007 04068.
[25] National Conference of State Legislatures. States With Religious and Philosophical Exemptions From School Immunization Requirements. URL https://www ncsl. org/
researchhheallth/seae school- immunization exemption-staat state--sttee-aae aspx
[26] A. Paullada, 1. D. Raji, E. M. Bender, E. Denton, and A. Hanna. Data and its (dis)contents: A survey of dataset development and use in machine learning rescarch. CoRR, abs/20012.0533 2020. URL attps://arxiv wwwvvorggaaae arg/abb/20122.53//
[27] Perspective. Perspective how it works, 2021. URL https: http///www..t http://wwwwwppppetteeeee www.peeppptttppp.....
com/how- 1t- 1ttrooooottt-ss works.
[28] Perspective.
Perspective Developers attributes & languages, 2021. URL https://
support pewportppeeseettiveeee wwtp:/pppwp.pppppappaaaaeeaaaaea -atttttttttee-------aatrrbttteee and- lndl-aaguggeease
[29] V. Sanh, T. Wolf, Y. Belinkov, and A. M. Rush. Learning from others' mistakes: Avoiding dataset biases without modeling them. CoRR, abs/ Abs/2012.01300, 2020. URL
attps: //arxi///axiv.../// http///axwvv.oa/ab/eaaaa wwrg//ab/222.... 01300.
[30] M. Sap, D. Card, S. Gabriel, Y. Choi, and N. A. Smith. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668-1678 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-110 URL https:/wwww.clla brg/ wtggaanthology/P19-111
[31] Surge AI. Human Intelligence. On Demand., 2021. URL https://surgehqqal
(32] United Nations. Himan-rignt
Human Rights.
URL
12

---

[33] United Nations Office of the High Commissioner. Human Rights enhancing equal- ity and countering discrimination, 2021. URL https://www Caw.aroorg//////oooee
aspx.
[34] U.S.
Department of Health & Human Services. Statutory Rape:
A Guide to State Laws and tercourse with Minors. ste guide state and Rexiaa - wtw.r-rrrrrrrrreeemmm minorrs
Reporting Requirements. Sexual In- URL
hhs http:///aapp......ooo..oooo
laws/
[35] U.S. Department of Justice. The United States Department of Justice elder abuse and elder financial exploitation statules. URL https://www. www. wttt::::/wwwjjusticc..... htt.:uutttc..eeeeeeeeeeeeee.
prosecutors/ senccuttor/sttuutee statutes.
[66] U.S. Food & Drug Administration. Buying & Using Medicine Safely medication health fraud. URL attps://wwH. fda. gov/ berruug/bbyiin... using-mediiine
nedication--eer hea teautu
[37] B. Vidgen, A. Harris, D. Nguyen, R. Tromble, S. Hale, and H. Margetts. Chal- lenges and frontiers in abusive content detection. In Proceedings of the Third Work-
shop on Abusive Language Online, pages 80 80-93, Florence, Italy, Aug. 2019. Asso- ciation for Computational Linguistics. doi: 10.18633/v1/W19-350 l/www. aclweb. org/ http//ooooooo//9-5500000
URL https:
[38] A. Xu, E. Pathak, E. Wallace, S. Gururangan, M. Sap, and D. Klein. Detoxifying language models risks marginalizing minorily voices. CoRR, abs/2200.063900 2021.
URL https: // /aaaxvivvvorro...aa orw./oorg/aooo http://b//114.00990000
[39] J. Xu, D. Jn, M. Li, Y. Bourcau, J. Weston, and E. Dinan. Recipes for safety in open- domain chatbots. CoRR, abs/2010.07079, 2020. URL https://arxii httpaaaww.....aass///////eesaaae 2010.
13


Reviews:
[{"review_id": "XJ0ciNcBCPx", "reviewer": "Reviewer_hQiK", "summary": "This paper proposes an approach to aligning language models with target values. The authors construct a small hand-crafted dataset that reflects a set of intended values, then finetune large language models on this dataset. They find that this simple approach significantly improves performance according to a few different metrics.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper addresses an important problem (controlling the outputs of language models, especially in ways that are more aligned with human values). It proposes a simple method for addressing this problem that indeed seems to improve performance significantly. The experiments generally seem convincing, and the improvement seems reasonably large. \n\nThe approach taken by the paper is *extremely* simple and essentially amounts to finetuning (albeit on a dataset constructed in a particular careful way). I think some would view this as a drawback, and it does admittedly makes the paper feel a little on the thin side in terms of originality, but for the most part I think this is a positive thing. Indeed, one of the main takeaways is that one can do a good job using this method without much effort by using an extremely small number of examples (in the appendix they claim that performance starts to converge at around only 60 samples, at least for the 175B model). \n\nThe paper itself is also well-written, with a very detailed appendix. Overall I think it's relatively straightforward, but well-executed, with a couple arguably surprising results, and makes meaningful progress in both addressing and understanding a very important problem. \n\nOther feedback:\n- I think it's an important observation that the approach becomes more effective with model size, but to the extent that larger models are more sample efficient this isn't too surprising. I'd like to know how much of this is due to larger models converging to a higher level of performance vs merely being more sample efficient than smaller models. To be fair, they do say on the last page that future work should assess scaling laws for this problem, which is basically what I'm advocating here. But it seems especially important in this context because it affects how scalable this approach will be moving forward. I'd be less excited about it if peak performance is pretty much the same but converges at 6 samples instead of 60 samples with GPT-4 or whatever.\n- I'm also curious to know how finetuning on this hand-crafted dataset affects performance out-of-distribution. For example, if you have a rather different input format (such as using the 175B model as a chat bot having a long back-and-forth conversation with a human), does it still tend to output more aligned outputs?\n\nI think it would be quite a bit stronger with these sorts of additional experiments, which is why I'm only giving it a 7, but I think it still probably deserves to be accepted.\n\nMinor comments;\n- ALMBFVD is a very cumbersome name, so I would seriously consider changing it. Even just FVD would be much better in my opinion.\n- Many of the figures could be much nicer (partly aesthetically, but especially in terms of ease of understanding what's going on).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper is largely focused on addressing a problem that is societally important. I think it mostly does a good job of this. One point I would probably want it to emphasize somewhat more is that the target values to finetune on are not only subject to some debate, but may also vary from application to application even if there is consensus (e.g. because it may depend on whether a model is making descriptive claims or normative claims). But especially given that they're mainly proposing a general framework/approach rather than, say, a particular dataset to finetune on, this is a pretty minor point.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "SzBvaFpjsmb", "reviewer": "Reviewer_LP2G", "summary": "This paper studies the feasibility of using a small, carefully-constructed corpus to guide large language modeling toward particular values embodied by the corpus. In particular, the authors perform a case study of whether such a process can make GPT-3 more sensitive, according to a Western, US American lens.\n\nThe authors describe an iterative procedure for collecting a dataset that reflects the targeted values and evaluating whether the language model has improved, according to the targeted standards. They apply their procedure to several topics and language models of varying sizes, showing that the process is generally successful for steering language models toward the desired values.\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: This paper is quite original, and is the first of my knowledge to contribute such an analysis for language models of the scale they study.\n\nClarity: I thought this paper was clear. I appreciated the authors’ efforts toward contextualizing the lenses / values that they are coming from, and recognizing that different people and different cultures have different lenses. The description of the data collection process and evaluation was also pretty easy to follow.\n\nQuality / Significance: I thought that this paper was reasonably thorough in controlling for potential confounds. I wish there was a bit more discussion about why high-quality data in general (the “control” text) decreases language model toxicity---do the authors think that this is specific to the particular values they target? I also appreciated the level of detail to which the authors described the data collection and evaluation process. I wish there was more space in the main paper to highlight results across different topics, but the paper is quite content-full as is.\n\nThere are clearly many open questions left: how well does this method work for values that aren’t as widely held? Perhaps this method only works well if the language model has seen text that is broadly representative of the target values during pretraining. How can we have more fine-grained control over what prejudices are replaced with (e.g., the top descriptive word for “Muslim” switching from a bias toward Islamism to a bias toward heterosexuality).\n\nDespite these questions, I think that this study will be of interest to the broader community, and that the work is significant.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, I found the discussion to be satisfactory, and this paper in particular is one that requires such a discussion.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "IXexlOzH6Zn", "reviewer": "Reviewer_gwYe", "summary": "Language models exhibit harmful biases. The authors try to address the toxic behaviors of the GPT-3 model as follows. First, they construct a set of examples that correspond to a large set of desired “values” (e.g., gender equality, realistic beauty standards). The examples are prompt-continuation pairs. Then, they finetune the language model to generate the continuation (which has the desired values) given the prompt. \n\nThe authors find that using a very small set of examples, the post-finetuning models would generate continuations that are less toxic, using both quantitative (human-eval scores and automatic toxicity scores) and qualitative evaluation methods that analyze word associations given a specific gender, religion, or race. \n\nThe paper targets U.S. values only which are not necessarily the universal values; however, the authors acknowledge and point this issue out, in the introduction and the discussion sections. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Issue on formatting: The NeurIPS website says “you must format your submission using the NeurIPS 2021 LaTeX style file which includes a preprint option for non-anonymous preprints posted online” but the font of the submitted article does not follow the provided tex file.\n\n(+) The sensitive topics are chosen according to official sources like the government and the United Nations guidelines. They cover a wide range of categories. \n\n(+) Significant testing of the quantitative results (when comparing the post-finetuning model and the pre-finetuning model) is comprehensive. \n\nLots of critical information, which is required to judge the quality of the paper, is hidden in the 30-page-long appendix; e.g., the methodology for choosing the number of prompts, how specifically the authors wrote the prompts. However, I appreciate the detailed explanations of the prompts in the appendix. \n\n(-) Hyperparameter tuning seems arbitrary (e.g., “training loss weight was 0.1 for the prompt and 1.0 for the completion”, “using learning rate 30x less than the default learning rate” in line 1325). Training details should be expanded. Important parts can be moved to the main text. It’s unclear how long the finetuning lasts. What are the criteria for “convergence”?\n\n(-) Currently the train/dev/test prompts are similar in terms of style. The models can generate better continuations when using top-p sampling (p=0.8) & when there is a prompt that follows a specific format (e.g., one sentence long, usually under 15 words, ending in question marks, etc.). Given the evaluation strategy, it’s unclear whether the improvement in values can transfer to downstream applications.\n\n(-) It’s unclear whether the post-finetuning models still do as well as the pre-finetuning models on GLUE/SuperGLUE classic benchmark tasks.\n\n(-) Missing citations: Nangia et al. (EMNLP 2020), https://arxiv.org/pdf/2010.00133.pdf, which deals with the discussion on categories of biases as well as the evaluation of LM bias. A comprehensive discussion on previous LM-bias-related works should be included; e.g., http://web.cs.ucla.edu/~kwchang/publications_area/.\n\nI would hope for more information related to human factors. \n- Who are the writers? How are the writers trained? \n- How do the authors validate the continuations? \n- Who are the human annotators? How are they trained? \n- How do the authors validate the annotation results? \n\nMinor:\n- Footnote 9: \"top-p controls diversity via nucleus sampling\" but top-p sampling and nucleus sampling are the same thing.\n- A space before bracket citations?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The project is done using American English, and the project caters to a specific set of U.S. values. However, the authors acknowledge the issue and claim that a similar procedure to fix a biased language model can be done in the context of other cultures as well. \n\nIF the post-finetuning models generalize to downstream tasks (i.e., the tasks still achieve the same performance; there is less bias in downstream tasks), then this work would be encouraging for the NLP/ethics community because it claims that using a small set of expert-written examples could greatly debias a language model. \n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3-4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper demonstrates that large language models (i.e., GPT-3) can be relatively cheaply fine-tuned so as to generate text continuations consistent with some specific values/toxicity-related goal. Reviewers had some concerns about how broadly these results would apply, but nonetheless all found the results to be significant and timely.
Decision: Accept (Spotlight)