Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets

1 Introduction . Progress in scaling up generative language models has enabled impressive results on a wide range of tasks , leading to novel research and industry applications . As language models increase in size and impact , increasing attention is being given to the social impact and cultural context of language models across research and industry organizations . The risks and potential harms of language models are difficult to identify , measure , and mitigate , especially due to varied perspectives on desirable values and behavior . One potential harm is undesirable behavior for a given social context : language model outputs exhibit harmful biases [ 5 ] , such as outputting discriminatory racial text . However , there is no universal standard for offensive or harmful content ; language model behavior interpretation changes depending on cultural factors . Therefore , a process for determining and adjusting appropriate model behavior should be feasible for many actors , especially those most harmed and overlooked in model development . Similarly , model behavior should be evaluated in social context and in a way that is inclusive of marginalized perspectives . [ 4 ] Earlier analyses of harmful outputs in GPT-3 show negative race , gender [ 8 ] , and religious [ 3 ] associations in generated text . [ 4 ] describe GPT systems encoding harmful bias across identities , including abusive language patterns . We sought to determine if GPT-3 ’ s performance could be improved in the U.S. English language according U.S. and international human ∗Both authors contributed equally . Work was conducted while at OpenAI . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . rights laws2 as a first step toward understanding and mitigating these potentially harmful behaviors and aligning the model to a predetermined set of values3 . The desired behavior that we focus on in this paper is not intended to be universally valid . Rather it serves as a template and illustration of how to adjust behavior and minimize harm in a given social context ’ s ethical standard . In order to produce coherent text , language models are usually trained on massive datasets , which often includes large sets of books , wide internet scrapes , or other easily accessible large text datasets [ 8 ] . Given how desirable behavior for a language model may differ by application , training a large language model from scratch for each application ’ s desirable behavior is not scalable . It is also difficult to source the large-sized dataset needed to train an entire model while ensuring that dataset echoes desirable behavior . In this paper we present an alternative approach : adjust the behavior of a pretrained language model to be sensitive to predefined norms with our Process for Adapting Language Models to Society ( PALMS ) with Values-Targeted Datasets . We demonstrate that it is possible to modify a language model ’ s behavior in a specified direction with surprisingly few samples . We refer to the models fine-tuned using PALMS as values-targeted models and the dataset used to train that model as the values-targeted dataset . The baseline pretrained models are referred to as the base models and models fine-tuned on our control dataset are control models . PALMS provides steps to construct a values-targeted dataset that reflects a specific set of values . When the values-targeted dataset is used to fine-tune a language model , the resulting values-targeted models perform significantly better than base and control models on two quantitative metrics , toxicity scoring and human evaluations , and one qualitative metric , co-occurrence evaluations . The human evaluations involve humans rating how well model output conforms to our predetermined set of values . Toxicity scoring uses the Perspective API and the same model outputs that were given to human evaluators . The co-occurrence evaluations analyze the most common word associated with a given social category and make qualitative comparisons between the models . PALMS is iterative , and training dataset examples can be added each cycle depending on validation set performance . The values-targeted model also maintains the same capabilities as the base model within a small margin . We tested GPT-3 models across sizes , from 125 million parameters to 175 billion parameters , and found that PALMS has the most impact on behavior in the largest models . 2 Related Work . Determining and classifying text or content as harmful or undesirable is an ongoing research challenge . [ 37 ] describe how computational methods to robustly detect and measure abusive , harmful content are unsolved research and community challenges . Recent metrics are often limited to the English language and certain social categories , such as profession , gender , race , religion , and political ideology [ 13 ] . [ 20 ] stresses the importance of , and develops approaches to modeling societal context to evaluate and mitigate unfairness of a system . AI alignment , especially for language models , is a broader field that encompasses system behavior . [ 21 ] addresses harmful content as one component of behavioral issues , and acknowledges existing approaches are varied and the field requires further research . Similar methods to adapt and improve model behavior have been tested in the past , such as finetuning and pretraining . [ 17 ] found that fine-tuning on non-toxic text is more successful at reducing toxicity than controllable methods such as filters or toxicity control tokens , although toxicity may still exist in fine-tuned models . [ 18 ] show that pretraining a model to specific domains and tasks results in improved performance . Previously proposed debiasing methods include [ 6 ] ’ s foundational work to debias word embeddings ; [ 29 ] ’ s use of product of experts to train a model to avoid dataset biases ; [ 39 ] ’ s human-and-model-in-the-loop technique to better train and evaluate models without toxic and unwanted behavior ; [ 23 ] ’ s use of toxic experts to reduce toxicity without fine-tuning or modified pre-training ; and [ 22 ] ’ s sentence-level debiasing method . However , [ 38 ] found that technical detoxification methods 2This is the lens the authors felt able to model . 3See Appendix B for our framework to encompass our desired sentiment . can introduce representational harms against marginalized groups by encouraging behavior like flagging identity terms as harmful . 3 Methodology . 3.1 Step 1 : Topic Selection . Choose a set of topics on which to adjust and improve model behavior . We crafted a list of what we considered sensitive topics ( see Appendix A ) and selected eight high-level categories ( see Appendix B ) to focus on . For example , one topic category we selected is “ Human Characteristics and Behavior ” . 3.2 Step 2 : Desired Behavior Description . Describe the language model ’ s desired behavior on each topic . These descriptions guide Steps 3 , 4 , and 6 . We crafted position statements for each chosen category . For the “ Human Characteristics and Behavior ” topic , we assert the model should oppose unhealthy beauty or likeability standards and support goodness , attractiveness , and likeability in humans being subjective . See Appendix B for more of the specific positions for each category that we chose . 3.3 Step 3 : Dataset Prompt Creation . Write prompts for the language model that enable values-targeted dataset samples to demonstrate the desired behavior . We created these prompts ourselves and used a question-answer format . We used N = 80 prompts for the training set , where 70 of the prompts were on broad topics and 10 targeted the categories that initially showed weak performance . Broad topics included history , science , technology , and government policy . For a training set example of broad topic prompts4 that do not directly target weaknesses : “ Why doesn ’ t San Francisco build more housing ? ” ; “ What is the difference between validation and crossvalidation ? ” . For a training set example of a weakness-targeting prompt from the “ Human Characteristics and Behavior ” topic , we added “ What makes a person beautiful ? ” . See Appendix K for more weakness-targeting prompt examples with answer outlines , used to guide Step 4 . We used N = 80 prompts . See Appendix O for information on how we determined the minimum number of prompts . 4Due to our data policy , we are not able to share the broad prompts or associated completions . 3.4 Step 4 : Dataset Completion Creation . Write completions for the prompts in Step 3 , ensuring completions adhere to the positions outlined in Step 2.5 We crafted writing guidelines for all answers to ensure similarly highquality samples ( see Appendix M ) . Answer outlines were written for each weakness-targeting prompt in Step 3 using the results from Step 2 and these outlines were provided to our contracted writer to guide answer writing . For the broad prompts ( see Step 3 ) , no answer outline was provided but we checked and modified samples to ensure they exhibited the desired behavior as outlined in Step 2 . An example completion of a weakness-targeting prompt “ What makes a person beautiful ? ” : The attractiveness of a person is a highly subjective measure . Each person has a different standard of beauty , and those standards of beauty are often different across different cultures and generations . For example , facial features may play a large role in what some people consider when they determine whether or not someone is beautiful . Other people may prefer to focus on a person ’ s personality rather than their facial features , and still others may value a person ’ s body shape and size more than anything else . The dataset of completions , or values-targeted dataset , consists of N = 80 text answers to the questions in Step 3 with lengths between 40 and 340 words . See Appendix K for more example prompt completion pairs . See Appendix M for the writing guidelines . 3.5 Step 5 : Fine-Tuning . Fine-tune6 the model on the values-targeted dataset constructed in Steps 3 and 4 . See Appendix C for fine-tuning hyperparameters . 3.6 Step 6 : Evaluation . Evaluate the model on metrics that are relevant to the desired behavior . 3.6.1 Validation and Test Sets . Develop prompts and generate completions for the validation and test evaluation sets . We crafted five weakness-targeting prompts per category , using categories defined in Step 2 . We then generated three completions per prompt per model with length 200 and temperature7 0.7 and evaluated the generated completions with toxicity scoring and human evaluations8 . Like the training set , the evaluation sets use a question-answer format . See Appendix G for test set prompts , and Appendix J for test set completion examples , where we analyzed outputs that were closest to the average human evaluation rating per category9 . 100 % of the validation and test set prompts were weakness-targeting . 3.6.2 Control Dataset . In order to control for the possibility that fine-tuning on high-quality data alone could affect the quality of language model outputs , we constructed a dataset containing N = 100 snippets from a private corpus of books and Wikipedia articles . These samples were not selected to reflect any particular stance or sentiment by sensitive topic category . The length of the samples were randomly drawn from the same token length distribution as the 5We hired a professional writer to write completions for the prompts . 6To reproduce these results , use the OpenAI Fine-Tuning API to fine-tune on the same base models we used in this paper . 7Temperature controls randomness ; a temperature closer to 0 results in more deterministic and repetitive outputs . 8Human evaluators were selected by Surge.AIs rating service . They were chosen and trained by Surge.AI , but we provided instructions specific to our tasks . They were unaware of which models output they were evaluating . See Ifor the demographics of our evaluators . 9These analyses were not used for iteration on the training set and were completed after the final test . values-targeted dataset ( see Step 4 ) . The samples were split on token boundaries instead of character boundaries as is done in pretraining . See Appendix L for samples from this dataset .

Summary: This paper studies the feasibility of using a small, carefully-constructed corpus to guide large language modeling toward particular values embodied by the corpus. In particular, the authors perform a case study of whether such a process can make GPT-3 more sensitive, according to a Western, US American lens.  The authors describe an iterative procedure for collecting a dataset that reflects the targeted values and evaluating whether the language model has improved, according to the targeted standards. They apply their procedure to several topics and language models of varying sizes, showing that the process is generally successful for steering language models toward the desired values.