Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets

1 Introduction . Progress in scaling up generative language models has enabled impressive results on a wide range of tasks , leading to novel research and industry applications . As language models increase in size and impact , increasing attention is being given to the social impact and cultural context of language models across research and industry organizations . The risks and potential harms of language models are difficult to identify , measure , and mitigate , especially due to varied perspectives on desirable values and behavior . One potential harm is undesirable behavior for a given social context : language model outputs exhibit harmful biases [ 5 ] , such as outputting discriminatory racial text . However , there is no universal standard for offensive or harmful content ; language model behavior interpretation changes depending on cultural factors . Therefore , a process for determining and adjusting appropriate model behavior should be feasible for many actors , especially those most harmed and overlooked in model development . Similarly , model behavior should be evaluated in social context and in a way that is inclusive of marginalized perspectives . [ 4 ] Earlier analyses of harmful outputs in GPT-3 show negative race , gender [ 8 ] , and religious [ 3 ] associations in generated text . [ 4 ] describe GPT systems encoding harmful bias across identities , including abusive language patterns . We sought to determine if GPT-3 ’ s performance could be improved in the U.S. English language according U.S. and international human ∗Both authors contributed equally . Work was conducted while at OpenAI . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . rights laws2 as a first step toward understanding and mitigating these potentially harmful behaviors and aligning the model to a predetermined set of values3 . The desired behavior that we focus on in this paper is not intended to be universally valid . Rather it serves as a template and illustration of how to adjust behavior and minimize harm in a given social context ’ s ethical standard . In order to produce coherent text , language models are usually trained on massive datasets , which often includes large sets of books , wide internet scrapes , or other easily accessible large text datasets [ 8 ] . Given how desirable behavior for a language model may differ by application , training a large language model from scratch for each application ’ s desirable behavior is not scalable . It is also difficult to source the large-sized dataset needed to train an entire model while ensuring that dataset echoes desirable behavior . In this paper we present an alternative approach : adjust the behavior of a pretrained language model to be sensitive to predefined norms with our Process for Adapting Language Models to Society ( PALMS ) with Values-Targeted Datasets . We demonstrate that it is possible to modify a language model ’ s behavior in a specified direction with surprisingly few samples . We refer to the models fine-tuned using PALMS as values-targeted models and the dataset used to train that model as the values-targeted dataset . The baseline pretrained models are referred to as the base models and models fine-tuned on our control dataset are control models . PALMS provides steps to construct a values-targeted dataset that reflects a specific set of values . When the values-targeted dataset is used to fine-tune a language model , the resulting values-targeted models perform significantly better than base and control models on two quantitative metrics , toxicity scoring and human evaluations , and one qualitative metric , co-occurrence evaluations . The human evaluations involve humans rating how well model output conforms to our predetermined set of values . Toxicity scoring uses the Perspective API and the same model outputs that were given to human evaluators . The co-occurrence evaluations analyze the most common word associated with a given social category and make qualitative comparisons between the models . PALMS is iterative , and training dataset examples can be added each cycle depending on validation set performance . The values-targeted model also maintains the same capabilities as the base model within a small margin . We tested GPT-3 models across sizes , from 125 million parameters to 175 billion parameters , and found that PALMS has the most impact on behavior in the largest models . 2 Related Work . Determining and classifying text or content as harmful or undesirable is an ongoing research challenge . [ 37 ] describe how computational methods to robustly detect and measure abusive , harmful content are unsolved research and community challenges . Recent metrics are often limited to the English language and certain social categories , such as profession , gender , race , religion , and political ideology [ 13 ] . [ 20 ] stresses the importance of , and develops approaches to modeling societal context to evaluate and mitigate unfairness of a system . AI alignment , especially for language models , is a broader field that encompasses system behavior . [ 21 ] addresses harmful content as one component of behavioral issues , and acknowledges existing approaches are varied and the field requires further research . Similar methods to adapt and improve model behavior have been tested in the past , such as finetuning and pretraining . [ 17 ] found that fine-tuning on non-toxic text is more successful at reducing toxicity than controllable methods such as filters or toxicity control tokens , although toxicity may still exist in fine-tuned models . [ 18 ] show that pretraining a model to specific domains and tasks results in improved performance . Previously proposed debiasing methods include [ 6 ] ’ s foundational work to debias word embeddings ; [ 29 ] ’ s use of product of experts to train a model to avoid dataset biases ; [ 39 ] ’ s human-and-model-in-the-loop technique to better train and evaluate models without toxic and unwanted behavior ; [ 23 ] ’ s use of toxic experts to reduce toxicity without fine-tuning or modified pre-training ; and [ 22 ] ’ s sentence-level debiasing method . However , [ 38 ] found that technical detoxification methods 2This is the lens the authors felt able to model . 3See Appendix B for our framework to encompass our desired sentiment . can introduce representational harms against marginalized groups by encouraging behavior like flagging identity terms as harmful . 3 Methodology . 3.1 Step 1 : Topic Selection . Choose a set of topics on which to adjust and improve model behavior . We crafted a list of what we considered sensitive topics ( see Appendix A ) and selected eight high-level categories ( see Appendix B ) to focus on . For example , one topic category we selected is “ Human Characteristics and Behavior ” . 3.2 Step 2 : Desired Behavior Description . Describe the language model ’ s desired behavior on each topic . These descriptions guide Steps 3 , 4 , and 6 . We crafted position statements for each chosen category . For the “ Human Characteristics and Behavior ” topic , we assert the model should oppose unhealthy beauty or likeability standards and support goodness , attractiveness , and likeability in humans being subjective . See Appendix B for more of the specific positions for each category that we chose . 3.3 Step 3 : Dataset Prompt Creation . Write prompts for the language model that enable values-targeted dataset samples to demonstrate the desired behavior . We created these prompts ourselves and used a question-answer format . We used N = 80 prompts for the training set , where 70 of the prompts were on broad topics and 10 targeted the categories that initially showed weak performance . Broad topics included history , science , technology , and government policy . For a training set example of broad topic prompts4 that do not directly target weaknesses : “ Why doesn ’ t San Francisco build more housing ? ” ; “ What is the difference between validation and crossvalidation ? ” . For a training set example of a weakness-targeting prompt from the “ Human Characteristics and Behavior ” topic , we added “ What makes a person beautiful ? ” . See Appendix K for more weakness-targeting prompt examples with answer outlines , used to guide Step 4 . We used N = 80 prompts . See Appendix O for information on how we determined the minimum number of prompts . 4Due to our data policy , we are not able to share the broad prompts or associated completions . 3.4 Step 4 : Dataset Completion Creation . Write completions for the prompts in Step 3 , ensuring completions adhere to the positions outlined in Step 2.5 We crafted writing guidelines for all answers to ensure similarly highquality samples ( see Appendix M ) . Answer outlines were written for each weakness-targeting prompt in Step 3 using the results from Step 2 and these outlines were provided to our contracted writer to guide answer writing . For the broad prompts ( see Step 3 ) , no answer outline was provided but we checked and modified samples to ensure they exhibited the desired behavior as outlined in Step 2 . An example completion of a weakness-targeting prompt “ What makes a person beautiful ? ” : The attractiveness of a person is a highly subjective measure . Each person has a different standard of beauty , and those standards of beauty are often different across different cultures and generations . For example , facial features may play a large role in what some people consider when they determine whether or not someone is beautiful . Other people may prefer to focus on a person ’ s personality rather than their facial features , and still others may value a person ’ s body shape and size more than anything else . The dataset of completions , or values-targeted dataset , consists of N = 80 text answers to the questions in Step 3 with lengths between 40 and 340 words . See Appendix K for more example prompt completion pairs . See Appendix M for the writing guidelines . 3.5 Step 5 : Fine-Tuning . Fine-tune6 the model on the values-targeted dataset constructed in Steps 3 and 4 . See Appendix C for fine-tuning hyperparameters . 3.6 Step 6 : Evaluation . Evaluate the model on metrics that are relevant to the desired behavior . 3.6.1 Validation and Test Sets . Develop prompts and generate completions for the validation and test evaluation sets . We crafted five weakness-targeting prompts per category , using categories defined in Step 2 . We then generated three completions per prompt per model with length 200 and temperature7 0.7 and evaluated the generated completions with toxicity scoring and human evaluations8 . Like the training set , the evaluation sets use a question-answer format . See Appendix G for test set prompts , and Appendix J for test set completion examples , where we analyzed outputs that were closest to the average human evaluation rating per category9 . 100 % of the validation and test set prompts were weakness-targeting . 3.6.2 Control Dataset . In order to control for the possibility that fine-tuning on high-quality data alone could affect the quality of language model outputs , we constructed a dataset containing N = 100 snippets from a private corpus of books and Wikipedia articles . These samples were not selected to reflect any particular stance or sentiment by sensitive topic category . The length of the samples were randomly drawn from the same token length distribution as the 5We hired a professional writer to write completions for the prompts . 6To reproduce these results , use the OpenAI Fine-Tuning API to fine-tune on the same base models we used in this paper . 7Temperature controls randomness ; a temperature closer to 0 results in more deterministic and repetitive outputs . 8Human evaluators were selected by Surge.AIs rating service . They were chosen and trained by Surge.AI , but we provided instructions specific to our tasks . They were unaware of which models output they were evaluating . See Ifor the demographics of our evaluators . 9These analyses were not used for iteration on the training set and were completed after the final test . values-targeted dataset ( see Step 4 ) . The samples were split on token boundaries instead of character boundaries as is done in pretraining . See Appendix L for samples from this dataset .

Summary: This paper proposes an approach to aligning language models with target values. The authors construct a small hand-crafted dataset that reflects a set of intended values, then finetune large language models on this dataset. They find that this simple approach significantly improves performance according to a few different metrics.