SlowGuess/NAACL_2022_OCR huggingface dataset:
Does Scaling-up LM help? We further investigate whether simply scaling up LMs can improve human values alignment. As shown in Table 2, scaled-up GPT-3 LM still suffers low alignment with human values even if we provide few-shot demonstrations (two positive and two negative). This demonstrates that the success of GPT-3 on knowledge-intensive tasks can not be fully transferred to tasks requiring value judgements (Brown et al., 2020). Without deliberate optimization, the benefit of fine-tuning GPT-3 on all data is not on par with SENSEI, but we find there is significant improvement over MLE on GPT-2 medium, potentially owing to the enlarged model capacity ( $\approx$ 4x larger  $\rightarrow$ $30\%$  improvement).
Table 2: Benchmark results of SENSEI on three human values alignment tasks. Compared with prior arts, SENSEI improves alignment performance by at most  $24\%$  in accuracy (ACC) and  $8\%$  in ROUGE-L (R-L), with affordable "alignment tax" (Askell et al., 2021) in perplexity (PPL). We also report the results of two naive methods (MLE training and Data Filtering), and GPT-3 (few-shot and fine-tuned) $^{12}$  for reference. We bold the best performing and underline the second best results (GPT-3 not included as it uses a much larger model (babbage, 1.3B)).
Task	Moral Story (25% Training Data)
+ # of Demo(s)	1 (▲)no demo	1 (▼)	2 (▲▼)	2 (▼▲)
MLE	↑ 1.29 30.6	↓ 1.07	↓ 2.17	↑ 1.23
Data Filtering	↑ 0.13 40.9	↓ 1.44	↓ 1.58	↑ 2.19
DiaIoGPT	↑ 0.13 46.3	↓ 1.44	↓ 2.58	↓ 1.19
Sensei (A+C)	↑ 0.10 53.7	↓ 0.07	↓ 0.73	↑ 0.93
Sensei (A)	↑ 0.14 49.4	↓ 0.12	↓ 1.01	↑ 1.11
GPT-3	↑ 0.10 61.5	↓ 0.20	↑ 1.1	↑ 3.4
Table 3: Robustness evaluation when using in-context demonstrations as prompts to query the LMs. SENSEI (Actor + Critic) is the most robust method (in alignment accuracy) under perturbation strategies on prompts, such as ending changing (ending in  $\triangle$ positive or  $\nabla$ negative demonstration), and number of in-context demonstrations (two pairs v.s. one demonstration).
4.2 SENSEI is Robust in Few-shot Scenarios
Prompt-based learning has been demonstrated to be useful for LMs to perform well, especially in few-shot scenarios (Liu et al., 2021a). When provided a prompt that contains a few demonstrations, the LM is more likely to recall similar data it has seen during training, which is also interpreted as introducing necessary inductive bias (Gao et al., 2021; Liu et al., 2021a). However, such prompt-based learning is reported to be non-robust: The generation after the prompt tends to closely correlate with the attribute of the demonstrations, which may be undesirable. For example, if we query MLE-trained GPT-2 with "My friend Amy is in trouble. I should not help her. My mum needs my help. I should",
the generation could be "not help her". This non-robustness could lead to unethical generations, and can be exploited for adversarial attacks.
In Table 3, we prepare several perturbation strategies on prompts to mislead few-shot trained LM generation. In general, SENSEI with both Actor and Critic is the least influenced method. Changing the ending demonstration seems to have significant influence on the alignment accuracy: the LM tends to generate sentences whose attribute is similar to the demonstrations near the end of the prompt, which can explain why negative-ending prompts can often lead to decrease in alignment accuracy (Zhao et al., 2021). More demonstrations help the in-context learning but they become more significant in zero-shot settings (see GPT-3 results).
4.3 Values Transfer Learning of SENSEI
Since data labeled with human values is costly and scarce, we explore whether the alignment on one value can be extended/transferred to another, which investigates the generalizability of SENSEI on unseen values. In Figure 2 we show two transfer matrices of alignment accuracy, from seen values (rows) to unseen ones (columns). SENSEI has better generalization, especially in Morality (M) and Deontology (D), and Data Filtering seems to obtain more gains on generalization from Non-Toxicity (N), potentially because the toxicity dataset (i.e., RealToxicityPrompts) covers some of the other values in an implicit way.
We conducted human evaluation on Amazon Mechanical Turk (MTurk) to validate the quality of SENSEI alignment. In total 210 participants were randomly assigned to evaluate the three tasks. Participants (57.1% male, 41.9% female, 1% unanswered) were all from the United States and above 18 years old, with an average age of 31.5 years old. Each participant was paid 1 dollar for completing 16 questions in each questionnaire (average completion time per questionnaire was about 5.07 minutes). They were properly informed that the collected data would be used for research purposes in the consent form at the beginning.
Results. We conducted paired sample  $t$ -tests to examine how much gain can be achieved by different alignment methods, in terms of 1) Alignment (i.e., "How much do you agree that the generated text is aligned with the human value:
Figure 2: Transfer learning on human values of (a) SENSEI, and (b) Data Filtering. M: Morality; D: Deontology; N: Non-Toxicity; HHH: Helpful, Honest, and Harmless; T: Trustfulness. We use the datasets mentioned in §3. Accuracy: alignment accuracy.
morality/deontology/non-toxicity?" The answer is a 7-point Likert scale from 1-totally disagree to 7- totally agree), 2) Readability (i.e., "How similar is the text to human-generated text?" From 1-not similar at all to 7-very similar), and 3) Overall quality of the generated text, from 1-low to 7-high.
As shown in Table 4, in terms of alignment performance, SENSEI achieves statistically significant improvements over the MLE baseline, while DialogGPT only obtains significant result in nontoxicity alignment. For readability, both SENSEI and baselines are rated lower than the MLE method but not significantly; we take this as the tax of alignment. SENSEI and DialogGPT both bring benefits in alignment with respect to the overall ratings, but the improvement from DialogGPT is not significant for deontology  $(p = 0.25)$ . These results further confirm that SENSEI better aligns the generation with human values with affordable "alignment tax".
Value Judgement in LMs. By querying LMs with manually created (Petroni et al., 2019; Kassner and Schütze, 2020) or automatically generated prompts (Shin et al., 2020), many studies present systematic analysis on how well pre-trained LMs can memorize knowledge in different domains, such as temporal numbers (Lin et al., 2020; Qin et al., 2021), abductive inference (Bhagavatula
Moral Stories	ETHIC: Deontology	RealToxicityPrompts
MLE	DF	DG	SENSEI	MLE	DF	DG	SENSEI	MLE	DF	DG	SENSEI
Alignment	Mean	4.77	5.22	4.81	5.25	4.57	4.85	4.73	4.91	5.23	5.25	5.45	5.61
p	-	.00*	.07	.00*	-	.04*	0.06	.03*	-	.30	.03*	.00*
Readability	Mean	5.64	5.52	5.31	5.42	5.33	5.27	5.19	5.25	4.96	4.88	4.93	4.90
p	-	.10	.05	.09	-	.23	.10	.22	-	.12	.18	.14
Overall	Mean	5.13	5.33	4.97	5.39	4.72	4.85	4.77	4.93	5.07	5.13	5.22	5.30
p	-	.02*	.05	.00*	-	.09	.25	.03*	-	.10	.02*	.00*
Table 4: Human evaluation results on Alignment, Readability, and Overall quality of SENSEI and other baselines. DF: Data Filtering. DG: DialogGPT. All results are compared with the MLE as it is the default pre-training method for most LMs. Scores are on a scale from 1-7.  $p$  value describes the significance of difference. (* corresponds to  $p < {0.05}$  ).
et al., 2020), and emotion reflection (Sap et al., 2019). All these works focus more on on the so-called "descriptive knowledge" (Emelin et al., 2021), while recent work have started to pay special attention to whether LMs are well-encoded with proper social values, which are typically abstract and sophisticated. For example, there are many studies on bias in NLP, including gender bias (Wang et al., 2019; Zhao et al., 2017), race bias (Sheng et al., 2019a), sentiment bias (Sheng et al., 2021; Huang et al., 2020), and etc. Everitt et al. (2018) give a review of Artificial General Intelligence (AGI) safety literature, discussing common problems for designing safe AGI caring about shared human values.
Human Values Alignment of LMs. Aligning human and language model objectives is seen as especially important for "embodied" AI agents which learn through active interaction with their environment (Tamkin et al., 2021; Kenton et al., 2021; Everitt et al., 2018). By continuously asking human feedback during evaluation, Christiano et al. (2017) are able to train an RL agent that is aware of human preferences. Irving et al. (2018) attempt to address AI safety and ethics problems by using two RL agents to debate and have it judged by humans. Aiming to tackle larger scale alignment problems, researchers have tried to decompose the problem into sub-problems (e.g., recursively summarizing chapters of a book to align with human preference) (Wu et al., 2021), or deploy a sequence of models while keeping humans in the loop (Leike et al., 2018). All these methods can be seen as online alignment methods, as they require human periodic human involvement. SENSEI, instead, learns human values from offline
data, making it less costly and easier to deploy.
SENSEI can be limited by the LM that it uses — for instance, in few-shot learning scenarios, the total length of in-context demonstrations + context is limited by the max sequence length of the LM used. Additionally, our work is focused on English, and SENSEI may require additional resources to accommodate the shared values in other languages and cultures. To handle cases where the context and generation are in different languages (e.g., machine translation to align with human values), SENSEI may require non-trivial modifications of its architecture. One could potentially extend SENSEI to these scenarios using multi-lingual sequence-to-sequence models such as multilingual-T5 (Xue et al., 2021).
In this work, we proposed SENSEI, a novel training framework aimed at aligning LM generation with human values. Given offline alignment datasets with human demonstrations, SENSEI jointly learns a reward distributor (Critic) and a conditional generator (Actor). Compared to several baselines, SENSEI shows superior performance on three human value alignment datasets and additional benefits for transfer learning of unseen human values.
Future work could explore more fine-grained human values and the value transfer ability of SENSEI on a larger scale. Another direction is to further study the integration of SENSEI with full-size foundation models, like GPT-3 (175B), or DeepMind's Gopher (Rae et al., 2021), to explore SENSEI's potential.
The goal of SENSEI is to provide a general-purpose human values alignment framework for generative LMs. Still, the generation of SENSEI can be affected by certain biases from the LM it is based on (Rae et al., 2021; Liu et al., 2021b), though these biases may be partially mitigated by the alignment itself. Another major ethical consideration is that SENSEI can mimic undesirable attributes of the target alignment demonstrations that could be non-contemporary and do not represent current norms and practices (Liu et al., 2021d; Sheng et al., 2019b)—and SENSEI has no scheme to diagnose these problems. Furthermore, our experiments and analysis are done in English, and therefore we do not claim that our findings will generalize across all languages and cultures, although our framework has the potential to be extended to other languages through necessary modifications.
Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. 2021. A general language assistant as a laboratory for alignment. ArXiv preprint, abs/2112.00861.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen-tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of "bias" in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454-5476, Online. Association for Computational Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. 2020. On the weaknesses of reinforcement learning for neural machine translation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 4299-4307.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics.
Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020. Plug and play language models: A simple approach to controlled text generation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Denis Emelin, Ronan Le Bras, Jena D. Hwang, Maxwell Forbes, and Yejin Choi. 2021. Moral stories: Situated reasoning about norms, intents, actions, and their consequences. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 698-718, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tom Everitt, Gary Lea, and Marcus Hutter. 2018. AGI safety literature review. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden, pages 5441-5449. ijcai.org.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3816-3830, Online. Association for Computational Linguistics.
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356-3369, Online. Association for Computational Linguistics.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 1856-1865. PMLR.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021. Aligning AI with shared human values. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. 2020. Reducing sentiment bias in language models via counterfactual evaluation. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 65-83, Online. Association for Computational Linguistics.
Geoffrey Irving, Paul Christiano, and Dario Amodei. 2018. Ai safety via debate. ArXiv preprint, abs/1805.00899.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics.
Nora Kassner and Hinrich Schütze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7811-7818, Online. Association for Computational Linguistics.
Zachary Kenton, Tom Everitt, Laura Weidinger, Jason Gabriel, Vladimir Mikulik, and Geoffrey Irving. 2021. Alignment of language agents. ArXiv preprint, abs/2103.14659.
Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019. Ctrl: A conditional transformer language model for controllable generation. ArXiv preprint, abs/1909.05858.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466.
Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. 2018. Scalable agent
alignment via reward modeling: a research direction. ArXiv preprint, abs/1811.07871.
Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xiang Ren. 2020. Birds have four legs?! NumerSense: Probing Numerical Commonsense Knowledge of Pre-Trained Language Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6862-6868, Online. Association for Computational Linguistics.
Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. ArXiv preprint, abs/2109.07958.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021a. Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing. ArXiv preprint, abs/2107.13586.
Ruibo Liu, Chenyan Jia, and Soroush Vosoughi. 2021b. A transformer-based framework for neutralizing and reversing the political polarity of news articles. Proc. ACM Hum.-Comput. Interact., 5(CSCW1).
Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, Lili Wang, and Soroush Vosoughi. 2021c. Mitigating political bias in language models through reinforced calibration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14857-14866.
Ruibo Liu, Lili Wang, Chenyan Jia, and Soroush Vosoughi. 2021d. Political depolarization of news articles using attribute-aware word embeddings. Proceedings of the International AAAI Conference on Web and Social Media, 15(1):385-396.
Ruibo Liu, Jason Wei, and Soroush Vosoughi. 2021e. Language model augmented relevance score. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6677-6690, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint, abs/1907.11692.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381-2391, Brussels, Belgium. Association for Computational Linguistics.
Volodymyr Mnih, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Timothy P. Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In Proceedings of the 33nd International Conference
on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1928-1937. JMLR.org.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc G. Bellemare. 2016. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 1046-1054.
Richard Yuanzhe Pang and He He. 2021. Text generation by learning from demonstrations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Fabio Petroni, Tim Rocttäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
Lianhui Qin, Aditya Gupta, Shyam Upadhyay, Luheng He, Yejin Choi, and Manaal Faruqui. 2021. TIME-DIAL: Temporal commonsense reasoning in dialog. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7066-7076, Online. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. ArXiv preprint, abs/2112.11446.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints.