Training Socially Aligned Language Models in Simulated Human Society
Authors: name: Ruibo Liu, name: Ruixin Yang, name: Chenyan Jia, name: Ge Zhang, name: Denny Zhou, name: Andrew M. Dai, name: Diyi Yang, name: Soroush Vosoughi

Social alignment in AI systems aims to ensure that these models behave
according to established societal values. However, unlike humans, who derive
consensus on value judgments through social interaction, current language
models (LMs) are trained to rigidly replicate their training corpus in
isolation, leading to subpar generalization in unfamiliar scenarios and
vulnerability to adversarial attacks. This work presents a novel training
paradigm that permits LMs to learn from simulated social interactions. In
comparison to existing methodologies, our approach is considerably more
scalable and efficient, demonstrating superior performance in alignment
benchmarks and human evaluations. This paradigm shift in the training of LMs
brings us a step closer to developing AI systems that can robustly and
accurately reflect societal norms and values.