[36] Aaron Wildavsky. Choosing preferences by constructing institutions: A cultural theory of preference formation. American political science review, 1987. (p. 2)
[37] CA Coello. Handling preferences in evolutionary multiobjective optimization: A survey. In CEC, 2000. (p. 2)
[38] Shalom H Schwartz et al. An overview of the schwartz theory of basic values. Online readings in Psychology and Culture, 2012. (p. 2)
[39] Marcos Nadal and Anjan Chatterjee. Neuroaesthetics and art's diversity and universality. Wiley Interdisciplinary Reviews: Cognitive Science, 2019. (p. 2)
[40] David Lopez-Paz, Diane Bouchacourt, Levent Sagun, and Nicolas Usunier. Measuring and signing fairness as performance under multiple stakeholder distributions. arXiv preprint, 2022. (p. 2)
[41] Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint, 2022. (p. 2)
[42] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional AI: Harmlessness from AI feedback. arXiv preprint, 2022. (p. 2)
[43] Robert Irvine, Douglas Boubert, Vyas Raina, Adrian Liusie, Vineet Mudupalli, Aliaksei Korshuk, Zongyi Liu, Fritz Cremer, Valentin Assassi, Christie-Carol Beauchamp, et al. Rewarding chatbots for real-world engagement with millions of users. arXiv preprint, 2023. (p. 2)
[44] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. (pp. 2, 5, 30, and 31)
[45] Leon Barrett and Srini Narayanan. Learning all optimal policies with multiple criteria. In ICML, 2008. (pp. 2, 3, and 9)
[46] Kaiwen Li, Tao Zhang, and Rui Wang. Deep reinforcement learning for multiobjective optimization. IEEE-T-CYBERNETICS, 2020. (pp. 2, 3, and 9)
[47] Michiel A. Bakker, Martin J Chadwick, Hannah Sheahan, Michael Henry Tessler, Lucy Campbell-Gillingham, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matthew Botvinick, and Christopher Summerfield. Fine-tuning language models to find agreement among humans with diverse preferences. In NeurIPS, 2022. (p. 2)
[48] Aviv Ovadya. Generative CI through collective response systems. arXiv preprint, 2023. (p. 2)
[49] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint, 2022. (pp. 2, 5, 9, and 31)
[50] Grgur Kovac, Masataka Sawayama, Rémy Portelas, Cédric Colas, Peter Ford Dominey, and Pierre-Yves Oudeyer. Large language models as superpositions of cultural perspectives. arXiv preprint, 2023. (p. 2)
[51] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint, 2021. (p. 2)
[52] Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale. Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback. arXiv preprint, 2023. (pp. 2, 9, 10, and 22)