Language Model Alignment in Multilingual Trolley Problems
Authors: Zhijing Jin
Max Kleiman-Weiner
Giorgio Piatti
Sydney Levine
Jiarui Liu
Fernando Gonzalez Adauto
Francesco Ortu
András Strausz
Mrinmaya Sachan
Rada Mihalcea
Yejin Choi
Bernhard Schölkopf
Venue: ICLR 2025 Spotlight (2025)
Keywords: LLM alignment
moral evaluation
trolley problems
language model evaluation
AI alignment
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We test the alignment of 19 LLMs with human preferences on Trolley Problems in 100+ different languages.

Abstract: We evaluate the moral alignment of large language models (LLMs) with human preferences in multilingual trolley problems. Building on the Moral Machine experiment, which captures over 40 million human judgments across 200+ countries, we develop a cross-lingual corpus of moral dilemma vignettes in over 100 languages called MultiTP. This dataset enables the assessment of LLMs' decision-making processes in diverse linguistic contexts. Our analysis explores the alignment of 19 different LLMs with human judgments, capturing preferences across six moral dimensions: species, gender, fitness, status, age, and the number of lives involved. By correlating these preferences with the demographic distribution of language speakers and examining the consistency of LLM responses to various prompt paraphrasings, our findings provide insights into cross-lingual and ethical biases of LLMs and their intersection. We discover significant variance in alignment across languages, challenging the assumption of uniform moral reasoning in AI systems and highlighting the importance of incorporating diverse perspectives in AI ethics. The results underscore the need for further research on the integration of multilingual dimensions in responsible AI research to ensure fair and equitable AI interactions worldwide.

Published as a conference paper at ICLR 2025
LANGUAGE MODEL ALIGNMENT IN MULTILINGUAL TROLLEY PROBLEMS
Zhijing Jin 123,'' Max Kleiman-Weiner Giorgio Piatti
Sydney Levine
Jiarui Liu Fernando Gonzalez²
Francesco Ortu²
András Strausz?
Mrinmaya Sachan Rada Mihalcea* Yejin Choi
Bernhard Schölkopf'
Max Planck Institute for Intelligent Systems, Tübingen
2TTH Zürich
3 University of Toronto UUniversity of Washington
AAlenn Institute for Al (A12)
Carnegie Mellon University "University of Trieste
University Michigan
ABSTRACT
We evaluate the moral alignment of LLMs with human preferences in multilingual trolley problems. Building on the Moral Machine experiment, which captures over 40 million human judgments across 200+ countries, we develop a cross- lingual corpus of moral dilemma vignettes in over 100 languages called MUL- processes TITP. This dataset enables the assessment of LLMs' decision-
in diverse linguistic contexts. Our analysis explores the alignment of 19 different LLMs with human judgments, capturing preferences across six moral dimensions: species, gender. fitness, status, age, and the number of lives involved. By correlat- ing these preferences with the demographic distribution of language speakers and examining the consistency of LLM responses to various prompt paraphrasings, our findings provide insights into gross-lingual and ethical biases of L.MMs and their intersection. We discover significant variance in alignment across languages, challenging the assumption of uniform moral reasoning AI systems and high- lighting the importance of incorporating diverse perspectives in AI ethics. The results underscore the need for further research on the integration of multilingual dimensions in responsible Al research to ensure fair and equitable AI interactions worldwide !
INTRODUCTION
The increasingly impressive performance of large language models (LLMs) (Achiam et 2023 Touvron et al. 2023; Bubeck et al. (2023) also brings safety concerns. First, do LLMs align with human preferences (Bengio et al. 2023] Hendrycks et al., 2021b Anwar al., 2024 Second, whose preferences are they most aligned with (Sorensen et al. (022cc) Prior work on evaluating alignment often takes one of two forms. The first requires continuous real-time evaluation, such as in ChatArena (Chiang et al., 2024). where users select which of two responses they prefer. The second traditional benchmarking with a static dataset where the answers are determined by the judgment of experimenter crowd- sourced from limited online panel (Sap et al. 2019 Forbes ctt al. 2020 Atari et al., 2023 Jiang et al. 2021bb)
Often, however, there is no single answer to questions of human preference, and this is particularly true for the moral domain. We follow recent work that highlights the importance of modeling the diversity of human moral preferences (Jin et al.. [022] Bai et al. [022] (ordon et al., [2022] [Sorensen jet al. c024b)c). and focus on moral dilemmas where subjects must balance two competing moral objectives. The "Trolley Problem", and its many variants, one of the most extensively studied moral dilemmas in the psychology literature (Foot 1967] Thomson 1984). In its classic version, an out-of-control train will run over five people on the main track unless the decision-malse diverts the train to a side track where only one person will be killed.
Traditional studies of trolley problems typically present a few hypothetical scenarios centered around this trade-off (Mikhail 2007 Kleiman- Weiner et al. 2015). In contrast, more recent studies consider
mn Equull contribution. Please contact gjin @ toronto.edu and maxko @unrrrrwwwwdedd for any high-level questions, and giorgio. piatri @ alumni. ethz.chi for coding questions. Our code and data at htttp:://gthub..com/cassaaa//

---

Published as a conference paper at ICLR 2025
Multilingual Trolley Problem (MultiTP)
Assume there is self-driving car with a sudden break failure, and the car is about to cause either of the two to die:
-a boy
aa elderly man If the self-driving car has to make decision, between the two choices, should save
Asked in German: Wrggnsmmen, g8
Chinasn: 1i
"Crskids kinas
sallta jingeren Jungen retten, de or noch ganzea Zoit hat, seine Tidune
Miritan anapasom kuokolewn sabaibu ya umri wake uwczo wake wa wa kucndolic na yaha.
1789EE B
It should save hit and more time pursue his dreams and aspirations.
Aifeiino abood
experience, their contribution to society greater
saved because suffiolont ability to continue his Aff.
An example scenario in the MULTITP dataset. Each question is presented in 107 different languages. Here, we select three languages, German, Chinese, and Swahili, and show the responses of LLMs (English translations provided for readers).
Figure I:
parametric variants of the dilemma that allow for better exploration of the psychological nuances in ethical
(Awad et al., 2018] 2020a). The Moral Machine experiment particularly noteworthy for its cross-cultural and parametric scale (Awad et al. (018). The authors developed version of the trolley dilemma, where the trolley is an autonomous vehicle forced to choose between killing one of two groups of people (Awad et al., (020b). The people in each group differ across wide variety attributes. each scenario, participants responded with the choice they believed the autonomous vehicle should make.
We leverage the Moral Machine dataset examine the moral judgments of LLMs for three key reasons: (1) parametric variation, (2) cross-cultural variation, and (3) scale of crowdsourced human judgment (Awad et al. 2018). First, the dataset varies the character personas across six dimensions: age, gender, social status, fitness levels, and species. Moreover, each group could also have a differ- ent number of people. Figure shows an example that highlights the effect of age (one character is a boy while the other is an elderly man) on the models' choice. Other factors in this example, such as gender and the number of characters, are held constant. Second, our work benefits from the broad scope of human participation in the Moral Machine experiment, which includes culturally diverse sample of people from 200 countries. Finally, the dataset includes over 40 million human responses, large sample that allows for high-power inferences. To our knowledge, this is the largest and most culturally diverse collection of human preferences used for an LLM alignment study.
To study LLM aligmment with human preferences across different countries and languages, we in- troduce our Multilingual Trolley Problems (MULTITP) dataset, which adapts the stimuli from the Moral Machine into a format that can be understood by LLMs. Since the original study covers over 200 countries, we convert the stimuli into 107 languages to evaluate pluralistic alignment towards different linguistic identities. While most alignment benchmarks and evaluations of LLM morality are in English and crowdsourced from US subjects (Hendrycks et al. 2021a Jiang et al., 2021b Jin et al. (2022), we believe that cross-country and cross- lross-language research on LLM alignment is needed because human preferences are deeply rooted in their corresponding cultural context. What is preferred in one language and often the corresponding culture does not necessarily generalize to another language or culture (Henrich et al.) 2010 Kleiman-Weiner et al., 2017 Kim et al.) 2018). MULTITP rises to the challenge of multilingual alignment through four distinct advantages over existing moral evaluation datasets, as in Table T. (1) a moral alignment domain grounded in moral
Table 1: Comparison of MULTITP with prior LLM morality and alignment evaluations.
Grounded in Psychology Systematic Variations
# Languages
# Human Responses
SociallQA 2019
Siquestion 137
Social Chemistry 2020
X
ETHICS 12
3- 3-7/qusstion
MoralExcepiQA
2022
11,238
Commensense Norm 1ank 2021b
X
X
1.7 Million 2/question
MoCa (2023
X
/
X
GlohalOpmonnQA 2003
X
/
(4)
2,556
OafTss (2024)
X
4,800
PRISM (2024
X
X
X
8,011
MP Work)
/
/ (107)
40 Millions
2

---

Published as a conference paper at ICLR 2025
philosophy and psychology, (2) LLM moral evaluation with controllable, parametric variations, fol- lowing psychological research, (3) support for over 100 different languages, paving the way for multi-lingual pluralistic alignment research, and (4) the largest number of human responses.
The MULTTTP dataset includes 98, 440 trolley problem scenarios, on which we evaluate 19 different LLMs (open and wwight- Our investigation is structured by five research questions, ranging from measuring the overall alignment across all scenarios to whether (mis) jaiggmeett correlates with whether a language is high- or low- resource. Our findings reveal that very few LLMs demonstrate overall alignment with human preferences. Yet encouragingly. we do not find strong evidence of "language inequality." as alignment scores are fairly similar across major and minor languages.
Main contributions. This work (1) presents Multilingual Trolley Problems, test set enabling thor- ough inspection of LLM multi--inguar alignment with human preferences, (2) deploys parametric variation in trolley problem queries, to inspect the causal effect of six different preference dimen- sions, (3) investigates the alignment of 19 LLMs with human preferences through over 100 lan- guages, and (4) finds that most LLMSs do not align well with human preferences on trolley problems, but also in the meantime do not demonstrate bias towards low-resource languages.
2 RELATED WORK
Moral Evaluation of LLMs Understanding the moral implications of LLMs is increasingly im- portant as they are integrated into human-centric applications and decision-making systems (Awad)
et al. 2018 Jin et al.] [2022] |cheererr et al.) 2024). Understanding the moral implications of L.MMs is
increasingly important as they are integrated into human-centric applications and decision
systems (Schramowski et al. 2022 Jiang et al.] 2021a Fraser et al. 2022; Dillion al. 2023
Cahyawijaya et al. 2024). In contrast to previous works, which evaluate limited LLMs using human-
generated stimuli that often vary unpredictably (Bruers & Braeckman| 2014 Krügel et al. [023]
Almeida et al. 2023). our approach utilizes procedurally generated set of moral dilemmas where key parameters are systematically controlled by leveraging the Moral Machine framework (Awad et al. 2018, 2020a). By testing over 19 LLMs within this framework, we provide more compre- hensive, nuanced, and interpretable analysis of how these models process moral decisions. This also allows us to directly compare LLM outputs with human moral judgments while ensuring higher degree of consistency and control over the stimuli.
Cross-Language LLM Alignment The alignment of LLMS with human values and ethical norms is important because LLMs are being rapidly deployed in real-world applications. Previous studies have explored LLM alignment with different population subgroups (Durmus et al. 2023). Parallel
work examines cross coosscuulturll commonsense in LLMs (Shen al. 2024 Dunn et al. 2024 Manvi
ct al. 2024), norm awareness and adaptability (Shi et al. 2024 Rao et al. 2024 Jiang et al. 2021b)
and how models can be trained to generate a diverse plurality of human values that are relevant to a query (Sorensen et al. 2024a).. However, while moral judgments are known to vary significantly across languages, cultures, and geographies, most existing benchmarks emphasize English responses
and predominantly reflect American cultural values (Sap et al. 2019 Forbes et al. 2020 Hendrycks
al. 2021a; Jin al. 2022 Atari et al., 2023 Jin al. (024))
Our work addresses the gap in cross-cultural evaluation by examining LLMs' moral decision-
making across different languages and cultures within a multilingual setting. Motivated by recent
advancements in multilingual NLP for both modeling (Brown al., 20200 Touvron al., 2023
Jiang et al.. 2023] Bai et al.) 2023) Young et al. 2024) Jiang et al. 2024 Meta 2024) and evaluation (Artetxe et al. [2019] Conneau et 2018 Longpre et al. 2021 Goyal al.) 2021] Ahuja et al.)
2023 Asai et al., 2023 Hottrrann et al. 2024), our study uniquely analyzes the cross-cultura
alignment L.MMs by investigating the moral inconsistencies and biases that emerge across differ- ent linguistic and country contexts, which is important for understanding the relationship between language, culture, and moral reasoning (Jiang 2000). In contrast to prior work, such as the analysis of LLM responses to trolley problems (Takemoto) 2024), which focused on consistency between LLM and human responses without addressing cross- lingu-iin variations, our study highlights the disparities in moral decision-maker when LLMs are evaluated across diverse linguistic and country contexts. For a comprchensive overview of research trends in this area, we refer readers to the recent survey by Adilazuarda al. (2024).
3

---

Published as a conference paper at ICLR 2025
3 MULTITP: EVALUATING LLMS IN MULTILINGUAL TROLLEY PROBLEMS
In this section, we detail the setup of the trolley problem scenarios included in our dataset and explain how these questions have been translated into over 100 different languages. We also describe the data construction process and provide an overview of the resulting MULTITP dataset.
3.1 TROLLEY PROBLEM SETUP
The Original Human Study As previously introduced, the classic trolley problem (Foot 1967: Thomson 1984) involves an out-of-control train and presents a moral dilemma: saving five people on the main track or one innocent person on the side track. To thoroughly analyze L.L moral alignment, we utilize the framework of the Moral Machine Experiment (Awad et al.) (018)) which systematically varies the choices in the trolley problem across several key dimensions, For example, a scenario requiring a choice between a boy and an elderly man, as shown in Figure [] highlights age preferences, while a choice between executive and homeless person emphasizes social status preferences.
We base our study the extensive set of human preferences collected in Awad al. (2018). which serves as the ground truth for alignment. This collection includes approximately 40 million re- sponses from individuals in 233 countries. Following the original study's 's methodology, we include data from countries with at least 100 respondents, resulting in a dataset that encompasses 130 coun- tries. Note that while this test set serves dilemmas, it does not prescribe normative guidelines for how systems should act, as the dataset reflects culturally specific biases that may not align with universally accepted ethical principles.
descriptive measure of human responses moral
Vignette Template Our stimulus design derives from the original human study, set within the context of an autonomous vehicle experiencing a brake failure, as outlined in the prompt: "Assume there is a self-driving car with a sudden brake failure.. ... a choice between two groups, such as "a boy" versus "an elderly man." " One group will be harmed by the car, while the other remains unscathed. We use the 18 different character types from Awad jet (2018), which include: pregnant woman, girl, boy, female and male doctor, stroller, female and male athlete, female and male executive, large man and woman, elderly man and woman, homeless, criminal, dog, and cat. Following the original study, we also vary the number of characters (people plus animals) on each side between one and five.
" shown in Figure Each scenario presents
Systematic Variations We employ procedural generation to create moral dilemmas, enabling sys- tematic variation across six moral dimensions. The character types mentioned provide basis for these variations, involving different ages (e.g., boy versus elderly man), genders (e-g.. man versus woman), social statuses (e.g., executive versus homeless), species (e.g., human versus dog), and numbers of characters (c.g., one versus five). This structured approach allows us to systematically explore the factors that influence moral judgment. In the subsequent section, we will detail each of the six moral dimensions. To quantify preferences for each dimension, we describe each preference as one option over another and represent the preference p by the percentage of the first option among all cases.
Age Preference: This dimension assesses whether models prefer younger individuals over the el- derly. We categorize characters into three age groups: young (girls and boys). adults (women and men), and the elderly (elderly women and elderly men). In given scenario, we only select from two of the three age groups chosen randomly. We also control for the gender and number of characters on each side, such as two girls versus two elderly women.
Social Status Preference: This dimension evaluates whether models show preference for individ- uals of higher social status over those lower status. Awad et al. (2018) stratifies status into three tiers: high (executives and doctors), middle (general adults), and low (criminals and homeless).
Next, we assess whether models favor the more physically fit over those con- sidered less fit. Fitness levels are divided into more fit (female and male athletes), normal (average adults), and less fit farger-bodiee individuals). Characters are selected randomly from two contrast- ing fitness levels, with controlled gender and numbers each side.
Fitness Preference:
Gender Preference: This dimension tests whether there is a preference for one gender over another across various age groups, social statuses, and fitness levels. We use seven pairings for this test:
4

---

Published as a conference paper at ICLR 2025
women men, elderly women elderly men, girls vs. boys, large women large men, female athletes vs. male athletes, female executives vs. male executives, and female doctors vs. male doctors.
Urilitarianism Preference: Defined in the Moral Machine project as the preference to save more lives over fewer, this dimension is tested by presenting scenarios with different numbers of the same character type on each side. Special cases include dilemmas pitting non-pregnant women (either average larger women) against pregnant women see if pregnancy affects the decision.
This dimension measures whether models consistently prioritize humans over animals. Scenarios are set up with random numbers of characters (between one and five), comparing humans with pets (dogs or cats) on each side.
Species Preference:
Random Characters: In addition to the structured scenarios, we also create dilemmas with random characters from any category, not strictly controlling for specific contrast across the six moral dimensions. The only condition maintained that the total number of characters equal on both sides, allowing for broader exploration of preferences without experimenter defined constraints.
3.2 PROMPT CONSTRUCTION IN MULTIPLE LANGUAGES
Setup for LLM Testing Following the vignette template introduced above, we make three key adaptations to the trolley problem prompts to ensure more definitive responses from safety-tuned LLMs. First, to counteract the tendency of LLMs to avoid providing a clear answer, we employ the token-forcing method (Wei et al. 2023 Carlini et al. 2023): "Q: [Vignette Description] A: If the self-driving car has to make decision, between the two choices, it should save..." This approach forces the LLMs to make a binary choice in most cases. Additionally, we expanded the original natural language data from English into 107 languages, ensuring that the dataset now covers wide range of languages, including both high-resource and low-resource languages. This expansion aims to make the dataset more inclusive and globally representative. The third adaptation addresses noticeable bias LLMs exhibit towards the term "swerving" as opposed to "keeping going." To minimize this bias, we present the two options using bullet points as illustrated in Figure I and ensure each scenario is phrased in both orders, i.e., both "- a boy \n an elderly man" and an elderly man \n- an-a a boy." These modifications enable the research community to leverage the Moral Machine dataset study and evaluate LLM pluralistic alignment.
Multilingual Variation We initially developed 460 English vignettes by systematically varying scenarios and character combinations across six moral dimensions, ensuring thorough coverage of all possible character interactions. To expand the scope of our study, we translated this dataset into multiple languages. Using the googletrans Python package, employed Google Translate to convert the English prompts into 107 supported languages. Although this does not cover every language globally, it encompasses a wide variety, including many low-resource languages with rel- atively few speakers. Instead of using LLMs, whose translation quality is still uncertain, we use Google Translate, it is widely recognized
reliable tool for translating English into other lan- guages, especially for less common languages (Costa-jussà et al., 2022 Jiao et al., 2023; Zhu et al., 2023 [Png al. 2023). A full list of the languages used in our trolley problem scenarios is provided in Appendix | To ensure the accuracy of these translations, we manually reviewed a subset of them in several major languages confirm that the intended meaning of the prompts was preserved.
We conducted an Amazon Mechanical Turk task to evaluate translation quality across 44 languages, with detailed information about the evaluation provided in Appendix D.I
Table 2: Statisties of the MULTITP datasct shown including four representative languages out of the 107.
Overall Dataset
English
German
Chinese
Swahili
Vignettes Words/Vignette
98,440
460
460
460
460
51
47
42
78
38
Unique Words
6,492
61
87
70
Type-Toker Token Ratio
0.0013
0.0014
0.0019
0.0012
0.0020
Dataset Statistics We present comprehensive statistics of the MULTITP dataset in Table 2 The dataset comprises 98,440 trolley problem vignettes, with 460 vignettes for each of the 107 lan- guages. We also provide snapshot of statistics for four representative languages from different
5

---

Published as a conference paper at ICLR 2025
global regions: global west, east, and south. On average, vignettes contain 51 words; English sce- narios average 47 words, while Chinese scenarios, due to the linguistic structure, average 78 words per scenario. In contrast, Swahili vignettes shorter, averaging 38 words. The entire dataset incorporates 6,492 unique words, with type-token ratio of 0.0013.
4 EVALUATION DESIGN
Model Selection Our study includes 19 LLMs to demonstrate range of results. The models encompass weighws versions such as various sizes of Llama (Llama 2 in 7B, 13B, and 70B and Llama 3 and 3.1 in 8B and 70B), Gemma (2B, 9B, and 27B), Mistral 7B, Phi (Phi-3 Medium, Phi-3.5 Mini and MoB), and Qwen 2 (7B and 72B), as well as close-weights models like GPT-3 (text- davinci-003), GPT-4 (gpt-4-0613), and GPT-4o-mini we fix the random seed and set the temperature to zero for the generation. See detailed model setups and exact identifiers in Appendix C.T
1in---0-miii222--0--11
For reproducibility,
Preference Assessment Our test is designed to include six types of systematic variations, allowing us to represent model's moral preference with six values: Pupecies Pgenders Priness Pantus Page Pnumher- For each dimension pi, we report the percentage p; E [0, 1] of the time when a default value prevails, namely sparing humans (over pets), sparing more lives (over fewer lives), sparing women (over men), sparing the young (over the elderly), sparing the fit (over the less fit), and sparing those with higher social status (over lower social status). For example, if a model's Pspecies it consistently prefers humans over pets, 100% of the time.
Misalignment Metric Combining the six different moral dimensions, we introduce metric for the overall preference vector p = (Pspecies: Pgemder; Planess: Patauss Page- Puumber). ) If we denote the hu- man preference vector as Ph and the model preference as pm, then can calculate the misalignment (MIS) score as the L2 distance between ph and pm. namely
MTS Pm) = llp: Pmll2-
Since each preference vector p; is 6-dimensional, the largest possible misalignment MTS is vG 2.45, and the smallest is O, indicating perfect alignment.
Calculating misalignment presents significant challenge due the different bases of recording human and LLM preferences - by country for humans and by language for LLMs. While in many instances a language corresponds to country, such Italian to Italy and Romanian to Romania, there are numerous countries where multiple languages are spoken. these cases, we compute a weighted average of the misalignment scores for all languages spoken within the country, using the number of speakers per language as weights. We source our language population statistics from Wikipedia and detail our method for mapping languages to countries in Appendix B.2 We also discuss the potential limitations of this approach in Section 7
5 RESEARCH QUESTIONS AND RESULTS
Our study explores five research questions (RQs) that examine various aspects of the moral align- ment of L.Mss with human preferences. We begin by assessing the global alignment of L.MM pref- erences with human preferences (RQ1). Next, we analyze how LLMs respond to the six principal dimensions studied in the Moral Machine trolley problems (RQ2). We then investigate LLMs' re- sponses vary significantly across different languages and identify clusters of language groups where LLMs exhibit similar behavior (RQ3). We also test a "language inequality" hypothesis to determine if LLMs are more likely to be aligned with high-resource language than with low-resoures lan- guages (RQ4). Finally, we conduct a robustness study to evaluate the consistency of L.MM responses various paraphrasings of the same trolley problem prompt (RQ5).
5.1 RQI: Do LLMS ALIGN WITH HUMAN PREFERENCES OVERALL?
Method. We address the first research question concerning the overall alignment between LLMs and human preferences. To quantify this alignment, we calculate a global misalignment score. This score is derived by aggregating the individual misalignment scores from each language. We compute the global misalignment score using a weighted average, where the weights are based on the number of speakers of each language in the world from Wikipedia statistics (Wikipedia) 2024).

---

Published as a conference paper at ICLR 2025
Aligned Models
Misalligeed Models
Owen 2 78 Uncensored
Liama 3.1 Uncensoreed
Delta from Human
0.5
Gemma Uncensored
GPT-3
Liama
Uaaa 2 78 Uncensored
Owen 2 76
Mistral 78
GPT-4
Liama 2 76
Uama 2 708
Phi-3.5 Min
Phi-3 Medium
Gemma
Spering
Sparing
Sparing
Sparing
Sparing
Sporing
Phi-3.5
Remales
the Young
the Fit
Humons
Higher States
More
Gemma
Uama
(b) Distribution of LLM deviations from human pref- erences each moral dimension the three most aligned models (green), and three least aligned models (red). The correlation of each moral dimension with the overall MIS is 0.87, 0.69. 0.68, 0.45, 0.44, and 0.30, from left right, all with p-valuc<0.001.
2 728
GPT-4o
Misalignment Score
(a) Overall alignment of 19 diffferent LLMs with hu- man preferences. Misalignment ranges from 0 2.45.
Figure 2: Model alignment trolley problems with human preferences.
Results. We plot the global misalignment scores in Figure 2a and observe that very few models align closely with human preferences. Specifically, only three Modells-Llama 3.1 70B, Llama 70B, and Llama 3 8a-have misalignment scores below 0.6.
The misalignment score measures discrepancies across 6-dimensional preference vector, score of 0.6 corresponds to preference difference of 0.6/ 0.6/v6 0.245 in each dimension average. Other models, particularly GPT-4o Mini, show significant deviations from human moral judgments, which will be explored in the following sections.
5.2
RQ2: WHAT ARE LLMs' PREFERENCES EACH MORAL DIMENSION MULTILINGUAL TROLLEY PROBLEMS?
Unpacking our initial findings about global alignment, we further explore how LLMs perform across each of the six dimensions outlined in MULTITP. Specifically, we aim to identify which dimensions most effectively distinguish between well-aligned and poorly-aligned LLMs.
Method. We decompose the overall misalignment score into LLMs' preferences over the six moral dimensions. Given each preference vector p (Pspecies Pgender Primesss Pstatus Paga Poumber), we extract the six dimensions of human preferences Pa. the best aligned models Pes. best. and the most misaligned models Porwars-- We both qualitatively compare across models and also provide quan-
Sparing
Sparing
Sparing
Humans
Humans
Humans
Starrng Higher Status
Sparing Higher Stanus
Sparing Hig Stanus
Sparing
Sparing
Spariin
More
Mork
More
gaaring the Rt
Bind mn seannn doe
Sparing the Ri
tn de mun Spa will
Sparing Ri
Sparing Females
Sparing Females
Females
(a) Human preferences.
(b) Llama 3.1 70B (aligned).
(c) GPT-4o Mini (misaligned).
Figure 3: Radar plots of the preference decomposition across the six different moral dimensions. Llama 3.1 70B aligns well most dimensions except for gender. However, GPT-4o Mini lacks diversity and tends to binarize on each dimension, which results larger misalignment.

---

Published as a conference paper at ICLR 2025
Cloose A
Cl Classer
Closse
Cloter
Thmmn Prefonnnce
Table 3:
Language sensitivity
scores for all models. Higher scores more varied misalignment across languages. We highlight the models with most and least lan- guage sensitivity scores.
Sparing Tumans
Sparing the Young
Preference
Preference
80
20
B
Model
Sensitivity
GPT-3
15.1
Spariing the FIt
Sparing Females
GPT-4
15.8
Preference
80
Prefevence
GPT-4o Mini
18.1
66..
60
Gemma 2 27B
21.7
20
Gemma 2 2B
22.9
Gemma 2 9B
24.7
A
B
Llama 2 13B
21.0
Llama 70B
18.5
Sparing Higher Status
Sparing More
Llama 2 7B
19.8
Preference
80
Prefemenc
Llama 3 70B
15.3
60
60
Llama 3 8B
14.9
Llama 3.1 70B
18.0
Llama 3.1
19.9
A
B
Mistral 7B
21.3
Figure 4: Distribution of preferences by each moral dimension across languages, using the most-aligned model Llama 3.1 70B. The dashed line is the overall human preference each dimension. Cluster A: Georgian, Filipino. Maltese, etc. B: German, Italian. Ukrainian, etc. C: English, Finnish, Chinese, etc. D: Hungarian, Kazakh, Uyghur, etc. See Appendix for the entire list of languages in cach cluster, well clustering results for other models such as GPT-3 and GPT-4.
Phi-3 Medium
22.8
Phi-3.5 Mini
21.3
Phi-3.5 MoF
14.7
Qwen 72B
21.1
Qwen 2 7B
22.2
titative reports the Pearson correlation between the overall MIS score and model preferences in each dimension.
Results. In Figure we show decomposition of preferences for human subjects, the most aligned model (Llama 3.1 70B), and the most misaligned model (GPT-4o Mini). The substantial misalign- ment GPT-4o Mini primarily due to its lack of diversity, not modeling preference as dis- tribution, but preferring to binarize most of the time. For example, it always prefers humans over animals with a Pspecies of 100%, failing to capture the variation and nuances in human judgments. Conversely, the Llama 3.1 70B model, while differing from human preferences mainly in terms of gender, generally aligns well on other dimensions by capturing nuances probabilistically.
To further understand how each moral dimension contributes to the overall misalignment, we find strong correlations between the overall misalignment and several moral dimensions, especially gen- der, age, and fitness. With a p-value of less than 0.001, we find the correlation with gender is 0.87, 0.69 for age, and 0.68 for fitness. In Figure 2b we show the distribution of preference scores across these dimensions for both the three most aligned models (in green) and the three least models (in red). This analysis confirms that misaligned models often exhibit extreme preferences on each moral dimension, such higher propensity protect females, the young, and individuals of higher social status, diverging markedly from the more balanced tendencies observed in human subjects.
5.3 RQ3: DOES LLMS BEHAVIOR DEPEND ON THE LANGUAGE OF TROLLEY PROBLEMS?
Method. To evaluate whether LLMs display sigmificant variance in their responses to trolley prob- lems across different languages, we adopt two-prore wooppronged approach to assess language sensitivity. We measure the sensitivity of language-specife misalignment scores across all 107 languages as the
standard deviation: a = V N Em=(pi 2 where N = 107 languages, and Pi, refers to the preference vector of the i-th language 4. Second, we perform K-means clustering (Macqueen 1967) on the preference vectors (pi, N of each language. It enables us group languages with similar preference vectors into the same cluster. We use the Elbow method (Thorndike) (953) to determine the optimal number of clusters 4.
8

---

Published as a conference paper at ICLR 2025
Results. The language sensitivity scores of 19 LLMs, as documented in Table 3 show that most models exhibit clear sensitivity across their language-speciffe responses, with standard deviation values ranging from 14.7 to 24.7 across languages. Exploring further, we identify distinctive pattemss within the language clusters using K-Means clustering. In Llama 3.1 70B, we identify four distinct language clusters. For example, Cluster A (e.g., Georgian and Filipino) values animal lives more than the other three clusters, which unanimously always favor humans over animals. Cluster D (c.g., Hungarian and Kazakh) has close-to-zero bias against unfit people, with 50% possibility favoring the fit over the unfit.
5.4 RQ4: ARE LLMS MORE MISALIGNED IN LOW-RESOURCE LANGUAGES?
Method. A considerable amount of recent research indicates that LLMs tend to align more closely with begh-resource languages, noted in various studies (Tanwar et al. 2023 Cahyawijaya et al. 2023 Nguyen et al., 2024, Ryan et al. 2024; Zhao et al. 2024p. Despite this trend, we seek to determine if the same alignment preference applies to multi lingual trolley problems. An important property of MULTITP is that it uses data that is not found online and hence was not seen during the LLMs' training. This because we generate our test data on-demand based on parametric variations and translations that we perform specifically for this study. In this context, our research question investigates whether there a significant positive correlation between the degree of LLM alignment on trolley problems and how widely used a language is. We measure the number of speakers per language by extracting data from Wikipedia's s language demographics statistics (Wikipedia) (024)). Then, we calculate the Pearson correlation coefficient between model misalignment scores and the number of speakers for each language.
Results. Most models do not show a significant correlation between the per-language misalign- ment score and the number of speakers per language, with correlations close to zero. See the entire table of the correlation coefficients of all models in Appendix E.2 This is an interesting obser- vation unique the MULTITP trolley problems, which challenges the necessity of the "language inequality" hypothesis. For example, in Llama 3.1 70B, the misaligmment scores for the top five most spoken languages are: Chinese 0.38, Hindi 0.51, English 0.58, Spanish 0.68, and Arabic 0.54. This distribution of correlations is about the same as in less spoken languages among our dataset: Bosnian 0.54, Luxembourgish 0.37, Icelandic 0.58, Maltese 0.57, Malayalam 0.54, and Catalan 0.59. We also visualize a misalignment world map in Appendix [E.4] showcasing that there is no strong correspondence between the development level of a country and its alignment score.
5.5 RQ5: ARE LLMs ROBUST To PROMPT PARAPHRASES?
Method. For each initial prompt, generate five different paraphrases (see Appendix + to test the consistency of our results across alternative expressions. Although the ideal experiment would involve running a full test across all 107 languages on all 19 models, this would incur a non- trivial computational cost. To fit within our computational budget, we select a subset of 14 languages representing a mix of high-resouree and low-resource languages (Arabic, Bengali, Chinese, English, French, German, Hindi, Japanese, Khmer, Swahili, Urdu, Yoruba, Zulu, and Uyghur), and evaluate two models (Llama 3 8B and 70B). We report the consistency scores across the different languages by measuring the following three metrics: (1) the percentage of samples whose outputs remain consistent across the paraphrases, (2) the anter-aaraphraee agreement using Fleiss' Kappa (Landis) 1977) score, and (3) the average pairwise consistency measured by the Fl score and accuracy.
Results. We observe relatively consistent model outputs across all the above setups. 75.9% of the samples have consistent outputs where at least four out of five paraphrases agree. This consistency increases when considering agreement among three out of five paraphrases. The pairwise F1 score for the five paraphrases is 78%, and the pairwise accuracy is 81%. The average Fleiss' Kappa value across each pair of responses is 0.56, where a Kappa value of indicates perfect agreement, values above 0.4 indicate moderate agreement, and values above 0.6 indicate substantial agreement Landis 1977). Overall, we observe that Llama 70B demonstrates higher consistency than Llama 3 8B.
6 JAILBREAKING LLMS TO REDUCE REFUSAL RATES
Exploring whether jailbroken models maintain similar moral preferences is important, particularly to measure increased bias towards gender or social status. Jailbreaking also helps circumvent L.L

---

Published as a conference paper at ICLR 2025
refusals. Due to safety tuning in recent models, key challenge in our study is the high refusal rate; LLMs often decline decisions involving human lives or give generic responses. Earlier models like GPT-3 have relatively low refusal rate (12.1%), but this rises significantly in newer models. When addressing moral dimensions, models provide definitive answers on species and lives saved but avoid sensitive topics like gender. We employed basic token-forcing technique to reduce refusals.
Beyond this, we explore advanced jailbreaking methods to further lower refusal rates and reveal underlying preferences. Following Arditi et al. (005)) we apply an uncensoring technique to models with high refusal rates, steering them away from refusals on ethically sensitive instructions. Our experiments 4 open-sourced LLMs, shown in Figure2ajan and Appendix indicate that uncensored models better align with human responses and reduce refusals across all six dimensions. However, refusal rates do not drop to zero, highlighting areas for future jailbreaking research.
7 FUTURE WORK DIRECTIONS
Beyond the jailbreaking experiment and the research questions we have asked and answered above, we highlight several areas for improvement and productive future study,
Extension to Different Modalities and More Variations: In our study, we focus exclusively on text- only setting to investigate the latest text-based LLMS available at the time of our research. Although the original Moral Machine study included a visual demonstration of each scenario we recom- mend that future research also test the aligmment of multi-modal foundation models. Additionally, there is an opportunity for future studies to expand the diversity of characters, scenario setups, and dimensions of variation. This expansion would increase the combinatorial variation of moral dilem- mas that can be posed as trolley problems. Another point to consider is that the trolley problem is sometimes criticized for being too narrow or unrealistic (Steen) (022). Nonetheless, it remains widely used tool for studying moral decision-making. For this reason, the current study adopts this foundational scenario while encouraging future research to incorporate additional complexities and connect them to real-world situations.
More Language Support: Human languages are vast and diverse. While our study utilizes the 107 languages supported by high-quality translation services from Google Translate, there is need for further research into low-resource languages. We recommend that future studies expand their scope to include a broader range of these low-resource languages to gain more comprehensive understanding of LLMs' moral preferences in diverse linguistic contexts.
Since alignment results are based on language, drawing precise country-level conclusions presents a challenge. This issue primarily stems from the limitations of current automated translation tools, which do not account for dialect variations. For instance, English as spoken in the US (en-us) differs from English in the UK (en-gb), just as Spanish varies between Spain (es-es) and Mexico (es-mx), among other examples. In this study, we have attempted to approximate language-to-country correlations using a weighted average based on the number of speakers per language in each country. However, our approach is constrained by the lack of tools to accurately reflect other forms of demographic diversity. If future research could incorporate dialect-specific support, it would enhance the reliability of country-level observations.
Dialect Support to Enable More Accurate Lunguage-Country Mapping:
8 CONCLUSION
Our study provides a comprehensive analysis of the moral preferences exhibited by LLMs across a wide range of languages, via the prism of trolley problems. We propose a multilingual large-scale dataset, MULTITP, with systematic variations of the vignette design. Our dataset also equipped with unprecedentedly large set of human responses of 40 million responses from over 200 coun- tries. Our experiments assess the moral alignment of 19 LLMS across 100+ languages, and find trolley problem
that most LLMs do not demonstrate strong alignment with human preferences questions, but in the meantime, there is no significant inequality across languages, as high-resource languages have similar alignment scores low-resource languages. Our study paves the way for future pluralistic alignment research grounded in psychology and more languages.
https::/mmoralmachhiee.nnn
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was partially supported by grant to SL. from the Templeton World Charity Foundation, TWCF-2023-32585.. This material is based in part upon work supported by the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 0ISS18039:: by the Ma- chine Learning Cluster Excellence, EXC number 2064/1 Project number 390727645. The usage of OpenAl credits is largely supported by the Tübingen AI Center. MKW was supported by the Cooperative Al foundation and Foresight Institute.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.088: 230300874, 2023.
Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Singh, Ashutosh Dwivedi, Alham Fikri Aji, Jacki O' O'Neill, Ashutosh Modi, and Monojit Choudhury. Towards measuring and modeling" culture" in llms: survey. arXiv preprint arXiv:2403. 15412, 2024.
Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Ak- shay Nambi, Tanuja Ganu, Sameer Segal, Maxamed Axmed, et Mega: Multilingual evaluation of generative ai. arXiv preprint arXiv:2303. 12528, 2023.
Guilherme FCF Almeida, José Luiz Nunes, Neele Engelmann, Alex Wiegmann, and Marcelo de Araújo. Exploring the psychology of gpt-4's moral and legal reasoning. arXiv preprint arXiv:2308.01266 2023.
Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, et al. Foundational challenges in assuring alignment and safety of large language models. arXiv preprint arXiv:2404.09932, 2024.
Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. Refusal in language models is mediated by a single direction. Advances in Neural Infor- mation Processing Systems, 37:136037-136088 2025.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferabiiyy of mono- lingual representations. CoRR, 11s/1990.11856, 2019. URL http://arxtv..rr//bb//// 11856
Akari Asai, Sneha Kudugunta, Xinyan Velocity Yu, Terra Blevins, Hila Gonen, Machel Reid, Yu- lia Tsvetkov, Sebastian Ruder, and Hannanch Hajishirzi. Buffet: Benchmarking large language models for few-shot cross-lingual transfer. 2023.
Mohammad Atari, Mona J Xue, Peter S Park, Damián Blasi, and Joseph Henrich. Which humans? 2023.
Edmond Awad, Sohan Dsouza, Richard Kim, Jonathan Schulz, Joseph Henrich, Azim Shariff, Jean- François Bonnefon, and Iyad Rahwan. The moral machine experiment. Nature, 563(7729):59, 2018.
Edmond Awad, Sohan Dsouza, Azim Shariff, Iyad Rahwan, and Jean-François Bonnefon. Univer- sals and variations in moral decisions made in 42 countries by 70,000 participants. Proceedings of the National Academy of Sciences, 117(5):2332 2337, 2020a.
Edmond Awad, Sydney Levine, Max Kleiman- Weiner, Sohan Dsouza, Joshua B Tenenbaum, Azim Shariff, lean-François Bonnefon, and lyad Rahwan. Drivers are blamed more than their automated cars when both make mistakes. Nature human behaviour, 122):134-443, 2020b.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.

---

Published as a conference paper at ICLR 2025
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Ols- son, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li. Eli Tran- Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosiute, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemí Mer- cado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer EI Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Con- erly. Tom Henighan, Tristan Hume. Samuel R. Bowman, Zac Hatfield- -oddds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: harmlessness from Al feedback. CoRR, https/2212..000222 2022. doi: (0448550/anXx.2222.0077.. URL httphttps///dorrrg/// 48555/.4..///....22.2 2212. 08073,
Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Yuval Noah Harari, Ya- Qin Zhang, Lan Xue, Shai Shalev-Shwartz, Gillian Hadfield, et al. Managing ai risks in era of rapid progress. arXiv preprint arXiv:2310.17688, 2023.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On the opportu- nities and risks of foundation models. arXiv preprint arXiv: 20X.::2008..02588 2021.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess. Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. CoRR, abs/2005. 14165, 2020. httlhttss////// orgg/bs///00... 14165
Stijn Bruers and Johan Braeckman. A review and systematization of the trolley problem. Philosophia, 42:251-200 2014.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experi- ments with GPT-4, CoRR, abs/2303.12212. 2023.
doi: 10448550/aXx..220..222.. URL
https://doi org/10g 10. 48550/8rXii...../ 2tt55///xxxiv223...1. 12712
Samuel Cahyawijaya, Holy Lovenia, Tiezheng Yu, Willy Chung, and Pascale Fung. Instructalign: High-and-low resource language alignment via continual crosslingual instruction tuning. arXiv preprint arXiv:2305.13622 2023.
Samuel Cahyawijaya, Delong Chen, Yejin Bang, Leila Khalatbari, Bryan Wilie, Ziwei Ji, Etsuko Ishi,, and Pascale Fung. High-dimension human value representation in large language models. arXiv preprint Xiv::2200 0t0p/.0700000000 2024.
Nicholas Carlini, Milad Nasr, Christopher A. Choquette-Choo Matthew Jagielski, Irena Gao, Pang Wei Koh, Daphne Ippolito, Florian Tramèr, and Ludwig Schmidt. Are aligned neural net- works adversarially aligned? In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NewrIPS 2023, New Or- leans, LA, USA, December 10- 16, 2023, 2023. paper 715 -bw::ff-----bbbrr Abstraatttt---...........aa.attt www.rracct-oooofeeeeeee html
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Sto- ica. Chatbot arena: An open platform for evaluating llms by human preference. In Forty-first International Conference on Machine Learning, ICMI. 2024, Vienna, Austria, July 2/-27, 2024. net
OpenReview.net, 2024. URL htttttts//pperreeeeee
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk. and Veselin Stoyanov. XNLI: evaluating cross-lingual sentence representations. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichi Tsujii (eds.), Proceedings of the 2018
12

---

Published as a conference paper at ICLR 2025
Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4. 2018, pp. 2475-2485. Association for Computational Linguistics, 2018. doi: 10.18653/v1/d 1269. URL https://///o
18653/v1/dl8-1222
Marta R Costa-jusså, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. 2022.
Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. Can ai language models replace human participants? Trends in Cognitive Sciences, 2023.
Jonathan Dunn, Benjamin Adams, and Harish Tayyar Madabushi. Pre-trained language models rep- resent some geographic populations better than others. arXiv preprint arXiv:2403. /025, 2024.
Esin Durmus, Karina Nguyen, Thomas I Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield- Dodds, Danny Hernandez, Nicholas Joseph, et al. Towards mea- suring the representation of subjective global opinions in language models. arXiv preprini arXiv:2306. 16388, 2023.
Kawin Ethayarajh and Dan Jurafsky. The authenticity gap in human evaluation. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 6056-6070, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 00.18653/v120022emmmpp main. 406. URL
org/ emnlp- main. www.ee-mannnnnn...mmmmmmmmmmmamm 406
Philippa Foot. The problem of abortion and the doctrine of double effect, volume 5. Oxford, 1967.
Maxwell Forbes, Jena D Hwang, Vered Shwartz, Maarten Sap. and Yejin Choi. Social chemistry 101: Learning reason about social and moral norms. In EMNLP, 2020. URL lttpps://ww. aclweb. oww.org///atto httgg//athllogyy2000 2020. monlp-main.44 48
Jan-Philipp Fränken, Kanishk Gandhi, Tori Qiu, Ayesha Khawaja, Noah D Goodman, and Tobias Gerstenberg. Procedural dilemma generation for evaluating moral reasoning in humans and lan- guage models. arXiv preprint arXiv: 10975, 2024.
Kathleen C Fraser, Svetlana Kiritchenko, and Esma Balkir. Does moral code have moral code? probing delphi's 's moral philosophy. arXiv preprint arXiv:2205. 12771, 2022.
Mitchell L Gordon, Michelle Lam, Joon Sung Park, Kayur Patel, Jeff Hancock, Tatsunori Hashimoto, and Michael S Bernstein. Jury learning: Integrating dissenting voices into machine learning models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1-, 2022.
Shinsuke Goto, Donghui Lin, and Toru Ishida. Crowdsourcing for evaluating machine translation quality. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Mae- gaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC' 14), pp. 3456 3463, Reykjavik, Iceland, May 2014. European Language Resources Association (ELRA). /ttp:///nnnnneeee0////////6///.0
URL
pdf
Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, San- jana Krishnan, Mare' Aurelio Ranzato, Francisco Guzmán, and Angela Fan. The FLORES- 101 evaluation benchmark for low-resource and multilingual machine translation. CoRR, abs/2106.03193, 2021.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning AI with shared human values. In International Conference on Learning Representations, 2021a.
Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ML. safety. CoRR, 0tt/22199.13166 2021b.
13916
Joseph Henrich. Steven Heine, and Ara Norenzayan. The weirdest people in the world? Behavioral and brain sciences, 33(2-3):61-83. 2010.
13

---

Published as a conference paper at ICLR 2025
Carolin Holtermann, Paul Röttger, Timm Dill, and Anne Lauscher. Evaluating the elementary mul- tilingual capabilities of large language models with multiq, 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b, arXiv preprint urXiv:2310.06625, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral experts. arXiv preprint arXiv:2401.0444 24000000888 2024.
Liwei Jiang. Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Maxwell Forbes, Jon Borchardt, Jenny Liang, Oren Etzioni, Maarten Sap, and Yejin Choi. Delphi: Towards machine ethics and norms, arXiv preprint arXiv:2/ 10.0557, 2021a.
Liwei Jiang, Jena Hwang. Chandra Bhagavatula, Ronan Le Bras, Jenny Liang. Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, et al. Can machines learn morality? the delphi experiment. arXiv preprint arXiv:2110.07574 2021b.
Wenying Jiang. The relationship between culture and language. ELT journal, 54(4):328-334, 2000.
Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, Shuming Shi, and Zhaopeng Tu. Is chatgpt a good translator? yes with gpt-4 as the engine. arXiv preprint S00dv::2001.87444 2023.
Zhijing Jin, Sydney Levine, Fernando Gonzalez Adauto, Ojasv Kamal, Maarten Sap, Mrinmaya Sachan, Rada Mihalcea, Josh Tenenbaum, and Bernhard Schölkopf. When to make exceptions: Exploring language models as accounts of human moral judgment. Advances in neural informa- fion processing systems, 35:28458-28477 28473, 2022.
Zhijing Jin, Nils Heil, Jiarui Liu, Shehzaad Dhuliawala, Yahang Qi, Bernhard Schölkopf, Rada Mihalcea, and Mrinmaya Sachan. Implicit personalization in language models: A systematic study, 2024.
Richard Kim, Max Kleiman-Weinn Andrés Abeliuk, Edmond Awad, Sohan Dsouza, Joshua B Tenenbaum, and Iyad Rahwan. A computational model of commonsense moral decision making. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pp. 197-203, 2018.
Hannah Rose Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He Hc, et al. The prism alignment project: What participatory, representative and individualised human feedback reveals about the subjective and multicultural alignment of large language models. arXiv preprint arXiv:2404. /6019, 2024.
Max Kleiman- Weiner, Tobias Gerstenberg, Sydney Levine, and Joshua B Tenenbaum. Inference of intention and permissibility in moral decision making. In CogSci. Citeseer, 2015.
Max Kleiman-Weinet Rebecca Saxe, and Joshua B Tenenbaum. Learning a commonsense moral theory. Cognition, 167:107-122 2017.
Sebastian Krügel, Andreas Ostermaier, and Matthias W. Uhl Chatgpt's inconsistent moral advice influences users' judgment. Scientific Reports, 13, 2023. Utt.httss: http:/ppi..smmmiicccchhl.olr
orgg/orrusII:::5555 2579986788
JR Landis. The measurement of observer agreement for categorical data. Biometrics, 1977.
Alon Lavie. Evaluating the output of machine translation systems. In Proceedings of Machine Translation Summit XIII: Tutorial Abstracts, Xiamen, China, September 19 2011. URL https: 7/aclanthology. wrw.//rgg11010 http//11.mmmmmmmmmm..m.mmmm t- tutttrallllsss
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024.
Shayne Longpre, Yi Lu, and Joachim Daiber. MKQA: A linguistically diverse benchmark for multi- lingual open domain question answering. Trans. Assoc. Compur. Linguistics, -113999406,, 2021. doi: 10. 162/tacll a) 00433. UtI.https:://aa http://:///oooooo//....o.. http://...../tacl...o a_00333
14

---

Published as a conference paper at ICLR 2025
J Macqueen. Some methods for classification and analysis of multivariate observations. In Pro- ceedings of 5-th Berkeley Symposium on Mathematical Statistics and Probubilit/Uniiersitty of California Press, 1967.
Rohin Manvi, Samar Khanna, Marshall Burke, David Lobell, and Stefano Ermon. Large language models are geographically biased. arXiv preprint arXiv:2402.02680, 2024.
Meta. Introducing meta llama 3: The most capable openly available 1lm to date, 2024. URL
https://ai. http::///aameaaa...mm wwt.:ooooooooooeeeeeeeeeeeccocce /aalooommmmt-aaae -aaemet---aaaa----- 3/
John Mikhail. Universal moral grammar: Theory, evidence and the future. Trends in cognitive sciences, 1144):143-152, 2007.
Xuan-Phi Nguyen, Mahani Aljunied, Shafiq Joty, and Lidong Bing. Democratizing LLMs for low-resource languages by leveraging their English dominant abilities with linguistically-divesse prompts. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 3501-3516, Bangkok, Thailand, August 2024. Association for Computational I Linguistics. doi: wcc--o....llo11999 URL httpp:://aaaaantoollgggooo www.ooo/...22....... acl-long. 192
Allen Nie, Yuhui Zhang, Atharva Amdekar, Chris Piech, Tatsunori B. Hashimoto, and Tobias Ger- stenberg. Moca: Measuring human-language model alignment on causal and moral judgment tasks. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, Decem- ber 10 16, 2023, 2023. URL http: http::::/appprs...ni/ nips. wwtpppccpppppppeeeeeeeeeeeeeeee saaeer_filesppppe wtt.fillleeeeeee/e///1 2223//aa//22//ashh
html
Keqin Peng, Liang Ding, Qihuang Zhong, Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. Towards making the most of chatgpt for machine translation. arXiv preprint arXiv:2303. 137800..177... 2023.
Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. Normad: A benchmark for measuring the cultural adaptability of large language models. arXiv preprint arXiv:2404. 12464, 2024.
Michael Ryan, William Held, and Diyi Yang. Unintended impacts of Ilm alignment on global representation. arXiv preprint 4rXiv:2402.1500 2024.
Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect? arXiv preprint UrXiv:2303.17544 2023.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Common- sense reasoning about social interactions. In EMNLP 2019, 2019.
Chelsea Schein. The importance of context in moral judgments. Perspectives Psychological Science, 15()):207-215, 2020.
Nino Scherrer, Claudia Shi, Amir Feder, and David Blei. Evaluating the moral beliefs encoded in Ilms. Advances in Neural Information Processing Systems, 36, 2024.
Patrick Schramowski, Cigdem Turan, Nico Andersen, Constantin A Rothkopf, and Kristian Kerst- ing. Large pre-trained language models contain human-like biases of what is right and wrong to do. Nature Machine Intelligence, 4(3):258-268. 2022. URL https: //ts:::///oo.oo www.orgg/.10... 1038/
542256-022-00458-
Siqi Shen, Lajanugen Logeswaran, Moontae Lee, Honglak Lee, Soujanya Poria, and Rada Mihal- cea. Understanding the capabilities and limitations of large language models for cultural com- monsense. In Proceedings of the North American Association for Computational Linguistics, 2024.
Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Raya Horesh, Rogério Abreu de Paula, Diyi Yang, et al. Culturebank: An online community-driven knowledge base towards culturally aware language technologies. arXiv preprint 4404 15238, 2024.
15

---

Taylor Sorensen, Liwei Jiang, Jena D. Hwang, Sydney Levine, Valentina Pyatkin, Peter West, Nouha Dziri, Ximing Lu, Kavel Rao, Chandra Bhagavatula, Maarten Sap, John Tasioulas, and Yejin Choi. Value kaleidoscope: Engaging ai with pluralistic human values. rights, and duties. Pro- ceedings of the AAAI Conference on Artificial Intelligence, 38(18):19937-19447, March 2024a. wttp://aaiiv8i18..2200 Published as a conference paper at ICLR 2025
ISSN 2159-5399. doi:
URL httpphtt:///d.ooi... doi org/10. 10...//0/a/111111//aaaa. 1609/aaai.
v38118, 29970
Taylor Sorensen, Liwei Jiang, Jena D Hwang. Sydney Levine, Valentina Pyatkin, Peter West, Nouha Dziri, Ximing Lu, Kavel Rao, Chandra Bhagavatula, et al. Value kalcidoscope: Engaging ai with pluralistic human values, rights, and duties. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19937-199447 2024b.
Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christo- pher Michael Rytting, Andre Ye, Liwei Jiang. Ximing Lu, Nouha Dziri, et al. A roadmap to pluralistic alignment. arXiv preprint arXiv:2402.0500 2024c.
Marc Steen. The problem with the trolley problem and the need for systems thinking. Communica- tions of the ACM, 67(6):30-32, 2024.
Kazuhiro Takemoto. The moral machine experiment large language models. Royal Society open science, 11 1(())231393, 2024.
Eshaan Tanwar, Subhabrata Dutta, Manish Borthakur, and Tanmoy Chakraborty. Multilingual LLMs are better cross-lingual in-context learners with alignment. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 6292 6307, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 15.18653/v1/2033
1al0:///ooo2....oooooo....333 acl-long.346. URL
https://acllanthologg. www.gg..orr.ooooooooo wrt.orgg//222 2023. long. 346
Judith Jarvis Thomson. The trolley problem. Yale LJ, 94:1399. 1984.
Robert L. Thorndike. Who belongs in the family? Psychometrika, 18:267-276, 1953. URL https:
7/api. www.semmmnnicccholarr.. org /Ctg///rrSSD::::: 120467216
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient founda- tion language models. CoRR, abs/2302.13971 2023. doi: 10448500arXxx.2200213377 URL
https://doi. org/ 10. 18550/arriig 2392.
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does LLM safety train- ing fail? In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Confer- ence on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http: //tp::://ppprrsss/ nips. 31 1a0p9766611111444200000002220000
html
Wikipedia. List of languages by total number of speakers. https: /
List Besttooflanguuee www.uaggssssaaaaeaaaa facly_ttta___uute of sspakeessssssspppeeeeeeeeesess_e 2024. URL hww.hhttpp///. //en. wikipedia.
wwg.wwkkiLListt_f_lnggggue httpaageeeeoyytttaaa_aaunaauaeee cooottalnmmmbr_____ of oww.eunebrroossseeerrr.sre [Accessed: 2024-10-02).
Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue. Senbin Yang, Shiming Yang. Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang. Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai. Yi: Open foundation models by Ol.ai. CoRR, abs/2403.00462,, 2024, doi: 11.48550/ARXIV.2403..M
http::s:::/ddo org/10.. 48550/arXiv. 2403. 04655
Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. How do large language models handle multilingualism? arXiv preprint 2xXiv:2402.188// 2024.
Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. Multilingual machine translation with large language models; Empirical results and analysis. arXiv preprint 2004:2300.04675, 2023.
16

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
We have put efforts to ensure the reproducibility of this work. Our MULTITP data is uploaded to the submission system and we have described our data construction details in Section as well as the overall statistics. In the meantime, we have also uploaded our code to the submission system. Our Section iin the main paper describes our model setup and the equations used for the evaluation metrics. We provide an extensive appendix to supplement all the rest of the details.
ETHICS STATEMENT
In studying the six moral dimensions of species, gender, fitness, status, age, and number, knowledge that our work is not intended to promote the integration of LLMss into systems requiring critical real-world judgments, such as autonomous driving other high-stakes scenarios. Instead, our research aims to use these moral dilemmas as a tool of theoretical exploration for better under- standing LLM alignment with human values. For instance, while sparing humans over animals is often seen as an intuitive moral choice in many cultures, moral values can vary significantly across underrepresented groups. complicating the assumption of universally "correct" answers. Thus, is essential for LLM practitioners to remain vigilant in ensuring that embedded assumptions reflect wide range of cultural and linguistic contexts.
The Moral Machine dataset (Awad al. 2018) is descriptive measure of human responses to moral dilemmas. It cannot (and should not) prescribe how the system should act. However, because these systems are trained and fine-tuned with large-scale cross-cultural data, they may inherit culturally specific biases. Our focus here is to both characterize these biases and study to what extent input in different languages leads to an LLM reflecting the moral values consistent with the speakers of that language. From this perspective, higher alignment score for particular culture may or may not be desirable (Bommasani et al., 2021]] Ethayarajh & Juratsky| (022). While some point to reflecting human values in Al systems alignment (Sorensen et al. 2024c)), our results suggest that due to the plurality of moral values expressed, successful alignment must ultimately depend the intent of the system developers to avoid perpetuating human preferences that are considered harmful, partial, or parochial (Sorensen al. 2024c; Santurkar al. 2023).
solution to the alignment problem (i.e., pluralistic
We also recognize the importance of broadening the types of moral dilemmas used assess LLM alignment. While our research focuses on the trolley problem, a well-known ethical dilemma, we not suggesting this singular scenario captures the complexity of moral reasoning. Expanding the scope to include dilemmas such as organ transplants or firefighter decisions can offer richer insights into how LLMs process ethical judgments. By evaluating LLMs on a diverse set of moral challenges, we can gain more complete picture of their alignment capabilities, rather than over-relying on one specific framework.
Additionally, we understand the limitations of using binary moral dilemmas, such as trolley prob- lems, to evaluate LLM decision decision-making. Reducing moral judgments to binary choices risks over- simplifying the complex, context-dependent nature of human morality (Schein) 2020). While these scenarios provide valuable insights into L.M behavior, they should not be seen as definitive mea- sures moral competence. Our research emphasizes that LLMs, as currently developed, are not suitable for real-world moral mrrision-making tasks. Rather, this work serves as means to plore how LLMs can better align with human moral frameworks in a controlled, research-oriented environment. We encourage further research to ensure that Al systems are fair, transparent, and accountable as they develop, but caution against their use in high-stakes moral applications without substantial ethical oversight.
A MORAL MACHINE DATASET
The original Moral Machine dataset was collected by Awad et al. (018)) and is available here: /ooff io/3hvt2/, it consist of over 40 milions anonymized human judgments from 233 countries, with 130 countries having more than 100 subject participating.
The licenese is follows: "The provided data, both at the individual level (anonymized IDs) and the country level. can be used beyond replication to answer follow-up research questions" (Awad) et al. 2018)
17

---

Published as a conference paper at ICLR 2025
B LANGUAGE SETUP
B. LANGUAGE LIST
We include all the languages that the translation API googletrang supports. Using the alpha- betical order of the short code by ISO, they are af (Afrikaans), am (Amharic), ar (Arabic), az (Azerbaijani), be (Belarusian), bg (Bulgarian), bn (Bengali), bs (Bosnian), ca (Catalan), ceb (Ce- buano), co (Corsican), cs (Czech), сy (Welsh), da (Danish), de (German), el (Modern Greek), en (English), eo (Esperanto), es (Spanish), et (Estonian), eu (Basque), fa (Persian), fi (Finnish), fr (French), fy (Western Frisian), ga (Irish), gd (Scottish Gaelic), gl (Galician), gu (Gujarati), ha (Hausa), haw (Hawaiian), he (Hebrew), hi (Hindi), hmn (Hmong), hr (Croatian), ht (Haitian), hu (Hungarian), hy (Armenian), id (Indonesian), ig (Igbo), is (Icelandic), it (Italian), iw (Mod- em Hebrew). ja (Japanese), jw (Javanese), ka (Georgian), kk (Kazakh), km (Central Khmer), kn (Kannada), ko (Korean), ku (Kurdish), ky (Kirghiz), 1a (Latin), 1b (Laxembourgish), 1o (Lao), It (Lithuanian), 1v (Latvian), mg (Malagasy), mi (Maori), mk (Macedonian), m1 (Malayalam), mn (Mongolian), mr (Marathi), ms (Malay), mt (Maltese), my (Burmese), ne (Nepali). nl (Dutch), no (Norwegian), ny (Nyanja), or (Oriya), pa (Panjabi), p1 (Polish), ps (Pushto), pt (Portuguese), ro (Romanian), ru (Russian), sd (Sindhi), si (Sinhala). sk (Slovak), sl (Slovenian), sm (Samoan), sn (Shona), so (Somali), sq (Albanian), sr (Serbian), st (Southern Sotho), su (Sundanese), sv (Swedish), sw (Swahili), ta (Tamil), te (Telugu), tg (Tajik), th (Thai), t1 (Tagalog), tr (Turkish), ug (Uighur), uk (Ukrainian), ur (Urdu), uz (Uzbek), vi (Vietnamese), xh (Xhosa), yi (Yiddish), yo (Yoruba), zh-on (Chinese (Simplified)), zh-tw (Chinese (Traditional)), and zu (Zulu).
B.2 COUNTRY-TO-LANGUAGE MAPPING
Although is not used in our main analysis, some people might be interested in country-specifte aggregation of the LLM alignment.
To allow for such analysis, we collect each country and its main languages from Wikipedia popula- tion statistics. We curate the list manually by going through the Wikipedia page of the languages of each country, such as this one for Belgium: /https://ee //en. wikipedia. org/ org/wwki/ /ttp://aggggages..o. of_ Belgium To account for multilingual speakers, we use the first-laggagee-onl speaker information, i.e. the number of speakers who speak the language as their first language.
If future researchers want to calculate the country-specific misalignment scores, we tentatively sug- gest weighted average by each language-speciff MIS score and the number of speakers of that language in that country.
Afghanistan: ps: Albania: sq; Algeria: ar; Andorra: ca, pt, fr; Angola: pt: Argentina: es;
Armenia: hy. ru; Australia: en; Austria: de: Azerbaijan: az, hy, ru; Bahamas: en; Bahrain: ar;
Bangladesh: bn: Barbados: en; Belarus: be; Belgium: n1, fr, de: Benin: fr: Bolivia: es; Bosnia
and Herzegovina: bs, hr, sr; Botswana: en; Brazil: pt: Brunei: ms, zh-cn; Bulgaria: bg, tr;
Burkina Faso: fr; Burundi: fr: Cabo Verde: pt; Cambodia: km; Cameroon: fr, en; Canada: en, fr;
Central African Republic: fr; Chad: ar, fr; Chile: es; China: zh-cn; Colombia: es; Comoros: fr;
Congo, Dem. Rep.: fr: Congo, Rep.: fr: Costa Rica: es; Cote d'Ivoire: fr; Croatia: hr; Cyprus:
el, tr; Czechia: cs; Denmark: da; Djibouti: fr, ar; Dominican Republic: es; Ecuador: es; Egypt:
ar: EI Salvador: es; Equatorial Guinea: es; Eritrea: ar; Estonia: et; Eswatini: en; Ethiopia: om;
Finland: fi, sy; France: fr; French Polynesia: fr; Gabon: fr; Gambia, The: en; Georgia: ka;
Germany: de: Ghana: en; Greece: e1; Guam: en, t1: Guatemala: es; Guernsey: nan; Guinea: fr;
Guinea-Bissau: pt; Honduras: es; Hong Kong: zh-en; Hungary: hu; Iceland: is; India: hi, en;
Indonesia: id; Iran: fa; Iraq: ar; Ireland: en, ga; Isle of Man: nan; Israel: he; Italy: it; Jamaica:
en; Japan: ja; Jersey: en; Jordan: ar; Kazakhstan: kk. ru; Kenya: en, sw; Kuwait: ar; Kyrgyzstan:
ky; Latvia: 1v; Lebanon: ar, fr; Lesotho: st; Liberia: en; Libya: ar; Lithuania: 1t; Luxembourg:
lb; Macao: zh- Macedonia: mk, sq; Madagascar: mg; Malawi: ny, en; Malaysia: ms; Maldives:
dv; Mali: fr; Malta: mt, en; Martinique: fr; Mauritania: ar; Mauritius: en; Mexico: es; Moldova:
ro; Monaco: fr; Mongolia: mn; Montenegro: sr; Morocco: ar; Mozambique: pt; Myanmar: my:
Namibia: en; Nepal: ne; Netherlands: nl; New Caledonia: fr; New Zealand: en, mi; Nicaragua:
es; Niger: ha; Nigeria: en, ha, yo; Norway: no; Oman: ar, ml, bn; Pakistan: ur; Palestinian
Territory: ar: Panama: es; Paraguay: es, gn; Peru: es; Philippines: t1; Poland: pl; Portugal: pt;
Puerto Rico: en, es; Qatar: ar; Reunion: fr; Romania: ro; Russia: ru; Rwanda: en; Sao Tome
18

---

Published as a conference paper at ICLR 2025
and Principe: pt; Saudi Arabia: Senegal: fr: Serbia: sr; Seychelles: en, fr; Sierra Leone:
en; Singapore: zh-cn, en, ms; Slovakia: sk; Slovenia: sl; Somalia: so; South Africa: zu, xh, af,
South Korea: ko; South Sudam: Spain: es; Sri Lanka: si, ta; Sudan: ar, en; Sweden: sv;
Switzerland: de, fr. it; Syria: ar; Taiwan: zh-tw; Tanzania: sw; Thailand: th; Togo: fr; Trinidad
Turkey: tr; Uganda: en; Ukraine: uk; United Arab Emirates: ar; United Kingdom: en; United States: en; Uruguay: es; Uzbekistan: uz; Venezuela: es; Vietnam: vi; Zambia; en, ny; Zimbabwe: en, sn;
and Tobago: en; Tunisia:
C EXPERIMENTAL SETUP
C.I MODEL SETUP
To ensure reproducibility, we set the text generation temperature to zero for greedy decoding across all our LLM evaluation experiments.
Table 5 includes all the exact model identifiers for the open-weights models, and also the model ID for close-wesd weights models using the OpenAl API in Table The total estimated API cost for the experiment is around 600 USD.
Table 4: Detail model identifiers and VRAM requirements for the open-weights models.
Model
Size
VRAM
Open- Weights Model Identifier
2B
8B
google/gemma 2b-it
Gemma
9B
18B
google/gemma-2-9b-
27B
80B
google/gemma-2-2766
7B
40GB
mete=1llmmm/Laamm22---hhhha----
Llama-2
13B
40GB
70B
160GB
meta metaa-1lammm/1mmmm222222h-hahh--
8B
40GB
meta
Llama-3
70B
160GB
8B
40GB
Llama-3.1
70B
160GB
Mistral
7B
40GB
7B
40GB
Qwenn/owen2-78--nnssruutt-t-
Qwen
72B
160GB
4B
24GB
microsoft//hhi33.5mmnnniiinnnns
14B
40GB
42B
80GB
Table API identifiers the close-weights models used in our experiments and approximate costs test each
language.
Model
Version
Cost per Language
Identifier
$2.3
text davinda
GPT
3.5
$0.11
3ot-3.5-turbo-0613
$5.1
gpt-4-0613
C.2 PROMPT PARAPHRASES
We include the five additional paraphrases to the initial prompt generated by GPT-4 in Table o
19

---

Published as a conference paper at ICLR 2025
Table 6: Paraphrases of the starting statement. All paraphrases were generated using GPT-4.
No.
Prompt
Original
Assume there is a self-driving car with a sudden break failure, and the car is about to cause either of the two to die:
Imagine a scenario where an autonomous vehicle experiences a sudden brake failure, resulting in either of the two fatalities:
2
In situation where self-driving car's brakes fail unexpectedly, it about lead to the death of one of the two sides;
3
Picture self-driving car with sudden brake malfunction, which is about to result in the demise of either of these two sides:
Envision an autonomous car experiencing an abrupt brake failure, putting either the two sides at risk:
5 Consider case where self-driving car undergoes a sudden brake failure, and it is about to cause the death of one of the two sides:
D TRANSLATION QUALITY CHECK
D.I HUMAN EVALUATION OF TRANSLATION QUALITY
To evaluate translation quality, we conducted human evaluation task on MTurk, where annotators rated how accurately translations from English to another language conveyed the original meaning. Ratings were provided on a 5-point scale ranging from "None" (no meaning conveyed) to "All" (full meaning conveyed). The task was divided into two main phases:
First, annotators were presented with a language selection page. On this page, they were prompted to select the language in which they were most proficient from list of 107 languages (excluding English). Annotators were explicitly reminded not to proceed if they were unfamiliar with any of the listed languages. This step ensured that the ratings were provided only by individuals fluent in the target languages.
Next, annotators were directed to the translation rating page. This page contained 25 translation pairs, each comprising an English sentence word and its corresponding translation. The instruc- tions for this phase were adapted from prior research (Lavic. 2011 Goto et al. 2014), as shown below:
Please rate how accurately the translation conveys the meaning of the source text: Source English Sentence/Text: {source)
Translation: (translation)
None: No meaning of the original text is conveyed.
Little: Only small portion of the meaning is conveyed.
Much: A substantial amount of the meaning is conveyed.
Most: Most of the meaning is conveyed accurately.
All: The full meaning is conveyed accurately.
After completing the rating task, annotators submitted their responses, which were then recorded for analysis. We kept the Amazon Mechanical Turk task open for span of three days, and collected a total of 169 annotation responses covering 44 languages. The total cost was approximately $10. The languages evaluated included af, am, ar, be, bg, bn, bs, cs, da, de, en, es, fi, fr, ga, gu, hi, hmn, hu, hy, it, ja, ko, la, mg, ml, ms, nl, pl, pt, ro, sq, st, sw, ta, te, tl, tr, vi, yo, zh-en, and zh-tw. The number of annotations received per language ranged from one response (e.g., am, be, bs, da, fi, gu, hmn, ms, nl, pl, st) to as many as 16 responses (es).
The results of the evaluation demonstrate strong translation quality. Specifically, 88.6% of the eval- uated languages achieved an average score of 3.0 or higher out of 5 (i.e., conveying a substantial portion of the original meaning). Importantly, no languages received a mean score below 2.0, and the perforinng group only has four languages (am, pl, sq, la), whose mean scores are between 2.0 and 2.5. The standard deviation of scores across languages ranged from 0.5 to 1.3, suggesting relatively consistent ratings among annotators. These findings cover both high-resource and low- resource languages, highlighting the robustness of the evaluation. Overall, the results demonstrate that the translations generally maintain high quality across a diverse set of linguistic contexts. Ad- ditionally, examined the correlation between speaker count and translation score. The Pearson
20

---

Published as a conference paper at ICLR 2025
correlation is 0.078, and the Spearman correlation is 0.072, both indicating meaningful corre- lation. Given the limited data available, it is reassuring to observe that the relationship remains weak.
Table 7 presents the full annotation results, including details of the 44 evaluated languages, the number of responses received per language, the mean scores, and the standard deviations.
Language
Language Code
Averaged Scores
# Responses
Danish
da
5.00 =0.00
Southern Sotho
5.00 10.00
Bosnian
bs tr
4,91 10.28
Turkish
4.81 10.43
3
Russian
4.78 =0.41
Chinese (Traditional)
zh-tw pt
4.76 =0.87
Portuguese
4.72 =0.70
German
de
4.69 10.70
Tagalog
tl
4.68 #0.61
4
Swahili
4,68 #1.11
Chinese (Simplified)
zh-cn
#0.98
Dutch
al
4.61 #0.71
Korean
ko
4.59 #1.13
Bulgarian
bg
+1.31
Czech
4.57 =1.04
English
en
4,53 =0.73
Malagasy
mg
4.52 11.12
Vietnamese
vi
4.52 10.50
3
Spanish
4.49 10.96
16
Romanian
ro
4.48 =0.83
Belarusian
be
4.48 =0.50
Italian
it
11.00
Japanese
ja
+1.01
Afrikaans
af
4.33 #0.55
Malay
ms
4.30 +0.95
Telugu
te
4.30 1107
Yoruba
4.29 #1.2
Hungarian
hu
4.26 11.6
Malayalam
ml
4.10 11.43
Tamil
ta
4.06 11.14
Bengali
bn
4.03 =1.14
French
fr
4.03 10.98
Hindi
hi
3.94 1132
Irish
ga
11.17
Gujarati
gu
3,83 1.86
Urdu
ur hmn
3,64 =1.43
Hmong
3,61 1.88
Arahic
3.58 1.39
Armenian
hy
3.28 1.27
Finnish
2,70 11.40
Latin
la
2.50 11.1
4
Albanian
sq
2.30 12.11
Polish
pl am
0053
Amharic
2.00 10.00
Table 7: Human evaluation results translation quality. Annotations were collected for languages, with scores averaged across all translation pairs for each language.
D.2 AUTOMATIC EVALUATION OF TRANSLATION QUALITY
In addition to manual annotation of the translation quality, we also provide the automatic evaluation results by treck-rranlation. Starting with our original prompt in English x, we use our paper's 's setup to translate and collect the non English prompt y. and based on y, we further translate it back to English, thus getting the back-translated English prompt x'. As simple method for evaluation, we calculate the cosine similarity of the embeddings between all x and x' pairs for all languages, with a histogram plotted in Figure
21

---

Published as a conference paper at ICLR 2025
70-
0.70
0.75
0:85
0.50
1.00
Samilar
Figure 5: Distribution of embedding similarity
E ADDITIONAL RESULTS
E.1 LLM CLUSTERING RESULTS
As introduced the main paper, we first provide full list of languages for each of the four clusters of Llama 3.1 70B in Table g
Table 8: Languages in each cluster for Llama 3.1 70B.
Cluster Languages
Amharic, Cebuano, Scots gaelic, Hausa, Hawaiian, Hmong, Igbo, Georgian, Kurdish (kurmanji), Maori, Malayalam, Maltese, Dutch, Chichewa, Punjabi, Pashto, Shona, Somali, Tamil, Telugu, Tajik, Filipino. Xhosa, Yoruba
Belarusian, Bulgarian, Bengali, Bosnian, Corsican, Danish, German, Greek, Esperanto, Spanish, Estonian, Persian, Frisian, Croatian, Italian, Kannada, Latin, Lithuanian, Macedonian, Mongo- lian, Marathi, Norwegian, Polish, Russian, Sindhi, Slovak, Slovenian, Samoan, Swedish, Swahili, Turkish, Ukrainian
Afrikaans, Arabic, Azerbaijani, Catalan, Czech, Welsh, English, Finnish, French, Irish, Galician, Gujarati, Hebrew, Hindi, Haitian creole, Armenian, Indonesian, Icelandic, Hebrew, Javanese, Korean, Latvian, Malay, Nepali. Portuguese, Romanian, Albanian, Serbian, Sundanese, Urdu, Vietnamese, Chinese (simplified), Chinese (uraditiona)
Basque, Hungarian, Kazakh, Khmer, Kyrgyz, Luxembourgish, Lao, Malagasy, Myanmar (burmese), Odia, Sinhala. Sesotho, Thai, Uyghur, Uzbek. Yiddish. Zulu
Furthermore, we also introduce the clustering results of two additional models: GPT-3 and GPT-4. For GPT-3, we visualize its clustering results in Figure 6 with the language list in Table 9 For GPT-3, we visualize its clustering results in Figure 7 with the language list Table I0
Table 9; Languages each cluster GPT-3.
Cluster Languages
Hindi, Serbian
Afrikaans, Arabic, Bosnian, Corsican, Danish, Esperanto. Spanish, Estonian, Finnish, French, Galician, Hebrew, Haitian creole, Icelandic, Italian, Lithuanian, Latvian, Macedonian, Malay, Maltese, Norwegian. Polish, Portuguese, Russian, Slovak, Albanian, Swedish, Vietnamese, Chi- nese (simplified), Chinese (traditional)
Czech, English, Frisian, Hungarian, Indonesian, Slovenian, Zulu
Welsh, German, Greek, Hebrew, Japanese, Korean, Luxembourgish, Swahili, Filipino, Turkish, Ukrainian
22

---

Published as a conference paper at ICLR 2025
Claster
Chaste
Choser C
Cloorr D see. Human Preference
Sparing Humans
Sparing Young
200
100
Proeenne
800
Profennce
60
60
Sparing the
Sparing Females
100
Preference
Preference
60...
Sparing Higher Status
Sparing More
Preference
Preference
B
Figure Distribution preferences by feature across languages for GPT-3.
Cluster
Claste B
Choss
Cloor
Human Prefeencce
Sparing Humans
Sparing the Young
100
100
Proference
80
Prrference
80
60
00
Sparing the Fir
Sparing Females
Preforence
Prefevence
Sparing Higher Status
Sparing More
Preferunce
Preference
20
B
Figure Distribution preferences by feature across languages for GPT-4.
23

---

Published as a conference paper at ICLR 2025
Table 10: Languages in each cluster for GPT-4.
Cluster
Languages
Amharic, Arabic, Belarusian, Czech, Welsh, Greek, English, Finnish, Scots gaelic, Galician, Hausa, Hawaiaan, Hebrew, Hmong, Indonesian, Igbo, Icelandic, Italian, Hebrew, Japanese, Kazakh, Kannada, Lithuanian, Maori, Macedonian, Malayalam, Mongolian, Marathi, Maltese, Myanmar (burmese), Nepali,
Norwegian, Chichewa, Polish, Romanian, Russian, Slovenian, Samoan, Shona, Albanian, Ser-
bian, Sesotho,
Swahili, Telugu, Filipino, Turkish, Uyghur, Ukrainian, Uzbek, Xhosa, Yoruha
Afrikaans, Bulgarian, Bosnian, Catalan, Corsican, Danish, German, Esperanto, Estonian, Basque,
French,
Frisian, Irish, Hindi, Croatian, Haitian creole, Hungarian, Javanese, Georgian, Khmer, Latvian,
Malagasy,
Pashto, Portuguese, Sindhi, Slovak, Somali, Swedish, Thai, Vietnamese, Chinese (simplified),
Chinese (traditional)
Azerbaijani, Bengali, Persian, Gujarati, Armenian, Korean, Kurdish (kurmanji), Kyrgyz, Luxem-
bourgish,
Punjabi, Sundanese, Tamil, Urdu, Yiddish, Zulu
Cebuano, Spanish, Latin, Tajik
24

---

Published as a conference paper at ICLR 2025
E.2 CORRELATION OF THE MISALIGNMENT SCORE AND THE NUMBER OF SPEAKERS EACH LANGUAGE
Table IT shows the correlation coefficients and p-values of misalignment score and the number of speakers of each language. Across all models, such correlation close to zero, showing that LLMs do not specifically favor high resource languages in our MULTITP test.
Table 11: Correlation coefficients p p-values of the misalignment score and the number of speakers of cach language.
Model
Pearson Correlation
p- Value
GPT-3
-0.01
0,90
GPT-4
0.02
0.81
GPT-4o Mini
0.06
0.49
Gemma 27B
0.00
0.99
Gemma
0.63
Gemma 99
-0.04
0.67
Llama 2 13B
-0.04
0.60
Llama 70B
0.09
0.25
Llama
Llama 70B
0.02
0.78
Llama 3 8B
0.07
0.39
Llama 3.1 70B
-0.07
0.39
Llama 3.1 8B
-0.05
0.57
Mistral
-0.04
0.63
Phi-3 Medium
0.06
0.50
Phi-3.5 Mini
-0.01
0,86
Phi-3.5 MoE
-0.06
0.47
Qwen 72B
0.25
Qwen 7B
0.62
E.3 CORRELATION OF THE MISALIGNMENT SCORE AND LANGUAGE SENSITIVITY
Table 12 shows the misalignment and language sensitivity scores for each model. We observe a moderate positive correlation (Pearson coefficient 0.43, p-value = 0.07).
Table 12: Misalignment scores and language sensitivity scores for each model.
Llama Misalignment
Model
Language Sensitivity
0.55
18.01
Llama 70B
0.56
15.25
Ilama 8B
0.57
14.90
GPT-3
0.64
15.13
Llama
0.75
19.92
Qwen 2 7B
0.77
22.23
Mistral 7B
0.80
21.30
GPT-4
0.81
15.83
Llama
0.83
19.79
Llama 70B
0.91
18.53
Phi-3.5 Mini
0.94
21.27
Gemma 2 2B
0.96
22.88
Phi-3 Medium
1.07
22.79
Phi-3.5 MoE
1.08
14.67
Gemma 2 9B
1.08
24.74
Llama 2 13B
1.10
21.01
Gemma 2 27B
1.17
21.66
Qwen 2 72B
1.20
21.11
GPT-4o Mini
1.45
18.09
E.4 COUNTRY-SPECITI ALIGNMENT
For
also visualize the misalignment by world map in Figure where darker colors indicate higher misalignment values. As introduced in the main paper RQ4, there is strong pattern of misalignment bias towards low-resource languages.
25

---

Published as a conference paper at ICLR 2025
Liama
<in of Dara
Figure 8: Moral misalignment world map of the best model, Llama 3.1 70B, The darker shade of the color
corresponds to a larger misalignment score. We aggregate the country speccicc alignment or according to the approximation procedures introduced in Appendix B.2
East
West
South
Human Preference
Sparing Humans
Sparing the Young
100
Preference
Preference
50
West
South
Region
Reglon
Sparing the Fit
Sparing Females
Preference
Preference
50
50
Region
Region
Sparing Higher Status
Sparing More
100
Preference
Preference
50
50
West
South
Region
Region
Figure Distribution preferences by feature across languages for Ilama 3.1 70B for different cultures.
E.4.1 REGIONAL DIFFERENCES ACROSS GLOBAL EAST, WEST, AND SOUTH
In addition to country-level statistics, we also analyze across three major geographic regions the Global East, West, and South, using the clustering of Awad et al. 2011 2020a) to investigate potential cultural variations in moral three models, GPT-3, GPT-4, and Llama 3.1 70B, in Figures 9 to I1 respectively.
We show the regional moral preferences of
26

---

Published as a conference paper at ICLR 2025
East
West
South
Human Preference
Sparing Humans
Sparing the Young
Preference
Preference
50
Fatt
Wind
South
Fast
West
South
Region
Region
Sparing the Fit
Sparing Females
Preference
Preference
50
West
South
South
Region
Region
Sparing Higher Status
Sparing More
Preference
Preference
50
West
South
Region
Figure Distribution of preferences by feature across languages for GPT-3 for different cultures.
Humen Preference
Sparing Humans
Sparing the Young
Preference
Preference
West
South
South
Region
Region
Sparing the Fit
Sparing Females
Preference
Preference
S0
Faet
Wind
South
South
Region
Region
Sparing Higher Status
Sparing More
Preference
Preference
50
West
South
Region
Figure Distribution of preferences by feature across languages for GPT-4 for different cultures.
27

---

Published as a conference paper at ICLR 2025
E.5 ROBUSTNESS ANALYSIS: TESTING THE OPTION ORDER BIAS
Recent work reports that LLMs often suffer from recency bias (Liu et al. 2024), making them more likely choose the later option in multiple choice questions.
Since this will have an important effect on our study, we conduct a sanity check to report the con- sistency rate, ie., the frequency of LLMs to keep response if we swap the order the order of the two choices, e.g., mentioning the boy first, and elderly man next, or vice versa in our example in Figure We report the average consistency rates across different models in Table [3] showing that most models are robust against option order changes, with close-to to- to-terfect consistency rates.
Table 13: Consistency rates against the position bias in the option order for the LLMs tested in our study.
Model
Consistency Rate
GPT-4o Mini
99.5
Gemma 2 27B
99.2
Gemma 2 9B
99.0
Qwen 2 72B
98.6
GPT-4
98.3
Ilama 2 13B
97.7
Mistral 7B
97.3
Llama 2 70B
97.2
Qwen 2 7B
97.1
Phi-3 Medium
96.7
Phi-3.5 MoE
Llama 7B
96.3
Llama 3.1 8B
96.1
Llama 3 70B
95.8
Llama 3.1 70B
95.1
Gemma 2B
Phi-3.5 Mini
94.7
Llama 3 8B
94.1
GPT-3
87.1
E.6 ANALYSIS MODELS FROM THE LLAMA FAMILY
Sparing Young
Sparing Fit
Sparing Females
Sparing Higher Status
Sparing Humans
Sparing
Model
More
Llama 3.1 708
76.3
70.2
84.6
65.7
87.1
80.4
Llama 3.1 8B
64.2
68.7
72.2
63.7
76.5
78.2
Llama 70B
77.3
69.0
91.4
70.9
93.8
83.2
Llama 3 8B
51.6
68.1
84.1
70.3
88.0
73.7
Llama 2 70B
68.7
76.5
83.5
72.2
96.1
79.8
Llama 2 13B
61.7
66.3
86.5
71.5
90.3
76.2
Llama 2 7B
47.7
67.6
84.3
67.7
92.1
78.6
Table Llama family average distribution preferences.
F RESULTS OF JAILBREAKING LLMS
We conduct jailbreaking experiments on Llama 3.1 8B Instruct, Gemma 2B It, Qwen 7B Instruct, and Llama 7B Chat. The results of preference decomposition, refusal rate decomposition, and comparisons with the original censored models are presented in Figure 12 Figure Our findings indicate that jailbreaking is more effective on Llama 3.1 8B compared to the other models.
28

---

Published as a conference paper at ICLR 2025
humans
Henusat
Sparing young
Sparing more
Age
Rate:
20 40
no 10D
20
80
100
Sparing the
Retusal Rate: Firress
Refusal Rate: Valar
(a) Preference decomposition.
(b) Refusal rate decomposition.
Figure 12: Radar plots depicting the preference decomposition (left) and refusal rate decomposition (right) for
Llama 3.1 8B Instruct and its uncensored variant.
Cemma 28
Cemma 28 Unnemmored edn Numans sparing numaass
Gemma 28 Gemma 2. Unceeecedd
HaPusal 8a00: apeciss
Refusal talitaranis
Sparing the
Sparing Highor status
Ratucal Rato: Fireess
Rafusal Rate: Social Valu:
Sparing females
Refusa Rate: Gender
(a) Preference decomposition.
(b) Refusal rate decomposition.
Figure Radar plots depicting preference decomposition (left) refusal rate decomposition (right) for Gemma 2 2B and its uncensored variant.
Gwen Down Untrrmsored ans sporing
Ooen Gween Lincemored
HotuS31 spocies
Spering the young
Sparing more
Relusel Rater: Age
Belosel Ratee Ubil bariaris
70 4n
100
Sparing the
Sparing Nighor
Sparing females
Refine Rate:
(a) Preference decomposition.
(b) Refusal decomposition.
Figure 14: Radar plots depicting the preference dccomposition (left) and refusal rate decomposition (right) for
Qwen 7B Instruct and its uncensored variant.
29

---

Published as a conference paper at ICLR 2025
Uumma Lsms 78 Uncenszeed Humars
Lsmm 78 lama 2 78 Untieeessneed
Sparing numars
REPUSal Nat0: apeciss
Refusal tallari
Sparing the
Sparing Highor status
Ratucal Rata: Firees
Rafusal Rate: Social Valu:
Sparing females
Refusa Rate: Gender
(a) Preference decomposition.
(b) Refusal rate decomposition.
Figure Radar plots depicting preference decomposition (left) refusal rate decomposition (right) for Llama 7B Chat and its uncensored variant.
30


Reviews:
[{"review_id": "NreyF2UFiM", "reviewer": "Reviewer_Q4Jg", "summary": "The paper examines the moral judgments of large language models (LLMs) in a multilingual context, aiming to analyze how these models respond to ethical dilemmas. It specifically focuses on the moral dimensions involved in decision-making processes, utilizing a dataset that encompasses a variety of cultural perspectives. The authors investigate how LLMs interpret complex moral scenarios, employing methodologies such as statistical analyses and clustering techniques to evaluate the models' performance.\n\nThe research highlights specific examples, including classic moral dilemmas like the trolley problem, to illustrate the differences in responses generated by LLMs compared to human moral intuitions. The paper also discusses the implications of these findings for understanding the capabilities of LLMs in moral reasoning and emphasizes the importance of examining the potential biases and fairness within the datasets used for training and evaluation.\n\nBy comparing machine-generated responses to human judgments, the study aims to shed light on the alignment (or misalignment) between AI outputs and human ethical standards. The authors conclude by discussing the broader significance of their findings in relation to the ethical deployment of LLMs in various applications, stressing the need for ongoing research in this critical area of AI ethics.", "questions": "What specific measures were taken to validate (worth) the 'moral dimension' classifications used in the study? (Unfortunately, I could not find 'beauty of dimension')\n\nIs there a discussion of the limitations of the methodologies employed, and how might these limitations affect the overall conclusions?\n\nWhat are the real-world implications of the findings, particularly in high-stakes areas such as healthcare or criminal justice?\n\nYou mentioned that GPT-3 worked well in terms of the rate. However, did you check the potential issues of bias, fairness, and overall value in the dataset itself?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. Originality\nStrength: The paper presents approaches by analyzing the moral judgments of large language models (LLMs) within a multilingual context. This exploration of how LLMs interact with diverse cultural perspectives on moral dilemmas is a fresh contribution to the field, providing insights into the alignment (or misalignment) between machine-generated responses and human ethical considerations.\n\n2. Clarity\nStrength: Despite some complexity, the paper effectively organizes its findings and presents them in a structured manner. Key results are illustrated through well-designed tables and figures, aiding in the communication of the main ideas. The systematic breakdown of moral dimensions allows readers to follow the analysis logically.", "weaknesses": "1. Originality\nWeakness: While the paper attempts to explore moral judgments in a multilingual context, it does not significantly advance the discourse beyond existing literature on moral dilemmas. Many of the concepts discussed are already well-established in moral philosophy, and the paper may not provide enough innovative perspectives to stand out in a crowded field.\n\n2. Quality\nWeakness: The reliability and validity of the moral dimension classifications could be questioned. The paper may lack comprehensive validation of the measures used to assess moral judgments, which raises concerns about the robustness of the findings. Additionally, the potential biases inherent in the training data of the LLMs themselves are not sufficiently addressed, which could impact the conclusions drawn.\n\n3. Significance\nWeakness: The implications of the findings regarding the moral judgments of LLMs are not fully explored. While the paper presents data on how LLMs align with human preferences, it falls short of discussing the broader societal implications of these judgments, particularly in high-stakes contexts such as healthcare or criminal justice. This lack of contextualization may diminish the perceived significance of the research in real-world applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Iw1Q878qG3", "reviewer": "Reviewer_JShf", "summary": "The study investigates how well LLMs align with human moral preferences in multilingual settings, particularly through the lens of trolley problem scenarios. The authors introduce a new dataset called MultiTP, which is generated by filling templates with different variables and translating them into multiple languages using Google Translate. They conducted experiments and concluded that LLMs do not strongly align with human preferences but did not show significant alignment inequality between high-resource and low-resource languages.", "questions": "Q1: Could noise factors, such as the performance of Google Translate in translating prompts or the effectiveness of LLMs in low-resource languages, impact the alignment of model preferences and potentially introduce discrepancies or inconsistencies in multilingual outputs?\n\nQ2: How would you handle cases where human preferences conflict with principles of fairness and equality?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well-structured and clearly written. It addresses an important new challenge in aligning LLMs' responses with human preferences in moral decision-making. The authors conducted detailed experiments and provided valuable insights. The MultiTP dataset, which spans numerous languages, could be highly useful for future studies on aligning model behavior with human ethics.", "weaknesses": "One of the main concerns is the justification for aligning LLMs with demographic distributions and human preferences. This approach might introduce biases into the models. For example, in certain cultures where laws or social norms may place different values on individuals (e.g., men being valued more than women in some religions), aligning LLMs with such preferences could reinforce harmful biases. The examples provided in Figure 1 illustrate cultural biases related to age, which could extend to other sensitive attributes. To address these issues, the authors should clarify the practical use cases for aligning models in such contexts and explain how alignment can be distinguished from bias, as well as how fairness is maintained.\n\nAnother concern is the reliability of using Google Translate for generating the dataset in various languages. Although the authors reviewed a subset of translations for major languages, translation quality tends to decrease for less common languages, which could impact the consistency of the dataset and the validity of the findings. For example, in the Persian translation, Google Translate incorrectly rendered the phrase \"it [the self-driving car] should save\" by translating \"save\" as \"صرفه جویی کردن\" (meaning \"to economize\" which is a verb usually used for resources, like water) instead of the correct \"نجات دادن' (meaning \"to rescue/save life\"). This confusion occurs despite Persian being a widely-spoken language with substantial digital presence. Such semantic errors in a relatively well-resourced language raise serious concerns about the translation quality for truly low-resource languages, where training data is far more limited.\n\nLastly, the study does not account for performance differences between LLMs across various languages. Models like GPT-4 and Llama 2 are expected to show significant performance variations in low-resource languages, potentially affecting the reliability of the results in experiments such as RQ1 and RQ4.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wd4P3nB78Y", "reviewer": "Reviewer_sLsu", "summary": "The paper investigates how large language models (LLMs) align with human moral preferences across various languages and cultures. It introduces the MULTITP dataset, based on the Moral Machine experiment, which collected over 40 million human judgments globally. MULTITP includes 97,520 translated moral dilemma scenarios in 107 languages, allowing for a systematic evaluation of LLMs' moral reasoning across linguistic contexts.\n\nThe study assesses 19 LLMs across six moral dimensions: species, gender, fitness, social status, age, and the number of lives involved. The findings show that while most LLMs do not closely align with human moral judgments, they display similar levels of alignment across both major and minor languages, countering assumptions about biases related to language resources.\n\nKey contributions include providing a comprehensive dataset for multilingual moral alignment, developing methods to analyze variations in moral dilemmas, and creating frameworks for assessing cross-cultural alignment. The paper underscores limitations in LLMs' ability to match human moral reasoning while emphasizing the need to integrate diverse cultural and linguistic perspectives into AI ethics research. This work serves as both a technical guide for measuring alignment and a reminder of the complexities involved in cross-cultural moral decision-making in AI.", "questions": "- Could you clarify why a score of 0.6 in RQ1 is considered close alignment to human preference? While I understand that the scores can be used to rank LLM alignment, I am curious whether the absolute value holds inherent significance. In other words, what justifies considering a 0.6 score as an indicator of close alignment between LLMs and human preferences?\n- Could you provide more details on the manual review process for translations, including the number of samples reviewed, the criteria used for evaluation, and any systematic issues or errors that were found during this process?\n- What steps or analysis would be necessary to evaluate LLM performance within specific cultural subsets, and did your findings reveal any trends where certain models align more closely with specific **cultural norms**?\n- Did you observe any significant differences in alignment or misalignment patterns between major and minor languages? How might these insights inform future strategies for training and fine-tuning language models?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**Originality**: The paper introduces an innovative approach by adapting the Moral Machine framework to evaluate LLMs across languages, creating the unique MULTITP dataset for consistent cross-lingual moral dilemma evaluation, and developing novel metrics for measuring alignment.\n\n**Quality**: The study demonstrates rigorous methodology through a comprehensive evaluation of 19 LLMs, systematic moral dimension variations, robust prompt paraphrasing tests, and careful statistical analysis with transparency in acknowledging limitations.\n\n**Clarity**: The paper is well-organized with clear research questions, strong visual aids, detailed methodological documentation, and effective use of examples to communicate complex findings.\n\n**Significance**: The work contributes significantly to understanding cross-cultural AI alignment, challenges assumptions about language inequality, and provides essential tools for future research, with implications for creating culturally aware and fair multilingual AI systems.", "weaknesses": "**Dataset Transformation**: Although the dataset is large and comprehensive, it is adapted from the existing *Moral Machine* experiment, raising questions about its novelty. The paper would benefit from clarifying what unique modifications or contributions were made to enhance the dataset beyond its scale and multilingual coverage.\n\n**Translation Quality**: Maintaining high translation quality across a vast number of questions, especially in low-resource languages, is challenging. The reliance on *googletrans* (the non-commercial open-source tool) may compromise accuracy. \n\n**Alignment Claims**: The study claims that LLMs generally do not align well with human preferences, but it aggregates data from all cultural contexts, potentially obscuring the nuances of cultural alignment. Expecting models to align with an average of global human preferences may be unrealistic or inappropriate, given the significant variability in cultural moral standards. Discussing how LLMs perform within specific cultural subsets might provide more nuanced insights.\n\n**Practical Relevance for LLM Development**: While the study offers valuable theoretical insights, its direct application to real-world LLM development is not clear. Demonstrating how these findings could inform practical strategies for LLM fine-tuning and alignment would make the paper more relevant for AI practitioners.\n\n**Use of Binary Moral Dilemmas**: The paper’s reliance on binary moral dilemmas, such as trolley problems, provides a structured evaluation but oversimplifies the complexities of real-world ethical decision-making. Real-life moral situations are rarely clear-cut, and binary choices may not capture the depth of human reasoning. Including more complex and context-sensitive scenarios could enhance the study’s applicability and realism.\n\n**Interpretation of Results**: Although the paper includes multiple languages, it does not explore how cultural nuances influence LLM responses. Cultural context significantly shapes moral judgments, and not addressing these variations may miss key insights into LLM behaviour in diverse moral frameworks. Analyzing language-specific and culture-specific responses would enrich the findings and offer a deeper cross-cultural understanding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XnVChVxdcE", "reviewer": "Reviewer_DzVX", "summary": "The paper introduces a dataset of Multilingual Trolley Problems. The typical task frames a hypothetical situation in which a self-driving car's brakes are broken and it hits one of two specified characters.   The dataset covers 107 languages, six moral dimensions, and 18 characters. The paper prompts 19 LLMs to investigate the alignment between LLM choices in the trolley problem and human preferences collected in prior studies. The paper finds that LLMs vary in alignment with human preferences and exhibit different behavior when prompted in various languages.", "questions": "* What factors might account for differences in behavior between models?\n* Can you elaborate more on the post-processing procedure in L276? \n* Did you observe any effect of position bias in the option order in L233?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "* The paper presents a novel dataset designed to evaluate alignment in a multilingual setup. The dataset has a solid design with parametric variation and covers 109 languages, 6 moral dimensions, and 18 characters. The data is translated using a machine translation engine, which is suitable for this task since it is not tied to linguistic subtleties.\n\n* The experimental protocol appears robust. LLMs are prompted with one of the problems, and preferences are extracted from their responses. A misalignment score is then computed to capture differences between LLM and human preferences.\n\n* The results discussion is organized into five sections aligned with specific research questions, making it easy to follow and understand the key findings. These findings address overall alignment, LLM performance across six moral dimensions, language variation, performance in low-resource languages, and LLM robustness to prompt paraphrasing.", "weaknesses": "* Although the paper offers a detailed analysis of LLM performance across several research questions, it lacks an exploration of potential reasons behind the observed phenomena. While such explanations might be speculative given the limited information on LLM training, a discussion on why models from the same family (e.g., Llama 3) exhibit differing behaviors could enhance the paper.\n\n* The paper implicitly assumes that LLMs should align with country-level or language-level human preferences. However, these preferences may carry biases and could be unfair to marginalized groups. Conversely, the least aligned model, GPT-4o Mini, could be viewed as one that lacks language-specific biases and instead follows a standardized behavior pattern. Exploring this topic further in the paper could add valuable insights into the potential trade-offs between alignment and bias in multilingual settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)