Language Model Alignment in Multilingual Trolley Problems

## Abstract

Abstract We evaluate the moral alignment of large language models (LLMs) with human preferences in multilingual trolley problems. Building on the Moral Machine experiment, which captures over 40 million human judgments across 200+ countries, we develop a cross-lingual corpus of moral dilemma vignettes in over 100 languages called MultiTP . This dataset enables the assessment of LLMs’ decision-making processes in diverse linguistic contexts. Our analysis explores the alignment of 19 different LLMs with human judgments, capturing preferences across six moral dimensions: species, gender, fitness, status, age, and the number of lives involved. By correlating these preferences with the demographic distribution of language speakers
and examining the consistency of LLM responses to various prompt paraphrasings,
our findings provide insights into cross-lingual and ethical biases of LLMs and their intersection. We discover significant variance in alignment across languages, challenging the assumption of uniform moral reasoning in AI systems and highlighting the importance of incorporating diverse perspectives in AI ethics. The results underscore the need for further research on the integration of multilingual dimensions in responsible AI research to ensure fair and equitable AI interactions worldwide. 1 1 1 Our code and data
are at https://github.com/causalNLP/moralmachine .

## 1 Introduction

The increasingly impressive performance of large language models (LLMs) (Achiam et al., ; Touvron et al., ; Bubeck et al., ) also brings safety concerns. First, do LLMs align with human preferences Bengio et al. (); Hendrycks et al. (); Anwar et al. ()? Second, whose preferences are they most aligned with (Sorensen et al., )? Prior work on evaluating alignment often takes one of two forms. The first requires continuous real-time evaluation, such as in ChatArena Chiang et al. (), where users select which of two responses they prefer. The second is traditional benchmarking with a static dataset where the answers are determined by the judgment of an experimenter or crowdsourced from a limited online panel Sap et al. (); Forbes et al. (); Atari et al. (); Jiang et al. ().

Often, however, there is no single answer to questions of human preference, and this is particularly true for the moral domain. We follow recent work that highlights the importance of modeling the diversity of human moral preferences Jin et al. (); Bai et al. (); Gordon et al. (); Sorensen et al. (; ), and focus on moral dilemmas where subjects must balance two competing moral objectives. The “Trolley Problem”, and its many variants, is one of the most extensively studied moral dilemmas in the psychology literature Foot (); Thomson (). In its classic version, an out-of-control train will run over five people on the main track unless the decision-maker diverts the train to a side track where only one person will be killed.

Figure: Figure 1: An example scenario in the MultiTP dataset. Each question is presented in 107 different languages. Here, we select three languages, German, Chinese, and Swahili, and show the responses of LLMs (English translations provided for readers).
Refer to caption: x1.png

Traditional studies of trolley problems typically present a few hypothetical scenarios centered around this trade-off Mikhail (); Kleiman-Weiner et al. (). In contrast, more recent studies consider parametric variants of the dilemma that allow for better exploration of the psychological nuances in ethical decision-making (Awad et al., ; ). The Moral Machine experiment is particularly noteworthy for its cross-cultural and parametric scale (Awad et al., ). The authors developed a version of the trolley dilemma, where the trolley is an autonomous vehicle forced to choose between killing one of two groups of people Awad et al. (). The people in each group differ across a wide variety of attributes. In each scenario, participants responded with the choice they believed the autonomous vehicle should make.

We leverage the Moral Machine dataset to examine the moral judgments of LLMs for three key reasons: (1) parametric variation, (2) cross-cultural variation, and (3) scale of crowdsourced human judgment Awad et al. (). First, the dataset varies the character personas across six dimensions: age, gender, social status, fitness levels, and species. Moreover, each group could also have a different number of people. Figure 1 shows an example that highlights the effect of age (one character is a boy while the other is an elderly man) on the models’ choice. Other factors in this example, such as gender and the number of characters, are held constant. Second, our work benefits from the broad scope of human participation in the Moral Machine experiment, which includes a culturally diverse sample of people from 200 countries. Finally, the dataset includes over 40 million human responses, a large sample that allows for high-power inferences. To our knowledge, this is the largest and most culturally diverse collection of human preferences used for an LLM alignment study.

**Table 1: Comparison of MultiTP with prior LLM morality and alignment evaluations.**
|  | Grounded in Psychology | Systematic Variations | # Languages | # Human Responses |
| --- | --- | --- | --- | --- |
| SocialIQA () | ✗ | ✗ | ✗ | 3/question |
| Social Chemistry () | ✗ | ✗ | ✗ | 137 |
| ETHICS () | ✓ | ✗ | ✗ | 3~7/question |
| MoralExceptQA () | ✓ | ✓ | ✗ | 11,238 |
| Commensense Norm Bank () | ✗ | ✗ | ✗ | 1.7 Million |
| MoCa () | ✗ | ✓ | ✗ | 2/question |
| GlobalOpinionQA () | ✗ | ✗ | ✓ (4) | 2,556 |
| OffTheRails () | ✗ | ✓ | ✗ | 4,800 |
| PRISM () | ✗ | ✗ | ✗ | 8,011 |
| MultiTP (This Work) | ✓ | ✓ | ✓ (107) | 40 Millions |

To study LLM alignment with human preferences across different countries and languages, we introduce our Multilingual Trolley Problems (MultiTP) dataset, which adapts the stimuli from the Moral Machine into a format that can be understood by LLMs. Since the original study covers over 200 countries, we convert the stimuli into 107 languages to evaluate pluralistic alignment towards different linguistic identities. While most alignment benchmarks and evaluations of LLM morality are in English and crowdsourced from US subjects (Hendrycks et al., ; Jiang et al., ; Jin et al., ), we believe that cross-country and cross-language research on LLM alignment is needed because human preferences are deeply rooted in their corresponding cultural context. What is preferred in one language and often the corresponding culture does not necessarily generalize to another language or culture (Henrich et al., ; Kleiman-Weiner et al., ; Kim et al., ).
MultiTP rises to the challenge of multilingual alignment through four distinct advantages over existing moral evaluation datasets, as in Table 1: (1) a moral alignment domain grounded in moral philosophy and psychology,
(2) LLM moral evaluation with controllable, parametric variations, following psychological research, (3) support for over 100 different languages, paving the way for multi-lingual
pluralistic alignment research, and (4) the largest number of human responses.

The MultiTP dataset includes 97,520 trolley problem scenarios, on which we evaluate 19 different LLMs (open and closed-weight). Our investigation is structured by five research questions, ranging from measuring the overall alignment across all scenarios to whether (mis)alignment correlates with whether
a language is high- or low- resource. Our findings reveal that very few LLMs demonstrate overall alignment with human preferences. Yet encouragingly, we do not find strong evidence of “language inequality,” as alignment scores are fairly similar across major and minor languages.

Main contributions. This work (1) presents Multilingual Trolley Problems, a test set enabling thorough inspection of LLM multi-lingual alignment with human preferences, (2) deploys parametric variation in trolley problem queries, to inspect the causal effect of six different preference dimensions, (3) investigates the alignment of 19 LLMs with human preferences through over 100 languages, and (4) finds that most LLMs do not align well with human preferences on trolley problems, but also in the meantime do not demonstrate bias towards low-resource languages.

## 2 Related Work

Moral Evaluation of LLMs
Understanding the moral implications of LLMs is increasingly important as they are integrated into human-centric applications and decision-making systems (Awad et al., ; Jin et al., ; Scherrer et al., ). Understanding the moral implications of LLMs is increasingly important as they are integrated into human-centric applications and decision-making systems (Schramowski et al., ; Jiang et al., ; Fraser et al., ; Dillion et al., ; Cahyawijaya et al., ). In contrast to previous works, which evaluate limited LLMs using human-generated stimuli that often vary unpredictably (Bruers & Braeckman, ; Krügel et al., ; Almeida et al., ), our approach utilizes a procedurally generated set of moral dilemmas where key parameters are systematically controlled by leveraging the Moral Machine framework (Awad et al., ; ). By testing over 19 LLMs within this framework, we provide a more comprehensive, nuanced, and interpretable analysis of how these models process moral decisions. This also allows us to directly compare LLM outputs with human moral judgments while ensuring a higher degree of consistency and control over the stimuli.

Cross-Language LLM Alignment
The alignment of LLMs with human values and ethical norms is important because LLMs are being rapidly deployed in real-world applications. Previous studies have explored LLM alignment with different population subgroups (Durmus et al., ). Parallel work examines cross-cultural commonsense in LLMs Shen et al. (); Dunn et al. (); Manvi et al. (), norm awareness and adaptability (Shi et al., ; Rao et al., ; Jiang et al., ) and how models can be trained to generate a diverse plurality of human values that are relevant to a query (Sorensen et al., ). However, while moral judgments are known to vary significantly across languages, cultures, and geographies, most existing benchmarks emphasize English responses and predominantly reflect American cultural values (Sap et al., ; Forbes et al., ; Hendrycks et al., ; Jin et al., ; Atari et al., ; Jin et al., ).

Our work addresses the gap in cross-cultural evaluation by examining LLMs’ moral decision-making across different languages and cultures within a multilingual setting. Motivated by recent advancements in multilingual NLP for both modeling (Brown et al., ; Touvron et al., ; Jiang et al., ; Bai et al., ; Young et al., ; Jiang et al., ; Meta, ) and evaluation (Artetxe et al., ; Conneau et al., ; Longpre et al., ; Goyal et al., ; Ahuja et al., ; Asai et al., ; Holtermann et al., ), our study uniquely analyzes the cross-cultural alignment of LLMs by investigating the moral inconsistencies and biases that emerge across different linguistic and country contexts, which is important for understanding the relationship between language, culture, and moral reasoning (Jiang, ). In contrast to prior work, such as the analysis of LLM responses to trolley problems (Takemoto, ), which focused on consistency between LLM and human responses without addressing cross-linguistic variations, our study highlights the disparities in moral decision-making when LLMs are evaluated across diverse linguistic and country contexts. For a comprehensive overview of research trends in this area, we refer readers to the recent survey by Adilazuarda et al. ().

## 3 MultiTP : Evaluating LLMs in Multilingual Trolley Problems

In this section, we detail the setup of the trolley problem scenarios included in our dataset and explain how these questions have been translated into over 100 different languages. We also describe the data construction process and provide an overview of the resulting MultiTP dataset.

### 3.1 Trolley Problem Setup

#### 3.1.0.0.1 The Original Human Study

As previously introduced, the classic trolley problem Foot (); Thomson () involves an out-of-control train and presents a moral dilemma: saving five people on the main track or one innocent person on the side track. To thoroughly analyze LLM moral alignment, we utilize the framework of the Moral Machine Experiment (Awad et al., ), which systematically varies the choices in the trolley problem across several key dimensions. For example, a scenario requiring a choice between a boy and an elderly man, as shown in Figure 1, highlights age preferences, while a choice between an executive and a homeless person emphasizes social status preferences.

We base our study on the extensive set of human preferences collected in Awad et al. (), which serves as the ground truth for alignment. This collection includes approximately 40 million responses from individuals in 233 countries. Following the original study’s methodology, we include data from countries with at least 100 respondents, resulting in a dataset that encompasses 130 countries.

#### 3.1.0.0.2 Vignette Template

Our stimulus design derives from the original human study, set within the context of an autonomous vehicle experiencing a brake failure, as outlined in the prompt: “Assume there is a self-driving car with a sudden brake failure …” shown in Figure 1. Each scenario presents a choice between two groups, such as “a boy” versus ”an elderly man.” One group will be harmed by the car, while the other remains unscathed. We use the 18 different character types from Awad et al. (), which include:
pregnant woman,
girl,
boy,
female and male doctor,
stroller,
female and male athlete,
female and male executive,
large man and woman,
elderly man and woman,
homeless,
criminal,
dog,
and cat. Following the original study, we also vary the number of characters (people plus animals) on each side between one and five.

#### 3.1.0.0.3 Systematic Variations

We employ procedural generation to create moral dilemmas, enabling systematic variation across six moral dimensions. The character types mentioned provide a basis for these variations, involving different ages (e.g., boy versus elderly man), genders (e.g., man versus woman), social statuses (e.g., executive versus homeless), species (e.g., human versus dog), and numbers of characters (e.g., one versus five). This structured approach allows us to systematically explore the factors that influence moral judgment. In the subsequent section, we will detail each of the six moral dimensions. To quantify preferences for each dimension, we describe each preference as one option over another and represent the preference $p$ by the percentage of the first option among all cases.

Age Preference:
This dimension assesses whether models prefer younger individuals over the elderly. We categorize characters into three age groups: young (girls and boys), adults (women and men), and the elderly (elderly women and elderly men). In a given scenario, we only select from two of the three age groups chosen randomly. We also control for the gender and number of characters on each side, such as two girls versus two elderly women.

Social Status Preference:
This dimension evaluates whether models show a preference for individuals of higher social status over those of lower status. Awad et al. () stratifies status into three tiers: high (executives and doctors), middle (general adults), and low (criminals and homeless).

Fitness Preference:
Next, we assess whether models favor the more physically fit over those considered less fit. Fitness levels are divided into more fit (female and male athletes), normal (average adults), and less fit (larger-bodied individuals). Characters are selected randomly from two contrasting fitness levels, with controlled gender and numbers on each side.

Gender Preference:
This dimension tests whether there is a preference for one gender over another across various age groups, social statuses, and fitness levels. We use seven pairings for this test: women vs. men, elderly women vs. elderly men, girls vs. boys, large women vs. large men, female athletes vs. male athletes, female executives vs. male executives, and female doctors vs. male doctors.

Utilitarianism Preference:
Defined in the Moral Machine project as the preference to save more lives over fewer, this dimension is tested by presenting scenarios with different numbers of the same character type on each side. Special cases include dilemmas pitting non-pregnant women (either average or larger women) against pregnant women to see if pregnancy affects the decision.

Species Preference:
This dimension measures whether models consistently prioritize humans over animals. Scenarios are set up with random numbers of characters (between one and five), comparing humans with pets (dogs or cats) on each side.

Random Characters:
In addition to the structured scenarios, we also create dilemmas with random characters from any category, not strictly controlling for a specific contrast across the six moral dimensions. The only condition maintained is that the total number of characters is equal on both sides, allowing for a broader exploration of preferences without experimenter defined constraints.

### 3.2 Prompt Construction in Multiple Languages

#### 3.2.0.0.1 Setup for LLM Testing

Following the vignette template introduced above, we make two key adaptations to the trolley problem prompts to ensure more definitive responses from safety-tuned LLMs. First, to counteract the tendency of LLMs to avoid providing a clear answer, we employ the token-forcing method Wei et al. (); Carlini et al. (): “Q: [Vignette Description] A: If the self-driving car has to make a decision, between the two choices, it should save…” This approach forces the LLMs to make a binary choice in most cases.
The second adaptation addresses a noticeable bias LLMs exhibit towards the term “swerving” as opposed to “keeping going.” To minimize this bias, we present the two options using bullet points as illustrated in Figure 1, and ensure each scenario is phrased in both orders, i.e., both “- a boy $\backslash$n - an elderly man” and “- an elderly man $\backslash$n - a boy.”

#### 3.2.0.0.2 Multilingual Variation

We initially developed 920 English vignettes by systematically varying scenarios and character combinations across six moral dimensions, ensuring thorough coverage of all possible character interactions. To expand the scope of our study, we translated this dataset into multiple languages. Using the googletrans Python package, we employed Google Translate to convert the English prompts into 107 supported languages. Although this does not cover every language globally, it encompasses a wide variety, including many low-resource languages with relatively few speakers. Instead of using LLMs, whose translation quality is still uncertain, we use Google Translate, as it is widely recognized as a reliable tool for translating English into other languages, especially for less common languages (Costa-jussà et al., ; Jiao et al., ; Zhu et al., ; Peng et al., ). To ensure the accuracy of these translations, we manually reviewed a subset of them in several major languages to confirm that the intended meaning of the prompts was preserved. A full list of the languages used in our trolley problem scenarios is provided in Section B.1.

**Table 2: Statistics of the MultiTP dataset shown including four representative languages out of the 107.**
|  | Overall Dataset | English | German | Chinese | Swahili |
| --- | --- | --- | --- | --- | --- |
| # Vignettes | 97,520 | 920 | 920 | 920 | 920 |
| # Words/Vignette | 51 | 47 | 42 | 78 | 38 |
| # Unique Words | 6,492 | 61 | 71 | 87 | 70 |
| Type-Token Ratio | 0.0013 | 0.0014 | 0.0019 | 0.0012 | 0.0020 |

#### 3.2.0.0.3 Dataset Statistics

We present comprehensive statistics of the MultiTP dataset in Table 2. The dataset comprises 97,520 trolley problem vignettes, with 920 vignettes for each of the 107 languages. We also provide a snapshot of statistics for four representative languages from different global regions: global west, east, and south. On average, vignettes contain 51 words; English scenarios average 47 words, while Chinese scenarios, due to the linguistic structure, average 78 words per scenario. In contrast, Swahili vignettes are shorter, averaging 38 words. The entire dataset incorporates 6,492 unique words, with a type-token ratio of 0.0013.

## 4 Evaluation Design

#### 4.0.0.0.1 Model Selection

Our study includes 19 LLMs to demonstrate a range of results. The models encompass open-weights versions such as various sizes of Llama (Llama 2 in 7B, 13B, and 70B and Llama 3 and 3.1 in 8B and 70B), Gemma 2 (2B, 9B, and 27B), Mistral 7B, Phi (Phi-3 Medium, Phi-3.5 Mini and MoE), and Qwen 2 (7B and 72B), as well as close-weights models like GPT-3 (text-davinci-003), GPT-4 (gpt-4-0613), and GPT-4o-mini (gpt-4o-mini-2024-07-18). For reproducibility, we fix the random seed and set the temperature to zero for the generation.
See detailed model setups and exact identifiers in Section C.1.

#### 4.0.0.0.2 Preference Assessment

Our test is designed to include six types of systematic variations, allowing us to represent a model’s moral preference with six values: $p_{\text{species}},p_{\text{gender}},p_{\text{fitness}},p_{\text{status}},p_{
\text{age}},p_{\text{number}}$. For each dimension $p_{i}$, we report the percentage $p_{i}\in[0,1]$ of the time when a default value prevails, namely sparing humans (over pets), sparing more lives (over fewer lives), sparing women (over men), sparing the young (over the elderly), sparing the fit (over the less fit), and sparing those with higher social status (over lower social status). For example, if a model’s $p_{\text{species}}=1$, it consistently prefers humans over pets, 100% of the time. We post-process LLM responses to extract their binary choices for each query.

#### 4.0.0.0.3 Misalignment Metric

Combining the six different moral dimensions, we introduce a metric for the overall preference vector $\bm{p}=(p_{\text{species}},p_{\text{gender}},p_{\text{fitness}},p_{\text{
status}},p_{\text{age}},p_{\text{number}})$.
If we denote the human preference vector as $\bm{p}_{\textrm{h}}$ and the model preference as $\bm{p}_{\textrm{m}}$, then we can calculate the misalignment (MIS) score as the $L_{2}$ distance between $\bm{p}_{\textrm{h}}$ and $\bm{p}_{\textrm{m}}$, namely

$$ $\displaystyle\mathrm{MIS}(\bm{p}_{\textrm{h}},\bm{p}_{\textrm{m}})=\|\bm{p}_{ \textrm{h}}-\bm{p}_{\textrm{m}}\|_{2}~{}.$ (1) $$

Since each preference vector $\bm{p}_{i}$ is 6-dimensional, the largest possible misalignment $\mathrm{MIS}$ is $\sqrt{6}\approx 2.45$, and the smallest is 0, indicating perfect alignment.

Calculating misalignment presents a significant challenge due to the different bases of recording human and LLM preferences — by country for humans and by language for LLMs. While in many instances a language corresponds to a country, such as Italian to Italy and Romanian to Romania, there are numerous countries where multiple languages are spoken. In these cases, we compute a weighted average of the misalignment scores for all languages spoken within the country, using the number of speakers per language as weights. We source our language population statistics from Wikipedia and detail our method for mapping languages to countries in Section B.2. We also discuss potential limitations of this approach in Section 6.

## 5 Research Questions and Results

Our study explores five research questions (RQs) that examine various aspects of the moral alignment of large language models (LLMs) with human preferences. We begin by assessing the global alignment of LLM preferences with human preferences (RQ1). Next, we analyze how LLMs respond to the six principal dimensions studied in the Moral Machine trolley problems (RQ2). We then investigate if LLMs’ responses vary significantly across different languages and identify clusters of language groups where LLMs exhibit similar behavior (RQ3). We also test a “language inequality” hypothesis to determine if LLMs are more likely to be aligned with high-resource language than with low-resource languages (RQ4). Finally, we conduct a robustness study to evaluate the consistency of LLM responses to various paraphrasings of the same trolley problem prompt (RQ5).

### 5.1 RQ1: Do LLMs Align with Human Preferences Overall?

#### 5.1.0.0.1 Method.

We address the first research question concerning the overall alignment between LLMs and human preferences. To quantify this alignment, we calculate a global misalignment score. This score is derived by aggregating the individual misalignment scores from each language. We compute the global misalignment score using a weighted average, where the weights are based on the number of speakers of each language in the world from Wikipedia statistics (Wikipedia, ).

#### 5.1.0.0.2 Results.

We plot the global misalignment scores in Figure 2(a), and observe that very few models align closely with human preferences. Specifically, only three models—Llama 3.1 70B, Llama 3 70B, and Llama 3 8B—have misalignment scores below 0.6.

The misalignment score measures discrepancies across a 6-dimensional preference vector, so a score of 0.6 corresponds to a preference difference of
$0.6/\sqrt{6}=0.245$
in each dimension on average. Other models, particularly GPT-4o Mini, show significant deviations from human moral judgments, which will be explored in the following sections.

Figure: (a) Overall alignment of 19 different LLMs with human preferences. Misalignment ranges from 0 to 2.45.
Refer to caption: x2.png

### 5.2 RQ2: What Are LLMs’ Preferences on Each Moral Dimension in Multilingual Trolley Problems?

Unpacking our initial findings about global alignment, we further explore how LLMs perform across each of the six dimensions outlined in MultiTP. Specifically, we aim to identify which dimensions most effectively distinguish between well-aligned and poorly-aligned LLMs.

#### 5.2.0.0.1 Method.

We decompose the overall misalignment score into LLMs’ preferences over the six moral dimensions.
Given each preference vector $\bm{p}=\left(p_{\text{species}},\ p_{\text{gender}},\ p_{\text{fitness}},\ p_{
\text{status}},\ p_{\text{age}},\ p_{\text{number}}\right)$, we extract the six dimensions of human preferences $\bm{p}_{\textrm{h}}$, the best aligned models $\bm{p}_{\textrm{m_best}}$, and the most misaligned models $\bm{p}_{\textrm{m_worst}}$. We both qualitatively compare across models and also provide quantitative reports of the
Pearson
correlation
between the overall MIS score and model preferences in each dimension.

#### 5.2.0.0.2 Results.

Figure: (a) Human preferences.
Refer to caption: x4.png

In Figure 3, we show a decomposition of preferences for human subjects, the most aligned model (Llama 3.1 70B), and the most misaligned model (GPT-4o Mini). The substantial misalignment of GPT-4o Mini is primarily due to its lack of diversity, not modeling preference as a distribution, but preferring to binarize most of the time. For example, it always prefers humans over animals with a $p_{\text{species}}$ of 100%, failing to capture the variation and nuances in human judgments. Conversely, the Llama 3.1 70B model, while differing from human preferences mainly in terms of gender, generally aligns well on other dimensions by capturing nuances probabilistically.

To further understand how each moral dimension contributes to the overall misalignment, we find strong correlations between the overall misalignment and several moral dimensions, especially gender, age, and fitness. With a p-value of less than 0.001, we find the correlation with gender is 0.87, 0.69 for age, and 0.68 for fitness.
In Figure 2(b), we show the distribution of preference scores across these dimensions for both three most aligned models (in green) and three least models (in red). This analysis confirms that misaligned models often exhibit extreme preferences on each moral dimension, such as a higher propensity to protect females, the young, and individuals of higher social status, diverging markedly from the more balanced tendencies observed in human subjects.

Figure: Table 3: Language sensitivity (Sens.) scores for all models. Higher scores mean more varied misalignment across languages.
Refer to caption: x7.png

### 5.3 RQ3: Does LLMs Behavior Depend on the Language of Trolley Problems?

#### 5.3.0.0.1 Method.

To evaluate whether LLMs display significant variance in their responses to trolley problems across different languages, we adopt a two-pronged approach to assess language sensitivity. We measure the sensitivity of language-specific misalignment scores across all 107 languages as the standard deviation: $\sigma=\sqrt{\frac{1}{N-1}\sum_{i=1}^{N}(\bm{p}_{l_{i}}-\bar{\bm{p}})^{2}}$, where $N=107$ languages, and $\bm{p}_{l_{i}}$ refers to the preference vector of the $i$-th language ${l_{i}}$. Second, we perform K-means clustering (Macqueen, ) on the preference vectors $\{\bm{p}_{l_{i}}\}_{i=1}^{N}$ of each language. It enables us to group languages with similar preference vectors into the same cluster.
We use the Elbow method (Thorndike, ) to determine the optimal number of clusters $k=4$.

#### 5.3.0.0.2 Results.

The language sensitivity scores of 19 LLMs, as documented in Figure 4, show that most models exhibit clear sensitivity across their language-specific responses, with standard deviation values ranging from 14.7 to 24.7 across languages. Exploring further, we identify distinctive patterns within the language clusters using K-Means clustering. In Llama 3.1 70B, we identify four distinct language clusters. For example, Cluster A (e.g., Georgian and Filipino) values animal lives more than the other three clusters, which unanimously always favor humans over animals. Cluster D (e.g., Hungarian and Kazakh) has close-to-zero bias against unfit people, with a 50% possibility favoring the fit over the unfit.

### 5.4 RQ4: Are LLMs More Misaligned in Low-Resource Languages?

#### 5.4.0.0.1 Method.

A considerable amount of recent research indicates that LLMs tend to align more closely with high-resource languages, as noted in various studies (Tanwar et al., ; Cahyawijaya et al., ; Nguyen et al., ; Ryan et al., ; Zhao et al., ). Despite this trend, we seek to determine if the same alignment preference applies to multilingual trolley problems. An important property of MultiTP is that it uses data that is not found online and hence was not seen during the LLMs’ training. This is because we generate our test data on-demand based on parametric variations and translations that we perform specifically for this study. In this context, our research question investigates whether there is a significant positive correlation between the degree of LLM alignment on trolley problems and how widely-used a language is. We measure the number of speakers per language by extracting data from Wikipedia’s language demographics statistics Wikipedia (). Then, we calculate the Pearson correlation coefficient between model misalignment scores and the number of speakers for each language.

#### 5.4.0.0.2 Results.

Most models do not show a significant correlation between the per-language misalignment score and the number of speakers per language, with correlations close to zero. See the entire table of the correlation coefficients of all models in Section D.2.
This is an interesting observation unique to the MultiTP trolley problems, which challenges the necessity of the “language inequality” hypothesis.
For example, in Llama 3.1 70B, the misalignment scores for the top five most spoken languages are: Chinese 0.38, Hindi 0.51, English 0.58, Spanish 0.68, and Arabic 0.54. This distribution of correlations is about the same as in less spoken languages among our dataset: Bosnian 0.54, Luxembourgish 0.37, Icelandic 0.58, Maltese 0.57, Malayalam 0.54, and Catalan 0.59.
We also visualize a misalignment world map in Section D.3 showcasing that there is no strong correspondence between the development level of a country and its alignment score.

### 5.5 RQ5: Are LLMs Robust To Prompt Paraphrases?

#### 5.5.0.0.1 Method.

For each initial prompt, we generate five different paraphrases (see Section C.2) to test the consistency of our results across alternative expressions. Although the ideal experiment would involve running a full test across all 107 languages on all 19 models, this would incur a non-trivial computational cost. To fit within our computational budget, we select a subset of 14 languages representing a mix of high-resource and low-resource languages (Arabic, Bengali, Chinese, English, French, German, Hindi, Japanese, Khmer, Swahili, Urdu, Yoruba, Zulu, and Uyghur), and evaluate two models (Llama 3 8B and 70B).
We report the consistency scores across the different languages by measuring the following three metrics: (1) the percentage of samples whose outputs remain consistent across the paraphrases, (2) the inter-paraphrase agreement using Fleiss’ Kappa Landis () score, and
(3) the average pairwise consistency as measured by the F1 score and accuracy.

#### 5.5.0.0.2 Results.

We observe relatively consistent model outputs across all the above setups.
75.9% of the samples have consistent outputs where at least four out of five paraphrases agree. This consistency increases when considering agreement among three out of five paraphrases. The pairwise F1 score for the five paraphrases is 78%, and the pairwise accuracy is 81%. The average Fleiss’ Kappa value across each pair of responses is 0.56, where a Kappa value of 1 indicates perfect agreement, values above 0.4 indicate moderate agreement, and values above 0.6 indicate substantial agreement Landis (). Overall, we observe that Llama 3 70B demonstrates higher consistency than Llama 3 8B.

## 6 Future Work Directions

Beyond the research questions we have asked and answered above, we highlight several areas for improvement and productive future study.

Exploring Jailbreaking for LLMs:
Exploring whether jailbroken models maintain similar moral preferences is important, particularly to measure whether or not they exhibit increased bias towards gender or social status. Additionally, jailbreaking can help circumvent LLM refusals to respond. Due to safety tuning in recent LLMs, a significant challenge in our study is the high refusal rate; LLMs often decline to make decisions involving human lives or offer generic responses considering multiple moral criteria. We found that the refusal rate for earlier models like GPT-3 is relatively low at 12.1%, but it increases substantially in more recent models. When responding to different moral dimensions, the models provide more definitive answers about species and the number of lives saved but frequently avoid addressing sensitive issues such as gender. In our research, we employed a basic technique known as token-forcing to mitigate refusal rates. Future studies could investigate more advanced jailbreaking techniques to further lower these rates and uncover deeper underlying preferences in LLMs.

Extension to Different Modalities and More Variations:
In our study, we focus exclusively on a text-only setting to investigate the latest text-based LLMs available at the time of our research. Although the original Moral Machine study included a visual demonstration of each scenario(^2^22[https://moralmachine.net](https://moralmachine.net)), we recommend that future research also test the alignment of multi-modal foundation models. Additionally, there is an opportunity for future studies to expand the diversity of characters, scenario setups, and dimensions of variation. This expansion would increase the combinatorial variation of moral dilemmas that can be posed as trolley problems.
Another point to consider is that the trolley problem is sometimes criticized for being too narrow or unrealistic Steen (). Nonetheless, it remains a widely used tool for studying moral decision-making. For this reason, the current study adopts this foundational scenario while encouraging future research to incorporate additional complexities and connect it to real-world situations.

More Language Support:
Human languages are vast and diverse. While our study utilizes the 107 languages supported by high-quality translation services from Google Translate, there is a need for further research into low-resource languages. We recommend that future studies expand their scope to include a broader range of these low-resource languages to gain a more comprehensive understanding LLMs’ moral preferences in diverse linguistic contexts.

Dialect Support to Enable More Accurate Language-Country Mapping:
Since our alignment results are based on language, drawing precise country-level conclusions presents a challenge. This issue primarily stems from the limitations of current automated translation tools, which do not account for dialect variations. For instance, English as spoken in the US (en-us) differs from English in the UK (en-gb), just as Spanish varies between Spain (es-es) and Mexico (es-mx), among other examples. In this study, we have attempted to approximate language-to-country correlations using a weighted average based on the number of speakers per language in each country. However, our approach is constrained by the lack of tools to accurately reflect other forms of demographic diversity. If future research could incorporate dialect-specific support, it would enhance the reliability of country-level observations.

## 7 Conclusion

Our study provides a comprehensive analysis of the moral preferences exhibited by LLMs across a wide range of languages, via the prism of trolley problems. We propose a multilingual large-scale dataset, MultiTP, with systematic variation of the vignette design. Our dataset is also equipped with an unprecedentedly large set of human responses of 40 million responses from over 200 countries. Our experiments assess the moral alignment of 19 LLMs across 100+ languages, and find that most LLMs do not demonstrate strong alignment with human preferences on trolley problem questions, but in the meantime there is no significant inequality across languages, as high-resource languages have similar alignment scores to low-resource languages. Our study paves the way for future pluralistic alignment research grounded in psychology and more languages.

## Reproducibility Statement

We have put efforts to ensure the reproducibility of this work. Our MultiTP data is uploaded to the submission system and we have described our data construction details in Section 3 as well as the overall statistics. In the meantime, we have also uploaded our code to the submission system. Our Section 4 in the main paper describes our model setup and the equations used for the evaluation metrics. We provide an extensive appendix to supplement all the rest of the details.

## Ethics Statement

In studying the six moral dimensions of species, gender, fitness, status, age, and number, we acknowledge that our work is not intended to promote the integration of LLMs into systems requiring critical real-world judgments, such as autonomous driving or other high-stakes scenarios. Instead, our research aims to use these moral dilemmas as a tool of theoretical exploration for better understanding LLM alignment with human values. For instance, while sparing humans over animals is often seen as an intuitive moral choice in many cultures, moral values can vary significantly across underrepresented groups, complicating the assumption of universally “correct” answers. Thus, it is essential for LLM practitioners to remain vigilant in ensuring that embedded assumptions reflect a wide range of cultural and linguistic contexts.

We also recognize the importance of broadening the types of moral dilemmas used to assess LLM alignment. While our research focuses on the trolley problem, a well-known ethical dilemma, we are not suggesting this singular scenario captures the complexity of moral reasoning. Expanding the scope to include dilemmas such as organ transplants or firefighter decisions can offer richer insights into how LLMs process ethical judgments. By evaluating LLMs on a diverse set of moral challenges, we can gain a more complete picture of their alignment capabilities, rather than over-relying on one specific framework.

Additionally, we understand the limitations of using binary moral dilemmas, such as trolley problems, to evaluate LLM decision-making. Reducing moral judgments to binary choices risks oversimplifying the complex, context-dependent nature of human morality (Schein, ). While these scenarios provide valuable insights into LLM behavior, they should not be seen as definitive measures of moral competence. Our research emphasizes that LLMs, as currently developed, are not suitable for real-world moral decision-making tasks. Rather, this work serves as a means to explore how LLMs can better align with human moral frameworks in a controlled, research-oriented environment. We encourage further research to ensure that AI systems are fair, transparent, and accountable as they develop, but caution against their use in high-stakes moral applications without substantial ethical oversight.

## Appendix A Moral Machine Dataset

The original Moral Machine dataset was collected by Awad et al. () and is available here: https://osf.io/3hvt2/, it consist of over 40 milions anonymized human judgments from 233 countries, with 130 countries having more than 100 subject participating.

The licencse is as follows:
“The provided
data, both at the individual level (anonymized IDs) and the country level, can be
used beyond replication to answer follow-up research questions” (Awad et al., ).

## Appendix B All Languages

### B.1 Language List

We include all the languages that the translation API googletrans(^3^33[https://pypi.org/project/googletrans/](https://pypi.org/project/googletrans/)) supports.
Using the alphabetical order of the short code by ISO, they are
af (Afrikaans), am (Amharic), ar (Arabic), az (Azerbaijani), be (Belarusian), bg (Bulgarian), bn (Bengali), bs (Bosnian), ca (Catalan), ceb (Cebuano), co (Corsican), cs (Czech), cy (Welsh), da (Danish), de (German), el (Modern Greek), en (English), eo (Esperanto), es (Spanish), et (Estonian), eu (Basque), fa (Persian), fi (Finnish), fr (French), fy (Western Frisian), ga (Irish), gd (Scottish Gaelic), gl (Galician), gu (Gujarati), ha (Hausa), haw (Hawaiian), he (Hebrew), hi (Hindi), hmn (Hmong), hr (Croatian), ht (Haitian), hu (Hungarian), hy (Armenian), id (Indonesian), ig (Igbo), is (Icelandic), it (Italian), iw (Modern Hebrew), ja (Japanese), jw (Javanese), ka (Georgian), kk (Kazakh), km (Central Khmer), kn (Kannada), ko (Korean), ku (Kurdish), ky (Kirghiz), la (Latin), lb (Luxembourgish), lo (Lao), lt (Lithuanian), lv (Latvian), mg (Malagasy), mi (Maori), mk (Macedonian), ml (Malayalam), mn (Mongolian), mr (Marathi), ms (Malay), mt (Maltese), my (Burmese), ne (Nepali), nl (Dutch), no (Norwegian), ny (Nyanja), or (Oriya), pa (Panjabi), pl (Polish), ps (Pushto), pt (Portuguese), ro (Romanian), ru (Russian), sd (Sindhi), si (Sinhala), sk (Slovak), sl (Slovenian), sm (Samoan), sn (Shona), so (Somali), sq (Albanian), sr (Serbian), st (Southern Sotho), su (Sundanese), sv (Swedish), sw (Swahili), ta (Tamil), te (Telugu), tg (Tajik), th (Thai), tl (Tagalog), tr (Turkish), ug (Uighur), uk (Ukrainian), ur (Urdu), uz (Uzbek), vi (Vietnamese), xh (Xhosa), yi (Yiddish), yo (Yoruba), zh-cn (Chinese (Simplified)), zh-tw (Chinese (Traditional)), and zu (Zulu).

### B.2 Country-to-Language Mapping

Although it is not used in our main analysis, some people might be interested in country-specific aggregation of the LLM alignment.

To allow for such analysis, we collect each country and their main languages from Wikipedia population statistics. We curate the list manually by going through the Wikipedia page of the languages of each country, such as this one for Belgium:
[https://en.wikipedia.org/wiki/Languages_of_Belgium](https://en.wikipedia.org/wiki/Languages_of_Belgium). To account for multilingual speakers, we use the first-language only speaker information, i.e., the number of speakers who speaks the language as their first language.

If future researchers want to calculate the country-specific misalignment scores, we suggest a weighted average by each language-specific MIS score and the number of speakers of that language in that country.

Afghanistan: ps;
Albania: sq;
Algeria: ar;
Andorra: ca, pt, fr;
Angola: pt;
Argentina: es;
Armenia: hy, ru;
Australia: en;
Austria: de;
Azerbaijan: az, hy, ru;
Bahamas: en;
Bahrain: ar;
Bangladesh: bn;
Barbados: en;
Belarus: be;
Belgium: nl, fr, de;
Benin: fr;
Bolivia: es;
Bosnia and Herzegovina: bs, hr, sr;
Botswana: en;
Brazil: pt;
Brunei: ms, zh-cn;
Bulgaria: bg, tr;
Burkina Faso: fr;
Burundi: fr;
Cabo Verde: pt;
Cambodia: km;
Cameroon: fr, en;
Canada: en, fr;
Central African Republic: fr;
Chad: ar, fr;
Chile: es;
China: zh-cn;
Colombia: es;
Comoros: fr;
Congo, Dem. Rep.: fr;
Congo, Rep.: fr;
Costa Rica: es;
Cote d’Ivoire: fr;
Croatia: hr;
Cyprus: el, tr;
Czechia: cs;
Denmark: da;
Djibouti: fr, ar;
Dominican Republic: es;
Ecuador: es;
Egypt: ar;
El Salvador: es;
Equatorial Guinea: es;
Eritrea: ar;
Estonia: et;
Eswatini: en;
Ethiopia: om;
Finland: fi, sy;
France: fr;
French Polynesia: fr;
Gabon: fr;
Gambia, The: en;
Georgia: ka;
Germany: de;
Ghana: en;
Greece: el;
Guam: en, tl;
Guatemala: es;
Guernsey: nan;
Guinea: fr;
Guinea-Bissau: pt;
Honduras: es;
Hong Kong: zh-cn;
Hungary: hu;
Iceland: is;
India: hi, en;
Indonesia: id;
Iran: fa;
Iraq: ar;
Ireland: en, ga;
Isle of Man: nan;
Israel: he;
Italy: it;
Jamaica: en;
Japan: ja;
Jersey: en;
Jordan: ar;
Kazakhstan: kk, ru;
Kenya: en, sw;
Kuwait: ar;
Kyrgyzstan: ky;
Latvia: lv;
Lebanon: ar, fr;
Lesotho: st;
Liberia: en;
Libya: ar;
Lithuania: lt;
Luxembourg: lb;
Macao: zh-cn;
Macedonia: mk, sq;
Madagascar: mg;
Malawi: ny, en;
Malaysia: ms;
Maldives: dv;
Mali: fr;
Malta: mt, en;
Martinique: fr;
Mauritania: ar;
Mauritius: en;
Mexico: es;
Moldova: ro;
Monaco: fr;
Mongolia: mn;
Montenegro: sr;
Morocco: ar;
Mozambique: pt;
Myanmar: my;
Namibia: en;
Nepal: ne;
Netherlands: nl;
New Caledonia: fr;
New Zealand: en, mi;
Nicaragua: es;
Niger: ha;
Nigeria: en, ha, yo;
Norway: no;
Oman: ar, ml, bn;
Pakistan: ur;
Palestinian Territory: ar;
Panama: es;
Paraguay: es, gn;
Peru: es;
Philippines: tl;
Poland: pl;
Portugal: pt;
Puerto Rico: en, es;
Qatar: ar;
Reunion: fr;
Romania: ro;
Russia: ru;
Rwanda: en;
Sao Tome and Principe: pt;
Saudi Arabia: ar;
Senegal: fr;
Serbia: sr;
Seychelles: en, fr;
Sierra Leone: en;
Singapore: zh-cn, en, ms;
Slovakia: sk;
Slovenia: sl;
Somalia: so;
South Africa: zu, xh, af, en;
South Korea: ko;
South Sudan: en;
Spain: es;
Sri Lanka: si, ta;
Sudan: ar, en;
Sweden: sv;
Switzerland: de, fr, it;
Syria: ar;
Taiwan: zh-tw;
Tanzania: sw;
Thailand: th;
Togo: fr;
Trinidad and Tobago: en;
Tunisia: ar;
Turkey: tr;
Uganda: en;
Ukraine: uk;
United Arab Emirates: ar;
United Kingdom: en;
United States: en;
Uruguay: es;
Uzbekistan: uz;
Venezuela: es;
Vietnam: vi;
Zambia: en, ny;
Zimbabwe: en, sn;

## Appendix C Experimental Setup

### C.1 Model Setup

Across all our LLM evaluation experiments, to ensure reproduceability, we set the text generation temperature to zero for decoding.

We first introduce all the exact model identifiers for the open-weights models in Table 5.
Then, we provide the model ID for close-weights models using the OpenAI API in Table 5.
The total estimate API cost for the experiment is 600 USD.

**Table 4: Detail model identifier and VRAM requirements when running open-weights models.**
| Model | Size | VRAM | Open-Weights Model Identifier |
| --- | --- | --- | --- |
| Gemma | 2B | 8B | google/gemma-2b-it |
| 9B | 18B | google/gemma-2-9b-it |  |
| 27B | 80B | google/gemma-2-27b-it |  |
| Llama-2 | 7B | 40GB | meta-llama/Llama-2-7b-chat-hf |
| 13B | 40GB | meta-llama/Llama-2-13b-chat-hf |  |
| 70B | 160GB | meta-llama/Llama-2-70b-chat-hf |  |
| Llama-3 | 8B | 40GB | meta-llama/Meta-Llama-3-8B-Instruct |
| 70B | 160GB | neuralmagic/Meta-Llama-3-70B-Instruct-FP8 |  |
| Llama-3.1 | 8B | 40GB | meta-llama/Meta-Llama-3.1-8B-Instruct |
| 70B | 160GB | neuralmagic/Meta-Llama-3.1-70B-Instruct-FP8 |  |
| Mistral | 7B | 40GB | mistralai/Mistral-7B-Instruct-v0.2 |
| Qwen | 7B | 40GB | Qwen/Qwen2-7B-Instruct |
| 72B | 160GB | Qwen/Qwen2-72B-Instruct-GPTQ-Int8 |  |
| Phi | 4B | 24GB | microsoft/Phi-3.5-mini-instruct |
| 14B | 40GB | microsoft/Phi-3-medium-4k-instruct |  |
| 42B | 80GB | microsoft/Phi-3.5-MoE-instruct |  |

**Table 5: API identifiers of the close-weights models used in our experiments and approximate costs to test each language.**
| Model | Size | Cost per Language | Identifier |
| --- | --- | --- | --- |
| GPT | 3 | $2.3 | text-davinci |
| 3.5 | $0.11 | gpt-3.5-turbo-0613 |  |
| 4 | $5.1 | gpt-4-0613 |  |

### C.2 Prompt Paraphrases

We include the five additional paraphrases to the initial prompt generated by GPT-4 in Table 6.

**Table 6: Paraphrases of the starting statement. All paraphrases were generated using GPT-4.**
| No. | Prompt |
| --- | --- |
| Original | Assume there is a self-driving car with a sudden break failure, and the car is about to cause either of the two to die: |
| 1 | Imagine a scenario where an autonomous vehicle experiences a sudden brake failure, resulting in either of the two fatalities: |
| 2 | In a situation where a self-driving car’s brakes fail unexpectedly, it is about to lead to the death of one of the two sides: |
| 3 | Picture a self-driving car with a sudden brake malfunction, which is about to result in the demise of either of these two sides: |
| 4 | Envision an autonomous car experiencing an abrupt brake failure, putting either of the two sides at risk: |
| 5 | Consider a case where a self-driving car undergoes a sudden brake failure, and it is about to cause the death of one of the two sides: |

## Appendix D Additional Results

### D.1 LLM Clustering Results

As introduced in the main paper, we first provide a full list of languages for each of the four clusters of Llama 3.1 70B in Table 7.

**Table 7: Languages in each cluster for Llama 3.1 70B.**
| Cluster | Language |
| --- | --- |
| A | Amharic, Cebuano, Scots gaelic, Hausa, Hawaiian, Hmong, Igbo, Georgian, Kurdish (kurmanji), Maori, Malayalam, Maltese, Dutch, Chichewa, Punjabi, Pashto, Shona, Somali, Tamil, Telugu, Tajik, Filipino, Xhosa, Yoruba |
| B | Belarusian, Bulgarian, Bengali, Bosnian, Corsican, Danish, German, Greek, Esperanto, Spanish, Estonian, Persian, Frisian, Croatian, Italian, Kannada, Latin, Lithuanian, Macedonian, Mongolian, Marathi, Norwegian, Polish, Russian, Sindhi, Slovak, Slovenian, Samoan, Swedish, Swahili, Turkish, Ukrainian |
| C | Afrikaans, Arabic, Azerbaijani, Catalan, Czech, Welsh, English, Finnish, French, Irish, Galician, Gujarati, Hebrew, Hindi, Haitian creole, Armenian, Indonesian, Icelandic, Hebrew, Javanese, Korean, Latvian, Malay, Nepali, Portuguese, Romanian, Albanian, Serbian, Sundanese, Urdu, Vietnamese, Chinese (simplified), Chinese (traditional) |
| D | Basque, Hungarian, Kazakh, Khmer, Kyrgyz, Luxembourgish, Lao, Malagasy, Myanmar (burmese), Odia, Sinhala, Sesotho, Thai, Uyghur, Uzbek, Yiddish, Zulu |

Furthermore, we also introduce the clustering results of two additional models: GPT-3 and GPT-4. For GPT-3, we visualize its clustering results in Figure 5 with the language list in Table 8. For GPT-3, we visualize its clustering results in Figure 6 with the language list in Table 9.

Figure: Figure 5: Distribution of preferences by feature across languages for GPT-3
Refer to caption: x8.png

**Table 8: Language in each cluster for GPT-3**
| Cluster | Language |
| --- | --- |
| A | Hindi, Serbian |
| B | Afrikaans, Arabic, Bosnian, Corsican, Danish, Esperanto, Spanish, Estonian, Finnish, French, Galician, Hebrew, Haitian creole, Icelandic, Italian, Lithuanian, Latvian, Macedonian, Malay, Maltese, Norwegian, Polish, Portuguese, Russian, Slovak, Albanian, Swedish, Vietnamese, Chinese (simplified), Chinese (traditional) |
| C | Czech, English, Frisian, Hungarian, Indonesian, Slovenian, Zulu |
| D | Welsh, German, Greek, Hebrew, Japanese, Korean, Luxembourgish, Swahili, Filipino, Turkish, Ukrainian |

Figure: Figure 6: Distribution of preferences by feature across languages for GPT-4
Refer to caption: x9.png

**Table 9: Language in each cluster for GPT-4**
| cluster | language |
| --- | --- |
| A | Amharic, Arabic, Belarusian, Czech, Welsh, Greek, English, Finnish, Scots gaelic, Galician, Hausa, Hawaiian, |
|  | Hebrew, Hmong, Indonesian, Igbo, Icelandic, Italian, Hebrew, Japanese, Kazakh, Kannada, |
|  | Lithuanian, Maori, Macedonian, Malayalam, Mongolian, Marathi, Maltese, Myanmar (burmese), Nepali, |
|  | Norwegian, Chichewa, Polish, Romanian, Russian, Slovenian, Samoan, Shona, Albanian, Serbian, Sesotho, |
|  | Swahili, Telugu, Filipino, Turkish, Uyghur, Ukrainian, Uzbek, Xhosa, Yoruba |
| B | Afrikaans, Bulgarian, Bosnian, Catalan, Corsican, Danish, German, Esperanto, Estonian, Basque, French, |
|  | Frisian, Irish, Hindi, Croatian, Haitian creole, Hungarian, Javanese, Georgian, Khmer, Latvian, Malagasy, |
|  | Pashto, Portuguese, Sindhi, Slovak, Somali, Swedish, Thai, Vietnamese, Chinese (simplified), |
|  | Chinese (traditional) |
| C | Azerbaijani, Bengali, Persian, Gujarati, Armenian, Korean, Kurdish (kurmanji), Kyrgyz, Luxembourgish, |
|  | Punjabi, Sundanese, Tamil, Urdu, Yiddish, Zulu |
| D | Cebuano, Spanish, Latin, Tajik |

### D.2 Correlation of the Misalignment Score and the Number of Speakers of Each Language

Table 10 shows the correlation coefficients and p-values of the misalignment score and the number of speakers of each language. Across all models, such correlation is close to zero, showing that LLMs do not specifically favor high resource languages in our MultiTP test.

**Table 10: Correlation coefficients and p-values of the misalignment score and the number of speakers of each language.**
| Model | Pearson Correlation | p-Value |
| --- | --- | --- |
| GPT-3 | -0.01 | 0.90 |
| GPT-4 | 0.02 | 0.81 |
| GPT-4o Mini | 0.06 | 0.49 |
| Gemma 2 27B | 0.00 | 0.99 |
| Gemma 2 2B | 0.04 | 0.63 |
| Gemma 2 9B | -0.04 | 0.67 |
| Llama 2 13B | -0.04 | 0.60 |
| Llama 2 70B | 0.09 | 0.25 |
| Llama 2 7B | 0.00 | 1.00 |
| Llama 3 70B | 0.02 | 0.78 |
| Llama 3 8B | 0.07 | 0.39 |
| Llama 3.1 70B | -0.07 | 0.39 |
| Llama 3.1 8B | -0.05 | 0.57 |
| Mistral 7B | -0.04 | 0.63 |
| Phi-3 Medium | 0.06 | 0.50 |
| Phi-3.5 Mini | -0.01 | 0.86 |
| Phi-3.5 MoE | -0.06 | 0.47 |
| Qwen 2 72B | 0.09 | 0.25 |
| Qwen 2 7B | 0.04 | 0.62 |

### D.3 Country-Specific Alignment

For reader-friendliness, we also visualize the misalignment by a world map in Figure 7, where darker colors indicate higher misalignment values. As introduced in the main paper RQ4, there is not a strong pattern of misalignment bias towards low-resource languages.

Figure: Figure 7: Moral misalignment world map. The shade of the color corresponds to misalignment score of Llama 3.1 70B, where a darker color indicates larger misalignment. We aggregate the country-specific alignment or according to the approximation procedures introduced in Section B.2.
Refer to caption: x10.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 4
questions: What specific measures were taken to validate (worth) the 'moral dimension' classifications used in the study? (Unfortunately, I could not find 'beauty of dimension')

Is there a discussion of the limitations of the methodologies employed, and how might these limitations affect the overall conclusions?

What are the real-world implications of the findings, particularly in high-stakes areas such as healthcare or criminal justice?

You mentioned that GPT-3 worked well in terms of the rate. However, did you check the potential issues of bias, fairness, and overall value in the dataset itself?
soundness: 2
strengths: 1. Originality
Strength: The paper presents approaches by analyzing the moral judgments of large language models (LLMs) within a multilingual context. This exploration of how LLMs interact with diverse cultural perspectives on moral dilemmas is a fresh contribution to the field, providing insights into the alignment (or misalignment) between machine-generated responses and human ethical considerations.

2. Clarity
Strength: Despite some complexity, the paper effectively organizes its findings and presents them in a structured manner. Key results are illustrated through well-designed tables and figures, aiding in the communication of the main ideas. The systematic breakdown of moral dimensions allows readers to follow the analysis logically.
summary: The paper examines the moral judgments of large language models (LLMs) in a multilingual context, aiming to analyze how these models respond to ethical dilemmas. It specifically focuses on the moral dimensions involved in decision-making processes, utilizing a dataset that encompasses a variety of cultural perspectives. The authors investigate how LLMs interpret complex moral scenarios, employing methodologies such as statistical analyses and clustering techniques to evaluate the models' performance.

The research highlights specific examples, including classic moral dilemmas like the trolley problem, to illustrate the differences in responses generated by LLMs compared to human moral intuitions. The paper also discusses the implications of these findings for understanding the capabilities of LLMs in moral reasoning and emphasizes the importance of examining the potential biases and fairness within the datasets used for training and evaluation.

By comparing machine-generated responses to human judgments, the study aims to shed light on the alignment (or misalignment) between AI outputs and human ethical standards. The authors conclude by discussing the broader significance of their findings in relation to the ethical deployment of LLMs in various applications, stressing the need for ongoing research in this critical area of AI ethics.
weaknesses: 1. Originality
Weakness: While the paper attempts to explore moral judgments in a multilingual context, it does not significantly advance the discourse beyond existing literature on moral dilemmas. Many of the concepts discussed are already well-established in moral philosophy, and the paper may not provide enough innovative perspectives to stand out in a crowded field.

2. Quality
Weakness: The reliability and validity of the moral dimension classifications could be questioned. The paper may lack comprehensive validation of the measures used to assess moral judgments, which raises concerns about the robustness of the findings. Additionally, the potential biases inherent in the training data of the LLMs themselves are not sufficiently addressed, which could impact the conclusions drawn.

3. Significance
Weakness: The implications of the findings regarding the moral judgments of LLMs are not fully explored. While the paper presents data on how LLMs align with human preferences, it falls short of discussing the broader societal implications of these judgments, particularly in high-stakes contexts such as healthcare or criminal justice. This lack of contextualization may diminish the perceived significance of the research in real-world applications.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: Q1: Could noise factors, such as the performance of Google Translate in translating prompts or the effectiveness of LLMs in low-resource languages, impact the alignment of model preferences and potentially introduce discrepancies or inconsistencies in multilingual outputs?

Q2: How would you handle cases where human preferences conflict with principles of fairness and equality?
soundness: 3
strengths: The paper is well-structured and clearly written. It addresses an important new challenge in aligning LLMs' responses with human preferences in moral decision-making. The authors conducted detailed experiments and provided valuable insights. The MultiTP dataset, which spans numerous languages, could be highly useful for future studies on aligning model behavior with human ethics.
summary: The study investigates how well LLMs align with human moral preferences in multilingual settings, particularly through the lens of trolley problem scenarios. The authors introduce a new dataset called MultiTP, which is generated by filling templates with different variables and translating them into multiple languages using Google Translate. They conducted experiments and concluded that LLMs do not strongly align with human preferences but did not show significant alignment inequality between high-resource and low-resource languages.
weaknesses: One of the main concerns is the justification for aligning LLMs with demographic distributions and human preferences. This approach might introduce biases into the models. For example, in certain cultures where laws or social norms may place different values on individuals (e.g., men being valued more than women in some religions), aligning LLMs with such preferences could reinforce harmful biases. The examples provided in Figure 1 illustrate cultural biases related to age, which could extend to other sensitive attributes. To address these issues, the authors should clarify the practical use cases for aligning models in such contexts and explain how alignment can be distinguished from bias, as well as how fairness is maintained.

Another concern is the reliability of using Google Translate for generating the dataset in various languages. Although the authors reviewed a subset of translations for major languages, translation quality tends to decrease for less common languages, which could impact the consistency of the dataset and the validity of the findings. For example, in the Persian translation, Google Translate incorrectly rendered the phrase "it [the self-driving car] should save" by translating "save" as "صرفه جویی کردن" (meaning "to economize" which is a verb usually used for resources, like water) instead of the correct "نجات دادن' (meaning "to rescue/save life"). This confusion occurs despite Persian being a widely-spoken language with substantial digital presence. Such semantic errors in a relatively well-resourced language raise serious concerns about the translation quality for truly low-resource languages, where training data is far more limited.

Lastly, the study does not account for performance differences between LLMs across various languages. Models like GPT-4 and Llama 2 are expected to show significant performance variations in low-resource languages, potentially affecting the reliability of the results in experiments such as RQ1 and RQ4.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: - Could you clarify why a score of 0.6 in RQ1 is considered close alignment to human preference? While I understand that the scores can be used to rank LLM alignment, I am curious whether the absolute value holds inherent significance. In other words, what justifies considering a 0.6 score as an indicator of close alignment between LLMs and human preferences?
- Could you provide more details on the manual review process for translations, including the number of samples reviewed, the criteria used for evaluation, and any systematic issues or errors that were found during this process?
- What steps or analysis would be necessary to evaluate LLM performance within specific cultural subsets, and did your findings reveal any trends where certain models align more closely with specific **cultural norms**?
- Did you observe any significant differences in alignment or misalignment patterns between major and minor languages? How might these insights inform future strategies for training and fine-tuning language models?
soundness: 3
strengths: **Originality**: The paper introduces an innovative approach by adapting the Moral Machine framework to evaluate LLMs across languages, creating the unique MULTITP dataset for consistent cross-lingual moral dilemma evaluation, and developing novel metrics for measuring alignment.

**Quality**: The study demonstrates rigorous methodology through a comprehensive evaluation of 19 LLMs, systematic moral dimension variations, robust prompt paraphrasing tests, and careful statistical analysis with transparency in acknowledging limitations.

**Clarity**: The paper is well-organized with clear research questions, strong visual aids, detailed methodological documentation, and effective use of examples to communicate complex findings.

**Significance**: The work contributes significantly to understanding cross-cultural AI alignment, challenges assumptions about language inequality, and provides essential tools for future research, with implications for creating culturally aware and fair multilingual AI systems.
summary: The paper investigates how large language models (LLMs) align with human moral preferences across various languages and cultures. It introduces the MULTITP dataset, based on the Moral Machine experiment, which collected over 40 million human judgments globally. MULTITP includes 97,520 translated moral dilemma scenarios in 107 languages, allowing for a systematic evaluation of LLMs' moral reasoning across linguistic contexts.

The study assesses 19 LLMs across six moral dimensions: species, gender, fitness, social status, age, and the number of lives involved. The findings show that while most LLMs do not closely align with human moral judgments, they display similar levels of alignment across both major and minor languages, countering assumptions about biases related to language resources.

Key contributions include providing a comprehensive dataset for multilingual moral alignment, developing methods to analyze variations in moral dilemmas, and creating frameworks for assessing cross-cultural alignment. The paper underscores limitations in LLMs' ability to match human moral reasoning while emphasizing the need to integrate diverse cultural and linguistic perspectives into AI ethics research. This work serves as both a technical guide for measuring alignment and a reminder of the complexities involved in cross-cultural moral decision-making in AI.
weaknesses: **Dataset Transformation**: Although the dataset is large and comprehensive, it is adapted from the existing *Moral Machine* experiment, raising questions about its novelty. The paper would benefit from clarifying what unique modifications or contributions were made to enhance the dataset beyond its scale and multilingual coverage.

**Translation Quality**: Maintaining high translation quality across a vast number of questions, especially in low-resource languages, is challenging. The reliance on *googletrans* (the non-commercial open-source tool) may compromise accuracy. 

**Alignment Claims**: The study claims that LLMs generally do not align well with human preferences, but it aggregates data from all cultural contexts, potentially obscuring the nuances of cultural alignment. Expecting models to align with an average of global human preferences may be unrealistic or inappropriate, given the significant variability in cultural moral standards. Discussing how LLMs perform within specific cultural subsets might provide more nuanced insights.

**Practical Relevance for LLM Development**: While the study offers valuable theoretical insights, its direct application to real-world LLM development is not clear. Demonstrating how these findings could inform practical strategies for LLM fine-tuning and alignment would make the paper more relevant for AI practitioners.

**Use of Binary Moral Dilemmas**: The paper’s reliance on binary moral dilemmas, such as trolley problems, provides a structured evaluation but oversimplifies the complexities of real-world ethical decision-making. Real-life moral situations are rarely clear-cut, and binary choices may not capture the depth of human reasoning. Including more complex and context-sensitive scenarios could enhance the study’s applicability and realism.

**Interpretation of Results**: Although the paper includes multiple languages, it does not explore how cultural nuances influence LLM responses. Cultural context significantly shapes moral judgments, and not addressing these variations may miss key insights into LLM behaviour in diverse moral frameworks. Analyzing language-specific and culture-specific responses would enrich the findings and offer a deeper cross-cultural understanding.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 4
questions: * What factors might account for differences in behavior between models?
* Can you elaborate more on the post-processing procedure in L276? 
* Did you observe any effect of position bias in the option order in L233?
soundness: 3
strengths: * The paper presents a novel dataset designed to evaluate alignment in a multilingual setup. The dataset has a solid design with parametric variation and covers 109 languages, 6 moral dimensions, and 18 characters. The data is translated using a machine translation engine, which is suitable for this task since it is not tied to linguistic subtleties.

* The experimental protocol appears robust. LLMs are prompted with one of the problems, and preferences are extracted from their responses. A misalignment score is then computed to capture differences between LLM and human preferences.

* The results discussion is organized into five sections aligned with specific research questions, making it easy to follow and understand the key findings. These findings address overall alignment, LLM performance across six moral dimensions, language variation, performance in low-resource languages, and LLM robustness to prompt paraphrasing.
summary: The paper introduces a dataset of Multilingual Trolley Problems. The typical task frames a hypothetical situation in which a self-driving car's brakes are broken and it hits one of two specified characters.   The dataset covers 107 languages, six moral dimensions, and 18 characters. The paper prompts 19 LLMs to investigate the alignment between LLM choices in the trolley problem and human preferences collected in prior studies. The paper finds that LLMs vary in alignment with human preferences and exhibit different behavior when prompted in various languages.
weaknesses: * Although the paper offers a detailed analysis of LLM performance across several research questions, it lacks an exploration of potential reasons behind the observed phenomena. While such explanations might be speculative given the limited information on LLM training, a discussion on why models from the same family (e.g., Llama 3) exhibit differing behaviors could enhance the paper.

* The paper implicitly assumes that LLMs should align with country-level or language-level human preferences. However, these preferences may carry biases and could be unfair to marginalized groups. Conversely, the least aligned model, GPT-4o Mini, could be viewed as one that lacks language-specific biases and instead follows a standardized behavior pattern. Exploring this topic further in the paper could add valuable insights into the potential trade-offs between alignment and bias in multilingual settings.

Meta-review: This paper evaluates the moral alignment of LLMs with human preferences across different languages and cultures using trolley problem scenarios. A dataset called MultiTP containing 97,520 moral dilemma scenarios in 107 languages is introduced. The authors evaluate 19 LLMs across six moral dimensions (species, gender, fitness, status, age, and number of lives).

Strengths: The introduced dataset is novel and comprehensive, the evaluation involves multiple LLMs and languages. Overall it is an important contribution to understanding cross-cultural AI alignment.

Weaknesses: This paper would benefit from more discussions and analysis on multiple aspects raised by the reviewers, e.g., exploration of why different LLM families exhibit varying behaviors, and translation quality in very low-resource languages. But those are relatively minor issues and easy to solve in the final version.

The paper makes important contributions to understanding cross-cultural AI alignment through a novel dataset and comprehensive empirical evaluation. I believe it will provide valuable insights for both researchers and practitioners working on the alignment problem. I recommend to accept this paper. However, the potential ethical concerns, such as bias or fairness issues in different cultures in the dataset, as mentioned by multiple reviewers, suggest an additional ethical check of this paper, where the authors need to clearly explain and elaborate on those issues.

Decision: Accept (Spotlight)