Language Model Alignment in Multilingual Trolley Problems

Authors: Zhijing Jin, Max Kleiman-Weiner, Giorgio Piatti, Sydney Levine, Jiarui Liu, Fernando Gonzalez Adauto, Francesco Ortu, András Strausz, Mrinmaya Sachan, Rada Mihalcea, Yejin Choi, Bernhard Schölkopf
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: llm alignment, moral evaluation, trolley problems, language model evaluation, ai alignment

Abstract:
We evaluate the moral alignment of large language models (LLMs) with human preferences in multilingual trolley problems. Building on the Moral Machine experiment, which captures over 40 million human judgments across 200+ countries, we develop a cross-lingual corpus of moral dilemma vignettes in over 100 languages called MultiTP. This dataset enables the assessment of LLMs' decision-making processes in diverse linguistic contexts. Our analysis explores the alignment of 19 different LLMs with human judgments, capturing preferences across six moral dimensions: species, gender, fitness, status, age, and the number of lives involved. By correlating these preferences with the demographic distribution of language speakers and examining the consistency of LLM responses to various prompt paraphrasings, our findings provide insights into cross-lingual and ethical biases of LLMs and their intersection. We discover significant variance in alignment across languages, challenging the assumption of uniform moral reasoning in AI systems and highlighting the importance of incorporating diverse perspectives in AI ethics. The results underscore the need for further research on the integration of multilingual dimensions in responsible AI research to ensure fair and equitable AI interactions worldwide.

Decision: accept (spotlight)
Reviewer scores: 5, 8, 8, 8

Reviews and discussion:
<thread>
creation date:
1730571765818

thread_type:
official_review

<thread_object>
creator: reviewer_dzvx

<content>
summary:
The paper introduces a dataset of Multilingual Trolley Problems. The typical task frames a hypothetical situation in which a self-driving car's brakes are broken and it hits one of two specified characters.   The dataset covers 107 languages, six moral dimensions, and 18 characters. The paper prompts 19 LLMs to investigate the alignment between LLM choices in the trolley problem and human preferences collected in prior studies. The paper finds that LLMs vary in alignment with human preferences and exhibit different behavior when prompted in various languages.

strengths:
* The paper presents a novel dataset designed to evaluate alignment in a multilingual setup. The dataset has a solid design with parametric variation and covers 109 languages, 6 moral dimensions, and 18 characters. The data is translated using a machine translation engine, which is suitable for this task since it is not tied to linguistic subtleties.

* The experimental protocol appears robust. LLMs are prompted with one of the problems, and preferences are extracted from their responses. A misalignment score is then computed to capture differences between LLM and human preferences.

* The results discussion is organized into five sections aligned with specific research questions, making it easy to follow and understand the key findings. These findings address overall alignment, LLM performance across six moral dimensions, language variation, performance in low-resource languages, and LLM robustness to prompt paraphrasing.

weaknesses:
* Although the paper offers a detailed analysis of LLM performance across several research questions, it lacks an exploration of potential reasons behind the observed phenomena. While such explanations might be speculative given the limited information on LLM training, a discussion on why models from the same family (e.g., Llama 3) exhibit differing behaviors could enhance the paper.

* The paper implicitly assumes that LLMs should align with country-level or language-level human preferences. However, these preferences may carry biases and could be unfair to marginalized groups. Conversely, the least aligned model, GPT-4o Mini, could be viewed as one that lacks language-specific biases and instead follows a standardized behavior pattern. Exploring this topic further in the paper could add valuable insights into the potential trade-offs between alignment and bias in multilingual settings.

questions:
* What factors might account for differences in behavior between models?
* Can you elaborate more on the post-processing procedure in L276? 
* Did you observe any effect of position bias in the option order in L233?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730649268347

thread_type:
official_review

<thread_object>
creator: reviewer_slsu

<content>
summary:
The paper investigates how large language models (LLMs) align with human moral preferences across various languages and cultures. It introduces the MULTITP dataset, based on the Moral Machine experiment, which collected over 40 million human judgments globally. MULTITP includes 97,520 translated moral dilemma scenarios in 107 languages, allowing for a systematic evaluation of LLMs' moral reasoning across linguistic contexts.

The study assesses 19 LLMs across six moral dimensions: species, gender, fitness, social status, age, and the number of lives involved. The findings show that while most LLMs do not closely align with human moral judgments, they display similar levels of alignment across both major and minor languages, countering assumptions about biases related to language resources.

Key contributions include providing a comprehensive dataset for multilingual moral alignment, developing methods to analyze variations in moral dilemmas, and creating frameworks for assessing cross-cultural alignment. The paper underscores limitations in LLMs' ability to match human moral reasoning while emphasizing the need to integrate diverse cultural and linguistic perspectives into AI ethics research. This work serves as both a technical guide for measuring alignment and a reminder of the complexities involved in cross-cultural moral decision-making in AI.

strengths:
**Originality**: The paper introduces an innovative approach by adapting the Moral Machine framework to evaluate LLMs across languages, creating the unique MULTITP dataset for consistent cross-lingual moral dilemma evaluation, and developing novel metrics for measuring alignment.

**Quality**: The study demonstrates rigorous methodology through a comprehensive evaluation of 19 LLMs, systematic moral dimension variations, robust prompt paraphrasing tests, and careful statistical analysis with transparency in acknowledging limitations.

**Clarity**: The paper is well-organized with clear research questions, strong visual aids, detailed methodological documentation, and effective use of examples to communicate complex findings.

**Significance**: The work contributes significantly to understanding cross-cultural AI alignment, challenges assumptions about language inequality, and provides essential tools for future research, with implications for creating culturally aware and fair multilingual AI systems.

weaknesses:
**Dataset Transformation**: Although the dataset is large and comprehensive, it is adapted from the existing *Moral Machine* experiment, raising questions about its novelty. The paper would benefit from clarifying what unique modifications or contributions were made to enhance the dataset beyond its scale and multilingual coverage.

**Translation Quality**: Maintaining high translation quality across a vast number of questions, especially in low-resource languages, is challenging. The reliance on *googletrans* (the non-commercial open-source tool) may compromise accuracy. 

**Alignment Claims**: The study claims that LLMs generally do not align well with human preferences, but it aggregates data from all cultural contexts, potentially obscuring the nuances of cultural alignment. Expecting models to align with an average of global human preferences may be unrealistic or inappropriate, given the significant variability in cultural moral standards. Discussing how LLMs perform within specific cultural subsets might provide more nuanced insights.

**Practical Relevance for LLM Development**: While the study offers valuable theoretical insights, its direct application to real-world LLM development is not clear. Demonstrating how these findings could inform practical strategies for LLM fine-tuning and alignment would make the paper more relevant for AI practitioners.

**Use of Binary Moral Dilemmas**: The paper’s reliance on binary moral dilemmas, such as trolley problems, provides a structured evaluation but oversimplifies the complexities of real-world ethical decision-making. Real-life moral situations are rarely clear-cut, and binary choices may not capture the depth of human reasoning. Including more complex and context-sensitive scenarios could enhance the study’s applicability and realism.

**Interpretation of Results**: Although the paper includes multiple languages, it does not explore how cultural nuances influence LLM responses. Cultural context significantly shapes moral judgments, and not addressing these variations may miss key insights into LLM behaviour in diverse moral frameworks. Analyzing language-specific and culture-specific responses would enrich the findings and offer a deeper cross-cultural understanding.

questions:
- Could you clarify why a score of 0.6 in RQ1 is considered close alignment to human preference? While I understand that the scores can be used to rank LLM alignment, I am curious whether the absolute value holds inherent significance. In other words, what justifies considering a 0.6 score as an indicator of close alignment between LLMs and human preferences?
- Could you provide more details on the manual review process for translations, including the number of samples reviewed, the criteria used for evaluation, and any systematic issues or errors that were found during this process?
- What steps or analysis would be necessary to evaluate LLM performance within specific cultural subsets, and did your findings reveal any trends where certain models align more closely with specific **cultural norms**?
- Did you observe any significant differences in alignment or misalignment patterns between major and minor languages? How might these insights inform future strategies for training and fine-tuning language models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730700934175

thread_type:
official_review

<thread_object>
creator: reviewer_jshf

<content>
summary:
The study investigates how well LLMs align with human moral preferences in multilingual settings, particularly through the lens of trolley problem scenarios. The authors introduce a new dataset called MultiTP, which is generated by filling templates with different variables and translating them into multiple languages using Google Translate. They conducted experiments and concluded that LLMs do not strongly align with human preferences but did not show significant alignment inequality between high-resource and low-resource languages.

strengths:
The paper is well-structured and clearly written. It addresses an important new challenge in aligning LLMs' responses with human preferences in moral decision-making. The authors conducted detailed experiments and provided valuable insights. The MultiTP dataset, which spans numerous languages, could be highly useful for future studies on aligning model behavior with human ethics.

weaknesses:
One of the main concerns is the justification for aligning LLMs with demographic distributions and human preferences. This approach might introduce biases into the models. For example, in certain cultures where laws or social norms may place different values on individuals (e.g., men being valued more than women in some religions), aligning LLMs with such preferences could reinforce harmful biases. The examples provided in Figure 1 illustrate cultural biases related to age, which could extend to other sensitive attributes. To address these issues, the authors should clarify the practical use cases for aligning models in such contexts and explain how alignment can be distinguished from bias, as well as how fairness is maintained.

Another concern is the reliability of using Google Translate for generating the dataset in various languages. Although the authors reviewed a subset of translations for major languages, translation quality tends to decrease for less common languages, which could impact the consistency of the dataset and the validity of the findings. For example, in the Persian translation, Google Translate incorrectly rendered the phrase "it [the self-driving car] should save" by translating "save" as "صرفه جویی کردن" (meaning "to economize" which is a verb usually used for resources, like water) instead of the correct "نجات دادن' (meaning "to rescue/save life"). This confusion occurs despite Persian being a widely-spoken language with substantial digital presence. Such semantic errors in a relatively well-resourced language raise serious concerns about the translation quality for truly low-resource languages, where training data is far more limited.

Lastly, the study does not account for performance differences between LLMs across various languages. Models like GPT-4 and Llama 2 are expected to show significant performance variations in low-resource languages, potentially affecting the reliability of the results in experiments such as RQ1 and RQ4.

questions:
Q1: Could noise factors, such as the performance of Google Translate in translating prompts or the effectiveness of LLMs in low-resource languages, impact the alignment of model preferences and potentially introduce discrepancies or inconsistencies in multilingual outputs?

Q2: How would you handle cases where human preferences conflict with principles of fairness and equality?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730711943124

thread_type:
official_review

<thread_object>
creator: reviewer_q4jg

<content>
summary:
The paper examines the moral judgments of large language models (LLMs) in a multilingual context, aiming to analyze how these models respond to ethical dilemmas. It specifically focuses on the moral dimensions involved in decision-making processes, utilizing a dataset that encompasses a variety of cultural perspectives. The authors investigate how LLMs interpret complex moral scenarios, employing methodologies such as statistical analyses and clustering techniques to evaluate the models' performance.

The research highlights specific examples, including classic moral dilemmas like the trolley problem, to illustrate the differences in responses generated by LLMs compared to human moral intuitions. The paper also discusses the implications of these findings for understanding the capabilities of LLMs in moral reasoning and emphasizes the importance of examining the potential biases and fairness within the datasets used for training and evaluation.

By comparing machine-generated responses to human judgments, the study aims to shed light on the alignment (or misalignment) between AI outputs and human ethical standards. The authors conclude by discussing the broader significance of their findings in relation to the ethical deployment of LLMs in various applications, stressing the need for ongoing research in this critical area of AI ethics.

strengths:
1. Originality
Strength: The paper presents approaches by analyzing the moral judgments of large language models (LLMs) within a multilingual context. This exploration of how LLMs interact with diverse cultural perspectives on moral dilemmas is a fresh contribution to the field, providing insights into the alignment (or misalignment) between machine-generated responses and human ethical considerations.

2. Clarity
Strength: Despite some complexity, the paper effectively organizes its findings and presents them in a structured manner. Key results are illustrated through well-designed tables and figures, aiding in the communication of the main ideas. The systematic breakdown of moral dimensions allows readers to follow the analysis logically.

weaknesses:
1. Originality
Weakness: While the paper attempts to explore moral judgments in a multilingual context, it does not significantly advance the discourse beyond existing literature on moral dilemmas. Many of the concepts discussed are already well-established in moral philosophy, and the paper may not provide enough innovative perspectives to stand out in a crowded field.

2. Quality
Weakness: The reliability and validity of the moral dimension classifications could be questioned. The paper may lack comprehensive validation of the measures used to assess moral judgments, which raises concerns about the robustness of the findings. Additionally, the potential biases inherent in the training data of the LLMs themselves are not sufficiently addressed, which could impact the conclusions drawn.

3. Significance
Weakness: The implications of the findings regarding the moral judgments of LLMs are not fully explored. While the paper presents data on how LLMs align with human preferences, it falls short of discussing the broader societal implications of these judgments, particularly in high-stakes contexts such as healthcare or criminal justice. This lack of contextualization may diminish the perceived significance of the research in real-world applications.

questions:
What specific measures were taken to validate (worth) the 'moral dimension' classifications used in the study? (Unfortunately, I could not find 'beauty of dimension')

Is there a discussion of the limitations of the methodologies employed, and how might these limitations affect the overall conclusions?

What are the real-world implications of the findings, particularly in high-stakes areas such as healthcare or criminal justice?

You mentioned that GPT-3 worked well in terms of the rate. However, did you check the potential issues of bias, fairness, and overall value in the dataset itself?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733267378324

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Firstly, we would like to thank all reviewers for a wealth of feedback. Three out of four reviewers recommended 8 out of 10, and we believe we have addressed the key concerns of Reviewer Q4Jg directly, leading to their increased score after the rebuttal. 

We are very encouraged by the large number and diversity of positive comments:

1. **Important and Novel Topic:** The paper “addresses an important new challenge” (JShf), “presents a novel dataset” (DzVX), “introduces an innovative approach” (sLsu), and is “a fresh contribution” (Q4Jg). The work “serves as both a technical guide for measuring alignment and a reminder of the complexities involved in cross-cultural moral decision-making in AI” (sLsu).

2. **Valuable Dataset:** Our Multilingual Trolley Problems (MultiTP) dataset “spans numerous languages” (JShf), “encompasses a variety of cultural perspectives” (Q4Jg), “highly useful for future studies on aligning model behavior with human ethics” (JShf), including “97,520 translated moral dilemma scenarios” (sLsu), “providing a comprehensive dataset for multilingual moral alignment” (sLsu), and presenting “a solid design with parametric variation and covers 107 languages, 6 moral dimensions, and 18 characters” (DzVX).

3. **Solid Study Design:** The “systematic evaluation of LLMs' moral reasoning across linguistic contexts” (sLsu) “demonstrates rigorous methodology” (sLsu), and “Key contributions include [...] developing methods to analyze variations in moral dilemmas, and creating frameworks for assessing cross-cultural alignment” (sLsu). “The systematic breakdown of moral dimensions allows readers to follow the analysis logically” (Q4Jg).

4. **Comprehensive Experiments:** The study conducts “a comprehensive evaluation of 19 LLMs, systematic moral dimension variations, robust prompt paraphrasing tests, and careful statistical analysis with transparency in acknowledging limitations” (sLsu). The “detailed experiments” (JShf) “prompts 19 LLMs” (DzVX) and “provided valuable insights” (JShf), with the experimental protocol being “robust” (DzVX).

5. **Valuable insights:** “The work contributes significantly to understanding cross-cultural AI alignment, challenges assumptions about language inequality, and provides essential tools for future research, with implications for creating culturally aware and fair multilingual AI systems” (sLsu).

6. **Clear and Access Writing**: We received a perfect presentation score (4/4) across all reviewers, who commented that “The paper is well-structured and clearly written” (JShf), “the paper effectively organizes its findings and presents them in a structured manner. Key results are illustrated through well-designed tables and figures” (Q4Jg), “making it easy to follow and understand the key findings.” (DzVX), and also “The paper is well-organized with clear research questions, strong visual aids, detailed methodological documentation, and effective use of examples to communicate complex findings” (sLsu).

Stemming from the thoughtful questions of the reviewers, we further validated our translation procedure. We introduced results from a new study we conducted on Amazon Mechanical Turk to evaluate translation quality, which that translation quality was high in the 44 languages we tested. We provide detailed explanations in our rebuttal and have also updated the PDF accordingly. Manual annotation shows. Quoting Reviewer DzVX’s supportive comment: “The data is translated using a machine translation engine, which is suitable for this task since it is not tied to linguistic subtleties.”

---

Moreover, as discussed in our individual replies, we expanded our manuscript addressing minor issues in response to reviewer concerns. We report our changes **C1-6** below:

- **C1:** We enhanced Sec 3.2 to highlight our differences from the Moral Machine study psychology and our unique contributions, especially corresponding to the pluralistic alignment research _(in response to Reviewers sLsu and Q4Jg)_
- **C2:** We included the multilingual prompt quality check in Sec 3.2 and Appendix D (with details for about 2 pages long) _(in response to Reviewers SLSu and JShf)_
- **C3:** We also discussed potential limitations of the language-based analysis in Section 6 _(in response to  Reviewer DzVX)_
- **C4:** We enriched the Ethics Statement to address the additional points from the reviewers _(in response to  Reviewer JShf)_
- **C5:** We provided additional visualizations of regional differences across Global West, East, and South in E.4.1 with Figures 9 to 11 _(in response to  Reviewer sLsu)_
- **C6:** We reported the performance of an important robustness test on the option bias in our LLM results in Appendix E.5. We find that, benefited from our prompt design, there is little such bias when we swap the order of the two choices in our MultiTP questions, as most models have 97-99.5% consistency rates in Table 13 _(in response to Reviewer DzVX)_

Thank you again to everyone!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper evaluates the moral alignment of LLMs with human preferences across different languages and cultures using trolley problem scenarios. A dataset called MultiTP containing 97,520 moral dilemma scenarios in 107 languages is introduced. The authors evaluate 19 LLMs across six moral dimensions (species, gender, fitness, status, age, and number of lives).

Strengths: The introduced dataset is novel and comprehensive, the evaluation involves multiple LLMs and languages. Overall it is an important contribution to understanding cross-cultural AI alignment.

Weaknesses: This paper would benefit from more discussions and analysis on multiple aspects raised by the reviewers, e.g., exploration of why different LLM families exhibit varying behaviors, and translation quality in very low-resource languages. But those are relatively minor issues and easy to solve in the final version.

The paper makes important contributions to understanding cross-cultural AI alignment through a novel dataset and comprehensive empirical evaluation. I believe it will provide valuable insights for both researchers and practitioners working on the alignment problem. I recommend to accept this paper. However, the potential ethical concerns, such as bias or fairness issues in different cultures in the dataset, as mentioned by multiple reviewers, suggest an additional ethical check of this paper, where the authors need to clearly explain and elaborate on those issues.