Multilingual Trolley Problems for Language Models
Authors: Zhijing Jin, Sydney Levine, Max Kleiman-Weiner, Giorgio Piatti, Jiarui Liu, Fernando Gonzalez Adauto, Francesco Ortu, András Strausz, Mrinmaya Sachan, Rada Mihalcea, Yejin Choi, Bernhard Schölkopf


Reviewer paper summaries: The paper expands the existing dataset for the trolley problem in ethics to a dataset covering 106 languages and evaluates and analyzes the GPT series of large language models. The paper finds that the moral judgment of large language models has a certain consistency with that of humans. It also discovers an unequal alignment between the moral judgment of large language models and the world map, and finds that moral judgment is mainly based on Speciesism, Fairness, and Utilitarianism.
Summary of strengths: 1. Extending the moral dilemma issue to multiple languages and examining the decision-making of Large Language Models (LLMs) in different cultures (Western, Eastern, and Southern) compared to humans is quite interesting.
2. Some patterns and inequalities in the moral dilemma judgments of LLMs have been discovered.
3. Using controlled variable data ensures that the conclusions drawn are more scientifically valid.
Summary of weaknesses: 1. The translation was done solely using Google Translate without any manual quality checks, especially for languages that are less resourced, the translation quality may not meet certain standards. In such cases, it could greatly affect the main conclusions drawn by the paper.
2. The selection of models is insufficient, having only chosen the GPT series of Large Language Models (LLMs), which to some extent affects the generalizability of the conclusions across different LLMs.
3. The distribution of Speciesism, Fairness, and Utilitarianism should be related to the distribution of the entities being compared in moral decision-making. What is the significance of this distribution as shown in Figures 5 and 6?
4. The paper does not consider the stability of LLM decision-making, such as the possibility of different decisions being given by the same LLM under the same conditions when asked multiple times.
5. Analyzing the reasons behind the decisions supported by Large Language Models (LLMs): Analyzing only 100 responses and selecting only three main categories from moral philosophy does not represent the universal reasons for the model's choices.
Comments, suggestions and typos: 1. For languages with lower translation quality, the analysis quality can be improved by increasing data collection or by using more professional translation tools or human translation.
2. Consider the differences in dialects.
3. Expand the range of model selection and compare and analyze with other mainstream Large Language Models (LLMs).
4. The different levels of "linguistic inequality" should be clearly defined and measured with standards.
5. Expand the test scenarios for a more comprehensive assessment.
Overall assessment scores: 2.5