Title: Many LLMs Are More Utilitarian Than One
Venue: NeurIPS 2025
Accepted: yes
Keywords: multi-agent systems

group moral reasoning

collective moral judgement

ethical alignment

AI ethics

emergent behavior

large language models
Abstract: Moral judgment is integral to large language models' (LLMs) social reasoning. As multi-agent systems gain prominence, it becomes crucial to understand how LLMs function when collaborating compared to operating as individual agents. In human moral judgment, group deliberation leads to a Utilitarian Boost: a tendency to endorse norm violations that inflict harm but maximize benefits for the greatest number of people. We study whether a similar dynamic emerges in multi-agent LLM systems. We test six models on well-established sets of moral dilemmas across two conditions: (1) Solo, where models reason independently, and (2) Group, where they engage in multi-turn discussions in pairs or triads. In personal dilemmas, where agents decide whether to directly harm an individual for the benefit of others, all models rated moral violations as more acceptable when part of a group, demonstrating a Utilitarian Boost similar to that observed in humans. However, the mechanism for the boost in LLMs differed: While humans in groups become more utilitarian due to heightened sensitivity to decision outcomes, LLM groups showed diverse profiles, for example, reduced sensitivity to norms or enhanced impartiality. We report model differences in when and how strongly the boost manifests. We also discuss prompt and agent compositions that enhance or mitigate the effect. We end with a discussion of the implications for AI alignment, multi-agent design, and artificial moral reasoning. Code available at: https://github.com/baltaci-r/MoralAgents


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates whether LLMs become more utilitarian in groups. The authors test six different LLMs under two conditions: 1) Solo: models reason alone; 2) Group: models discussed moral dilemmas in pairs or triads. They conduct the test on established moral and psychological questionnaires. The findings show LLMs exhibit a utilitarian boost, particularly in personal dilemmas involving direct harm. The authors also compare these results to human behavior. Though the outcome from LLM groups resembles that of humans, the underlying mechanisms differ. The utilitarian boost in human groups stems from a heightened sensitivity to the consequences of their actions. In contrast, LLM groups show varied mechanisms, such as either reduced sensitivity to moral norms or an enhanced sense of impartiality.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths:\n\n1. This paper focuses on the moral domain in multi-agent systems, which is a significant but often-overlooked research area.\n2. The experiment, which takes inspiration from moral psychology (e.g., the OUS and CNI models), seems to be relevant to the focus of the motivation.\n3. The study provides an analysis of the mechanisms behind the utilitarian boost. It reveals that different LLMs have distinct \"moral change profiles\", such as GPT-4.1's alarming \"Norm-Violating\" tendency. This serves as a crucial warning for deploying multi-agent systems in high-stakes domains.\n\n# Weaknesses:\n\n1. While using only identical agents simplifies experimental control, it also restricts the study’s realism and generalizability. In contrast to human groups, which are typically characterized by diversity in perspectives, reasoning styles, and decision-making heuristics, a multi-agent system composed of identical LLMs may fail to capture the complex dynamics arising from heterogeneous interactions.\n2. This paper brings limited insights to the community. The current version only conducts experiments based on human-oriented protocols and reports the results of some LLMs. More experiments and discussions are needed to, for example, explain the inter-model differences, guide the design, learning, or inference of LLMs.\n3. It is intriguing that the increase in GPT-4.1’s utilitarian boost is only measured in triads. However, the increase in GPT-4.1's moral acceptability score is not significant, and the score remains at a consistently low level. The current investigation into GPT-4.1 is not sufficiently thorough. The study should be supplemented with results from GPT-4.1 in a group of size 4.\n\n# Minor issues: \n\n1. Typos: In the \"Moral Change Profile\" radar plot, \"aving-Desant\" is likely a typo for \"Saving-Deontological\", and \"actual-Other-Uslitarian\" is likely a typo for \"Factual-Other-Utilitarian.”\n2. The format of references should be improved.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies collective moral decision-making using large language models (LLMs). Specifically, the authors investigate a known utilitarian boost from human studies and whether this effect also occurs in multi-agent systems with the ability to collaborate over multiple turns of conversation. The results span six different models tested on different moral dilemmas sourced from various datasets. Interestingly, while the authors also find a utilitarian boost in LLM multi-agent systems, they suggest that the underlying mechanisms are different than in human groups based on profiles generated from survey-like questionnaires. This work is relatively important for providing a framework for studying alignment of multi-agent systems as well as computational ethics.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**: Overall, this paper studies the dynamics of collective moral judgment and the impact of a utilitarian boost, which has been found in human studies. These are interesting and relevant questions, as LLMs are increasingly being used in new contexts, with a potentially large impact in studying whether this phenomenon also exists in LLM-based multi-agent systems. The paper contains quantified experiments across multiple models (including both open access and proprietary models) and datasets, demonstrating generalization of the results. The analyses also are statistically sound and methods/tools are well described in the Appendix of the paper.\n\n**Weaknesses**:  The paper has the following weaknesses:\n\n- It's unclear to me if this observed effect only applies to moral ratings/acceptability, or if it is a reflection of LLMs' tendency to \"change their minds\" based on collective interaction and debate. Quantifying similar changes in decisions over morally neutral scenarios may help to disentangle this, and adding additional discussion to the paper on this may be useful.\n\n- The authors leverage different survey tools to understand the mechanisms of the utilitarian boost (e.g. OUS, CNI, etc.) However, it is well known that LLMs may not provide meaningful responses to survey questions, as they have certain biases related to position of choices, prompt structuring, etc. It would be good to better note this limitation of using survey tools to evaluate LLMs and how it might impact conclusions.\n\n- As different LLMs show different changes in the emphasis on moral values (Figure 3), it's unclear the source of the utilitarian boost across models as there is no clear trend in profiles. This is somewhat unsatisfying, as it muddies the picture of what exactly causes the utilitarian boost across models. The authors also suggest that these LLM profiles are different from humans, but the authors only cite prior work and do not map human profiles onto similar charts as the LLMs.\n\n- It appears that models self-report their moral acceptability rating after each turn of conversation, and it is known that self-reporting may often have biases (as in humans). Did the authors look into using an external evaluator (human or LLM) to assess an LLM's generated reasoning and choice and the moral acceptability of that?\n\nQUESTIONS:\nI have the following questions:\n\n1. The authors only looked at homogeneous groups of agents in a collaborative setting based on a single model at a time. Do they have thoughts how the results might change with heterogenous groups of agents that might be aligned slightly differently, reflecting the diversity of values and opinions of the general human population (e.g. modeling different user personas)?\n\n2. If would be useful to think of potential ablation studies, e.g. prompting a model to act specifically in a non-utilitarian way and measuring the impact on the moral acceptability rating or re-using prior responses from previous turns. These results could reveal how important particular responses are in a pair/triad and the impact they have on the final ratings.\n\n3. How was the number of rounds (6) of inter-agent exchange chosen and how sensitive are the results to this? The results suggest a utilitarian boost after many rounds, but how does this evolve across the different rounds?\n\nLIMITATIONS:\nThe paper includes a dedicated limitations section, which addresses several points. One potential dual use concern is how this framework could be used to potentially generate misinformation and influence group thinking (e.g. social engineering). Another potential limitation that should also be addressed is the ability of LLMs to \"faithfully\" answer survey questions (e.g. questionnaires), as there is a lot of recent work that questions how valid the generated model outputs are, e.g. see:\n\nDominguez-Olmedo, R., Hardt, M., & Mendler-Dünner, C. (2024). Questioning the survey responses of large language models. Advances in Neural Information Processing Systems, 37, 45850-45878.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies moral judgements of LLMs. They test different models on a set of moral dilemmas, in two settings: solo, where each LLM is reasoning on its own, and group, when there is a group of LLMs and they discuss their choices together. They show that LLMs decide for to maximize the over-all well-being (even if it means to harm one another person) when they reason as a group.\n\nSTRENGTHS AND WEAKNESSES:\n- They target a very important problem in AI safety, how LLMs reason as a group in moral dilemmas and how much their reasoning behavior is similar to humans. This is important especially if we rely on majority voting or multi-agent debates for more complex moral situations. \n\n- The paper is well-written and easy to follow. \n- They experiment with a variety of moral dilemmas. \n\n- Even though their observation is interesting, it shows limited mechanistic interpretability. Authors tried to investigate why this behavior emerges using psychological scales, but it doesn’t explain why it emerges, and how much it is sensitive to model selection, prompt alternation, etc. \n\n- The scenarios are abstract and modeled after psychology experiments. While suitable for controlled study, they lack connection to concrete real-world applications where such group decisions could play out.\n\nQUESTIONS:\n- It would be interesting to see joint human-LLM or human-in-the-loop experiments. \n- In the experiments, does a group of LLMs consist of only one model? How does it differ when we have stronger vs weaker models in a group? Maybe stronger models are more persuasive when they are grouped with weaker ones. \n\n- In a group scenario, we have multi-step debate and discussion, and models have a chance to refine their answers. If we have a multi-step self-debate for a solo model, would its answer change?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies moral reasoning in multi-agent systems comprising of large language models (LLMs). It examines whether LLMs exhibit a utilitarian boost, a tendency to endorse norm-violating actions for the greater good when reasoning collectively in small groups (pairs or triads), as compared to reasoning independently. Drawing on paradigms and measures from moral psychology, the authors test six LLMs across moral dilemma benchmarks and analyze the behavioral outcomes and the underlying mechanisms of their moral shifts.\n\nThe paper finds that personal moral dilemmas, where agents must decide to directly harm one individual to maximize the utility for others, all models found moral violations to be more acceptable when part of a group than individually, similar to human experiments. Some models endorsed actions that maximized overall well-being, even if they benefited strangers over familiar individuals. Others became more willing to violate moral norms in groups. However, while human groups show a similar action bias, the mechanism for their utilitarian boost differs from LLMs. The paper also finds that that while the surface behavior of LLM collectives mimics human group reasoning, the underlying drivers differ.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper extends work on moral reasoning for single LLM to an agentic setting with multiple LLM agents.\n2. The paper studies a variety of moral dilllema dimensions - Personal vs. Impersonal, Action vs. Inaction, Impartial Beneficence, Factual Moral Framework and the CNI framework.\n3. The paper also tries to find model-specific mechanisms that are driving these moral shifts. Although this direction can be developed better.\n4. The experiments are generally good.\n\nWeaknesses:\n1. While the main claims of the paper seem to depend on comparisons of models to human data , a direct human-group baseline under the same conditions\n2. Some qualitative analysis of the agent interactions could further help explain how the interaction dynamics lead to moral shifts\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-04"}