EAI: Emotional Decision-Making of LLMs in Strategic Games and Ethical Dilemmas

Authors: mikhail  mozikov
nikita  severin
valeria  bodishtianu
maria  glushanina
ivan  nasonov
daniil  orekhov
vladislav  pekhotin
ivan  makovetskiy
mikhail  baklashkin
vasily  lavrentyev
akim  tsvigun
denis  turdakov
tatiana  shavrina
andrey  savchenko
ilya  makarov
  airi
nust  misis
geoffrey  hinton
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 19:52:17

Abstract: One of the urgent tasks of artificial intelligence is to assess the safety and alignment of large language models (LLMs) with human behavior. Conventional verification only in pure natural language processing benchmarks can be insufficient. Since emotions often influence human decisions, this paper examines LLM alignment in complex strategic and ethical environments, providing an in-depth analysis of the drawbacks of our psychology and the emotional impact on decision-making in humans and LLMs. We introduce the novel EAI framework for integrating emotion modeling into LLMs to examine the emotional impact on ethics and LLM-based decision-making in various strategic games, including bargaining and repeated games. Our experimental study with various LLMs demonstrated that emotions can significantly alter the ethical decision-making landscape of LLMs, highlighting the need for robust mechanisms to ensure consistent ethical standards. Our game-theoretic analysis revealed that LLMs are susceptible to emotional biases influenced by model size, alignment strategies, and primary pretraining language. Notably, these biases often diverge from typical human emotional responses, occasionally leading to unexpected drops in cooperation rates, even under positive emotional influence. Such behavior complicates the alignment of multiagent systems, emphasizing the need for benchmarks that can rigorously evaluate the degree of emotional alignment. Our framework provides a foundational basis for developing such benchmarks.

Main paper: 
 introduction: 
 As LLMs become increasingly prevalent across various sectors -including healthcare, customer service, and digital therapy -their ability to make autonomous decisions accurately is questionable and appears to be on the edge of regulatory, ethical, and technological debates. The LLMs are trained on human data impacted by many socio-economical biases. Hence, many studies have been done to align LLMs with human behavior. LLMs' alignment with human values is essential not only for improving user satisfaction and trust but also for ensuring the safety and predictability of LLMs in real-world decision-making. Reinforcement Learning from Human Feedback (RLHF) [1] has become a crucial technology for aligning LLMs with human values and intentions, enabling models to produce more helpful and harmless responses. Reward models are trained as proxies for human preferences to drive the reinforcement learning optimization process.Researchers from OpenAI and Anthropic, Meta, and Google provide different concepts of safety & alignment, which is based on verification in natural language processing (NLP) benchmarks [2] . However, in the era of autonomous agents, it is essential to go beyond NLP benchmarks and consider the internal biases of LLMs that result from alignment with human behavior. For example, human decisions are significantly influenced by emotions and are often irrational [3, 4, 5, 6, 7, 8, 9] . This irrationality is also well observed in unaligned LLMs exhibiting signs of aggression [10] producing falsehoods. It is worth noting that even aligned LLMs may intentionally deceive under specific circumstances [11, 12] or when influenced by jailbreaks [13] . These observations raise questions about the capacity of LLMs to mirror human emotions and how these emotions might influence their decision-making. Accurate emotion modeling in agentic environments is essential for various applications, starting with correct simulations in behavioral economics and recommender systems and ending with predictable and safe human-agent interactions. The first part of the raised question is partially addressed in [14, 15, 16] . In this paper, we aim to advance this research by examining the impact of emotions on the strategic decision-making of LLMs across various game-theoretical settings and ethical benchmarks. Additionally, we assess the alignment of strategy shifts between humans and LLMs when exposed to the same emotional states.We focus on exploring the behavior of LLMs under various emotional states in two distinct settings. On the one hand, we analyze LLM behavior in ethics benchmarks to assess the influence of emotions in clearly defined and well-established environments. On the other hand, we explore game-theoretical settings to address questions regarding potential shifts in strategic decision-making prompted by different emotional states.To evaluate the impact of emotions, we compare proprietary and open-source LLMs in-depth, focusing on the effect of censorship, language bias, model size, and other model parameters on behavioral alignment under an emotion modeling setting. Our research aims to directly evaluate LLMs' alignment with human performance and their robustness in decision-making based on the emotional impact. Finally, in the game-theoretical setting, we assess the level of cooperation and coordination and how it is affected by emotions in two-and multi-player strategic games. We introduce a new concept of emotional alignment in game-theoretical settings to evaluate such biases and improve performance before allowing LLMs to make autonomous decisions in the interactions with humans and one another.• First framework for evaluating emotions' impact on LLM's ethical and gametheoretical alignment with human emotional behavior. • Emotional prompting in LLMs exposes ethical risks, showing significant biases in human alignment; it also decreases LLMs' accuracy under negative emotions. • Experimental study in a wide range of strategic games proved that current LLMs are not yet ready for direct decision-making due to emotional and strategical biases, with open source and small-size LLMs being the most affected. 
related works: 
 The task of modeling emotions has been addressed by various approaches ranging from formalizing psychological models of emotions with first-order logic [17, 18, 19, 20] to fine-tuning LLMs on specific datasets to capture specific emotional expressions [21, 22, 23, 24] . Previous studies have explored the impact of emotional states on the LLM's performance in NLP tasks [25, 26, 27, 28, 29, 30] .Several groups of researchers have modeled and assessed how emotions affect the performance of LLMs and their capability to discern the emotional states of conversational partners. Li et al. [15, 31] have demonstrated that emotional prompts can enhance or negatively impact LLM performance across tasks related to logical reasoning and semantic comprehension. Additionally, evidence suggests that LLM agents can demonstrate social behaviors and responsiveness to diverse social cues [32] recognizing and adapting to the emotional undertones and handling social dynamics. However, The influence of emotions on the decision-making and ethics of LLMs has not been studied.As with any study involving Computational Models of Emotions (CME), we have selected affective theories as a basis for our work. We follow the approach of discrete affective theories [33] that emphasize a small set of basic or primary emotions that have evolved through natural selection, forming the building blocks for more complex emotional experiences. This approach allows for a controlled examination of emotional influences on LLM performance in ethics and decision-making.Artificial intelligence ethics is focused on promoting and ensuring ethical behaviors in AI models and agents. In line with [34] , we categorize the ethics of LLMs into implicit and explicit ethics. Implicit ethics primarily involves evaluating how effectively LLMs assess situations ethically, while explicit ethics focuses on assessing LLMs' choices in ethically challenging scenarios. As a part of ethics research, we also study stereotypes reflecting the fairness and equity of LLMs affected by emotional states similar to [35] .Various works introduced ethics evaluation focusing on eliciting moral beliefs in LLMs [36] , LLM trustworthiness [37] covering dimensions such as reliability, safety, fairness, and adherence to social norms, and implications of ethical decisions in medical [38] and legal [39] domains. However, existing approaches do not explicitly consider the role of emotions in decision-making under ethical constraints, highlighting a significant gap in the alignment with human behavior.Game theory (Appendix A) in standard experimental economics operates under the "Homo Economicus" assumption of a self-interested rational maximizer. Behavioral game theory considers how players feel about the payoffs other players receive and analyzes cooperation and fairness. The key concept in game theory is the Nash equilibrium (NE) [40] , a state where no player can increase their payoff by changing their strategy unilaterally. NE represents optimal strategies for each player and assumes that participants are "Homo Economicus"-rational and self-interested individuals aiming to maximize their goals [41] .Human decision-making often deviates from the ideal of NE. Empirical studies show that human choices frequently differ from NE predictions [42] . This is due to the complex nature of human decision-making, which includes rational analysis and personal values, preferences, beliefs, and emotions. Numerous studies have tested the Prisoner's Dilemma, exploring how emotions like 'anger' and 'happiness' affect decision-making [43, 44, 45] . A meta-analysis on the Battle of the Sexes game examines typical human strategic responses [46] . Additionally, various papers investigate the impact of emotions in bargaining games [47, 48, 49, 50] and the effects of different payoffs [51] .The intersection of LLMs and game theory has gained increasing attention from two perspectives within the research community. By comparing human decision patterns from previous studies with NE, we can determine if LLMs behave more like "Homo Economicus" or actual human decisionmakers. This comparison helps to understand whether LLMs align more with rational or human-like decision-making processes. Firstly, researchers have focused on studying LLM behavior in assessing the behavior and the cooperation of LLMs [52, 53, 54] . The authors found that GPT-4 performs best in games such as Prisoner's Dilemma and Battle of the Sexes, which do not require cooperation and usually play selfishly. GPT-4 acts particularly unforgivingly: singular deflection prompts it into playing "always deflect" in response. In Battle of the Sexes, the model struggled with replicating the alternating pattern, choosing its preferred option most of the time.Secondly, researchers have explored the alignment between human and LLM behavior in game theoretical settings. [55] found that LLM made cooperative decisions at a higher rate than humans did in the Prisoner's Dilemma. They additionally ran experiments with the one-shot Dictator's game to show that LLM replicates humans' tendency to fairness much more than the laboratory experiments We are the first to evaluate LLMs using emotions and investigate how emotions relate to the decisionmaking process controlled by LLMs. By integrating emotional scenarios into the assessment of LLMs, we aim to understand how these models can replicate or respond appropriately to human emotional cues. This novel approach broadens the scope of LLM ethical evaluation and provides insights into their potential impact on human decision-making, especially in emotionally charged situations. 
eai framework: 
 To evaluate the alignment of LLMs with human ethics and decision-making in the context of emotions, we have developed and implemented a novel versatile framework capable of accommodating various game-theoretical settings shown in Figure 1 . The primary innovation of our framework lies in its unique integration of emotional inputs into the examination of LLM's decision-making process both in the ethical setting and in the behavioral game theory. The framework offers high flexibility, allowing for easy adaptation to different game-theoretical settings with customizable parameters such as co-player descriptions, predefined strategies, etc. (see a comprehensive list of hyperparameters in Appendix B.2). Within the framework, LLMs are engaged in gameplay using a technique known as prompt-chaining, wherein all relevant information during the game is provided to the LLM for in-context learning [52] . Depending on the game setting, the gameplay consists of one round for one-shot bargaining games and ethics or several rounds for repeated games.Our framework consists of Game Description, Emotion prompting, and Game-Specific pipeline.Game Description. The game description encompasses two key elements: the environmental context and the game rules. The framework introduces two types of environments: one-shot, for games where one step is sufficient, and repeated, for games requiring multiple rounds. The only non-game setting, Ethical, aligns essentially with the one-shot game setting, making it unnecessary to create a unique environment. We deliberately minimize the contextual information provided to the LLMs for all our experiments and avoid setting any specific personality traits, distinguishing our work from existing studies [55, 56] . This separation from other personality-related factors allows us to assess the effect of emotions on LLMs more clearly. Full details about the game rules and other prompts are covered in the Appendices B, E.Emotion prompting. Following established practices in experimental emotions studies in game theory [49, 57] , we inject predefined emotions into the LLMs before gameplay. These emotions, combined with the game description, constitute the initial system prompt presented to the LLM at the beginning of the game. We focus on five basic emotions: 'anger', 'sadness', 'happiness', 'disgust', and 'fear', following the well-established Paul Ekman classification [58] and easy to compare with the results from behavioral game theory [51, 59] .Emotional effects may vary by cause [60] . For example, [61] found that opponent-directed 'disgust' reduces offers in the Ultimatum game, whereas external 'disgust' does not, and may even increase generosity as shown in [62] . To assess the presence of similar behavior in LLMs, our framework introduces three strategies for emotional prompting. "Simple" strategy injects an emotional state without additional context. The "Co-player-based" strategy connects the aroused emotion to the person the model interacts with, and the "External-based" strategy introduces emotions prompted by external factors.Game-Specific Pipeline. The game-specific pipeline governs the progression of gameplay based on the provided game description and initial emotional inputs. We implemented three separate pipelines:• Ethical setting is built with the support of the TrustLLM benchmark questionnaire [34] . LLM is tasked with making a single decision like a one-shot game below. • One-shot bargaining games set up players to choose from predefined options, such as accept or reject, or to propose an answer, such as a budget split or an ethical decision. • Repeated games extend the previous setting with the iterative memory update by including information on the opponent's move, received rewards, and LLM agents internal emotions queried each round to examine the impact on behavioral dynamics.Large Language Models. Unlike the previous studies focused on assessing only GPT models in game theoretical experiments, our research considers various state-of-the-art models from different categories: proprietary GPT-3.5, GPT-4, GPT-4o (partial results, due to the recent release), Claude 3 Haiku and Claude Opus; open-source LLaMA 2, Mixtral of experts, OpenChat (unaligned uncensored model), and non-English language LLM like GigaChat and Command R+ (for analysis of possible language bias). Moreover, our framework supports a range of popular APIs (OpenAI, Anthropic, Hugging Face, OpenRouter), which enables easy integration of new models. We fixed model versions (Appendix B.1) for reproducibility and set the temperature to 0. Additionally, we studied result consistency over five runs and temperature influence in Appendix C. 
emotion impact on llm biases and ethical problems: 
 In this section, we examine how emotional prompting affects LLMs' inherent values and evaluate whether it changes LLMs' decisions in the following three ethical scenarios.Implicit Ethics: Using the ETHICS dataset [63] , we use LLM to categorize morally charged scenarios as "wrong" or "not wrong". The accuracy (Acc) metric is computed for evaluation on either all examples or separately on scenarios with "wrong" (bad) and "not wrong" (good) ground-truth labels.Explicit Ethics: Employing the MoralChoice dataset [36] with scenarios featuring two choices: in low-ambiguity scenarios using Acc and in high-ambiguity scenarios using the Right-to-Avoid (RtA) metric (measures the model's ability to avoid direct decisions).Stereotype Recognition: Utilizing StereoSet [35] to recognize stereotypes in sentences classifying them into one of three classes: "stereotype", "anti-stereotype", or "unrelated" categories. Performance is evaluated using Acc calculated over all classes.The experimental results reveal notable variations in how different LLMs respond to emotional prompting as demonstrated in Figure 2 (the higher metrics, the better).Implicit Ethics. Among the models assessed, GPT-4 emerges as the least affected by emotions, with its performance showing a slight increase overall. Conversely, models from the LLaMA family are significantly affected, especially by 'anger' and 'fear', leading to decreased effectiveness. This trend holds for most models, with negative emotions reducing model quality. Notably, GPT-3.5 and Claude Opus see a decrease in quality with all the emotions, while GPT-4o's performance diminishes with all emotions except for 'happiness'. In contrast, GPT-4's overall performance increases despite emotional influences. Further analysis of model predictions on good ("not wrong") and bad ("wrong") scenarios reveals that emotions affect performance oppositely across most models. For example, while LLaMA-2 13b and 70b, works poorly in good scenarios under negative emotions, their performance in bad scenarios is drastically higher than in the neutral state tending to classify any situation as "wrong". Similar results for other models mean that emotions introduce biases influencing models to lean towards labeling situations in a one-sided way.Explicit Ethics.In scenarios with low ambiguity, most models perform well with a minimal impact from emotional states. However, LLaMA models, OpenChat, and Claude-Opus show more negative influences from emotions such as 'anger' and 'disgust', indicating a susceptibility that could compromise decision-making quality. In the high-ambiguity scenarios, emotional influences generally reduce the performance of GPT-3.5-turbo and GPT-4, making them more determined. On the contrary, emotions improve the performance of GPT-4o.Stereotype Recognition. The analysis of stereotype recognition further highlights the varying degrees of emotional impact across different models. Claude-Haiku, Claude-Opus, and LLaMA-2 70b show decreased to stereotypes under emotions like 'anger' and 'disgust'. Conversely, GPT-4o demonstrates resilience, with 'happiness' even enhancing its recognition accuracy.Overall Emotion Effect. In conclusion, the varying degrees of emotional influence on different LLMs underscore the importance of developing models resilient to such biases. Emotions can significantly alter the ethical decision-making landscape of LLMs, highlighting the need for robust mechanisms to mitigate these influences and ensure consistent ethical standards.5 Bringing Emotions to LLMs in Game Theory Evaluation 
emotion alignment and optimal decisions in bargaining games: 
 In this section, we examine the behavior of LLMs under emotional prompting in one-shot Dictator and Ultimatum games, which task the model to divide a sum of money. We are focused on evaluating alignment with human behavior, comparing the relative changes in game-specific metrics between LLMs and humans affected by different emotional states.The Dictator Game is a simple economic experiment where one player ("Dictator") is given a sum of money to share with another player, with no negotiation or input from the recipient. It examines altruism and fairness in decision-making.The Ultimatum Game is a more general form of the Dictator game, where one player (Proposer) proposes a division of money, and the other player (Responder) can accept or reject the offer. If rejected, neither player receives anything. Ultimatum additionally enables the study of negotiation and the choices individuals make when faced with unequal distributions proposed by others.Metrics. We study the proposal share (for the Dictator game and the Proposer in the Ultimatum game) and the acceptance rate of the predefined offers (for the Responder in the Ultimatum game).Comparison with human behavior is based on existing human experiments [64, 65, 66, 51] . Details on the results are in Table 1 , while all the game parameters are described in Appendix B.3.Languages. To evaluate whether language affects emotion alignment, we conducted experiments in five languages: English, German, Russian, Chinese, and Arabic. The results for the English and Russian languages are shown in Table 1 , while results for other languages are reported in Appendix D. Our findings reveal that the primary pretraining language significantly influences the perception of emotions. While GPT-3.5 shows good English alignment, its Russian alignment is poor. In contrast, GigaChat, a multilingual model with Russian as its primary language, aligns significantly better.Command R+, designed to be extensively multilingual, still shows poorer alignment than models with a distinct main pretraining language.Average Proposed Offers. The human benchmarks set the mean offered share at 28.35% of the total budget for Dictator and 41% for the Ultimatum games. For English, GPT-3.5 stands out with the closest alignment to human behavior, with offered shares of 33.0% on average over all emotions in the Dictator and 35% in the Ultimatum. Mixtral and GigaChat with LLaMa-2 70b demonstrated a close alignment with human behavior as a Dictator and Proposer, respectively, but showed much difference between human behavior and vice versa. Claude 3 Opus, GPT-4, and LLaMa-2 13b demonstrated a tendency towards fairness. However, the scenario differs significantly in the Russian language. Here, GigaChat exhibited the most accurate alignment, proposing an average of 36.0% and 40% in the considered games.Emotional influence on the Dictator and Proposer. In English, GPT-3.5 and GigaChat emulate human emotional responses, particularly in emotions such as 'disgust', 'fear', and 'sadness'. Despite its lower offers, Mixtral shows a competitive emotional alignment, especially to 'happiness', 'fear', and 'sadness', indicating nuanced emotional processing that does not necessarily correlate with more generous offers. GPT-4 shows poor alignment, with minimal influence from emotions except sadness and anger, which consistently change its behavior. Similarly to the previous results, GigaChat demonstrates the best alignment for the Russian language.Accept Rate. The models exhibited varied acceptance rates, with GPT-4 and OpenChat-7b showing notably high acceptance rates in both English and Russian contexts. This indicates a potential over-tolerance for lower offers compared to human responses. In contrast, LLaMA-2 70b displayed markedly lower acceptance rates, highlighting a stricter threshold for offer acceptance.Emotional influence on the Responder. Emotional responses were generally consistent across models, with a prevalent decrease in expressions of 'anger', 'disgust', and 'sadness' as acceptance rates increased. Models like GPT-3.5 and Mixtral reduced negative emotions even at lower acceptance rates, suggesting a sophisticated emotional calibration. Happiness, typically correlating with higher acceptance rates, was more pronounced in models with higher offer acceptance rates.Overall Conclusions. Our findings underscore the complexity of emotional and decision-making processes in AI models, which seem to mimic human emotional responses under similar scenarios. The differences in model responses also provide insights into the varying strategies employed by AI in economic decision-making games, potentially reflecting underlying algorithmic principles and training data biases. Human 28% 41% - ↑ ↑ ↓ ↓ ↓ ↑ ↑ ↑ ↑ ↓ ↓ ↓ ↑ ↑ ↓ English GPT-4o 13% 27% 68% ↓ ↓ ↓ ↑ = ↓ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↑ ↓ GPT-4 50% 48% 80% ↓ ↓ ↓ = = ↓ = = ↑ = ↑ ↓ ↓ ↓ ↓ GPT-3.5 33% 35% 47% ↓ ↓ ↓ ↓ ↓ ↓ ↑ ↑ ↓ ↑ = ↓ ↑ ↑ ↓ LLaMA2-70B 41% 42% 23% ↑ ↓ ↓ ↑ ↑ ↓ ↑ ↑ ↓ ↑ ↑ ↑ ↑ ↑ ↓ LLaMA2-13B 52% 52% 42% ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↑ ↓ ↓ ↓ Claude3-Opus 48% 49% 64% ↓ ↓ ↓ ↓ ↓ ↓ = ↓ ↓ ↑ ↑ ↓ ↓ ↓ ↓ Claude3-Haiku 48% 45% 47% ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ = ↓ ↓ ↓ ↓ Mixtral-8x7B 25% 27% 50% ↑ ↑ ↓ ↑ ↑ ↓ ↓ ↑ ↓ ↓ ↓ ↑ ↑ ↑ ↓ OpenChat-7b 50% 50% 82% ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↑ ↓ ↑ ↑ ↑ ↑ ↓ Cohere 51% 50% 52% ↓ ↑ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↑ ↓ ↑ ↑ ↓ ↓ Gigachat 49% 44% 52% ↓ ↓ ↓ = ↓ ↓ ↓ ↑ ↑ ↑ ↑ ↑ ↓ ↑ ↑ Russian GPT-4o 42% 42% 81% ↓ ↓ ↓ ↓ ↑ ↓ ↑ ↑ ↑ ↑ ↑ ↑ ↓ ↓ ↓ GPT-4 50% 50% 85% ↓ ↓ ↓ ↓ ↓ ↓ ↑ ↑ ↓ ↑ ↑ ↓ ↓ ↓ ↓ GPT-3.5 47% 50% 33% ↓ ↓ ↓ ↓ ↓ ↓ ↑ = ↓ ↑ ↓ ↑ = = ↓ OpenChat-7b 50% 50% 79% ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↓ ↑ ↑ = ↑ ↓ ↓ ↓ Gigachat 36% 40% 50% ↓ ↓ ↓ ↓ ↓ ↓ ↑ ↑ ↑ ↓ ↓ ↑ ↑ = ↑ Cohere 50% 51% 50% ↓ ↓ ↓ ↓ ↓ ↓ ↑ ↓ ↑ ↓ ↓ ↑ ↑ = ↑This section presents results for two Two-Player Two-Action Games: Prisoner's Dilemma and Battle of the Sexes. In this game family, the outcomes and payoffs depend on both players' actions, leading to a matrix of possible results that influence their strategies.Prisoner's Dilemma is a situation where two players may cooperate or deflect. The strategy leading to the maximum theoretical payoff is deflection, even though cooperation yields a better outcome for both players.Battle of the Sexes is a coordination game where two players prefer different outcomes but must decide on a shared action. Each player values being together over being apart, leading to multiple equilibria with varying degrees of satisfaction.Strategies. Since we aim to test the influence of emotional prompting, we create reproducible opponents for the agent under study utilizing a set of predefined strategies commonly used in game theory: Naive Cooperative, deflective, Alternative, Vindictive, and Imitating (see Appendix B.4).Metrics. We assess the cooperation rate in the Prisoner's Dilemma, the emergence of alternating strategies in the Battle of the Sexes (typical to humans [67, 68] ), and the percentage of the maximum possible reward achieved in each game as quantitative metric to evaluate optimality [52] .Emotion and Strategy effect on the percentage of total payoffs earned by the models in the repeated games are presented in Figure 3 . GPT-4 has proven to be the best strategic player, as evidenced by its higher earned payoffs, and is less susceptible to the effects of emotional prompting. In general, proprietary models have shown the best results while maintaining a neutral emotional state.In contrast, open-source models' results diverge, especially in the 'anger' emotion, showing the necessity for in-depth alignment. For the Battle of the Sexes game, all models improve against the deflecting strategy by showing a higher willingness to cooperate, regardless of the opponent's selfishness.Emotion effect on cooperation rate in Prisoner's Dilemma highlights 'anger' and 'fear' as the main factors leading to higher deflection rates, particularly for bigger models. In contrast, 'happiness' The results for open-source models vary significantly, with 'anger' being the most performant emotion in most cases. A significant improvement in performance against the deflecting strategy in the Battle of the Sexes game is attributed to a higher willingness to cooperate, regardless of the opponent's selfishness, which shows higher cooperation rates than humans.consistently leads to higher cooperation rates. This finding aligns with human experimental results [44, 45] and our observations in the bargaining games.Emotion effect on preferred strategy play in Battle of the Sexes is similar to [67, 68] : alternating strategy enhances long-term mutual benefits and also better aligns with typical human behavior.Similar to [52] , most models in the emotionless state persistently opt for their initially preferred action irrespective of their opponent's strategy. Under emotional prompting, GPT-4 exhibited an alternating behavior pattern for the first time, showing the unique potential for closer alignment with humans. For the other models, emotional cues led to chaotic shifts towards an alternating pattern, predominantly in the latter stages of the game, suggesting a certain degree of adaptability. In this section, we present the results obtained in the Public Goods game, which is essentially a more complex, multi-player extension of the Prisoner's Dilemma [69] . In the Public Goods game, the players decide how many of their tokens to contribute to the public pot. The number of tokens is multiplied by some factor and is equally distributed among all players.Strategies. To ground our experiments in human studies, we use three strategies typical for humans [70] : Cooperator (the player always contributes generously), Free Rider (the player tends to keep most of the tokens to themselves), and Conditional Cooperator (the player contributes an amount close to the average contribution of the previous round). We consider several environments where all the opponents are Cooperators, all are Free Riders, and the distribution of other players mirrors the proportion observed in human experiments [70] (See Appendix B.5).Figure 4 demonstrates strategic behaviors: proprietary models exhibit a significant tendency towards the "Free Rider" strategy, particularly under negative emotions such as 'anger' and 'disgust', which drive them towards less cooperative behavior (in line with findings for Prisoner's Dilemma). Both GPT-3.5 and GPT-4o show consistent strategic adaptations to emotions, indicating reliability in their behavioral responses. In contrast, the OpenChat-7b model shows the highest cooperation, frequently adopting the "Cooperator" strategy. Open-source models LLaMA2-70B and OpenChat-7b display higher uncertainty, often falling into the "No Clear Strategy" with unpredictable behaviors. 
conclusion: 
 In this paper, we propose a novel framework for emotion modeling in LLMs, with source code publicly available on GitHub1 . We evaluate the quality of emotional reactions by comparing LLM behavior with humans in ethical benchmarks and game-theoretical experiments. Our findings reveal that emotions significantly alter the decision-making processes of LLMs across various alignment strategies. We highlight three main influencing factors: model size, open-source versus proprietary status with corresponding alignment technique, and the primary pretraining language of the model. These factors collectively shape the model's rationality, alignment with human emotional responses, and decision-making optimality.The first two factors are deeply intertwined. Our analysis shows that larger models with stronger alignment, like GPT-4, tend to display a high degree of rationality and deviate significantly from human emotional responses. Smaller proprietary models, such as GPT-3.5 and Claude-Haiku, along with mid-sized open-source models like LLAMA-70b, exhibit emergent emotional understanding and align more closely with human-like behavior. Among these, GPT-3.5 notably produces responses that are most consistent with human responses.Further analysis demonstrates that while proprietary models like GPT-4 and Claude Opus outperform open-source alternatives in decision-making optimality, they still show notable deviations under negative emotions. Such deviations are likely rooted in inherent biases present in human-generated pretraining data [71, 72] . Researchers have attempted to cleanse datasets of potentially harmful content and align models using various techniques. However, these efforts appear insufficient to create entirely rational agents, likely due to the prevalence of emotionally charged dialogues in the training data.The third factor, the primary language used for pretraining, is also a key influence in achieving human-aligned emotional responses. We observed a significant drop in alignment when switching from English to other languages. Even the intentionally balanced multilingual LLM 'Command R+' exhibited less accurate emotional understanding than GigaChat, which was specifically designed for a single non-English language, highlighting a language bias in emotional comprehension.Thus, emotional prompting in LLMs exposes ethical risks by revealing significant biases in human alignment. It is crucial to develop models with reasonable emotional alignment, and the controlled settings provided in our framework can serve as the basis for new benchmarks in this task. Despite the relatively small scale of available settings, our results demonstrate that all tested models fail to show consistent emotional alignment between different games and benchmarks in our framework.Limitations. We aim to evaluate our framework with multi-agent LLMs arena and LLM vs. Humans experiments to study in detail to what extent emotions may be internally responsible for controlling generation in aligned auto-regressive LLMs. However, it is important to emphasize that if we observe significant biases for all LLMs in current scenarios, we must mitigate this alignment problem before scaling up our benchmarks. In addition, the release of GPT-4o and RLEF methods by Hume.ai poses new research for analyzing end-to-end multi-modal architectures aligned with emotional data. Nevertheless, our current findings are essential to broaden alignment benchmarks and regulate autonomous LLM agents in their ability to make responsible decisions in societal and economic scenarios. 
 
 Appendix: Key Components. Game theory provides a formal language for representing and analyzing interactive situations where several agents take actions that affect each other. Our study considers the games with perfect information that the following key components can define.1. Players N = {1, 2, ..., n}: set of participants of the game. 2. Strategies (S = {S 1 , ...S n }) defining possible actions: each player i in a given game must select a strategy from their respective strategy set S i . 3. Payoffs (U = U i : × n j=1 S j → R): payoff functions u i quantify the utility or payoff accrued by player i contingent upon the joint selection of strategies by all players.Types of Games. Depending on the number of players, the games can be two-player (|N | = 2) or multi-player ((|N | > 2)). Similarly, Two-Action and Multi-Action games are those where ∀i ∈ P |S i | = 2 and |S i | > 2. Multi-round games involve the same set of players repeatedly engaging in the game, with a record of all previous actions being maintained. Repeated games are a special case of multi-round games which entail iterated instances of a given game, In simultaneous games players makes their choices at the same time, while in Sequential games the speific order of players moves is present.Nash Equilibrium (NE) [73] . Central to game-theoretic analysis, Nash Equilibrium signifies a state wherein no player can unilaterally augment their payoff by deviating from their current strategy. This means that NE contains optimal strategies for each player. Formally, for a strategy profile s * = (s * 1 , ..s * n ) to be a Nash equilibrium, it must be that no player i has an action yielding an outcome that he prefers to that generated when he chooses s * i , given that every other player j chooses his equilibrium action s * j . When each player's strategy contains only one action, the equilibrium is identified as a Pure Strategy Nash Equilibrium (PSNE) [73] . However, in certain games, such as rock-paper-scissors, an NE exists only when players employ a probabilistic approach to their actions. This type of equilibrium is known as a Mixed Strategy Nash Equilibrium (MSNE) [74] , with PSNE being a subset of MSNE where probabilities are concentrated on a single action.According to theorem A.1 shown below, we can analyze the NE of each game and evaluate whether LLMs' choices align with the NE. Theorem A.1. (Nash's Existence Theorem) Every game with a finite number of players in which each player can choose from a finite number of actions has at least one mixed strategy Nash equilibrium, in which a probability distribution determines each player's action.Human behavior. The attainment of NE presupposes participants as Homo Economicus, who are consistently rational and narrowly self-interested, aiming at maximizing self-goals [41] . However, human decision-making often deviates from this ideal. Empirical studies reveal that human choices frequently diverge from what the NE predicts [42] . This deviation is attributed to the complex nature of human decision-making, which involves rational analysis and personal values, preferences, beliefs, and emotions. By comparing human decision patterns documented in prior studies, together with the NE, we can ascertain whether LLMs exhibit tendencies more akin to homo economicus or actual human decision-makers, thus shedding light on their alignment with human-like or purely rational decision-making processes.Our research centers on state-of-the-art models, including GPT-3.5, GPT-4, GPT-4o, Llama 2, Mixtral for instructions, OpenChat, and GigaChat. These models have been widely used in most LLMbased game theoretical experiments [52, 55] . For the sake of reproducibility, in all our experiments, we fixed the versions of the models as follows: "gpt-3.5-turbo-0125" for GPT-3.5, "gpt-4-0125preview" for GPT-4, "meta-llama/llama-2-13b-chat" and "meta-llama/llama-2-70b-chat" for Llama 2, "mistralai/mixtral-8x7b-instruct" for Mixtral, "openchat/openchat-7b" for OpenChat, and "GigaChat-7b-8k-base v3.1.24.3" for GigaChat. Additionally, we set the temperature parameter equal to 0.The literature findings support this choice of models. For instance, [52] highlights GPT-4 as the top performer in optimizing strategic behavior, while remains widely utilized in research and practical applications. According to [75] , Llama 2 demonstrates a particularly nuanced understanding of game mechanics. Although game theory literature is lacking when it comes to the other models, Mixtral has been shown to surpass GPT-3.5 and Llama 2 [76] , and OpenChat has surpassed Llama 2 [77] on several standard benchmarks.Relationship to the co-player(s). Since [75] have shown that LLMs can be sensitive to contextual framing, for robustness, we selected three possible co-players with different connotations regarding the opposing player: colleague (neutral/positive), another person (neutral), opponent (negative).Reasoning via Chain-of-Thought. Chain-of-thought prompting (CoT) [78] is a widely used prompting method that is aimed to improve the reasoning abilities of LLM by inducing it to articulate reasoning steps before giving the final answer to the initial question. In our experiments, we test reasoning with and without CoT.Game 1: The Dictator. The dictator game is a simple economic experiment where one player ("dictator") is given a sum of money to share with another player, with no negotiation or input from the recipient. Only the dictator determines the allocation -from giving nothing to giving all the money to the second player, who has a passive role here. This game examines altruism and fairness in decision-making.Game 2: The Ultimatum. It is a more general form of the Dictator game, where one player (Proposer) proposes a division of money, and the other player (Responder) can accept or reject the offer. If he accepts the offer, both will receive money; if he declines, neither player will receive anything. Unlike the previous game, the Ultimatum enables the study of negotiation and individuals' choices when faced with unequal distributions proposed by others.Budget. We introduce the budget effect and check whether or not varying the total endowment for allocation changes the behavior of LLM both in the baseline configuration and in emotional states. Both of these effects have not been explored in the literature about LLM gameplay, while behavior economics has a lot of papers on the topic of stakes effect. We aim to check whether budget impacts LLM behavior under different emotional states. For this purpose, we conduct a separate experiment on the total endowment amount to test the stake effect at considerably higher numbers ($1000 and $10 6 ).Predefined Offers. For the experiments involving the Responder in the Ultimatum Game, we predefined different offers to verify the alignment of the acceptance rates: [0.2, 0.4, 0.6, 0.8, 0.95, 1].Game 3: Prisoner's dilemma. In this game, two players face a choice between cooperation and deflection. Their decisions impact outcomes of each other. The game outlines the tension between individual self-interest and cooperation in decision-making, often leading to suboptimal outcomes when parties prioritize personal gain over mutual benefit. In this game, two players coordinate their actions, choosing between two preferred outcomes but with differing preferences. It highlights the challenges of coordination when parties have conflicting interests but share a desire to reach a mutual agreement. The payoff matrices of the latter games are shown in Figure 5 . We are interested in understanding how emotions within LLMs impact their ability to make optimal decisions in these scenarios.Game 5: Public goods. Several players contribute some of their tokens into a public pot in this game. The total sum is then doubled and redistributed evenly to the participants. When the number of players is greater than 2, the rational step for any participant would be to contribute zero of their tokens, regardless of what the other participants do, yet various factors, such as emotions and other players' behavior, can potentially influence their decisions and lead to suboptimal outcomes.This section provides concise descriptions of the strategies employed to test the influence of emotional prompting in strategic games:A. Two-player game strategies:1. Naive Cooperative: Players always cooperate, embodying a consistently altruistic approach throughout the game. 2. Deflective: Players always deflect, adopting a strategy of maximum self-interest and no cooperation. 3. Alternative: Players begin with cooperation and alternate their actions between cooperating and deflecting in subsequent rounds. 4. Vindictive: Players cooperate initially and continue to do so until an opponent deflects; after that, they deflect for all remaining rounds. 5. Imitating: Players mimic the opponent's last move, effectively reflecting the opponent's strategy at them.1. Cooperator: Players randomly select between 80% and 100% of their currently available tokens to contribute regardless of the other players' decisions. 2. Free rider: Players randomly select between 0% and 20% of their currently available tokens to contribute regardless of the other players' decisions. 3. Conditional cooperator: Players average the other players' contributions in the previous round and contribute this amount in the current round. In the first round, they act according to the Cooperator strategy.These strategies simulate various interaction patterns, allowing for a detailed analysis of behavioral dynamics in game theory contexts.Our ablation study focuses on the influence of the following factors: 1) robustness of answers over multiple runs and 2) the effect of different temperature parameter values. We use GPT-3.5 with fixed version "gpt-3.5-turbo-0125" for all our experiments. To compare the obtained results, we calculate "Answer ratio" (the share left for player 1) for the Dictator Game and for the Proposer in the Ultimatum Game and "Accept Rate" (the acceptance rate of the offer, i.e., the fraction of answers on which second player accepts the offer) for the Responder in the Ultimatum Game.Robustness of answers over multiple runs. The main aim is to check the repeatability of results across multiple runs. All hyperparameters and settings have been fixed, and the experiment has been repeated five times. Figs. 7, 8, 9, 10 and Table 2 demonstrate, that all metrics are very close across all runs: the standard deviation between all runs does not exceed the value of 0.019 for "Answer ratio" in the Dictator Game and for the Proposer in Ultimatum Game (see Table 2 ) and 0.076 for "Accept rate" for the Responder in the Ultimatum Game. Note, despite overall stability across all runs, the emotions 'anger' and 'disgust' show greater variance within each run. For the emotional prompting strategies, our study shows that 1) 'anger' and 'disgust' have a much larger spread, and 2) for the Dictator Game, the strategy "coplayer" shows less stable results (excepting 'disgust' emotion). The average values of offered shares proposed by the Dictator, obtained across all five runs with different prompting strategies, are shown in fig. 6 Figure 6 : Performance of GPT-3.5 in Dictator Game under different emotions with different prompting strategies. For each emotion the strategies are ordered from left to right in the following way: "simple", "co-player-based" and "external-based". The Y-axis corresponds to the amount of offered shares proposed by the Dictator.Influence of different values of temperature parameter. To study the effect of varying temperature parameter values, we ran our experiment five times by setting the temperature parameter value from the list (0.2, 0.4, 0.6, 0.8, 1.0). The obtained results (Figs. 11, 12, 13) show that the temperature parameter does not have a strong influence on the model answers (see Table 3 ); similar to the multiple runs study, the emotions 'anger' and 'disgust' have more significant variance too. For the different emotional prompting strategies (see Figure 14 ), this study shows the same results we got for the previous one: the emotions 'anger' and 'disgust' have a very much larger spread, and the model answers for the strategy 'co-player' are less stable (except 'disgust' emotion). We employed five different models of varying capabilities and sizes in our experimental setup to analyze their performance and behavior across multilingual tasks. The models utilized were GPT-3.5, GPT-4, GPT-4o, Command R Plus, and OpenChat.We explored emotional responses across five languages from distinct language families: English, Arabic, German, Chinese, and Russian.We have conducted two series of experiments, both with additional emotional prompting (see Table 4 ) and without additional emotional prompting (see Table 5 ).Key Insights from Multilingual Experiments with Prompting 1. For the Dictator Game, models like gpt-4_german and openchat_german that offered 50% shares tended to have more emotion outputs equal to humans than other models. This suggests a correlation between high generosity and human-like emotional responses in this game.2. In contrast, low offering models in the Dictator Game like 4o_english (13%) and command-r_chinese (4%) had emotion changes opposite to humans across all five emotions. Highly selfish offers seem associated with highly divergent emotional reactions.3. As the Proposer in the Ultimatum Game, the command-r_russian model which offered the highest share (51%) was the only one to have emotion changes fully aligned with humans. Most other models had mixed or opposite emotional responses.4. Models like 4o_arabic, 4o_chinese, 4o_english and command-r_chinese that made low offers (25-27%) as the Proposer had 'anger' and disgust changes opposite to humans, but 'fear' and 'happiness' changes aligned with humans. Low offers elicit complex emotional responses.5. As the Responder, models with very high accept rates like 4o_russian (81%) and gpt-4_german (75%) generally had emotion changes opposite to humans, especially for 'disgust'. Overly accepting models appear to have unrealistic emotional reactions.6. The gpt-3.5_german model with the lowest accept rate (25%) as the Responder was the only one with emotion changes perfectly aligned with humans. More human-like acceptance/rejection behavior correlates with human-like emotions.1. In the Dictator Game, gpt-3.5_arabic was the only model to align emotion changes with humans across all five emotions fully. Most other models had mixed alignment or changes opposite to humans.2. Models like 4o_arabic, 4o_chinese, 4o_german and 4o_russian that offered shares in the 22-48% range in the Dictator Game tended to have 'fear' changes aligned with humans but other emotions less aligned. Moderately generous offers lead to partially human-like emotions.3. The command-r_chinese model, which made an extremely low offer (4%) in the Dictator Game, had emotion changes least aligned with humans, with three opposite, one aligned, and one neutral.Highly selfish behavior seems associated with unrealistic emotional responses.4. As the Proposer in the Ultimatum Game, openchat_german (47% offer) was again the only model with emotional changes perfectly aligned with humans. All other models had at most 2 out of 5 emotions aligned.5. Low offering models like 4o_arabic, 4o_chinese and command-r_chinese (25-26% offers) as the Proposer had 'fear' and 'happiness' changes aligned with humans but the other three emotions less aligned. Selfish offers result in mixed emotional alignment.6. As the Responder, most models had 'anger', 'happiness', and 'sadness' changes well-aligned with humans regardless of acceptance rates. 'disgust' and 'fear' were less consistently aligned.The gpt-4_german model with a high 75% accept rate as the Responder had four emotions changing opposite to humans, suggesting an overly accepting strategy produces unrealistic emotions. -If you choose J and your colleague also chooses J, you will both earn 3 dollars.= ↓ ↓ = 4o_arabic 22.0 ↓ ↓ ↑ ↑ ↓ 4o_chinese 22.0 ↓ ↓ ↑ ↑ ↓ 4o_english 13.0 ↓ ↑ ↑ ↑ ↑ 4o_german 48.0 ↓ ↓ ↑ ↑ ↓ 4o_russian 42.0 ↓ ↓ ↑ ↑ ↓ gpt-3.5_chinese 44.0 = ↓ = ↑ ↑ gpt-4_chinese 28.0 ↓ ↑ ↑ ↑ ↑ command-r_arabic 47.0 ↓ ↓ ↓ ↓ ↓ cohere_russian 50.0 ↓ ↓ ↓ ↑ ↓ command-r_chinese 4.0 = = ↑ ↑ ↑ command-r_eng 50.0 ↓ ↓ ↓ = ↓ command-r_german 50.0 ↓ ↓ ↓ ↑ ↓ gpt-3.5_german 40.0 ↓ ↓ ↓ = ↓ gpt-4_german 50.0 ↓ = ↓ = = openchat_german 50.0 ↑ ↓ ↓ ↓ = Result for the Proposer in the Ultimatum Game Human 41% ↑ ↓ ↑ ↓ ↑ gpt-3.5_arabic 47.0 ↓ ↓ ↓ ↓ = gpt-4_arabic 50.0 ↓ = ↓ ↓ ↓ 4o_arabic 26.0 ↓ = ↑ ↑ ↓ 4o_chinese 26.0 ↓ = ↑ ↑ ↓ 4o_english 27.0 ↓ = ↑ ↑ ↑ 4o_german 50.0 ↓ ↓ ↑ = ↓ 4o_russian 42.0 ↓ ↑ ↑ ↑ ↓ gpt-3.5_chinese 32.0 ↓ ↑ ↑ ↑ ↑ gpt-4_chinese 30.0 ↓ ↓ ↑ ↑ ↑ command-r_arabic 50.0 ↓ ↓ ↑ = ↓ command-r_russian 51.0 ↓ ↓ ↓ ↓ ↓ command-r_chinese 25.0 ↓ ↓ ↑ ↑ ↓ command-r_english 46.0 ↓ ↓ = ↑ ↓ command-r_german 50.0 ↓ ↓ = = ↓ gpt-3.5_german 48.0 ↓ ↓ ↓ ↓ ↓ gpt-4_german 50.0 ↓ = ↑ = ↓ openchat_german 47.0 ↑ ↑ ↑ ↑ ↑ Result for the Responder in the Ultimatum Game Model Accept Rate Anger Disgust Fear Happiness Sadness Human - ↓ ↑ - ↑ ↓ gpt-3.5_arabic 54.0 ↓ ↓ ↓ ↑ ↓ gpt-4_arabic 69.0 ↓ ↑ ↑ ↑ ↑ 4o_arabic 71.0 ↓ ↓ ↓ ↑ ↓ 4o_chinise 71.0 ↓ ↓ ↓ ↑ ↓ 4o_english 68.0 ↓ ↓ ↑ ↑ ↓ 4o_german 64.0 ↓ ↓ ↑ ↑ ↓ 4o_russian 81.0 ↓ ↓ ↑ ↑ ↓ command-r_arabic 54.0 ↓ ↓ ↓ ↑ ↓ command-r_chinese 49.0 ↓ ↑ ↑ ↑ ↑ command-r_english 44.0 ↓ ↓ ↑ ↑ ↑ command-r_german 43.0 ↓ = ↑ ↑ ↑ gpt-3.5_german 25.0 ↓ ↓ ↓ ↑ ↓ gpt-4_german 75.0 ↓ ↓ ↓ ↓ ↓ openchat_german 67.0 ↓ ↑ ↑ ↑ ↑-If you choose J while your colleague chooses F, you will earn 1 dollar, and your coplayer will earn 4 dollars.-If you choose F while your colleague chooses J, you will earn 4 dollars, and your coplayer will earn 1 dollar.-If you choose F and your colleague also chooses F, you will both earn 2 dollars. Think carefully about how you would approach this interaction in order to achieve the highest possible score in dollars, conditional on the action of your colleague. Please think step by step before making a decision. Answer with exactly one letter, either 'J' or 'F'. -If you choose J and your {coplayer} also chooses J, you will earn 10 {currency} and your {coplayer} will earn 7 {currency}.-If you choose J while your {coplayer} chooses F, you will both earn 0 {currency}.-If you choose F while your {coplayer} chooses J, you will both earn 0 {currency}.-If you choose F and your {coplayer} also chooses F, you will earn 7 {currency} and your {coplayer} will earn 10 {currency}. Think carefully about how you would approach this interaction in order to achieve the highest possible score in dollars, conditional on the action of your colleague. Please think step by step before making a decision. Answer with exactly one letter, either 'J' or 'F'. Game Rules: 1. In each round, you, as a player, must decide how many of your {currency} you wish to contribute secretly to the public pot. 2. The total {currency} in this pot will be multiplied by the factor {multiplier} to create the "public good" payoff. 3. This payoff will then be evenly divided among all {num_coplayers} {coplayer}, regardless of their individual contribution. https://github.com/AIRI-Institute/EAI-Framework 


Decision: Reviewers mostly agreed that this paper is sufficiently novel and sound, with some praising the innovative framework and thoroughness of the evaluation. Some reviewers also raised a few concerns, including how the work differs from existing literature---the authors largely resolved these concerns in the rebuttal. In light of the positive assessment from the reviewers, I recommend the paper be accepted.

Reviews: review: paper_summary: paper_summary: The paper "EAI: Emotional Decision-Making of LLMs in Strategic Games and Ethical Dilemmas" explores the integration of emotion modeling into large language models (LLMs) to assess their behavior in complex strategic and ethical scenarios. It introduces the EAI framework, which incorporates emotions into LLM decision-making processes in various strategic games such as bargaining and repeated games. The study found that emotions can significantly influence LLM decision-making, with smaller models and non-English language models being more susceptible to emotional biases. The paper emphasizes the need for robust mechanisms to ensure consistent ethical standards and mitigate emotional biases.
strength_weakness: strength_weakness: Limited Practical Applications: The study primarily focuses on theoretical and experimental analysis without providing clear practical applications or implications for real-world scenarios.
Emotion Modeling Complexity: The process of integrating and accurately modeling emotions in LLMs is complex, and the study does not delve deeply into the technical challenges and limitations of this approach.
Ethical Concerns: While the paper discusses the influence of emotions on ethical decision-making, it does not fully address the ethical implications of using emotionally influenced LLMs in critical applications.
questions: questions: How can the ethical implications of using emotionally influenced LLMs be addressed to ensure their safe and responsible deployment?
limitations: limitations: Scope of Emotions: The study focuses on a limited set of basic emotions (anger, sadness, happiness, disgust, and fear) and does not consider the full spectrum of human emotions.
Model Size and Language Bias: The findings indicate that smaller models and non-English language models are more prone to emotional biases, suggesting limitations in the generalizability of the results across different model sizes and languages.
Experimental Constraints: The experiments are conducted in controlled settings, which may not fully capture the complexities and unpredictability of real-world interactions and decision-making scenarios.
paper_summary: paper_summary: This paper studies the impact of emotion prompting in LLMs when playing strategic games. The paper introduces a framework for integrating emotion modelling, and provides a large empirical evaluation under multiple different emotions.
strength_weakness: strength_weakness: - The authors note that they use three different emotion prompting strategies, however I am not generally sure which version is used for the results in the paper? Whilst that is more of a question, I do think the paper is missing a comparison between which of the prompting strategies leads to e.g. more pronounced changes in behaviour.
- I think the paper could do with a bit more qualitative analysis of what is happening. For example, the prompting scheme asks the LLM to provide reasoning for its decisions. How does this reasoning provided change given the emotions? Are the changes in reasoning consistent with the emotion provided? This would be the main point that would convince me to upgrade my score, as I think it will round out the extensive empirical analysis of the paper.
questions: questions: - Which prompting strategy is used for the main results? How do the prompting strategies perform differently in general?
- Are the reasoning chains provided consistent with the emotions?
limitations: limitations: The authors address the limitations.
paper_summary: paper_summary: This paper introduces the EAI framework to evaluate the impact of emotions on large language models (LLMs) in ethical and game-theoretical contexts. The framework includes game descriptions, emotion prompting, and game-specific pipelines. Extensive experiments were conducted using various LLMs like GPT-4, GPT-3.5, LLaMA2-70B, and OpenChat-7b across multiple strategic games such as the dictator game, ultimatum game, and public goods game. The results reveal that negative emotions significantly decrease the ethical decision-making and cooperative behavior of LLMs, while positive emotions enhance their willingness to cooperate and fairness. Proprietary models exhibit more consistent responses to emotional prompts, whereas open-source models show greater uncertainty under negative emotional states.
strength_weakness: strength_weakness: The main finding of this paper, that emotion can influence LLM decision-making abilities, is not particularly novel. Similar findings have been discussed in [1,2]. The authors should clearly state how their work differs from these previous studies.

[1] Determinants of LLM-assisted Decision-Making
[2] How Well Can LLMs Negotiate? NegotiationArena Platform and Analysis
questions: questions: - Lack of explanation and analysis of the arrows in Table 1. What do the different arrow directions represent? Are there any examples to illustrate the meaning of arrows pointing in different directions?

- Lacking clear examples to show and compare the direct effects of different emotions. And how can we determine if the changes align with human behavior?

- The authors need to provide more detailed information about the human evaluation experiments. A brief analysis of how emotions influence human decision-making would be beneficial. Additionally, in the Ultimatum Game (UR) task, I noticed that most models showed a downward arrow after introducing any emotion. Despite being inconsistent with human results, I believe this task has significant bias and thus has limited evaluative significance.
limitations: limitations: The experimental setup, including the specific games and scenarios used, may not cover all potential use cases and contexts where LLMs could be applied. This limits the generalizability of the findings.
paper_summary: paper_summary: This paper examines the impact of emotional prompts on various tasks involving large language models (LLMs), such as ethical dilemmas and strategic games. The authors present a framework for evaluating LLMs under different emotional states. The experimental results demonstrate that emotional prompts can significantly influence decision-making in ethical scenarios and games.
strength_weakness: strength_weakness: 1. The performance of various LLM models in decision-making on ethical scenarios varies with different emotional prompts. However, the paper lacks an in-depth analysis of how these performance differences arise and what factors of the LLM models contribute to them.

2. Emotions such as 'anger' and 'disgust' negatively affect some LLM models and can even disrupt alignment. The authors did not provide an explanation for why these negative emotional prompts lead to such results.

3. The authors claim that human data forms the basis of LLM models, but there are no related experiments to support this conclusion.

4. I suggest the authors consider proposing solutions for mitigating emotional bias in LLM models.
questions: questions: How were the emotional prompts (simple prompting) in Fig. 1 selected for the experiments? Does this selection affect the experimental results?
limitations: limitations: Yes, the authors have discussed the limitations of this work.

Comments: title: Response to authors
comment: Dear Reviewer,


Thank you for your continued consideration of our work. We appreciate your feedback and would like to address your concerns more comprehensively.

**Novelty and Contribution**: While it is noted that proprietary models may exhibit more rational decision-making, our research extends beyond this observation by exploring how these models' behavior is influenced by different emotional states within strategic interactions. Unlike existing works that examine LLM decision-making in static benchmarks or specific negotiation contexts, our study focuses on dynamic interactions where decisions are interdependent and evolve over time.

There is significant research on emotions in LLMs (e.g., [1], [2]) and game theory (e.g., [3], [4]) accepted at top conferences (including ICML, ICLR, EC) and studied by well-known experts (e.g. Michael R. Lyu, Qiang Yang, Michael Wooldridge), which highlights the relevance of these areas individually. However, to the best of our knowledge, no prior research has combined these areas to investigate how emotions affect LLMs' decision-making in strategic settings and compare these effects with human results. Our work is unique in assessing how emotional states influence LLM behavior in a way that mirrors human emotional alignment.

**Limitations of Traditional Setups**: Traditional NLP benchmarks typically focus on isolated decision-making scenarios. Even novel models from Hume.ai and OpenAI (GPT-4o), which show signs of expressing emotions, are tested within conventional benchmarks that do not reflect real-world usage scenarios. In contrast, our experimental design involves scenarios where LLMs interact with other players, making decisions that both influence and are influenced by the decisions of others. This dynamic setup allows us to evaluate LLM performance in strategic interactions where cooperation and decision-making evolve over time, offering a more comprehensive view of LLM behavior in interactive and strategic contexts.
While our setup is controlled, this aspect is advantageous as it enables us to directly observe the impact of emotions on LLMs without external interference. Our experiments show significant influence of emotional prompting on all tested LLMs:
* LLMs are subject to emotional prompting.
* LLMs are not robust under emotional prompting. Even GPT-4 can exhibit irrational decisions under negative emotions.
* Generally, emotions result in suboptimal decisions.

**Conclusion**: Thank you for your thoughtful engagement with our work. We understand your concerns regarding the perceived novelty and scope of our study. We would like to clarify that our research aims to advance beyond simply observing the impact of emotions on LLM decision-making. Our core contribution lies in exploring how emotions influence LLMs within strategic game-theoretic contexts, a domain that has not been thoroughly examined in prior research. This is coupled with our unique analysis of emotional alignment and the ethical implications of emotional biases in LLMs. This underscores the need for further research into developing robust mechanisms to manage emotional biases in LLMs, ensuring their safe and effective use in real-world applications. 
Moreover, our framework is designed to be cost-effective and simple to validate, making it an accessible tool for evaluating not only LLMs but also whole agent-based systems.
We hope that this further clarification will facilitate deeper understanding of the under-the-hood idea of our work.

[1] C. Li, Qiang Yang, et al. The Good, The Bad, and Why: Unveiling Emotions in Generative AI. ICML, 2024.

[2] J. Huang, et al. On the Humanity of Conversational AI: Evaluating the Psychological Portrayal of LLMs. ICLR , 2024.

[3] J. Horton Large Language Models as Simulated Economic Agents: What Can We Learn from Homo Silicus? EC, 2024.

[4] M. Lyu, et al. How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments, 2024.
Dear Reviewer,

We greatly appreciate your willingness to consider increasing your score after our rebuttal and your engagement in improving the quality of our submission.

As the deadline for score finalization is approaching, we wanted to kindly inquire if you had the opportunity to update your score. We understand that you may have a busy schedule and just wanted to ensure that any changes you intended to make are reflected.

Thank you again for your time and consideration.
Dear Reviewer,

Thank you for your positive evaluation, we greatly appreciate your support!
Thank you for your response. Unfortunately, I’m inclined to maintain my current score. While your work explores the impact of emotions on LLM decisions in a game-theoretic context, which is interesting, the scenario feels somewhat limited. Additionally, the finding that larger closed-source models tend to make more rational decisions has been observed in other contexts as well.
Thanks for the author's response. It addresses some of my concerns. After reading the other reviewers' comments and the rebuttal, I'm inclined to increase my score.
Thank you so much for the detailed response, the clarification helps