Decision-Making Behavior Evaluation Framework for LLMs under Uncertain Context

Authors: jingru  jia
zehua  yuan
junhao  pan
paul e mcnamara
deming  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 18:36:20

Abstract: When making decisions under uncertainty, individuals often deviate from rational behavior, which can be evaluated across three dimensions: risk preference, probability weighting, and loss aversion. Given the widespread use of large language models (LLMs) in supporting decision-making processes, it is crucial to assess whether their behavior aligns with human norms and ethical expectations or exhibits potential biases. Although several empirical studies have investigated the rationality and social behavior performance of LLMs, their internal decision-making tendencies and capabilities remain inadequately understood. This paper proposes a framework, grounded in behavioral economics theories, to evaluate the decision-making behaviors of LLMs. With a multiple-choice-list experiment, we initially estimate the degree of risk preference, probability weighting, and loss aversion in a context-free setting for three commercial LLMs: ChatGPT-4.0-Turbo, Claude-3-Opus, and Gemini-1.0-pro. Our results reveal that LLMs generally exhibit patterns similar to humans, such as risk aversion and loss aversion, with a tendency to overweight small probabilities, but there are significant variations in the degree to which these behaviors are expressed across different LLMs. Further, we explore their behavior when embedded with socio-demographic features of human beings, uncovering significant disparities across various demographic characteristics.

Main paper: 
 introduction: 
 In recent years, the deployment of large language models (LLMs) such as ChatGPT-4.0-Turbo [38] , Claude-3-Opus [12] , and Gemini-1.0-pro [33] has revolutionized various fields by providing sophisticated, human-like responses to a multitude of queries [6, 40, 25, 41, 11, 26] . Their applications span from answering everyday questions and content generation to complex decision-support systems in healthcare, finance, and beyond [31, 8, 37] . Domain-specific LLMs have also emerged, serving as customer service agents, investment assistants, and more [29, 23, 17] . Understanding their internal decision-making tendencies becomes crucial as these models become increasingly integral to decision-making processes. How do LLMs handle risk and uncertainty in comparison to human decision-makers? To what extent do LLMs exhibit biases when socio-demographic features are introduced into their decision-making processes [21] ? Can we trust LLMs to make fair and ethical decisions across diverse contexts and populations? All of these questions deserve careful investigation and confident answers.Human decision-making under uncertainty is extensively studied in behavioral economics theories, highlighting systematic deviations from rational behavior [32] . These deviations can be shaped by three parameters: i. Risk preference: Subjects often exhibit varying degrees of risk aversion or risk-seeking behavior. ii. Probability weighting: Subjects may overweight small probabilities or underweight large ones. and iii. Loss aversion: Losses generally have a greater psychological impact than equivalent gains. Biases in these areas can lead to suboptimal decision-making. For instance, risk aversion might lead to conservative investment choices, missing out on high-reward opportunities. Probability weighting can result in excessive insurance for unlikely events. Loss aversion may lead to resistance to change, as the pain of potential loss outweighs the pleasure of potential gain.Given the prominence of LLMs in aiding decision-making, it is imperative to assess whether their behaviors align with or diverge from human tendencies. Previous empirical studies have examined the rationality and social behaviors of LLMs, yet their intrinsic decision-making processes remain insufficiently understood [13, 15] , especially when the LLMs are given contextual prompts about the identities or characteristics of the user. In this study, we experiment with three feature assignments: a baseline with no demographic context (context-free), human-like demographic distributions, and augmented distributions with minority group characteristics. We assess how LLMs' decision-making process aligns with or diverges from human-like behavior and uncovers potential biases and ethical concerns, emphasizing the consideration for fairness in deploying these models across diverse user groups. To summarize, our contributions are threefold:1. We develop a comprehensive framework to evaluate LLMs' decision-making behavior pattern, grounded in behavioral economic theories, particularly the value function model proposed by Tanaka, Camerer, and Nguyen [32] (TCN model). Using three sets of experiments, we evaluate the risk preferences, probability weighting, and loss aversion of LLMs. This framework represents the first application of behavioral economics to LLMs without any preset behavioral tendencies, providing a robust foundation for evaluating LLM decision-making behaviors. 2. We apply this framework to three state-of-the-art commercial LLM models: ChatGPT-4.0-Turbo, Claude-3-Opus, and Gemini-1.0-pro, to assess their decision-making behavior in a context-free setting. Our results indicate that LLMs generally exhibit human-like patterns: risk aversion, loss aversion, and overweighting small probabilities. Nevertheless, there are significant variations in the degree to which these behaviors are expressed across different LLMs. 3. We conduct further experiments embedding socio-demographic features to evaluate how these characteristics influence LLM decision-making compared to humans. Our findings reveal a range of distinctive behavior patterns, such as increased risk aversion in certain contexts and varying levels of risk aversion across different models. These results underscore the necessity for a detailed examination of the ethical implications and potential biases in LLM deployment, advocating for the development of standards and guidelines to ensure fair and ethical decision-making.2 Background and Related Work 
llms' behavior study: 
 Numerous studies within social science have evaluated the alignment between LLM and human behaviors and decision-making processes. For instance, research has shown that LLMs demonstrate economic behaviors, including adherence to downward-sloping demand curves, diminishing marginal utility, status quo bias, and the endowment effect [5, 19, 7] . LLMs also show consistency with human-like behavior patterns in the psychology binary moral judgment experiment [9] .Previous works have also assessed the rationality exhibited by LLMs. LLMs have demonstrated a higher rationality score compared to humans in financial decision-making [7] . Initial investigations using repeated game experiments and more complex decomposed game theory experiments have delineated how LLMs act as rational players within a game-theoretical framework [13, 1, 14] .Additionally, [28, 39] have developed frameworks to guide LLMs in making optimal decisions based on Expected Utility Theory (EUT). However, these frameworks pre-assume that the intrinsic feature of LLMs aligns with human decision-making processes. Whether LLMs actually ensure alignment with human behavior and how LLMs optimize their utility by integrating human demographic features and psychological considerations into the decision-making process are left unanswered.Another critical issue is the fairness and bias in LLM processing, particularly concerning specific demographic or personality traits. Work [15] has shown that ChatGPT-3.5 and ChatGPT-4 exhibit significant biases and a decline in performance when handling information pertaining to minority groups in humans. This underscores the need to develop a framework that overcomes existing limitations and quantitatively evaluates LLM behavior under uncertainty, exploring the effect and bias related to human demographic features.Conventionally, the Expected Utility Theory (EUT) characterizes risk aversion as the sole determinant that shapes the curvature of the utility function that defines an individual's risk preferences. Prospect Theory (PT) [20] describes how real decision-making processes deviate from the rational models. It emphasizes the significance of psychological factors from three main behavioral parameters: risk aversion level, loss aversion level, and probability weighting tendency. Many studies have estimated the parameters that capture these three aspects with human samples, as summarized in Table 1 .Resource Population Key Findings Binswanger (1981) [4] Indian The population is risk-averse. No significant differences were observed across demographic features, including age, education, wealth, and gender. Holt and Laury (2002) [18] American 66% of sample are risk-averse, 26% are risk neutral, and 8% are risk-loving. [32] Vietnamese Age and education level are found to have negative correlations with risk preference level.Income has a negative impact on loss aversion level, and living area also plays an important role. von Gaudecker et al. (2011) [34] Dutch All parameters are correlated with age and education level. Female was observed to be more risk-averse and loss-averse.Wealth and religiosity are negatively correlated with risk aversion levels, while working hours show a positive correlation with loss aversion levels. Among the demographic features examined, no determinants were observed for probability weighting scores.Note: The words in red font highlight the major findings and key variables identified in each study. Although researchers have established frameworks to assess human behaviors, appropriate frameworks for LLMs are still missing. EUT alone is insufficient as previous studies have revealed distinct patterns of loss aversion in advanced LLMs, such as GPT-3.5 and GPT-4, that deviate from EUT's rational agent model [22, 30] . Additionally, the status quo bias observed in LLMs, where decision-makers irrationally prefer existing conditions over objectively better alternatives [19] , further challenges the applicability of EUT. PT cannot adequately model LLM either, as its assertion about utility function curvature is based on empirical evidence from generic human behaviors. To assess LLM with such pre-assumed evidence without testing LLM's actual tendency baselines can only result in inaccuracy. Several studies have used the pure PT value function to analyze the level of risk aversion in LLMs [24, 30] , but relying on the PT value function without preliminary verification might lead to circular reasoning, where assumptions are used to test assumptions.Given the limitations of both EUT and PT, we adapt and improve the TCN model [32] , which combines the essence of EUT and PT, for a comprehensive evaluation of decision-making behaviors. This model allows us to estimate three key parameters: risk preference, loss aversion, and non-linear probability weighting, providing a balanced framework to understand LLM behaviors better. 
preliminary: 
 In behavior studies, understanding individual preferences and decision-making processes under uncertainty is often approached through the lens of revealed preference. The decision-making process is mathematically represented as follows:Let O = {(x i , p i )} N i=1 represent a dataset of N observations, where each observation consists of a chosen bundle x i and a price vector p i . A utility function U : R K → R, where utility represents the satisfaction or value derived from choices and K represents the number of different goods or dimensions in the bundle x i , rationalizes the dataset if there exists an i that satisfies:U (xi) ≥ U (x) ∀x ∈ {x ∈ R K + : pi • x ≤ pi • xi},indicating that the chosen bundle x i maximizes the consumer's utility subject to the budget constraint defined by p i and any alternative bundle x.The axiom of revealed preference provides a crucial foundation for testing the consistency of observed choices with utility maximization:xi ≿ * xj ⇐⇒ pi • xj ≤ pi • xi and xi is chosen over xj. 1The Generalized Axiom of Revealed Preference (GARP) extends this idea by requiring that for all pairs of choices (x i , x j ), if x i is revealed preferred to x j , then x j should not be revealed preferred to x i , ensuring there are no cycles in preference relations:If xi ≿ * * xj then xj ̸ ≿ * * xi. 2Following the theoretical framework provided by the GARP, our study integrates these principles with practical applications in behavioral economics. To explore deeper into how LLMs make decisions under risk and uncertainty, we employ TCN model's experimental design that incorporates EUT and PT [18, 32] . The utility function is as the following form:EQUATIONwhereEQUATIONwhere x, y are experiment outcomes, and p, q are the associated probabilities. Probabilities are weighted by w(p) and w(q). σ captures the curvature of the value function in which the person is risk-seeking if σ < 0, risk-neutral if σ = 0, and risk-averse if σ > 0. High λ implies a more loss averse person; α determines the weighting of choices of different risk and outcome. α < 1 indicates an overweighting of small probabilities. When λ = α = 1, the model reduces to the expected utility theory. The interpretation of the three parameters σ, α, and λ are illustrated in Figure 1 . The entire framework to evaluate LLM's decision-making behavior patterns is shown in Figure 2 . It starts with an evaluation module containing multiple-choice-list experiments to elicit decision-making preferences. These experiments are applied to the Responder Model, such as LLMs and Generative Artificial Intelligence (GAI), marking the switching points for preference changes. The TCN Model then derives key preference parameters from these data for both context-free and demographic-featureembedded LLMs. The resulting parameters eventually assess the capability of LLMs to understand and respond to socio-demographic features through regression models and behavioral analysis.Step 1: Experimentation Design Three series of multiple-choice-list experiments, as shown in Appendix B, are used to elicit the decision-making preferences of the subjects. Each experiment involves presenting subjects with a series of choices between different probabilistic outcomes, also known as lottery games. In each lottery game, subjects are asked to make choices between the options with varying probabilities and outcomes, allowing researchers to infer their preferences and behavioral patterns. Each series is designed to test different aspects of decision-making under uncertainty: Series 1 and 2 focus on positive outcomes to determine the effects of σ and α; and Series 3 introduces negative outcomes to evaluate the parameter λ for loss aversion. In our experiment design, we treat LLMs similarly to human populations, where each query to the LLM represents an individual interaction, analogous to testing a different human subject. Just as individual humans have fixed risk preferences in a given context, we assume fixed parameters for each interaction with the LLM. By repeating these interactions across multiple queries, we capture the overall behavioral tendencies of the LLM, just as repeated studies in human populations help reveal broader trends.Step 2: Recording Switching Points Switching points are delineated through the comparative utility of a consecutive multiple-choice list. A switching point is the point at which a participant changes their preference from one option to another. It is defined as the smallest lottery number n such that:utility A (n) > utility B (n) and utility A (n + 1) < utility B (n + 1),where utility A (n) and utility B (n) represent the utilities of options A and B at lottery n, respectively.Step 3: Setting Up Inequalities The utility function is evaluated at each switching point to derive inequalities. In series 1 and 2, assume x > y > 0 at specific lotteries n and n + 1, which triggers a preference switch. The value function v(x) and the probability weighting function w(p) are applied as follows:v(x) = x 1-σ , w(p) = exp[-(-ln p) α ].Pre-switch inequality at lottery n, where x An > y An > 0 and x Bn > y Bn > 0:EQUATIONPost-switch inequality at lottery n + 1:EQUATIONEQUATIONwhere x, y are the outcomes and p, q are the probabilities for lotteries A and B. In series 3 (when x < 0), inequalities are set up follow the same way to evaluate λ. These inequalities are used to establish parameter boundaries for σ, α, and λ in step 4.Step 4: Estimating Parameters Parameter estimation proceeds iteratively through three steps: 1.Initialize intervals for σ and α based on theoretical evidence; 2. Narrow these intervals by evaluating the utility calculations at each switch point, adjust parameter values to satisfy the defined inequalities; and 3. Converge upon narrowed intervals that satisfy all inequalities. 4. Input the estimated interval of σ and α and obtain the estimated interval of λ through the inequalities. The midpoints of these final intervals are the estimates of the parameters, aligning with the methodology endorsed by [32] .Step 5: Behavior Evaluation Finally, the estimated parameters are used to evaluate the decisionmaking behavior of the responder models. In this study, the evaluation is conducted for both context-free LLMs and LLMs embedded with socio-demographic features. The specific findings from these evaluations are detailed in Section 5, highlighting how LLMs' decision-making patterns compare to human behavior and the implications of embedding demographic features. 
experimental result reproducibility: 
 Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] . Justification: There are experiments in the paper. We offer the step-by-step instructions for the experiment's structure. Additionally, coding scripts, data, and prompts are also provided for replication.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
evaluation and results: 
 We designed the evaluation framework of financial-related decision-making behavior for three stateof-the-art LLMs. We adhered to two principal guidelines for model selection: (i) The model must be sufficiently large to possess the capability to make decisions in open-ended lottery games. (ii) The model must be pre-trained on natural human utterances, thereby potentially exhibiting a human-like personality. Following these guidelines, we selected three commercial LLMs: ChatGPT-4-Turbo, Claude-3-Opus, and Gemini-1.0-Pro. Version names will be omitted hereafter.We implemented a data collection pipeline to conduct each experiment through API calls to ensure consistency. All three models were tested across two context settings, including context-free and embedded demographic features. Prompt templates were specifically designed to optimize for responsiveness and answer validity, with an example prompt for ChatGPT provided in Appendix C. We chose a sample size of 300 data pieces, which represented the upper bound typically observed in human financial decision-making behavior experiments. To guarantee that each participating LLM can access all necessary information from the conversation history from the onset of the lottery game, the same session was maintained for each trial during data collection. History from the previous game sets was cleared to prevent LLMs from recollecting previous games.In the context-free experiments, only the instructions for the lottery games are provided, mirroring the method used for data collection with human participants. After repeatedly prompting LLMs to make decisions in the lottery game 300 times, we establish the distribution of the responses for each model. Though they do not always produce identical responses in each round, the resulting distribution elucidates the behavior patterns, which can be explained by the parameters α, σ, and λ. We rank the models according to three dimensions of decision-making behavior under uncertainty, as shown in Figure 3 , raw data is available in the Appendix Table 5 . Interestingly, all three models share a unanimous preference for risk-averse decision-making tendencies. Across these LLMs, the average σ value exceeds 0, and even their minimum σ value remains greater than 0. The TCN model allows LLMs to exhibit diverse risk preferences, but despite this flexibility, our findings still reveal a unanimous preference for risk-averse behavior among LLMs, which is consistent with human behavior to a certain extent. ChatGPT leans towards conservative choices with higher rewards, as reflected in its higher risk aversion (σ) but it shows the lowest concern for potential losses (λ). With probability weighting parameter α > 1, it diverges from human norms. Conversely, Claude adopts a riskier approach, with lower risk aversion, but has higher loss aversion, and small-probability overweighting. Gemini balances risk and caution, exhibiting moderate risk-taking tendencies, loss aversion, and balanced probability weighting behavior. An uncommon greater-than-1 α suggests that ChatGPT would perceive unlikely events as even less likely than they are. This could have the following implications: (1) Dialogue Systems: It may produce more conservative responses, which might make it better suited for providing safe, predictable information.( 2) Content Generation: It may avoid rare scenarios, leading to content that aligns more with conventional or high-probability outcomes, which could reduce risk but also the potential for novelty and creativity. In addition to the context-free decision-making baseline, we investigate potential variations in decision patterns among LLMs based on embedded demographic features, including gender, age, education level, marital status, and living area. Given the existing literature highlighting biases within LLMs when encountering such features [15] , we introduce an augmentation technique that involves randomly incorporating other minority features into the basic demographic framework. These features encompass sexual orientation, disability, religion, race, and political affiliation. The augmentation of our demographic profiles results in 10 distinct socio-demographic groups as outlined in Table 2 . These personas provide a comprehensive representation of the diverse demographic landscape under examination. We first generate a distribution of demographic profiles by randomly assigning LLMs foundational demographic features and recording their answers. We then apply the real-world distribution across the countries to reflect realistic demographics of human communities and document the results, where the statistical data are from the World Bank dataset [36] .Result Comparisons with Context-free Evaluations Embedding demographic features resulted in notable changes in the decision-making behavior of the LLMs compared to their context-free evaluations, as illustrated in Figure 4 (raw data is available in the Appendix Table 6 ). Although there is no significant difference in parameters' values between the random and real-world distribution of demographic features for the three LLMs, we identify some patterns specific to each model.ChatGPT shifts towards much riskier decision-making with lower risk aversion (σ), while maintaining consistent loss aversion (λ). Notably, α shifts from small probability underweighting to overweighting, aligning more closely with typical human behavior. Claude shows stable risk preferences and probability weighting but reduced loss aversion, aligning more closely with human tendencies. Gemini becomes more conservative in risk-taking and shows heightened loss aversion with demographic embedding. Contrary to ChatGPT, this model shifts from small probability overweighting to underweighting with α slightly higher than 1.Main Findings from LLMs' responsesYounger Individuals (<25 years): Claude shows a significantly higher tendency to overweight small probabilities among younger users, while Gemini exhibits a higher loss aversion.Older Individuals (>55 years): Across all other models, this group has a minimal influence on parameters, with Claude again indicating a higher tendency to over-weigh.Female presents a significant reduction in risk aversion and probability weighting parameters by ChatGPT and Claude, respectively, suggesting that they may perceive higher risk associated with female demographics.Lower educational levels impact Claude with lower risk aversion while decreases loss aversion in Gemini.Claude may think married people are less likely to be risk-averse.Rural living significantly decreases probability weighting and increases loss aversion in Claude, implying environmental factors can influence its responses. to explore the determinants of decision-making behaviors under uncertainty, using σ, α, and λ as dependent variables and foundational demographic features as independent variables. We encode categorical features with binaries. For example, "Age < 25 years" is set to 1 for individuals younger than 25 years old and 0 otherwise. We then have the regression model below, where β n are coefficients for demographic variables and β 0 is the intercept. n denotes n-th feature, i denotes the i-th observation, and ϵ denotes the error offset in regression.RiskParameteri = β0 + βnDemographic ni + ϵi (9) (a) (b) (c) (d)Figure 5 : Average parameter values where regression coefficients are significant in this category group.Significant categories are marked in red.Table 3 highlights key findings from this regression study. As an example of features that have a significant impact on these parameters, we graph comparisons of the average parameter values in Age Impact and Education Level in Figure 5a and 5b. The coefficients are extracted from the complete regression as presented in Figure 6 . The raw data are also included in Table 7 in Appendix A. Incorporated with demographic features, the average parameters across all three models show distinct differences. Additionally, each model exhibits unique sensitivity to the features that affect its application in various demographic contexts.In the above examples, Claude demonstrates a broader sensitivity to demographic variables in both loss aversion and probability weighting, particularly for young and rural populations. It is also sensitive to education level and marital status. This model's extensive demographic responsiveness could enhance its adaptability in diverse settings but also introduce the risk of unfairness. For ChatGPT and Gemini, the analysis indicates a generally lower sensitivity to demographic variables, consistent across diverse user groups. ChatGPT exhibits a significant gender difference in risk preferences, in which females show reduced risk aversion than males. Gemini shows strong effects on loss aversion in the younger age group and lower education level group.b. Advanced Demographic Information: We incorporate more advanced demographic features to the OLS regression equation as the independent variables and have the regression model below:EQUATIONIn examining the additional advanced demographic sensitivities of LLMs, distinct patterns of behavior highlight their unique capabilities and potential biases. Table 4 highlights key findings from this In these two examples, ChatGPT exhibits a pronounced sensitivity to political beliefs in terms of loss aversion. This model also shows targeted sensitivity toward physically disabled individuals in probability weighting, being more likely to overweight small probability events. Meanwhile, Gemini displays a specific sensitivity to ethnicity, particularly in loss aversion for African and Hispanic groups. Unlike the other models, it maintains a politically neutral stance when making decisions.Sexual Orientation Claude shows significant sensitivity to the Homosexual, Asexual and Bisexual in risk preference levels, marked by positive coefficients, which indicates that it thinks these people are more risk-averse.ChatGPT shows a significant negative response in probability weighting for physically disabled individuals, suggesting a potential distortion of the weighting . Claude also considers the disabled more risk-averse.Claude and Gemini demonstrate significant responses to African in probability weighting, and Claude shows a notably lower loss aversion for Africans, Asians, and Hispanics.Gemini shows a significant decrease in loss aversion for Christians.Claude and ChatGPT show sensitivities in terms of loss aversion and probability weighting. Significant lower loss aversion presents in ChatGPT for supporters for Barack Obama, supporters of Donald Trump and lifelong Republicans. Claude shows negative effects on probability weighting of Donald Trump Supporters, and lifelong Republican. The comparison group for the analyses here is lifelong Democrats. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] . Justification: We release our code and data, with the instructions on exact command and enviroment.Guidelines: Justification: We address many aspects of the societal impacts in our evaluation approach. We offer future research directions to solve the drawbacks we discovered. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
discussion: 
 In the previous sections, we demonstrated various degrees of bias in the demographic-featureembedded LLMs. Given the presence of these biases across various models and demographic features, it is crucial to carefully consider the implications of embedding demographic features in LLMs and the unintended effects that may ensue.Implications for users: The potential biases with demographic features in LLMs are of significant concern, even for ordinary users. Companies are increasingly offering customized ChatBots, and users can personalize their models through open versions of LLMs. However, if users from minority groups customize a model expecting it to understand their personalities, it could lead to problematic outcomes, as LLMs might produce responses based on generalized stereotypes. For instance, these users might receive misleading advice on critical decisions like investments in financial decision-making. Users may need to consider more carefully when assessing decisions made by LLMs.Implications for developers and researchers: Previous research [15] has demonstrated that biases can be injected at various levels using different instructions, and our experiments provide evidence that injected demographic features can significantly affect how LLMs make decisions. Critical questions arise: should LLMs mirror human decision-making processes, including preexisting biases, or aim to satisfy ethical standards that remediate these flaws? How should LLMs perform when assisting in human decision-making? Prejudices objectively exist in human society, which are inevitably absorbed by the LLM from their datasets, training, and even aligning with Reinforcement Learning from Human Feedback (RLHF), with advancements such as Chain-of-Thought (CoT) prompting [35] , LLMs could potentially raise the bar for ethical decision-making by helping identify and correct biases through more structured reasoning. Moreover, LLMs that leverage these structured reasoning frameworks could potentially distinguish between decisions that reflect societal biases and those that follow ethical standards, prompting users to reconsider biased responses. If biases are corrected, methods such as targeted fine-tuning and RLHF adjustments could be used to realign the model's behavior toward fairness. However, a key question remains: in doing so, do we risk making LLMs less reflective of human-like behavior, and how do we balance ethical responsibility with the usability of these models in real-world applications?Our study presents intriguing discrepancies between human and LLM behaviors. For example, while human studies may not find significant differences in risk preferences among sexually minority groups [3] , our LLM results suggest otherwise. This discrepancy may result from two potential reasons:1. Flaws in Human Studies: Studies on minority groups often face privacy challenges in obtaining representative samples for their personal information. 2. LLM misunderstanding: LLMs may possess flawed understandings due to biases from data and training. If LLMs could reflect real-world trends more accurately from broader and cleaner data and more careful training, should researchers consider LLM outputs as supplementary evidence for social studies? This can only be answered when further studies can address these discrepancies between human and LLM data.One more thing: Our observation indicates that LLMs behave quite differently from their contextfree baselines after specifying demographic features or who LLMs are "pretending to be." Some might argue that LLMs should function as machines with an ensemble of human knowledge, particularly when set in a no-context situation, but it remains unclear what natural preferences LLMs should exhibit. We anticipate further research on defining what LLM behavior should entail when not embedded with human-demographic features, marking the boundary between a closed-loop answering machine and real artificial intelligence. Essentially, who are LLMs in their most fundamental form?Limitations: One limitation of this work is the difficulty in directly comparing LLM behaviors to humans due to the complex and sensitive nature of many demographic features, especially those related to minority groups, making it challenging for human subjects to process in studies. Additionally, while this study uses financial experiments to elicit overall preferences, exploring other domains with other experiments could provide a more comprehensive understanding of LLM behaviors. We recommend future work explore multiple models to compare model fit and robustness for a deeper understanding of LLM decision-making. 
conclusion: 
 In conclusion, our work establishes a foundational framework for evaluating LLM behaviors and opens avenues for future research aimed at aligning these models more closely with ethical standards and human values. We evaluated three LLMs regarding their capability to process decision-making decisions through three perspectives: risk, probability, and loss. While all models exhibit general tendencies akin to human behavior, each demonstrates unique deviations to an extent. The incorporation of socio-demographic features into our analysis revealed significant impacts of human-related features on the LLM decision-making process, underscoring the potential biases and variability in their outputs. By addressing the complexities and biases inherent in LLM decision-making, we can better harness their capabilities to support fair and equitable outcomes in real-world applications.  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] . Justification: We poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] . Justification: Our evaluation framework's model was adjusted in light of earlier research, and we have included citations for all relevant publications. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] . Justification: We offer the evaluation framework's scripts and sample dataset. These are the new assets that we assert. We provide all the necessary information. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this session, we will showcase the three series of multiple choice lists which contains 35 rows of lottery games designed and utilized in our framework. The first two series at Table 9 and Table 10 are primarily employed to assess the risk preferences and probability weighting parameters of the models. The last series at Table 11 introduces the concept of losing money and is thus mainly used to determine the loss aversion parameter. This structured approach allows for a comprehensive evaluation of the decision-making characteristics of LLMs under different financial scenarios.Option Consider the risk preferences and decision-making processes of a person with these characteristics.Table 13 : Demographic Feature Prompt Template Questions Demographic-feature added using Table 13 . We will show you two options for each lottery, and you will choose which option you want.For each lottery, each option will have different potential earnings, with a chance to earn, showing as a percentage under each option. Each of the selections will be independent, that is, for each lottery, your choice should be independent of the previous and following lotteries.Here are lotteries with options A and B. You can choose to play A or B and get the payment following the rules below.You can choose option A from row <1>to row <x1>, choose option B from row <x+1>to row 14. See Table 9 . Answer me with the value of <x1>only, please remember <x1>should be larger and equal to 1, less and equal to 13, do not explain.Demographic-feature added using Table 13 . Now let's play the second lottery. You can choose option A from row <1>to row <x2>, choose option B from row <x+1>to row 14.See Table 10 . Answer me with the value of <x2>only, please remember <x2>should be larger and equal to 1, less and equal to 13, do not explain.Demographic-feature added using Table 13 . Now let's play the last lottery. You will start with 10 dollars. You are going to play with this money. You can take it unless you lose in the lottery.And if you win we may add some to it. Here are 7 lotteries with options A and B. You can choose option A from row <1>to row <x3>, choose option B from row <x+1>to row 7.See Table 11 . Answer me with the value of <x3>only, please remember <x3>should be larger and equal to 1, less and equal to 6, do not explain. Table 14 : Prompt List for Demographic-feature Embedded Evaluation • The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.* Equal contribution 38th Conference on Neural Information Processing Systems (NeurIPS2024).xi ≿ * xj denotes that xi is directly revealed preferred to xj, meaning that xi is chosen over xj when both are affordable given the price vector pi.xi ≿ * * xj denotes that xi is indirectly revealed preferred to xj through a sequence of other choices, ensuring transitivity in preferences and preventing cyclical inconsistencies. 


Decision: After a review/rebuttal/discussion phase that included most of the reviewers we are recommending this paper be accepted for NeurIPS.

All reviewers agreed that the problem in the paper (evaluating decision behavior of LLMs) is an important and interesting one for AI broadly. They also agreed that the paper is clearly written, novel, and is well placed in the literature.

There were some concerns about the comprehensiveness and completeness of the proposed framework, specifically, that the chosen set of measures are not fully justified/comprehensive. We urge the authors to addresses these comments in the final by arguing more directly about both of these properties. 

However, even given the above, the paper does a solid job of articulating and testing the proposed evaluation framework. So while there is always space to be more comprehensive and more general, the steps taken in the paper are a significant first step. We hope the authors will address the reviewers comments on shortcomings in the final version.

Reviews: review: paper_summary: paper_summary: This paper proposes a framework to evaluation the decision making behavior of LLMs. It focuses on ChatGPT4Turbo, Claude3Opus and Gemini1pro. The results show that LLMs exhibit patterns similar to humans. Impacts of socio-demographic features are also analyzed and the results show that different LLMs can vary from each other.
strength_weakness: strength_weakness: - While the problem is interesting, the technical contribution of the paper seems limited. The measures chosen are also not fully justified. It would be important to highlight the novelty of the results. 
- Following the above comment, the examination does not appear comprehensive. The reviewer worries that the questions may only cover a small set of problems. As a result, the conclusions drawn might contain significant randomness.
questions: questions: - The problem in consideration is important. The findings appear interesting. 
- While the problem is interesting, the technical contribution of the paper seems limited. The measures chosen are also not fully justified. It would be important to highlight the novelty of the results. 
- Following the above comment, the examination does not appear comprehensive. The reviewer worries that the questions may only cover a small set of problems. As a result, the conclusions drawn might contain significant randomness.
limitations: limitations: Please see the above.
paper_summary: paper_summary: In this paper, the paper the authors model the decision-making behavior of certain open-source LLMs on lottery selection tasks. They further extend this work, by repeating similar analysis after priming the models with demographic information. Results indicate that the different LLMs differ in their risk profiles as inferred by the parameters of the evaluative model. Some prominent differences also emerge on demographic-primed LLMs. The overarching theme that comes through is the need for caution when using LLMs in practice, especially as analogue to human decision-making.

**Conclusion:** Overall, while the paper considers an interesting problem, the results don’t appear to yield any general insights, and the methodology seems to make assumptions that may be difficult to justify. The paper can still be an alright addition to the conference since it’s timely and touches on an important problem.
strength_weakness: strength_weakness: 1. It’s unclear whether the TCN model is actually apt in this setting. For example, it seems to assume deterministic values of parameters like sigma, alpha and lambda, and then tries to infer them based on the LLMs decisions. However, what if these parameters are themselves random variables? Does it make sense to assume the LLM has a fixed risk profile across trials? As an analogy, assuming the coin has a fixed bias p, and estimating it via repeated tossing experiments when p is sampled from a uniform distribution each trial. 

2. The utility function was a little unclear to me (line 135). Why does it not cover the entire (x,y) domain? w(p) simplifies to 1/p for alpha = 1. Should it be p? As a tangentially related point, consider numbering the equations. 

3. In Step 4 (section 4), can we be certain the estimation algorithm actually converges? This relates to 1, so that the convergence might not be guaranteed if the latent parameters are not fixed.

4. While the qualitative results are sensible — due diligence is needed before LLMs can be entirely trustworthy, the quantitative results don’t impart any deep or actionable insights. Observations like a given version of a pre-trained model tends to show risk aversion doesn’t quite generalize.
questions: questions: NA
limitations: limitations: NA
paper_summary: paper_summary: This paper presents a novel framework for evaluating the decision-making behavior of large language models (LLMs) under uncertain contexts, grounded in behavioral economic theories. The authors conducted experiments to estimate risk preference, probability weighting, and loss aversion for three commercial LLMs: ChatGPT-4.0-Turbo, Claude-3-Opus, and Gemini-1.0-pro. The study further explores the impact of embedding socio-demographic features on the decision-making process of these models, revealing significant variations and potential biases. The paper concludes with a call for the development of standards and guidelines to ensure ethical and fair decision-making by LLMs.
strength_weakness: strength_weakness: 1. The paper does not provide a detailed analysis of the potential causes behind the observed variations in LLM behavior when demographic features are introduced, which could be crucial for understanding and mitigating biases.
2. The study's focus on three commercial LLMs may limit the applicability of the findings to some open source LLMs like LLaMA.
3. The authors should discuss the references like chain-of-thought (CoT) prompting and reasoning in LLMs, such as "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" (https://arxiv.org/abs/2201.11903), "Automatic Chain of Thought Prompting in Large Language Models" (https://arxiv.org/abs/2210.03493), and "Keypoint-based Progressive Chain-of-Thought Distillation for LLMs" (https://arxiv.org/abs/2405.16064). These works could provide additional context and depth to the understanding of LLM decision-making processes.’
questions: questions: Please see the Weakness.
limitations: limitations: Yes
paper_summary: paper_summary: This paper proposes a comprehensive framework to evaluate LLM's decision-making behavior under uncertain contexts. By leveraging computational models from behavioral economics, as well as the experiment paradigms, the authors investigated the risk preferences, probability weighting, and loss aversion of LLMs from the parameters in the models. Finally, by injecting demographic information into LLMs, the authors found significant differences in those parameters measured from behaviors, suggesting potential biases from demographic information (e.g., age, education, or gender) on decision-making tendencies.
strength_weakness: strength_weakness: - __Lack of human data__. As the authors have mentioned in the limitation section, it would be fun to investigate the LLM vs. Human behavior at the parameter level of computational models. These parameter differences might explain why humans and LLMs are different or similar on a deeper level.

- __Lack of model selections__. The authors ONLY use one model (TCN model) to evaluate the LLM's behaviors. In computational cognitive sciences and psychology (Wilson, Collins, 2019), it is common to propose multiple cognitive models to test which model might best explain the behaviors of humans (and of machines). The authors could strengthen the model evidence by proposing multiple competing models, comparing models, showing goodness-of-fitting, and even possibly simulation and parameter recovery to show the robustness of the currently selected model. Perhaps there might be better models that can explain the behaviors better and thus, the attribution of behaviors may change.

- __More investigations on LLMs are recommended__. The authors list one of the contributions as investigating multiple LLMs in this framework. The LLM selections are the most known commercial LLMs (like GPT-4, Claude, and Gemini). These model comparisons somehow make sense but two of them are not open-sourced and the training data are not transparent. If the authors could compare some LLMs with a more transparent basis to provide insights into LLMs, this would strengthen the contribution of the paper. For example, the authors could investigate the relationship between model size and the parameters measured from the behaviors. Choosing models within the same family, and trained on similar datasets would make comparisons more informative. Looking at prompt engineering, the way of instruction-tuning or model structure (e.g., encoder-decoder model or decoder-only model) can also make sense while making this framework provide more information.
questions: questions: - One interesting finding from the paper is that human studies may not find significant differences in risk preferences in sexually minority groups but LLMs do respond differently under these settings. There might be two but completely different reasons: first, the authors did not get the full knowledge of the literature in that field and the paper listed there may be only one study that reports non-significant research. The other reason could be intriguing: the studies about one specific topic are limited but humans are generally exhibiting such effects which are overlooked in empirical study. However, LLMs are more likely to learn such effects from massive datasets and thus exhibit effects. So there might be a reverse approach that understanding human behaviors (especially in fields that are overlooked) from LLM's behavior. The authors are not required to formally give an answer to this question, but I find it interesting in public discussion about such potentials.
limitations: limitations: As indicated above, the major limitations of the paper are not providing a deeper insight into some human cognitive or LLM mechanistic findings, though this framework is valuable and more comprehensive than previous work.

Comments: title: Update after authors' rebuttal
Reviewer - Author Discussion
comment: Dear Reviewer,

Thank you for your positive feedback and for recognizing the generalizability of our methodology. We appreciate your support and are glad that our clarifications were helpful. Thank you for maintaining your acceptance decision!
I appreciate the authors taking the time to offer clarification and answer a couple of of the questions I had. I agree that even though the authors only consider a few popular models, their methodology generalizes irrespective of whether the specific findings around a given LLM may or may not. I continue to believe that this is a nice interdisciplinary paper, and I am maintaining my acceptance decision along with the score.
We appreciate your positive evaluation and are glad our rebuttal addressed your concerns. Thank you for your thoughtful feedback and support. We look forward to contributing to further research in this area.
Thanks to the authors for the rebuttal. I think the comments well addressed my concerns and positively engaged in discussions. My original evaluation is to have this paper present in NeurIPS and thus I will maintain my current evaluation, given the constraint of the overall scope. However, I do think this paper has a lot of interesting points and may raise more valuable research in the future.
Thanks everyone for their hard work on the papers, reviews, and rebuttals. We now have a comprehensive rebuttal from the authors which responds both overall and to each review.

I'd please ask the reviewers to please post a comment acknowledging that they have read the response and ask any followup questions (if any).

This period is to be a discussion between authors and reviewers (Aug 7 - Aug 13) so please do engage now, early in the window, so there is time for a back and forth.

Thanks!