Exploring the Sensitivity of LLMs' Decision-Making Capabilities: Insights from Prompt Variations and Hyperparameters
Authors: Manikanta Loya
Divya Anand Sinha
Richard Futrell
Venue: EMNLP 2023 Findings (2023)
Keywords: Language Models
Decision-Making
Cognitive Psychology
Chain of Thought
TL;DR: This study shows that Large Language Models' decision-making abilities vary based on prompts and hyperparameters, contrary to previous findings.

Abstract: The advancement of Large Language Models (LLMs) has led to their widespread use across a broad spectrum of tasks, including decision-making. Prior studies have compared the decision-making abilities of LLMs with those of humans from a psychological perspective. However, these studies have not always properly accounted for the sensitivity of LLMs’ behavior to hyperparameters and variations in the prompt. In this study, we examine LLMs’ performance on the Horizon decision-making task studied by Binz and Schulz (2023), analyzing how LLMs respond to variations in prompts and hyperparameters. By experimenting on three OpenAI language models possessing different capabilities, we observe that the decision-making abilities fluctuate based on the input prompts and temperature settings. Contrary to previous findings, language models display a human-like exploration–exploitation tradeoff after simple adjustments to the prompt.

Exploring the Sensitivity of o Decision-Making Capabilities: Insights from Prompt Variation and Hyperparameters
Manikanta Loya* manikanl@uci.ee edu
Divya Anand Sinha" dasinha@uci. edu University of California, Irvine
Richard Futrell rfutrell@uci.ce
Abstract
However, such work has not always paid due tention the fact that LLM responses can highly variable and sensitive to the details of the prompt used and to hyperparameters such as temperature. Limited interactions with LLMs-such as interac- tions using only one prompt-can be misleading (Bowman, 2023). In this work, we follow up on the behavioral experiments conducted by Binz and Schulz (2023), who studied LLMs' decision mak- ing using analogues of number of well-known human experimental paradigms, finding strong di- vergences from human behavior. However, the experiments in the previous work used only one prompt per task, and did not study the effects of hyperparameters. We adopt the same task as the previous work, but systematically vary prompts and temperature.
The advancement Large Language Mod- els (LLMs) has led their widespread use across a broad spectrum of tasks, including decision-making. Prior studies have compared the decision-making abilities of LLMSs with those of humans from a psychological perspec- tive. However, these studies have not always properly accounted for the sensitivity of LLMs behavior to hyperparameters and variations in the prompt. In this study, we examine LLMs performance on the Horizon decision-making task studied by Binz and Schulz (2023), analyz- ing how L.MSs respond variations in prompts and hyperparameters. By experimenting on three OpenAl language models possessing dif- ferent capabilities, we observe that the decision- making abilities fluctuate based on the input prompts and temperature settings. Contrary to previous findings, language models display human-like apploration-expliiaatio tradeoff after simple adjustments to the prompt."
Our aims are both substantive we seek find whether, with basic changes to the prompt, mod- cls show human-like behavior in these decision making tasks-and methodological: we wish to emphasize that psychological LLM research must consider variability function of prompt and hyperparameters.
1 Introduction
The recent success of large language models (LLMs) at variety of tasks has led to curios- ity about their cognitive abilities and characteris- tics. As LLMs are increasingly integrated in daily life both as conversation partners and economic decision-makent (Munir et al., 2023; Chaturvedi et al., 2023; Yang et al., 2023), such studies are necessary for understanding the limits and char- acteristics of such agents. An understanding of LLMs at psychological level may also provide strategies for improved prompting and training. To this end, number of researchers have recently adopted methods from cognitive psychology and behavioral economics to evaluate language models in the same way that humans have been evaluated (e.g. Linzen et al., 2016; Miotto et al., 2022; Phelps and Russell, 2023, among many others).
Background and Related Work
As mechanistic understanding and control of LLMs remains complex, researchers have increasingly adopted methods from human behavioral sciences for characterizing LLMs' behavior: in the same way that the human brain is largely a black box that must be probed using experimental methods and constructs, LLMs may be studied in the same way. In addition to studies that have used the meth- ods of cognitive psychology to understand LLMS' reasoning and grammatical abilities (c.g., Linzen et al., 2016; Futrell et al., 2019; Cai et al., 2023), searchers have increasingly adapted methods from psychometrics (Miotto et al., 2022; Bodroza et al., 2023; Abramski et al., 2023), which seek to charac- terize LLMs in terms of personality variables such
"Equal Contribution
*Code is available at the following github link.

---

are now regarded as best practices for such tasks, for example the use of Chain-of-Thought (CoT) prompting (Wei et al., 2022) the original prompt forces the LLM to choose a machine in the next token generated, without deliberation. Below, we investigate the behavior of LLMs on this task under systematic variations of temperature and prompt.
You are going casine that owns two sot machines, You carn money each time pla these machines.
You have resiived the felloing amount dollars when playing in paet:
frlcchine delivered 51 dollars..
- RMlcchin derivered zo dallars.
Mrachine delivered dollars. MMchinne delivered
Your goal is maximbze sum of recelved dollars within additional rounds.
macchm At Machine
O Which machine do you choose?
3 Horizon Task Experiments
Figure Original Horizon task prompt (Binz and Schulz, 2023).
The Horizon Task as shown in Binz and Schulz (2023) is a special case of the Multi-Armer bandit (MAB) setting. MAB problems (Sutton and Barto, 2018, Ch. 2) are one of the common problems in the area of Reinforcement Learning. This game involves an agent interacting with a slot machine possessing h arms, Each arm the agent pulls has a reward associated with it defined by underlying probability distribution. This game is played over multiple episodes with the goal of maximizing the accrued rewards.
as agrecableness and conscientiousness, and meth- ods from behavioral economics (Cartwright, 2018; Phelps and Russell, 2023; Horton, 2023), which characterize LLMs' decision-making in terms of preferences for risk and reward.
Prior research on prompting techniques (Wei et al., 2022; Wang et al., 2023) has shown that sub- tle modifications in input prompts can lead to varied outcomes in reasoning tasks (Cobbe et al., 2021). Srivastava et al. (2022) revealed that Large Lan- guage Models (LLMs) are notably susceptible to the precise wording of natural language questions, especially when presented a multiple-choice set- ting. In a recent study, Ouyang et al. (2023) empha- sized the influence of temperature adjustments on LLM's performance in code generation tasks. Un- like previous studies, our research delves into the sensitivity of LLMs concerning economic decision- making abilities.
One approach involves persistently selecting the arm that has delivered the maximum amount of re- wards in the past. An alternative strategy involves thorough exploration of all arms discern their respective underlying probability distributions, fol- lowed by the selection of the arm with the highest potential for reward. While this feasible, every turn spent discerning the underlying distribution, diverts from the primary goal of reward maximiza- tion. The former and latter strategies are known exploitation and exploration respectively, and the exploration exploration-cxxllotttioo exploitation dilemma is fundamen- tal concept in decision-making that arises in many domains.
Our work is a focused followup on Binz and Schulz (2023), investigating the sensitivity of one of their results to changes in prompt and hyper- parameters. Binz and Schulz (2023) evaluated on decision-making information search, delibera- tion, and causal reasoning text-davinci-O0 (Brown et al., 2020) by presenting it with prompts such as the one shown in Figure 1. We follow up on the tasks from the information search area, instanti- ated in the Horizon task, described in the following section. In this task, humans show characteristic trade-off of exploration and exploitation (Wilson et al., 2014), favoring exploration in early trials and exploitation later, whereas Binz and Schulz (2023) find that LLMs do not.
To explore the extent to which humans use these strategies, Wilson et al. (2014) reports experiments where participants were asked to play the Horizon Task. This task consists of a set of two-armed ban- dit problems, where participants are presented with two options, each associated with noisy rewards. The task comprises either five or ten trials, and in each trial, participants must select one option, receiving corresponding reward feedback. In the initial four trials of the task, participants have only one option and are provided with the corresponding reward feedback. These forced-choier trials create two distinct information conditions: "unequal in- formation" and "equal information." In the unequal information condition, one option is played three times, while the other option is played only once. In the equal information condition, both options are
The results of Binz and Schulz (2023), however, are based on single prompt and setting, limiting the generality of their results. Furthermore, ob- serving the Horizon task prompt (and the others used throughout the paper), does not follow what

---

random
random
random
Hofzon Honzon
0.0
Moan regret
Meaa regret
1.0
2-
5
Trials
Trials
Trials
text-davinci-002
(b) text-davinci-000
(c) 5pt-3.5-turbo
Figure 2: Mean regret obtained in the Horizon (multi-trial multi-armed bandit) task by humans and LLMs with varying temperature, using the prompt from The solid black line indicates human performance; others L.MMs. Eror bars show the standard error of the mean.
Quasi-CoT
Q: Which machine do you choose? At the end, summarize your output as "So, chose Machine [machine name]".
Q: Which machine do you choose?
A: Thinking step by step chose Machine
A: Let's think stop by stop
CoT-exploit
CoT-explore
The following hints will help you make good decision:
The fallowing hints will help you make good decision:
1. In each round you choose elther Machine or Machine J and receive reward from that machine.
1. In each round you choose elther Machine or Machine and receive reward from that machine.
2. You must choose machine with highest average of delivered dollars.
3. Average 1, 2, calculated first by computing sum all observations which
2. If you have more than rounds left, you choose machine lesser number of past observations.
dividing by number observations
6/3 =.
positive integer then
higher.
Your maximize the received dollars
Your goal is maximize the of received dolllars within additional round. Which machine do choose?
within additional round.
Q: Which machine do you choose?
A: Let's think step by step
A: Let's think step step
Figure 3: Modifications in prompt for the Horizon task. Horizon prompt is shown. In of CoT, CoT-Exploit & CoT-Explore we explicit ask the model to summarize its choice at the end by appending the entire prompt with "Answer the following question and summarize your choice the end as "Machine[[machin Laaaiieenaaaela____aa at the beginning.
played twice. The five-trial setting denoted Hori- zon 1, indicating that participants make decisions only once, while the ten-trial setting is referred to as Horizon 6, as participants make decisions over six rounds.
man behavior favors exploitation in Horizon 1, but a gradual shift from exploration to exploitation Horizon
3.1 Varying Temperature
The impact of various temperature settings (0.0, 0.0.0.5,1000 tested is illustrated in Figure 2. It is clear that the behavior of each model, as indicated by the mean regret line, differs according the temper- ature. For Horizon 1, the lowest regret is ob- tained for temperature zero across all three models. Further, unlike text-dave text-davinci-002 and as shown in Binz and Schulz (2023), the mean regret is
Binz and Schulz (2023) applied this experimen- tal design to language models using the prompt in Figure 1, and we follow their experimental setup exactly except for variations to the prompt and hyperparameters. The performance of LLMs is assessed by measuring the mean regret across mul- tiple runs. The regret is defined the difference between the optimal reward, which corresponds to the machine with the higher reward, and the actual reward obtained from the selection process. Hu-
all three OpenAl models2

---

Fandolf
Fanddom
random
Herizon
Herizon
Humans
Without Col
Moan regret
Moan regrat.
CoT
Quasi CoT
2
Trials
(a) text-davinci-802"
(b) text-davinci-002
(c) g.t--..5ttuute
Figure 4: Mean regret obtained by humans and LLMs on the Horizon task, varying prompt. 'Quasi-CoT" means prompt of the form 'Thiniing step-by-step. I choose Machine ... which does not enable true chain-of-thowe th-uuht reasoning. The temperatures for GPT-2, GPT-3, and GPT-3.5 are 1.0, 0.5, and 1.0 respectively. These temperatures show the greatest learning effect (negative slope) in the Horizon 6 task.
lower than humans for both text-davinci- and gpt-3. 5-turbo. In the case of Horizon there is notable rise in the inital mean regret, suggesting that higher temperatures result in subop- timal decision-making However, increasing tem- perature demonstrates more pronounced learning effect, evidenced by greater negative slope.
text-davin siet-dvvinci--i n0i022 we find that altered prompts yield the human-likk negative slope, indicating an exploration-explotaiion trade-off.
3.3 CoT Prompting with Hints
To overcome the identified limitations in LLMs, such as their inaccuracies in computing averages (Razeghi et al., 2022; Imani et al., 2023) and sub- optimal exploration capabilities, we introduce ad- ditional hints within the input prompt to guide the pecision-making process. Specifically, we de- signed two prompts, namely CoT-Exploit and CoT- Explore, which aim facilitate explicit exploita- tion and exploration. The hints associated with these prompts are shown in Figure 3.
3.2 Varying Prompt
To encourage deliberation during incorporate variations in the input prompt. Specifically, we explore two different variants of the Chain of Thought (CoT) prompting technique (Wei et al., 2022)-CoT and Quasi-CoT. In Fig- ure 3, we illustrate the modifications made to the original prompt. The variant referred to as Quasi- CoT utilizes the prompt "Thinking step by step choose Machine", which forces the machine to make decision before fully processing its reason- ing. On the other hand, the CoT variant makes decision only after fully processing its reasoning. The Quasi-CoT condition allows us to disentangle the effects true step-by-step reasoning in CoT from the effects of prompting the LLM to think carefully.
In the CoT-Exploit prompt, we instruct the model to base its decisions on the average of observed experiences and equip it with the required mathematical calculations to make a decision. Likewise, in the CoT-EExllore approach, we explicitly direct the model to select machine with lower frequency among the observed experiences.
The alteration in the behavior of LLMs due to changes in the input prompt is depicted in Figure 4. Across all models, CoT demonstrates lower-regret compared both Quasi-CoT and the original prompt, whereas Quasi- performs worse than original prompt. Furthermore, even in
The performance of gpt-3. 3.5--.rbb.. using var- ious CoT prompting variants, is compared in Fig- ure 5. As anticipated, CoT-Exploit outperforms CoT-Explore, displaying consistent decrease in slope. However, CoT--Exllre performs signifi- cantly worse than random decision-making CoT- Explore primary concentrates on getting more in- formation about each machine rather than overall rewards.
'Experiments with text-davinci-e50 using CoT prompt failed due to its inability to summarize its choice at the end.

---

and Schulz (2023), is highly sensitive to the way these LLMs are queried. The non-human-lik, be- havior observed by Binz and Schulz (2023) van- ishes under simple variations of prompt, and super- human performance in terms of minimizing regret is easily achievable. Going forward, we urge care- ful consideration in the LLM psychology literature of the fact that model behavior can diverge under different settings.
Horizon
CoT
Horizon 6
CoT-Exploit
Humans
CoT-Explore
Quasi CoT
10
Mean regret
6
5 Limitations
We have presented focused extension of one of the studies from Binz and Schulz (2023), demon- strating sensitivity to prompt and hyperparameters which was overlooked in the previous work. How- ever, our work is limited in that (1) we have only examined one of the tasks from Binz and Schulz (2023), (2) we have only presented a few variations of temperature and prompt, and (3) we have only experimented with some of the models available to us as of June 2023, selecting high-profile closed- source models over open-source models. Never- theless, we believe that our overarching point that LLM psychology needs to take into account hy- perparameters, prompts, and variability remains valid.
2
2
3
5
Trials
Figure 5-trrb-- behavior under different variants of CoT prompts temperature I.0.
3.4 Discussion
Through our experiments, we have discovered that the decision-making capabilities of LLMs are in- fluenced by both the prompts used and the tem- perature settings, more by the choice of prompt rather than the temperature. This highlights the importance of varying prompts to clicit the desired behavior from LLMs during decision-making tasks, and that studies which have used only one kind of prompt are potentially misleading.
Ethics Statement
This work involves psychological studies of LLMs in economic decision making contexts. If LLMSs really deployed economic decision makers, then ethical issues could result from biases and limitations of the models. We urge caution in such applications.
Intriguingly, we observed that the model gpt-3. 11133.5-turbe 5-turbo with the Quasi-CoT prompt (Fig- ure 5) exhibits the closest resemblance to human behavior. This prompt alerts the model to the need for reasoning, but does not give it the space to actu- ally perform any reasoning. The similarity of the Quasi-CoT result to humans suggests that humans may also struggle fully process the associated information and reasoning.
References
Katherine Abramski, Salvatore Citraro, Luigi Lombardi. Giulio Rossetti, and Massimo Stella. 2023. Cognitive network science reveals bias in gpt-3, chatgpt, and gpt-4 mirroring math anxiety in high-school students. arXiv preprint arXiv:2305. 2
Furthermore, by providing hints to guide the decision- makinn process, we have observed that su- perhuman performance can be achieved, as demon- strated by the CoT-Exploit variant (Figure 5). This result suggests that language model behavior in these tasks is potentially highly controllable.
Marcel Binz and Eric Schulz. 2023. Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences. (000))22218523112
Bojana Bodroza, Bojana Dinic, and Ljubisa Bojic. 2023. Personality testing of gpt-3; Limited temporal reliability, but highlighted social desirability of gpt- 3's personality instruments results. arXiv preprint arXiv:2306.0433
4 Conclusion
Samuel R Bowman. 2023. Eight things know about large language models. arXiv preprint arXiv:2304.00611.
We have demonstrated that the psychological be- havior of LLMs, as previously explored by Binz

---

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 1877-1901.
Shuyin Ouyang, Jie M. Zhang, Mark Harman, and Meng Wang. 2023. Lim is like a box of chocolates: the non- determinism of chatgpt in code generation.
Steve Phelps and Yvan Russell, 2023. Investigating emergent goal-like behaviour in large language mod- els using experimental economics. arXiv preprint 2rXiv:3305.79770
Zhenguang G Cai, David A Haslett, Xufeng Duan, Shuqi Wang, and Martin J Pickering. 2023. Does ChatGPT resemble humans in language use? arXiv preprint 233iv22333.80/14
Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022. Impact of pretraining term frequencies on few-shot numerical reasoning. In Findings of the Association for Computational Lin- guistics: EMNLP 2022. pages 840-854, Abu Dhabi, United Arab Emirates. Association for Computa- tional Linguisties.
Edward Cartwright. 2018 Behavioral Economics. Routledge.
Rijul Chaturvedi, Sanjeev Verma, Ronnie Das, and Yo- gesh K Dwivedi. 2023. Social companionship with artificial intelligence: Recent trends and future av- enues. Technological Forecasting and Social Change, 193: 122644.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint orXiv:2206.04611
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word prob- lems.
Richard Sutton and Andrew G Barto. 2018. Reinforce- ment Learning: An Introduction, MIT press.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models.
Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pages 32-42, Minneapolis, Minnesota. Association for Computational Linguistics.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arX 19903.
Robert C Wilson, Andra Geana, John M White, Elliot A Ludvig, and Jonathan D Cohen. 2014. Humans use directed and random exploration to solve the explore- exploit dilemma. Journal of Experimental Psychol- ogy: General, 143
John Horton. 2023. Large language models as sim- ulated economic agents: What can we learn from homo silicus? arXiv preprint 33X
Shima Imani, Liang Du, and Harsh Shrivastava. 2023. MathPrompter: Mathematical reasoning using large language models. In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track), pages 37- 42, Toronto, Canada, Association for Computational Linguistics.
Hui Yang. Sifu Yue, and Yunzhong He. 2023. Auto-gpt for online decision making: Benchmarks and addi- tional opinions. arXiv preprint arXiv:2306.0222
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of Istms to learn syntax- sensitive dependencies. Transactions of the Associa- tion for Computationna Linguistics, 4:521-535.
Marilù Miotto, Nicola Rossberg, and Bennett Klein- berg. 2022. Who gpt-3? an exploration of per- sonality, values and demographics. arXiv preprint arXiv:2209.14338
Iqbal Munir et al. 2023. Artificial intelligence chatgpt in medicine. can it be the friend you are looking for? Journal of Bangladesh Medical Association of North America (BMANA) BMANA Journal, pages 01-04.


Reviews:
[{"review_id": "zxE6QOZ8Sy", "reviewer": "Reviewer_zACN", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">Poorly motivated direction: I am a bit unsure why minor alterations to temperature and prompting elucidate the “psychological behavior of LLMs” as the authors claim. I feel as though this research could have been more ambitious—simply demonstrating that the use of a slightly different prompting approach alters model outputs does not feel to be a substantive finding (especially given the numerous other research exploring model’s sensitivity to prompting strategies in more expansive domains and using much more robust methodology)\n\nRegarding motivation, a more ambitious and extensive study would certainly be worthwhile, but we believe the present contribution is enough for a short paper making a focused contribution. Our motivation is to show that cognitive behavior varies according to prompt and temperature settings – indicating that tests related to these tasks should consider prompt and temperature before drawing conclusions.\n\n>The authors do not provide details necessary to judge the robustness of their experiments: number of trials, reasoning for temperatures chosen, reasoning for development of Quasi-CoT prompt.\n\nThe number of trials is 6, following the setup from Binz and Schulz (2023) exactly. We will add this to the paper. The motivation for Quasi-CoT is to disentangle the effect of real CoT prompting from any effect of simply instructing the model to reason carefully. \n\n\nWe will add further discussion and background on multiple bandits, basing our review on Sutton & Barto (2018).\n\n>Missing citations for key claims made; for example:\nln. 94-96: “In this task, humans show a characteristic trade-off of exploration and exploitation, favoring exploration in early trials and exploitation later…”\nln. 179-181: “...the identified limitations in LLMs, such as their inaccuracies in computing averages and sub-optimal exploration capabilities…”\n\nWe will cite Wilson et al. (2014) for the claim on human psychology and Razeghi et al. (2022) for the lack of arithmetic accuracy in LLMs.\n\nWe will address all the grammar and presentation issues mentioned.\n\n**References**\n\n1. Sutton, Richard S., and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018.\n2. Wilson RC, Geana A, White JM, Ludvig EA, Cohen JD. Humans use directed and random exploration to solve the explore-exploit dilemma. J Exp Psychol Gen. 2014 Dec;143(6):2074-81. doi: 10.1037/a0038199. Epub 2014 Oct 27. PMID: 25347535; PMCID: PMC5635655.\n3. Razeghi, Yasaman, et al. \"Impact of pretraining term frequencies on few-shot numerical reasoning.\" Findings of the Association for Computational Linguistics: EMNLP 2022. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors explore an interesting intersection of language modeling and human psychology. ", "reasons_to_reject": "Major weaknesses\n* Poorly motivated direction: I am a bit unsure why minor alterations to temperature and prompting elucidate the “psychological behavior of LLMs” as the authors claim. I feel as though this research could have been more ambitious—simply demonstrating that the use of a slightly different prompting approach alters model outputs does not feel to be a substantive finding (especially given the numerous other research exploring model’s sensitivity to prompting strategies in more expansive domains and using much more robust methodology)\n* The authors do not provide details necessary to judge the robustness of their experiments: number of trials, reasoning for temperatures chosen, reasoning for development of Quasi-CoT prompt\n* Related works does not address multi-armed bandit setups which are critical for comprehension of methods\n* Missing citations for key claims made; for example:\n* 1) ln. 94-96: “In this task, humans show a characteristic trade-off of exploration and exploitation, favoring exploration in early trials and exploitation later…”\n* 2) ln. 179-181: “...the identified limitations in LLMs, such as their inaccuracies in computing averages and sub-optimal exploration capabilities…”\n\n\nMinor weaknesses\n* Multiple grammatical errors and confusingly written at times\n* Figure 3 is difficult to parse and contains major grammatical errors in the LM prompts\n* The description of the Horizon task (described in Section 3) is difficult to parse. More precise language and motivation would really help better elucidate the Horizon task’s setup and utility. \n\n\nThe authors may want to consider reframing the work to include results for a different set of psychology benchmarks (ones that previous work has utilized in tandem use with LLMs) and demonstrate that conclusions vary as a function of the hyperparameters (i.e. showing the challenges with reproducibility when working with LLMs). ", "questions_to_authors": "", "paper_topic_and_main_contributions": "In this paper, the authors study the role hyperparameter and prompt tuning plays on the Horizon decision-making task–a pairwise comparison task in which agents operate under either “unequal information” or “equal information” settings–from Binz and Schulz (2023). The authors recreate the experiments from the paper and alter the original prompts and temperature of the tested LLMs. They find that models performance varies across different prompting and temperature setups. \n", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Although there were many presentation errors/grammatical errors/typos; a few were:\n* ln 29: “such” has no referent\n* ln 58: missing “the” before “prompt”\n* ln 57-63: very hard to parse \n\n\nFigure 3, a few examples are: \n* lesser → fewer\n* summarization of output template is not a complete sentence\n* choose → will choose\n* computing sum → computing the sum\n\namong many others…\n"}}, {"review_id": "FvJ1ZHEqX6", "reviewer": "Reviewer_57ap", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Other factors in terms of prompt engineering can also play a role and hard to justify and do an exhaustive study on all variants. But more discussion on that will be helpful.\n\nOur main motivation is to show that cognitive behavior varies according to prompt and temperature settings – indicating that tests related to these tasks should consider prompt and temperature before drawing conclusions. A more thorough study would certainly be worthwhile, but we believe the present contribution is enough for a short paper making it a focused contribution. We will make sure to investigate other factors that might influence the results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- psychologically guided prompt variants are interesting.\n- comprehensive experiments ", "reasons_to_reject": "- Other factors in terms of prompt engineering can also play a role and hard to justify and do an exhaustive study on all variants. But more discussion on that will be helpful. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper investigate on LLM's performance on the Horizon decision-making task, analyszing how LLMs respond to variations in prompts and hyperparameters.  LMs display a human-like exploration-exploitation tradeoff after simple adjustments to the prompt\n\nThe hyperparameters explored include:\ntemperature (0, 0.5, 1) and two variants of the Chain of Thought prompting technique (CoT and Quasi-CoT).\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9IRAjMm5Zy", "reviewer": "Reviewer_AUjq", "summary": "", "questions": "A) Could you please explain why in Figure 2, the Horizon 1 and Horizon 6 tasks have a huge difference under \"Trial 1\"", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Could you please explain why in Figure 2, the Horizon 1 and Horizon 6 tasks have a huge difference under \"Trial 1\"?\n\nIn Figure 2, we believe the Horizon 1 and Horizon 6 tasks have a large difference under \"Trial 1\" because of the exploit-explore tradeoff. The prompt differs between the tasks (number of additional rounds left) and we'll make sure to show the differences in the appendix. In the Horizon 1 task, because the prompt explains there is only one trial, the LLM favors exploitation rather than exploration. In the Horizon 6 task, the prompt is as in Figure 1, so exploration is preferred.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Accepting it would allow the community to be more aware about considering the temperature and prompting strategies in their LLM research.", "reasons_to_reject": "I dont see any reason to reject this paper", "questions_to_authors": "A) Could you please explain why in Figure 2, the Horizon 1 and Horizon 6 tasks have a huge difference under \"Trial 1\"", "paper_topic_and_main_contributions": "The Paper explores how different LLMs behave under different temperature and prompting strategy settings. They focus on the exploration-exploitation capabilities of these settings and compare it with human behavior.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "G5Yl5Acvbp", "reviewer": "Reviewer_nAFL", "summary": "", "questions": "(1) What is the temperature in the experiments shown in Fig 4; and why did you choose that value for this experiment? [I'd recommend adding this information to the paper]\n\n(2) Can you comment on what should be a good strategy regarding the choice of temperature?\n\n(3) is there a specific reason you did not study other tasks from Binz and Schulz (2023)?\n\n(4) Why did you choose the Horizon decision-making task? i.e. instead of any of the other tasks in Binz and Schulz (2023)?\n\n(5) is there a repo (even if it consists of simple scripts) with the code to reproduce the results (and download the datasets, etc)? \n\nI understand that such a repo would require the users to have access to OpenAI APIs. But I also guess it could easily be adapted to e.g. Llama2?\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> What is the temperature in the experiments shown in Fig 4; and why did you choose that value for this experiment?\n\nIn Fig.4 the temperatures for gpt-2, gpt-3, gpt-3.5 are 1.0, 0.5, 1.0 respectively. We chose these temperatures because they showed the greatest learning effect (negative slope) in the horizon 6 task, based on Figure 2. We will add this information in the final version of the paper.\n\n> Can you comment on what should be a good strategy regarding the choice of temperature?\n\nIt depends on how we want the model to perform. For example, in gpt-3.5 we found that when the temperature is set to 0 the model performs optimally. On the other hand, a temperature of 1.0 makes the model more human-like.\n\n> Is there a specific reason you did not study other tasks from Binz and Schulz (2023)? \n\n> Why did you choose the Horizon decision-making task? i.e. instead of any of the other tasks in Binz and Schulz (2023)?\n\nWe found these results from the original paper to be the most worthy of further investigation because they involve the models’ ability to make economic decisions and implicitly calculate a regret function. These topics are under-explored in other work on LLM psychology.\n\n> Is there a repo (even if it consists of simple scripts) with the code to reproduce the results (and download the datasets, etc)?\n\nWe have made the code available to reviewers as the supplementary material. Will make it available on Github upon publication.\n> I understand that such a repo would require the users to have access to OpenAI APIs. But I also guess it could easily be adapted to e.g. Llama2?\n\nModifying the current code to carry out identical experiments on open-source models such as Llama 2 is straightforward, and we plan to incorporate comprehensive testing compatibility with open-source models in the final GitHub repository.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The papers is concise, well written and makes a simple yet important point, namely that any study analyzing the characteristics, features and abilities of LPLM -- specially those studies coming out of the NLP community --- should consider the effect of prompting and hyperparameter variation on the LPLM performance. \n\nI think it's a good short paper; and it complies nicely with the short paper definition of EMNLP.", "reasons_to_reject": "As mentioned by the authors in the limitations section, their work only examines one of the tasks in Binz and Schulz (2023). It would have been interesting to see similar investigations wrt. the other tasks.\n\n", "questions_to_authors": "(1) What is the temperature in the experiments shown in Fig 4; and why did you choose that value for this experiment? [I'd recommend adding this information to the paper]\n\n(2) Can you comment on what should be a good strategy regarding the choice of temperature?\n\n(3) is there a specific reason you did not study other tasks from Binz and Schulz (2023)?\n\n(4) Why did you choose the Horizon decision-making task? i.e. instead of any of the other tasks in Binz and Schulz (2023)?\n\n(5) is there a repo (even if it consists of simple scripts) with the code to reproduce the results (and download the datasets, etc)? \n\nI understand that such a repo would require the users to have access to OpenAI APIs. But I also guess it could easily be adapted to e.g. Llama2?\n", "paper_topic_and_main_contributions": "The paper revisits the performance of large pretrained language models (LPLM) on a Horizon decision-making task studied by Binz and Schulz (2023). In the latter study it was found that there can be strong divergences between human and GPT3 performance on that task. The authors claim however that Binz and Schulz (2023) did not analyze the effects of prompt modification and temperature on the performance of LPLMs. The paper goes on to precisely study the change in performance of LPLM wrt. a Horizon decision-making task under prompt and temperature variation. \n\nTheir results empirically show that the decision-making capabilities of LPLM are indeed affected by both prompt and temperature. The former having a stronger effect than the latter. They also observe that, with adequate prompting, LPLM can display human-like exploration-exploitation trade-off.\n\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "(1) Can you add (e.g. in the appendix) the equivalent of Fig 5 but for text-davinci-003 and text-davinci-002? \n\nand, if the reason why you didn't add this figure is because the results are too similar, can you add a comment mentioning this?\n\n(2) Can you add (e.g. in the appendix) the equivalent of Fig 4 but for the other two temperatures?"}}]

Author rebuttal:

Meta-review: This work investigates how Large Language Models (LLMs) perform on a decision-making task that requires balancing exploration and exploitation. The authors show that LLMs’ behavior depends on the input prompts and the temperature settings, and that they can achieve human-like or superhuman results with simple adjustments. The authors suggest that LLM psychology studies should account for the variability of LLMs’ responses under different conditions.

Pros:
Demonstrated that LLMs can achieve superhuman performance in terms of minimizing regret on the Horizon task, which can have implications for designing more efficient and effective decision-making systems based on LLMs.

Cons:
Insufficient details and justifications for the experimental design and analysis
Decision: Accept-Findings