LLM Agents Do Not Replicate Human Market Traders: Evidence from Experimental Finance

## Abstract

Abstract In this study, we compare Large Language Models (LLMs) with human traders in a classic experimental-finance paradigm where prices are determined endogenously. Using a well-established asset-trading design, we run homogeneous (mono-agent) markets with single-model LLM agents and heterogeneous “Battle Royale” markets with multiple LLM models. Our findings reveal that LLMs generally exhibit a “textbook-rational” approach, pricing the asset near its fundamental value and showing only a muted tendency toward bubble formation, whereas humans deviate substantially and consistently generate bubbles. Additional treatments, including dividend shocks and repeated-exposure (‘experienced’) runs, show that these differences persist across various experimental settings. Further analyses of LLM-generated strategy text indicate lower variance, reduced bias, and stronger reliance on fundamentals relative to humans’ more heuristic-driven trading. These results highlight the risk of using LLM-agents to model human-driven market phenomena, as key behavioral features such as large, emergent bubbles are not reproduced.

## 1 Introduction

Financial markets are complex, dynamic coordination systems shaped by individual decisions made under uncertainty. While traditional financial theory assumes rational behavior, decades of behavioral finance research has revealed persistent biases, such as herding and overconfidence, that can give rise to mispricings and asset bubbles. These deviations underscore the challenges of achieving optimal outcomes in uncertain and evolving market environments.

The rise of agentic systems, particularly those driven by Large Language Models (LLMs), introduces a novel and previously unexplored dimension to this problem. As LLMs increasingly participate in domains traditionally navigated by humans, including financial systems and markets, understanding their behavioral tendencies has become critical. Questions such as; Do LLM-agents exhibit similar behavioral biases, behave more rationally, or fall victim to entirely different biases not seen in humans?

This paper presents the first comprehensive study comparing ”out-of-the-box” LLM and human behavior within a single experimental finance paradigm. We adopt a classical trading environment where participants trade a risky asset with a controlled fundamental value and examine the behavior of LLM-agents in both homogeneous (mono-agent) markets, populated entirely by instances of the same model, and heterogeneous ”Battle Royale” markets, where agents based on different base LLMs compete. These experimental markets are entirely endogenous, with prices determined solely by the buy-and-sell orders submitted by participants, and without any external price anchor or market maker. This enables us to dive deeper into how the underlying agent behavior affects the market, as we can control all market parameters.

Previous research offers conflicting perspectives on AI-human behavioral alignment. Some works suggest a significant difference between human and LLM interactions (Anlló et al. (2024)), while others indicate that LLM behavior could supplement or even substitute for human studies (Leng (2024); Horton (2023); Jia et al. (2024)). We address this tension in the literature through rigorous experimental comparison in experimental finance, a domain that has long served as a testbed for understanding financial decision-making under risk and uncertainty.

##### Contributions

Our findings reveal clear behavioral differences between LLM and human-driven markets. LLM markets demonstrate substantially more “textbook-rational” behavior than their human counterparts. Human markets consistently generate significant price bubbles and deviations from the asset’s fundamental value, while LLM-agents generally trade much closer to fundamental value and exhibit markedly reduced trading variance. We also find that LLMs appear to incorporate less bias into their future price forecasts and employ strategies more focused on fundamental value, rather than strategies commonly used by humans. These results challenge the notion that ”out-of-the-box” LLMs can faithfully replicate human market dynamics, particularly in terms of the emergence of market-level phenomena such as bubbles and crashes. As agentic systems increasingly participate in and influence real financial markets, understanding these behavioral divergences becomes essential for effective regulation, risk management, and market design.

## 2 Background

Numerous works have studied LLMs in strategic environments such as social dilemmas and iterated games Brookins & DeBacker (2024); Horton (2023); Akata et al. (2023). In finance, most existing studies focus on empirical applications—LLMs trading in real markets or analyzing financial data (e.g., Xiao et al. (2024); Li et al. (2024); Yu et al. (2024a); Yun (2024); Liu & Lo (2025); Ding et al. (2024); Abdelsamie & Wang (2024); Chen (2025); Zhou et al. (2025)). However, to our knowledge, no prior work has analyzed LLM behavior in experimental markets where prices are endogenously generated by agent interaction (see Appendix [C](https://arxiv.org/html/2502.15800v3#A3) for a more comprehensive review).

In finance, LLMs have regularly been applied to real-world scenarios. Papers such as; Zhang et al. (2023); Bond (2023); Kirtac & Germano (2024) explored LLMs’ ability to process financial sentiment and predict market movements. Fatouros et al. (2024) introduced MarketSenseAI, which leverages GPT-4 for stock selection, illustrating the utility of LLMs in navigating complex financial data. While these studies demonstrate the promise of LLMs in financial settings, there is still a need to understand how these agents perform in experimental financial settings where strategic interactions drive outcomes.

The study of market dynamics in controlled experimental environments has revealed critical insights into behavioral biases and inefficiencies. For instance, Smith et al. (1988) demonstrated the formation of bubbles and crashes in experimental markets, even under conditions of perfect information. Subsequent work by Bostian & Holt (2009) highlighted how individual behaviors, such as overconfidence and herding, contribute to these phenomena. Incorporating AI agents into such paradigms presents a unique opportunity to evaluate whether these systems replicate, mitigate, or exacerbate market inefficiencies.

## 3 Experiment

We focus on a classic experimental finance paradigm with a constant fundamental value, the mathematical price of an asset; see Equation [B](https://arxiv.org/html/2502.15800v3#A2) in the Appendix. We adopt the experimental design from Smith et al. (2014) and Holt et al. (2017), which involves an endogenous experimental asset market where subjects trade a risky asset (STOCK) in an open-call market for 30 periods operated using the oTree experiment platform (Chen et al. (2016)). Participants are given 4 shares of STOCK and 100 units of CASH. Participants have two options: either invest CASH in STOCK, which in each trading period pays out dividends of either 0.4 or 1 units of CASH with equal probability, or hold CASH, which generates interest at 5% each trading period. The price is determined solely by the trading activity of the experimental market participants.

After the experiment, the risky asset is redeemed for a fixed value of 14 units of CASH, representing the fundamental value of the STOCK (See Appendix [B](https://arxiv.org/html/2502.15800v3#A2) for full formulation).

Because of these parameter decisions (i.e., dividend schedule, CASH interest rate, and redemption price), the fundamental value remains constant throughout the entirety of the experiment, enabling us to objectively classify when the market is in a “bubble” state Bostian et al. (2005). This differs starkly from applied financial studies, where the fundamental value of a real-world asset is unknown. Further discussion of the experimental details can be found in Appendices [A](https://arxiv.org/html/2502.15800v3#A1).

This task was assigned to both LLM agents and human traders in separate markets. In adapting the task to LLM-agents, our primary goal was to best emulate the experiment presented to human subjects. The description of the task setting mirrors the original instruction video, shown to human participants (See Appendix [H.2](https://arxiv.org/html/2502.15800v3#A8.SS2)). In every round of the experiment, each agent simultaneously submits orders to the market as well as expectations regarding current and future market prices.

A common challenge in adapting multi-round games to LLM-agents is the persistence of memory between rounds, which is critical to learning and adapting strategies. Building off previous works (Fish et al. (2024)), we allow agents to communicate with future versions of themselves by allowing them to read and write their own “Insights” and “Thoughts” at each round. This structure incorporates Chain-of-Thought (CoT) reasoning (Wei et al. (2022)), which has been shown to help elicit the reasoning ability of LLMs, providing an interpretable window into how agents perceive and interact with the market. Further prompting details are presented in Appendix [I](https://arxiv.org/html/2502.15800v3#A9).

To frame our analysis, we categorized potential outcomes into three groups: rational (R), where models do not generate bubbles and trade close to the fundamental value; human (H), where models generate bubbles in a manner similar to humans; and erratic (E), where models perform inconsistently, failing to adhere to any consistent strategy, whether rational or human-like.

## 4 Single-Model LLM Market Performance

In this section, we evaluate how single-model LLM markets behave along two critical dimensions: rational adherence to fundamental asset value and similarity to human market dynamics. Table [1](https://arxiv.org/html/2502.15800v3#S4.T1) and Figure [1](https://arxiv.org/html/2502.15800v3#S4.F1) summarize our key findings.

### 4.1 Experimental Setup

We conducted three markets using agents based on six different API-based commercial LLMs: Anthropic’s Claude-3.5-Sonnet Anthropic (2024), OpenAI’s GPT-3.5 and GPT-4o OpenAI (2025), xAI’s Grok-2 xAI (2024), Mistral-Large AI’s Mistral-Large AI (2024), and Google’s Gemini-1.5 Pro DeepMind (2024). Each market consisted of 20 identical instances of the same LLM agent. The agents were given the same prompt and were unaware of the identities of their counterparts. Each instance was instructed to maximize its profit.

For comparison, we also ran nineteen human-only markets, with identical experimental conditions (though the humans also conducted a short risk-elicitation task in between each round). Participants were a mix of [UNIVERSITY] students and staff, as well as online participants recruited through prolific.com. The human participants were given instructions that mirror the LLM prompts; full instructions are shown in Appendix section [H.2](https://arxiv.org/html/2502.15800v3#A8.SS2). The average plot from these experiments is shown alongside the LLM-only Battle Royale market results in Figure [5.1](https://arxiv.org/html/2502.15800v3#S5.SS1).

#### 4.1.1 Single-Model Market Analysis

As shown in Table [1](https://arxiv.org/html/2502.15800v3#S4.T1), Claude-3.5 Sonnet and GPT-4o yield the lowest MSEs and nearly zero correlation with human price paths, keeping prices close to fundamentals and mapping to the Rational (R) hypothesis. Both also show low portfolio variance, suggesting consistent strategies across instances.

By contrast, Grok-2 and GPT-3.5 show the largest deviations from fundamentals and the highest correlations with human markets. Grok-2 exhibits bubble–crash cycles, while GPT-3.5 generates a monotonic run-up without correction (Figure [1](https://arxiv.org/html/2502.15800v3#S4.F1)), a pattern occasionally seen in humans.

Mistral-Large also produces bubbles but with smaller peaks and sharper crashes, dipping below fundamentals late in the session. While more rational in absolute terms (MSE $\approx$ 5.7 vs. 429.8 for humans), its dynamics align with the Human (H) hypothesis.

Finally, Gemini-1.5 Pro departs from both rational and human-like behavior, trading well below fundamentals before correcting upward. This so-called “reverse bubble” (negative PCC) suggests erratic dynamics rather than coherent strategy, mapping most closely to the Erratic (E) hypothesis.

**Table 1: Market performance metrics**
| Model | MSE (Fundamental) | PCC (Human Avg) | Closest Hypothesis | PV Variance |
| --- | --- | --- | --- | --- |
| Claude-3.5-Sonnet | 0.536 | 0.001 | R | 26.15 |
| GPT-4o | 0.789 | 0.050 | R | 22.76 |
| Grok-2 | 17.325 | 0.558 | H | 39.43 |
| Mistral-Large | 5.694 | -0.112 | H | 49.96 |
| GPT-3.5 | 26.367 | 0.490 | H | 30.44 |
| Gemini-1.5 Pro | 3.103 | -0.401 | E | 45.58 |

Figure: Figure 1: Mono-Agent Price Charts compared to Humans
Refer to caption: figs/all_markets.png

## 5 Battle Royale: Comparing LLM Market Performance

To evaluate how various LLM agents fare when competing in the same market, we conducted *Battle Royale* market experiment with a heterogeneous mixture of the models. By introducing diverse LLMs into a single environment, we aimed to explore:

- •
Which LLM is most effective at maximizing profits in a competitive, multi-agent setting? Which Agent is the best trader?
- •
How a mixed-LLM market behaves, and whether it will exhibit emergent phenomena when models with distinct trading strategies interact?
- •
Whether heterogeneous LLM interpretation and strategy differences are enough to generate financial bubbles?

### 5.1 Experimental Setup

We conducted markets involving the same six LLM models as in the Single-Model section. Each market included a total of 24 agents (four agents per model). The agents operated under identical conditions and were tasked with maximizing their profits over 30 trading periods.
The risky asset in these markets maintained a constant fundamental value of 14 units, consistent with prior experiments. Figure [2](https://arxiv.org/html/2502.15800v3#S5.T2) shows the three ’Battle Royale’ market results.

Figure: Figure 2: Battle Royale Market Performance with Human Comparison
Refer to caption: figs/battle_royale_markets_overlaid_no_title.jpeg

### 5.2 Battle Royale Market Analysis

Across three heterogeneous markets, two converged on prices near fundamentals, while one produced a modest bubble resembling human markets but with a lower peak. Trading volume spiked at the end of this bubble run, echoing the late-round rush seen in human experiments (Figure [5.1](https://arxiv.org/html/2502.15800v3#S5.SS1)). Overall, these dynamics align most closely with the Rational (R) hypothesis, although they demonstrate that competitive interaction can occasionally generate bubble-like patterns.

At the agent level (Table [2](https://arxiv.org/html/2502.15800v3#S5.T2)), portfolio values cluster tightly, with no model consistently dominating. The spread between best and worst traders remains small, suggesting convergence on broadly similar strategies and limited inequality of outcomes in mixed markets. This shows that none of these models are able to consistently best the others in competitive trading.

## 6 Trading Strategy Analysis

After each round, LLM agents were prompted to provide details of the strategy they used to drive their trading behavior. This was done both to allow us to analyze their behavior more deeply and to enable the LLMs to develop long-running strategies, utilizing Chain-of-Thought-like reasoning. In contrast, human subjects were asked what approach they employed only at the end of the experiment. More details of our data cleaning techniques and additional analysis can be found in Appendix [F](https://arxiv.org/html/2502.15800v3#A6).

We first examined the word frequency of both humans and LLMs in the mono-agent runs. Of the top six words for each group, five are the same, though in a different order (”sell”, ”buy”, ”stock”, ”price”, and ”market”; see Table [6](https://arxiv.org/html/2502.15800v3#A6.T6) in Appendix for details). The top ten most popular words for the human subjects include ”low” and ”high”, which do not appear among the top ten words used by the LLMs. Conversely, the words ”continue” and ”buyback” are among the most common for the LLMs (see also word clouds in Figure [22](https://arxiv.org/html/2502.15800v3#A6.F22) in the Appendix). This pattern suggests that human traders follow a ”buy low, sell high” strategy in an attempt to outperform the market. In contrast, LLM agents adopt a longer-term approach that emphasizes the asset’s intrinsic value.

### 6.1 Text Mining and Strategy Comparison to Humans

To assess whether LLMs engaged in coordination or converged on shared narratives, we ran a text–mining analysis of the “insight” and “plan” files. We found no explicit coordination language (0 hits on terms such as ”coordinate” or ”work together”), and only a handful of generic references to ”anticipating other agents.” By contrast, we observed several instances of near–orthogonal language between agent pairs (cosine similarity $\leq 0.1$; e.g., “buy aggressively” vs. “sell off”), suggesting divergence rather than cooperation. Overall, mixed–model markets appear to consist of independent traders reacting to shared price signals rather than colluding.

We then directly compared LLM and human strategies. A classifier trained on the insights/plans text achieved $>$99% accuracy in identifying which model produced which text. Extending this to a seven–way classifier including humans (using their end of experiment reflections), we found that $\sim$33% of Gemini-1,5 Pro and GPT-4 reflections were misclassified as human, while all other models were below 10%. Binary classifiers confirmed this pattern: GPT-4 text was misclassified as human nearly 50% of the time (Mistral-Large 36%; GPT-3.5 28%), while other models were below 3%. Despite superficial stylistic overlap, cluster analysis of strategy words showed almost no clusters containing both humans and LLMs (in one run, only 8/497 humans clustered with any model), suggesting that the underlying reasoning remains distinct.

To probe this distinction further, we ran a Latent Dirichlet Allocation (LDA) Blei et al. (2003) on all strategies combined. Using the Python gensim library Řehůřek & Sojka (2010), we assigned each strategy to one of two topics. The first topic, identified by the words orders, sell, buy, shares, stock, denoted strategies aligned with a “buy low; sell high” practice. The second topic, identified by the words stock, adjust, buyback, closely, current, reflected a more fundamental value focused approach. The LDA assigned 89.3% of human strategies to the first topic but only 36.6% of the LLM strategies; conversely, 63.4% of LLM strategies (10.7% of humans) fell into the second category. A t-test confirmed this difference is highly significant ($p<.001$). Figure [23](https://arxiv.org/html/2502.15800v3#A6.F23) shows this distribution.

### 6.2 Linguistic Audit of Model Strategies

We also wanted to identify why specific models generate larger, more human-like bubbles, while others adhere relatively closely to fundamentals. To do so, we ran a finer-grained linguistic audit of the ”insight” and ”plan” outputs. Each statement was tagged as *speculative*, *fundamental*, or *generic* based on keyword dictionaries.

**Table 3: Distribution (%) of speculative, fundamental, and generic text in agent strategy statements. The behavior column reflects observed market outcomes.**
| Model | Speculative | Fundamental | Generic | Behavior |
| --- | --- | --- | --- | --- |
| Grok-2 | 68.4 | 16.0 | 15.6 | Bubble-prone |
| GPT-3.5 | 28.4 | 18.0 | 53.6 | Bubble-prone (No Crash) |
| GPT-4o | 17.9 | 59.6 | 22.5 | Bubble-dampening |
| Claude-3.5 Sonnet | 14.5 | 82.8 | 2.7 | Bubble-dampening |
| Gemini-1.5 Pro | 20.5 | 58.7 | 20.8 | Bubble-dampening |
| Mistral-Large | 35.8 | 52.0 | 12.2 | Mixed |

Grok-2 and GPT-3.5 seem to create bubbles because speculative narratives dominate their reasoning, whereas GPT-4o, Claude-3.5, and Gemini-1.5 Pro seem to dampen exuberance by emphasizing intrinsic value anchors. Mistral-Large oscillates between the two, producing smaller, transient bubbles with a trajectory of value $\rightarrow$ momentum $\rightarrow$ uncertainty. These linguistic signatures provide a possible mechanistic explanation for the divergent market outcomes observed across models.

## 7 Market Treatments

Beyond the baseline constant–fundamental design, we tested whether LLM’s rationality profiles held under more complex and dynamic market conditions, such as when payoffs and fundamental values shift over time.

To test whether our findings extend beyond constant fundamental value, we conducted two separate mono-agent market experiments (each with three sessions) featuring a mid-experiment dividend shock. Beginning in Round 15, we either doubled ([7.1](https://arxiv.org/html/2502.15800v3#S7.SS1)) or halved ([7.2](https://arxiv.org/html/2502.15800v3#S7.SS2)) the dividend and redemption value, which doubles/halves the fundamental value of the asset. The agents were not informed of this change prior to the experiment and were told at Round 15 that the dividend and redemption value had changed (either doubled or halved) and would remain so for the rest of the experiment. This mirrors classic shock tests in experimental finance, and simulates how unexpected ”news” can impact the value of an asset, allowing us to assess whether LLMs adjust appropriately when the fundamental value changes. Note we were unable to run Grok 2 as the public endpoint has been deprecated.

### 7.1 Dividend (”News”) Shock Doubling

Results show that all models except GPT-3.5 quickly converged toward the new fundamental value. Claude-3.5 Sonnet, GPT-4o, Gemini-1.5 Pro, and Mistral-Large all incorporated the new information, though the process was not instantaneous, stabilizing prices near $28. GPT-3.5, however, lagged behind: its forecasts and orders remained anchored near the pre-shock equilibrium. Notably, Gemini-1.5 Pro occasionally overshoots the FV, creating minor bubbles and crashes.

Figure: Figure 3: Mono-Agent Market Price Charts Dividend Shock Doubling
Refer to caption: figs/quint_up.png

### 7.2 Dividend (”News”) Shock Halving

As shown in Figure [4](https://arxiv.org/html/2502.15800v3#S7.F4), all models except GPT-3.5 converged toward the new fundamental value. Claude-3.5 Sonnet, Gemini-1.5 Pro, and Mistral-Large all incorporated the new information within 2–3 rounds, stabilizing prices near $7. GPT-3.5 and GPT-4o, however, lagged behind, with GPT-3.5’s prices remaining anchored near the pre-shock equilibrium, and GPT-4o incorporating the information slowly.

Figure: Figure 4: Mono-Agent Market Price Charts Dividend Shock Halving
Refer to caption: figs/quint_down.png

These treatments reinforce our main result: most LLMs exhibit a fundamental value oriented adjustment, while GPT-3.5 is uniquely prone to inertia and bubble-like dynamics. We notice a slight increase in the propensity for the models to create bubbles, but they still occur infrequently. It may appear that the models converge more quickly to downward shocks than upward shocks. Please keep in mind that upward shocks are larger in absolute magnitude, and our experiment design limits orders to $\pm$ $3 of the current price. This was done to make the experiment accessible to humans and carried over to our LLM experiments for parity.

We also conduct multiple robustness checks in the Appendix, ensuring that slight experimental differences (such as removing the risk-elicitation task for LLMs [D.2](https://arxiv.org/html/2502.15800v3#A4.SS2)) or LLM trading experience ([D.1](https://arxiv.org/html/2502.15800v3#A4.SS1)) do not materially alter our conclusion. While these treatments represent only a small subset of the numerous possible experimental modifications, they demonstrate that our conclusion regarding LLM rationality remains robust to various changes in market conditions.

## 8 Rationality Properties of Individual LLM Forecasts

In addition to analyzing the pricing behavior of LLMs, we were also interested in how these models generate forecasts for future price periods. We captured each model’s forecasts, which allowed us to assess not only the rationality of their forecasts, whether they align with fundamental valuation principles, but also whether the LLMs themselves behave rationally in response to their expectations. We compare this analysis to that of our human traders to understand whether LLMs, in the mono-agent market setting, make similar or different forecasting mistakes compared to their human counterparts. The mathematical formulation for calculating the mean and median forecasts for any given session is provided in the Appendix (Section [G.3](https://arxiv.org/html/2502.15800v3#A7.SS3)).

Forecast errors are defined as $E_{i,t}^{h}=P_{t+h}-f_{i,t}^{\,t+h}$, where $P_{t+h}$ denotes the actual price at round $t+h$ and $f_{i,t}^{\,t+h}$ is agent $i$’s forecast formed at round $t$ for horizon $h$; positive values indicate underprediction. Full distributions and horizon-specific test results are reported in Appendix [G.3](https://arxiv.org/html/2502.15800v3#A7.SS3). Humans exhibit substantial systematic underprediction: the mean one-step error ($h=1$) is $1.67$. By contrast, all LLMs’ mean errors are close to zero. Consistent with this, Appendix Figure [26(b)](https://arxiv.org/html/2502.15800v3#A7.F26.sf2) shows that LLM forecast-error distributions are more tightly centered around zero than those from human markets, indicating higher forecast precision.

Figure: Figure 5: Mean Forecast Error by Round
Refer to caption: figs/appendix_forecast/Figure_MeanForecastError_May11.png

We also applied three additional diagnostics to compare the forecast-rationality profiles of LLM agents and human subjects: (i) unbiasedness—whether forecast errors are centered on zero (no systematic bias); (ii) zero autocorrelation—whether errors in one round fail to predict errors in later rounds; and (iii) errors uncorrelated with forecasts—whether the forecast itself fails to predict the size of the error (no optimism/pessimism bias).

Results are summarized in Table [4](https://arxiv.org/html/2502.15800v3#S8.T4) and expanded upon in Section [G.3](https://arxiv.org/html/2502.15800v3#A7.SS3) of the Appendix. Results suggest that LLMs generally forecast more accurately than humans in the short run, but their rationality profiles differ between models. GPT-3.5 is highly biased and correlated; Claude-3.5 and GPT-4o are the most consistently rational; Gemini-1.5 Pro, Grok-2, and Mistral-Large fall in between. Importantly, no model fully reproduces the human error pattern.

**Table 4: Forecast Accuracy and Rationality Properties of Humans and LLMs.**
| Model | Forecast Error | Unbiasedness | Zero Autocorr. | Errors $\perp$ Forecasts |
| --- | --- | --- | --- | --- |
| Humans | 1.67 | 0.56 | 0.17 | 0.31 |
| Claude-3.5 Sonnet | 0.09 | 0.46 | 0.08 | 0.16 |
| Gemini-1.5 Pro | 0.14 | 0.41 | 0.18 | 0.41 |
| GPT-3.5 | –1.54 | 0.00 | 0.60 | 0.00 |
| GPT-4o | 0.04 | 0.40 | 0.60 | 0.03 |
| Grok-2 | –0.31 | 0.56 | 0.18 | 0.27 |
| Mistral-Large | 0.00 | 0.44 | 0.14 | 0.36 |

## 9 Potential Limitations and Broader Impacts

Our results should be interpreted in light of several caveats.
First, LLM agents do not experience incentives, emotions, or online learning in the human sense; their “strategies” emerge from the prompt structure and frozen weights, rather than endogenous reasoning or reinforcement. Second, we study stylized markets with a single asset and a short horizon. How LLM traders respond to longer horizons, short selling, or richer strategic interactions remains an open question. Finally, we evaluated only a handful of foundation models without fine-tuning. Model choice, prompting tactics, or retrieval-augmented memory may materially alter behavior.

Despite these limitations, our work has three areas of immediate impact. First, in live markets, several hedge funds have begun deploying LLM-driven order flow. As machine agents come to represent a measurable share of trading volume, both regulators and practitioners will need empirical baselines to understand the dynamics they generate. Second, almost all significant findings in behavioral finance (loss aversion, herding, mental accounting) were initially discovered through lab experiments (Kahneman & Tversky (2013), Thaler (1985)). As we attempt to understand how LLMs trade and the biases they exhibit, we thought it best to adopt the same approach. Third, for experimental design, recent papers have explored the idea of substituting LLMs for human subjects in laboratory finance experiments (Cui et al.); our evidence of systematic departures from human behavior underscores the risk of doing so without rigorous validation. Therefore, we advise caution in replacing human subjects with ”off-the-shelf” LLM agents. We acknowledge these limitations and thus view this study as a *starting point, rather than an endpoint, for incorporating LLM agents into financial and behavioral research*.

## 10 Discussion and Conclusions

In this study, we compare the behavior of Human and Large Language Models (LLMs) in an experimental finance paradigm that has historically elicited asset bubbles and crashes when conducted with human participants. We found that LLM-agents typically converge on prices near the fundamental value, thereby exhibiting more “textbook-rational” behavior than human subjects. Although some single-model markets did display mild bubble-like phenomena, particularly GPT-3.5, which inflated prices without a clear crash, and Mistral-Large and Grok 2, which produced modest run-ups, these tendencies were weaker than what is commonly observed in human-only markets. Moreover, individual outcomes in LLM-driven markets tended to cluster more tightly around similar final portfolio values, indicating less strategic heterogeneity than seen among humans. We found that these findings persisted even under changing market conditions, news/dividend shocks. These findings collectively suggest that although LLMs may serve as rational traders, they seldom replicate the pronounced boom-and-bust patterns that emerge from human behavioral biases. We also analyze the forecasting behavior of these LLM agents and find that the majority of the LLMs generate more accurate forecasts of future price behavior than human traders; however, we observe some heterogeneity among LLM agent models in their ability to forecast rationally. Across the board, the LLM agents performed differently from humans on most tasks; therefore, our results caution against using LLMs as replacements or for piloting for human participants in experimental finance studies. Future work may explore whether new prompting strategies or incentives can induce more human-like trading patterns from LLM-agents. However, for now, it remains clear that these models trade with a consistency and rationality that diverge significantly from the behaviors displayed by human counterparts.

## 11 Ethics Statement

We do not believe there are any ethical concerns with this work, all human data was collected under review of the IRB at [University]. The experiments with large language models involved only synthetic agents and did not expose participants or sensitive populations to potential harm. All analyses follow standard practices in experimental economics and machine learning. We believe this work adheres fully to the ICLR Code of Ethics and raises no issues regarding privacy, fairness, security, or potential misuse.

## 12 Reproducibility Statement

All code will be released alongside the camera-ready submission. Complete experimental details, including instructions provided to agents and human participants, as well as the full set of prompts, are documented in Appendix [A](https://arxiv.org/html/2502.15800v3#A1) and Appendix [I](https://arxiv.org/html/2502.15800v3#A9). These materials, together with the statistical procedures and robustness checks reported in Section 4 and Appendices E–G, ensure that all results can be readily reproduced. Our goal is not only to enable replication of the experiments presented here, but also to provide a foundation that future work can extend and build upon.

## 13 LLM Disclosure

Large Language Models (LLMs) were used in a limited capacity to assist with LaTeX formatting and debugging. They were not used for research ideation, experimental design, data analysis, or substantive writing. The authors take full responsibility for the content of this paper.

## References

- Abdelsamie & Wang (2024)
M. Abdelsamie and H. Wang.
Comparative analysis of llm-based market prediction and human expertise with sentiment analysis and machine learning integration.
In *2024 7th International Conference on Data Science and Information Technology (DSIT)*, pp. 1–6. IEEE, December 2024.
- AI (2024)
Mistral AI.
Mistral large (24.11).
[https://docs.mistral.ai/getting-started/models/models_overview/](https://docs.mistral.ai/getting-started/models/models_overview/), 2024.
Accessed: 2025-05-11.
- Akata et al. (2023)
Elif Akata, Lion Schulz, Julian Coda-Forno, Seong Joon Oh, Matthias Bethge, and Eric Schulz.
Playing repeated games with large language models.
*arXiv preprint arXiv:2305.16867*, 2023.
URL [https://arxiv.org/abs/2305.16867](https://arxiv.org/abs/2305.16867).
- Anlló et al. (2024)
Hernán Anlló et al.
Do large language models reason like us?
*Communications Psychology*, 2024.
URL [https://communities.springernature.com/posts/do-large-language-models-reason-the-way-we-do](https://communities.springernature.com/posts/do-large-language-models-reason-the-way-we-do).
- Anthropic (2024)
Anthropic.
Introducing claude 3.5 sonnet.
[https://www.anthropic.com/news/claude-3-5-sonnet](https://www.anthropic.com/news/claude-3-5-sonnet), 2024.
Accessed: 2025-05-11.
- Bhat & Jain (2024)
R. Bhat and B. Jain.
Stock price trend prediction using emotion analysis of financial headlines with distilled llm model.
In *Proceedings of the 17th International Conference on PErvasive Technologies Related to Assistive Environments*, pp. 67–73, June 2024.
- Bird et al. (2009)
Steven Bird, Ewan Klein, and Edward Loper.
*Natural language processing with Python: analyzing text with the natural language toolkit*.
” O’Reilly Media, Inc.”, 2009.
- Blei et al. (2003)
David M Blei, Andrew Y Ng, and Michael I Jordan.
Latent dirichlet allocation.
*Journal of machine Learning research*, 3(Jan):993–1022, 2003.
- Bond (2023)
Shaun A. Bond.
Large language models and financial market sentiment.
*SSRN Electronic Journal*, 2023.
doi: 10.2139/ssrn.4584928.
URL [https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4584928](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4584928).
- Bostian et al. (2005)
AJ Bostian, Jacob Goeree, and Charles A Holt.
Price bubbles in asset market experiments with a flat fundamental value.
In *Draft for the Experimental Finance Conference, Federal Reserve Bank of Atlanta September*, volume 23, pp. 2005, 2005.
- Bostian & Holt (2009)
Andrew A. Bostian and Charles A. Holt.
Price bubbles with discounting: A web-based classroom experiment.
*Journal of Economic Education*, 40(1):27–37, 2009.
doi: 10.3200/JECE.40.1.27-37.
- Brookins & DeBacker (2024)
Philip Brookins and Jason DeBacker.
Playing games with gpt: What can we learn about a large language model from canonical strategic games?
*Economics Bulletin*, 44(1):25–37, 2024.
URL [http://www.accessecon.com/Pubs/EB/2024/Volume44/EB-24-V44-I1-P3.pdf](http://www.accessecon.com/Pubs/EB/2024/Volume44/EB-24-V44-I1-P3.pdf).
- Chen et al. (2016)
Daniel L. Chen, Martin Schonger, and Chris Wickens.
otree—an open-source platform for laboratory, online, and field experiments.
*Journal of Behavioral and Experimental Finance*, 9:88–97, 2016.
ISSN 2214-6350.
doi: https://doi.org/10.1016/j.jbef.2015.12.001.
URL [https://www.sciencedirect.com/science/article/pii/S2214635016000101](https://www.sciencedirect.com/science/article/pii/S2214635016000101).
- Chen (2025)
Q. Chen.
Stock price prediction with llm-guided market movement signals and transformer model.
In *FinTech and Sustainable Innovation*, pp. 1–7, 2025.
- Cheng & Chin (2024)
J. Cheng and P. Chin.
Empirical asset pricing with large language model agents.
*arXiv preprint arXiv:2409.17266*, 2024.
- Cobbe et al. (2021)
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.
Training verifiers to solve math word problems.
*arXiv preprint arXiv:2110.14168*, 2021.
URL [https://arxiv.org/abs/2110.14168](https://arxiv.org/abs/2110.14168).
- (17)
Ziyan Cui, Ning Li, and Huaikang Zhou.
Can AI replace human subjects? a large-scale replication of psychological experiments with LLMs.
URL [https://papers.ssrn.com/abstract=4940173](https://papers.ssrn.com/abstract=4940173).
- DeepMind (2024)
Google DeepMind.
Gemini 1.5: Unlocking multimodal understanding across millions of tokens.
*arXiv preprint arXiv:2403.05530*, 2024.
URL [https://arxiv.org/abs/2403.05530](https://arxiv.org/abs/2403.05530).
- Ding et al. (2024)
Q. Ding, H. Shi, J. Guo, and B. Liu.
Tradexpert: Revolutionizing trading with mixture of expert llms.
*arXiv preprint arXiv:2411.00782*, 2024.
- Fatouros et al. (2024)
George Fatouros, Kostas Metaxas, John Soldatos, and Dimosthenis Kyriazis.
Can large language models beat wall street? evaluating gpt-4’s impact on financial decision-making with marketsenseai.
*Neural Computing and Applications*, December 2024.
ISSN 1433-3058.
doi: 10.1007/s00521-024-10613-4.
URL [http://dx.doi.org/10.1007/s00521-024-10613-4](http://dx.doi.org/10.1007/s00521-024-10613-4).
- Fish et al. (2024)
Sara Fish, Yannai A. Gonczarowski, and Ran I. Shorrer.
Algorithmic collusion by large language models, 2024.
URL [https://arxiv.org/abs/2404.00806](https://arxiv.org/abs/2404.00806).
- Hendrycks et al. (2021)
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.
Measuring massive multitask language understanding.
*arXiv preprint arXiv:2009.03300*, 2021.
URL [https://arxiv.org/abs/2009.03300](https://arxiv.org/abs/2009.03300).
- Holt et al. (2017)
Charles A. Holt, Megan Porzio, and Michelle Y. Song.
Price bubbles, gender, and expectations in experimental asset markets.
*European Economic Review*, 100:72–94, 2017.
doi: 10.1016/j.euroecorev.2017.05.005.
URL [https://doi.org/10.1016/j.euroecorev.2017.05.005](https://doi.org/10.1016/j.euroecorev.2017.05.005).
- Horton (2023)
John J. Horton.
Large language models as simulated economic agents: What can we learn from homo silicus?, 2023.
URL [https://www.nber.org/papers/w31122](https://www.nber.org/papers/w31122).
- Jia et al. (2024)
J. J. Jia, Z. Yuan, J. Pan, P. McNamara, and D. Chen.
Decision-making behavior evaluation framework for llms under uncertain context.
In *Advances in Neural Information Processing Systems*, volume 37, pp. 113360–113382, 2024.
- Kahneman & Tversky (2013)
Daniel Kahneman and Amos Tversky.
Prospect theory: An analysis of decision under risk.
In *Handbook of the fundamentals of financial decision making: Part I*, pp. 99–127. World Scientific, 2013.
- Kang et al. (2024)
Y. Kang, G. Wang, X. Yang, Y. Wang, and M. Liu.
Can large language models effectively process and execute financial trading instructions?
*arXiv preprint arXiv:2412.04856*, 2024.
- Kim (2025)
B. Kim.
Large language model driven technical analysis: Enhancing predictive accuracy and interpretability in stock trading.
*International Journal of Internet, Broadcasting and Communication*, 17(1):205–215, 2025.
- Kirtac & Germano (2024)
Kemal Kirtac and Guido Germano.
Sentiment trading with large language models.
*arXiv preprint arXiv:2412.19245*, 2024.
URL [https://arxiv.org/abs/2412.19245](https://arxiv.org/abs/2412.19245).
- Kuruvilla & Mythily (2025)
J. S. Kuruvilla and M. Mythily.
Financial llm for stock price analysis and investment recommendation.
*Journal of Telematics and Informatics*, 13(1), 2025.
- Leng (2024)
Yan Leng.
Can llms mimic human-like mental accounting and behavioral biases?
*SSRN Electronic Journal*, 2024.
doi: 10.2139/ssrn.4705130.
URL [https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4705130](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4705130).
- Li et al. (2025)
X. Li, Y. Zeng, X. Xing, J. Xu, and X. Xu.
Hedgeagents: A balanced-aware multi-agent financial trading system.
In *Companion Proceedings of the ACM on Web Conference 2025*, pp. 296–305, May 2025.
- Li et al. (2024)
Y. Li, B. Luo, Q. Wang, N. Chen, X. Liu, and B. He.
A reflective llm–based agent to guide zero–shot cryptocurrency trading.
*arXiv preprint arXiv:2407.09546*, 2024.
- Liu & Lo (2025)
K. M. Liu and M. C. Lo.
Llm-based routing in mixture of experts: A novel framework for trading.
*arXiv preprint arXiv:2501.09636*, 2025.
- OpenAI (2025)
OpenAI.
Openai api, 2025.
URL [https://openai.com/api/](https://openai.com/api/).
Accessed: 2025-05-11.
- Oprea & Bâra (2024)
S. V. Oprea and A. Bâra.
A multi-agent system based on llm for trading financial assets.
*Ovidius University Annals, Series Economic Sciences*, 24(2), 2024.
- Pang et al. (2002)
Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan.
Thumbs up? sentiment classification using machine learning techniques.
*arXiv preprint cs/0205070*, 2002.
- Řehůřek & Sojka (2010)
Radim Řehůřek and Petr Sojka.
Software Framework for Topic Modelling with Large Corpora.
In *Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks*, pp. 45–50, Valletta, Malta, May 2010. ELRA.
[http://is.muni.cz/publication/884893/en](http://is.muni.cz/publication/884893/en).
- Saqur (2024)
R. Saqur.
What teaches robots to walk, teaches them to trade too–regime adaptive execution using informed data and llms.
*arXiv preprint arXiv:2406.15508*, 2024.
- Shi & Hollifield (2024)
J. Shi and B. Hollifield.
Predictive power of llms in financial markets.
*arXiv preprint arXiv:2411.16569*, 2024.
- Siddique et al. (2025)
M. T. Siddique, S. S. Jamee, A. Sajal, S. N. Mou, M. R. H. Mahin, M. O. Obaid, M. Hasan, et al.
Enhancing automated trading with sentiment analysis: Leveraging large language models for stock market predictions.
*The American Journal of Engineering and Technology*, 7(03):185–195, 2025.
- Smith et al. (2014)
Alec Smith, Terry Lohrenz, Justin King, P. Read Montague, and Colin F. Camerer.
Irrational exuberance and neural crash warning signals during endogenous experimental market bubbles.
*Proceedings of the National Academy of Sciences*, 111(29):10503–10508, 2014.
doi: 10.1073/pnas.1318416111.
URL [https://www.pnas.org/doi/abs/10.1073/pnas.1318416111](https://www.pnas.org/doi/abs/10.1073/pnas.1318416111).
- Smith et al. (1988)
Vernon L. Smith, Gerry L. Suchanek, and Arlington W. Williams.
Bubbles, crashes, and endogenous expectations in experimental spot asset markets.
*Econometrica*, 56(5):1119–1151, 1988.
doi: 10.2307/1911361.
- Sun et al. (2010)
Youfa Sun, Xiao Liang, Xu Guo, and Cai Liu.
Algorithm of call auction price.
*Applied Mechanics and Materials*, 20-23, 01 2010.
doi: 10.4028/www.scientific.net/AMM.20-23.981.
- Thaler (1985)
Richard Thaler.
Mental accounting and consumer choice.
*Marketing science*, 4(3):199–214, 1985.
- Wei et al. (2022)
Jason Wei, Xuezhi Wang, et al.
Chain of thought prompting elicits reasoning in large language models.
*arXiv preprint arXiv:2201.11903*, 2022.
- xAI (2024)
xAI.
Grok-2 beta release.
[https://x.ai/blog/grok-2](https://x.ai/blog/grok-2), 2024.
Accessed: 2025-05-11.
- Xiao et al. (2024)
Y. Xiao, E. Sun, D. Luo, and W. Wang.
Tradingagents: Multi-agents llm financial trading framework.
*arXiv preprint arXiv:2412.20138*, 2024.
- Yu et al. (2023)
X. Yu, Z. Chen, Y. Ling, S. Dong, Z. Liu, and Y. Lu.
Temporal data meets llm–explainable financial time series forecasting.
*arXiv preprint arXiv:2306.11025*, 2023.
- Yu et al. (2024a)
Y. Yu, H. Li, Z. Chen, Y. Jiang, Y. Li, D. Zhang, and K. Khashanah.
Finmem: A performance-enhanced llm trading agent with layered memory and character design.
In *Proceedings of the AAAI Symposium Series*, volume 3, pp. 595–597, May 2024a.
- Yu et al. (2024b)
Y. Yu, Z. Yao, H. Li, Z. Deng, Y. Jiang, Y. Cao, and Q. Xie.
Fincon: A synthesized llm multi-agent system with conceptual verbal reinforcement for enhanced financial decision making.
*Advances in Neural Information Processing Systems*, 37:137010–137045, 2024b.
- Yun (2024)
S. Yun.
Pretrained llm adapted with lora as a decision transformer for offline rl in quantitative trading.
*arXiv preprint arXiv:2411.17900*, 2024.
- Zhang et al. (2024a)
C. Zhang, X. Liu, Z. Zhang, M. Jin, L. Li, Z. Wang, Y. Zhang, et al.
When ai meets finance (stockagent): Large language model-based stock trading in simulated real-world environments.
*arXiv preprint arXiv:2407.18957*, 2024a.
- Zhang et al. (2024b)
W. Zhang, L. Zhao, H. Xia, S. Sun, J. Sun, M. Qin, B. An, et al.
A multimodal foundation agent for financial trading: Tool-augmented, diversified, and generalist.
In *Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining*, pp. 4314–4325, August 2024b.
- Zhang et al. (2023)
Xiao-Yang Zhang, Weiguang Ma, Shuo Liu, Runhan Wu, Jinyang Jiang, Weiqi Li, Yong Zhu, Dong Chen, Ming Wang, and Hongyang Zhang.
Instruct-fingpt: Financial sentiment analysis by instruction tuning of large language models.
*arXiv preprint arXiv:2306.12659*, 2023.
URL [https://arxiv.org/abs/2306.12659](https://arxiv.org/abs/2306.12659).
- Zhao et al. (2024)
H. Zhao, Z. Liu, Z. Wu, Y. Li, T. Yang, P. Shu, T. Liu, et al.
Revolutionizing finance with llms: An overview of applications and insights.
*arXiv preprint arXiv:2401.11641*, 2024.
- Zheng et al. (2023)
Lianmin Zheng, Sky Huang, Sewon Min, Zhao Zhang, Han Zha, Tuo Zhao, Joseph E Gonzalez, Ion Stoica, and Pieter Abbeel.
Judging llm-as-a-judge with mt-bench and chatbot arena.
*arXiv preprint arXiv:2306.05685*, 2023.
URL [https://arxiv.org/abs/2306.05685](https://arxiv.org/abs/2306.05685).
- Zhou et al. (2025)
L. Zhou, Y. Zhang, J. Yu, G. Wang, Z. Liu, S. Yongchareon, and N. Wang.
Llm-augmented linear transformer–cnn for enhanced stock price prediction.
*Mathematics*, 13(3):487, 2025.

## Appendix A Additional Experiment Details

We now elaborate on the details of the experimental setup. There are 3 practice trading rounds followed by 30 rounds of trading in the main experiment. The practice rounds have no impact on the main experiment rounds and serve only to familiarize participants with the market setting. During each round, participants interact with the market through a sequence of screens.

Our experimental design involves an asset market where subjects trade a risky asset in an open-call market for 30 periods (Smith et al., 2014; Holt et al., 2017). Participants are given 4 shares of stock and 140 units of cash. Participants have two options: either invest cash in the risky asset, which pays out dividends of either 0.4 or 1 with equal probability, or hold cash, which accrues interest at a rate of 5% each trading period. The dividend payment is drawn independently for each round.

The experiment consists of a sequence of four screens in this order, as depicted in the Figures below:

- 1.
Order Submission Screen (20 seconds)(Figure [6](https://arxiv.org/html/2502.15800v3#A1.F6)): Participants submit orders to buy or sell the asset
Figure 6: Order Submission Page
- 2.
Forecast Screen (30 seconds)(Figure [7](https://arxiv.org/html/2502.15800v3#A1.F7)): Participants submit their expectations regarding current and future market prices.
Figure 7: Forecast Page
Participants are asked to submit their estimation for the current market period and periods 2, 5, and 10 rounds ahead.
- 3.
Round Results Screen (10 seconds)(Figure [8](https://arxiv.org/html/2502.15800v3#A1.F8)): Participants see the actual realized price for that period, how many shares were traded in total, and how many shares they traded.
Figure 8: Round Results Page

After the experiment, the risky asset is redeemed for a fixed value of 14 units, representing the fundamental price, calculated using the formula presented in Equation [7](https://arxiv.org/html/2502.15800v3#A2.E7). Note that our experiment paradigm was designed such that this fundamental value remains constant throughout the entirety of the experiment. This enables us to objectively classify when the market is in a “bubble” state, given we always have a constant fundamental value to compare the market price to.

### A.1 Price Formation

We now describe the procedure used in computing the market-clearing price for the call auctions in our experiment. For every session $s$ and round $r$, the market price is determined as the price that maximizes volume Sun et al. (2010). A bid submitted by subject $i$ in round $r$ of session $s$ is a tuple consisting of price and quantity, $(p,q)$. Here, $i=\{1,2,3\}$ indicates that a subject may submit up to 3 bids per round. The set of all bids submitted in a given market session and round is represented by $B_{s,r}$. Likewise, the set of all asks is denoted by $A_{s,r}$, with similar notation for individual asks, $(p,q)$.

Let $P_{s,r}$ be the set of all prices submitted as bids or asks,

$$ $P_{s,r}=\{p\mid(p,q)\in B_{s,r}\cup A_{s,r}\},$ (1) $$

where $p((p,q))$ is a function that returns the price portion of a given order tuple (bid or ask).

Define the cumulative buy quantity of a given price $p$ as the total quantity of all bids at or above that price:

$$ $Q^{B}(p)=\sum_{(p^{\prime},q)\in B_{s,r}}q\cdot\mathbb{1}_{p^{\prime}\geq p},$ (2) $$

where $q((p,q))$ is a function that returns the quantity portion of a given order tuple (bid or ask). This represents the total quantity of shares that all participants are willing to purchase at a given price.

The cumulative sell quantity is likewise defined as:

$$ $Q^{A}(p)=\sum_{(p^{\prime},q)\in A_{s,r}}q\cdot\mathbb{1}_{p^{\prime}\leq p},$ (3) $$

and is the total quantity of shares that all participants are willing to sell at a given price.

For a given price $p$, volume is determined by

$$ $V(p)=\min(Q^{B}(p),Q^{A}(p)).$ (4) $$

The market price is then

$$ $p_{s,r}^{*}=\arg\max_{p\in P_{s,r}}V(p).$ (5) $$

All prices are rounded down to the nearest integer value, a consideration made to simplify the user interface.

If there is no intersection between the bid and ask, the mid-point between the highest bid and the lowest ask is reported as the market price for that period.

### A.2 Compute Details For Reproducibility

For our experiments, we leverage commercially available API endpoints from Anthropic’s Claude-3.5-Sonnet (Anthropic, 2024), Google’s Gemini 1.5 Pro (DeepMind, 2024), Mistral’s Mistral-Large 24.11 (AI, 2024), OpenAI’s GPT-3.5 and GPT-4o (OpenAI, 2025), and xAI’s Grok-2 (xAI, 2024) to underpin the agents trading in our markets.

We note that our results are reproducible only insofar as these models remain accessible via their respective APIs. To contextualize the compute requirements for our experiments, Table [5](https://arxiv.org/html/2502.15800v3#A1.T5) reports the approximate token usage per model. These figures include tokens consumed during pilot runs, which primarily involved the OpenAI and xAI models. Token usage for Gemini 1.5 Pro could not be accessed through the API console.

**Table 5: Approximate token usage across models.**
| Provider | Model | Approx. Token Usage |
| --- | --- | --- |
| Anthropic | Claude-3.5-Sonnet | 17.9M |
| Google DeepMind | Gemini 1.5 Pro | Not Available |
| Mistral | Mistral-Large 24.11 | 18.7M |
| OpenAI | GPT-3.5-turbo-0125 | 30.1M |
| OpenAI | GPT-4o-2024-08-06 | 24.7M |
| xAI | Grok-2 | 33.9M |

## Appendix B Fundamental Value Calculation

For completeness, we present a derivation of the fundamental value (FV) of the asset trading in our experiments. We show that (1) the FV is constant and (2) the FV of the asset matches exactly the redemption value. We assume minimal familiarity with financial economics terminology.

In each period, a unit of stock pays a stochastic dividend drawn i.i.d. from a Bernoulli distribution over $\{0.4,1.0\}$, with equal probability. Therefore, the expected dividend is:

$$ $\mathbb{E}[D_{t}]=0.5\cdot 0.4+0.5\cdot 1.0=0.7$ $$

Cash holdings earn a constant risk-free return of $r=5\%$ per period. Consequently, the present value of receiving a dividend payment $D_{t}$ (or any transfer in general) some $t$ rounds in the future is:

$$ $\frac{D_{t}}{(1+r)^{t}}$ $$

Let $T$ be the final trading period, and suppose the asset is redeemed at time $T$ for a fixed amount $V_{T}=14$. Then the fundamental value of the asset in any round $1\leq t<T$ is the expected discounted value of all future dividends plus the redemption value:

$$ $P_{t}^{*}=\frac{V_{T}}{(1+r)^{T-t}}+\sum_{k=1}^{T-t}\frac{\mathbb{E}[D_{k}]}{(1+r)^{k}}$ (6) $$

While Equation [6](https://arxiv.org/html/2502.15800v3#A2.E6) is elegant in that it builds only off of the present values of expected dividend payments and redemption value, it is not immediately clear why $P_{t}^{*}$ is constant in our setting. We now show this.

$$ $P_{t}^{*}=\frac{V_{T}}{(1+r)^{T-t}}+\mathbb{E}[D_{k}]\sum_{k=1}^{T-t}\frac{1}{(1+r)^{k}}$ $$

We may now compute the standard finite geometric sum.

$$ $P_{t}^{*}=\frac{V_{T}}{(1+r)^{T-t}}+\mathbb{E}[D_{k}]\cdot\frac{1}{1+r}\big(\frac{1-(\frac{1}{1+r})^{T-t}}{\frac{r}{1+r}}\big)$ $$

Manipulating the expression, we obtain the following:

$$ $P_{t}^{*}=\frac{V_{T}}{(1+r)^{T-t}}+\mathbb{E}[D_{k}]\cdot\big(\frac{1-(\frac{1}{1+r})^{T-t}}{r}\big)$ $$

$$ $P_{t}^{*}=\frac{V_{T}}{(1+r)^{T-t}}+\frac{\mathbb{E}[D_{k}]}{r}-\frac{\mathbb{E}[D_{k}]}{r(1+r)^{T-t}}$ $$

Recall that $V_{T}=\mathdollar 14=\frac{\mathdollar 0.7}{0.05}=\frac{\mathbb{E}[D_{k}]}{r}$. Thus, we can complete our simplification to obtain:

$$ $P_{t}^{*}=\frac{\mathbb{E}[D_{k}]}{r}$ (7) $$

Notice that Equation [7](https://arxiv.org/html/2502.15800v3#A2.E7) is time independent. Hence, we have shown (1). Further, given the experiment parameters $P_{t}^{*}=\mathdollar 14$, which is exactly the redemption value, demonstrating (2).

## Appendix C Extended Related Works

Here we provide a more comprehensive overview of prior work at the intersection of LLMs, strategic reasoning, and finance. While the main text highlights only the studies most relevant to our contribution, this appendix details the broader landscape for transparency.

LLMs in strategic environments. Several studies analyze whether LLMs replicate human-like behaviors in canonical game-theory tasks. Brookins & DeBacker (2024) examine GPT-3.5 in the Prisoner’s Dilemma, showing tendencies toward cooperation that exceed typical human baselines. Horton (2023) study LLMs in public goods games, highlighting context-dependent altruism. Akata et al. (2023) explore GPT-4 in iterated games, where it adapts between cooperative and competitive strategies, suggesting emergent long-term planning.

Empirical finance applications. A large and rapidly growing literature investigates LLMs as tools for real-world financial prediction and decision-making. Studies include LLM agents for trading and market making Xiao et al. (2024); Li et al. (2024; 2025); Yu et al. (2024b), financial data analysis Yu et al. (2024a); Yun (2024); Liu & Lo (2025); Ding et al. (2024), multimodal integration Zhang et al. (2024b); Yu et al. (2023); Kim (2025), and forecasting tasks Chen (2025); Zhou et al. (2025); Zhang et al. (2024a); Kang et al. (2024); Siddique et al. (2025); Saqur (2024); Bhat & Jain (2024); Kuruvilla & Mythily (2025); Oprea & Bâra (2024); Shi & Hollifield (2024); Cheng & Chin (2024); Zhao et al. (2024). This literature demonstrates the usefulness of LLMs in financial contexts but focuses on exogenous markets and benchmarks.

Experimental and behavioral finance. Our work differs from the above by situating LLMs in endogenous experimental markets, where prices emerge solely from agent interaction. This connects more directly to the tradition of laboratory asset markets Smith et al. (1988); Bostian & Holt (2009), which have long been used to isolate human biases such as herding, overconfidence, and bubble formation. To our knowledge, ours is the first study to apply this paradigm systematically to LLM-agents.

Evaluation Frameworks for LLMs
Conventional benchmarks for LLMs focus on static environments, emphasizing either knowledge-oriented tasks (e.g., Hendrycks et al. (2021)) or reasoning-oriented challenges (Cobbe et al. (2021)). While these benchmarks provide valuable insights, they often overlook strategic reasoning, a key facet of human intelligence. Recent efforts, such as Chatbot Arena (Zheng et al. (2023)), introduce pairwise comparison frameworks to assess LLM interactions dynamically.

## Appendix D Additional Treatments/Robustness

### D.1 Experienced Sessions

We tested whether prior exposure changes agent behavior via an experienced-session variant of the mono-market. Each model completed two full 30-round sessions with the standard environment (i.i.d. dividends, 5% cash interest, redemption at 14; prices fully endogenous to submitted orders). In Session 2, the prompt additionally provided the complete market history from Session 1 (prices/volumes/clearing outcomes) and the agent’s own PLANS/INSIGHTS files from that session. Chain-of-thought was enabled in both sessions. Dividend paths were re-sampled; agent sampling (e.g., temperature) was not fixed across runs.

Performance in the “experienced” Session 2 was essentially unchanged relative to Session 1: mean absolute deviation from fundamental value, realized profit, and forecast accuracy remained at similar levels for all models. Figure [9](https://arxiv.org/html/2502.15800v3#A4.F9.fig4) shows the price paths for the experienced runs. The incidence and magnitude of bubbles did not show a consistent increase or decrease. In short, exposure to the entire prior market, together with perfect recall of that history, did not materially alter trading outcomes. This suggests that substantive learning (if any) may require changing objectives (e.g., fine-tuning/reward shaping) or the environment (e.g., information asymmetries, cross-asset substitution, transaction costs). Note: Grok 2 has been deprecated, and thus we were unable to run this treatment with that model. Mistral-Large had technical errors that will be rerun for camera-ready submission.

Figure: Figure 9: Experienced Market Trajectories for different LLMs.
Refer to caption: figs/Appendix_Treatments/claude_exp.png

### D.2 Risk Elicitation

We tested whether inserting a standard risk-elicitation task primes LLM trading. For each model, we ran one mono-market session (30 rounds, i.i.d. dividends, 5% cash interest, redemption value at 14) and added Holt–Laury–style lotteries at the same cadence used in our human studies (interleaved prompts between rounds). The task was exogenous: lottery choices did not affect cash balances, inventory constraints, or market clearing. Prompts otherwise matched the baseline; prices remained fully endogenous to submitted orders.

Adding risk-elicitation did not materially change outcomes relative to the baseline per model, as shown in Figure [10](https://arxiv.org/html/2502.15800v3#A4.F10.fig4). Mean absolute deviation from fundamental value, bubble incidence/magnitude, realized profit, turnover, and forecast accuracy were all at similar levels; no consistent directional shift appeared in any model class.

In this environment, brief risk-preference queries do not prime LLM agents toward systematically different trading behavior. This aligns with our human-protocol rationale (the task is orthogonal to trading) and supports the symmetry claim: omitting the lotteries for LLMs in the main experiments does not explain the LLM–human differences we report. Note: Grok 2 has been deprecated, and thus we were unable to run this treatment with that model. Gemini-1.5 had technical errors that will be rerun for camera-ready submission.

Figure: Figure 10: Risk-Elicitation Priming Market Trajectories for different LLMs.
Refer to caption: figs/Appendix_Treatments/gpt4_risk.png

## Appendix E Additional Analysis

### E.1 Human Market Trajectories

Figure: Figure 11: Human Market Trajectories
Refer to caption: figs/05_round_v_price.png

### E.2 Detailed individual market analysis

- •
Gemini 1.5 Pro: Gemini markets seem to generate a reverse bubble trend, where the asset is initially sold off below fundamental and then bid back up as redemption grows closer (Figure [12](https://arxiv.org/html/2502.15800v3#A5.F12)). While overall rationality given by the MSE ($\sim$ 3.1) from the fundamental metric is low, the behavior is the exact opposite of what we have seen in the majority of human experiments and indicates that the model likely does not truly understand the task nor model any biases that humans may exhibit.
Figure 12: Gemini 1.5 Pro Markets
This market seems closest to the Erratic (E) hypothesis. This is also supported by the Pearson correlation coefficient (PCC) as compared to the average human market price of -0.4009, indicating that there are large differences between this model’s performance and that of humans.
We can also analyze the uniformity of the market, as you can see in Figure [13](https://arxiv.org/html/2502.15800v3#A5.F13), compared to other markets, the traders do seem to have a more varied earning profile. This is also shown in the Portfolio Value variance, which is 45.58, higher than all but one of the other single-model markets.
Figure 13: Gemini 1.5 Pro Portfolio Variance
- •
Mistral-Large 24.11 Mistral seems to generate markets that appear mostly like the type of bubbles we’ve seen in our human markets, though the size of the asset bubble is greatly reduced from our median human market and the price crashes below fundamental during the final rounds (Figure [14](https://arxiv.org/html/2502.15800v3#A5.F14)).
Figure 14: Mistral-Large 24.11 Portfolio Markets
This behavior, while not economically rational (purely rational agents would only buy a price below 14 and would sell any price above 14) does correspond somewhat with the behavior we’ve seen out of human players, though the Pearson correlation coefficient (PCC) as compared to the average human market price of -0.1118 are large differences. This negative PCC likely pertains to the behavior in these markets after the bubble crashes. This market seems closest to the Human (H) hypothesis.
We can also analyze the uniformity of the market, as can be seen in the below graph (Figure [15](https://arxiv.org/html/2502.15800v3#A5.F15)), compared to other markets, the traders do seem to have a more varied earning profile. This is also shown in the mean Portfolio Value variance, which is 49.96, the highest of the single-model markets.
Figure 15: Mistral-Large 24.11 Portfolio Variance
- •
Claude-3.5-Sonnet: Claude markets seem to demonstrate highly ‘rational’ behavior, the price remains very close to the fundamental value throughout the market (Figure [16](https://arxiv.org/html/2502.15800v3#A5.F16)). This indicates that the model likely understands the “rational” strategy. This market is the “most” rational based on the MSE metric with the lowest MSE from fundamentals of all the markets ($\sim$ 0.53).
Figure 16: Claude-3.5-Sonnet Markets
This market seems closest to R. Which is further supported by the Pearson correlation coefficient (PCC) as compared to the average human market price of being $\sim$ 0.0013, showing that there are large differences between this model’s performance and that of humans.
We can also analyze the uniformity of the market, as can be seen in the below graph (Figure [17](https://arxiv.org/html/2502.15800v3#A5.F17)), compared to other markets, the traders do seem to have a more uniform earning profile. This is also shown in the mean Portfolio Value variance, which is 26.15, lower than most of the other single-model markets.
Figure 17: Claude-3.5-Sonnet Portfolio Variance
- •
GPT-3.5: Demonstrates bubble behavior without the crash, where prices rise steadily with no indication of a return to fundamental value (Figure [18](https://arxiv.org/html/2502.15800v3#A5.F18)). Unlike human markets, trading volume almost disappears toward the end of the experiment, whereas humans generally trade more as the end approaches. This model is the least “rational” by the MSE metric, having a far and away the highest MSE of all the models ($\sim$ 26.36).
Figure 18: GPT-3.5 Markets
This market seems closest to H, as some bubbles without crashes were observed in human experiments. Interestingly, this model has a relatively high Pearson correlation coefficient (PCC) as compared to the average human market price of $\sim$ 0.4897, but it is clear that there are still large differences between this model’s performance and that of humans.
The Portfolio Value variance for this market is 30.44 (Figure [19](https://arxiv.org/html/2502.15800v3#A5.F19)), one of the lower values among single-model markets, indicating generally uniform behavior.
Figure 19: GPT-3.5 Portfolio Variance
- •
GPT-4o: Demonstrates mostly rational behavior, with prices remaining very close to the fundamental value throughout the market (Figure [20](https://arxiv.org/html/2502.15800v3#A5.F20)). This indicates that the model likely understands the “rational” strategy. This is emphasized quantitatively by the model having the second lowest MSE compared to fundamental at ($\sim$ 0.79). This market seems closest to the Rational (R) hypothesis.
Figure 20: GPT-4o Markets
Additionally, unlike human markets, almost all the agents performed very similarly to each other, indicating that there was not a large amount of behavior variance among the agents. This is also shown in the mean Portfolio Value variance of 22.76 (Figure [21](https://arxiv.org/html/2502.15800v3#A5.F21)), which is the lowest amongst the single-model markets.
Figure 21: GPT-4o Portfolio Variance

## Appendix F Additional Natural Language Processing Information

After each round, LLM agents were prompted to provide details of the strategy they were using for their trading behavior, both to enable us to analyze behavior and also to allow the LLMs to develop long-running strategy making use of Chain-of-Thought-like reasoning. In contrast, Human subjects were asked what approach they employed only at the end of the experiment. All comments gathered were preprocessed by converting to lower case, removing punctuation and new line characters, and removing stop words. (English stop words from Python’s nltk library) For the human subjects, this part was optional. Even so, we received 442 responses from 514 subjects (85.99%), averaging 18.95 words each. The 620 individual LLM agents generated 18,060 responses, averaging 64.75 words each.

First, we looked at the most frequently used words in each group of participants (humans and LLMs). Of the top six words for each group, five are the same, though in a different order (”sell”, ”buy”, ”stock”, ”price”, and ”market”) (Table [6](https://arxiv.org/html/2502.15800v3#A6.T6)). The top fifteen most popular words for the human subject include ”low”, ”high”, and ”early”, which do not appear among the top fifteen words used by the LLMs, while the words ”continue”, ”monitor”, ”closely”, and ”buyback” are among the most popular words for the LLMs. (Popular words are highlighted in word clouds shown in Figure [22](https://arxiv.org/html/2502.15800v3#A6.F22).) This suggests that the human traders might follow a ”buy low, sell high” strategy in an attempt to beat the market, while the LLM agents take a longer investment approach, keeping the fundamental value of the asset in mind.

To confirm this idea, we ran a Latent Dirichlet Allocation (LDA) Blei et al. (2003) on all strategies combined. We used the Python gensim library Řehůřek & Sojka (2010) to assign the strategies to one of two topics. The topic identified by the words orders, sell, buy, shares, stock denoted strategies that implement a ”buy low; sell high” practice. We noticed that $89.3\%$ of human strategies were assigned this topic by the LDA analysis, while only 36.6% of the LLM strategies. A t-test of this difference is highly significant (p $<$ .001). The other topic was identified by the words stock, adjust, buyback, closely, current and suggests a more measured and conservative approach. The LDA analysis assigned $63.4\%$ of the LLM strategies (10.7% of humans) to these topics. Figure [23](https://arxiv.org/html/2502.15800v3#A6.F23) shows this assignment.

**Table 6: Most popular words used by human participants and by the LLM Agents**
|  | Human | LLM-Agent |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Word | Count | Freq | Word | Count | Freq |
| Rank |  |  |  |  |  |  |
| 1 | sell | 230 | 5.60% | price | 88707 | 6.04% |
| 2 | buy | 202 | 4.92% | market | 63445 | 4.32% |
| 3 | stock | 177 | 4.31% | buy | 38996 | 2.66% |
| 4 | price | 141 | 3.43% | sell | 38141 | 2.60% |
| 5 | tried | 129 | 3.14% | orders | 31335 | 2.14% |
| 6 | market | 79 | 1.92% | round | 28385 | 1.93% |
| 7 | low | 75 | 1.83% | stock | 28130 | 1.92% |
| 8 | high | 70 | 1.70% | cash | 26469 | 1.80% |
| 9 | would | 57 | 1.39% | continue | 23318 | 1.59% |
| 10 | interest | 53 | 1.29% | adjust | 19250 | 1.31% |
| 11 | shares | 51 | 1.24% | value | 18669 | 1.27% |
| 12 | strategy | 48 | 1.17% | shares | 18476 | 1.26% |
| 13 | cash | 44 | 1.07% | buyback | 17555 | 1.20% |
| 14 | early | 41 | 1.00% | monitor | 16582 | 1.13% |
| 15 | much | 41 | 1.00% | current | 14595 | 0.99% |

Figure: (a) Human Strategies
Refer to caption: figs/word_cloud_h.png

Figure: Figure 23: Percentage of human and LLM strategies assigned to one of two topics.
Refer to caption: figs/topcis_percent_hvb.png

### F.1 Individual Token counts

#### F.1.1 How the models’ language reveals distinct trading playbooks

##### Method.

For every agent, we concatenate the round-by-round Plan and Insight fields, convert them to lowercase, strip punctuation, and tokenize.

Four interpretable features are then computed:

- 1.
Fundamental token rate – occurrences of *fundamental, intrinsic, value,
mean‑revert, FV, 14, discounted cash‑flow* (14 keywords).
- 2.
Speculative token rate – *pump, breakout, leverage, surge, moon, momentum,
Fomo* (18 keywords).
- 3.
Trade‑verb density – *buy, sell, long, short, liquidate*.
- 4.
Risk $\mathbf{/}$ Profit ratio – risk‑related tokens divided by profit‑related tokens
(*risk, volatility, drawdown* vs. *profit, gain, return*).

Counts are normalized per 1,000 tokens

**Table 7: Language–based strategy fingerprints (plans+insights).**
| Model | Fund. /1000 | Spec. /1000 | Trade verbs/1000 | Risk /Profit |
| --- | --- | --- | --- | --- |
| Claude‑3.5 Sonnet | 61.9 | 4.3 | 21.7 | 1.98 |
| GPT‑4o | 26.0 | 4.1 | 31.5 | 0.90 |
| Gemini-1.5‑Pro | 40.1 | 7.0 | 34.8 | 1.01 |
| Grok‑2 | 29.9 | 5.0 | 17.8 | 2.30 |
| Mistral‑Large | 25.7 | 6.5 | 34.2 | 1.12 |
| GPT‑3.5 | 19.6 | 10.1 | 3.8 | 7.55 |

##### Findings.

- •
Value‑anchored rationalists. Claude‑3.5 (62 fundamental tokens) and GPT‑4o (lowest Risk/Profit) talk most about intrinsic value and trade in balanced lots, keeping price within $1–2 of FV.
- •
Momentum bubble–builders. Grok‑2 and Mistral-Large use twice the speculative language of Claude and issue 18–34 trade verbs per 1,000 → collective buying pushes price up to $20–22 before a late crash.
- •
Risk‑obsessed inflator. GPT‑3.5 has the leanest trade language but a Risk/Profit ratio $\approx$8 ×. Agents buy cautiously, hold, and never realize gains, so the market inflates to $18.8 yet never panics.
- •
Net‑seller undervaluation. Gemini’s speculative share is moderate, but its sell verbs outnumber buys (net-buy = –0.19), driving the only “reverse bubble” that trades below FV.

A simple multinomial logistic regression on these four features classifies the six models with 83% 5-fold CV accuracy (Appendix [F](https://arxiv.org/html/2502.15800v3#A6)), confirming that each family follows a linguistically—and behaviorally—distinct playbook.

### F.2 Sentiment Analysis

Sentiment analysis (or opinion mining) is a natural language processing technique that measures the emotional tone of a passage of text. The aim is to assign a score that is either negative or positive to a passage to express whether it conveys a negative or positive affect. We use nltk (Bird et al., 2009) to perform our analysis. We did not word stem or remove stopwords after Pang et al. (2002).

We ran a sentiment analysis on the strategies and found that human and LLM agents produce a similar distribution of sentiment (Figure [24](https://arxiv.org/html/2502.15800v3#A6.F24)). This similarity might be due to the relatively banal nature of the subject. However, the human subjects did produce a slightly wider variance of sentiment ($\sigma_{h}=0.167>\sigma_{b}=0.131$).

Figure: Figure 24: Distributions of sentiment for human and LLM agents
Refer to caption: figs/sentiment_hist_human_and_bot.png

The strategies of the LLM agents were collected after each round, whereas the human subjects were asked to provide a strategy once after the experiment. This difference allows us to examine the changes in sentiment over time and market conditions. We noticed that sentiment is significantly different when the market price is below the fundamental value (Figure [25](https://arxiv.org/html/2502.15800v3#A6.F25)). We define mispricing as $M_{sr}=MP_{sr}-FV_{s}$ where $M_{sr}$ is the mispricing for session $s$ in round $r$, $MP_{sr}$ is the market price in session $s$ round $r$, and $FV_{s}$ is the fundament value for session $s$ (which is 14 for all sessions). We ran a fixed effects regression of sentiment scores on round number and mispricing with standard errors clustered on the LLM (table [8](https://arxiv.org/html/2502.15800v3#A6.T8)). The results show that the models do tend to adjust the tone of their strategy based on market conditions, even when controlling for the specific LLMs.

**Table 8: Fixed effects regression of sentiment score on round number and level of mispricing. Standard errors are clustered on LLM and shown in parentheses. Sentiment was multiplied by 100 to reduce the significant digits in the estimates.**
|  | Sentiment x 100 |
| --- | --- |
| Mispricing | 0.294^∗∗ |
|  | (0.0674) |
| Round | -0.136^∗∗ |
|  | (0.0275) |
| Constant | 8.684^∗∗∗ |
|  | (0.358) |
| Observations | 18600 |
| Standard errors in parentheses |  |
| ^∗ $p<0.05$, ^∗∗ $p<0.01$, ^∗∗∗ $p<0.001$ |  |

Figure: Figure 25: Sentiment scores are plotted against the mispricing of the market compared to the fundamental value of $14
Refer to caption: figs/sentiment_by_mispricing.png

## Appendix G Rationality Properties of Individual LLM Forecasts

In addition to analyzing the pricing behavior of LLMs, we were also interested in how these models generate forecasts for future price periods. We captured each model’s forecasts, which allowed us to assess not only the rationality of their forecasts, whether they align with fundamental valuation principles, but also whether the LLMs themselves behave rationally in response to their expectations. We compare this analysis to that of our human traders to understand whether LLMs, in the Single-LLM market setting, make similar or different forecasting mistakes compared to their human counterparts. The mathematical formulation for calculating the mean and median forecasts for any given session is provided in the appendix (Appendix [G.3](https://arxiv.org/html/2502.15800v3#A7.SS3)).

### G.1 Forecasting Framework

Fix a session (we will suppress session notation). For each period $t$, participants provide forecasts before the price revelation. The forecast made in period $t$, denoted $f_{i,t}^{t}$, predicts the price $P_{t}$. The forecast for future periods, made in period $t$, is denoted as $f_{i,t}^{(t+h)}$, where $h\in\{0,2,5,10\}$ is the prediction horizon (or how far into the future agents are forecasting).

The forecast error for an individual $i$ in period $t$, forecasting $P_{t+h}$, is given by:
$E_{i,t}^{h}=P_{t+h}-f_{i,t}^{(t+h)}$
A positive value indicates that the agent underforecasted the price, while a negative value indicates an overforecast.
In the appendix, Table [10](https://arxiv.org/html/2502.15800v3#A7.T10) presents the full descriptive statistics of prices and forecasts, while Table [11](https://arxiv.org/html/2502.15800v3#A7.T11) reports the forecast errors for each model and for our human participants.

For next‐round forecasts, the Mistral-Large model achieves the smallest average error (0.00), highlighting its most rational performance. GPT-4o follows with an error of 0.04, Claude-3.5-Sonnet with 0.09, and Gemini-1.5-Pro with 0.14. Grok-2 and GPT-3.5 record errors of –0.31 and –1.54, respectively. By comparison, human participants exhibit a one‐round forecast error of approximately 1.67. Thus, all AI models outperform human forecasters in terms of forecast accuracy.

### G.2 Distribution of Forecast Errors

We investigate the distribution of forecast errors for each AI model across each horizon, grouped by experimental rounds. Figure [26(b)](https://arxiv.org/html/2502.15800v3#A7.F26.sf2) demonstrates that these distributions vary both by model and by prediction horizon.

At the model level, all the models exhibit more tightly centered error distributions around 0 when compared to human markets. While GPT-3.5’s errors are the most dispersed, it has notably more concentrated distributions than human participants. GPT-3.5’s distribution also shows a leftward shift, most notably at the five- and ten-period horizons, indicating an increasing underforecast bias over successive rounds, a pattern that somewhat parallels the trend among human forecasters. The other models do not show this shift.

Next, we compute the mean forecast error for each round and horizon across all AI models and human participants. Figure [5](https://arxiv.org/html/2502.15800v3#S8.F5) shows how this average error evolves over successive rounds. Among the models, Claude-3.5-Sonnet performs best—its mean error remains closest to zero at every horizon. GPT-4o initially overforecasts but steadily converges toward zero in later rounds, reflecting increasing accuracy. By contrast, GPT-3.5 consistently overforecasts as rounds progress, making it the least accurate model. Notably, none of the AI models replicate the human pattern of early underforecasting and then subsequent overforecasting in later rounds.

Figure: Figure 26: Comparison of forecast errors across rounds for human participants and AI models
Refer to caption: figs/appendix_forecast/Figure_Distribution_Forecast_Error_May11_p1.png

#### G.2.1 Unbiasedness

At the individual agent level, we test whether the mean forecast error $E_{i,t}^{h}$ is zero across $t$ for each horizon $h$. This yields four test statistics per agent, corresponding to the four horizons. For each case, we conduct $t$-tests to determine whether the mean forecast error is significantly different from zero, i.e., whether it is biased. For each horizon, we then count the number of agents whose mean forecast error is not significantly different from zero, indicating unbiased forecasts, and compute the percentage of unbiased agents for each model. The first block of Table [9](https://arxiv.org/html/2502.15800v3#A7.T9) presents, for each model and horizon, the percentage of agents whose forecasts are unbiased. The corresponding results are also shown in Figure [31](https://arxiv.org/html/2502.15800v3#A7.F31). Among the models, GPT-3.5’s forecasts are the most biased, as all forecast errors for the four horizons differ from 0. Claude-3.5-Sonnet, Gemini-1.5-Pro perform most similarly to humans; the bias measure decreases as the horizon increases. GPT-4o performs best at horizon = 0, indicating the most rational forecasting and the least bias.

**Table 9: Rationality test results**
| Test | Horizon | Human | Claude-3.5<br>Sonnet | Gemini-1.5<br>Pro | GPT-3.5 | GPT-4o | Grok-2 | Mistral-<br>Large |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Unbiasedness | 0 | 0.619 | 0.833 | 0.778 | 0 | 1.000 | 0.657 | 0.972 |
| 2 | 0.623 | 0.389 | 0.583 | 0 | 0.597 | 0.554 | 0.528 |  |
| 5 | 0.570 | 0.306 | 0.194 | 0 | 0.000 | 0.516 | 0.139 |  |
| 10 | 0.438 | 0.292 | 0.069 | 0 | 0.000 | 0.547 | 0.111 |  |
| Zero autocorrelation | 0 | 0.420 | 0.069 | 0.208 | 0.819 | 0.458 | 0.031 | 0.194 |
| 2 | 0.092 | 0.236 | 0.208 | 0.847 | 0.648 | 0.016 | 0.208 |  |
| 5 | 0.068 | 0.000 | 0.083 | 0.806 | 0.403 | 0.062 | 0.139 |  |
| 10 | 0.086 | 0.028 | 0.208 | 0.861 | 0.486 | 0.109 | 0.097 |  |
| Forecast error<br>uncorrelated<br>with forecast levels | 0 | 0.625 | 0.028 | 0.389 | 0 | 0.056 | 0.438 | 0.403 |
| 2 | 0.349 | 0.308 | 0.431 | 0 | 0.069 | 0.203 | 0.472 |  |
| 5 | 0.145 | 0.175 | 0.586 | 0 | 0.000 | 0.141 | 0.306 |  |
| 10 | 0.117 | 0.123 | 0.643 | 0 | 0.000 | 0.125 | 0.250 |  |

#### G.2.2 Zero Autocorrelation

To test for zero autocorrelation, we examine whether forecast errors from period $t$ should be independent of forecast errors in subsequent periods. The guiding principle is that future forecast errors should not depend on the information available at the time the forecast was made.

To test this, we estimate the regression:
$E_{t,0}=\beta_{0}+\beta_{1}E_{t-1,0}+\epsilon_{t}$, where the hypothesis is $\beta_{1}=0$. This zero-autocorrelation condition extends similarly to longer forecast horizons.

We divide the data at horizon $h=0$ by agent and use ordinary least squares regression to estimate $\beta_{1}$ for each agent at different horizons. For each model and horizon, we then count the number of agents whose $\beta_{1}$ is not significantly different from zero—indicating zero autocorrelation and thus rationality—and calculate the corresponding percentage.

The second block of Table [9](https://arxiv.org/html/2502.15800v3#A7.T9) presents the results of the autocorrelation test (see also Figure [28](https://arxiv.org/html/2502.15800v3#A7.F28)). We see some heterogeneity among the models; GPT-3.5 appears to be the most rational, with the highest percentage of agents showing no autocorrelation. In contrast, Claude-3.5-Sonnet and Grok-2 exhibit the strongest autocorrelation, with the lowest percentage of agents passing the test. Humans show autocorrelation when the horizon is 0, a pattern similar to GPT-4o (humans: 42%, GPT-4o: 45.8%).

#### G.2.3 Correlation Between Forecast Errors and Forecasts

Forecast errors, $P_{t+h}-f_{i,t}^{(t+h)}$, should be uncorrelated with forecasts:
$\text{cor}(P_{t+h}-f_{i,t}^{(t+h)},f_{i,t}^{(t+h)})=0$
This principle is grounded in two key intuitions. First, the current forecast is part of an agent’s information set. If any variable within that set can predict future forecast errors, a rational agent should incorporate it to reduce those errors. Second, a negative correlation between forecast errors and forecasts suggests a systematic bias: when an agent is optimistic, making a high forecast, they tend to overestimate the actual outcome, resulting in a negative forecast error.

To test this, we compute these correlations separately for each forecast horizon and agent by estimating the regression:
$E_{i,t}^{h}=\beta_{0}+\beta_{1}f_{i,t}^{(t+h)}+\epsilon_{i,t}^{h}$

For the four values of $h$, we count the number of agents whose $\beta_{1}$ is not significantly different from zero for each model and horizon, indicating no correlation between forecast errors and forecasts, which is consistent with rationality.

The third block of Table [9](https://arxiv.org/html/2502.15800v3#A7.T9) reports the results of the correlation test across different horizons (see also Figure [33](https://arxiv.org/html/2502.15800v3#A7.F33)). When the horizon is 0, humans perform the best, followed by Grok-2. In contrast, GPT-3.5 and GPT-4o perform the worst, as their forecast errors are consistently correlated with their forecasts across nearly all horizons.

In summary, LLMs generally produce more stable and centered forecast errors than humans. GPT-3.5 shows high bias and strong forecast-error correlation but low autocorrelation. Claude-3.5-Sonnet and Gemini-1.5-Pro are closest to human behavior, while GPT-4o performs well at short horizons. Each model has its strengths and weaknesses, but no model fully replicates human behavior across all tests.

**Table 10: Descriptive statistics for Price and Forecasts by source.**
| Source | Variable | Mean | SD | Min | Max | Q1 | Q3 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| claude-3.5-sonnet | Price | 14.65 | 1.00 | 13.00 | 20.00 | 14.00 | 15.00 |
|  | Forecast current | 14.67 | 0.98 | 13.00 | 20.00 | 14.00 | 15.00 |
|  | Forecast 2 rounds later | 14.78 | 1.12 | 14.00 | 21.00 | 14.00 | 15.00 |
|  | Forecast 5 rounds later | 14.90 | 1.49 | 14.00 | 22.00 | 14.00 | 16.00 |
|  | Forecast 10 rounds later | 15.13 | 2.04 | 14.00 | 25.00 | 14.00 | 16.00 |
| gemini-1.5-pro | Price | 13.43 | 2.02 | 7.00 | 20.00 | 13.00 | 14.50 |
|  | Forecast current | 13.68 | 1.91 | 6.00 | 21.00 | 13.00 | 15.00 |
|  | Forecast 2 rounds later | 13.87 | 1.77 | 7.00 | 22.00 | 14.00 | 15.00 |
|  | Forecast 5 rounds later | 14.26 | 1.48 | 8.00 | 23.00 | 14.00 | 15.00 |
|  | Forecast 10 rounds later | 14.55 | 1.11 | 9.00 | 22.00 | 14.00 | 15.00 |
| gpt-3.5 | Price | 16.48 | 0.92 | 14.00 | 20.00 | 16.00 | 17.00 |
|  | Forecast current | 17.98 | 2.15 | 14.00 | 35.00 | 17.00 | 18.00 |
|  | Forecast 2 rounds later | 19.92 | 2.06 | 14.00 | 38.00 | 19.00 | 21.00 |
|  | Forecast 5 rounds later | 22.69 | 2.42 | 15.00 | 45.00 | 21.00 | 24.00 |
|  | Forecast 10 rounds later | 25.28 | 2.76 | 17.00 | 55.00 | 24.00 | 27.00 |
| gpt-4o | Price | 14.94 | 0.87 | 14.00 | 20.00 | 15.00 | 15.00 |
|  | Forecast current | 14.95 | 0.91 | 14.00 | 21.00 | 15.00 | 15.00 |
|  | Forecast 2 rounds later | 15.20 | 1.10 | 13.00 | 22.00 | 15.00 | 15.00 |
|  | Forecast 5 rounds later | 15.98 | 1.45 | 14.00 | 25.00 | 15.00 | 16.00 |
|  | Forecast 10 rounds later | 17.06 | 2.10 | 14.00 | 28.00 | 16.00 | 18.00 |
| grok-2 | Price | 18.17 | 2.13 | 14.00 | 22.00 | 16.00 | 20.00 |
|  | Forecast current | 18.15 | 2.45 | 14.00 | 23.00 | 16.00 | 20.00 |
|  | Forecast 2 rounds later | 18.39 | 2.64 | 14.00 | 24.00 | 16.00 | 20.00 |
|  | Forecast 5 rounds later | 18.90 | 2.76 | 14.00 | 26.00 | 17.00 | 21.00 |
|  | Forecast 10 rounds later | 19.07 | 3.02 | 14.00 | 28.00 | 17.00 | 21.75 |
| mistral-large | Price | 15.17 | 2.09 | 12.00 | 20.00 | 14.00 | 16.00 |
|  | Forecast current | 15.34 | 2.21 | 10.00 | 22.00 | 14.00 | 16.00 |
|  | Forecast 2 rounds later | 15.77 | 2.45 | 9.00 | 23.00 | 14.00 | 17.00 |
|  | Forecast 5 rounds later | 16.60 | 2.75 | 9.00 | 25.00 | 14.00 | 18.00 |
|  | Forecast 10 rounds later | 17.76 | 3.23 | 14.00 | 27.00 | 15.00 | 19.00 |
| Human | Price | 31.31 | 16.42 | 12.00 | 102.00 | 19.00 | 40.00 |
|  | Forecast current | 15.50 | 8.66 | 1.00 | 30.00 | 8.00 | 23.00 |
|  | Forecast 2 rounds later | 16.50 | 8.08 | 3.00 | 30.00 | 9.75 | 23.25 |
|  | Forecast 5 rounds later | 18.00 | 7.21 | 6.00 | 30.00 | 12.00 | 24.00 |
|  | Forecast 10 rounds later | 20.50 | 5.77 | 11.00 | 30.00 | 15.75 | 25.25 |

**Table 11: Descriptive statistics for Forecast errors by source.**
| Source | Variable | Mean | SD | Min | Max | Q1 | Q3 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| claude-3.5-sonnet | Forecast error current | 0.09 | 0.49 | -2.00 | 3.00 | 0.00 | 0.00 |
|  | Forecast error 2 rounds | -0.01 | 0.68 | -2.00 | 3.00 | 0.00 | 0.00 |
|  | Forecast error 5 rounds | -0.12 | 1.10 | -4.00 | 4.00 | -1.00 | 1.00 |
|  | Forecast error 10 rounds | -0.38 | 1.52 | -7.00 | 3.00 | -1.00 | 1.00 |
| gemini-1.5-pro | Forecast error current | 0.14 | 0.88 | -5.00 | 6.00 | 0.00 | 0.50 |
|  | Forecast error 2 rounds | -0.12 | 0.84 | -4.00 | 4.00 | -1.00 | 0.00 |
|  | Forecast error 5 rounds | -0.64 | 1.36 | -5.00 | 5.00 | -1.00 | 0.00 |
|  | Forecast error 10 rounds | -1.23 | 1.69 | -6.00 | 6.00 | -2.00 | 0.00 |
| gpt-3.5 | Forecast error current | -1.54 | 1.92 | -16.00 | 2.00 | -2.00 | -0.50 |
|  | Forecast error 2 rounds | -3.42 | 1.73 | -18.00 | 0.50 | -4.00 | -2.50 |
|  | Forecast error 5 rounds | -6.12 | 2.24 | -26.00 | 1.50 | -7.00 | -5.00 |
|  | Forecast error 10 rounds | -8.64 | 2.68 | -39.00 | 0.00 | -10.00 | -7.00 |
| gpt-4o | Forecast error current | 0.04 | 0.41 | -2.00 | 2.50 | 0.00 | 0.00 |
|  | Forecast error 2 rounds | -0.19 | 0.46 | -2.00 | 2.00 | 0.00 | 0.00 |
|  | Forecast error 5 rounds | -0.94 | 0.94 | -6.00 | 1.00 | -1.00 | 0.00 |
|  | Forecast error 10 rounds | -2.05 | 1.68 | -11.50 | 2.00 | -3.00 | -1.00 |
| grok-2 | Forecast error current | -0.31 | 1.37 | -3.00 | 6.00 | -1.00 | 0.00 |
|  | Forecast error 2 rounds | -0.31 | 1.49 | -3.00 | 6.00 | -1.50 | 0.00 |
|  | Forecast error 5 rounds | -0.42 | 1.67 | -5.00 | 5.00 | -2.00 | 1.00 |
|  | Forecast error 10 rounds | 0.00 | 2.53 | -7.50 | 7.00 | -2.00 | 2.00 |
| mistral-large | Forecast error current | 0.00 | 1.04 | -4.00 | 4.00 | 0.00 | 0.00 |
|  | Forecast error 2 rounds | -0.36 | 0.95 | -4.00 | 3.00 | -1.00 | 0.00 |
|  | Forecast error 5 rounds | -1.14 | 1.50 | -9.00 | 4.00 | -2.00 | 0.00 |
|  | Forecast error 10 rounds | -2.38 | 2.96 | -14.00 | 6.00 | -3.00 | 0.00 |
| Human | Forecast error current | 1.67 | 8.70 | -84.00 | 92.00 | -1.00 | 2.00 |
|  | Forecast error 2 rounds | 1.69 | 11.12 | -79.00 | 95.00 | -3.00 | 6.00 |
|  | Forecast error 5 rounds | -0.10 | 17.49 | -149.00 | 96.00 | -7.00 | 9.00 |
|  | Forecast error 10 rounds | -0.53 | 25.44 | -178.00 | 96.00 | -12.00 | 15.00 |

Figure: Figure 27: Percentage of Forecasts Without Autocorrelation Bias for Each Model Across Horizons
Refer to caption: figs/appendix_forecast/Figure_Unbiased_May11.png

Figure: Figure 28: Percentage of Unbiased Forecasts for Each Model Across Horizons
Refer to caption: figs/appendix_forecast/Figure_ZeroAutocorrelation_May11.png

Figure: Figure 29: Results Showing the Correlation Between Forecast Errors and Forecasts
Refer to caption: figs/appendix_forecast/Figure_UncorrelatedForecastLevel_May11.png

### G.3 Additional Forecast Information

#### G.3.1 Mean and Median Forecasts

The mean forecast for session $s$ at period $t+h$ is:

$$ $m_{t}^{(t+h)}=\frac{\sum_{i=1}^{N_{s}}f_{i,t}^{(t+h)}}{N_{s}}$ $$

Where $N_{s}$ is the number of participants in session $s$.

The median forecast is calculated as follows:
Arrange the values of $f_{t,t+h}^{(i)}$ in ascending order:

$$ $f_{t,t+h}^{(1)},f_{t,t+h}^{(2)},\dots,f_{t,t+h}^{(N_{s})}$ $$

where

$$ $f_{t,t+h}^{(1)}\leq f_{t,t+h}^{(2)}\leq\dots\leq f_{t,t+h}^{(N_{s})}$ $$

If $N_{s}$ is odd:

$$ $\text{Median}=f_{t,t+h}^{\left(\frac{N_{s}+1}{2}\right)}$ $$

If $N_{s}$ is even:

$$ $\text{Median}=\frac{f_{t,t+h}^{(N_{s}/2)}+f_{t,t+h}^{(N_{s}/2+1)}}{2}$ $$

#### G.3.2 Additional Distribution of Forecast Errors Graph

Figure: Figure 30: Additional Forecast Error Distribution Graph
Refer to caption: figs/appendix_forecast/Distribution_of_Forecast_Error_by_Model_and_Round.png

### G.4 Median Forecast Error

We calculate the median forecast error for each round across different horizons. Figure 5 in the main text illustrates how the median forecast error evolves over time for each horizon. Claude-3.5-Sonnet performs the best, as its mean forecast for each horizon remains closest to 0 compared to other models. GPT-4o shows a trend of initially overforecasting prices before gradually shifting toward 0 in later rounds, indicating a movement toward greater accuracy. In contrast, GPT-3.5 performs the worst, consistently overforecasting prices as the rounds progress.

#### G.4.1 Unbiasedness

At the individual agent level, we test whether the mean forecast error $E_{i,t}^{h}$ is zero across $t$ for each horizon $h$. This provides four test statistics for each value of $h$. We conduct $t$-tests to determine whether the mean forecast error is statistically different from zero.

Specifically, we test whether the mean forecast error for each model and horizon is significantly different from zero. We use the t test at agent level for four horizons and get the p value. Then we count the number of agents whose mean forecast error is significantly different from 0 at each horizon, and calculate the percentage of each model. Figure [31](https://arxiv.org/html/2502.15800v3#A7.F31) presents the percentage of no difference with 0 of each model at each horizon. Among the models, gpt-3.5 performs worst, as all forecast error for four horizons are all different from 0, Claude-3.5-Sonnet, Gemini-1.5-Pro performs most similarly to humans, the ratio gets lower as horizon increase. GPT-4o performs best at horizon = 0, indicating rational forecasting.

Figure: Figure 31: The percentage of no autocorrelation bias of each model at each horizon
Refer to caption: figs/appendix_forecast/Figure_Unbiased_May11.png

##### Zero Autocorrelation

To test for zero autocorrelation, we examine whether forecast errors from period $t$ should be independent of forecast errors in subsequent periods. The guiding principle is that future forecast errors should not depend on the information available at the time the forecast was made.

For example, the forecast error in period 8 is:

$$ $E_{i,8}^{0}=P_{8}-f_{i,8}^{8}$ $$

This error should be uncorrelated with the forecast error in period 9:

$$ $P_{9}-f_{i,9}^{9}$ $$

A similar zero-autocorrelation principle applies for longer horizons. To test this, we estimate the regression:

$$ $E_{t,0}=\beta_{0}+\beta_{1}E_{t-1,0}+\epsilon_{t}$ $$

where the hypothesis is $\beta_{1}=0$.

We divide the data of horizon $h=0$ by agent and use ordinary least squares (OLS) regression to estimate $\beta_{1}$ at the agent level for different horizons. We then count the number of agents whose $\beta_{1}$ is not significantly different from 0 ($p$-value $>0.05$) for each model and each horizon and calculate its percentage.

Figure [32](https://arxiv.org/html/2502.15800v3#A7.F32) presents the results of the autocorrelation test. Among the models, GPT-3.5 appears the most rational, as it has the highest percentage of agents showing no autocorrelation. In contrast, Claude-3.5-Sonnet and Grok-2 exhibit the most autocorrelation, with the lowest percentage of agents passing the test.

Figure: Figure 32: The results of the autocorrelation test.
Refer to caption: figs/appendix_forecast/Figure_ZeroAutocorrelation_May11.png

##### Correlation Between Forecast Errors and Forecasts

Forecast errors, $P_{t+h}-f_{i,t}^{(t+h)}$, should be uncorrelated with forecasts:

$$ $\text{cor}(P_{t+h}-f_{i,t}^{(t+h)},f_{i,t}^{(t+h)})=0$ $$

This principle is based on two intuitions:
1. The current forecast is part of an agent’s information set. If something in their information set can predict future errors, they should use it to reduce the error.
2. A negative correlation between forecast errors and forecasts implies that when an agent is optimistic (high $f_{i,t}^{(t+h)}$), they tend to be too optimistic (negative $P_{t+h}-f_{i,t}^{(t+h)}$).

To test this, we compute these correlations separately for each forecast horizon and agent by estimating the regression:

$$ $E_{i,t}^{h}=\beta_{0}+\beta_{1}f_{i,t}^{(t+h)}+\epsilon_{i,t}^{h}$ $$

for the four values of $h$. We then count the number of agents whose $\beta_{1}$ is not significantly different from zero ($p$-value $>0.05$) for each model and horizon, indicating no correlation between forecast errors and forecasts.

Figure [33](https://arxiv.org/html/2502.15800v3#A7.F33) presents the results of the correlation test across different horizons and models. In this test, Gemini-1.5-Pro performs best, as it has the highest percentage across all four horizons compared to other models. In contrast, GPT-3.5 and GPT-4o perform worse, as most agents from these models exhibit forecast errors that are correlated with forecast levels.

Figure: Figure 33: Results of the correlation between forecast error and forecast
Refer to caption: figs/appendix_forecast/autocorrelation.png

**Table 12: Descriptive statistics and forecast errors for each model and humans.**
| Source | Variable | Mean | SD | Min | Max | Q1 | Q3 |
| --- | --- | --- | --- | --- | --- | --- | --- |
| claude-3.5-sonnet | Price | 14.65 | 1.00 | 13.00 | 20.00 | 14.00 | 15.00 |
|  | Forecast current | 14.67 | 0.98 | 13.00 | 20.00 | 14.00 | 15.00 |
|  | Forecast t+2 | 14.78 | 1.12 | 14.00 | 21.00 | 14.00 | 15.00 |
|  | Forecast t+5 | 14.90 | 1.49 | 14.00 | 22.00 | 14.00 | 16.00 |
|  | Forecast t+10 | 15.13 | 2.04 | 14.00 | 25.00 | 14.00 | 16.00 |
|  | Forecast error current | 0.09 | 0.49 | -2.00 | 3.00 | 0.00 | 0.00 |
|  | Forecast error t+2 | -0.01 | 0.68 | -2.00 | 3.00 | 0.00 | 0.00 |
|  | Forecast error t+5 | -0.12 | 1.10 | -4.00 | 4.00 | -1.00 | 1.00 |
|  | Forecast error t+10 | -0.38 | 1.52 | -7.00 | 3.00 | -1.00 | 1.00 |
| gemini-1.5-pro | Price | 13.43 | 2.02 | 7.00 | 20.00 | 13.00 | 14.50 |
|  | Forecast current | 13.68 | 1.91 | 6.00 | 21.00 | 13.00 | 15.00 |
|  | Forecast t+2 | 13.87 | 1.77 | 7.00 | 22.00 | 14.00 | 15.00 |
|  | Forecast t+5 | 14.26 | 1.48 | 8.00 | 23.00 | 14.00 | 15.00 |
|  | Forecast t+10 | 14.55 | 1.11 | 9.00 | 22.00 | 14.00 | 15.00 |
|  | Forecast error current | 0.14 | 0.88 | -5.00 | 6.00 | 0.00 | 0.50 |
|  | Forecast error t+2 | -0.12 | 0.84 | -4.00 | 4.00 | -1.00 | 0.00 |
|  | Forecast error t+5 | -0.64 | 1.36 | -5.00 | 5.00 | -1.00 | 0.00 |
|  | Forecast error t+10 | -1.23 | 1.69 | -6.00 | 6.00 | -2.00 | 0.00 |
| gpt-3.5 | Price | 16.48 | 0.92 | 14.00 | 20.00 | 16.00 | 17.00 |
|  | Forecast current | 17.98 | 2.15 | 14.00 | 35.00 | 17.00 | 18.00 |
|  | Forecast t+2 | 19.92 | 2.06 | 14.00 | 38.00 | 19.00 | 21.00 |
|  | Forecast t+5 | 22.69 | 2.42 | 15.00 | 45.00 | 21.00 | 24.00 |
|  | Forecast t+10 | 25.28 | 2.76 | 17.00 | 55.00 | 24.00 | 27.00 |
|  | Forecast error current | -1.54 | 1.92 | -16.00 | 2.00 | -2.00 | -0.50 |
|  | Forecast error t+2 | -3.42 | 1.73 | -18.00 | 0.50 | -4.00 | -2.50 |
|  | Forecast error t+5 | -6.12 | 2.24 | -26.00 | 1.50 | -7.00 | -5.00 |
|  | Forecast error t+10 | -8.64 | 2.68 | -39.00 | 0.00 | -10.00 | -7.00 |
| gpt-4o | Price | 14.94 | 0.87 | 14.00 | 20.00 | 15.00 | 15.00 |
|  | Forecast current | 14.95 | 0.91 | 14.00 | 21.00 | 15.00 | 15.00 |
|  | Forecast t+2 | 15.20 | 1.10 | 13.00 | 22.00 | 15.00 | 15.00 |
|  | Forecast t+5 | 15.98 | 1.45 | 14.00 | 25.00 | 15.00 | 16.00 |
|  | Forecast t+10 | 17.06 | 2.10 | 14.00 | 28.00 | 16.00 | 18.00 |
|  | Forecast error current | 0.04 | 0.41 | -2.00 | 2.50 | 0.00 | 0.00 |
|  | Forecast error t+2 | -0.19 | 0.46 | -2.00 | 2.00 | 0.00 | 0.00 |
|  | Forecast error t+5 | -0.94 | 0.94 | -6.00 | 1.00 | -1.00 | 0.00 |
|  | Forecast error t+10 | -2.05 | 1.68 | -11.50 | 2.00 | -3.00 | -1.00 |
| grok-2 | Price | 18.17 | 2.13 | 14.00 | 22.00 | 16.00 | 20.00 |
|  | Forecast current | 18.15 | 2.45 | 14.00 | 23.00 | 16.00 | 20.00 |
|  | Forecast t+2 | 18.39 | 2.64 | 14.00 | 24.00 | 16.00 | 20.00 |
|  | Forecast t+5 | 18.90 | 2.76 | 14.00 | 26.00 | 17.00 | 21.00 |
|  | Forecast t+10 | 19.07 | 3.02 | 14.00 | 28.00 | 17.00 | 21.75 |
|  | Forecast error current | -0.31 | 1.37 | -3.00 | 6.00 | -1.00 | 0.00 |
|  | Forecast error t+2 | -0.31 | 1.49 | -3.00 | 6.00 | -1.50 | 0.00 |
|  | Forecast error t+5 | -0.42 | 1.67 | -5.00 | 5.00 | -2.00 | 1.00 |
|  | Forecast error t+10 | 0.00 | 2.53 | -7.50 | 7.00 | -2.00 | 2.00 |
| mistral-large | Price | 15.17 | 2.09 | 12.00 | 20.00 | 14.00 | 16.00 |
|  | Forecast current | 15.34 | 2.21 | 10.00 | 22.00 | 14.00 | 16.00 |
|  | Forecast t+2 | 15.77 | 2.45 | 9.00 | 23.00 | 14.00 | 17.00 |
|  | Forecast t+5 | 16.60 | 2.75 | 9.00 | 25.00 | 14.00 | 18.00 |
|  | Forecast t+10 | 17.76 | 3.23 | 14.00 | 27.00 | 15.00 | 19.00 |
|  | Forecast error current | 0.00 | 1.04 | -4.00 | 4.00 | 0.00 | 0.00 |
|  | Forecast error t+2 | -0.36 | 0.95 | -4.00 | 3.00 | -1.00 | 0.00 |
|  | Forecast error t+5 | -1.14 | 1.50 | -9.00 | 4.00 | -2.00 | 0.00 |
|  | Forecast error t+10 | -2.38 | 2.96 | -14.00 | 6.00 | -3.00 | 0.00 |
| Human | Price | 31.31 | 16.42 | 12.00 | 102.00 | 19.00 | 40.00 |
|  | Forecast current | 15.50 | 8.66 | 1.00 | 30.00 | 8.00 | 23.00 |
|  | Forecast t+2 | 16.50 | 8.08 | 3.00 | 30.00 | 9.75 | 23.25 |
|  | Forecast t+5 | 18.00 | 7.21 | 6.00 | 30.00 | 12.00 | 24.00 |
|  | Forecast t+10 | 20.50 | 5.77 | 11.00 | 30.00 | 15.75 | 25.25 |
|  | Forecast error current | 1.67 | 8.70 | -84.00 | 92.00 | -1.00 | 2.00 |
|  | Forecast error t+2 | 1.69 | 11.12 | -79.00 | 95.00 | -3.00 | 6.00 |
|  | Forecast error t+5 | -0.10 | 17.49 | -149.00 | 96.00 | -7.00 | 9.00 |
|  | Forecast error t+10 | -0.53 | 25.44 | -178.00 | 96.00 | -12.00 | 15.00 |

## Appendix H Human Experiment Details

### H.1 Design Details

In the base experiment, all human participants saw the exact same instructions; there were no treatment variables. The study was conducted using a web-based experimental software developed by one of the authors.

There were two types of subjects participating in each market: in-lab participants, collected at the university, and online participants using Prolific. For each market, the in-lab participants had biometric data measured using Empatica E4 SCR devices and newer (acquired April 2024) Empatica EmbracePlus devices. We do not analyze any biometric data in this paper.

Before participating in the experiment:

- 1.
Subjects sign a consent form agreeing to participate in the experiment.
- 2.
Subjects fill in a short socio-demographic questionnaire where they provide information on their age, gender, study background, work experience, and level of English. The full survey is shown in the appendix.
- 3.
Subjects watch a video with the experimental instructions on how to trade in the software. Participants were provided with written instructions and had to respond to 5 comprehension questions on the experimental software. If participants could not answer at least one question correctly, they were unable to participate. This was done to ensure that we only allowed participants who understood the instructions of the experiment. The full quiz is shown in the appendix.
- 4.
Participants then logged in to the market to perform the actual trading experiment. In-lab participants will have their biometric data measured using either an Empatica E4 SCR device or a newer (acquired April 2024) Empatica EmbracePlus device.
- 5.
Participants were then asked to respond to a post-experimental survey, where they were asked to qualitatively outline their strategy. The full survey is shown in the appendix.

Subjects received payments for their participation:

- •
A base payment of $20 and an average bonus of approx. $10
- •
Bonus payment is calculated based on: (i) participants’ trading results at a rate of 200 experimental CASH units to $1, (ii) forecasting ability (i.e. each price forecast that is within 2.5 units of actual price is rewarded with 5 CASH), (iii) a randomly chosen lottery that participants select from the experiment and (iv) the quiz result (subject receive $0.25 for each of the 5 comprehension questions answered correctly.

Sample size:
We collected data from 19 markets, with an average of 18 subjects/market, resulting in a total of 342 participants. In each market session, we collected biometric data from 3-5 in-lab participants.

### H.2 Instructions

Instructions were given by a video which can be viewed on YouTube: [REDACTED FOR DBL REVIEW]. Below is the transcript of those instructions.

## Appendix I Prompts

### I.1 Additional Prompting Considerations

In adapting the task to LLM-agents, our primary goal was to imitate as best as possible the experiment as presented to human subjects. To this end, the description of the task setting mirrors the original instruction video, which details both the experiment premise as well as how to participate. Additional instructions are either specific to the text-based format of the market for LLM agents or general tips aimed at ensuring that agents understand the setup. They are told that their primary objective is to maximize total earnings. To reduce the necessary computation for the task, we make the following adjustments to the “screen” sequence for human subjects outlined in Appendix [A](https://arxiv.org/html/2502.15800v3#A1):

- 1.
We combine the Order Submission and Forecast screens
- 2.
We defer presenting round results until order entry of the following round

As no additional market data is revealed between the Order Submission and Forecast screens, modification (1) does not meaningfully change the experimental design. Similarly, modification (2) still provides updated market data before querying future market decisions, and so does not impact the setting. Thus, every round of the experiment, each agent simultaneously submits orders to the market as well as expectations regarding current and future market prices. At each round, agents enter their market orders (if any) and price forecasts by filling in the provided JSON template.

We also include a “Practice Reflection” portion of the experiment, which invites agents to share specific insights and/or lessons from the practice rounds to be accessible for the remainder of the experiment. Such emulates the learning focus of the practice rounds for human subjects as well as the rhetorical questions posed after the practice rounds. Following the experiment, agents are also asked to complete a “Final Reflection” to distill down any key lessons learned through their participation.

### I.2 Portfolio Data Format

### I.3 Standard Prompt Shell

### I.4 Market Data Format

### I.5 Fundamental Value Shock Treatment Prompt

For the two fundamental value shock treatments, we retain the same prompts, with the exception that from Round 15 onwards, the possible dividend outcomes as well as the redemption value are adjusted. Models are not informed of this change prior to Round 15 and thus, this operates as a ”shock” to all participants.

Mechanically we inject the following prompt after the round recap screen is shown before Round 15.

## Peer Reviews

confidence: 4
contribution: 4
initial_rating: 6
presentation: 4
questions: In Section 8, it seems like the prediction task the LLMs face is easier, because the prices in the LLM markets exhibit less volatility than the prices in the human markets. Is there possibly a way to control for this? For example, perhaps one could task each LLM with the prediction tasks of all the other markets (including the human markets).
soundness: 4
strengths: The experimental design allows for a sensible comparison of human versus LLM behavior. Even though the LLMs tested are relatively outdated at this point, the fact that a broad array of different LLMs from different providers are tested, and the separation between human and LLM behavior is so clear, means the results strike me as credible and generalizable.
summary: This paper compares the trading behavior of humans and an array of LLMs in an experimental asset market. Subjects trade the asset with each other, and the asset price depends on trading activity, allowing for the possibility of bubble formation. The paper finds that the LLMs it tests are less prone to bubble formation than human traders, this result holds both in a "mono-agent" (only copies of agents based on the same LLM) and in a "battle royale" (different LLMs competing) setting. Moreover, the two most capable LLMs tested (GPT-4o, Claude 3.5 Sonnet) exhibit the least tendency towards bubble formation. The paper also conducts a textual analysis to uncover reasons behind differences in LLM and human trading behavior.
weaknesses: The textual trading strategy analysis is interesting but perhaps a little rudimentary, focusing mostly on keyword matching. Moreover, the results at the start of Section 6.1 (that the LLMs and humans all write in different styles) is not really surprising. This analysis would be strengthened by, e.g. (1) a more fine-grained semantic text analysis, (2) additional experiments in the style of Section 7, e.g. checking how LLMs' trading behavior changes if the content of the insight/plan part of the prompt changes.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Would a larger LLM (e.g., Claude 3 Opus) or a modified RLHF model exhibit bubble-like behavior?

How do the results apply to markets involving high-frequency trading or multi-asset portfolios?

Does the human participants' prior financial knowledge influence the results?

Can LLMs more accurately simulate inexperienced traders?
soundness: 3
strengths: 1.  First systematic comparison of LLMs and humans in endogenous experimental markets, bridging behavioral finance and AI alignment

2. Rigorous methodology: a lot of markets vs. 6 LLM models (Claude-3.5, GPT-4o, etc.), with controls for dividend shocks and experience.

3. Well-structured with clear visualizations and statistical tests.

4. Challenges the use of off-the-shelf LLMs as human proxies in finance experiments.
summary: Experimental Design: Conducts homogeneous (single-model) and heterogeneous ("Battle Royale") LLM-agent markets.

Behavioral Divergence: Demonstrates that LLMs exhibit "textbook-rational" behavior, while humans consistently generate price bubbles.

Robustness Tests: Validates findings across dividend shocks, experienced sessions, and linguistic analysis of trading strategies.

Forecast Rationality: Shows LLMs produce more accurate and less biased price forecasts than humans.
weaknesses: Are larger models (e.g., GPT-4 Turbo, Claude 3 Opus) and LLMs that may exhibit different behaviors excluded?

Why LLMs are anchored to fundamentals is not explored.

The simplified single-asset design lacks real-world characteristics (e.g., short selling, information asymmetry).

The incentives of human participants (e.g., monetary rewards) may not align with the "profit maximization" imperative of LLMs.

The impact of imperative engineering (e.g., explicit bubble-inducing directives) is not explored.

---

confidence: 4
contribution: 1
initial_rating: 2
presentation: 2
questions: - LLMs have inherent prompt sensitivity. Is there any ablations on how that will affect the results (e.g. one counterargument from the papers that 'claim' to use LLM for human simulation might say that they are explicitly asking LLM to take on human persona and roles - will that change behaviors systematically)

- Can the authors articulate the contribution beyond documenting "LLMs look rational under explicit FV prompts"? What is the core conceptual/empirical advance for experimental finance or agent-based markets, and how does it differ from prior work and adds new insights to the community?

- Results may hinge on PLANS/INSIGHTS + CoT and looser time budgets for models. Is there results on no-memory/no-cot agents, or, another extreme, on more up-to-date advanced reasoning models?
soundness: 2
strengths: The paper covers several topics, there's comparison to human baselines and careful analysis.
summary: The paper conducted LLM experiments on experimental finance settings where there's a known fundamental value and compare it with human experimental data. The paper finds that LLM stick close to fundamentals and rarely form large bubbles, which departs from human behaviors. The paper also tested settings on battle royale markes and forecasts, showing LLM-human differences.
weaknesses: I'm a bit unsold on the motivation and contributions.

- First of all, the exact types of experimental data and 'correct'/'expected' behavior is well within the training data of the LLM. In addition, prompts in the experiment disclose redemption value / fundamental value mechanisms, so it is largely unsurprising to observe the discovered LLM phenomenon. In other words, "textbook-rational" seem unsurprising given that LLMs are trained on the said textbooks, as well as the CoT reasoning paradigm that they are trained to use. 
- Given the above, I’m not sure that the mere discovery of LLM "rationality" is new or persuasive for this community. Nowadays, models already hit near–top human accuracy on competitive math (gold medal in IMO) and similar benchmarks; so the fact that an LLM looks more rational than the average human-especially when the task is framed explicitly with a redemption value-doesn’t feel surprising. I do see value in documenting this within experimental finance, but the acceptance bar likely requires more than this. 
- Most experiments are run on (or include) older model generations. In line 1028 the authors noted that "Grok 2 has been deprecated, and
thus we were unable to run this treatment with that model. Gemini-1.5 had technical errors that will be rerun for camera-ready submission". I think especially given the framing of the papers in line 450, any deployment in real markets will minimally use one of the up to date reasoning models. 

Some minor points:
- Humans face strict per-screen time limits (20 seconds to trade and 30 seconds to forecast). There seem to be a difficulty translating that to time and budget for LLMs. In the prompt used in the paper, the authors described (line 971) "PLANS/INSIGHTS" files that agents make on top of chain of thought. That's a notable difference in experimental conditions.
- There are some inconsistent findings across models (e.g., Gemini-1.5 Pro vs. GPT-3.5 vs. GPT-4o), suggesting model-specific idiosyncrasies that may not hold across generations. The paper’s own summary tables/plots show materially different error patterns and dynamics by model.
- The paper also notes "Token usage for Gemini 1.5 Pro could not be accessed through the API console". This is a minor point but is not consistent with my experience with using it.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: See the weaknesses.
soundness: 2
strengths: 1. The findings show clear behavioral differences: LLM-driven markets exhibit far more “textbook‑rational” pricing than human‑driven markets.
2. The paper shows that LLMs incorporate less bias into their price forecasts and rely more on fundamental‑value strategies, unlike the heuristic‑driven approaches common among human traders.
3. These results challenge the assumption that “out‑of‑the‑box” LLMs can reliably replicate human market dynamics, especially the emergence of phenomena like bubbles and crashes.
summary: This study compares Large Language Models (LLMs) with human traders in experimental markets where prices emerge endogenously. LLMs consistently price assets near their fundamental values and show little bubble formation, while humans generate substantial bubbles. These patterns persist across different market conditions. Analysis of LLM strategies shows lower variance, reduced bias, and stronger reliance on fundamentals. The authors conclude that LLMs are poor proxies for modeling human market behavior, as they do not replicate key phenomena like large bubbles.
weaknesses: 1. The experimental setup appears overly simplified, which undermines the strength of the authors’ claims. The simulated market environment does not adequately capture the complexity of real-world trading scenarios, making it difficult to draw robust conclusions about performance differences between human traders and LLMs.
2. The configuration of LLM agents in the experiment also seems simplistic and may not fully reflect the models’ ability to understand and develop trading strategies. The implementation approach resembles prompt engineering, but it is unclear to what extent prompt design was influenced by human intervention or biases, which could affect the validity of the results.
3. The paper does not provide sufficient detail about the human participants involved in the study. Without clear information on their backgrounds, trading experience, or demographic characteristics, it is difficult to assess the credibility of the reported differences and the generalizability of the findings.


Decision: Reject