Evidence from the Synthetic Laboratory: Language Models as Auction Participants

Authors: Kehang Zhu, Anand V. Shah, Yanchen Jiang, John Joseph Horton, David C. Parkes
Primary area: learning theory
Keywords: Language Model, Auction, Behavioral Economics, learning

Abstract:
This paper investigates the behavior of simulated AI agents (large language mod-
els, or LLMs) in auctions, validating a novel synthetic data-generating process
to help discipline the study and design of auctions. We begin by benchmarking
these LLM agents against established experimental results that study agreement or
departure between realized economic behavior and predictions from theory; i.e.,
revenue equivalence between first-price and second-price auctions and improved
play in obviously strategy-proof auctions. We find that when LLM-based agents
diverge from the predictions of theory, they do so in a way that agrees with behav-
ioral traits observed in the existing experimental economics literature (e.g., risk
aversion, and weak play in ‘complicated’ auctions). Our results also suggest that
LLMs are bad at playing auctions ‘out of the box’ but can improve their play when
given the opportunity to learn. This learning is robust to various prompt specifi-
cations and holds across a variety of settings. We run 2,000+ auctions for less
than $250 with GPT-4o and GPT-4, and develop a framework flexible enough to
run auction experiments with any LLM model and a wide range of auction design
specifications.

Decision: reject
Reviewer scores: 8, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730079459125

thread_type:
official_review

<thread_object>
creator: reviewer_qbpd

<content>
summary:
This paper studies the auction behavior of LLMs. They benchmark LLM agents against two kinds of existing experimental results. The first result is that realized human biddings differ from theoretical results in first-price and second-price auctions. The second result is that clock auctions induce more rational play in humans than second-price auctions. The authors systematically designed prompts and their results show that the behavior of LLMs conforms with the aforementioned experimental results.

strengths:
1. The study of LLMs in auctions are interesting and relatively new.

2. In the appendix, the authors include robustness checks, which make the results more convincing.

3. The main structure of the paper is clear. Each benchmark begins with the setting, proceeds to existing theoretical and empirical results, and ends with simulation outcomes, which make it easy to be understood.

weaknesses:
1. My main concern is that the technique contribution of this paper is limited. The method and design of experiments are simple and not novel.

2. The results claimed in abstract are not clearly presented. 
  * The authors claim that 
>Our results also suggest that LLMs are bad at playing auctions ‘out of the box’ but can improve their play when given the opportunity to learn.
>
   On one hand, it not clear which part of their results shows that LLMs are bad at playing auctions ‘out of the box’. On the other hand, in sec 3.2.4, they mention that 
>Interestingly, we see little evidence of learning over time.
>
This seems to contradict with the conclusion in the abstract and I think clarification is needed.

* The authors claim that 
>..., validating a novel synthetic data-generating process to help discipline the study and design of auctions.
>
And the authors actually ask two research questions:
>The present work examines this question for auctions: how well do LLMs substitute for humans in auctions, especially when behavioral traits like risk-aversion or bounded rationality matter? And, more generally: how can we use LLMs to improve the design of economic mechanisms?
>
The first question is answered well. But it is not clear that which part of their results discusses using this process to improve the design of economic mechanisms. 

Overall, I suggest the authors summarize their research questions and contributions in a more explicit way.

questions:
1. Why the authors use LOESS-smoothed data for regression? Is it standard? I think a few explanations are needed here.

2. In sec 3.2.4, the auctions repeat 15 rounds and the LLM agents seem not improve their understanding of the mechanisms by bidding closer to their true value over time. Did the author try to repeat the auction more than 15 rounds?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730546792460

thread_type:
official_review

<thread_object>
creator: reviewer_mwn6

<content>
summary:
This paper investigates the auction behavior of simulated AI agents (LLMs) and validates a synthetic data-generating process for auctions.
Precisely, this paper benchmarked LLM agents against established experimental results in auctions, such as revenue equivalence between first-price and second-price auctions and behavior in obviously strategy-proof auctions. It is found that when LLMs diverge from theory, their behavior aligns with some human behavioral traits observed in experimental economics literature (e.g., risk aversion). Also it is shown that LLMs can improve their play with learning opportunities and this learning is robust across settings.

strengths:
1. Provides insights into how LLMs perform in auction scenarios, which is relevant as LLMs are increasingly being considered for various economic applications.
2. Conducts a detailed analysis of LLMs' behavior in different auction formats and settings, including semantic analysis and counterfactual experiments to understand their decision-making processes.

weaknesses:
1. "Each setting is repeated 15 times with values drawn randomly each time." It is hard to say whether 15 samples are enough to represent a random distribution.
2. It is a pity there is no theoretical result can be derived or developed based on the experiments.

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1730846499807

thread_type:
official_review

<thread_object>
creator: reviewer_evbn

<content>
summary:
This paper studies how well LLMs can simulate human behavior in auctions. The key motivation behind this work is that generating synthetic data using LLMs could potentially offer a cheaper alternative to traditional human-subject experiments.    

The findings show that LLMs sometimes deviate from the predictions of economic theory, but these deviations often align with the behavioral traits observed in real-world experiments with human participants. In particular, the LLMs tend to underbid in second price auctions and overbid in first price auctions, yielding a counter argument to revenue equivalence. This can be explained by risk aversion in LLM bidding behavior. Additionally, the paper reveals that LLMs perform better in simpler auction formats (clock auctions), which supports the argument behind the obvious strategy-proofness.    

The authors suggest that LLMs could be a valuable tool for studying a wide range of economic mechanisms, including those that are currently difficult or impossible to test experimentally.  They highlight the potential of LLMs to generate data for large-scale experiments that would be too costly or raise ethical concerns if conducted with human participants.

strengths:
This paper moves along an exciting application of LLMs in experimental economics, and could be a valuable and unique plus to ICLR. The economic properties being examined in this paper are fundamental and the observations from LLMs are interesting.

weaknesses:
One major challenge to this line of research (LLMs as an alternative to human-subjects in experiments) is that the behavior of the LLMs could be highly sensitive to (1) training data and (2) prompt, which may put the reliability of observations from experiments with less known results at risk. More importantly, the complete prompts used in this work are not shared and there seems no robustness test of the LLM behavior against the prompts.

Taking the first prompt template in Appendix A.1 as an example, where “RULE EXPLANATION”, “INSTRUCTIONS”, “PERSONA” are not given. I used this template by only filling the rule explanation part and observed very different results (i.e., the bidding plan from the LLM). 
* If I describe the auction rule of second price auction in a less formal way (in the tongue of ordinary people), the LLM replies things like slightly overbidding and randomized bidding etc. 
* If I describe the auction rule of second price auction in a formal way (in the tongue of an auction theorist, e.g., using “sealed bid second price auction”), the LLM immediately says “my dominant strategy is to bid my true valuation of the item”.

From the simple test above, we can see that response from the LLM could depend on (1) whether they learned how to play this game from training data (they may know the literature of the experiment already) and (2) whether the prompt properly recalls the corresponding memory.

Therefore, a crucial component of studies like this would be a robustness test.

questions:
What are the prompts used as “RULE EXPLANATION”, “INSTRUCTIONS”, “PERSONA” in this work?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731056447713

thread_type:
official_review

<thread_object>
creator: reviewer_e3av

<content>
summary:
This manuscript asks, "how well do LLMs substitute for humans in auctions"? - a question motivated by the cost of experimenting with human subjects: the c. 1,000 auctions run for the manuscript cost less than US\\$100 on GPT-4 and GPT-4o - in contrast to the US\\$15,000 price of the human experiments (with 404 subjects) in Li (2017).

The paper first describes the LLM simulation environment (§2).  Then §3 forms the bulk of the paper, introducing the auction designs (first and second price sealed bid - FPSB and SPSB - as well as the strategically equivalent ascending auction, AC, and a 'blind' variant, AC-B, which does not report when bidders drop out).  For each, theoretical and (human) experimental results are presented.  These are compared to the LLM results.

The findings are:
1. consistent with theory, LLMs bid higher in the SPSB than in the FPSB, although with a "smaller separation ... than would be predicted by theory" - largely due to overbidding relative to theory in the FPSB;
1. in AC - which is 'simpler' than the strategically equivalent auction SPSB - LLM play is closer to theoretically predicted play.  This is found in both independent private values (IPV) and affiliated private values (APV) settings.

strengths:
**originality**

The manuscript is original in the sense of being the first I have seen to assess LLMs as human substitutes in auctions.

**quality**

The research is well reasoned, conducted and written up.

**clarity**

The manuscript is generally clear.

**significance**

To use the expression of Dell'Acqua et al. (2024), the capabilities of LLMs form a 'jagged frontier' relative to human performance, requiring task-by-task analyses.  This manuscript contributes to that endeavour.

weaknesses:
1. above all, I wonder what problems are solved by the discovery that LLMs in stylized environments play roughly like humans do.  Thinking aloud, some guesses:
   1. testing failure modes in advance of high-profile auctions.  Even given the relatively low costs of LLM use, this seems an inefficient way of proceeding relative to e.g. randomly generating bid data or formally proving properties (q.v. Caminati, Kerber, Lange and Rowat, 2015).  To be convinced, I would want to see how well LLMs perform in more asymmetric auctions, e.g. calibrated to real high-profile auctions, or perhaps the Combinatorial Auction Test Suite (Leyton-Brown, 2024).
   1. assistance in gaining intuitions in theoretical analysis of novel auction formats (q.v. Dütting, Feng, Narasimhan, Parkes, and Ravindranath, 2024).  Here, the manuscript would be more convincing if it showed us insights into novel auction formats - instead of just the most common ones.
1. It is claimed (p.2) that LLMs may exhibit risk-aversion even when prompted not to.  The evidence for this claim is in Appendix A.4, which is not part of the submission.  Thus, I am unable to assess this claim.  I would, in particular, like to see it compared to other hypotheses.
1. It is claimed (p.3) that Li (2017) found "that human subjects tend to be more truthful in second price sealed bid auctions than in ascending clock auctions".  Again, I would want to see which other hypotheses were considered by Li: one can imagine sunk cost arguments affecting ascending clock bids, but not SPSB.  (A similar comment arises on p.8, when it is claimed that humans "are less truthful" under AC-B: alternatively, might they not just have more difficulty with the Bayesian calculation?)
1. It would be nice to see performance considered for varying $n$.  The manuscript indicates that $n$ is "often" three (p.3) or always three (p.4), which - I would guess - may account for around half of the examples in an LLM's training data.  Thus, it is possible that performance deviates from human considerably outside of the training data.
1. Other interesting robustness tests could include:
   1. compare results for auctions identical up to the 'currency' units (e.g. dollars, billions of cowrie shells, etc.). 
   1.  compare results with and without the first step of the simulation procedure (p.3), which asks the LLM to generate a bidding plan explicitly: which result is closer to amateur/expert human bidding?
1. I would probably cite Vickrey's original paper for the results on his eponymous auction, rather than Krishna's 2009 textbook (p.5).
1. The panels in Figure 1 are reversed: that on the left, for which the bid = value in theory, is the SPSB, not the FPSB.
1. Given the importance of Li's OSP (2017), I would like a bit more space given to its explanation (e.g. maybe removing the table at the bottom of p.4, if needed).

questions:
1. footnote 5 mentions setting a non-zero temperature to induce variation.  _Prima facie_, this might seem to correspond to something other than a BNE - like a trembling hand, or QRE?  (If so, than the comparison to BNE theoretical results is inappropriate.)
1. valuations in the FPSB and SPSB are drawn uniformly, but bids are constrained to be integers (p.4).  In general, equation (2), the FPSB BNE bid, does not resolve to an integer.  Is there any reason to believe that whatever rounding strategy is used does not introduce artefacts into the results?
1. there is evidence that, with training, humans learn to play close to Nash (pp.5, 8).  While the abstract claims that LLMs "can improve their play when given the opportunity to learn", the text (p.8) finds, "little evidence of learning over time".  Which of these is correct?  Either way, it may be useful to be more careful to compare like-for-like humans and LLMs (e.g. amateur/untrained v experienced/trained).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732811240542

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Robustness checks

comment:
We welcome the suggestion to contribute further robustness checks and thank the reviewers for comments that improved the paper. Inspired by reviewer comments, we have run the following three sets of additional experiments for robustness:

1. Comparing play under the FPSB and SPSB auctions (in IPV settings) along currencies other than the dollar (for now, the euro and ruble). We were pleased to see that changing the currencies did not suggest any serious change in our results (and in fact, a greater separation between FPSB and SPSB play than was reported in the main text). 

More specifically:
- For the variation with euros instead of dollars, see Fig 5 in the updated draft.
- For the variation with rubles instead of dollars, see Fig. 6 in the updated draft.
- For context, the original plots from Section 3.1 are in Fig.1.

These are the only two currency variations we ran. Both are reported in Appendix A.5.

2. Additionally, we also compared play under the FPSB and SPSB auction in languages other than English (for now, Spanish and Chinese). Once again, we were pleased to see that changing the language did not suggest any change in our original results.

- For the variation with Spanish instead of English prompts, see Fig 7 in the updated draft.
- For the variation with Chinese instead of English prompts, see Fig 8 in the updated draft.

These are the only two language variations we ran. Both are reported in Appendix A.5.

3. Running our existing prompts along various numbers of agents (4 and 5 agents). We only ran these simulations for the first-price auction setting, as the BNE prediction in the FPSB depends on N. Extensions could also report results for other auction settings where the equilibrium prediction changes on N (e.g., third-price auctions, all-pay auctions, etc). We were pleased to see that LLM bids actually increased as N increased! I.e., evidence corroborating the BNE prediction of (N-1)/N increasing in N.

- For the results of the FPSB auction with 4 and 5 agents, see Fig 4 in the updated draft.

These results have been added to the paper and are reported in Appendix A.4.

Thank you once again to the reviewers for comments that inspired these robustness checks. We believe they improved the credibility of our results.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732811344118

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Improving play from LLMs via learning

comment:
A second theme in the reviewer responses were questions on learning. Namely, 1) When is there evidence of learning?, 2) what does it take to obtain learning behavior?

To answer both, we added a new Appendix section to the paper: Appendix A.7 (Learning). 

Here, we repeat the results of the SPSB experiments in Section 3.1 but remove the ability for agents to conduct any chain of thought reasoning – we call such agents ‘Out-the-box’ agents. 

Then, we compare the play of the 'Out-the-box' agents against 'Chain-of-thought' agents in the SPSB, and find: 

1) Chain-of-thought agents perform better in the long run (after round 2, they consistently beat Out-the-box agents), and 
2) Chain-of-thought agents demonstrate significant improvement in the first few rounds and then stabilize to a lower rate of mistakes, while Out-the-box agents demonstrate little to no improvement across 15 rounds of play. 

We found it interesting that Chain-of-thought agents start with a much higher rate of mistakes in the first two rounds, suggesting that something about endowing agents with planning capacity causes them to 'tile the space of possible plans' more extensively when first trying out strategies.

Thank you again to the reviewers for the comments that inspired this additional work. We believe it greatly improved the paper.

Note: We made Appendix A.7 an extension of the setting in Section 3.1 (SPSB) because the SPSB auction has an equilibrium in dominant strategies of bidding one’s value. Hence, it's natural to plot the average absolute value deviation between bid and value (e.g., average |Bid - Value|) by round as a measure of ‘mistakes’ made by round -- if there is learning, we expected mistakes to go down by round (which they did for Chain-of-thought agents). Extensions could conduct a similar analysis by plotting mistakes to the theoretical benchmark under any equilibrium notion.
</content>
</thread_object>

</thread>


Metareview: