STEER-ME: Assessing the Microeconomic Reasoning of Large Language Models

Authors: Narun Krishnamurthi Raman, Taylor Lundy, Kevin Leyton-Brown, Jesse Perla
Primary area: datasets and benchmarks
Keywords: large language models, microeconomics, benchmarking, decision-making, economic agents, llm agents

Abstract:
Large language models (LLMs) are increasingly being applied to economic tasks like stock picking and financial analysis. Existing LLM benchmarks tend to focus on specific applications and often fail to describe a rich variety of economic tasks. Raman et al. (2024) offer a blueprint for comprehensively benchmarking strategic decision-making. However, their work failed to address the non-strategic settings prevalent in micro-economics. We address this gap by taxonomizing micro-economic reasoning into $58$ distinct elements, each grounded in up to $10$ distinct domains, $5$ perspectives, and $3$ types. The generation of benchmark data across this combinatorial space is powered by a novel LLM-assisted data generation protocol that we dub auto-STEER which generates a set of questions by adapting handwritten templates to target new domains and perspectives. By generating fresh questions for each element, auto-STEER  helps reduce the risk of data contamination, ensuring that \model evaluations remain valuable over time. We leveraged our benchmark to evaluate $15$ LLMs over each of the instantiated elements, examined their ability to reason through and solve microeconomic problems and compared LLM performance across a suite of adaptations and metrics. Our work provides insights into the current capabilities and limitations of LLMs in non-strategic economic decision-making and a tool for fine-tuning these models to improve performance.

Decision: reject
Reviewer scores: 5, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730130488935

thread_type:
official_review

<thread_object>
creator: reviewer_vlwy

<content>
summary:
The paper introduces a novel benchmark, STEER-ME, to assess LLM's microeconomic reasoning in non-strategic settings. Using an LLM-assisted data generation method, Auto-STEER, the benchmark includes 57 distinct microeconomic elements across multiple domains and perspectives, minimizing data contamination. The authors evaluated 15 LLMs on these benchmarks, highlighting strengths and limitations in handling non-strategic economic decision-making.

contributions:
1. Novel benchmark development: The STEER-ME benchmark assesses LLM performance in non-strategic microeconomics, a domain not extensively covered by previous benchmarks that focused mainly on game theory or strategic decision-making.

2. Comprehensive taxonomy: The paper categorizes microeconomic reasoning into 57 elements, each tested across diverse contexts, such as finance and public policy, to evaluate broad economic reasoning capabilities.

strengths:
1. proposed a benchmark on economics reasoning, which is relatively novel compared to classical benchmarks.

2. evaluated quite a few LLMs on the benchmark and have achieved some results.

weaknesses:
Section 2: There are numerous economic concepts that may be challenging for machine learning researchers to follow, particularly the rationale behind constructing the taxonomy shown in Figure 1.

Section 3.1: Details about the dataset are lacking. Key statistics, such as the distribution of questions and question lengths, are not provided. Additionally, there is no discussion on how the dataset’s quality is ensured.

Section 5: The analysis of experimental results is limited. There is no case study or error analysis presented to offer deeper insights into model performance.

Code / Data:  i have not seen it uploaded.

questions:
See weakness. In addition, i have questions below

1. Why is it necessary to develop these benchmarks for LLMs? Existing finance-related benchmarks, such as FinQA, FinEval, FinBench, FinanceBench, and FAMMA, already appear to cover similar topics to those in STEER. Additionally, general reasoning datasets in common scenarios may address some overlapping subjects.

2. What specific insights does this benchmark provide for LLM researchers?



--------------------------------------------------
I raise to 5 for the clarification about the motivation and dataset.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730272307029

thread_type:
official_review

<thread_object>
creator: reviewer_4bwc

<content>
summary:
This study fills a gap in the assessment of existing Large Language Models (LLMs) for non-strategic microeconomic reasoning tasks. To achieve this goal, the authors propose a new STEER-ME benchmark that categorizes microeconomic reasoning into 57 elements, generates a large number of multiple-choice questions and dynamically generates test data using the LLM-assisted data generation protocol (auto-STEER) to reduce the risk of data contamination. The experimental results show that the performance of different LLMs in the microeconomic reasoning task varies significantly, with even the larger models falling short; the only model that consistently performs well is o1-preview.

strengths:
1. This work identifies a gap in the evaluation of current LLMs for non-strategic microeconomic reasoning tasks with high practical needs and research value.

2. A structured STEER-ME benchmark and a dynamic data generation protocol auto-STEER are proposed to ensure test diversity and data cleanliness.

3. The experiments cover a wide range of models and adaptation strategies, revealing differences in the performance of different LLMs in non-strategic microeconomic reasoning.

weaknesses:
1. In the AUTO-STEER section, the authors may need to clarify which type of LLM was used for data generation. Additionally, it is important to discuss the impacts of the benchmark results by different LLMs for data generation.

2. The authors note that LLMs struggle with basic mathematical problems, like calculating the deadweight loss of a monopoly[Line 445-450], and suggests that this may be attributed to the use of incorrect formulas. To strengthen this claim, the authors should present specific evidence, such as analyses of error cases.

3. The authors need to provide more information on which open-source LLMs were tested (they did not list the LLMs), particularly whether they included models from the fields of mathematics and economics. It would also be important to discuss any differences in results between these specialized LLMs and general LLMs.

questions:
1. It would be better to explore the effects of LLMs under in-context learning conditions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730342253963

thread_type:
official_review

<thread_object>
creator: reviewer_ihdk

<content>
summary:
The paper presents a new benchmark specifically designed to evaluate large language models (LLMs) in non-strategic microeconomic tasks, addressing a gap in existing benchmarks that focus more on strategic decision-making. The authors create a taxonomy of 57 microeconomic reasoning elements across multiple domains, perspectives, and types, and introduce a data generation protocol called auto-STEER to produce fresh, contamination-resistant questions for each element. By evaluating 15 LLMs using this benchmark, the study reveals strengths and weaknesses in the models' ability to handle microeconomic concepts such as optimization and marginal analysis.

strengths:
1. The creation of a taxonomy with 57 elements specifically for non-strategic economics provides a strong theoretical basis for the benchmark. This comprehensive taxonomy ensures that the benchmark is grounded in a thorough understanding of microeconomic reasoning, enabling a detailed and structured evaluation of LLMs in this domain.
2. The benchmark is solid, due to its broad variation in testing angles and use of multiple testing metrics.

weaknesses:
1. The paper fails to discuss the correlation between the results of the proposed benchmark and those of related benchmarks, highlighting the significance of this research.
2. There are still some issues that need to be addressed in the writing of the paper, such as:
- The paper lacks hyperlinks for tables, figures, and sections, making it difficult for readers to locate referenced content.
- The title of Figure 2 contains misleading positional descriptors for the two images, referring to them as ‘on top’ and ‘on the right.’

questions:
1. The analysis in the results section appears incomplete. Why is there only a separate section on "Domain Robustness" without any mention of "type robustness" or "perspective robustness"?
2. Although the paper generates a large number of questions for testing purposes, is this amount really necessary? For instance, the paper creates 100 templates by rephrasing, only modifying exact words or objects in the questions. Will these modifications lead to significant differences in results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730739635457

thread_type:
official_review

<thread_object>
creator: reviewer_8xwi

<content>
summary:
This paper presents a new benchmark specifically tailored to evaluate LLMs' performance in non-strategic microeconomic reasoning. This benchmark, STEER-ME, addresses limitations in existing economic benchmarks by categorizing microeconomic reasoning tasks into 57 distinct elements, covering various domains and perspectives. The authors also propose an LLM-assisted data generation protocol that dynamically generates questions to prevent data contamination. The study evaluates 15 LLMs across these elements to identify both strengths and limitations in their reasoning capabilities, providing insights into current LLM performance on fundamental economic tasks.

strengths:
1. The paper offers a highly comprehensive benchmark, breaking down microeconomic reasoning into 57 distinct elements that span a wide range of domains and perspectives.
2. Significant resources and computational effort were invested in the study, with $5,896.33 spent on API requests to OpenAI and Anthropic, and 6.81 GPU years of compute used to evaluate open-source models.
3. All model outputs are made publicly available, promoting reproducibility and enabling further research and contributions to evaluation practices.
4. The study employs a wide range of evaluation metrics, including calibration-related metrics like Expected Calibration Error (ECE).
5. The paper highlights the critical issue of data contamination in LLM benchmarks

weaknesses:
1. The primary contribution—expanding benchmarks to cover non-strategic microeconomic settings—leans more toward an economic contribution than a machine learning one, which may affect the paper's resonance with ICLR’s core audience.
2. The paper asserts that auto-STEER addresses data contamination but lacks empirical evidence or a detailed explanation of how it mitigates this problem effectively.
3. The benchmark is limited to multiple-choice questions, whereas real-world financial assistant LLMs are often required to handle open-ended generation tasks relevant to economic analysis. I would recommend expanding more experiments with more types of datasets, such as the free-text generation QA datasets.
4. The extensive number of elements and perspectives, while thorough, could complicate the task of providing a clear, overall model recommendation.
5. A substantial portion of the paper focuses on introducing economic rationality elements, which might not align well with the technical expectations of ICLR’s machine-learning audience. But overall, the idea is pretty interesting. 

I am willing to raise my score if the paper could include more analysis (e.g., more evaluations on different types of datasets) on the perspectives of LLM-wise or machine learning-wise.

questions:
Please refer to the weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732589089127

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We would like to thank all reviewers for their thoughtful comments and for highlighting areas where our work could be further improved. We recognize that some aspects of our benchmark's insights may not have been fully clear, and we appreciate the opportunity to elaborate. Here, we outline some of the key insights our initially submitted work offered:

1. Reasoning over Options: One particularly interesting finding is that LLMs demonstrate an ability to recognize when a question can be resolved using the provided options. This allows them to "short-circuit" difficult computations by leveraging the structure of multiple-choice questions, a behavior that suggests nuanced reasoning strategies.
2. Challenges with Real-Valued Exponents: Real-valued exponent computations, which are prevalent in economic reasoning (e.g., interest rates, discounting, elasticities), present a significant challenge for LLMs. This reveals a gap in the models' ability to handle key mathematical operations that are critical for solving common economics problems.
3. Domain-Specific Conditioning: We also observe that certain domains influence LLM performance. For example, when questions are framed in more realistic, real-life scenarios, models often fail to recognize what is being asked. They tend to equivocate or misinterpret the problem, indicating potential biases introduced by the contextual framing of the task.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733270056169

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Highlights from the Resubmitted Paper

comment:
# Additions to the Evaluation:

1. We have tested 12 additional LLMs as listed in the table in the appendix.
2. We have created an additional adaptation, free-text QA, and have evaluated all models across all elements on this adaptation.

# Additions to the Results:


1. Conducted a quantitative analysis on the impact of multiple-choice options on chain-of-thoughts showing that in nearly all cases for all of the closed-source models that offering options increase a model's ability to answer correctly.
2. Conducted case studies diving deeper into the impact of MCQA on performance

    a. Showed that in elements that dealt with aggregation, the inclusion of options aided the LLMs' abilities to condition on the right reasoning path.

    b. Offered a number of new insights through the free-text QA adaptation on a subset of elements that had high performance gaps between MCQA hidden and shown. 
3. Added a deeper case/error analysis of the mathematical errors on the deadweight loss element using the free-text QA adaptation as the diagnostic tool. We then characterized and detailed the specific error cases. 
4. Added a type robustness section diving deeper into the issues LLMs had with different commonly used functional families.
5. Showed that the choice of generation LLM had negligible impact on downstream performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733270402839

thread_type:
official_comment

<thread_object>
creator: area_chair_4ny1

<content>
title:
End of reviewer-author discussion phase

comment:
Dear reviewers,

As we near the conclusion of the reviewer-author discussion phase, I wanted to kindly follow up to see if you’ve had a chance to review the author responses on your comments. Could you confirm that you’ve read it and, if needed, update your review and scores accordingly?

Thank you for your time and effort!

Your AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary:
The paper introduces STEER-ME, a benchmark for evaluating LLMs' microeconomic reasoning capabilities in non-strategic settings. Key claims include:
- A taxonomy of 57 distinct microeconomic reasoning elements across multiple domains and perspectives
- Introduction of auto-STEER, an LLM-assisted data generation protocol to reduce data contamination risks
- Evaluation of 15 LLMs (later expanded to 27) showing varying performance levels in microeconomic reasoning tasks
- Finding that even larger models struggle with certain economic reasoning tasks, with o1-preview being the only consistently strong performer
- Discovery that LLMs particularly struggle with real-valued exponent computations and certain mathematical operations critical for economics

Strengths:
- Comprehensive taxonomy breaking down microeconomic reasoning into well-defined elements
- Significant computational resources invested ($5,896.33 on API requests, 6.81 GPU years)
- Public availability of model outputs promoting reproducibility
- Diverse evaluation metrics including calibration metrics
- Novel approach to generating fresh questions through auto-STEER protocol
- Broad coverage of domains and perspectives in economic reasoning

Weaknesses:
-  Limited quantitative evaluation of free-text QA adaptation, with analysis restricted to case studies
-  Insufficient empirical evidence demonstrating auto-STEER's effectiveness in mitigating data contamination
- Primary focus on multiple-choice questions may not fully reflect real-world economic reasoning tasks
- Complex taxonomy with numerous elements may complicate clear model recommendations
- Lack of comprehensive correlation analysis with related benchmarks
- Limited discussion of specialized economic LLMs' performance compared to general models

Reasons for Rejection:
-  Methodology Limitations: The heavy reliance on multiple-choice questions without sufficient quantitative evaluation of free-text generation limits the benchmark's practical applicability.
- Incomplete Validation: The paper lacks robust empirical evidence supporting auto-STEER's effectiveness in preventing data contamination.
- Scope vs. Depth Trade-off: While comprehensive in breadth, the large number of elements dilutes the depth of analysis for each component.
- Technical Innovation: The contribution leans more toward economic taxonomy than advancing machine learning methodology.