Cost-of-Pass: An Economic Framework for Evaluating Language Models
Authors: Mehmet Hamza Erol
Batu El
Mirac Suzgun
Mert Yuksekgonul
James Zou
Venue: ICLR 2026 Poster (2026)
Keywords: economic evaluation framework
language-model evaluation
cost‑performance trade‑off
inference time techniques
Primary area: datasets and benchmarks
TL;DR: Introduces cost-of-pass (dollars per correct answer) and its frontier to measure LM cost-efficiency. Analyzes model families across task types, reports rapidly falling frontier, and reveals limited cost-effectiveness of common inference-time methods.

Abstract: The widespread adoption of AI systems in the economy hinges on their ability to generate economic value that outweighs their inference costs. Evaluating this tradeoff requires metrics that account for both performance and costs. Building on Farrell's theory of productive efficiency, we develop an economically grounded framework for evaluating language models' productivity by combining accuracy and inference cost. We formalize cost-of-pass, the expected monetary cost of generating a correct solution. We then define the frontier cost-of-pass as the minimum cost-of-pass achievable across available models or the human-expert(s), using the approximate cost of hiring an expert. Our analysis reveals distinct economic insights. First, lightweight models are most cost-effective for basic quantitative tasks, large models for knowledge-intensive ones, and reasoning models for complex quantitative problems, despite higher per-token costs. Second, tracking this frontier cost-of-pass over the past year reveals significant progress, particularly for complex quantitative tasks where the cost has roughly halved every few months. Third, to trace key innovations driving this progress, we examine counterfactual frontiers—estimates of cost-efficiency without specific model classes. We find that innovations in lightweight, large, and reasoning models have been essential for pushing the frontier in basic quantitative, knowledge-intensive, and complex quantitative tasks, respectively. Finally, we assess the cost-reductions from common inference-time techniques (majority voting and self-refinement), and a budget-aware technique (TALE-EP). We find that performance-oriented methods with marginal performance gains rarely justify the costs, while TALE-EP shows some promise. Overall, our findings underscore that complementary model-level innovations are the primary drivers of cost-efficiency, and our economic framework provides a principled tool for measuring this progress and guiding deployment.

Published as a conference paper at ICLR 2026
COST-OF-PASS: AN ECONOMIC FRAMEWORK FOR EVALUATING LANGUAGE MODELS
Mehmet Hamza Erol Stanford University mhanza@stanford.cmm edu
Batu El
Mirac Suzgun
Stanford University
Stanford University
edu
msuzgunêst cantene@ettafrddddo .nnoodd.edu
Mert Yükksekööüül Stanford University mer mertyg@stanforde
James Zou
Stanford University Jamesz@stanford.ccm
ABSTRACT
The widespread adoption of AI systems in the economy hinges on their ability to generate economic value that outweighs their inference costs. Evaluating this tradeoff requires metrics that account for both performance and costs. Building on Farrell's 's theory of productive efficiency, we develop economically grounded framework for evaluating language models' productivity by combining accuracy and inference cost. We formalize cost-of-pass, the expected monetary cost of generating a correct solution. We then define the frontier cost-of pass as the mini- mum cost-of-pas pass achievable across available models or the human-expert(s), using the approximate cost of hiring an expert. Our analysis reveals distinct economic insights. First, lightweight models are most cost-effective for basic quantitative tasks, large models for snowledge-intensive ones, and reasoning models for com- plex quantitative problems, despite higher per-token costs. Second, tracking this frontier cost-of-pass over the past year reveals significant progress, particularly for complex quantitative tasks where the cost has roughly halved every few months. Third, to trace key innovations driving this progress, we examine counterfactual frontiers-end estiemats of cost-efficiency without specific model classes. We find that innovations in lightweight, large, and reasoning models have been essential for pushing the frontier in basic quantitative, mnowledge-intensive and complex quan- titative tasks, respectively. Finally, assess the cost-reductions from common inference- techniques (majority voting and self-refinementt, and a budget- aware technique (TALE-EP). We find that performance-orient marginal performance gains rarely justify the costs, while TALE-EP shows some promise. Overall, our findings underscore that complementary model-level inno- vations are the primary drivers of cost-efficiency, and our economic framework provides principled tool for measuring this progress and guiding deployment.
methods with
INTRODUCTION
The recent progress in generative AI, particularly language models (LMs), has sparked significant interest in their potential to transform industries, automate cognitive tasks, and reshape economic productivity (Brynjolfsson et al.) 2025 Eloundou et al., 2024 Acemoglu 2024). The widespread adoption of these Al systems the economy hinges on whether the economic benefits generated by the tasks they can perform outweigh the associated inference costs, and whether those inference costs are lower than the cost of equivalent human labor. Consequently, two priorities have emerged the forefront of LM research: advancing capabilities and reducing costs. These goals. however, often involve trade-offs with more powerful models or test-time techniques that offer higher accuracy at the expense of greater computational and monetary cost (Chen al. 2024 Parashar et al. 2025 Madaan ct al. 2023 Wang et al., 2023; Kapoor et al. 2024). While standard metrics capture accuracy or other system capabilities, they fail to account for cost, leading to incomplete picture of progress.
"Co-first authors.
CCosennio authors.
https: http://hhhb.ccc/////aaaaaaaaooat

---

Published as a conference paper at ICLR 2026
Concepts
(A) Frontier Cost-of-pass & Human Expert Baseline
(C) Essentialness of Model Families to Task Categories
Expart Baselno Cost (S) Frontlar Cost (5)
Light
Large
Reasoning
Cost-of-Pass: Expected cost of producing output.
Bavi:
Ouardtative
70%
0%
10%
k(m.p)
(jonet_pp)) Baccurary
Gr(r)
Knowledge
[P(
Basod
Basic Quardiative
Knowledge Based
Complex
Complex
Human Expert Baseline Cost:
Quantbative
Cast hinin a human expert produce correct output.
(B) Progress as Frontier Cost-of-Pass over Time
(D) Cost Reductions with Inference Time Techniques
w(expert.p) p) Casen (p)
Lighiwight
Self-Reffie
Majority Vote: K-3
Majorty Volo: k-d
Frontier Cost-of-Pass: Lowest cost-of-pass given available set of human expert baseline.
Lange
Basic
Quandiative
0%
0%
5%
min (V,LM), ofexpert, 2)]
Knowledge
(WM 2014
K:MM = ((ma))
Bert ConschPaas
Sep
Comptex
Frontier Cast-of-Pass (14)
Quanstarvve
on
Figure 1: Highlights the cost-of-pass framework and empirical analyses. Core concepts (left) set foundations for: (A) Comparing the Human Expert Baseline to the frontier achieved by the single most effective LM per task category. (B) Tracking the reduction in frontier cost-of-pass over time, indicating progress driven by new model releases (color-coded by family). (C) Quantifying the essential contribution of each model family: lightweight (less than $1 per million tokens), large, and reasoning; the current cost-efficiency frontier, measured by the percentage each family's contribution. (D) Assessing the economic benefit (relative cost reduction) achieved by applying common inference-time techniques over the baseline model frontier (which rarely results in meaningful gains).
Ultimately, what matters to the users is not just raw capability, but the value delivered relative to cost and the standard has been to interpret and report these separately. As the ecosystem of models grows, it is essential to assess new models not in isolation, but in the context of broader ecosystem, where marginal improvements may or may not justify higher costs, and do so an easy-to-interpret manner.
Prior attempts to incorporate cost into evaluation have typically relied on fixed inference bud- gets (Wang al. 2024), heuristic performance scores (McDonald et al. 2024), or non- monetryy proxies such as conciseness (Nayab et al. 2024). These choices tie conclusions to specific constraints or heuristies, limiting the generality and economic interpretability of such evaluations. To system- atically investigate the trade-off between cost and performance and analyze the LM ecosystem as a whole, we draw insights from a well-established and foundational framework from economics: production frontiers. Economists have long studied these frontiers, which map set of inputs the maximum output attainable under a given technology (Farrell 1957). In Farrell's 's original formulation, producer is technically efficient if no input can be reduced without lowering output, and price efficient if the input mix minimizes cost given input prices. Together, these conditions yield the lowest possible cost per unit of output. Extending this framework, (Aigner al. (1977) introduced stochastic frontier production functions, in which the relationship between inputs and output is modeled as stochastic rather than deterministic, practically accounting for potential defective outputs that do not pass evaluation criteria due to factors beyond the producer's control.
These economic concepts are highly relevant to modern LMs, which inherently function as stochastic producers: for a given input, they yield desired output (e.g., correct solution) stochastically (Brown) et al. 2024). Common practices such as employing scaffolds or more computationally intensive inference techniques (Snell et al. 2024 Madaan et al.] 2023 Wang et al. 2023) represent efforts to manipulate this production process. These strategies seek to increase the probability of success but typically do so at the expense of higher computational cost, directly mirroring the economic trade-offs inherent in production efficiency. Motivated by these parallels and the economic goal of minimizing cost per successful output under uncertainty, we adapt the ideas from economically grounded production theory to develop a quantitative evaluation framework tailored to LMs.
We summarize contributions follows.
Concepts. We introduce cost-of-pass (2.2)) which quantifies the expected monetary cost to achieve successful output for given problem. Building on this concept and incorporating human-expert
2

---

Published as a conference paper at ICLR 2026
cost baseline, define the frontier cost-of-pass ($2.4) across all available options (LMs and human-expert) for that problem. We show these reveal distinct economic niches for model families (e.g., lightweight reasoning models) on different tasks, which accuracy comparisons alone obscure ($3.2).
the minimum achievable cost-of- cest-of-past pass
Tracking progress with frontier cost-of-pass. Using the cost-of-pass and frontier cost-of-pass, we analyze economic improvements across three task categories from May 2024 to February 2025. We observe exponential decrease in frontier cost-of-pass across all tasks, though the trends vary. Notably, we observe that, over the past year, the expected cost of generating a correct solution to complex quantitative problems has been cut in half every few months. We find that the frontier cost-of-pass, is driven primarily by lightweight models and reasoning models ($3.3).
Counterfactual frontier in the absence of model families. We show that our analysis reveals the complementary roles of different model types in driving recent progress. Innovations in lightweight models have been instrumental in reducing costs on basic quantitative tasks. Large models, by contrast, have been most impactful for knowledge-based benchmarks like GPQA-Diamond (Rein et al., 2024). Meanwhile, reasoning models have been central advances on complex quantitative reasoning challenges such as AIME (Mathematical Association of America, 2024) and MATH (Hendrycks et al. 2021 (..4..
Impact of post-hoc inference time techniques. We observe that common test-time techniques such as self-refinement (Madaan et al. (023> and majority voting (self-conssstency; Wang et al. 2023) to improve performance offer either limited or economic benefits, while budget-aware technique TALE-EP (Han et al.) 2025) delivers some benefits. These indicate that the recent reductions in frontier cost-of-pass have been mostly driven by model-level innovations 3.5
2
SETUP
2.1 ECONOMIC THEORY OF PRODUCTION EFFICIENCY
Classical production theory examines how producers efficiently convert inputs (resources) into outputs. A central concern is understanding the maximum output attainable with given set of inputs, conversely, the minimum inputs (and thus cost) required achieve specific target output level.
Consider a set producers F= = = such that each producer fi E F can transform an input vector x E R (e.g.. quantities of different resources) into an output. The inputs used by producer fi have associated prices, represented by price vector w; E B When focusing on achieving a specific target output level, say u units, economists are interested in the frontier cost V.. This represents the absolute minimum monetary cost required to produce at least units, considering all input vectors x capable of achieving this output across all available producers with their respective pricings. This frontier cost is formally defined as:
V₂ min (w x f((x 2 u) J.eT
Farrell (1957) used these core concepts to formalize definitions for technical and price efficiency in a production ecosystem for producers. Critically, Aiinnr et al. (1977) extended this framework to handle stochustic production functions, where output is probabilistic for given input.
Building on this economic foundation, we adapt the core concept of a frontier cost (V.) to represent the minimum achievable cost for obtaining correct solution using LMs. To better reflect LM behavior, which is inherently stochastic, we incorporate this variability into our cost-efficiency metric. This aligns our framework with core production concepts and enables assessment of the economic impact stochastic LM producers.
2.2 COST-OF-PASS, AN EFFICIENCY METRIC FOR LMS
Here we instantiate the economic framework for LMs. Consider a specific problem p. where the unit of production is correct solution. We define model inference pipeline using an LM, acting
3

---

Published as a conference paper at ICLR 2026
stochastic producer. Two quantities characterize its efficiency problem p.
R. (p) Prob. of m producing a correct answer p, C. (p) = Expected cost of one inference attempt by m on p-
In the context of LMs, the inputs x correspond to resources like prompt and generated tokens, while the input prices represent the costs per token charged by the provider. The total cost of these inputs for a single inference attempt by model m on problem p is captured by Cn (p), effectively instantiating the term w' from the theory in the previous section.
Since the model output is stochastic, the expected number of attempts to obtain the first correct solution is I/Rm(.)) (p), assuming independent trials. This yields the cost-of-pass, defined as the expected monetary cost to obtain one correct solution for problem
v(m,p) p) = (m)
Rm (p)
The cost-of-pass, pass integrates both performance (R., (p)) and cost (Cm into a single economically interpretable metric: it quantifies how efficiently financial resources are converted into correct outputs. This formulation mirrors classical production theory, where the goal is to assess the cost of achieving a specific target output (Farrell ) in our case, the target is correct solution. When a model cannot produce one (Rm = 0), the cost-of-pase pass becomes infinite, appropriately signaling infeasibility.
2.3 THE LM FRONTIER COST-OF-PASS
While cost-of-pass ($ 2.2) evaluates a single model's 's efficiency, understanding the overall state of LM capabilities for given problem requires assessing the collective performance of the entire available LM ecosystem. Therefore, analogous to the frontier cost Va (Eq. D we define the LM frontier cost-of-pass for problem p as the minimum cost-of-pass achievable using any available LM strategy m from the set M:
(z(M) = min u(m,p).
mEM
VM(M) quantifies the minimum expected cost to solve problem p using the most cost-effective model currently available within the set M. If no LM in M can solve p (i.e., Rm = 0 for all E MM) then (M) xc.
2.4 GROUNDING EVALUATION: ESTIMATED HUMAN-EXPERT BASELINE
The LM frontier cost-of-past pass (MM reveals the best LM performance but lacks context: does not show if LMs are economically advantagcous over human labor. Morcover, the LM frontier cost-of-pass can be infinite if no LM succeeds. To address both, we introduce human-expert baseline as a reference point, by considering human-expert annotator as a specific strategy: Mespent Let Mo { represent this baseline set. We assume experts typically achieve near-perfect correctness (p) 1) for tasks they are qualified for Thus, the cost-of-pass pass for qualified expert is approximately their labor cost per problem:
v(expert, p) = (p)
The estimation of Cespent (p) involves considering required expertise, time per problem, and appropriate compensation rates (detailed $2.6.I). By incorporating this baseline, we define the frontier cost-of- pass for problem p, considering both LMs (M) and the human-expert alternative (Mo):
MM) min (nin(V (M), ())
(5)
This frontier cost-of-pass represents the true minimum expected cost to obtain correct solution for problem using the best available option, whether it's an LM or a human. Crucially, V, U MM) always finite (assuming finite human-expert cost and capability).
The qualified expert is intended to match the benchmark's 's label creators, for whom we implicitly assume near-perfect correctness. Appendix D.3 discusses associated limitations with possible relaxations.
4

---

Published as a conference paper at ICLR 2026
2.5 MEASURING PROGRESS AND VALUE GAIN
To track improvements against the best available option over time, let Mr denote the total set of available strategies time I, encompassing both the of LM strategies released up to time and the human-expert baseline Mo, that is, M₁ = {mmg) M The frontier cost-of-pan pat-of-pass achievable at time can be calculated as:
(((M)) = min v(m,p), p).
(6)
meM.
As new LM models {m²} are released, the set expands such that M₁ M (m, Consequently. the frontier cost-of-pass Vp, (J,, forms a non non-inceeen inn--iceeasiigg sequence over time t. tracking the reduction in the minimum cost needed to solve particular problem p-
To quantify the economic impact of new developments, we define the gain. When a new set of models {m,) becomes available time (often single model), the gain for problem p is the reduction it causes in the frontier cost-of-pass:
Gp((me),,MMan (y(M.-1) ((,MM-)))(((M--UUU 1 (m,)))
Note that G5 measures how much cheaper the new model(s). {m,}. make solving p compared to prior best options, including humans. Hence, large Gp value indicates significant economic contribution solving p. This notion underlies our experiments, analyzing the value generated by models relative to the human baseline and tracking the evolution of the overall frontier.
Extending to a distribution. Although measuring frontier cost-of-pase and value gain for individual problems can be informative, particularly through fine-grained perspective, often care about more than a single instance. Let P = }-= be n problems drawn i.i.d. from D. We treat P as the empirical distribution that puts mass 1/n on each element. We can then extend our definitions for such a distribution through the following:
(poD(Mi) EprrlV((Mo)],
(8) (9)
Gp-D(lme). Mc-1) z Ep (tm)),
2.6 ESTIMATING THE ECONOMIC EFFICIENCY
To operationalize our overall framework for any given distribution of problems, we introduce the following recipe:
(I) Estimate success rates. For each model- model-problem pair (m,p) generate a number of independent attempts to approximate Rm (p). Use the same prompt and model settings across these attempts, varying only factors necessary to ensure independence (e.g.. internal sampling randomness).
(2) Estimate per-attempt cost. Track the average number of tokens (prompt generation) consumed per attempt, multiply by the current token price (which can differ by model provider or usage level), and add any extra charges (c.g., third-part API calls, external reasoning modules, etc.). This sum yields C. (p).
(3) Compute cost-of-pass, For each model m, calculate v(m, p) (p)(p) (R. yields v(m,p) p) = oo.)
(4) Determine frontier cost-of-pass, Estimate human-expert cost v(expert, (see below). Find (M U Mo) for a given set of strategies M.
(5) Analyze over benchmarks. Aggregate (y(M) across problems p D estimate Vp-D(Mi) Track progress over time (for Ma) and compute gain GpoD for new models.
2.6.1 ESTIMATING HUMAN-EXPERT COST
To estimate o(expert, p). the plausible cost obtaining a correct human-expert answer, we systemati- cally determine the required qualifications, appropriate hourly compensation, and average time for typical problem p per dataset. We determine these quantities based on a hierarchy of evidence by prioritizing the dataset's creation process associated studies (e.g., reported annotation pay/time (Parrish et al. 2022)). When direct data is absent, we leverage findings from closely related work (Zhang et al.] 2024) infer parameters from the dataset's context (e.g., deriving time-per-problem
5

---

Published as a conference paper at ICLR 2026
Compensation rates are informed by reported study payments (Rein 2024) or relevant market rates for comparable expertise (e.g., specialized tutoring rates (TuorCruncher
from contest rules (Art of Problem Solving 2023)
2025
Wyzant 2025
Basic Quantitative
Knowledge Based
Complex Quantitative
Model Category
2-Digit Add.
GSM8K
BBQ GPQA Dia.
MATH 500
AIME24
Lightweight Models
1lama-3.1-8B
4.8e-5
0.19
2.7e-2 2
18.58
3.38
15.33
GPT-4o mini
5.4e-5
0.22
1.3e 2
25.38
2.06
14.67
Llama-3.3-7000
1.6e-4
0.16
7.4e-3 3
18.58
1.31
10.67
Large Models
Llama-3, 1lama-3.1-405B 1-405B
6.9e-4
0.14
6.7e-3
10.43
1.13
8.67
Claude Sonnet-3.5
2.1e-3
0.19
6.4e-3
14.06
2.54
14.67
GPT-4o
-.3e-3
0.17
6.2e-3
14.07
0.96
14.01
Reasoning Models
OpenAl ol-mini
5.4e-3
0.17
1.3e-
12.27
0.50
4.80
OpenAl ol
1.9e-2
0.22
4.3e-2 2
8.07
0.90
2.85
DeepSeek-RI
1.8e 3
0.17
1.5e 2
14.57
0.21
3.41
OpenAl 03-mini
1.1e-
0.11
1.1e-2 2
8.18
0.76
2.03
Table 1: Frontier dollar cost-of-pan per model / dataset. Each entry is the expected dollar cost of problem Mo). Per column, the 3 entries
p ~ D with the presence of the model and a human expert: Vp-D(( with the lowest value (i.c. best fronticr cost-of-pass have blue highlights. Different model families emerge as cost-effective at different task categories, highlighting the strengths of our evaluation.
EXPERIMENTS
3.1 EXPERIMENT SETUP
Models. We consider three categories of models:
(1) Lightweight models: We use the per-token cost as a proxy and select models with a cost less than
$1 million input and output tokens (see Table 4 Llama-3,1-80 (Grattafiori et al. 2024). GPT-4o
mini OpenAA 2024), and Llama-3.3-70B
(Meta-Al
2024).
(2) Large models: We select large general-purpose LMs: Llama-3.1 -40BB (Grattafiori et al. 2024). Claude Sonnet-3.5 (Anthropic 2024). and GPT-4o (Hurst et al. 2024).
(3) Reasoning models: We select models with special reasoning post-training, including OpenAI's ol-mini (Jacch et al. (2024), ol (Jaech et al. 20244), and o3-mini (OpenAl 2025), as well as DeepSeek
R1 Guo al. 2025).
Within each category, we select three to four representative models released between the second half of 2024 and early 2025. To preserve the integrity of our temporal analysis, prioritize the earliest stable releases and exclude research previews or experimental versions.
Datasets. We evaluate models across three sets of tasks:
(1) Basic quantitative tasks: These involve basic numerical reasoning. We include an arithmetic dataset (Two Digit Addition) assess basic numerical computation, and GSMBK (Cobbe al. 2021) to evaluate multi-step grade-school level problem solving.
(2) Knowledge-based tasks: These require recalling and reasoning over factual knowledge. We include a scientific inonlevge-intensive question answering task (GPQA-Di amond (Rein et al. 2024)) evaluate models' ability to recall and utilize complex scientific facts, and bias benchmark (BBQ (Pariish et al. ) to evaluate whether models rely on stereotypical knowledge or can disambiguate factual responses from biased defaults.
(3) Complex quantitative reasoning tasks: These require complex mathematical reasoning and problem solving. We use MATH-500 (Hendrycks al. 2021 Lightman al. 2023 assess models
The full derivation, justification, and sources for approach are detailed in Appendix A The resulting
estimates in Table

---

Published as a conference paper at ICLR 2026
Two Digit Addition
BBQ
MATH500
daade-15
clace-1.
deade335
Oaga 3.408
S 1-355
GSM8K
GPQA Diamond
AIME 2024
32440 duwwa
25
cludo-8
20
60
Cluude-3.2
Tarra 4000
larad 3-10
02-181
5
20
3ara3 1-1ee
3-20 donosool
Figure 2:
The frontier dollar cost-of-pane pass-ppssssss (i.. (M))) steadily decreases with new model releases, spanning models released between May 2024 and February 2025. Y-axes normalized (divided by Vy--n(MM)) shown percentage (%)).
on competition-leve maths problems, and AIME-24 Mathematical Association of America 2024 to evaluate performance on challenging problems from the 2024 American Invitational Mathematics Examination.
Evaluation protocol. All implementation details including model API providers, per-token pricing, prompt template, sampling budget, and accuracy/cost calculation details are shared in Appendix B
3.2 FRONTIER COST-OF-PASS WITH A SINGLE MODEL
In this experiment, we aim to quantify the economic value each model m generates on different distributions of problems p ~ D. For this, we take exment as baseline and quantify the frontier cost-of-pan passsss--ssssooo of problem in the presence of the model m: VP-D((m) U Mo).
The results in Table I highlighting the lowest three instances per dataset, show that our frontier cost-of-par pass effectively captures how different model families offer economic advantages across various task categories. We find that lightweight models yield the lowest frontier cost-of-pass on basic quantitative tasks, such as Two Digit Addition. This outcome aligns with the observation that all model familics achieve high accuracy on this dataset (see Table 5) which in turn makes the least expensive models appear most cost-effective. In contrast, for knowledge-basse models achieve lower frontier cost-of-pass pass compared to lightweight ones. While the reasoning models, such as o1, are priced significantly more expensively compared to both large and lightweight models, they lead to significant performance improvements, which, overall, result in reductions the cost-of-pass mainly in complex quantitative tasks.
tasks, larger
In contrast, when either task performance (pm(p~ D)) or cost (Cm(p ~ D)) is solely taken into account (Tables such metrics tend favor either reasoning lightweight models respectively due to their significant edge per criteria, without assessing the nuances in the economic impact they induce. This effectively highlights the sophistication of our metric and evaluation framework.
3.3 TRACKING FRONTIER COST-OF-PASS WITH NEW RELEASES
In this experiment, we track the improvements the frontier cost-of-par -ass- for problem. Figure shows the trends of the cumulative gain per dataset VP=D(MM) each updated by the corresponding model release (M₃-1 U {m,)). We observe steady decline in the frontier cost-of-pass, for complex quantitative tasks. In contrast, knowledge-bert and basic quantitative tasks typically exhibit a sharp initial drop in frontier cost-of-pass with the early releases of models, followed by plateau. To
7

---

Published as a conference paper at ICLR 2026
quantify the cost reduction trends, we empirically fit an exponential decay curve of the form:
(x(M)) = ae ae +c,
(10)
where denotes time in months since the first model release, and a, b, and are fit parameters. From this, compute the time for the exponential component of the cost drop by 50%: Ti/2 in(2) /b. Using this formulation, we find that for complex quantitative tasks, between May 2024 and February 2025, the frontier cost-of- pass for 500 halved approximately every 2.6 months, whereas for AIME-2024, the halving time was 7.1 months; indicating consistent cost reductions over the past year.
3.4 ESSENTIALNESS OF MODEL. FAMILIES: COUNTERFACTUAL FRONTIER COST-OF-PASS
showed the frontier cost-of-pase -fa-ss-passsss decreasing over time with new model releases. To understand which model families were most critical to this progress, we conduct a counterfactual analysis that quantifies the impact of removing each family. Defining Mg as a family of models (lightweight, large, or reasoning), we measure the counterfactual contribution of family 9 on dataset D by calculating the relative improvement in frontier cost-of pass attributable its inclusion:
Section 3.3
GpDD(Mg:Mp MM)
(11)
Vp~D(Mr \ (Mg)
Here, MM includes all models used in our experiments. This metric represents the relative improve- ment in the final frontier cost-of-pass (p~D((m) attributable to the model family MMg with higher values indicating greater essentialness of that family for achieving the current frontier.
Basic Quantitative
Knowledge Based Diamond
Complex Quantitative MATH500
Digit Additonn
Lightwwight
33.7
2.9
0.2
Model Family
Large
0.0
0.0
44.7
33.3
0.1
0.1
Reasoning
74.4
81.0
Figure 3: The relative improvement (%) in frontier cost-of-pass pass attributable to cach model family g, calculated under counterfactual setting where M2 is removed. Higher values signify greater essentialness for maintaining the current frontier.
Figure iillsstrate our main findings, revealing distinct roles across model families. Lightweight models help reduce the frontier cost-of-pass basic quantitative tasks, while large models are only essential in knowledge-intensive tasks. Reasoning models play a key role advancing the frontier for complex quantitative reasoning and also improve performance GPQA-Di ammond,, as well as GSM8K, which benefits from small reasoning models like o3-mini.
These findings highlight that progress different task types is driven by different model paradigms. While large models have brought clear gains on knowledge-intensive tasks (e.g. GPQA), improvements in cost-efficiency, especially in more quantitative domains, appear largely driven by advances in lightweight and reasoning models. Together, these suggest that the current cost-efficiency frontier, as reflected our framework, shaped mainly by (i) lightweight models and (ii) reasoning models.
3.5 IMPACT OF INFERENCE TIME TECHNIQUES ON FRONTIER COST-Or-PASS
We now assess whether common inference-time techniques provide meaningful economic benefits. Specifically, ask: is it cost-effective improve model performance through these techniques, compared to relying on the models' baseline performance? To explore this, we focus on the set of lightweight and large models, denoted by Mr. First, we determine the frontier cost-of-pass achieved
by M, without any modifications. We then apply given inference- time technique uniformly across all models in Mr. yielding modified set Mÿ, The gain from this technique, measured relative to
8

---

Published as a conference paper at ICLR 2026
Basic Quantitative
Knowledge Based
Complex Quantitative
Inference Time Technique
Two Digit Addition
GSMSK
BBQ
GPQA Diamond
MATH500
ATME24
TALE-
24.5
Self- Refinement
0
0
6.7
24.9
0
0
Majority Voting (k=3)
Majority Voting (k=4)
0
0
0
Table 2: Relative performance gains (%) from different inference time techniques across datasets.
the original frontier cost-of-pass, can be computed follows:
Gp-D(Mi, M.)
(12)
(p-
We consider two popular techniques: self-refinement Madaan al. (2023) and majority voting (a.k.a. self-consistency; Wang et al., 2023), with and 4 votes. Moreover, we evaluate a budget-aware inference-time technique: TALE-EP Han et al. (2025) well. As shown in Table self-refinement shows some economic benefit on knowledge-intensive tasks, considerably 24.9% improvement on GPQA-Di amond. In contrast, majority voting (despite potentially enhancing accuracy) does offer relative economic improvement across the tested models and datasets. Meanwhile, the budget- aware technique contributes meaningfully in many more of the tasks to reducing the frontier cost-of-pass.
Collectively, these findings suggest that, for the evaluated techniques, the costs by performance- oriented methods often outweigh accuracy gains when measured by the frontier cost-of- pass. By contrast, TALE-EP ELEEEP (conditioning generation on a self preddcteddedd token budget) yields visible reductions on subset of tasks, though benefits are uneven. This implies that such common inference-time approaches may currently offer limited economic benefits within our evaluation framework.
RELATED WORKS
Economic perspectives and broader impacts. The efficiency of LMs carries significant economic implications, they are viewed as general-pun purposs technologies impacting productivity and labor (Eloundou et al. 2024 (Brynjolisson et al. 2025). Recent scholarship in economics further examines how Al changes which tasks are automated, how expertise organized, and how R&D productivity evolves across domains (Svanberg et al. 2024 Autor & Thompson 2025 [Beiiroglu et al. (024). Complementary economic analyses explore provider strategies regarding pricing and product design (Bergemann et al., 2025), trade-off's between training and inference compute (Villalobos & Atkinson 2023 and user-side maaiing involving ROI, token costs, and success probabilities (Xexéo al. 2024).
Our cost-of-pass metric serves as a bridge between these technical realities of model performance and their economic consequences. By providing the expected monetary cost successfully complete a task, it allows for quantifying the economic contribution of specific AI systems and informs rational model selection for achieving economic viability, and provides quantitative perspective the economic evolution of the LM ecosystem.
LM resource consumption, efficiency optimization and benchmarking, Research increasingly recognizes the importance of LM resource consumption and efficiency. Empirical studies already connect benchmark performance to monetary cost by tracking LLM inference prices and historical price-perr piicepperfomance trends across tasks and benchmarks (Cottier et al., 2025 Gundlach et al. 2025). These empirical evaluations are complemented by work that have quantified operational costs like tokens (Chen et al.) 2023) and energy (Maliakel et al. (025). revealing task-dependent performance and potential diminishing returns from high expenditure (Miserendino al., (005). This focus has intensified with the rise of reasoning methodologies (Sui et al. 2025) and inference-time techniques (c.g., Madaan et al. (2023); Wang et al. (2023)), which often trade increased computational cost for potential accuracy gains.
Concerns like "overthinking." where lengthy processing fails to improve results (Chen et al. 2024 Cuadron et al.) 2025), have spurred efforts optimize resource through methods like dynamic token budgeting (Han et al. 2025), specialized training (Arora Zanette 2025), prompt engineering
(Xu et al. 2025
Aytes al. 2025) researching optimal reasoning lengths (Wu et al. 2025

---

Published as a conference paper at ICLR 2026
Yang et al. 2025 Concurrently, evaluation methodologies have evolved beyond pure accuracy correctness measures.
Recognizing its insufficiency, researchers have incorporated cost via fixed budgets (Wang et al. 2024 performance heuristics (McDonald et al. 2024), or non-monetary metrics like conciseness (Nayab for
al. 2024). Kapoor et al. (2024) strongly advocated for using real dollar costs and accounting factors central to our approach. Benchmarking efforts have also highlighted
diminishing returns from simply scaling inference computation (Parashar et al., 2025). While these works underscore the need for cost-aware analysis, they often rely on specific constraints (e.g., fixed budgets) heuristic metrics.
Our cost-of-pass framework seeks to advance this by providing single, interpretable metric that adapts economic production principles, offering unified way assess the economic viability of different models and techniques without predefined budget assumptions or proxy metrics.
5 CONCLUSION
We introduced an economic framework designed to evaluate language models by integrating their performance with inference cost. Drawing from production theory, conceptualize language models as stochastic producers, and assess their efficiency using our proposed cost-of-patt pass metric, which measures the expected cost per correct solution. Our analysis utilizes this metric alongside the frontier cost-of-pass, defined as the minimum achievable cost compared to human expert baseline. This approach reveals distinct economic roles played by different model classes. For instance, retrospective and counterfactual evaluations demonstrate that lightweight models primarily drive efficiency on basic tasks, whereas reasoning models are essential for complex problem-solving. Critically, our findings show that common inference-tine techniques typically increase the cost-of-pass, thus failing to provide net economic benefits when compared the progress made by improving the underlying models themselves. We discuss the limitations of our methodology, outline directions for future work, and consider practical implications our framework in Appendix [D] Taken together, these insights underscore the value of our framework in offering a principled foundation for measuring language model innovation economic terms. It serves as a valuable tool for guiding model selection and aligning AI development with real-world value.
ACKNOWLEDGMENTS
We thank Federico Bianchi, Dan Jurafsky, Daniel E. Ho, Can Yeşildere, Semyon Lomasov and Aaron Scher for valuable comments and discussions in the early stages of this project. MHE gratefully acknowledges support from the Fulbright Foreign Student Program. BE gratefully acknowledges the support of the Stanford mnight-Hennessy Scholarship. MS gratefully acknowledges the support of an HAI-SAP Fellowship and a Google PhD Fellowship.
REFERENCES
Daron Acemoglu. The Simple Macroeconomics of AI. NBER Working Papers 32487, National Bureau Economic Research, Inc, May 2024. URL https::/ideas. wwwp./ppe...r///////bbbeeeeeeeee
32487 html
Dennis Aigner, C.A.Knox Lovell, and Peter Schmidt. Formulation and estimation of stochastic
frontier production function models. Journal of Econometrics, 6(1):21-37, 1977. ISSN 0304- 4076. doi: Ccmp://seeeee//ttte////////44110
https://ww..cciencddrree..
Anthropic. Claude 3.5 sonnet announcement, 2024. URL https://ww.aattoooic.coo//eeeeee
Accessed: Feb. 2025.
Daman Arora and Andrea Zanette. Training language models reason efficiently. arXiv preprint arXiv:2502.04463 2025.
Art of Problem Solving. American invitational mathematics examination - AoPS wiki,
2023. URL htw.s: http:::tttttttofaaeeees httttp////artrffrrolllssslliiii https//wwww..aoccooo/aaaaaeeeeea wtwp:e/wwwwpppttteeeeeeeee.....e Invi tational
10

---

Published as a conference paper at ICLR 2026
Mathematicn Examination States AIME is 15 questions in 3 hours ( [22 min per problem). Accessed 2025-05-14.
David Autor and Neil Thompson. Expertise. Journal of the European Economic Association, 23(4): 2025. doi: 10.1093/jeea//vaf02.. 4/1203/8955555 Published June 26, 2025.
Simon A Aytes, Jinheon Back, and Sung Ju Hwang. Sketch-of-thought: Efficient IIm reasoning with adaptive cognitive-inspined sketching. arXiv preprint arXiv:2503.05/77, 2025.
Dirk Bergemann, Alessandro Bonatti, and Alex Smolin. The economics of large language models: Token allocation, fine-tuning, and optimal pricing. arXiv preprint arXiv:2502.077736 2025.
Tamay Besiroglu, Nicholas Emery-Xu, -u, and Neil Thompson. Economic impacts of ai-augmented r&d. Research Policy, 53(7):105037, 2024.
Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling, 2024. http:://s///a org/ aaaaabbs/22.ee 21787
Erik Brynjolfsson, Danielle Li, and Lindsey Raymond. Generative work. The Quarterly Journal of Economics, 140:889-942, 2025.
Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint 0tt.::233000077666 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes. Ariel Herbert-Voss William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin. Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. URL
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of ol-like 1lms. arXiv preprint arXiv:2412.21187, 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jemry Tworek, Jacob Hilton, Reïchhro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Dataset licensed under the MIT License.
Ben Cottier, Ben Snodin, David Owen, and Tom Adamczewski. Llm inference prices have fallen rapidly but unequally across tasks. Technical report, Epoch AI, 2025. URL https://epoch... ai/ wwwpsiiihhhtlmmmmeeeeeeeee price- price-trends Accessed: 2025-11-17.
Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang. Yichuan Wang, Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, et al. The danger of overthinking; Examining the reasoning-action dilemma in agentic tasks. arXiv preprint arXiv:2502.088 08235, 2025.
Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. Gpts are gpts: Labor market 2024.
impact potential of Iims. Science, (0166770)::3361111
Michael James Farrell. The measurement of productive efficiency. Journal of the royal statistical society: series A (General), 120(3):253-28)) 1957.
Irena Gao, Percy Liang, and Carlos Guestrin. Model equality testing: Which model is this API serving? The Thirteenth International Conference on Learning Representations, 2025. URL
https://openrevieer net./forum2ideqqdd777335

---

Published as a conference paper at ICLR 2026
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama herd of models. arXiv preprint arXlv:2407.21783, 2024.
Hans Gundlach, Jayson Lynch, Matthias Mertens, and Neil Thompson. The price of progress: Algorithmic efficiency and the falling cost of ai inference. In NeurlPS 2025 Workshop on Eval- uating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling, 2025. URL https://openrevieww http OpenReview. Accessed: 2026-02-17.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl incentivizes reasoning in I1ms through reinforce- ment learning. Nature, 645(808 1):6336336, 2025.
Tingxu Han, Zhenting Wang, Chunrong Fang, Shiyu Zhao, Shiqing Ma, and Zhenyu Chen. Token- budget-aware IIm reasoning, 2025. UKL//tttss///arr..rr//////222112
Andreas Haupt and Erik Brynjolfsson. Position: Al should not be imitation game: Centaur evaluations. In Forty-secoodd International Conference on Machine Learning Position Paper Track, 2025. URL https://opeenreview.. wtw.p//wweeteoorrmmmdddd/nooooo.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track Dataset licensed under
(Round 2), 2021. the MIT License.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
InputiX. Learn about data entry rates per hour in india, 2020. URL Accessed: 17 February 2026.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint 1rX1::2412.16620, 2024.
Sayash Kapoor, Benedikt Stroebl, Zachary S Siegel, Nitya Nadgir, and Arvind Narayanan. Ai agents that matter. arXiv preprint arXiv:240.001502, 2024.
Michelle A. Kirchner. The effects of implementing Xtra Math program on student achievement. URL htRL/hhtss///vteerr wttpeeciiio////////eeeeeeeeeeeee 84988/doww /wwnnoowwoood States that the school district designated proficiency as 20 correct problems in one minute by end of year. Accessed 2026-02-18.
Master's thesis, Viterbo University, 2017.
contentdm. oww.. org/ htt//igiitat tal/
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step, 2023. URL
https://arxxv.org//
20050
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with seld-feedback. Advances in Neural Information Processing Systems, 36:46534 46594, 2023.
Paul Joe Maliakel, Shashikant Ilager, and Ivona Brandic. Investigating energy efficiency and perfor- mance trade-offs in 1lm inference across tasks and dvfs settings. arXiv preprint arXiv:2501.082) 2025.
Mathematical Association of America. MAA invitational invitaiionall competitions: American invitational math- ematics examination (AIME), 2024. URL https:://maa.rgg/////// http::/mwwa..rggaaaaaiiiiattaaaa compenliooommttiiiooon Accessed: 2025-05-14.
Tyler MeDonald, Anthony Colosimo, Yifeng Li, and Ali Emami. Can afford the perfect prompt? balancing cost and accuracy with the economical prompting index. arXiv preprint arXiv:2412.01660, 2024.
12

---

Published as a conference paper at ICLR 2026
Mcta-AL. Llama 3.3 70b instruct model, 2024. URL https://huggingfaaee http://gnnnnnacccco///eeeeeeeaaa 3-70B- Instruct License: Llama 3.3 Community License Agreement, see htttss://huuugiiggaacc co/meta- Lateettallamaaaamaaaa 3. 708--------00000 /ttoo//rruuttbbbo//////aLLaLCaL-
Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. Swe-lancer: Can frontier Ilms earn $1 million from real-world freelance software engineering? arXiv preprini 2211:12502./21155, 2025.
Sania Nayab, Giulio Rossolini, Marco Simoni, Andrea Saracino, Giorgio Buttazzo, Nicolamaria Manes, and Fabrizio Giacomelli. Concise thoughts: Impact of output length on Ilm reasoning and cost. arXiv preprint arXiv:2407.19825 2024.
OpenAL. Gpt-4o mini: Advancing cost-efficient intelligence, 2024. URL.intpss///peaai.co////oo hlc-444rhhii..aaaaciioooooof-f-- Ligence/
OpenAl. Openai o3-mini system card. Technical report, OpenA1, 2025. URL wtt.hhtt:://ppai..cmm
index/o3-mini-siee -as---istsssttccrrddd System card. Published January 31, 2025.
Shubham Parashar, Blake Olson, Sambhav Khurana, Eric Li, Hongyi Ling, James Caverlee, and Shuiwang Ji. Inference-time computations for I1m reasoning and planning: A benchmark and insights. arXiv preprint arXiv:2502. 2025.
Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. BBQ: A hand- built bias benchmark for question answering. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Findings of the Association for Computational Linguisties: ACL 2022. pp. 2086-2105, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. (0.118653/ htt.///.22n2/ii//// findings-acl. URL pettts://aclanntholoo org/2022. find22..ininggs-cca. acl. 165/ Dataset licensed under CC BY 4.0.
David Rein. Can good benchmarks contain mistakes? NYU Aligmment Research Group Blog. May 2024. URL https:///pp nyu. edu/ ha////rgg//// /aaarr////aaaggeeeeeeeeeee loan-good-beed htw--beechaark---ttiii---nn www.-oonnnnnnnniineiiiee Reveals GPQA expert pay (S100/hr) and non-expert solve times. Accessed 2025-05-14.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof qa benchmark. In
First Conference on Language Modeling, 2024. Dataset licensed under CC BY 4.0.
Rocket Math. Math fact fluency benchmarks (written testing). Technical report, Rocket Math, 2022. URL https://www. www.e tmath. wwtpc//wowwooteeeeeeee http:///nnnneeeeoooo/o////////// Fat.ttah.aaa.aaeeee.....
Benchmarks.200 2024. pdf) Early-grade grade benchmarks include addition 25 problems/min (1st grade)
to 30 problems/min (2nd and 3rd grade). Accessed: 2026-02-18.
SalaryExpert. Data annotator salary in india, 2025. http:://ww.sllaressstttcoc/ss//
job/data job/data-annotator annotator/india Accessed: 17 November 2025.
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03311. 2024.
Yang Sui, Yu-Neng Chuang, Guanchu Wang, Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Hanjie Chen, Xia Hu, et al. Stop overthinking: A survey on efficient reasoning for large language models. arXiv preprint arXiv:2503. 164 2025.
Mirac Suzgun, Mert Yuksekgonul, Federico Bianchi, Dan Jurafsky, and James Zou. Dynamic cheatsheet: Test-time learning with adaptive memory. arXiv preprint arXiv:2504.0793 2025.
Maja Svanberg, Wensu Li, Martin Fleming, Brian Goehring, and Neil Thompson. Beyond AI exposure: Which tasks are cost-effective to automate with computer vision?, 2024. Available at
SSRN:
TeacherOn. Maths olympiad tutors in india, 2025a. URL httpp::/www.eechhrrn..coo//////e Accessed: 17 November 2025.
-lympiad-tutors-inne
TeacherOn. Maths tutors in india, 2025b. URL htwp://wcchheeoo..oa/aaaaaatttoe
Accessed: 17 November 2025.
13

---

Published as a conference paper at ICLR 2026
TeacherOn. JEE maths tutors in india, 2025c. URL maths- iistors-iinindiii Accessed: 17 November 2025.
TutorCruncher. Average tutoring rates usa: How much do tutors charge per hour?, 2025. URL https://tutorccunncher httprrrrrnnccc..ccoo///////eeeee hatpll/gvvvrrae-tuutto rates Reports $45 $100/hr as typical range for test-prep tutoring. Accessed 2025-05-14.
Upwork. How much does a data entry specialist cost?, 2025a. URL httpp::/www.upwerr http://ww..puork.ccm/mireree (atar-entry-peciialsst/oooo Median $13/hr for data entry freelancers; $10 $20/hr typical range. Accessed 2025-05-14.
Upwork. Hire the best scientific researchers in india, 2025b. URL https://ww..pporrr.cc//rrrr Scceettific-rrssarchhrr///// Accessed: 17 November 2025.
U.S. Bureau of Labor Statistics. Customer service representatives: Occupational out-
look handbook, 2025. URL httpps///ww.bbr www..bll..ov/oooo/offcccc... and adwinistratiiveeuuoor customer sepresee
wtt.:oo//ooofiii-aaaaaaaaaaiiiii htm Accessed: 2025-09-23.
Pablo Villalobos and David Atkinson. Trading off compute in training and infer-
ence. Technical report, Epoch AI, 2023. URL. https://epoch: https://eoch.aa//uuuiicttt ai/publications/
ittprittteiniittiinnnnnnnnnnnn training and- innfieeene Accessed: 2025-11-18.
Junlin Wang, Siddhartha Jain, Dejiao Zhang, Baishakhi Ray, Varun Kumar, and Ben Athiwaratkum. Reasoning in token economies: Budget-aware evaluation of Ilm reasoning strategies. arXiv preprint arXiv:2406.064661 2024.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL.
https://openr Met:forum?id=1PLINNIN INNINNMM
Yuyang Wu, Yifei Wang, Tianqi Du, Stefanie Jegelka, and Yisen Wang. When more is less: Under- standing sreain-of-thougeee length in llms. arXiv preprint arXiv:2502.00266 2025.
Wyzant, New jersey math tutors cost $33-$55 per hour on average, 2025. URL httpss://ww. wyzant com/New_Jersey weatthtuutors.com aspx) Average private tutoring rates for math (K-12 and competition). Accessed 2025-05-14,
Geraldo Xexéo, Filipe Braida, Marcus Parreiras, and Paulo Xavier. The economic implications of large language model selection on earnings and return on investment: A decision theoretic model. arXiv preprint arXiv:2405. 2024.
Silei Xu, Wenhao Xie, Lingxiao Zhao, and Pengcheng He. Chain of draft: Thinking faster by writing less. arXiv preprint arXiv:2502. 18600, 2025.
Wenkai Yang. Shuming Ma, Yankai Lin, and Furu Wei. Towards thinking-optine optimall scaling of test-time compute for I1m reasoning. arXiv preprint arXiv:2502.18083 2025.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. r-bench: benchmark for tool- agent-user interaction in real-world domains, 2024. h/tps://t/// org/abs////6..... 12045)
Zendesk. What is average handle time (aht) and how do you calculate it?, August 2025. URL zendesk. ctw/blog/ aaer handle time/ Accessed: 2025-09-23.
Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, and Summer Yue. A careful examination of large language model performance on grade school arithmetic, In NewrIPS 2024 Datasets and Benchmarks Track, 2024.
14

---

Published as a conference paper at ICLR 2026
A DETAILS OF HUMAN EXPERT COST ESTIMATION
In this section, we provide a detailed analysis of how the human expert costs in Table are calculated per dataset.
Dataset
Qualification Requirements
Hourly Rate
Time per Question
Est. Cost
AIME
Advanced high-school contest math skills
$45-$100
12 minutes
$9-$20
BBQ
General familiarity with social biases
$15
0.44 minutes (24 sec)
$0.10
GPQA Dia.
Graduate-level domain exper-
$100
35 minutes
$58
tisc
GSM8K MATH500
Basic arithmetic reasoning
$33-855
3.7 minutes
$2-$3.50
Strong competition-lveel
$35-$60
12 minutes
$7-$12
problem-solving
Two-Digit Add.
Basic numeracy
$10-$20
00.04 minutes (~~2-3 sec)
$0.01-$0.02
Table 3: Estimated costs of hiring human expert to solve one question from each dataset, based on typical qualifications, hourly rates, and time per question.
AIME (American Invitational Mathematics Examination) consists of 15 challenging math problems in 3-hour contest (administered in two separate sections: AIME I & II), giving an average of about 12 minutes per problem (Art of Problem Solving 2023). In practice, expert math tutors for competitions like AIME command high hourly fees in the range of $45- reflecting intensive test-preparation rates (TutorCruncher, 2025). This rate range aligns with specialized test prep tutoring in the US, which is higher than regular tutoring due the advanced problem-solving skills required (TutorCruncher 2025). At roughly 12 minutes AIME question on average, solver could handle about five such problems per hour under exam conditions (Art of Problem Solving 2023).
BBQ (Bias Benchmark for QA) contains short question-answer scenarios targeting social bias. Crowdworkers annotating BBQ have been paid around $15 per hour, a rate chosen to exceed U.S. minimum wage (Parrish et al. 2022). Because each task includes multiple BBQ questions, workers were able to answer roughly 5 questions in 2 minutes (Parrish et al., 2022) i.e. <~24 seconds per question, about 0.4 minutes per question. This fast per-question time reflects the fact that BBQ items are short multiple-choice queries, allowing human annotator to complete approximately 150 BBQ questions in an hour at that pay rate (Parrish et al. 22022).
-riamond consists of extremely difficult graduate-level science questions, so human experts demand high compensation. In one case, domain experts were paid about $100 per hour to contribute and validate GPQA questions (Rein et al. 2024). These questions are "Google prooff" and time- consuming: skilled non-expert participants spent over 30-35 minutes average per question when attempting to solve GPQA problems with unrestricted web access (Rein et al. 2024). This long duration per question underscores GPQA's 's complexity: at most questions could be solved in hour even by motivated annotators, which justifies the premium expert hourly rate (Rein 2024).
GSM8K contains grade-school level math problems. Solving these is relatively time-efficient for adults: in one study, crowdworkers under time pressure managed to solve about 4.07 GSM8K problems in 15 minutes on average (Zhang al. 2024), or roughly 3.7 minutes per question. The required skill is comparable to general math tutoring at the K-8 level, for which typical U.S. tutor rates are about $33-$55 per hour on platforms like Wyzant (Wyzant 2025). At such rate, paying a person to solve GSM8K problems would be economical, given that proficient solver can complete approximately 16 questions in one hour (Zhang al. (024)).
MATH-500 is set of 500 advanced competition math problems (drawn from the larger MATH dataset). These span a range of difficulty levels (from relatively routine competition questions up to AIME-level problems) so strong general competition-math tutor has typically sufficient qualifications. As with AIME, well-prepared human might spend on the order of 10-15 minutes per problem, roughly 12 minutes on average for hard competition question (Art of Problem Solving (023). Tutors capable of solving and teaching such competition math problems often charge rates on the order of $50 per hour (with typical range of $35-$60 for competition math tutoring)
15

---

Published as a conference paper at ICLR 2026
Therefore, solving roughly five MATH-500 problems could cost about $50 and take around an hour, consistent with the per-question time and high skill required.
(Wyzant 2025
Two-Digit Addition consists of simple two-digit addition problems, which very quick for humans to solve. Early elementary school students are often expected to complete about 20-30 basic addition problems in minute (Kirchner) 2017 Rocket Math 2022). This corresponds to roughly 2-3 seconds per addition (~0.04 minutes per question). Because the task so elementary, the labor to solve large numbers of such problems be valued at lower hourly rate. Simple data-entry style work or basic math tasks on freelance platforms pay on the order of $10-$20 per hour (Upwork) 2025a). At $15/hour, for example, worker could theoretically solve several hundred 2-digit additions within the hour, given the ~23 second solution time (Kirchner 2017: Rocket Math 2002)
B DETAILS OF EVALUATION
For each dataset in our evaluation, we sample to 128 instances and run each mode - 8 times estimate the expected runtime cost and accuracy per sample. We use temperature of 0.7 and top-p of 1.0 for all models except OpenAl's reasoning models, for which we set the temperature 1.0 without applying top_p. Additionally, we the default maximum token generation limits provided by each model. Following Suzgun et al. (2025). we use a concise but descriptive instruction prompt for models follow:
Experiment Prompt
Please solve the following question. You can explain your solution before presenting the final answer. Format your final answer as:
<answer>
</answer>
Instructions:
For multiple-choice: Give only the letter (e.g., (A)).
For numeric: Give only the number (e.g., 42).
For free-response: Provide the full final answer text.
INPUT:
{input}
In our experiments, define the pass I'm (p) as whether the model obtains a correct answer after single run or not (0 or 1), and the cost Cm (p) as:
Cy (p) = 7amn (m. p) (m)) + Tout (m, p) Cour (m)
(13)
where n= nm,,, p) denotes the number of input / output tokens consumed / generated by the model m on problem p, and (mm)) denotes the dollar costs per input output tokens consumed generated by the model m (see T the pricing). For the expert costs, utilize the estimations from 3 and set the rates the upper-bound value to ensure the approximation of the expert accuracy being I. Finally, shown in Table 4 open-source models are queried through a single provider for consistency and simplicity (TogetherAI, in our case).
we access proprietary models via their original providers, while
$ Here, the short-form "model" refers to the underlying model together with its inference pipeline (prompt, decoding settings, etc.). Comparisons throughout the paper are done this basis, and Appendix shares the adopted details.
16

---

Published as a conference paper at ICLR 2026
Cost (per million tokens)
Category
Model
Release Date
Input Tokens
Output Tokens
Llama-3. 1--na-3.1-8B
7/23/2024
S0.18
S0.18
Lightweight Models
GP'I-40 Mini
7/18/2024
$0.15
$0.60
Llama 3.B-70B
12/6/2024
$0.88
$0.88
Llama-3. 1-40B
7/23/2024
$3.50
$3.50
Large Models
GP
5/13/2024
$2.50
$10.00
Claude Sonnet-3.5
6/20/2024
$3.00
$15.00
OpenAl mi-mini
9/12/2024
$1.10
$4.40
OpenAl 03-min
1/31/2025
$1.10
$4.40
Reasoning Models
DeepSeek-KI
1/20/2025 12/5/2024
$7.00
$7.00
OpenAl
$15.00
$60.00
Table 4: Per-token inference costs with release dates, Each model name links the utilized provider.
C ADDITIONAL RESULTS
C.I EXPECTED ACCURACY AND INFERENCE COSTS
As discussed Section 3.2 we report the expected accuracy and cost for each model per dataset, denoted Rm (p ~ D) and (m D). To compute these, following the methodology in Section 2.5 we use the i.i.d. sampled set D of problems per dataset and approximate the expectation by averaging the accuracy Rm and cost (p) across problem instances. The results Tables and reveal skewed preference for particular model families under each metric, suggesting that these metrics alone are insufficient capture the economic impact of models.
Basic Quantitative
Knowledge Based
Complex Quantitative
Model Category
2-Digit Add.
GSMSK
BBQ GPQA Dia.
MATH 500
AIME24
Lightweight Models
Llama-3.1-88
89.45
75.78
21.48
17.87
37.30
12.50
GPT-4o mini
99.90
88.57
53.32
18.07
70.02
14.58
I.lmm-3.3-70B
99.90
92.09
85.06
46.48
72.75
33.33
Large Models
Llama-3.1-4055
99.71
93.95
85.74
44.14
67.87
31.67
Claude Sonnct-3.5
100
94.43
92.58
55.37
64.75
15.83
GPT-4o
99.71
91.99
90.04
47.07
73.14
14.58
Reasoning Models
OpenAl ol-mini
99.51
92.58
85.74
49.12
85.94
53.33
OpenAl o1
100
94.04
95.02
73.83
89.45
72.50
DeepSeek-R1
100
93.36
83.69
54.88
93.85
60.83
OpenAl o3-mini
100
92.77
83.79
71.68
88.57
77.08
Table 5: Accuracy (%) per model per dataset: Rm ~ D). In each column, the 3 entries with the highest accuracy have blue highlights.
C.2 EVALUATION ON A REAL-WORLD DOMAIN
We evaluate our framework on Tau-bench Yao et al., 2024), a benchmark that targets tool use, agent behavior, and user interaction in real-world domains. We sample 8 tasks per category (airline, retail), totaling 16 tasks, and run each model as an agent under the evaluation protocol described in the original paper. We exclude DeepSeck-RI due its visible chain-of-thouu messages, which contaminates responses under this protocol. We apply the cost modeling based on total tokens consumed generated per round, and we aggregate costs interaction rounds. Estimates are averaged over 4 independent trials per run.
being mixed with user
For the human-exper baseline, consider the "retail or call-center communication" qualification, with an hourly wage of $20.59 (U.S. Bureau of Labor Statistics 2025) and an average minutes per task (Zendesk) 2025), which yields $2.06 per task.
17

---

Published as a conference paper at ICLR 2026
Basic Quantitative
Knowledge Based
Complex Quantitative
Model Category
2-Digit Add.
GSM8K
BBQ GPQA Dia.
MATH 500
AIME24
Lightweight Models
Llama-3.1-8B
4.2e-5
7.4e-5
5.2e-5
1.8e-4
1.5e-
2.2e-4
GPT-4o mini
5.4e-5
1.9e-4
1.0e-4
3.9e-4
3.7e-4
5.6e-6
Alama-3.3-708
1.6e-4
3.3e-4 4
3.1e-4
9.6e
6.7e-4
1.le-3 3
Large Models
Llama-3. 3.1BB
6.9e-4
1.4-3 3
1.0e-3
3.0e-3
2.4e-3
3.7e-3
Claude Sonnet-3.5
2.1e-3
3.7e-3
3.0e-3
6.9e-3
5.9e-3
7.5e-3
GPT-4o
-.3e-3
4.5e-3
2.7e-3
0.01
8.7e-3
0.01
Reasoning Models
OpenAl ol-mini
5.4e-3
8.4e-3
7.6e-3
0.02
0.02
0.07
OpenAl ol DeepSeck-R1
0.02
0.03
0.04
0.25
0.13
0.52
1.8e-3
5.1e-3
4.6e-3
0.04
0.01
0.04
OpenAl o3-mini
1.le-3
2.1e-3
2.6e-3
0.01
5.4e-3
0.02
Table 6: Dollar cost incurred per model per dataset: Cm D). In each column, the 3 entries with the lowest cost have bluc highlights.
Lightweight Models
Large Models
Reasoning Models
Llama-3.1-80 1-8B
1.6770
Alama-331-4053
1.8875
OpenAl o1-mini
1.8230
GPT-4o mini
1.2944
Claude Sonnet-3.5
1.5135
OpenAI ol
1.6406
Llama-3.3-700
1.6897
GPT-4o
1.2247
OpenAl o3-mini 1.2703
Table 7: Frontier dollar cost-of- pass per model Tau-bench real wollworld tasks. Each pair of columns lists models (left) their frontier cost-of-pass pass with respect to human expert baseline (right): (p-D((m) U.Ma). The lowest three values are highlighted in blue, indicating that all the model families have an economic merit in this task.
We repeat the analyses in Sections 3.2
3.3 and 3,4 and share the results in Tables 7 respectively.
Our overall findings indicate that (1) model families have merit this task, (2) the evolution of the frontier cost-of-pass still follows exponential decay (similar to other tasks), and (3) none of the model familics are significantly essential in driving progress.
C.3 RELATIVE GAIN PER MODEL RELEASE
Figure presents the relative improvement in temporal frontier cost-of-pass for each model release, illustrated using bar plots. Namely, we calculate:
Fo-D(Ime,,Maa Vp-D(Mi-1)
(14)
The results indicate that the reasoning models demonstrate notable advancements, particularly on complex quantitative tasks. In contrast, lightweight models exhibit marked gains on basic tasks. These findings support the observations from our experiments (Sections 3.2 3.4). Notably, The substantial improvement observed for GPT-4o is likely due being the first model included in
analysis, resulting in a pronounced leap relative to the baseline cost associated with human expert annotation.
May 13
Jun 20
Jul 18
Jul 23
Sep 12
Dec
Dec 6
Jan 31 OpenAl
Claude Sonnet-3.5 3.5
GPT-4o
Allama-3.1-88
OpenAl
Llama-3.3
OpenAl
GPT-4o
mini
Llama-3.1 3.1BB0
ol-mini
70B
ol
o3-mini
1.2247
1.1900
0.8411
0.8127
0.8127
0.8021
0.7668
0.7311
Table 8: Frontier dollar cost-of-pass -aass model release dates Tau-bench. Each column reports the best- to-date frontier value Vp-D(M,) after incorporating models released on the indicated date. The trajectory continues to follow an exponential decay, consistent with other tasks. This table is the tabular version of the time-evolution figure (see Fig. for example).
18

---

Published as a conference paper at ICLR 2026
Lightweight
Large
Reasoning
Essentialness (%)
22.5
13.2
6.0
Table Essentialness model families on au-bench (metric from Section 3.4.
The results show that
Lightweight models are the most essential, but overall, none of the families are strongly essential in driving progress for the frontier cost-of-pass.
Two Digit Addition
BBQ
MATH500
100
80
40
20
g0t-40
clarde go-4o-iii
Bammae f-4038
of Foniiei
dt demedd 9-708
(engnedert
grt-io chanda 1.5
got- vourina
Itana E-a8
Baema 1-4050
Bemrac Semmac33--18 3-998 despoalrr
ot
unt-io
clasdd 1.8
git- rerii
Hema 6-088
Haana f-409p
a kintnl
of Beman 3-5--
GSMBK
GPQA Diamond
AIME 2024
40
20
Bi-40
dleat-33
Lamt-1 1-4090
1-10
Honn-3 5-718 dmppaccr
comiei
urt-do
dlsude--.5 ast mivimm
tarria fal
Gano 14000
Ronad R 5-1110 despselrr
65-iill
git-4o
d noviii
term-3 1-10
Barr 1-458
Bomac 5-m1 dmpstek--
of
dbminl
Figure 4: Bar plot showing the percentage of change in frontier cost-ol-pass per model release (i.e. Gpe p(m (m.).M,-1)
Vp-D(M,-1)
C.4 ESSENTIALNESS HUMAN EXPERT BASELINE
Adapting the methodology in Section 3.4 for each task. We treat the human-expert baseline as separate family, Mg. and compare it to the remaining models, .M Mo. via
we quantify the essentialness of the human- baseline
Gp-D(Ma, Mr \o) VpnD(Mp (o)
(15)
Under this definition, essentialness 100% if there exists least one instance in the distribution that no model in Mr \ MM can solve, so the frontier requires Mo on some part of the distribution. Conversely, if every instance solved strictly lower cost by the LMs, essentialness is 0%.
Applying this analysis, we find that human experts remain fully essential for GSM8K, GPQA-Da -og amond, MATH-500, and AIME-2024, and non-essential (0%) for BBQ and Two-Digit Addi tion. Interestingly, there is no task where human experts are partially necessary (in between 0-100%).
C.5 ESSENTIALNESS OF SINGLE MODELS
In this section, following the methodology outlined in Section 3.4 we quantify the relative improve- ment in frontier cost-of-pass, using counterfactual approach. Specifically, for each model m,, calculate the following:
GpnD((me), Mr \ fm.))
(16)
Vp~ \ (m+))
quantifying the essentialness of the model m.. The results presented the contributions of most individual models largely compensable by the remaining models.
that
19

---

Published as a conference paper at ICLR 2026
Furthermore, observe similar coarse-level trend, noted in Section 3.4 indicating that different model families provide greater benefits in specific task categories.
Basic Quantitative TwsDigiAAdditinn
Knowledge Based
Complex Quantitative
Diamond
MATH500
1ama-3.1-88
0.2
0.0
Lightwelgh ght
9pt-4o-mini
0.2
0.0
lama-3.3 3-3-00
liama 3.1 1-00B
0.1
Model Left Out
Large
dlsude-3.5
0.0
spl-4o
0.0
0.0
o1-mini
0.0
0.0
0.0
Roasoning
01
0.0
0.0
24.8
deepseak-r1
0.0
0.0
0.0
4.2
0.0
63-mini
0.0
33.0
0.0
0.8
0.3
5.0
Figure 5: The relative improvement (%) in frontier cost-of-pass under a counterfactual setting, removing a model m. from the model set My. High values mean that the model is essential for maintaining the current frontier.
C.6 REGIONAL ABLATION FOR HUMAN EXPERT COST ESTIMATION
As realistic sensitivity analysis of the human expert cost estimates, we change the region India and re-estimate the hourly rates for the stated qualifications in Table The estimates in Table TO report the updated hourly rates and the resulting estimated cost per question for all datasets. We then repeat the analysis in Section 3.2 and present the results in Table [1] Our findings indicate that while the main interpretation of the analysis does not change, there is a slight shift preference toward lightweight models when the region is set to India. We interpret this as effect of the human-expert cost influencing the penalty for failure: when the human-expen cost is lower, the framework becomes more forgiving of eπrors, thereby increasing the relative appeal of lightweight models.
Dataset
Hourly Rate (INR)
Reference
Est. Cost (INR)
Est. Cost (USD)
AIME BBQ GPQA Dia.
1.250-2.000
TeacherOn
2025a
250-400
$2.84 $4.55
SalaryExperi
2025
2,97
$0.034
3,520-4,440
Upwork
2025b
2,053-2,567
$23.33-$29.1
GSM8K
350-800
TeacherOn
2025b
21.59
$0.25-$0.56
MATH500
6,201.200
(TeacherUn
2025c
120- 240
$1.36-$2.
Two- Two-Digit Add.
175-275
(InputiX
2020)
0.12 0.18
$0.0013 30.000200002
Table 10: Human Human-expert cost estimation when the region changed to India. The qualification requirements
and time per question are used from Table The conversion rate of INR88:$1 is used.
C.7 CONFIDENCE INTERVALS FOR FRONTIER COST-Or-PASS ESTIMATES
To test statistical significance of our results in Section 3.2 and Table I calculate 95% bootstrap percentile confidence intervals for each frontier cost-of -f-sssf-pass estimate in the table by simulating the sampling procedure from the models 10,000 times. By considering the midpoints of the confidence intervals as new estimates, we conduct the same analysis in the main paper. The analysis in Table [12 mostly mirror the original findings with minor difference that GPQA Diamond, o1-mini achieves
20

---

Published as a conference paper at ICLR 2026
Basic Quantitative
Knowledge Based
Complex Quantitative
Model Category
2-Digit Add.
GSM8K
BBQ
GPQA Dia.
MATH 500
AIME24
Lightweight Models
Llama-3.1-8B
4.8e-5
0.031
0.009
9.34
0.768
3.49
GPT-4o mini
5.4e-5
0.035
0.005
12.76
0.47
3.34
L.ama-3.3-708
1.6e-4
0.027
0.003
9.35
0.30
2.43
Large Models
Llama-3.1-405
6.9e-4
0.023
0.003
5,25
0.261
1.98
Claude Sonnet-3.5
0.002
0,034
0.004
7.08
0.585
3.34
GPT-4o
0.002
0.031
0.005
7.09
0.231
3.2
Reasoning Models
OpenAl ol-mini
0.002
0.035
0.01
6.19
0.137
1.19
OpenAl ol
0.002
0,061
0.032
4.24
0.313
1.29
DeepSeek-RI
0.002
0.033
0.009
7.36
0,062
0.836
OpenAl o3-mini
0.001
0.02
0.006
4.13
0.178
0.488
Table 11: Using the same experimental setup as in Table 1 except with the region set to India (for which we re-estimate the human-ews expertt baseline costs in Table [T0]. The results majorly mirror those in Table with slight shift toward our metric favoring lightweight models.
lower frontier cost-of-pass. than Llama-3. 1--0BB. Therefore, this indicates the statistical significance of our findings.
Basic Quantitative
Knowledge Based
Complex Quantitative
Model Category
2-Digit Add.
GSMBK
BBQ
GPQA Dia.
MATH 500
AIME24
Lightweight Models
Llama-3,1-88
(1. (1.8e-5, 2.0e-4) 4)
(0.192, 0.328)
(0.034, 0,042)
(23.11. 28 09)
(3.84, 4.59) (2.06,
(15.33, 16.67) (10.67, 12)
(5.3e-5, 5. 5.50-5)
(0.219, 0.274)
0.017)
(28.55, 32.63)
Lluma-3.3-70B
(1.8c 4, 1.6c 4)
(0.164, 0.1)
(7.4c 9.79 3)
(19.49, 22.2)
(1.41, L.78)
Large Models
Llama-3.
16.9e-1.1 7,00e-1
(0.138, 0.165)
(6.7e-3. 3.3e-3)
16.77)
L888)
10.67)
Claude Son
(2. 1e-1, 32.1 2.2e-3)
(0.195, 0.195)
16.3e-3.1 7.4e-3)
(14.96, 17.68)
(2.63, 3.1) (1.14,
(14.67, 15.34)
GPT-4o
(2.3e-3, 2.3e-3)
(0.169,
18.14)
Reasowing Models
OpenAl ol-mini
(5.30 5.5n-3)
(0.172, 0.2)
(6.013, 0.014)
(12.73, 15.44)
(0.497, 0.779)
(4.77. 6.11
OpenAl ol DeepSeck-R1
0.018)
(0.221, 0.223)
(0.042. 0.044)
(8.08, 9,85)
(0.885, 0.988)
(2.75, 3.98)
(1.8e-3. 1.8e-3)
(0.17, (0.17.0.2)
(0.014, (0.014.0.017)
(15.46, 17.73)
(0.205, 0.394)
(3:4, 4.74)
OpenAl 03-mini
(1. (1.1e-3. 1.1e-3)
(0.112, 0,167)
(0.011. 0.014)
(8.63, 10.89)
(0.756, 0.944)
(2.03, 2.7)
Table 12: The same experimental sctup as in Table 1 where we report 95% bootstrap percentile confidence
intervals (10,000 bootstrap samples). For cach column, the three entries with the lowest midpoint (i.e., best frontier cost-of-pass) highlighted in blue. The results closely mirror those in Table except that GPQA Diamond ol ol-mini achieves lower frontier cost-of-pass than Llama-3, 1-40BB; indicating statistically significant results.
C.8 BENCHMARK HACKING: CHEAP RANDOM GUESSERS ON FORGIVING MULTIPLE-CHOICE TASKS
Our framework is designed to reward strategies that convert monetary cost into correct outputs efficiently. However, sufficiently forgiving settings its canonical form has structural weak spot that extremely cheap yet essentially random agent can appear economically optimal, despite being useless (or even malicious) from an application perspective.
Consider a dataset composed of multiple-choice questions with & options each (e.g. A: == 4). We assume a model that has just enough capability to follow formatting instructions (c.g. always returning single option token such as "A", "B". "C", or "D"), but whose behavior the task is indistinguishable from uniform random guessing, which would lead to R (p) = k- Its cost-of-pass would then be
Ciraus (p) -Cmma
Mrradd p)
Rimas (p)
When we consider C.raa (p) to be significantly small (for example, for low -rw-price model consuming small number of tokens and emitting O(1) answer tokens), then u(Mrand, p) can be lower than the
21

---

Published as a conference paper at ICLR 2026
cost-of-past pass of more capable models that actually reason about the problem but incur higher token costs and/or higher prices per token.
Lowest 3 frontier cost-of-pass
Dataset
Adversarial baseline
3rd
2nd
1st
BBQ
6.7e-3 3
6.4e-3
6.2e-3
7.9e-3
GPQA-Diamond
10.43
8.18
8.07
2.1e-4
Table 13: Comparison of the three lowest frontier dollar cost-of-pass values per dataset from Table with an adversarial random-uuessing baseline (Llama-3.1 3.18BB configured mmma) on the multiple-choide chiicce benchmarks (BBQ, (ogA-Diamondj) The adversarial baseline attains artificially low cost-of pass, illustrating how a cheap random guesser can exploit forgiving multiple-choice evaluations.
To make the example concrete, we instantiate M/add directly with Llama-3. 1-8B's -B's setup. We configure the model to only output random answer in the required format by following the instructions the prompt in Section B We then repeat the same analysis as in Section 3.2 on the multiple-choiee tasks in our framework: BBQ and GPQA-Diamond Results in Table 13 show that this adversarial baseline attains a substantially lower frontier cost-of-pe pass than any other reported model, i.c., it appears to have made significantly more progress on top of the human-expert baseline than the genuinely competent systems. This concretely illustrates the benchmark-hackinn
concern formalized above.
To mitigate such benchmark hacking in practice we recommend (i) explicitly excluding impractical baselines such as random guessers unreliable systems from the set of strategies considered on the frontier, (ii) employing robustness-oriented dataset augmentation that makes random guessing unstable (for example by evaluating each question under several independently shuffled option orderings and requiring the model to produce consistent semantic answer across these variants) (ii) using stricter success criteria for a pass such as pass" k, which exponentially penalize low-accuracy models, and (iv) incorporating explicit penalties for failed attempts into the cost model. Some of these extensions are elaborated Section
D PRACTICAL IMPLICATIONS, LIMITATIONS, AND FUTURE DIRECTIONS
In this section, we acknowledge the limitations of our framework and evaluations, share practical perspectives together with directions for future extensions.
D.1 EXTENDING MODELING ACROSS COMPLEX AND DIVERSE DIMENSIONS
Our experiments consider a common but relatively simple cost and performance modeling, which may not seem clear for practitioners to adapt to their more complex settings. To start with, our analyses use per-token API prices that can be represented by Cm. (p) = w xm, (p) (Section 2.2). where w contains prices (input/output tokens) and x., (p) contains the corresponding quantities. In practical scenarios, one may include other components of the evaluation pipeline by placing their unil cost in and their per-atlempi quantity in to enrich the definition. Examples include: verification costs per attempt (c.g. human or automatic checks), costs associated with unsuccessful outputs, tool-usage fees, orchestration overhead (c.g. queue time, cold-start penalties, inter-service latency), and amortized fixed costs per attempt (training, hardware depreciation, maintenance, KV caching). Additionally, depending on the nature of the model inference, the costs associated with input and output token consumption can be adjusted (for example, to reflect cost reductions from batched inputs or outputs).
Regarding the success metric, one may replace accuracy with a stricter iitabilitt--rientee metric (e.g., pass"k k (Yao et al. 2024), requiring k consecutive successes) or more lenient metric (e.g., pass@k k (Chen et al. (021). rewarding any success within & attempts). Such alternatives are useful in settings where consistency, robustness, partial correctness matter. Similarly, for classification- based tasks where precision- recall trade-off's are central, one can treat the desired base criteria (e.g. enforcing minimum recall at acceptable false-positive rate) the definition of pass and compute cost-of-pase accordingly.
To clearly illustrate how such extensions could be incorporated. let's take canonical evaluation on Tau-Bench in Section C.2 and imagine its extension to a real-world customer support center that automates part of its customer support workflow with LMs. While our evaluation already
22

---

Published as a conference paper at ICLR 2026
incorporates multiple turns (by accounting for all consumed and generated tokens across rounds) and thus improves on single input/output evaluations, it can still be extended to better represent economic objectives. For instance, one could assign cost to each tool invocation, capturing both the direct cost of using the tool and a monetary representation of the time it takes to use it. Moreover, one might incorporate verification by qualified expert who reviews the interaction history and fixes problems, with the expert's 's labor cost included in the cost formulation. For failures where the expert either manually provides fix or fails identify/fix the issue, the cost model could be extended to include the associated losses (e.g. an extra charge for non- catasstttoppiic corrections, or larger losses due to catastrophic errors affecting the customer, such as misbooking an airline ticket). If such a system handles queries in batches and uses common system prompt, these effects can be incorporated by adjusting the per-token costs and amortizing the system prompt cost across the batch. Finally, other amortized costs from the underlying system (such maintenance of any infrastructure) can be incorporated into the cost modeling on a per-response basis. Regarding performance, one could apply the reliability-oriente pass' k metric to better and more reliably represent success in real-world deployments when benchmarking different systems or models.
For some applications, alternative units per attempt (FLOPs, time, latency, energy) may matter more than dollar cost, and the application of our framework may not be immediately visible. If an oracle system m' guarantees a non-zero success rate Rm D) with measurable expenditure, one may treat it as a baseline (analogous to the human expert baseline for costs) and apply our analyses, yielding an alternative unit dollars. If such an oracle does not exist, assuming that for each p D there exists some m E .M with Rm (p) >> O, several analyses in this paper (e.g. essentialness and impact) still apply. In such alternative units, the relative economic desirability of different techniques can also change. For instance, if latency is the main expenditure of interest, some inference time techniques like majority voting (Wang et al.) (223) can become very effective when the votes are executed in parallel, benefiting from improved expected correctness while incurring roughly the latency of a single vote.
D.2 INTERPRETABILITY OF OUR FRAMEWORK
Since our cost-of-pass metric is v(m,p) p) C.. for given problem instance p and model m, (p.(p)
with (p) w xm (p), improvements can arise from (i) lowering unit prices w, (ii) reducing resource use Xm (p), or (iii) increasing success probability R.,. (p). practice, cheaper tokens reduce w via pricing changes, distillation, quantization, or optimized serving: fewer tokens reduce x. (p) via prompt compaction, dynamic budgets, or instructions that promote concise generation; and higher accuracy increases Rm.(p) through better prompting, light test-time techniques, or improved model / training. Thus, the metric and framework capture these practical dimensions and quantify them in an interpretable way.
While these dimensions explain directional changes, the formulation of v(m, p) still reports estimates (for cost and performance) solely through expectations and therefore does not capture variance. Two strategies with identical expected cost-of-pass may entail very different variances, and hence different risks. Similarly, extending the frontier cost-of-past pass and gain (Equations (6) and (7)) to distributions (Equations (8) and (9) also operates purely in expectation. Augmenting the metric with measures of variance or risk-adjusted objectives, replacing expectation with alternatives such median or
worst-case values, could enhance interpretability and practical usefulness. We leave these extensions future work.
D.3 LIMITATIONS
We present limitations associated with both the framework and our evaluations. While covered in Section D our evaluations instantiate simple formulations for costs and success. This is reasonable proxy from user perspective and extends gracefully, but it still omits indirect and context-speeiffcc terms (like cvaluatioon/veiifiattio overheads, wait times, invocation retrics, tool-call charges etc.). Our framework remains compatible with these terms via the vectorized cost view, but we do not include them in our core results.
Regarding the success metric, framework assumes binary success/failure criterion, thus con- tinuous or composite notions of success are not modeled directly. This binary success modeling centers measurements the cost per unit of successful production. We believe this perspective
23

---

Published as a conference paper at ICLR 2026
is especially useful in real-world domains where each successful umit of production has reasonably similar relationship to the underlying economic objective. For example, in a support-center setting, a successful unit of production could be defined as whether a customer's query is resolved, with each resolved query contributing in a broadly similar way to the overall business goal. In scenarios that require more sensitive distinctions between different outcomes (for instance, when responses vary in quality or are evaluated along multiple dimensions), it may be beneficial to extend our framework to model cost per marginal unit of utility. The main idea would be to define a utility function over outcomes (potentially aggregating several quality dimensions into single score) and study how much per unit of utility costs for a typical attempt. This utility could be expressed in dollar terms or in another unit, but the key idea is to make it continuous rather than purely binary, which can better capture complex objectives. Many high-stakes and complex domains, such as healthcare and finance place asymmetric value on different error types (e.g. false positives vs. false negatives), and such preferences can be encoded directly into the utility function. We leave a rigorous study of such formulations and their applications to future work.
Both pricing and performance can vary across API providers (Gao et al. 2025). especially for open- source models hosted by third parties. Treating each mooider-mmdel (i.e., inference pipeline) pair as a distinct strategy and either (i) reporting all results from the same provider consistently or (ii) providing multiple provider snapshots per model can make benchmarking and comparisons more robust.
Throughout evaluations, we fix single concise instruction and sampling arguments (e.g.. temperature, top-p). We chose this to reduce degrees of freedom and enable comparability across models. However, results may be sensitive to these choices. Future work can study prompt and decoding sensitivity by evaluating small prompt ensembles per model and conducting sweeps over decoding settings.
Our empirical analysis focuses on widely used public benchmarks that serve as the community's current measures of progress. However, the evaluated models may have been partially exposed to these datasets (e.g. during pretraining). Therefore, these benchmarks are not a clean measure of outto--distribution generalization. As result, our findings are best interpreted as quantifying cost-effecttiveness on canonical and widely familiar problems, and they may provide an optimistic view of the attainable economic frontier. In high-stakes deployments, or in any setting where data contamination is a major concern, practitioners should explicitly account for this potential training-test overlap.
Model selection in our evaluations can introduce temporal and categorical bias. Due to budget, compactness, and coverage considerations; we evaluated a subset of releases. For this, we fixed a short time window and chose representative models per major family capture broad trends. A more exhaustive design is beyond our scope, but two extensions are natural: (i) broadening coverage to include historical and subsequent releases, and (ii) sampling more densely within fixed horizon (more models at closely spaced release dates).
Our family distinction between lightweight and large models is based on per-token prices. Alternative categorizations (parameter count, open/closed status, deployment modality) are possible. We prioritize transparency and reproducibility; as sizes are often undisclosed, and openness does not map directly to user-incurred costs. We also keep the analysis prototypical by focusing on user-facing, common-case models (omitting quantizations or distillations). Future work can adopt alternative categorizations to quantify economic impact under different groupings.
The human expert baseline assumes that qualified annotators always succeed given sufficient time and compensation. In our experiments, we instantiate this by matching the expert with the benchmark's label creators and implicitly treating them as near-perfect. Extremely challenging problems (or domains with scarce expertise) may violate this assumption, and more realistic estimates of the human cost-of-pass would require dedicated human-subject studies that jointly measure success rates and labor costs. Our evaluations are further restricted to two strategy types: standalone model pipelines and the human expert baseline, omitting hybrid human-Al workflows in which models and humans jointly solve tasks and can outperform either component alone (Haupt & Brynjolfsson 2025). Within our framework, such workflows can be represented as additional strategies (c.g. Mhybrid M), with the frontier cost-of-part of-pass computed over this expanded set, Accurately estimating
24

---

Published as a conference paper at ICLR 2026
the cost-of-pass for these strategies would again require controlled numann-ubbeed studies that capture both Al inference and human labor, which we leave to future work.
Finally, empirical conclusions about inference-time techniques drawn under canonical and common setup that risk-neutral, where we primarily optimize dollar cost per unit of success. In domains with extreme penalties for errors, where success is defined more stringently (for example, emphasizing reliability), our framework could instead favor strategies that employ heavier inference-time techniques to achieve very high reliability. Moreover, many such techniques can have non-trivial practical benefits (such as parallelizable majority voting (see Appendix [..]) once richer cost and success models are adopted. A more extensive, risk-aware empirical analysis of these techniques is an important direction for future work.
Despite these caveats, the framework's abstract, modular design means each of the above extensions can be implemented by plugging in refined cost functions, richer success metrics, or additional variability terms. At the same time, our core analysis remains practical baseline, as per-token API pricing reflects actual user-side costs, and binary pass/fail captures minimal utility in many applications. We hope future work adapts the framework along these lines and develops datasets that jointly stress cost and performance dimensions.
25


Reviews:
[{"review_id": "tVHjVhqT2V", "reviewer": "Reviewer_ucAV", "summary": "This paper proposes an economic framework for evaluating language models that jointly considers accuracy and inference cost. The authors introduce \"cost-of-pass” as the expected monetary cost to generate a correct solution and \"frontier cost-of-pass”. The framework is applied to analyse 11 language models across 6 benchmarks. Results indicate that: 1) different model families (lightweight, large, reasoning) are cost-effective for different task types, 2) frontier cost-of-pass has decreased exponentially, particularly for complex quantitative tasks and 3) model-level innovations drive progress more than inference-time techniques.", "questions": "1. Being a little disingenuous here: what would a naive baseline model that is one random selection call (~$0.00001 per attempt) score as a cost-of-pass? Does there need to be some weighting or other term added since the low cost term dominates as the cost tends to 0, especially in MCQ tasks?\n2. The paper assumes Rexpert(p) ≈ 1 (line 190), but provides no empirical validation that human experts actually achieve near-perfect accuracy on these benchmarks. Where does this assumption come from?\n3. If a model consistently fails on certain problem types, repeated attempts won't help, so won't 1/Rm(p) be an overestimate of required attempts? These aren’t independent tests.\n4. How were the 6 benchmarks selected? Why not include code generation, long-context, or creative tasks?\n5. Which specific versions of each model were used? (GPT-4o has had multiple versions)\n6. Have the authors considered median or worst-case cost-of-pass instead of expectation? This might be useful in many fields.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper has strong theoretical grounding in established production theory.\n2. It is clearly formulated and easy to follow, even for those working in parallel fields.\n3. It has practical and timely relevance as models become increasingly expensive and real-world decisions become more critical.\n4. Experiments are comprehensive.", "weaknesses": "1. There seems to be no discussion or handling of (KV or similar) caches. These can significantly reduce the cost of inference. Similarly, no discussion of batching opts that have large cost effects. These are critical limitations that need to be highlighted.\n2. The binary assumption does limit the practical application. There is no discussion or proposal on how this might apply when precision/recall trade-offs are important. Related to this, no discussion of how different domains might care about different error types, such as healthcare vs finance vs research. The authors partly acknowledge this but I think the paper would be much stronger if it was given more comprehensive consideration.\n3. Similarly, the quality of the answer can have a significant difference. If two models both give the right answer but only one of them gives detailed explanations with in-depth knowledge, that might be much more valuable than the other simple answer. Another limitation that isn’t discussed sufficiently.\n4. The cost-of-pass might be falling not because models are more efficient, but because the problems are getting stale and contaminated in training data. Models (including smaller/cheaper ones) can get “unnaturally” high scores on some benchmarks because they target them specifically. What is the risk that MATH-500 is increasingly used in training data for newer, smaller models, artificially increasing their score on them?\n5. It is not clear to me if the framework handles amortisation in real LM deployment sufficiently. In classical production making 10 products = 10x cost of 1 product (in the same factory). But LM deployment is more dynamic when it comes to problem solving, each one is unique but can leverage learnings from previous ones to improve performance, e.g. through prompt improvements. Can these be assumed to be approximately the same? This thought/question is partly inspired by Learning by Doing by Kenneth Arrow.\n6. Writing sometimes feels like patch-work. Abbreviations like “language models (LMs)” are defined three times. This only needs to be done once.\n7. No analysis of how sensitive the conclusions are to variations in human-expert cost estimates. Given the wide variability in expert compensation ($15-$100/hour for different expertise levels, per Appendix A), this might be significant.\n8. No confidence intervals or standard deviations reported for cost-of-pass estimates. Hard to judge variance and significance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yQEbRbkHjX", "reviewer": "Reviewer_YBga", "summary": "The paper introduces the cost-of-pass framework for evaluating the usefulness of language models in an economic light. They show that progress has been driven by lightweight and reasoning models for most domains. They also show that the impact of inference time techniques is nonexistent or minimal on the frontier cost of pass with the exception of cost-aware techniques like TALE-EP.", "questions": "A more multidimensional discussion of inference time improvements would be helpful. Many techniques, for instance, majority vote, allow parallel processing, which dramatically increases cost but reduces wall clock time. What are the upsides to these techniques, or should people not use them in practice?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "The paper engaged much more thoroughly with the economics than other literature. Particularly, evaluating human performance as part of the frontier cost of passing. I liked the attribution of progress to different model classes. It's interesting to see that reasoning models have led to progress adjusted for cost. I also thought the examination of inference time techniques was particularly interesting. Overall, a novel perspective, and I’d like to see more work like this engaging with economic constraints.", "weaknesses": "Ideally, the frontier cost of pass would include joint human+AI ability to solve problems. For instance, maybe a person with an AI chatbot can beat both another human and an AI chatbot. Human-AI collaboration ability is quite hard to measure experimentally, but some discussion of the literature in this area would be useful, ie, https://digitaleconomy.stanford.edu/wp-content/uploads/2025/06/CentaurEvaluations.pdf\n\nI would probably try to summarize the related work section in the introduction (this is mostly already done), so I’d probably remove this section. There is a lot of mathematics to justify quite intuitive results. I’d probably try to move some of the exact definitions to the appendix and try to incorporate the limitations and further work in the main body of the paper. I’d also try to make the limitation section stronger (see the questions below).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ELm50WQFnd", "reviewer": "Reviewer_GCEw", "summary": "This paper posits the cost of generating a correct solution to a problem should be a central metric for understanding AI progress and it's potential economic impact. This can differ substantially from measures like price per token because the number of tokens generated for a response can vary substantially between models. However, it also differs from price per query because the model requires a correct answer and allows for many samples given some probability that one is correct. This is similar to a pass@k measure. Using this model they estimate price-at-pass costs for several quantitative tasks and also gather data on the cost of a human performing those tasks. This measure shows different niches for differing model sizes and non-trivial tradeoffs between accuracy and price. Further some inference time methods are valuable under this measure, and some like majority voting are useless (since the pass@k type measure would simply be able to select the correct answer from the majority voting system). They track this value over time showing rapid exponential decrease in cost-of-pass, but the rate of cost decrease is very non-uniform across their task domains.", "questions": "- I would love to see more discussion and exploration on the potential costs of finding/verifying the correct answer. (I now see there is some of this in Appendix D, I'd rather have it addressed in the main paper and even more consideration given in the appendix). It is especially unclear to me that Tau-bench in C.2 is applicable here. Wrong computer execution or airline bookings can be disastrous if a wrong trace is produced.\n- It is very strange to me that your prices for MATH500 and AIME are so different. I believe these are extremely similar types of problems and skills involved and I would expect these to be performed by the same people and thus the tutoring wage should be the same.\n- I'm pretty sure your method makes any majority vote or similar method trivially useless. Thus I don't see why you have both 3 and 4 player voting and so much exposition on it. \n- I believe there is work by Epoch AI and Neil Thompson that seem relevant to this paper and the authors may be interested in looking into.\n\n-- Regarding price performance increases in LLMS\n\nBen Cottier, Ben Snodin, David Owen, and Tom Adamczewski. LLM inference prices have fallen rapidly but unequally across tasks, 2025. URL https://epoch.ai/data-insights/llm-inference-price-trends. Accessed: 2025-09-03.\n\nThe Price of Progress\nHans Gundlach, Jayson Lynch, Matthias Mertens, Neil Thompson \nhttps://openreview.net/pdf?id=JEsU87WUUb\n\n-- regarding inference time compute methods\n\nPablo Villalobos and David Atkinson. Trading Off Compute in Training and Inference, 2023. URL145\nhttps://epoch.ai/blog/trading-off-compute-in-training-and-inference. Ac-146\ncessed: 2025-09-02.\n\n\n-- regarding economic modeling of tasks and how to think about when they might be economically competitive with people\n\nBeyond AI Exposure:\nWhich Tasks are Cost-Effective to Automate with\nComputer Vision?\nMaja S. Svanberg, Wensu Li, Martin Fleming, Brian C. Goehring, Neil C. Thompson\n\nNBER WORKING PAPER SERIES\nEXPERTISE\nDavid Autor, Neil Thompson\n\nEconomic impacts of AI-augmented R&D\nTamay Besiroglu, Nicholas Emery-Xu, Neil Thompson", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper poses an interesting metric that seems useful in understanding model improvement in a practical way\n- Historical trends are analyzed over standard models and benchmarks, and human baseline costs are estimated", "weaknesses": "- The human price estimates do not seem very rigorously studied or calculated\n- cost-to-pass has a number of limitations which the authors do address fairly well in Appendix D. I would love to see more discussion of what assumptions are needed for this metric to be a good reflection of reality and what domains that might hold true in.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aK6btVAdrD", "reviewer": "Reviewer_bX6f", "summary": "The paper proposes Cost-of-Pass, a metric for evaluating the economic efficiency of language models (LMs). The metric is defined as the expected monetary cost of producing a correct solution. The authors also define a frontier cost-of-pass, representing the minimum cost across available LMs and a human-expert baseline, and track how this frontier evolves over time and across model families (lightweight, large, reasoning). They report that lightweight models dominate simple tasks, large models excel at knowledge-based problems, and reasoning models are most efficient for complex quantitative reasoning. The work claims to ground LM evaluation in production theory and provide a principled economic framework for assessing progress.", "questions": "1. How can you model different tasks that LLMs are already really good at and are already solving in the real world?\n2. How can your framework provide either machine learning insights or economics insights about future LLMs?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper tries to address an increasingly relevant topic: economic efficiency and cost-awareness in LLM evaluation. The framework provides an intuitive and interpretable scalar metric that combines cost and accuracy. The evaluation also Includes multiple model families and task types, offering a broad empirical view. The use of “human-expert cost” as a reference baseline is a reasonable and communicable idea.", "weaknesses": "1. Theoretical novelty: The paper's formalism rephrases classic production frontier concepts without introducing new theoretical insights. The \"cost-of-pass\" is simply a ratio of cost per correct answer and the \"frontier\" is just the minimum of these values over models. This is an intuitive but elementary restatement, not providing any novel theoretical insights into the LLM industry. As a result, the so-called “economic foundation” adds rhetorical flavor but little intellectual substance.\n2. Empirical metrics: The paper defines the cost purely as API pricing multiplied by token counts. However, these prices are commercial, provider-dependent, and change frequently by many economic factors. Thus, the analysis reflects vendor pricing strategies, not intrinsic model efficiency. The “economic” quantities are therefore non-fundamental and insightful. Also, “correctness” is defined as a binary pass/fail outcome per benchmark. This oversimplified notion of correctness ignores nuanced forms of reasoning quality, partial credit, or structured evaluation that are essential for emerging multimodal, interactive, or agentic tasks.\n3. Experimental results: The empirical results primarily plot “frontier cost” over time or remove model families to show counterfactual trends. These are descriptive and unsurprising that lightweight models are cheaper for simple math, etc. The human-expert baseline is estimated from tutoring websites and contest durations, which introduces large uncontrolled variance. Most importantly, these insights have been largely discussed in previous work in cost-aware inference and routers and so on. The paper's results don't provide significant new insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)