Credit-Budgeted ICPC-Style Coding: When Agents Must Pay for Every Decision
Authors: Lingfeng Zhou
Junhao Shi
Jin Gao
Dequan Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Coding Agents
Large Language Models
Agent Evaluation
Interactive Environment
Primary area: datasets and benchmarks
TL;DR: We build USACOArena, a competitive programming arena to evaluate coding agents' decision-making skills under resource constraints, revealing strategic profiles that go beyond simple code correctness.

Abstract: As autonomous agents and agent swarms become increasingly capable at complex coding tasks, the focus must shift from mere accuracy to real-world efficiency. Unconstrained agents often waste massive resources and time, incurring high opportunity costs. To be practical, agents must manage a generalized "credit" budget covering generated tokens, local tests, and elapsed time. To evaluate this resource awareness, we introduce USACOArena, an interactive ACM-ICPC style arena where every decision consumes credits from a fixed pool. This transforms a static coding benchmark into a dynamic resource management challenge. Our experiments reveal that frontier models—including the Codex framework and leading single agents—struggle to optimally balance these economic constraints. Through competitive profiling and self-play, we uncover distinct, path-dependent decision strategies. Ultimately, enforcing a strict credit economy is essential for developing highly efficient, cost-aware agent swarms.

Published as a conference paper at ICLR 2026
CREDIT-BUDGETED ICPC-STYLE CODING: AGENTS MUST PAY FOR EVERY DECISION
WHEN
Lingfeng Zhou's Junhao Shilt,
Jin Gao', Dequan Wang! 1.2t
Shanghai Jiao Tong University
Shanghai Innovation Institute
ABSTRACT
Current evaluations of autonomous coding agents assume an unrealistic, infinite- resource environment. However, real-world software engineering is a resource- bound competition. As scale toward large agent swarms, ignoring compute and time costs risks catastrophic budget exhaustion. To shift the focus from iso- lated accuracy to cost-aware problem-solving, we introduce USACOArena an interactive ACM-ICPC-style arena driven by a strict "credit" economy. Every generated token, local test, and elapsed second depletes fixed budget, forcing agents to make strategic trade-offs. Our comprehensive profiling reveals that fron- tier single agents and swarms currently fail to optimally balance accuracy with these constraints, exhibiting divergent, path-dependent behaviors. Ultimately, US- ACOArena provides an essential dynamic training ground for developing highly efficient, resource-aware agent architectures.
INTRODUCTION
Autonomous agents like Claude Code and Codex have become highly capable programmers, rou- tinely solving complex coding tasks. However, this rapid progress masks critical flaw: current evaluations focus almost exclusively on static coding accuracy, implicitly assuming an idealized, infinite-resource environment. In reality, software engineering is a resource-bound competition. As the field scales toward deploying large agent swarms, ignoring opportunity costs-such as API tokens, execution time, and local testing compute- will lead to catastrophic resource exhaustion (Figure [T). Therefore, evaluating true agentic intelligence requires fundamental paradigm shift: from isolated code correctness to competitive, cost-aware resource management.
will
To bridge this gap, agents require dynamic feedback signal that quantifies the real-world penalty of their actions. We argue for the necessity of a generalized "credit" system-a unified budget encom- passing every decision an agent makes. Under this framework, agents must pay for every generated token, every local compilation, and every passing second. This scarcity-driven competition forces agents make strategic, metacognitive trade-offs: they must constantly evaluate whether to expend their depleting resources exploring novel solution path or to halt and submit the current best at- tempt. Fostering this rigorous resource awareness in single agents today an essential prerequisite for ensuring that future agent swarms can collaborate efficiently without exhausting shared budgets.
Accurately measuring an agent's s ability balance these constraints requires a rigorous, frictionless testing environment. Real-world software engineering benchmarks (Jimenez et al. (023 Yang et al.) 2024 2025al JJain et al. [2025b; Zhou et al.) 2025; Badertdinov et al. (025)) are invaluable, but they inherently introduce confounding variables- such missing documentation, ambiguous require- ments, or complex repository To isolate and evaluate the resource-allocation capabilities of agents, turn to the ACM-ICPC competitive programming format. Competitive programming inherently models a self-contained game with perfect information, where the rules, objective metrics, and striet computational budgets are fully disclosed upfront.
that obscure the pure observation
decision-makin costs.
"qqual contribution 'Coreesponing author: dequanwang @ sjtu.cdu.ccn

---

Published as a conference paper at ICLR 2026
Ignorant of Cost
Compute Cost @
Budget Limit
Cost-Aware Autonomy
Deployed
Unified credit feedback
Time
Figure 1: From Temporal Vacuum to Cost-Aware Autonomy. Without economic feedback, tradi- tional swarms waste time and compute, inevitably crashing into real-world Budget Limits (top). USACOArena solves this by translating absolute time, tokens, and testing overhead into a Unified Credit Feedback signal (bottom-left funnel). By forcing swarms to "pay" for every decision, it in- duces efficient, cost-aware coordination for successful deployment (bottom right).
Based on these principles, we introduce USACOArena, an interactive coding arena driven by strict credit economy. In USACOArena, every action consumes credits from fixed, depleting pool. This framework translates a static coding test into a dynamic resource management survival challenge. compels agents utilize their remaining budget optimize the delicate balance between speed, cost, and accuracy.
continuous feedback signal, pushing them to
We conduct comprehensive experiments in USACOArena evaluate how current frontier models handle these economic and competitive constraints. Our empirical analysis yields three key insights. First, we benchmark variety of leading single-model agents and profile their decision-making pro- cesses, uncovering distinctly divergent strategies: for instance, -e5-po exhibits an aggres- sive, high-exploration approach, whereas GPT-S-Codex demonstrates highly conservative resource management profie. Second, we conduct self-play experiments to observe emergent behaviors. We find that identical agents, when operating under the same environmental pressures, discover highly diverse and path-dependent routes to solve the identical problem. Finally, by evaluating contempo- rary agent frameworks and early swarm implementations, we discover that currently, neither single agents nor agent swarms optimally balance coding accuracy with strict credit limits. These sults underscore a fundamental gap in current agent architectures, demonstrating that both single agents and collaborative frameworks currently fail optimally manage resources.
In summary, our contributions are as follows:
1. A Shift Credit Feedback: We identify the lack of resource awareness as a major bottle- neck for the real-world deployment of agents and swarms. We shift the evaluation focus from simple code correctness dynamic, cost-effective problem-solving under competi- tive constraints.
2. The USACOArena Arena: strict credit budget. It rigorously enforces constraints by translating API tokens, local test limits, and elapsed time into a unified, quantifiable feedback metric.
We present an interactive, ICPC-style environment driven by
3. Strategic Profiling of Frontier Models: We provide a comprehensive empirical analysis of how leading single agents (e.g., Gemini-2.5-pa GPT-5-Codex handle the trade-offs between speed, cost, and correctness, highlighting their disparate strategies and current limitations.
4. Pathways for Future Swarms: Through self-play and multi- experiments, we demon- strate that strict credit economy elicits complex, path-dependent behaviors. US- ACOArena serves as a vital dynamic training ground to build highly efficient and resource- aware agent swarms in the future.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Our research is positioned at the intersection of automated code generation and the rapid emergence of autonomous agent swarms. We introduce a novel evaluation paradigm that bridges critical gap: shifting the focus from static code correctness to cost-aware autonomy and strategic resource management under strict economic feedback.
2.1 LLMS FOR CODE AND THE ILLUSION OF STATIC BENCHMARKS
Recent breakthroughs in I Large Language Models (LLMs) have significantly advanced automated
code generation OpenAA| 2025b; Anthropic 2025a by Guo et al.) 2024 Wei et al. 2024 DeepSeek-
AT ct al., 2024 Cummins et al. 2024 Liu et al. 2024). To assess these capabilities. bench-
marks evolved from fundamental functional correctness (Chen et al.] 2021 Austin et al. (021) to algorithmic competitions (Li al., 2022; Shi al.) (024) and real-world software engineering
tasks (Jimenez et al. 2023 Li et al. 2025] Liu et al. 2025). Recent efforts further enhanced evalua- tion rigor with live problem sets (Jain et al. 2024 Zheng al. 2025). robust ranking systems (Quan
et al. 2025 Yang et al. 2025b), multiple function calls (Zhuo et al. 2025). language- driggne---iennn-e cod- ing Deng al. 2025), and interactive debugging (Yuan al. 2025). However, these benchmarks
share a fundamental limitation: they evaluate models in a temporal vacuum. By focusing exclu- sively on the static accuracy of the final code output, they fail to measure the economic realities of software engineering, leaving factors like compute cost, temporal efficiency, and strategic trade-offs completely unevaluated.
2.2
FROM INDIVIDUAL AGENTS TO CODING SWARMS
To move beyond simple code generation and drastically compress delivery time, the community is rapidly pivoting toward coordinated multi-agent swarms. Sophisticated frameworks feature diverse architectures, including coder-tester co-evolution (Wang et al. 2025b), generator-verifer pairs (Jain) et al. 2025a), policy-critic models (Xic et al. 2025), and retriever-generic
pipelines (Wang al,, 2025a), alongside internal reasoning mechanisms such as feedback loops and explanation -rivve repair (Jiang al.] [2025] Gehring al. 2025). Additionally, multi-agent systems built on platforms like AutoGPT (Significant Gravitas), MetaGPT (Hong et al. [2024), Verse (Chen et al. 2024b) and OpenHands (Wang al. [0244 Ma et al.
2024 Gao et 20.4 Chen et al. 2024a Yang al.]
2024 Xia et al. 2024]] Aggarwal et al. (2025) employ complex workflows to parallelize intelligence. Despite this architectural complexity, swarm effectiveness still judged using static benchmarks.
This creates a severe disconnect: swarm might successfully solve a task but waste massive amounts of compute or succumb to heavy coordination overhead. The true strategic competence of these swarms remains largely unquantified.
2.3 THE MISSING PARADIGM: ECONOMIC FEEDBACK AND COST-AWARE AUTONOMY
In Reinforcement Learning, learning from experience in interactive environments is crucial for achieving superhuman performance. Works like SWE-rebench (Badertdinov et al. 2025 and R2B- Gym (Jain et al., 2025b) provide executable training grounds, while evaluation frameworks like ColBench (Zhou et al. (025) and TheAgentCompany (Xu et al. 2025) measure multi-step task per- formance. However, they primarily serve as execution sandboxes that ultimately reward only the final artifact.
In contrast, USACOArena is explicitly designed to instill procedural wisdom through direct eco- nomic feedback. By theoretically framing the arena as Budget-Constrained Partially Observable Markov Decision Process (POMDP), we force swarms to pay for every decision, inference token, and elapsed second. While existing benchmarks measure Crystallized Intelligence (applying known patterns), USACOArena rigorously tests Fluid Intelligence. This isolates swarm's 's ability to opti- mize algorithmic reasoning, manage risk aversion, and mitigate coordination tax under strict uncer- tainty to achieve true cost-aware autonomy.
3

---

Published as a conference paper at ICLR 2026
Fast Time & High Compute
Agents
BUDGET LIMIT
Final Score
Slow Time Low Compute
NV
4
Buy Hint
Figure 2: The Unified Credit Economy of USACOArena. Our environment evaluates agents on cost- aware decision-making rather than static correctness. The agents face distinct strategic trade-offs: aggressively parallelizing to minimize absolute delivery time at the expense of high inference costs (top), conserving compute while burning valuable time credit (middle) or spending resources to acquire strategic hints to overcome bottlenecks (bottom). Every action-along with the continuous passage wall-clock time drains a shared Budget Limit. To maximize the Final Score, the swarm must optimally balance speed, compute, and reliability without going bankrupt.
USACOARENA: AN ICPC-INSPIRED ARENA FOR CODING AGENTS
This section details the design USACOArena. is an interactive environment built evaluate the strategic choices of coding agent swarms. We move beyond simple correctness checks. We aim evaluate how well agents manage delivery time and compute costs. First, we explain our ACM- ICPC-style design and the USACO problem set (Section 3.1). Next, we define our unified credit model (Section 3.2). This model unifies delivery time, inference tokens, and testing overhead into single credit economy. Finally, we explain our system architecture and communication protocol (Section 3.3
3.1 FOUNDATIONAL DESIGN: ADAPTING THE ACM-ICPC FORMAT
We model USACOArena the ACM-ICPC format. This format matches real-world software gineering well. The ACM-ICPC rules use an all-or-nothing scoring system. This rewards fast and bug-free code generation. This differs from the IOI format, which focuses theoretical limits. prooiidde more details.
We make main changes for agent swarms. First, official ACM-ICPC problems too hard for current models. We use the USACO problem corpus instead. Its difficulty ranges from Bronze to Platinum. This allows us test different agent capabilities clearly. Each match uses 12 problems to simulate an ACM-ICPC World Final.
Mitigating Data Contamination. approach. USACO releases four new contests every year. We update our dataset with the newest season. This tests the agents new, unseen problems.
To prevent train-test overlap, use "Living Benchmark"
Second, we add absolute delivery time into a unified credit economy. In the real world, fast delivery important correct code. We give each agent fixed credit budget. Actions and the passing of time both cost credits. This forces agents to balance speed, compute cost, and reliability. To join the main test, agent must first solve simple Bronze problem (Section 4.1
4

---

Published as a conference paper at ICLR 2026
The agent's policy outputs tuple: (Score, Consumed Credit). The agent maximize its score without exceeding the credit limit:
max (core( nT)
subject Caction (T) + Cime Σ C limit
n
3.2 SCORING AND CREDIT MODEL: OPERATIONALIZINE ACM-ICPC RULES
To implement these ideas, USACOArena uses explicit scoring and credit system. This system mirrors the real-world economic pressures of software engineering.
Ranking and Scoring. As shown in Figure we adapt the ACM-ICPC rules. Total score is the primary ranking metric. Consumed credit is the tie-breaker. The score is a weighted sum of all accepted (AC) problems. Harder problems give more points. We give no partial credit for failing any test case.
The Unified Credit Model. To implement these ideas, USACOArena uses an explicit scoring and credit system. This system mirrors the real-world economic pressures of software engineering.
Ranking and Scoring. As shown in Figure we adapt the ACM-ICPC rules. Total score the primary ranking metric. Consumed credit is the tie-breaker. The score is a weighted sum of all accepted (AC) problems. Harder problems give more points. We give no partial credit for failing any test case.
The Unified Credit Model. The credit system is unified economy. It includes action costs, delivery time costs, and penalties.
Action Costs are the resources spent solve problems.
LLM Inference Cost: This is the compute cost. We normalize it by the API price (Ap- pendix H.. Expensive models use more credit. Agents must balance thinking quality and cost,
Hint and Test Cost: These represent the effort for reading documents or local debugging (Appendix K.
Delivery Time Cost is our key addition evaluate speed. We track the absolute wall-clock time T from the start to the end of the agent's run. We multiply T by an adjustable time coefficient a. This converts absolute delivery time into credit. By changing o, scenarios. A high a values fast delivery, while low values compute efficiency.
test different competition
Penalty Costs happen when agent submits wrong code (e.g., Wrong Answer). This stops agents from guessing randomly, The termination rule is important. An agent stops only when its action and time costs exceed the budget (Caction + aT < Cima). However, the final tie-breaker uses the total consumed credit. This includes penalties (C (Cconsumed Caction + aT + Openaty). This forces agents to deliver correct code quickly and cheaply.
3.3 SYSTEM ARCHITECTURE AND COMMUNICATION PROTOCOL
We build our system using a turn-based loop based on the Model Context Protocol (MCP) (An- thropic 2024). Every turn, the USACOArena server sends the game state to the agent as a ISON object. This includes credit, score, and the leaderboard. The agent replics with action, like SUBMIT SUBMIT_SOLUTION SOLUTION. This protocol ensures all agents see the same data and use the same actions. We run all submitted code in secure online judge sandbox. This makes platform safe and easy to reproduce.
We choose MCP solve the reproducibility crisis in agent research. MCP is independent of pro- gramming languages. Researchers can test any agent easily. also supports multi-tun ttrtttum chats. This helps build future multi-agent ACM-ICPC teams easily.

---

Published as a conference paper at ICLR 2026
Average Score 25
15.0
10'
13.6
Comsumed Credit
12.0
10"
4,8
2,0
1,6
1,0
0.4
10
Gemini-2.5pp 2-po
GFT-3-Codes
GLM-455
Owent-230 2355h
Kimi-K2
Claude Clande-sseenet
Gemini-2.5-pm
GPT-S-Cldee
GLM-4.5
Qven
Kimi-Ka
(a) USACO 2024 December
(b) USACO 2025 Jamuary
20
Average Score
Awcoge (ftr
Crevanee
10"
18
16
14.6
Consumed Credit
12.0
12
10
10
10
3,2
1.0
0,5
0.5
0.5
10
2.0
10
Gemini-2. Gemini-255pp 2.ppr
GPdes-Gades
GLM-4.5
Owens--235 b
Claude 4-smmet
DeepSeck-111
DeepSenk.r3
KimixK2
Gomini-2.5 pro
Codes
Qweat-2350
GLM-4.5
(c) USACO February
(d) USACO Open
Figure 3: Average agent scores and consumed credit across the four contests of the 2024- 2025 USACO season. Each subplot shows the results for single contest, with agents sorted by rank. Blue bars represent the average score (left axis), while the orange line indicates the average consumed credit (right axis, log scale). Error bars and the shaded area denote the standard deviation over five independent runs; for clarity, only agents that achieved non-zero average The results reveal a stable and significant performance hierarchy: across all four contests varying difficulty, Gemini-2.5-ppm and GPT-5-Codex consistently rank first and second, respectively.
shown.
4 EXPERIMENTS
Our evaluation demonstrates the utility of USACOArena. We first detail the experimental setup (Section[4.] Next, we present the main competition results across diverse LLMs (Section 4.2p. We explore emergent behaviors in self-play (Section Furthermore, we conduct systematic ablations of our benchmark (Section[4.4) Finally, we isolate the impact of agent swarm and evaluate absolute delivery time using Codex (Section 4.5
4.1 EXPERIMENTAL SETUP
All experiments follow the ICPC-inspired competition rules detailed in Section
Problem Corpus and Qualification. The evaluation uses 48 problems from the 2024-202 2025 US- ACO season. To ensure baseline competency, an agent must successfully solve the easiest Bronze- level problem from a contest to qualify. Detailed difficulty baselining and qualification results are provided in Appendix and
Agent Construction. Agents are constructed by pairing an LLM with non-prescriptive prompt outlining the objective rules, ensuring strategies are emergent rather than hard-coded (Appendix J.
4.2 CROSS-MODEL BENCHMARKING: STRATEGIC PROFILES UNDER STRICT COMPUTE BUDGETS
While Section 4.5 demonstrates the critical role of absolute delivery time in agents, evaluating di- verse commercial APIs introduces unfair network latency. Therefore, to isolate and analyze the pure strategic reasoning capabilities of different foundation models, temporarily set 0 and focus entirely on compute efficiency. To ensure the robustness and generalizability. we evaluate agents across the four distinct contests of the 2024-2025 USACO season. Each contest is five times,

---

Published as a conference paper at ICLR 2026
and the results presented in Figure are the average of those runs. For this experiment, we select GPT-5-Codex as the representative for the GPT-5 series, as it is specifically optimized for agentic coding tasks (OpenAl 2025a).
A Stable Two-Tier Hierarchy. The results reveal consistent two-tier performance hierarchy across all four contests. Gemini-2.5-pp and GPT-5-Codex invariably secure the first and second ranks, respectively, demonstrating a significant and reliable capability gap between these top-tier agents and the rest of the field, whose performance is far more volatile. For clarity, the figure only displays agents that achieve a non-zero average score in each contest.
In-Depth Analysis of Top-Tier Agents. While the main competition results establish a clear per- formance hierarchy, they do not fully explain why one agent consistently outperforms another. The final scores are merely the outcome of a complex, dynamic pecision-makingg process. To understand the underlying strategic differences that lead victory, this section provides an in-depth analysis of the two top-performing agents: Gemini-2.5-pro and GPT-5-Codex.
Capability Strategy Gap and Human Baselines. A critical observation is that the benchmark is far from saturated. The theoretical maximum score per contest is 54 points (weighted across Bronze to Platinum levels). Currently, top-tier agents score around 15 points. To contextualize this against human baselines, we map agent scores to the official USACO division structure:
Novice (Score < 3, Bronze): Struggling to clear entry-level algorithmic tasks.
Intermediate (Score 3-9, Silver): Capable of solving foundational problems.
Current state-of-the-ar agents (Score ~15) fall firmly into this tier. They are comparable to talented high school competitors who have cleared intermediate stages but struggle with complex algorithmic reasoning.
Advanced (Score 9-24, Gold):
Expert (Score > 24, Platinum): The world-finals frontier that current models have yet to consistently breach.
Remarkably, GPT-5-Codex has occasionally achieved peak scores of up 29 in isolated runs, prov- ing the intrinsic capability to solve Platinum problems exists. The bottleneck keeping its average score much lower is purely strategic-a failure to optimally allocate budget and manage risk.
This analysis resolves the performance paradox: Gemini-2.5-prr wins by being a more effective competitor, not necessarily superior problem-solver. This distinction is best understood through the iitoraation-xpliiaatiio trade-off. Gemini-2.5-pro s strategy is one of aggressive exploration; it attempts many problems to maximize broad coverage and its cumulative score, accepting lower precision rate as necessary cost. In contrast, GPT-5-Codee pure exploitation. Its risk-averse approach limits its attempts to only high-confidence problems, causing it to miss many scoring opportunities and turning its precision into strategic liability.
cautious perfectionism is a form of
This distinction highlights key finding of our work. Gemini-2.55pro's success demonstrates that in competitive setting, strategy of broad exploration can outperform more capable but overly conservative exploitation strategy. This implics that optimal performance in a complex, resource- constrained environment like USACOArena requires more than just raw problem-solving accuracy. For the field to advance, this suggests that developing an agent's decision-making framework its ability to assess risk and manage resources is important enhaneing its core capabilities. True expertise in this domain lies in an agent's ' ability to dynamically balance the trade-off between exploring all viable opportunities and exploiting the most promising ones.
Impact of Contest Difficulty. The varying difficulty of the contests highlights the extent of this performance gap. While more agents achieve non-zero scores in accessible contests, the most chal- lenging one-the USACO 2025 US Open-showcases dominant performance by Gemini-2.5-pro. Its score of 14.6 establishes vast lead over GPT-5-Codex (3.0), suggesting that high-difficulty problems strongly accentuate the top agent's strengths. A detailed breakdown is in Appendix F
Limitations in Agent Self-Assessment. However, a deeper analysis of agent behavior reveals a widespread deficiency in strategic self-assessment. Lower-ranked agents often mismanage resources

---

Published as a conference paper at ICLR 2026
A deeper look at the data reveals a fascinating para- dox. As summarized in Table [ GPT-5-Codex demonstrates significantly higher peak capabil- ity, achieving a maximum score of 29 compared to Gemini-2.5-pro's 19. This confirms its potential to solve more difficult, higher-value problems. How- ever, it is Gemini-2.5-pm that achieves better aver- age rank and win rate more than double that of its competitor (68.4% vs. 31.6%). This performance inversion points directly to the decisive role of com- petitive strategy.
Table 1: Comparison of Agent Profile Metries for Gemini-2.5 20minn-2.55-pro and GPT-5-Codex.
Agent
Asg. Rank
Win Rate Max Score
Min Score
Gemini-2.5.pp
1.3 0.47
70.0%
GPTes-Codex
4: 0.47
30.0%
Submission Counts
Credit Consumption
800.0
The strategic profiles in Figure explain this out- come, revealing a clear difference. Gemini-2.5- pro's profile is characterized by an aggressive, high- volume strategy. Its plot extends outward on the At- tempted Problems and Submission Counts axes, indicating it attempts more problems to maximize scoring opportunities. This "breadth-first" approach treats credit as resource to be actively spent in ex- change for broader coverage.
Attempted
Submission
Problems
Precision
Ganini S-pro
GPT-5-Coddx
Problems Solve Rate
First-Submit Accuracy
Figure 4: Strategic Profiles of Top-Tier Agents. Submission Precision is the percent- age of AC submissions out of all submission attempts; Problems Solve Rate is the percent- age of AC problems out of all attempted prob- lems; and First-Submit Accuracy is the per- centage of problems solved on the first at- tempt out of all successfully solved problems.
In stark contrast, GPT-5-Codes adopts a conserva- tive, "perfectionist" strategy. Its profile is heav- ily skewed toward near-perfect First-Submit Ac- curacy and Problems Solve Rate. This risk-averse approach prioritizes precision, but severely limits problem coverage, causing it to forego attempts on many potentially solvable problems.
by attempting problems far beyond their capabilities, forgoing points on easier tasks. Paradoxi- cally, the top-performing agent exhibits the opposite flaw: despite being capable of solving high- value Platinum problems (see Appendix D). Gemini-2.5-pre consistently defaults to safer, lower- scoring problems. This suggests that even the most advanced agents currently lack sophisticated risk-assessment and strategic planning.
Further ablation studies within the GPT-5 family (detailed in Appendix G confirm that agentic frameworks significantly improve performance precision, albeit at higher resource consumption.
4.3 EMERGENT BEHAVIORAL DIVERSITY IN SELF-PLAN
To investigate whether top agent's performance deterministic, we conduct a series of self-play experiments, pitting identical instances of gemi ni-2 5-pro against each other. The results reveal a striking diversity of behaviors. As shown in Figure []). the outcomes across 18 competi- tors are highly variable. rarely ending tie, and showing no simple correlation between credit consumed and final score. A trajectory analysis of a single match (Figure ((b) provides a granular explanation for this variance. It shows how different, path-dependent strategic choices such as methodical, bottom-up approach versus getting stuck on difficult problems can lead to decisive win-loss outcome.
These findings yield key insight: USACOArena is not simple puzzle but complex and sensi- tive environment that can reveal critical inconsistencies in an agent's decision-making. This demon- strated behavioral diversity, where single policy produce wide range of outcomes, is crucial prerequisite for improvement through learning- -aase methods. Therefore, our self- experiments validate USACOArena not only robust evaluation testbed but also suggest its potential dy- namic training ground for future research into cultivating more capable agents.
A granular trajectory analysis (Figure 5(b)) reveals the underlying mechanics of this variance, which we term the First-Move Effect. Divergence is highly correlated with the outcome of the agent's 's first attempted problem. If the agent stumbles upon solvable problem early, it enters Success Loop,
8

---

Published as a conference paper at ICLR 2026
200
Compeiton
Wimss
Points
1.0
17.3
Competion
30
150
Sotto hoobers
Gald gaosoes
Conumed Credit
6.8-
Loset
12 Points
H55
12.3
6.6-
Soots
6.4
Quudds adre? Boxane
Trapedd by the
62-
Comanned Cralii
6.
pocpantd extac betnoes
Coppetton
Coopectiv
Conpeetier
Competttor
Cooputtton
Conpectite
Compettion
Compentic
Coopeettie
Conpeetter
Conpritte
Compaste
Stap
(b)
Figure 5: Emergent behavioral diversity and strategic divergence in self-play. (a) Final scores and credit consumed across nine competitions between identical gemini-2.5-p agents, re- vealing wide spectrum of outcomes with no trivial correlation between and performance. (b) A trajectory analysis of single match provides a granular explanation, showing how different strategic paths lead decisive win-loss result. This demonstrated diversity, resulting from com- plex path-dependent decisions, validates USACOArena's suitability as rich training environment.
gaining the "confidence" (score padding) to take calculated risks harder problems. Conversely, an identical agent that randomly selects a mathematically intractable problem first often enters a Panic Loop: it fails, rapidly depletes its budget, and adopts an overly conservative fallback strategy, leading to performance collapse. This rich, path-dependent divergence validates USACOArena as powerful environment with sufficient reward gradients for future Reinforcement Learning (RI.) based agent optimization.
4.4 SYSTEMATIC ABLATIONS AND PROMPT SENSITIVITY
To validate that USACOArena measures intrinsic strategic intelligence rather than artifacts of arbi- trary hyperparameter tuning, conducted an extensive ablation campaign across 8 models and environmental dimensions on the 2025 February Contest (detailed results in Appendix B
Capability Saturation vs. Constraint Effect. By systematically varying the Credit Limit, we ob- served a strict Constraint Effect: reducing the budget to 10M significantly degrades the top model's (Gemini-2.5-pro) performance (dropping from 13.2 8.3). However, doubling the budget to 40M yields no improvement (remaining at 13300) This confirms that current tate-of-thhe-a art agents are Capability-Lmiited not Budget-Limited. They hit a "reasoming wall" long before they hit the credit limit, proving the hierarchy is robust. Furthermore, weaker models exhibit a clear floor effect, scoring near 1.0 regardless of resource abundance.
Prompt Sensitivity and Intrinsic Alignment. We additionally tested whether simple prompt en- gineering could fix the strategic deficits of models like Gemini-2.5-pro. We evaluated four prompt variations, ranging from enforced Chain-of-Thought (P1.1) to explicit "Aggressive/Opportuiiss instructions (P2. 1). Even the most aggressive prompt yielded only marginal score improvements (13.2 to 14.7), while overly verbose Few-Shot prompts actually degraded performance by consum- ing excessive token credits without improving decisions. This confirms that the observed "Conser- vative" versus "Aggressive" profiles are intrinsic the models' alignment and architectural priors, not merely a product of prompt formulation.
4.5 PROFILING AGENT SWARM: THE ECONOMIC TRADE-OFFS OF COST-AWARE AUTONOMY
Beyond single-agent performance, we investigate the ultimate resource management challenge: scal- ing to agent swarms, where parallel execution drastically accelerates both problem-solving speed and budget depletion. To isolate the impact of multi-agent coordination strategies from underlying model variations, we use swarms of Codex agents to evaluate their efficiency under strict credit constraints. We configured the swarms with three distinct strategic profiles: (1) Speedy Spendthrift:

---

Published as a conference paper at ICLR 2026
Time Compute
Expenditure Profiling
Profile
Coding Tokens
Toni Token Eapendrar (Millano)
0
Speedy Spendhiiitt (Secee: 3)
Thinking Tokens
Similegia
Frigai Perfectionist (Scor: 8)
Tokets (Millumm))
Communication Overhead
15
Delivery Time (Minits)
Speecitinn
Storeght
Pertentoni
(b)
Figure 6: Performance and resource profiling of Codex agent swarms. (a) Absolute Time vs. Cost: Illustrates the trade-off between delivery time, token expenditure, and final score. The Speedy Spendthrift minimizes time but suffers catastrophic performance drop (Score: 3), whereas the Frugal Perfectionist achieves high score (Score: at the cost of impractical delays ( 300 mins). (b) Token Breakdown: Reveals that blindly maximizing parallel workers (Speedy) triggers mas- sive explosion in Communication Overhead (orange hatched area). This rapidly depletes the budget without improving coding accuracy, empirically highlighting the critical need for dynamic, cost- aware resource management in future multi-agent systems.
Maximizes parallel workers to minimize absolute delivery time, disregarding token costs. (2) Frugal Perfectionist: Uses cautious, sequential approach to minimize token expenditure. (3) Cost-Aware Strategist: Dynamically balances parallel exploration and sequential execution based on the remain- ing budget.
Figure ((a) illustrates the fundamental trade-off between absolute delivery time and token expendi- ture. The Speedy Spendthrift effectively compresses delivery time but suffers from extreme token inflation. Conversely, the Frugal Perfectionist strictly conserves tokens at the cost of time delays.
Figure b(b) reveals how resource mismanagement dictates final match performance under strict credit limits. Despite its speed, the Speedy Spendthrift achieves poor parallelization triggers a massive communication tax, prematurely draining its budget on inter-agent coordination rather than effective problem-solving. By dynamically interpreting economic feed- back, the Coxt-Aware Strategis! mitigates this coordination overhead and efficiently allocates its budget. It successfully achieves an optimal winning score in fraction of the time required by the sequential approach.
of 3. Its aggressive
5 CONCLUSION
In this work, introduce USACOArena, interactive evaluation framework that translates abso- lute delivery time, inference tokens, and testing overhead into a unified credit economy. By forcing agents to pay for every decision, our experiments reveal that satteoff-he-ar performance relies less on raw coding capability and more on path-dependent strategies that carefully balance aggressive exploration with conservative precision.
However, our evaluation reveals critical limitations in current AI systems. Even top-tier models exhibit profound metacognitive deficits, frequently misallocating resources on intractable problems and ignoring strategic hints when trapped in failure loops. Furthermore, our ablation studies in- dicate that state-of-the-an agents are fundamentally capability-limiin limited rather than budget-limited; even with abundant credits, they hit a "reasoning wall" and fail to translate surplus resources into higher scores. They also demonstrate a widespread lack strategic self-assessment, oscillating be- tween reckless gambles on advanced tasks and overly conservative choices despite being capable of more. Moving forward, the rich, non-deterministic behavioral divergence observed in our self-play experiments validates USACOArena more than static benchmark. It serves dynamic Re- inforcement Learning training ground, providing the essential economic reward gradients needed to cultivate self-aware cost-efficient agents.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research is supported by the Key R&D Program of Shandong Province, China (224CXGCC102133, We express our gratitude the funding agency for their support. We thank all the anonymous reviewers for their valuable suggestions.
ETHICS STATEMENT
Our problem dataset is exclusively sourced from the publicly accessible USA Computing Olympiad (USACO) archives, which are distributed for educational and training purposes. Our usage falls strictly under non-commercial academic research, consistent with standard fair-use practices in the machine learning community. We do not redistribute the data for profit. The source code for the USACOArena environment, agent wrappers, and decision logs will be open-sourced ensure full reproducibility.
REPRODUCIBILITY STATEMENT
We are committed ensuring the reproducibility of our work. The core methodology our envi- ronment and the complete experimental setup are detailed in Section 3 and Section respectively. The appendix provides exhaustive details necessary for reproduction, including: the full problem corpus from the 2024-2025 USACO season (Appendix D): comprehensive qualification and main competition results (Appendix E and F a list of all large language models used (Appendix HH): all competition hyperparameters (Appendix DD; and the exact system prompt provided to the agents (Appendix D.. The complete source code for the USACOArena environment, agent wrappers, and evaluation scripts will be made publicly available in open-source repository upon publication.
REFERENCES
Vaibhav Aggarwal, Ojasv Kamal, Abhinav Japesh, Zhijing Jin, and Bernhard Schölkopf. Dars: Dynamic action re-sampling to enhance coding agent performance by adaptive tree traversal. arXiv preprint arXiv:2503. 14269, 2025.
Anthropic.
Introducing the model context protocol. 2024. URL https::/www.attthrogg
Accessed: 2025-09-21.
Anthropic. Introducing Claude 4. 2025a.
05
Anthropic. Claude Opus httppp//www..athhhpippc///////// 08 2025b.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXhv:2108.07733: 07733, 2021.
Ibragim Badertdinov, Alexander Golubev, Maksim Nekrashevich, Anton Shevtsov, Simon Karasik, Andrei Andriushchenko, Maria Trofimova, Daria Litvintseva, and Boris Yangel. Swe-rebench: An automated pipeline for task collection and decontaminated evaluation of software engineering agents, 2025. URL
250.. 20411
Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong, Artem Aliev, Jie Wang, Xiao Cheng, Guangtai Liang, Yuchi Ma, Pan Bian, Tao Xie. and Qianxiang Wang. Coder: Issue resolving with multi-agent and task graphs, 2024a.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint 1000:22077.03374 2021.

---

Published as a conference paper at ICLR 2026
Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang. Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian. Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelfth International Conference on Learning Representations, 2024b. URL htttps://openrevve
net wtw.
Chris Cummins, Volker Seeker, Dejan Grubisic, Baptiste Roziere, Jonas Gehring, Gabriel Synnaeve, and Hugh Leather. Meta large language model compiler: Foundation models of compiler opti- mization, 2024.
org/
DeepSeek-AI, Qihao Zhu, Daya Guo, Zhihong Shao. Dejian Yang, Peiyi Wang, Runxin Xu, Y. Wu, Yukun Li, Huazuo Gao, Shirong Ma, Wangding Zeng, Xiao Bi, Zihui Gu, Hanwei Xu, Damai Dai, Kai Dong, Liyue Zhang. Yishi Piao, Zhibin Gou, Zhenda Xie, Zhewen Hao. Bingxuan Wang, Junxiao Song, Deli Chen, Xin Xie, Kang Guan, Yuxiang You, Aixin Liu, Qiushi Du, Wenjun Gao, Xuan Lu, Qinyu Chen, Yaohui Wang, Chengqi Deng, Jiashi Li, Chenggang Zhao, Chong Ruan, Fuli Luo, and Wenfeng Liang. Deepseck-coder--V:: Breaking the barrier of closed-source models in code intelligence, 2024. httpssss:://axxi org/ abs /206 11931
Le Deng, Zhonghao Jiang, Jialun Cao, Michael Pradel, and Zhongxin Liu. Nocode-bench: A bench- mark for evaluating natural language-driven feature addition, 2025. URL https: pattps://arxi:. org/abs/ 2507. 18130
Dawei Gao, Zitao Li, Xuchen Pan, Weirui Kuang, Zhijian Ma. Bingchen Qian, Fei Wei, Wenhao Zhang, Yuexiang Xie, Daoyuan Chen, Liuyi Yao, Hongyi Peng, Ze Yu Zhang, Lin Zhu, Chen Cheng, Hongzhu Shi, Yaliang Li, Bolin Ding, and Jingren Zhou. Agentscope: A flexible yet robust multi-agent platform. CoRR, abs/2402. 14034, 2024.
Jonas Gehring, Kunhao Zheng, Jade Copet, Vegard Mella, Taco Cohen, and Gabriel Synnaeve. RLEF: Grounding code LLMs in execution feedback with reinforcement learning. In Forty- second International Conference on Machine Learning, 2025. htttttttt///peeeeesss
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. Deepseek-coder, When the large language model meets programming the rise of code intelligence, 2024. URL https:
//arxiv.orgg/bb//22000 14196
Sirui Hong. Mingehen Zhuge. Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for a multi-agent collaborative frame-
work, 2024.
http:///aaaw....or///////////00 00352
Arnay Kumar Jain, Gonzalo Gonzalez-Pumariega, Wayne Chen, Alexander M Rush, Wenting Zhao, and Sanjiban Choudhury. Multi-turn code generation through single-step rewards. In Forty- second International Conference on Machine Learning, 2025a. /
net Leet/foruu22l--aaeLLLe...
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezaama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint 2rXiv:2403.0797. 2024.
Naman Jain, Jaskirat Singh, Manish Shetty, Tianjun Zhang, Liang Zheng, Koushik Sen, and Ion
Stoica. R2e-gym: Procedural environment generation and hybrid verifiers for scaling open- weights SWE agents. In Second Conference Language Modeling, 2025b. URL https: //openreview. net http://oommmmdd////mmmme
Nan Jiang, Xiaopeng Li, Shiqi Wang, Qiang Zhou, Soneya Binta Hossain, Baishakhi Ray, Varun Kumar, Xiaofei Ma, and Anoop Deoras. Ledex: training 1lms to better self-debug and explain code. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS '24, Red Hook, NY, USA, 2025. Curran Associates Inc. ISBN 9798331314385.
12

---

Published as a conference paper at ICLR 2026
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? arXiv preprint vrXi::2310.06770. 2023.
Bowen Li, Wenhan Wu, Ziwei Tang. Lin Shi, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, Zhiyin Yu, He Du, Ping Yang, Dahua Lin, Chao Peng. and Kai Chen. Prompting large language models tackle the full software development lifecycle: A case study. In Owen Rambow, Leo Wanner, Marianna Apidianaki. Hend AI-Khalifa. Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the 31st International Conference on Com- putational Linguistics, pp. 751 1-331, Abu Dhabi, UAE, January 2025. Association for Compu- tational Linguistics. URL httpt: setttps:/acclantho. ogy. ogy. wwl.orgg22222 col Lnn-manmann 502/
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alphacode. Science, 378(6624):1092-1097 2022.
Jiawei Liu, Songrum Xie, Junhao Wang, Yuxiang Wei, Yifeng Ding, and LINGMING ZHANG. Evaluating language models for efficient code generation. In First Conference Language Mod- eling, 2024. URL http///pppeeeeeeeeeeee/ee net/ / for rum? fommid=ii d=TRCMMeAhmC
Kaiyuan Liu, Youcheng Pan, Yang Xiang, Daojing He, Jing Li, Yexing Du, and Tianrun Gao. Pro- jectEval: A benchmark for programming agents automated evaluation on project-level code gen- eration. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, pp. 20205 Vienna, Austria, July 2025. Association for Computational I Linguistics. ISBN 979-8-9116-256 5. doi: 10. 10.18653/v1/2025.15 finds.isniinneeee 1036. URL (tRthtt:s:: http//caaannaccccocceee org.oo/00....2222.../.// Findings-acl. 1036/
Yingwei Ma, Qingping Yang, Rongyu Cao, Binhua Li, Fei Huang, and Yongbin Li. Alibaba ling- maagent: Improving automated issue resolution via comprehensive repository exploration. arXiv
preprint arXiv:2406.0142 2024.
OpenAl. Gpt-5-codex, 2025a.
URL Accessed: 2025-09-23
-ereaddess----ccee todexxxx
OpenAl. Introducing
08
2025b.
Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, et al. Codeelo: Benchmarking competition-level code generation of lims with human-comparable elo ratings. arXiv preprint arXiv:2501.01257 2025.
Quan Shi, Michael Tang, Karthik Narasimhan, and Shunyu Yao. Can language models solve olympiad programming? arXiv preprint arXiv:2404.10952 2024.
Significant Gravitas. AutoGPT. URL httts::/ http::///...taaaacccc/////aaaaaa oGt
Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma. Mingzhang Zheng. Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. OpenHands: An Open Platform for AI Software Developers as Generalist Agents, 2024. URL htthht:t:///aa org/ abs/2407. 16741
Yanlin Wang, Yanli Wang, Daya Guo, Jiachi Chen, Ruikai Zhang, Yuchi Ma, and Zibin Zheng- RLCoder: Reinforcement Learning for Repository-Level Code Completion In 2025 IEEF/ACM 47th International Conference on Software Engineering (ICSE), pp. 1140-1152, Los Alamitos,
CA, USA. May 2025a. IEBE Computer Society. doi: 10. 10.1109//CSE55534.222225 2T0.:334447222000101.. URL http::/:/coo.eeecommmtteeeeeeees
/0///11..00 10.1119//CSE55377.. 7022.. 00014
Yinjie Wang, Ling Yang, Ye Tian, Ke Shen, and Mengdi Wang. Co-evolving Ilm coder and unit 03136
tester via reinforcement learning, 2025b. URL
13

---

Published as a conference paper at ICLR 2026
Yuxiang Wei, Zhe Wang, Jiawei Liu, Yifeng Ding, and Lingming Zhang. Magicoder: Empow- ering code generation with OSS-instruct. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceed- ings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 52632-52657 PMLR, 21-27 Jul 2024. URL https:
//proceedings.. m
mlr wrwss/ Dxees//235/wei11
html
Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying llm-based software engineering agents. arXiv preprint, 2024.
Zhihui Xie, Jie chen, Liyu Chen, Weichao Mao, Jingjing Xu, and Lingpeng Kong. Teaching lan- guage models to critique via reinforcement learning. In Forty-second International Conference Machine Learning. 2025. URL wtt.hhttss/// Tlettps:/openreviiee
Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang. Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie. Shuyan Zhou, and Graham Neubig. Theagentcompany: Benchmarking Ilm agents on consequential real world tasks, 2025.
URL httpttttss//aiii..oo.. www..ar.//////////.//.../......e
abs /241//24412 2412. 122..11666 14161
John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528-5008 2024.
John Yang, Carlos E Jimenez, Alex L Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R Narasimhan, Diyi Yang, Sida Wang, and Ofir Press. Swe-bench multimodal: Do ai systems generalize visual software domains? In The Thirteenth International Conference on Learning Representations, 2025a. BRLlattpss///peneeviie htt.:/Eorummidxriiiiqee..
Lei Yang, Renren Jin, Ling Shi, Jianxiang Peng, Yue Chen, and Deyi Xiong. Probench: Bench- marking large language models in competitive programming. arXiv preprint urXiv:2502. 2025b.
Xingdi Yuan, Morgane M Moss, Charbel El Feghali, Chinmay Singh, Darya Moldavskaya, Drew MacPhee, Lucas Caccia, Matheus Pereira, Minseon Kim, Alessandro Sordoni, and Marc- Alexandre Côté. debug-gym: A text-based environment for interactive debugging, 2025. URL
ablg:/bb//2550 21557
Zihan Zheng, Zerui Cheng, Zeyu Shen, Shang Zhou, Kaiyuan Liu, Hansen He, Dongruixuan Li, Stanley Wei, Hangyi Hao, Jianzhu Yao, et al. Livecodebench pro: How do olympiad medalists judge iims in competitive programming? arXiv preprint arXiv:2506. 1928, 2025.
Yifei Zhou, Song Jiang, Yuandong Tian, Jason Weston, Sergey Levine, Sainbayar Sukhbaatar, and Xian Li. Sweet-rl: Training multi-ture turn 1lm agents collaborative reasoning tasks, 2025. URL
https://arxx
org/ abws////..3 15478
Terry Yue Zhuo, Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen GONG, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng. Jiawei Liu, Qian Liu, Zijian Wang. David Lo, Binyuan Hui, Niklas Muennighoff, Daniel Fried, Xiaoning Du, Harm de Vries, and Le- andro Von Werra. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference Learning Representations, 2025. URL http://pnweeeeeeeeeeeeeeooraaaaa
14

---

Published as a conference paper at ICLR 2026
A STATEMENT ON LLM USAGE
Throughout the preparation of this manuscript, we utilized large language model (Google's Gem- ini) collaborative writing assistant. The model's contributions were primarily focused the articulation and presentation of our research. Specific tasks included refining the paper's ' narrative structure and logical flow, condensing and polishing sentences for clarity and impact, suggesting alternative terminology to strengthen key arguments, and providing feedback on the design and captions of figures. The core scientific comtributions-inee including the initial research ideation, the design and implementation of the USACOArena environment, the execution of experiments, and the analysis of the results-wee were conceived and conducted entirely by the human authors. The role of the LLM was that of writing partner and editor. We are disclosing its use here to maintain full transparency regarding our research and writing process.
B SYSTEMATIC ABLATIONS AND ROBUSTNESS ANALYSIS
To address potential concerns regarding the sensitivity of our benchmark specific hyperparameter choices, we conducted comprehensive ablation campaign. We systematically varied the envi- ronment's Credit Limits, Economic Costs (Hints and Testing), and Incentive Structures (Scoring Weights) across all models.
Rationale for Problem Selection: We executed these ablations on the USACO 2025 February Contest. Unlike the extremely difficult US Open or the easier earlier contests, the February Contest offers balanced difficulty distribution (a "Goldilocks" zone). This allows most agents solve early problems but forces them to make difficult, observable trade-off's later ones, making behavioral shifts decision-makien highly apparent. Due to computational resource constraints, the ablation studies presented below are averaged over 3 independent runs.
B. THE MEGA-ABLATIE MATRIX
Table presents the complete results of our sensitivity analysis across 7 environmental dimensions.
Table The Mega-Ablation Matrix on USACO 2025 February Contest. The "Main Result" column
reflects the standard configuration used in Section 4.2 "Exp. Score" exponential scoring
weights (1, 10, 100, 1000) instead of the default (1, 2, 5, 10}.
Model
Result (Table 10 config)
Low Credit (LOM)
High Credit (40M)
Free Test (50)
High Test (S1k)
Free Hiat (50)
High Hiat (100x)
Fiat Score
Exp. Score
Gemini--255ppo
13.2
13.0
19.7
GPT-5-Codex
12.0
5.7
4.3
3.7
3.7
16.3
GLM-4.5
3.2
1.0
0.7
1.0
1.0
1.0
1.7
1.3
1.0
Qwen3-235B
2.8
3,3
1.7
2.7
0.0
Clande
DeepSeek-
0.5
0.7
1.0
1.0
0.3
1.0
1.0
0.7
1.0
DeepScck-V3
0.3
0.3
1.0
0.3
0.7
0,3
Kimi-K2
B.2 SCIENTIHIC INSIGHTS FROM THE ABLATION SPECTRUM
The extensive data from the Mega-Ablation Matrix confirms that the USACOArena evaluation framework successfully isolates Strategic Intelligence from environmental noise. The key scien- tific findings include:
Credit Limits (Constraint vs. Capability): When reducing the budget to 10M (Low Credit), Gemini-2.55pro's performance drops significantly (from 13.2 8.3). This proves that the credit budget is highly meaningful constraint that accurately simulates roal-world scarcity and forces necessary trade-offs. Conversely, doubling the budget to 40M (High Credit) does not improve performance (13.2 to 13.0). This indicates that current state- of-the-ar art agents are Capability Limited, not Budget-Limited where throwing more computational resources (or money) at the problem yields sharply di- minishing returns. Agents frequently terminate with a surplus of budget in the 40M setting, getting stuck in reasoning loops rather than effectively burning resources for progress.
They hit a "reasoning wall"
15

---

Published as a conference paper at ICLR 2026
Robustness of the Model Hierarchy: The performance gap and overarching hierarchy (Gemini-2.5-pra > GPT-5-Codex > Others) remain invariant across drastic changes to Test Costs, Hint Costs, and Scoring Weights. This definitively proves that superior performance in USACOArena stems from intrinsic model capability and architectural fit, rather than overfitting specific parameter tuning exploiting arbitrary pricing structures.
The Floor Effect Weaker Models: Models with comparatively weaker coding rea- soning capabilities (e.g.. DeepSeek, GLM, Kimi) hover near a score of ~~10 regardless of the environmental settings (even with free hints free tests). This validates that US- ACOArena effectively filters out models that lack the prerequisite Fluid Intelligence even form a basic strategy. Only models that pass the initial capability threshold can participate in the deeper strategic meta-game measured by the arena.
C FOUNDATIONS IN COMPETITIVE PROGRAMMING STANDARDS
The design of USACOArena is deeply rooted in the well-established standards of human compet- itive programming. This appendix provides detailed rationale for our two foundational design choices: (I) the adoption of a competitive programming format as the evaluation paradigm, and (2) the specific selection of the ACM-ICPC ruleset over other formats, such as the IOI. These choices are crucial for creating an evaluation framework that measures the capabilities most relevant the practical application of coding agents in real-world software engineering contexts.
C.I THE RATIONALE FOR COMPETITIVE EVALUATION PARADIGM
Current static benchmarks for code generation, such as HumanEval (Chen et al. [021 and MBPP (Austin et al.j 2021), primarily evaluate an agent's ability to produce functionally correct code for isolated, well-defined problems. While valuable, this static pass/fail approach fails to capture the dynamic, resource-constraing decision- makinng process that defines true autonomous agency. It assesses what an agent can produce, but not how why it arrives at solution.
For decades, programming competitions have served as the de facto standard for assessing human intelligence in computational problem-solving They provide holistic evaluation by creating an environment where participants must:
Manage Finite Resources: Operate under strict constraints (e.g., time, computational re- sources), forcing strategic allocation of effort.
Prioritize Tasks: Analyze set of diverse problems, assess their difficulty, and strategi- cally decide the order in which to tackle them.
Balance Trade-offs: Make critical decisions between solution optimality, implementation speed, and correctness risk.
By situating agents in a competitive arena, we move beyond measuring mere correctness and begin to quantify these crucial agentic abilities. The competitive format transforms the evaluation from simple test of knowled inolledge into rigorous assessment of strategy, efficiency, and performance under pressure, providing a far richer and more meaningful signal of an agent's true capabilities.
C.2 WHY ACM-ICPC OVER IOI? A FRAMEWORK FOR ENGINEERING-ALIGNED EVALUATION
While both the ACM International Collegiate Programming Contest (ICPC) and the International Olympiad in Informatics (IOI) are premier competitions, their underlying philosophies and mechan- ies are tailored measure different skills. We deliberately model USACOArena on the ACM-ICPC because its format is substantially more aligned with the values and demands of professional soft- ware engineering. The goal is evaluate coding agents as potential engineering collaborators, not as pure research tools. This alignment is evident across several key dimensions.
Evaluation Paradigm: Breadth and Pragmatism vs. Depth and Originality. fundamentally test of problem-solving breadth and rapid implementation. A typical contest.
The ACM-ICPC
16

---

Published as a conference paper at ICLR 2026
features large number of problems (8-13) to be solved within tight five-hour window. This structure rewards broad, practical knowledge of standard algorithms and data structures, and the abilitt to quickly recognize a problem pattern and apply a known, reliable solution. This directly parallels the day-to-day reality of a software engineer, who must efficiently address a wide variety of tasks, from implementing new features to fixing bugs, by applying the right tool for the job.
In contrast, the IOI is a test of algorithmic depth and creative invention. With only a few (typ- ically three) highly complex problems per day, challenges participants to devise novel or highly optimized algorithms, often pushing the boundaries of known techniques. This format is more akin to academic research or work in a specialized R&D lab. For an agent intended to serve general- general-purpose purpose coding assistant, the broad-based, pragmatic skill set measured by the ICPC is a more relevant benchmark.
Scoring Philosophy: The Imperative of Zero-Bug Correctness. defining feature of the ACM- ICPC is its all-or-nothing scoring system. A submission eams credit if and only if it passes every single hidden test case. A solution that fails on single edge case equivalent to one that fails com- pletely. This binary outcome brutally enforces the concept of robustness and correctness, which is the bedrock of reliable software engineering. In a production environment, code with a "99% pass rate" is simply buggy code, and single critical failure can have catastrophic consequences. The ICPC format, therefore, directly measures an agent's 's ability to produce zero-bug solutions.
Furthermore, the ICPC's penalty system wyschh-which adds fixed time penalty for each incorrect sub- mission on a problem that is eventually solved-explicity disincentivizes a careless trial-and-error approach. It rewards careful planning, local testing, and a deep consideration of edge cases before submission, all of which are hallmarks of a disciplined engineering process.
The IOI, conversely, employs partial-credit system, awarding points based on the number of test cases a solution correctly handles. This is excellent for measuring incremental progress and rewarding clever heuristics that solve a subset of the problem. However, does not instill the same absolute imperative for correctness that the ICPC format does. For an agent destined for real-world deployment, the ICPC's unforgiving standard of correctness is a far more meaningful and critical measure of its reliability.
Alignment with Engineering Values: Efficiency and Delivery over Theoretical Optimality. The intense time pressure and large problem set the ACM-ICPC naturally encourage competi- tors to find the simplest, most direct path to a correct solution. The goal is not necessarily to write the most theoretically optimal or elegant code, but to write correct and efficient enough code to pass within the given constraints, and to do quickly. This mindset perfectly mirrors the agile, celivery-focused nature of modern software development, where delivering a working, maintainable feature schedule is paramount, and premature optimization is well-known anti-pattems
The 1OI's focus on a small number of extremely difficult problems, in contrast, incentivizes the pursuit of theoretical optimality. The challenge often lies in shaving off logarithmic factors in complexity or designing a complex algorithm that precisely meets stringent time and memory limits. While an exceptional display of algorithmic prowess, this is often a form of over-engineering in typical software development context. An agent that rapidly delivers a correct O(N log N) solution often more valuable than one that spends immense resources discover a complex O(N) solution, especially when the former is sufficient for the task at hand.
In summary, by adopting the ACM-ICPC format, we are explicitly choosing to evaluate coding agents against set of criteria that prioritize the core values of software engineering: broad appli- cability, rigorous correctness, and the efficient delivery of robust solutions under constraints. This makes UCACAArea not just a test of algorithmic knowledge, but direct measure of an agent's potential as a practical and reliable engineering tool.
D PROBLEM CORPUS AND DIFFICULTY BASELINING
Corpus Philosophy The problem corpus for USACOArena is not fixed, static set. is living collection that mirrors the official USACO contest schedule. For each of the four contests a USACO season, we adopt the official 12-problem set-three problems each for Bronze, Silver, Gold,
17

---

Published as a conference paper at ICLR 2026
and Platinum levels the basis for distinct USACOArena competition. This approach ensures that our benchmark stays current with the evolving difficulty and style of competitive programming problems and avoids any potential bias from manual problem curation.
Difficulty Baselining Study To provide an empirical baseline of difficulty for the novel problems in the 2024-2025 season, we conduct an analysis using top-tier agent, Gemini-2.5-pro. Each of the 48 problems is run with ample resources to assess its inherent solvability by state-of-the-art model. The results of this study, presented in Table 3 are used select or filter problems, but rather to provide grounded reference for analyzing agent performance in the main experiments. For example, this data helps us understand when an agent fails on problem that is known to be solvable, indicating potential strategic failure rather than fundamental capability gap.
Table 3: Difficulty baselining results for the 48 problems of the USACO 2024-2025 season, using Gemini-2.5-pra This data provides a grounded reference for expected problem difficulty in our main experiments.
Problem ID
Level
Result
Test Cases
Cons. Credit
LLM Calls
Sub.
1526
platinum
WA
1/20
10167024
48
37
1525
platinum
WA
1/15
10306989
45
36
1524
platinum
TLE
1/16
10018138
63
49
1523
gold
TLE
1/20
10151386
1522
gold
AC
15/15
1446214
6
1521
gold
AC
25/25
2818862
18
1520
silver
AC
17/17
2936761
17
14
1519
silver
AC
12/12
7132837
36
27
1518
silver
WA
0/18
10131801
55
45
1517
bronze
AC
11/11
510085
3
1516
bronze
AC
11/11
204672
2
1515
bronze
AC
12/12
137433
3
1502
platinum
WA
0/20
10164077
45
1501
platinum
WA
1/19
10035673
55
42
1500
platinum
WA
1/13
10065047
54
1499
gold
AC
18/18
252597
2
1498
gold
TLE
1/20
10012120
52
45
1497
gold
AC
21/21
2939562
10
1496
silver
AC
12/12
583964
4
1495
silver
AC
18/18
628954
4
1494
silver
TLE
1/18
10099233
53
35
1493
bronze
AC
13/13
214167
2
1492
bronze
AC
11/11
201263
2
1491
bronze
AC
16/16
104275
1478
platinum
WA
2/19
10175416
45
38
1477
platinum
TLE
1/14
10031850
46
28
1476
platinum
AC
23
735288
4
1475
gold
WA
4/18
10129632
50
1474
gold
AC
23/23
2377487
12
10
1473
gold
AC
16/16
1723281
8
1472
silver
AC
15/15
1368787
10
8
1471
silver
AC
16/16
411089
1470
silver
AC
23/23
420372
3
1469
bronze
AC
13/13
153411
2
1468
bronze
AC
11/11
420033
4
1467
bronze
AC
12/12
472262
4
1454
platinum
WA
1/20
10146203
70
26
1453
platinum
TLE
1/18
10067438
84
19
1452
platinum
AC
15/15
571806
Continued on next page
18

---

Published as a conference paper at ICLR 2026
Table 3: continued from previous page
Problem ID
Level
Result
Test Cases
Cons. Credit
LLM Calls
Submissions
1451
gold
AC
16/16
1151155
1450
gold
AC
23/23
4026173
26
13
1449
gold
AC
20/20
1119907
1448
silver
AC
13/13
2734345
11
1447
silver
AC
11/11
2035808
8
1446
silver
AC
11/11
1649536
8
1445
bronze
AC
13/13
373173
1444
bronze
AC
16/16
102214
1443
bronze
AC
13/13
311834
2
1
1430
platinum
TLE
4/24
10106758
55
45
1429
platinum
TLE
1/22
10170320
56
41
1428
platinum
AC
25/24
7526267
40
32
1427
gold
AC
20/20
1213337
7
4
1426
gold
AC
20/20
479578
1425
gold
AC
23/23
374396
3
2
1424
silver
AC
16/16
528027
1423
silver
AC
15/15
951701
5
2
1422
silver
WA
1/21
10032808
74
44
1421
bronze
TLE
2/11
10098563
66
51
1420
bronze
AC
11/11
173543
2
1419
bronze
AC
26/26
221140
E PARTICIPANT QUALIFICATION RESULTS
To ensure that the agents evaluated in main experimentspossess baseline of functional com- petency, we implemented a qualification stage for each of the four USACO contests. The core requirement for agent to successfully solve the single easiest Bronze-level problem from the respective contest set. Only agents that achieved an 'Accepted' (AC) status on this prerequisite task were included the full, multi-problem competitive runs analyzed in the main paper.
The following tables (Table 4 through Table 7 provide the detailed performance results for this qualification task across the four contests. These results only justify our selection of participants for the main analysis but also offer a preliminary glimpse into the vast performance disparities among the models. Even on these relatively simple entry-level problems, we observe significant variance in resource consumption (Consumed Credit) and efficiency (Submissions), foreshadowing more complex strategic differences analyzed the main text.
Table 4: Representative qualification results for the USACO 2024 December Contest. Agents are required to solve the easiest Bronze problem 1445. Cons. Credit means Consumed Credit.
Model
Result
Cons. Credit
LLM Calls
Submissions
Qualified
Gemini-22 Pro
AC
373,173
3
2
Yes
GPT-5-Codex
AC
26,986
1
Yes
Claude-4-Sonnet
AC
57,405
2
Yes
DeepSeek-V3. V3.1
69,514
Yes
DeepSeek- V3
AC
536,880
130
47
Yes
Kimi K2-0K0-
AC
449,065
35
17
Yes
Qwen3-235B
AC
495,102
49
7
Yes
GL.M-.55
AC
7,725
1
Yes
19

---

Published as a conference paper at ICLR 2026
Table 5: Representative qualification results for the USACO 2025 January Contest. To qualify, agents were required to solve at least one of the three available Bronze problems (1467, 1468, 1469).
Model
Result
Cons. Credit
LLM Calls
Submissions
Qualified
Gemini-2.5-PM GPT-5-Codex
AC
472,262
4
2
Yes
AC
177,127
1
Yes
WA
10,035,455
168
90
Claude-4-Sonnet
TLE
10,037.109
508
232
No
TLE
10,013,656
227
127
DeepSeek-Vi V3.1
497,429
46
Yes
Kimi K2-0905
AC
2,605,011
196
89
Yes
Qwen3-235B
AC
99,574
10
5
Yes
GLM-4.5
AC
245,624
29
5
Yes
Table 6: Representative qualification results for the USACO 2025 February Contest. Agents are required solve the easiest Bronze problem 1491. Cons. Credit means Consumed Credit.
Model
Result
Cons. Credit
LLM Calls
Submissions
Qualified
Gemini-2.5-Pro
AC
104,275
2
1
Yes
GPT-5-Codex
AC
19,739
1
Yes
Claude-4-Sonnet
AC
57,486
2
Yes
DeepSeek-V3.
V3.1
AC
14,048
1
Yes
DeepSeek-V2 V3
AC
6,918
4
1
Yes
Kimi-K2-0905
AC
213,684
18
9
Yes
Qwen3-235B
AC
122,844
3
Yes
GLM-4.5
AC
7,474
1
Yes
Table 7: Representative qualification results for the USACO 2025 US Open Contest. Agents are required to solve the easiest Bronze problem 1515. Cons. Credit means Consumed Credit.
Model
Result
Cons. Credit
LLM Calls
Submissions
Qualified
Gemini-2.5-Pro
AC
137433
Yes
GPT-5-Codex
AC
37583
Yes
Claude-4-sonect
AC
813746
20
9
Yes
DeepSeek-V3.1
AC
1153921
257
47
Yes
DeepSeek-V3
AC
239910
Yes
Kimi-K2
AC
1366068
95
23
Yes
Qwen3-235B
AC
308213
24
4
Yes
GLM-4.5
AC
245624
29
5
Yes
F DETAILED RESULTS OF MAIN COMPETITION
This section provides the detailed results from our main experiment, where each qualified agent competed in the full 12-problem UCAAAAren contest. To account for the stochastic nature agent performance and potential variations in LLM API responses, each agent completed the competition five times. The results presented in the main paper are the average of these five runs.
Table g presents the aggregated performance metrics for each agent, including the average rank, score, and consumed credit, along with their standard deviations. We also provide a breakdown of the average credit consumption across the main categories LLM inference, hints, and penalties- to offer deeper insight into each agent's prevailing strategy. The agents are sorted by their final average rank, determined first by average rank and then by average score and consumed credit.
The raw, run-by-run data for each agent, including detailed action logs and final scores for each of the five trials, available in the supplementary material for full reproducibility.
20

---

Published as a conference paper at ICLR 2026
Table 8: Aggregated results from the main experiment, averaged over runs across four contests. The data shows each agent's final rank, score, and credit consumption, reflecting their strategic priorities. Values are presented mean standard deviation.
Model
Avg. Rank
Avg. Score
Avg. Consumed Credit
Inference Credit
Hint Credit
Penalty Credit
Gemini-2.5-poo
1.30
0.47
13,762,787
13.76M 4.3M
2.5K 2.5K
4.1K
GPT-5-Codex
1,70 0.47
9.39 7.59
4,707
3.1M
4,71M = 3.IM
1.KK + 2.0K
0.3K + 0.3K
Owen3-235b
4.00 1.59
1.61 0.92
11,732.391 6.9M
11.17M + 6.5M
560.2K : 375.7K
3.6K 2.6K
GLM-4.5
4.35
1.57
2.33 2.28
7,249,215 4.0M
7,06M + 3.9M
161.7K 93.2K
22.7K + 16.8K
Ve
5.70 1.13
0.11 = 0.32
194,030 = 0.2M
0.17M + 0.2M
19.8K = 24.9K
1.2K = 1.3K
DeepSeck-Y V3.1
6.00
1,30
0.06 0.24
253.011
0.23M +: 0.4M
14K 2.7K
Kimi K2 0905
6.00
1.45
0.72 + 1.18
1,337,561 2.0M
1.32M + 2.0M
10.0K # 17.2K
3.0K 4.2K
Cluude--sounee
6.95
1.36
0.39 0.61
1,285,766 0.8M
1,28M + 0.8M
1.4K + 1.7K
L.K 0.6K
Table 9: Performance and Strategy Comparison within the GPT-5 Agent Family. parenthesis indicates the head-to-head win rate for each agent within its matchup.
The value in
Agent
Win Rate
Avg. Score
Avg. Credit Consumed
Attempted Problems
Submission Precision (%
Experiment 1: Generalist vs. Specialist
GPT-5 (Base)
100% %
20.3
14M
26
12.9%
GPT-5-Codex
0%
8.0
8M
16
68.2%
Experiment 2: Specialist vs. Agentic Framework
GPT-5-Codex
0%
5
4M
16
44.4%
Codex-CLI 100%
9.5
15M
21
52.4%
The aggregated results highlight key strategic differences. For example, while GPT-5 and Gemini- 2.5-pro are the clear top performers, GPT-5 consistently consumes less credit across all categories, indicating a more efficient problem-solving process. The credit breakdown also reveals that lower- tier models often accumulate significant penalty credit without corresponding increase score, suggesting tendency towards inefficient trial-and-ene -nd--rrro strategies.
G PROBING AGENT ARCHITECTURE: CASE STUDY ON THE GPT-5 FAMILY
To isolate the impact of different design choices, we conduct controlled "civil war" experiments within the GPT-5 family. We run two head-to-head matchups: the base GPT-5 against code- specialized variant, GPT-5-Codex; and GPT-5-Codex against Codex-CI.I, version augmented with an agentic framework. Each matchup is run three times on the challenging US Open contest problem set, with the averaged results reported in Table In our analysis, submission precision is defined as the ratio of correct submissions to the total number of attempts.
Our experiments reveal key trade-offss in agent development. In the first matchup, the specialized GPT-5-Codex adopts a far more cautious approach than its base model. It frequently uses the TEST action to ensure high submission precision, but its reluctance to attempt uncertain problems limited its overall score. This suggests that code-specialization enhances reliability, potentially at the cost of problem-solving initiative. The second matchup shows that the agentic framework Codex-CLI significantly improves performance, achieving a much higher score and win rate while maintaining the same high precision. This demonstrates that well-designed architecture can directly boost model's 's performance, though this may come with higher resource consumption.
H LARGE LANGUAGE MODEL DETAILS
Our evaluation leverages diverse suite of Large Language Models (LLMs) to ensure comprehen- sive analysis of agent capabilities within the USACOArena. Table providdes a detailed breakdown of the models employed in study, including their provider and associated costs. All pricing data,
21

---

Published as a conference paper at ICLR 2026
specified in U.S. dollars per million input and output tokens respectively, retrieved from Arti- ficial Analysis in September 2025. This selection represents a cross-section of the contemporary LLM landscape, encompassing models with varied architectures, parameter scales, and economic costs, thereby facilitating robust and multifaceted analysis agent performance.
Table 10: Specifications of Large Language Models used in our evaluation. Costs are denoted in USD per million tokens.
Provider
Model
Input Cost
Output Cost
OpenAl
GPT-5-2025-08-07
$1.25
$10.00
GFT-5-Codex
$1.25
$10.00
Google
Gemini 2.5 Pro
$1.25
$10.00
Anthropic
2Laude--ommet-442255611
$3.00
$15.00
DeepSeek
DeepSeek-v3
$0.27
$1.10
DecpSeck-v3.1
$0.27
$1.10
Alibaba Cloud
$0.70
$2.80
Moonshot
Kimi -kmi-02-0905
$1.00
$2.75
Zhipu AT
G1MM44.5
$0.59
$2.19
htttes//arrtiffcialannlggss..
22

---

Published as a conference paper at ICLR 2026
I USACOARENA HYPERPARAMETERS
The main experiments conducted in this study utilize a standardized default configuration for the USACOArena environment. This configuration, which is highly customizable facilitate diverse research questions, is detailed in Table
Table 11: USACOArena Competition Configuration Parameters
Parameter
Description
Default Value
Basic Setup
max_ max__redits_per_partiieee
Maximum credits per participant
20,000,000
Scoring System brenze_score si silver_score geld_score platinum_score score
Points for Bronze problems
1
Points for Silver problems
2
Points for Gold problems
5
Points for Platinum problems
LLM Inference agent_temperature
Model generation lemperature
0.7
Hint Request Costs
level_0_his 1_hnnt
Strategy hints cost Textbook knowledge cost Knowledge-specifie content cost
500
level_1_hine
1,000
level_ 1_nnn___ level_3_hint Levei_4_hint
1,000
Similar problems cost Example problems cost
1,500
1,500
Test Code Costs
test_codee
Base cost per test request
Penalty System
wA_penalts
Penalty for Wrong Answer Penalty for Runtime Error Penalty for Compile Error Penalty for Time Limit Exceeded Penalty for Memory Limit Exceeded
100
RE 3E_penaaty
100
CE je__ennlyty
100
TLE eLE_penalty
100
MLE_penalty
100
Problem Distribution
total_probles
Total number problems Bronze difficulty count
brenze_problemt silver silver_problems
3
Silver difficulty count
3
gold problems
Gold difficulty count
3
platinum_proh ems
Platinum difficulty count
3
PROMPT FOR USACOARENA EVALUATION
The following box details the complete prompt structure provided agents in the USACOArena competition. The prompt is designed as purely objective specification of the environment. It comprehensively delineates the foundational components the competition: the governing rules, the format for communicating game state, the complete set of available actions, and the structure of action results. Crucially, the prompt deliberately refrains from offering any strategic guidance or heuristics. This ensures that all observed strategies are properties of the agent's 's autonomous decision-making process, rather than reflection of guidance embedded in the instructions
23

---

Published as a conference paper at ICLR 2026
An Example Prompt for Evaluating Agentic LLMs in USACOArena
SYSTEM PROMPT
You competitive programming agent participating in coding competition. You will receive the rent state of the competition and results of your previous actions. Your goal is to solve as many problems as possible (achieve 'Accepted'' status).
Your final ranking is determined first by your total score. The score is weighted sum of the problems solve, with harder problems (c.g., Platinum) being worth more than casier ones (c.g., Bronze). If scores are tied, the agent with the lower total of (actual consumed credit penalties) ranks higher. You start with limited credit budget, and many actions consume credit. You will terminated from the competition when your actual consumed credit reaches the limit. Credit is consumed in three main ways:
1. LLM Inference: Generating responses, which consumes credit based the number of tokens you use.
2. Purchasing Hints; Using hints to help solve problems.
3. Testing Code: Running your code against test cases before final submission.
IMPORTANT:
Penalties from wrong submissions affect your ranking tie-breaker but do NOT count toward termination.
In this competition, solving problems is much more important than minimizing the consumed credit. So you should try your best to solve many problems possible.
Please respond with JSON object containing actionn and "parameters' ficlds.
USER PROMPT
COMPETITION RULES
Credit System:
Each participant starts with total of 20,000,000 credit limit.
Credit consumed by three main sources: LLM Inference, Purchasing Hints, and Testing Code.
Your participation ends when your actual consumed credit reaches the limit.
Scoring Rules:
Your Final Score is the sum of points from all problems you solve completely.
No partial credit is awarded.
Points are weighted by difficulty: Bronze (1), Silver (2), Gold (5), Platinum (10).
Penalties: A penalty of 100 points is incurred for CE, MLE, RE, TLE, and WA submissions.
Ranking and Tie-Breaking: Rank is determined by Final Score. Ties are broken by the lower
(Actual Consumed Credit Penalties).
Programming Languages: C++17, Java, and Python3 are available.
YOUR STATUS
Name: <agent_ramee
Consumed Credit: <consumed_credita
Solved Problems: <solved_list
Current Score: <score>
Penalty: <penalty>
AVAILABLE PROBLEMS
1p
<probl em em_id_2>
CURRENT RANKINGS
1. <Agent 1>: Score <81>, Credit+Penalty:: <C1> [ACTIVE]
2. <Agent 2>: Score <s2>, Credit+Fenalty: <C2> TERMINATED
24

---

Published as a conference paper at ICLR 2026
AVAILABLE ACTIONS
1. WEW_PPRBLEM: View problem details.
2. GET_HINT Get hint for prohlem (consumes credit). Levels 0-4 available.
3. SUBMIT SUBMIT_SOLUTTONN SOLUTION: Submit solution.
4. DEST_CODE: Test code with custom cases (consumes credit).
5. TERMINATE: End participation.
RESPONSE FORMAT
Please respond using the following JSON format:
"action"; "caction_name>"
"paransters"; { // Fill in parameters according the action type
K THE FIVE-TIERED HINT SYSTEM IN USACOARENA
Empirically, while current top-tier agents rarely utilize this hint infrastructure, its design is far from over-engineered; rather, it serves critical Capability Probe. The observation that agents con- sistently fail to purchase hints even when hopelessly stuck in loop reveals a fundamental Metacog- nitive Deficit. Current models lack the self-awwreness to evaluate their own progress and recognize when they require external theoretical intervention. The five-tiered system is an essential instrument for measuring the exact moment future architectures bridge this metacognitive gap.
To rigorously evaluate agent's ability to make strategic decisions under resource constraints, we engineered a sophisticated five-tiered hint system within USACOArena. This system not merely help feature; it functions costs and utilities. Agents must perform a cost-benefit analysis to decide if, when, and what type of hint to purchase. This design allows us to observe and quantify an agent's resource management and problem- solving strategies.
economic model where information is commodity with varying
Level 0: Strategic Guidance (500 Credit) formation about competitive programming.
This foundational hint provides high-level, static in-
Function: It delivers a pre-compiled document containing the core philosophy of com- petitive programming, a comprehensive debugging checklist, and general contest strategies (e.g., time management). The contents are derived from USACO Guide
Mechanism: The system retrieves the full content from a static JSON file (/dataset/corpuses /UUAACSSsstaaataee (("hint_level*) 0)).
json). The API call is parameter-free
This low-cost hint is intended for the early stages of competition, allowing an agent to establish baseline understanding of the meta-game without spending significant resources.
Strategic Purpose:
Level I:
Problem-Specific Textbook Content (1,000 Credit) This hint offers theoretical knowl- edge directly relevant to specific problem.
Function: It provides concise, relevant excerpt from competitive programming text- book that explains the theoretical concepts or algorithms needed for a given problem.
Mechanism: Upon receiving a problem_iie the system automatically extracts key al- gorithmic and data structure terms from the problem description. It then employs BM25 search algorithm to find the most relevant section in a 2.8MB textbook corpus which is de-
https://usaco. guide
25

---

Published as a conference paper at ICLR 2026
rived from Algorithms for Competitive Programming The top result is returned, truncated to 1,000 characters.
Strategic Purpose: This is for agents that can identify a knowledge gap related to a prob- lem but do not know the name of the required algorithm. It tests the agent's ability to recognize when it needs theoretical grounding.
Level 2:
Knowledge-Targete Textbook Content (1,000 Credit) Similar to Level 1, this hint also retrieves textbook content, but with key difference in agent interaction.
Function: It provides detailed explanation of a specific algorithm or data structure ex- plicitly named by the agent.
Mechanism: Instead of a problem_aa id, the agent must provide hint_ settt_nnowwedeee keyword (e.g.. "segment tree"). The system uses this keyword directly in its BM25 search against the same textbook corpus.
This hint for a more advanced scenario where an agent correctly identifies the required algorithm by name but needs learn its implementation details. tests an agent's 's self-awareness of its specific knowledge deficits.
Strategic Purpose:
Level Similar Problem Retrieval (1,500 Credit) This high-cost hint provides a concrete, solved example of similar problem.
Function: returns full problem description, along with a complete, vetted solution and explanation, for problem that semantically similar to the one the agent currently working on.
The system uses the current probl 1 em i''s text (description and samples) query for BM25 search against the entire USACO problem library derived from US- ACO Guide excluding problems from the current competition. The most similar problem is returned.
Mechanism:
This is powerful tool for agents that completely stuck the problem-solving approach. Its high cost forces the agent to consider whether viewing a direct analogy is worth the significant credit expenditure.
Strategie Purpose:
Level 4: Curated Example Problems (1,500 Credit) This is the most targeted hint, designed to provide practice on a specific topic a specific difficulty.
Function: It retrieves complete example problem (description, solution, complexity anal- ysis) that matches both a user-specified difficulty level and a knowledge keyword.
Mechanism: The system filters the entire USACO problem library based on both problem_cifficule difficulty (e.g.. "Bronze") and aiin__knwweddg. (e.g.. "complete search") tags provided by the agent.
This hint allows an agent to request a targeted exercise, simulating human's 's process of looking for practice problems. It tests the agent's 's ability to formulate precise learning objective.
Strategic Purpose:
httos:://op-aloorithmg www.sigrrriimmmtmmom com
https://usaco.guide
26


Reviews:
[{"review_id": "mLMVD5Tw82", "reviewer": "Reviewer_FRAP", "summary": "This paper introduces USACOArena, a competitive programming benchmark that evaluates coding agents on strategic decision-making under resource constraints rather than just code correctness. The authors translate time into a \"credit\" budget where every action costs credits. Experiments across USACO contests reveal that Gemini-2.5-pro's aggressive exploration strategy outperforms GPT-5-Codex's conservative perfectionism despite the latter having higher peak capability. Self-play experiments show behavioral diversity, suggesting the arena's potential as both evaluation testbed and training environment.", "questions": "1. Is there a theoretically optimal strategy for USACOArena given the credit model and problem distribution? How do current agents compare to this theoretical benchmark? Understanding the optimum would help assess whether observed strategies are actually good or just relatively better.\n\n2. You note agents lack strategic self-assessment, with Gemini defaulting to easier problems despite being capable of harder ones. What interventions improve self-assessment? Would meta-prompting about strategy help, or do we need fundamentally different architectures?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper tackles a genuinely underexplored problem in coding agent evaluation. While existing benchmarks focus on correctness or long-horizon planning, this work examines resource-constrained strategic decision-making, which matters for production deployment where every API call has real cost. The credit-based abstraction is creative and well-motivated as an agent-native alternative to wall-clock time.\n\n\n\n2. The practical significance is evident. The finding that Gemini-2.5-pro wins through broader exploration despite lower per-problem accuracy challenges assumptions about optimizing for correctness. This insight could influence how we design and deploy coding agents in cost-sensitive production environments.", "weaknesses": "1. My main concern is that this reads more like a well-executed technical report paper than a deep scientific contribution. The analysis remains largely descriptive, showing what strategies emerge without explaining why they succeed or fail. While we learn that Gemini-2.5-pro's aggressive strategy beats GPT-5-Codex's conservative approach, we don't understand when each strategy would be optimal or what problem characteristics favor exploration versus exploitation. The paper doesn't deeply analyze the actual decision-making process despite claiming this as a core contribution.\n\n2. The most significant gap is the complete absence of ablation studies. A lot of ablations are essential to understand whether the benchmark measures genuine strategic competence or just exploits specific parameter choices. Without them, we can't assess the robustness of the findings or generalize beyond the particular configuration tested.\n\n3. The benchmark relies on closed-source API models that may change or become unavailable, raising reproducibility concerns. Agent performance may be highly sensitive to prompt formulation, but there's no discussion of prompt optimization or robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Sf0fAyWWoR", "reviewer": "Reviewer_hZpW", "summary": "This paper introduces USACOArena, an ICPC-style coding environment where LLM agents operate under a limited credit budget for each action (prompts, compilations, tests, rollbacks). This framework is used to evaluate how coding agents strategize under resource constraints, revealing distinct decision-making behaviors beyond just producing correct solutions.", "questions": "1. Could the authors clarify how the specific credit costs (for prompts, compilations, tests, etc.) were chosen? For instance, is the cost of a compilation or a test run proportional to some real-world metric, or was it tuned experimentally? It would be helpful to know if they conducted any sensitivity analysis on these cost parameters. \n2. Does the paper propose any method to improve agents' decision-making under budget constraints, or is it strictly an evaluation of existing models without optimization guidance?\n3. How well might the proposed framework extend to other types of tasks or more complex scenarios? The current benchmark uses ICPC-style problems with binary outcomes. Have the authors considered tasks with partial credit or multi-phase projects to see how the budgeting concept applies there?\n4. What metric defines an agent's success beyond final correctness (e.g., a cost-adjusted score), and how is the \"know when to stop\" behavior evaluated quantitatively?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a realistic cost-budgeted coding challenge environment that addresses a clear gap in current LLM coding benchmarks.\n2. The study reveals novel insights into agent behavior (e.g., balancing exploratory attempts vs. conserving budget) that are not captured by conventional unlimited-attempt evaluations.\n3. The authors provide a reproducible benchmark, including code and decision logs, to support future research on resource-aware coding strategies.\n4. The experimental evaluation is thorough and compares multiple agents, highlighting differences in their decision-making under cost constraints.", "weaknesses": "Please refer to Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gSvgjKQ5mu", "reviewer": "Reviewer_15eu", "summary": "LLM Agents are typically graded by correctness. However, in real-world applications, there is always an associated cost budget that needs to be respected. This is particularly true for coding agents. In this work, the authors therefore propose the USACOArena benchmark that tests coding agents in a variety of software engineering tasks under budget constraints. In contrast to existing benchmarks like SWE Bench, which tests for a knowledge gap, USACOArena tests for a capability gap. Their benchmark is built on the ACM International Collegiate Programming Contest (ICPC) and frontier models as well as strong open-weights alternatives are evaluated.", "questions": "1. How can you ensure that there is no train-test set overlap in frontier models? Is the performance gap between proprietary frontier models and open-weight alternatives due to their training data? Of course, you cannot look inside those models, but is there any way to ensure that there is no overlap? I would be interested in your thoughts. This is particularly interesting, because GPT-5 Codex performs well for USACO 2024 December, 2025 January, 2025 February, and then performance suddenly drops for the 2025 US Open. This suggests there might be some overlap.\n2. Can you clarify how saturated the benchmark already is? I.e., what does it mean to achieve an average score of 15 out of 20?\n3. How do the performances compare to the human baseline?\n4. Do you have permission to use the ACM-ICPC problem instances? I am not an expert in how such compliance topics must be handled, and I did not find information in the manuscript. Therefore, I would like to hear your approach.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors propose a new software engineering benchmark that reflects the actual constraints in real software engineering benchmarks. This is achieved by translating the core human constraint of time to the equivalent agentic constraints of credit. I believe this benchmark will be useful for measuring the abilities of future agents.\n2. The benchmark is based on ACM-ICPC rules, a proven real contest for human software engineers. \n3. The benchmark architecture allows agents to interact through the standardized MCP protocol, which facilitates real-world adoption.\n4. The benchmark allows for a strategic analysis of the decision-making strategies employed by different agents (e.g., Gemini’s breadth-first vs Codex's perfectionist strategy). This is a particularly interesting component of the benchmark when compared to previous benchmarks.\n5. They compare frontier models and reveal a performance gap between proprietary models and open-weight alternatives.", "weaknesses": "1. The main concern I have is that the benchmark seems saturated with the best agents achieving an average score of ~15 of the possible 20. Therefore, it is unclear how long the benchmark will serve as a reliable testbed for agent performance. Moreover, it is unclear whether the strong performance of proprietary frontier models stems from a train-test overlap. \n2. There is no comparison to the human baseline. This comparison would be valuable to better understand how far the current generation of models is from human experts.  \n3. There are no runtime analyses of the compared models. It would be valuable to also see the wall-clock times for solving these challenges.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pgW3zGbMbp", "reviewer": "Reviewer_y3i4", "summary": "This paper introduces USACOArena, an interactive benchmark for evaluating LLM coding agents under resource constraints. The key innovation is translating the human constraint of time into an agent-native \"credit\" budget, where every action (LLM inference, testing, hints, submissions) consumes credits. Drawing from ACM-ICPC competitive programming, the arena forces agents to make strategic trade-offs rather than simply pursuing correctness. The authors evaluate leading models across four USACO contests, revealing distinct strategic profiles: Gemini-2.5-pro's aggressive exploration strategy versus GPT-5-Codex's conservative perfectionism. Self-play experiments demonstrate emergent behavioral diversity, suggesting the arena's potential as both a benchmark and training environment.", "questions": "Please address questions raised in weaknesses section. Can you provide results showing how agent rankings change under different credit limits (e.g., 10M, 20M, 40M)? Does the current limit meaningfully constrain behavior, or do agents typically finish with surplus credits? How often do agents actually use the five-tier hint system? If usage is negligible, the elaborate design may be over-engineered. If usage is significant, showing how top agents strategically employ hints would enrich the analysis.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper addresses a genuine gap in current evaluation methodologies. Existing benchmarks like HumanEval and SWE-bench measure \"what\" code is produced but ignore \"how\" agents arrive at solutions. The shift to evaluating decision-making under resource constraints is conceptually valuable and well-motivated. The credit-based operationalization of resource constraints is clever—it sidesteps the unfairness of using wall-clock time for API-based models while maintaining the pressure of scarcity.\n\nThe strategic profiling in Section 4.2 provides genuine insight. The finding that Gemini-2.5-pro wins despite GPT-5-Codex having higher peak capability (max score 29 vs 19) is compelling, and the exploration-exploitation framework effectively explains this paradox. \n\nThe paper is well-written with effective visual aids. Figure 1's comparison of SWE-Bench versus USACOArena clearly communicates the knowledge gap versus capability gap distinction. The motivation is compelling, and the ACM-ICPC grounding provides solid theoretical justification for the design choices.", "weaknesses": "I have some concern about missing ablations. The environment contains numerous hyperparameters (credit limits, penalty costs, hint prices, scoring weights) that appear somewhat arbitrary. Without systematic ablation studies, we cannot assess: Sensitivity of rankings to credit budgets (Would Gemini-2.5-pro still dominate with half or double the credit limit? Does the 20M threshold meaningfully constrain behavior, or could it be 10M or 40M without changing outcomes?), Impact of penalty structure (All penalties are uniformly 100 credits. Why not differentiate between compilation errors), Hint pricing justification, Scoring weight rationale, etc. There are a lot of design choices that needs more justification.\n\nThe paper would be substantially strengthened by at least one ablation study demonstrating that agents respond meaningfully to environmental parameters, validating that the credit system actually shapes behavior rather than being a post-hoc measurement wrapper.\nSection 3.3 mentions using the Model Context Protocol but doesn't explain why this specific protocol is necessary or superior to alternatives. What does MCP provide that a simple JSON API wouldn't? This feels like name-dropping a trendy framework without demonstrating its value.\nSection 4.4 shows that identical agents produce diverse outcomes but doesn't deeply investigate why. The trajectory analysis in Figure 5(b) is anecdotal (one match). More systematic analysis could reveal: What environmental factors cause divergence? Do certain problem orderings reliably lead to different outcomes? Can we predict which \"personality\" an agent will exhibit based on early decisions? The claim that this validates USACOArena as a \"training environment\" is unsupported—no actual training/RL experiments are conducted.\nWhile the paper reports standard deviations, there's no formal statistical testing. Are the performance differences between Gemini-2.5-pro and GPT-5-Codex statistically significant across the five runs? Pairwise comparisons with confidence intervals would strengthen claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)