AetherCode: Evaluating LLMs’ Ability to Win In Premier Programming Competitions
Authors: Zihan Wang
Jiaze Chen
Zhicheng Liu
Haojie Pan
Markus Mak
Yidi Du
Geonsik Moon
Aaron Tua
Kunshuo Peng
Jiayi Lu
Boqian Zou
Chenyang Ran
GuangTian
Shoutai Zhu
Duan Yeheng
Zhenghui Kang
Zhenxing Lin
Lishangshu
Qiang Luo
Qingshen Long
Zhiyong Chen
Yihan Xiao
Yurong Wu
Daoguang Zan
Mingxuan Wang
Ming Ding
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Model
Reasoning
Code LLM
Benchmark
Primary area: datasets and benchmarks

Abstract: Competitive programming has emerged as a critical benchmark for evaluating the reasoning and coding capabilities of Large Language Models (LLMs). Despite impressive progress on existing benchmarks, we argue that current evaluations overstate model proficiency, masking a substantial gap between LLMs and elite human programmers. This gap arises from two key limitations: insufficient difficulty and scope of benchmark problems, and evaluation bias from low-quality test cases. To address these shortcomings, we present **AetherCode**, a new benchmark that draws problems from premier programming competitions such as IOI and ICPC, offering broader coverage and higher difficulty. AetherCode further incorporates comprehensive, expert-validated test suites built through a hybrid of automated generation and human curation, ensuring rigorous and reliable assessment. By combining challenging problem design with robust evaluation, AetherCode provides a more faithful measure of LLM capabilities and sets a new standard for future research in code reasoning.

Published as a conference paper at ICLR 2026
AETHERCODE: EVALUATING LLMS' ABILITY TO WIN IN PREMIER PROGRAMMING COMPETITIONS
Zihan Wang" 1:2 Jiaze Chen 3 Zhicheng Liu", Haojie Pan" 3 Markus Mak', Yidi Du Geonsik Moon", Aaron Tua Kunshuo Peng", Jiayi Lu Boqian Zou", Chenyang Ran", 3 Guang Tian", Shoutai Zhu 3 Yeheng Duan 3 Zhenghui Kang". Zhenxing Lin Shangshu Li Qiang Luo", Qingshen Long", Zhiyong Chen', 3, Yihan Xiao Yurong Wu 3 Daoguang Zan Mingxuan Wang", Ming Ding"
1 "ttate Key Laboratory of General Artificial Intelligence, Peking University
2 School of Intelligence Science and Technology, Peking University
ByteDDance
zh. wangestu, pku. edu. en, shen. ka dbyyteanceeecoom
ABSTRACT
Competitive programming has emerged as a critical benchmark for evaluating the reasoning and coding capabilities Large Language Models (LLMs). Despite impressive progress on existing benchmarks, we argue that current evaluations overstate model proficiency, masking substantial gap between LLMs and elite human programmers. This gap arises from two key limitations: insufficient diffi- culty and scope of benchmark problems, and evaluation bias from low-quality test cases. To address these shortcomings, that draws problems from premier programming competitions such as IOI and ICPC, offering broader coverage and higher difficulty. AetherCode further incor- porates comprehensive, expert-validated test suites built through a hybrid of auto- mated generation and human curation, ensuring rigorous and reliable assessment. By combining challenging problem design with robust evaluation, AetherCode provides more faithful measure of L.M capabilities and sets a new standard for future research in code reasoning.
present AetherCode, a new benchmark
INTRODUCTION
Competitive programming is widely regarded as a crucial benchmark for evaluating the reasoning and coding capabilities of Large Language Models (LLMs) (OpenAl et al. (025).. Solving com- plex competitive programming problems demands not only sophisticated reasoning abilities but also knowledge from diverse domains, including mathematics, data structures, and algorithms. Recent years have witnessed rapid advancements in the reasoning capabilities of LLMs, key indicator of which is their success on a majority of existing code reasoning benchmarks. State-of-the-ar mod- els now achieve over 90% Pass@1 / accuracy on MBPP (Austin et al. 2021) and HumanEval (Chen) jet al. 2021), and over 80% on LiveCodeBench (Jain et al. 2025). These encouraging developments might lead one to ask: has competitive programming been mastered by LLMs?
In this paper, we argue that a significant gap still exists between the performance of LLMs and top-tier human competitors in programming contests. We propose that the perception of LLM dom- inance stems primarily from the limitations in the breadth and rigor of current code reasoning bench- marks, which are no longer sufficient fully assess the capabilities of today's increasingly powerful models. Specifically, we identify two main shortcomings in existing benchmarks:
Insufficient Difficulty and Scope. Early benchmarks such as HumanEval (Chen et al., 2021 and MBPP (Austin al. 2021) consist of basic coding tasks, for instance, sorting or reversing a list which present minimal reasoning challenges for ofate-of-the-arr LLMs. More recent "competition-leeell benchmarks often source problems from limited set websites. For example, LiveCodeBench (Jain et al.) 2025) collects problems mainly from LeetCode and AtCoder, while CodeELOO (Quan et al. 2025) and BiveCodeBench Pro

---

Published as a conference paper at ICLR 2026
(Zheng al. 2025 originate solely from CodeForces. The problems from these web- sites have inherent limitations. LeetCode problems are generally casier and often require only the implementation of single function rather than complete program. CodeForces contests, which typically feature 5-7 problems within a 2-3 hour timeframe, constrain the design space for problem setters, for example, leading to scarcity of problems that require complex, large-scale implementations.
Evaluation Bias from Low- Low-Quality Test Cases. Inaccurate verifiers introduce bias into the evaluation (Vendrow et al., 2025). The correctness of a piece of code verified us- ing a comprehensive set of test cases (input-output pairs). An incomplete test suite may fail to detect incorrect submissions, particularly those with subtle flaws, such as the mis- handling of corner cases solutions that exceed time limits under specific, extreme ditions. Consequently, designing high-quality test cases is a huge challenge that requires deep understanding of potential failure points, skill typically honed through extensive competitive programming experience. Most past benchmarks lack sufficiently rigorous cases. HumanEval (Chen et al.) (202]) and MBPP (Austin et al., (021), for instance, rely on a small number of handwritten test cases. Others, including EvalPlus (Liu et al.) 2023), CodeContests (Li et al., 2022), and LiveCodeBench (Jain et al. 2025). employ naive test case generation pipelines, such random mutation, which fall far short of the qual- ity of expert-designed test suites. Furthermore, recent research [Wang et al., 2025b) has revealed issues with test case correctness itself; for example, many test cases in the Code- Contests dataset do not adhere to the problem's 's constraints, causing even correct solutions to fail. It is worth noting that some recent benchmarks, such CodeELO (Quan et al. 2025) and LiveCodeBencch Pro (Zheng et al. 2025), have attempted to leverage the official CodeForces judging service indirectly access its high-quality, expert-crafted test cases. However, this approach presents two significant issues. First, it raises compliance risks, as CodeForces explicitly prohibits the use of crawlers on its judging interface. Second, this method is constrained by submission frequency limits, which impedes agile and flexible experimentation. Therefore, we contend that an open-source benchmark with high-quality, self-contained test cases remains critically important for the LLM community.
To address these challenges, we introduce AetherCode, a new benchmark with the following key contributions:
Problem Curation from Top-Tier Competitions. AetherCode is the first benchmark to system- atically collect latest problems from premier programming competitions worldwide. including the Olympiad in Informatics (OI) and the International Collegiate Programming Contest (ICPC). Our process involved comprehensive collection, meticulous cleaning, and format conversion of prob- lems from PDF to a Markdown+LaTE structure. Each problem statement was manually proofread for correctness, and team of competitive programming experts annotated each problem with clas- sification tags.
High-Quality Test Case Generation. We developed a hybrid methodology, combining automated generation with expert annotation, to create high-qualib, test cases for every problem. We evaluated the correctness and comprehensiveness of our test cases by validating them against large corpus of collected solutions, enforcing a standard of zero false positives and zero false negatives.
This paper is organized as follows: Section 2 details the benchmark curation process. Section presents our evaluation results. Section paper with comments for future research.
4 presents some related work, and Section 5 concludes the
BENCHMARK CURATION
This Section details the curation process of the AetherCode Benchmark. The overall curation pro- cess is illustrated in Fig. [] Then, Sections 2.1 and 2.2 describe the specifies of problem collection and categorizing, respectively. Section 2.3 explains how we construct high-quality test cases for each problem.
2

---

Published as a conference paper at ICLR 2026
Table 1: Comparison between ActherCode and other code reasoning benchmarks
Dataset
Difficulty
# Problems
Updates
Test Cases
Souree
Construction
HumanEval Chen
2021
164
Handcrafted
Original
MBPP Austin
2021
974
Handcrafted
Original
APPS Henddyycks
2021
5,000
Crawled
CodeForces, AtCoder etc.
USACO
307
Publicly accessible
USACO
CodeConiests el
2022
165
Mutation
CudeForces, AlCoder etc.
LiveCodeBench (Jain ot
2025
**
1055
Semi-automatic
LectCode, ArCoder
CodeELO Quan et
***
387
CodeForees
LiveCodeBench Pro Zheng et al.
2025
***
584
CodeForces
AodherCode
456
G-V Agent & Experts
Premier Contests
Statement Processing
MD
foundaunnntion
LaTeX
Prootreading
Verified Statement
tot INFORMATICS
Problem Categorizing
Statements
IAVI
No
Amnolator
Category
Dificulty
Time'rrganzer
Images
Construction
Submissions
deda USACO
Veriled
Acourary Check
Export
G-V Agant
Promier Contests
Figure/ 1: Curation process of AetherCode. (a) We begin by converting the collected problem statements from PDF to Marklown-LaTeX format, which is then manually proofread for accuracy. (b) Each problem is then categorized by its algorithm type and difficulty level, and also compile additional metadata, such the time and the organizer of the contest. (c) To ensure quality, we use G-V Agent conjunction with human experts to annotate test cases. The accuracy of these test cases is then evaluated against a collected set of solutions and further audited by gold medalists and professional problem setters.
2.1 PROBLEM COLLECTION
We source our problems from premier programming competitions worldwide rather than from on- line programming websites. Based their target audience, these competitions can be broadly categorized into two main series: the Olympiad in Informatics (OI) series, which is aimed at pre- college school students, and the International Collegiate Programming Contest (ICPC) series. which is designed for college students.
OI Series. The Olympiad in Informatics a series of competitions aimed at popularizing computer science knowledge among high-school students and cultivating outstanding talents in computer sci- ence. The OI competitions usually require participants to solve algorithm-related problems by pro- gramming. Take the International Olympiad in Informatics (IOI), the top-level event of OI, as example. Each contestant competes individually, and each country can send up to players. During the two-day competition, players need to independently solve 3 problems within hours each day, mainly using C++. Furthermore, various countries and regions host their own national or regional Ol competitions, such as the National Olympiad in Informaties (NOI) in China and the USA Com-

---

Published as a conference paper at ICLR 2026
puting Olympiad (USACO) in the United States. hop-performing contestants in these competitions earn the opportunity to advance to the IOI.
ICPC Series. The ICPC is the oldest, largest, and most prestigious university-level programming contest in the world. Each team consists of up to 3 students and uses one computer to solve 10 13 problems in 5 hours, using programming languages such C, C++, Java, Python. The team that correctly solves the most problems with the least total time wins.
The world is divided into several regions for the ICPC. In Europe, there are Central Europe (CERC), North Europe (NWERC), South-East Europe (SEERC), and South- Europe (SWERC) regions. Other regions include Asia-Pacific,, Asia East Continent, North America, Latin America, Africa, and Arab region, etc. The ICPC is a multi-tiered event. First, there are regional contests held worldwide from September to November each year. The top-performing teams in the regional contests advance to the regional finals or championships. Then, the best teams from these finals or championships qualify for the ICPC World Finals, which is usually held from April to June each year. This is the highest-level stage of the ICPC, where the best teams from around the world compete for the championship.
In addition to the official ICPC events, we also incorporated problems from other large-scale and renowned collegiate programming contests, such as the China Collegiate Programming Contest
(CCPC).
For each problem, we collected the following components:
Problem Statement. The statement typically comprises title, a detailed problem descrip- tion, input/output specifications, sample inputs and outputs with explanations, data range constraints, and time/memory limits. The majority of the problem statements was origi- nally PDF format. To enhance comprehension for LLMs, we converted these PDFs into a Markdown format with LaTeX for mathematical notations. Each converted file was then manually proofread ensure its accuracy.
Solutions. We curated collection of over 30,000 Human--ritted solutions for these prob- lems, encompassing both correct and incorrect submissions. For each problem, we ensured a minimum of 5 correct and 20 incorrect solutions. The primary purpose of collecting these solutions is to evaluate the quality of the subsequently generated test cases, process detailed in Section 2.3
Test Cases. A minority of the competitions, e.g., USACO, publicly released their official test cases, which collected and standardized. For problems where official test cases were not available, we constructed our high-quality test cases. The methodology for this construction is described in Section 233
Metadata. We also gathered auxiliary information, such the date of the competition (for decontamination purposes) and human contestant performance data (to facilitate difficulty assessment), among other available data points.
The data charactistics and statisties of AetherCode v1 are presented Table
2.2 PROBLEM CATEGORIZATION
Beyond curating problems, an equally critical step in constructing AetherCode was the systematic categorization of each problem to ensure comprehensive coverage and facilitate fine- the-grained evalua- tion. To this end, we adopted a multi- muulti-dimensionall categorization framework designed with the input of competitive programming experts:
Difficulty Segmentation. Problems were divided into four levels of difficulty: Easy, Mediam, Hard, and Extreme. This classification was guided by expert judgment as well as official contest results. Most of the problems are distributed roughly evenly among Easy, Medium, and Hard. Notably, problems that human contestant was able to solve during competition were specially classified as Extreme, representing challenges that push the boundaries of algorithmic reasoning. The number of problems under each difficulty level is presented in Fig, 2 This difficulty classification is judged entirely from the perspective of humans rather than being classified by LLM''s performance. This is because want to provide perspective to study how the difficulty for LLLMs differs from the
4

---

Published as a conference paper at ICLR 2026
difficulty in the eyes of humans. Specifically, rank problems within the same contest based on the number of participants who successfully solved them. For contests without leaderboards, as well as for determining the relative difficulty order across different contests, rely expert evaluation. Finally, based on the overall difficulty ranking of all problems, we divide the dataset into three roughly equal categories: Easy, Medium, and Hard.
Temporal and Contextual Dimensions. Each problem was annotated with metadata to enable both decontamination and longitudinal analysis of model performance: (I) Date of the contest, allowing chronological tracking of trends in problem design and model capabilities. (2) Organizer and competition type, primarily distinguishing between Olympiad in Informatics (OI) and International
Collegiate Programming Contest (ICPC) series. (3) Competition scope, categorizing contests as regional-level, national-level, or worldwide.
Problem Properties. Some problems require additional considerations beyond standard in- put-output interface: (1) Problems dependent on visual or image-based input were excluded from the benchmark. (2) Problems requiring special judges (custom checkers) were explicitly labeled to ensure proper handling during evaluation.
Algorithmic and Domain Categories. To capture the breadth of algorithmic knowledge tested programming contests, we implemented a hierarchical taxonomy as shown in Appendix [B] The first level consists of ten categories: Algorithm Basics, Search, Dynamic Programming. Strings, Mathematics, Data Structures, Graph Theory, Computational Geometry, Common Techniques, and Problems on Trees. The number of problems corresponding these ten major categories is shown in the 2 The second level has 144 categories, which are presented in Appendix B It includes more detailed algorithm tags. For example, the major category "Mathematics" contains several sub- categories such as Number Theory, Linear Algebra, Probability, Game Theory, Combinatorics, and Polynomials. Problems can belong multiple categories to reflect their criss-discyylinarry nature.
This structured categorization enables targeted evaluation of model strengths and weaknesses while also ensuring that AetherCode serves as scalable resource for future research. In particular, allows progress to be tracked across difficulty levels, problem types, and algorithmic domains, pro- viding a more comprehensive understanding of model capabilities.
2.3 TEST CASE CONSTRUCTION
Recent studies (Liu et al. 2023 Wang et al. 2025b) have highlighted concerns regarding the quality of test cases in several existing code datasets. For instance, benchmarks such MBPP (uustnn et al. 2021) and HumanEval (Chen et al. 2021) include only a limited number of handwritten test cases per problem. Others, like CodeContests (Li et al., (022) and EvalPlus (Liu al.) (023), rely on naive methods such as mutation to generate test cases. Consequently, such test cases are insufficient for comprehensively evaluating the correctness and efficiency of program. Therefore, we contend that the quality of test case construction is a critical factor determining the overall quality of benchmark.
Notably, some recent benchmarks (Quan et al. 2025 Zheng et al. 2025) directly utilize the Code- Forces's judging service for evaluation. This approach allows them indirectly access high-quality test cases created by professional problem setters, thereby circumventing the challenge of test case construction. However, this method presents potential compliance risks, CodeForces explicitly prohibits the use of crawlers on its judging interface. Furthermore, this approach is constrained by submission frequency limits, which impedes agile and flexible evaluation. Therefore, we argue that benchmark equipped with its own high-quality test cases remains critically important for the LLM community.
To ensure AetherCode possesses sufficiently high-quality test cases, we approached the task from two perspectives. First, we established more stringent evaluation criteria for test case quality, which is presented in Section 2.3.1 Second, employed hybrid approach, combining automated gen- eration with expert annotation, to construct the test cases, which are presented in Sections 2.3.2 and 2.3.3 A deteiled procedure of test case generation is presented in Appendix C
5

---

Published as a conference paper at ICLR 2026
2.3.1 TEST CASE QUALITY ASSESSMENT
Previous research on test case quality has predominantly focused on quantity, operating under the assumption that a greater number of test cases correlates with higher quality et al. 2022 2023). However, recent studies (Wang al., 2025b) indicate that quantity not a direct proxy for quality. This discrepancy arises from two primary issues. First, test cases in some older datasets, despite their volume, suffer from significant correctness issues, often violating the problem's explicit constraints. Second, conventional test case generation methods that merely amass large volumes of random data fail to provide adequate coverage of various special and corner cases.
Consequently, we depart from evaluating test cases by their quantity and instead propose direct assessment of their ability to discriminate between correct and incorrect solutions. In our framework, we conceptualize the entire test suite for a problem as a binary classifier, that is, classifier that distinguishes between correct and incorrect solutions. We then evaluate the performance of this classifier using& large, curated collection of both correct and incorrect submissions. We adopt the True Positive Rate (TPR) and True Negative Rate (TNR) as our primary evaluation metrics.
True Positive
Number of Passed Correct Solutions
TPR True Positive False Negative
Number of Correct Solutions
True Negative
Number of Rejected Incorrect Solutions
TNR True Negative
False Positive
Number of Incorrect Solutions
The TPR measures the correctness of the test cases; a high TPR indicates that correct solutions are not erroneously failed, which is expected when the test cases themselves valid. Conversely, the TNR measures the comprehensiveness or coverage of the test cases, quantifying their ability to detect (or "hack") incorrect solutions.
By employing a hybrid approach that combines automated generation with expert curation, we have achieved 100% TPR and 100% TNR on our collected solution set. This signifies that all collected correct solutions pass our test cases, while all collected incorrect solutions are successfully rejected. To the best of our knowledge, AetherCode the first benchmark that sets such high standard for test cases.
2.3.2 AUTOMATIC CONSTRUCTION OF TEST CASES
We employed the Generator-Valicn (G-V) Agent System (Wang et al. 2025b) to automatically construct cases for these problems. This is multi-agent system composed of two interacting agents: a validator and a generator. The generator agent writes test case generator program to produce diverse test cases, including random and various corner cases. The validator agent writes validator program to ensure that the test cases produced by the generator are correct and adhere to the problem's constraints.
Previous research (Wang et al. 2025b) has pointed out that some past code datasets blindly increased the quantity of test cases while ignoring their validity, thereby introducing significant bias into eval- uations. Therefore, the validator plays a crucial role in ensuring the correctness of the test cases. To further guarantee quality, we have added a manual -nman--n- step to review and correct the validator programs, ensuring that all of our test cases valid.
In this test case generation task, the G-V agent system alone achieves a TNR of 89.9%. Furthermore, due to the incorporation of additional human verification for the validator, it attains TPR of 100%. Recognizing that this Automatic Construction phase could not achieve a 100% TNR on its own, we introduced additional expert annotation stage further strengthen the test cases.
2.3.3 EXPERT ANNOTATION OF TEST CASES
To this end, we recruited 67 competitive programming experts. The majority of them hold Code- forces ratings above 2000, with a few experts exceeding 2600 and achieving the title of International Grandmaster. These experts were tasked with constructing targeted test cases specifically designed to fail the various incorrect solutions we had collected. These manually crafted test cases were then merged with the automatically generated ones to form the final test suite.

---

Published as a conference paper at ICLR 2026
Table 2: Data characteristics of AetherCode v1 (2401-2505)
Feteema (7))
(24)
Pank: (221)
Tech (147)
Eagy (159)
CATEGORY
METRIC
COUNT
Diffieulty
Category
2024
400
Gouph (64)
Year
2025
56
DP(110))
DS. (120)
Origin
(145)
Su. (20)
(a6k
ICPC
380
Test Cases
Avg. Tests
47.15
Figure 2: Difficulty and category distributions of the Aether- Code v1 (2401-2505). The definitions of the category abbre- viations are in Table 4
Categories
Categories
Tags
144
Furthermore, we recognized that for certain problems with limited number of collected incorrect solutions (fewer than 50), achieving 100% TNR might not sufficiently guarantec the robustness of the test cases. To address this, we subjected the test cases for problems to a manual quality audit by specialized review team. Each member of this elite team holds least three ICPC gold medals and has a minimum of two years experience in competitive programming problem-setting Their deep understanding of potential pitfalls and common errors in each problem allows them to leverage their extensive experience to further ensure the quality and comprehensiveness of the test cases. Specifically, this elite team further supplements missing corner various incorrect and inefficient solutions to verify the comprehensiveness of the test cases.
and additionally writes
Additionally, for problems that accept multiple valid outputs, customized judging scripts (a.k.a. checker, or special judge) were provided and thoroughly reviewed by these experts to ensure correct evaluation.
EVALUATION
Our evaluation includes reasoning models and 6 non-reasoning models. The reasoning mod- els comprise 04-mini minii-iigh (OpenAl c), Gemini 2. 5 Pro FFashh (Comanici et al. 2025) Seed- 1. 6 6-thinking (Chen et al. 2025a), DeepSeek R1 Guo et al. (2025), GLM-4.5 (Zeng and Owen Yang al. 2025a), among others. The non-
ct al. 2025 reasoning models consist of GPT-4 OpenAl a), GPT-40 (OpenAl) b), Kimi-K2 (Kimi-Team
let al. 2025), DeepSeek-yy [Liu al., 2024). claude-4- 4--de-44-oone. (without thinking), and 3wen Coooer. All models are configured with a maximum output length of 32,768 tokens. Each model is evaluated four times in each problem, and the average numbers are reported. Detailed settings of the experiment are presented in Appendix
3.1
MAIN RESULT
Table presents a comprehensive performance evaluation of several prominent models on Aether- Code. For full results, please refer to the online leaderboard. The analysis yields the following key conclusions:
Significant Performance Gap between Models. aa--mni-highh and Gemi mwm.iii-2.55.. 2.5-Proo de- liver exceptional performance, establishing an elite tier with significant gap over other models. They are notably two of the three models capable of tackling the "Extremely Difficult" problems. This consistent, substantial lead across difficulty tiers underscores the high degree of discrimina- tion provided by the AetherCode benchmark.
Reasoning Models Comprchensively Outperform Non-Reasoning Models. As anticipated, rea- soning models demonstrate markedly superior performance compared to non-reasoning models. For instance, models from the Qwen3 series, such as Owen3-32B, outperform several non-reasoning models despite having fewer parameters. More notably, even with four sampling attempts (Pass @ 4), the performance roa-reasoning models still falls short of that achieved by reasoning models. This phenomenon indicates that for complex tasks like coding competitions, the solution space

---

Published as a conference paper at ICLR 2026
Table Performance comparison between reasoning models and non-reasoning models on Acther- Code vl (%, 2401-2505). The Difficulty and Year columns show the models' pass @ scores on problems of varying difficulty levels and from different years. The Pass @N column displays the models' Pass @ Pass 2 and Pass @4 scores.
MODEL
Easy
Medium
Extreme
2024
2025
2
Reasoning Models
04 mini high
65.3
32.1
8.0
3.8
35.8
32.6
35.5
43.0
46.6
Gemini-2.5.89
2.5
25.0
Seed Thinkiinnnngggg 0715
53.9
20.2
4.7
28.3
14.7
26.6
33.0
38.5
DeepSeck-B 1-0022
46.2
16.0
3.8
23.4
14.3
22.3
27.4
32.4
Dwwa2-2259BAA2B-Thinnee
43.1
23.6
11.6
22.2
28.9
36.0
Seminn-22.5-Flan GLM-455
42.1
15.2
2.7
0
22.0
8.0
20.3
24.5
28.5
9.8
Owen3 2055--333--2222
37.6
12.4
1.9
19.1
7.1
17.6
21.7
25.2
Qwen3-32B
34.8
10.9
6.7
16.3
20.4
23.9
Claude 4.5 Sonnet thinking
36.8
8.8
17.1
10.3
16.3
19.8
23.3
Claxde-4-Opus-thine
30.0
5.2
1.0
13.1
7.6
12.4
15.6
18.2
Qwen33-B
23.7
4.8
2.7
13.0
13.5
Non-Reasoning
GPT-4.1
23.9
5.7
11.3
4.5
10.5
13.2
15.3
Kimi-K2
23.1
4.7
10.6
4.0
9.8
12.2
14.5
DeepScok-V3-OFe V3--234
20.8
4,0
8.9
5.4
8.5
10.5
12.3
Owen3-Coder-t80B
Claude 4. Somnet nothinking
18.4
2.6
0.8
7.9
4.5
7.5
9.1
11.0
GPT-4o
11.6
1.0
1.3
4.4
5.6
7.0
ploration capabilities of non-reasoning models constrained, making it difficult to find correct solutions through limited sampling. This bottleneck is particularly pronounced in weaker models.
comparison of Pass and Pass @4 scores reveals that increasing the number of samples yields more substantial performance improve- ment for top-tier models. For example, 04- 04-mini mini-high -high's is score improved by 11.1% (from 35.5% to 46.6%), whereas the weaker Qwen3- -33B only saw gain of 7.6% (from 16.3% to 23.9%). Particularly noteworthy Gemini-22 5-Pro, which achieved a remarkable performance increase of 13.3% (from 32.5% to 46.0%) This demonstrates its vast exploration potential in solving com- plex programming problems, enabling it to generate more diverse and high-quality solutions through multiple attempts.
Top-Tier Models Exhibit Great Exploration Potential.
3.2 PERFORMANCE ACROSS ALGORITHMS
The performance comparison in Table reveals significant differentiation in model capabilities across various problem categories. All models, regardless of being reasoning or non-reasoning types, uniformly excel pattern-based tasks such "Basic Algorithms" and "Strings". However, their limitations become equally apparent when handling highly abstract problems. Most mod- els struggle to tackle "Computational Geometry" and "Tree Structures", with the performance of o4-mini-he -iiii in computational geometry being notable exception. Furthermore, the short- comings non-reasoning models are particularly pronounced, their capability bottlenecks extend into domains that also demand deep logic and abstract thinking, such as "Dynamic Programming" and "Mathematics".
worth noting that, due to the inconsistent distribution of problems across categories, individual categories (such Tree) may happen to be particularly difficult, resulting in lower model scores. The difficulty distribution for each category is presented Appendix B
Generally speaking, models with higher overall scores also tend to be stronger across nearly every subcategory, with 04-mini-high ranking first in them. However, this analysis also allows us to identify the weaknesses of certain models. For example, while GPT-4.1 I has the highest overall score among the non-reasoning models, its performance on mathematical problems significantly weaker.
8

---

Published as a conference paper at ICLR 2026
Table 4: Performance comparison (Pass @ @1) between reasoning models and non non-reasoning models across 10 major categories: Algorithm Basics (Basic), Search, Dynamic Programming (DP), Strings (Str.), Mathematics (Math), Data Structures (DS), Graph Theory (Graph), Computational Geometry (Geo.), Common Techniques (Tech.), and Problems Trees (Tree).
Basic
Search
Math
Graph
Geo.
Reavoning Models
of-mini-high
38.1
28.5
27.7
35.6
31.8
25.8
28,5
27.1
26.9
7.3
Gemnini
32.2
17.0
17.3
26.0
24.2
17.9
18.8
12.5
19.2
1.0
DeepSeck-R1-0522
26.3
14.6
19.3
15.6
10.4
7.3
26.2
14.5
15.0
20.2
21.1
14.8
15.6
11.8
15.1
4.2
Gemini-2.5-Flash
24.1
16.5
11.8
19.2
16.7
16.3
17.2
13.2
11A
4.2
GLM-4.5
13.0
2.1
Owen3-235B-A222
22.2
13.0
8.4
20.2
13.5
11.0
12.5
11.1
9.4
4.2
Qwen3
19.7
10.9
Clande-4 (.nndl-,,5-Soneetee 5-Soonntt-hiininng
20.7
11.5
8.2
17.3
9.1
944
10.6
11.1
11.4
0
Clandd-t-Ouusthinking
16.0
10.0
5.7
17.3
6.3
7.7
0
Qwen3-8B
13.3
3.9
15.4
7.6
6.3
1.4
1,0
Non-Rensoning Models
13.9
4.2
Kimi K2
13.7
7.5
3.6
15.4
7.0
8.1
0.7
0
DeepSeok-V3-000 V3-0324
12.1
1.8
14.4
3.9
6.3
0
0
Twen3-Coder--80BAAT
11.1
14.4
4.2
1.4
1.0
Claudb-4-Somsx--othiie
10.9
8.0
1.8
13.5
2.6
5.0
3.5
2.1
3.4
0
GPT-4o
3.3 DIAGNOSIS FAILURE REASONS
We categorize all model failure cases into four types. Wrong Answer means the program outputted an incorrect result. Time Limit Exceeded means the program failed to output an answer within the given time limit. Runtime Error means the program encountered an error during runtime, including Segmentation Error, exceeding the memory limit, etc. Compile Error means the program has syn- and could not be successfully compiled. The statistical results presented in Appendix Table
For the majority of models, the primary enror type is Wrong Answer, accounting for approximately 70% to 80% cases, followed by Time Limit Exceeded. The Claude series is slightly different, with Wrong Answer and Time Limil Exceeded each accounting for roughly half of the errors. We conducted study on the failure cases of the Claude models and found that, on difficult problems, they tend design algorithms that are correct but inefficient, rather than prioritizing adherence to the problem's 's time complexity constraints.
Compile Error rates vary significantly across different models. Most models maintain Compile Error rate within 10%, with the Claude series achieving the lowest. However, some models ex- hibit particularly high Compile Error rates, such as GLM-4.5. Our analysis of GLM-4.5's Compile Error cases revealed that over half were caused by the model using the incorrect programming lan- guage; for example, it writes a Python program while being instructed to use C++. This indicates deficiency in GLM-4.5's ability to follow programming language instructions.
To further analyze the causes of model failures, we conducted a more granular attribution er- ror types. We performed a qualitative analysis of reasoning cases from o4 o4-mini -iiih The primary failure reasons identified include: incorrect algorithmic logic, failure to handle corner cases, insufficient algorithmic efficiency, and implementation errors. Furthermore, 04-mini- -high sometimes acknowledges its inability to solve problem rather than providing an incorrect answer. Relevant problems and corresponding cases provided in the Appendix E
found that

---

Published as a conference paper at ICLR 2026
4 RELATED WORK
4.1 CODE BENCHMARKS
Coding ability is one of the important capabilities of LLMs. How to evaluate the coding ability of LLMs has also received widespread attention from researchers in recent years. Existing code benchmarks can be roughly divided into three categories: basic benchmarks, code reasoning bench- (SWE) benchmarks. Representative of the basic benchmarks are HumanEval (Chen et al.) 2021). MBPP (Austin et al. 202I). CoderEval (Yu et al. 2024), PPM (Chen et al. 2024). DynaCode (Hu et al.) 2025). and DyCodeEval (Chen et al. 2025b). which con- tain some fundamental programming tasks such as sorting and simple sequence operations. Because they are relatively basic, they are also suitable as observation metries for model capabilities during the pre-training stage. Software-engineering (SWF) benchmarks mainly focus on repository-level engineering code tasks, examining the model's s agentic capabilities, environmental interaction, code comprehension, task planning. instruction following, and long-context abilities. Representative ex- amples in this category include SWE-bench (Jimenez et al., 2024), SWE-bench Verified (Chowd-) hury et al. 2025), Multi-SWE-bench (Zan et al.] 2025), and EvoCodeBench (Li et al. 2024). Code reasoning benchmarks taneously evaluate a model's 's reasoning and coding abilities, testing its capacity for deep reasoning. Representative examples include CodeContests (Li et al. (0022 and LiveCodeBencch (Jain et al., 2025). In this paper, we mainly focus on code reasoning tasks.
marks, and coftware-engineene
primarily composed of competitive-
programming tasks that simul-
4.2 CODE REASONING BENCHMARKS
Code reasoning benchmarks primarily consist of competition-level programming problems and used to evaluate the deep reasoning capabilities of LLMs. Many existing benchmarks collect prob- lems from online programming websites like LeetCode and CodeForces, including CodeContests (Li et al. 2022) (from Aizu, AtCoder, Codechef, CodeForces, HackerFarth). LiveCodeBench (Jain) al. (022)) (from AtCoder and LeetCode), CodeELO (Quan al. 2025) (from CodeForces), Live- CodeBench Pro (Zheng et al. (2025) (from CodeForces), and ProBench (Yang et al. (025b) (from CodeForces, Luogu, Nower) Some efforts have also been made to gather problems from ma- jor competitions, including USACO Bench (Shi et al. 2024). LLM-Pros (Hossain et al., 20255) OJBench (Wang et al., (2025a), and ICPC-Eval (Xu et al.) 2025); however, these are limited to a few specific contests, and some rely outdated data, posing a significant risk of data contamination. For example, Eval only includes 11 ICPC contests from 2023 to 2024: USACO Benchmark includes USACO problems from 2011 to 2023; OJBench only includes 4 ICPC contests with NOI problems from 2016 to 2023; LLM-Pros includes 14 ICPC contests from 2011 to 2024. To our knowledge, AetherCode is the first benchmark to comprehensively collect latest problems from pre- mier competitions around the world, surpassing previous work in both breadth and depth.
CONCLUSION
In this paper, introduced AetherCode, a challenging, rigorously evaluated benchmark purpose- built to assess LLMs' coding and reasoning capabilities. AetherCode distinguishes itself by sourc- ing all its problems from premier global programming competitions, including OI series and ICPC series, which ensures high degree of challenge and relevance. Furthermore, it features a compre- hensive and meticulously validated suite of test cases, created through a hybrid model of automated generation and expert curation. By validating against a dataset of over 30,000 human submissions, our test suite achieves 100% TPR and 100% TNR on our collected solution set, guaranteeing excep- tional accuracy and reliability in evaluation.
Our comprehensive evaluation of several leading-edge models on AetherCode yielded critical in- sights. We observed significant performance disparity among models, with top performers like 04-mo-ie 11-mini-hit seii-highh thigh and Gemini- 2. 5-rro establishing a distinct upper tier. Reasoning models demonstrated clear and consistent advantage over their non-reasoning counterparts across all diffi- culty levels, highlighting the crucial role of logical deduction in solving complex algorithmic prob- lems. Overall, even the most advanced models today can only solve a small fraction of problems in AetherCode. This indicates that current LLMs still have considerable room for improvement in reasoning and coding, and there remains a significant gap compared to top human experts.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We thank Siyao Liu, Jinxin Chi, Haojie Pan, Jingjing Xu, Stanley Wong, Ge Zhang, Wenhao Huang, Yonghui Wu, as well as other colleagues at ByteDance, and more importantly, the anonymized competitive programming expert team, for their support for the AetherCode project.
REFERENCES
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski. David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoe Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2 108. 077327 2021.
Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiang- peng Wei, Wenyuan Xu, Yufeng Yuan, Yu Yue, Lin Yan, Qiying Yu, Xiaochen Zuo. Chi Zhang, Ruofei Zhu, Zhecheng An, Zhihao Bai, Yu Bao, Xingyan Bin, Jiangjie Chen, Feng Chen, Hong- min Chen, Riwei Chen, Liangqiang Chen, Zixin Chen, Jinsong Chen, Siyan Chen, Kaiyuan Chen, Zhi Chen, Jin Chen, Jiecao Chen, Jinxin Chi, Weinan Dai, Ning Dai, Jiahui Dai, Shihan Dou, Yantao Du, Zhengyin Du Jianhui Duan, Chen Dun, Ting-Han Fan, Jiazhan Feng, Junda Feng, Ziyuan Feng, Yuwei Fu, Wenqi Fu, Hanjie Fu, Hao Ge, Hongyi Guo, Mingji Han, Li Han, Wen- hao Hao, Xintong Hao, Qianyu He, Jerry He, Feng He, Wen Heng, Zehua Hong, Qi Hou, Liang Hu, Shengding Hu, Nan Hu, Kai Hua, Qi Huang, Ziyue Huang. Hongzhi Huang, Zihao Huang. Ting Huang, Wenhao Huang. Wei Jia, Bin Jia, Xiaoying Jia, Yuhua Jiang, Haobin Jiang, Ziheng Jiang. Kaihua Jiang, Chengquan Jiang, Jianpeng Jiao, Xiaoran Jin, Xing Jin, Xunhao Lai, Zheng Li. Xiang Li, Liyi Li, Hongkai Li, Zheng Li, Shengxian Wan, Ya Wang, Yunshui Li, Chenggang Li, Niuniu Li, Siyu Li, Xi Li, Xiao Li, Aoyan Li, Yuntao Li, Nianning Liang, Xinnian Liang, Haibin Lin, Weijian Lin, Ye Lin, Zhicheng Liu, Guanlin Liu, Guanlin Liu, Chenxiao Liu, Yan Liu, Gaohong Liu, Juncai Liu, Chundian Liu, Deyi Liu, Kaibo Liu, Siyao Liu, Qi Liu, Yongfei Liu, Kang Liu, Gan Liu, Boyi Liu, Rui Long, Weiqiang Lou, Chenwei Lou, Xiang Luo, Yao Luo, Caiping Lv, Heyang Lv, Bole Ma. Qianli Ma, Hongzhi Ma, Yiyuan Ma, Jin Ma, Wenchang Ma, Tingting Ma, Chen Mao, Qiyang Min, Zhe Nan, Guanghan Ning, Jinxiang Ou, Haojie Pan. Ren- ming Pang. Yanghua Peng. Tao Peng. Lihua Qian, Lihua Qian. Mu Qiao. Meng Qu, Cheng Ren, Hongbin Ren, Yong Shan, Wei Shen, Ke Shen, Kai Shen, Guangming Sheng. Jinlong Shi, Wenlei Shi, Guang Shi, Shuai Shuai Cao, Yuxin Song, Zuquan Song, Jing Su, Yifan Sun, Tao Sun, Zewei Sun, Borui Wan, Zihan Wang, Xiaohui Wang, Xi Wang. Shuguang Wang, Jun Wang, Qinlong Wang, Chenyuan Wang, Shuai Wang, Zihan Wang, Changbao Wang, Jiaqiang Wang, Shihang Wang. Xuwu Wang, Zaiyuan Wang, Yuxuan Wang. Wenqi Wang, Taiqing Wang, Chengzhi Wei, Houmin Wei, Ziyum Wei, Shufa Wei, Zheng Wu, Yonghui Wu, Yangjun Wu, Bohong Wu, Shuang Wu, Jingqiao Wu, Ning Wu, Shuangzhi Wu, Jianmin Wu, Chenguang Xi, Fan Xia, Yuqiao Xian, Liang Xiang. Boren Xiang, Bowen Xiao, Zhen Xiao, Xia Xiao, Yongsheng Xiao, Chao Xin, Shulin Xin, Yuwen Xiong, Jingjing Xu, Ziwen Xu, Chenyin Xu, Jiayi Xu, Yifan Xu, Wei Xu, Yufei Xu, Shikun Xu, Shipeng Yan, Shen Yan, Qingping Yang. Xi Yang, Tianhao Yang. Yuchang Yang, Yuan Yang, Ximing Yang, Zeyu Yang, Guang Yang. Yifan Yang. Xuesong Yao, Bairen Yi, Fan Yin, Jianian Yin, Ziqiang Ying, Xiangyu Yu, Hongli Yu, Song Yu, Menghan Yu, Huan Yu, Siyu Yuan, Jun Yuan, Yutao Zeng. Tianyang Zhan, Zheng Zhang. Yun Zhang, Mofan Zhang, Wang Zhang, Ru Zhang, Zhi Zhang, Tianqi Zhang, Xinyi Zhang, Zhexi Zhang, Sijun Zhang, Wen- qiang Zhang, Xiangxiang Zhang. Yongtao Zhang, Yuyu Zhang, Ge Zhang, He Zhang, Yue Zhang, Renjie Zheng, Ningxin Zheng, Zhuolin Zheng, Yaowei Zheng, Chen Zheng, Xiaoyun Zhi, Wan- jun Zhong, Cheng Zhong, Zheng Zhong, Baoquan Zhong, Xun Zhou, Na Zhou, Huan Zhou, Hang Zhu, Defa Zhu, Wenjia Zhu, and Lei Zuo. S-edI.5- Advancing superb reasoning models with reinforcement learning, 2025a. htthttss: http://awwvvv...g.oo abs 2504. 13914
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards. Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec
11

---

Published as a conference paper at ICLR 2026
Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Me- Grew, Dario Amodei, Sam McCandlish, llya Sutskever. and Wojciech Zaremba. Evaluating large org/
language models trained code, 2021. URL
www.oo/ggg/aaaaeeeeeeeee http//bb/22000 03374
Simin Chen, Xiaoning Feng. Xiaohong Han, Cong Liu, and Wei Yang. PPM: automated genera- tion of diverse programming problems for benchmarking code generation models. Proc. ACM Softw. Eng., ((FSE): 1915, 2024, doi: 10. 10.1145/3643780, URL //tp::://oo.... org/1.....g......
1145/3643788
Simin Chen, Pranav Pusarla, and Baishakhi Ray. Dynamic benchmarking of reasoning capabilities in code large language models under data contamination, 2025b. htt.:https:/aattt org/
abs/2503, 04149
Yao Cheng, Jianfeng Chen, Jie Chen, Li Chen, Liyu Chen, Wentao Chen, Zhengyu Chen, Shijie Geng, Aoyan Li, Bo Li, Bowen Li, Linyi Li, Boyi Liu, Jiaheng Liu, Kaibo Liu, Qi Liu, Shukai Liu, Siyao Liu, Tianyi Liu, Tingkai Liu, Yongfei Liu, Rui Long, Jing Mai, Guanghan Ning, Z. Y. Peng, Kai Shen, Jiahao Su, Jing Su, Tao Sun, Yifan Sun, Yunzhe Tao, Guoyin Wang, Siwei Wang, Xuwu Wang. Yite Wang, Zihan Wang, Jinxiang Xia, Liang Xiang, Xia Xiao, Yongsheng Xiao, Chenguang Xi, Shulin Xin, Jingjing Xu, Shikun Xu, Hongxia Yang. Jack Yang, Yingxiang Yang, Jianbo Yuan, Jun Zhang, Yufeng Zhang, Yuyu Zhang, Shen Zheng, He Zhu, and Ming Zhu. Fullstack bench: Evaluating lims as full stack coders, 2025. URL
org
ab1/2412. 00535
Neil Chowdhury, James Aung, Chan Jun Shern, Oliver Jaffe, Dane Sherburn, Giulio Starace, Evan Mays, Rachel Dias. Marwan Aljubeh, Mia Glaese Carlos E. Jimenez. John Yang. Leyton Ho, Tejal Patwardhan, Kevin Liu, and Aleksander Madry. Introducing SWE-bench verified, 2025.
URL httpss://ppentt
com/ hoom/index: wwwpnndeeeiitttniiinnno.mme beech fied/
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan- Jiang Jiang, Krishna Haridasan, Ahmed Omran, Nikunj Saunshi, Dara Bahri, Gaurav Mishra, Eric Chu, Toby Boyd, Brad Hekman, Aaron Parisi, Chaoyi Zhang, Kornraphop Kawintiranon, et al. Gemini 2.5: Pushing the frontier with advanced reasoning. multimodality, long context, and next generation agentic capabilities, 2025. URLhttps: /ttp:///axiiv.... org wwtp://..wwrabb/.......o 66261
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.1244 2025.
Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazcika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace Hc, Dawn Song, and Jacob Steinhardt. Mcasuring coding challenge competence with APPS. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track Datasets and Bench- marks 1, NeurlPS Datasets and Benchmarks 2021, December 2021, virtual, 2021. URL https: /ttt:///aattttsssss www.he www.aarkksssoooooeoeooeceee www.ieuur..... wwwerips..ccppppe..ee.. pttpp///pp////21///// boaaaaa 2atol/aa002-bbAraactttaa Abstract round2. html
Md. S. Hossain, Anika Tabassum, Md. Fahim Arefin, and Tarannum Shaila Zaman. LIm-pros: Analyzing large language models' performance in competitive problem solving. In IEEE/ACM International Workshop on large Language Models for Code, LLM4Codee ICSE 2025, Ottawa, ON, Canada, May 3, 2025, pp. 80 87. IEEE, 2025. doi: 11100091LLAA00007777770002220000 URL wttphhtp://eect /ol. http:::///ooo.oo/////// 1109/ A1A.:///MAACC..663.....2........
Wenhao Hu, Jinhao Duan, Chunchen Wei, Li Zhang, Yue Zhang, and Kaidi Xu. Dynacode: A dynamic complexity-aware code benchmark for evaluating large language models in code gen- eration. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.). Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 August 1, 2025, pp. 21980-21997, Assoccation for Computational Linguistics, 2025.
URL https: //acl antho
fiddiifiiiiinnn www.dningnall......m.. 1133/
.13333
12

---

Published as a conference paper at ICLR 2026
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang. Armando Selar-Lezama, Koushik Sen, and lon Stoica. LiveCodeBench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025.
URL wtt.hhtps::// hhtttps://oppnreviiee
Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. Swe-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https:: httpps://ppeneeview.co net wtw.:/uuwwweeeeouuunmo forum? VIF8yNQM66
Kimi-Team. Yifan Bai. Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi K2: Open agentic intelligence. arXiv preprint arXiv:2507.20:3 2025.
Jia Li, Ge Li, Xuanming Zhang, Yihong Dong, and Zhi Jin. Evocodebench: An evolving code gen- eration benchmark aligmed with real-world code repositories, 2024. URL https:
org/abs/20d /200. 00599
Rongao Li, Jie Fu, Bo- Wen Zhang. Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. TACO: Topics in Algorithmic COde generation dataset, December 2023. URL http:
/ //arxiv. wttp://orggaab////aeeeeee
14852 arXiv:2312. 14852 version:
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cy- prien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Rob- son, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition- level code generation with AlphaCode. Science, 378(6624) 2022. doi: 10.1126/ 100./..111cccc./ ence.
science.abo aerence.abql13. 1158. URL https: www. sci www.sccceierrooooo.oo.oo.ooo.oee www.ccciecce.org//b/////a/// abqq 158
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NewrIPS 2023, New Orleuns, LA, USA, December 10 pttp://ee///pp/////////// 312.66/077c222255555333333333-00
http:apppprrrriissccccppppapppaa 16, 2023, 2023. URI. ottpp/////..tt////ppppppp.aaaaa.
html
OpenAl. httPa///htpppp/ggeaaaaca///////p
a. Accessed: 2025-04-14.
OpenAl. GPT-4o. ht.pps//pgenal..ommmdeeeeeeeeeee 03-13.
b. Accessed: 2024-
OpenAl.
Introducing----an---e 04-mini-high.
https://opena: com. /dex/ c. Accessed: 2025-04-16.
OpenAl, Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song. Hunter Lightman. Ignasi Clavera, Jakub Pachocki, Jerry Tworek, Lorenz Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohaninejad, Nat McAleese, o3 con- tributors, Oleg Mürk, Rhythm Garg, Rui Shu, Szymon Sidor, Vineet Kosaraju, and Wenda Zhou. Competitive programming with large reasoning models, 2025. URL btttts:///axiiv...
abss labs/2502.00 06807
Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, et al. CodeElo: Benchmarking competition code generation of Ilms with human-comparable Elo ratings. arXiv preprint arXiv:2501.01257, 2025.
13

---

Published as a conference paper at ICLR 2026
Quan Shi, Michael Tang, Karthik Narasimhan, and Shunyu Yao. Can language models solve olympiad programming? arXiv preprint arXiv:2404.10993 2024.
Joshua Vendrow, Edward Vendrow, Sara Beery, and Aleksander Madry, Do large language model benchmarks test reliability?, 2025.
03461
Zhexu Wang, Yiping Liu, Yejie Wang, Wenyang He, Bofci Gao, Muxi Diao, Yanxu Chen, Kelin Fu, Flood Sung, Zhilin Yang, Tianyu Liu, and Weiran Xu. Ojbench: A competition level code bench- mark for large language models, 2025a. URL httpt:tttttttiiiteee wwwpaawwww.w.orraaooooaoo abs 2ttpa////555666333333 16395
Zihan Wang, Siyao Liu, Yang Sun, Ming Ding, and Hongyan Li. CodeContests+: High-quality test case generation for competitive programming. In Christos Christodooulppoulos,, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Findings of the Association for Computa- tional Linguistics: EMNLP 2025, pp. 5576-5600, Suzhou, China, November 2025b. Association for Computational Linguistics. ISBN 8888888-77373555 URL org/2025. findiiggnnennnnnn
299/]
Shiyi Xu, Yiwen Hu, Yingqian Min, Zhipeng Chen, Wayne Xin Zhao, and Ji-Rong Wen. Iepc- eval: Probing the frontiers of IIm reasoning with competitive programming contests, 2025. URL.
https://arxi: org/ abs 2506. 04894
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint 20X0025505.09388, 2025a.
Lei Yang. Renren Jin, Ling Shi, Jianxiang Peng, Yue Chen, and Deyi Xiong, Probench: Benchmark- ing large language models in competitive programming, 2025b. URL htttp:://arxitt org/
abs/ / 2202 20
Hao Yu, Bo Shen, Dezhi Ran, Jiaxin Zhang, Qi Zhang, Yuchi Ma, Guangtai Liang, Ying Li, Qianx- iang Wang, and Tao Xie. Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In Proceedings of the 46th IEEE/ACM International Conference Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024, pp. 37:1-37:1 ACM, 2024. doi: 10. 00.1145/55755033623333
wR..tttss: doi www.oo/g///// 10. 01155/55950005 36231166663111
Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, Siyao Liu, Yongsheng Xiao, Liangqiang Chen, Yuyu Zhang, Jing Su, Tianyu Liu, Rui Long, Kai Shen, and Liang Xiang. Multi-swe-bench: multilingual benchmark for issue resolving, 2025. htthttps::/ arxiv. org/ www.//s/////oooooooooo//eeeeeee/ 2504 02605
Aohan Zeng, Xin Lv, Qinkai Zheng. Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng. Jiajie Zhang, Kedong Wang. Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, Yean Cheng. Yifan An, Yilin Niu, Yuanhao Wen, Yushi Bai, Zhengxiao Du, Zihan Wang, Zilin Zhu, Bohan Zhang, Bosi Wen, Bowen Wu, Bowen Xu, Can Huang, Casey Zhao, Changpeng Cai, Chao Yu, Chen Li, Chendi Ge, Chenghua Huang. Chenhui Zhang, Chenxi Xu, Chenzheng Zhu, Chuang Li, Congfeng Yin, Daoyan Lin, Dayong Yang. Dazhi Jiang, Ding Ai, Erle Zhu, Fei Wang. Gengzheng Pan, Guo Wang. Hailong Sun, Haitao Li, Haiyang Li, Haiyi Hu, Hanyu Zhang, Hao Peng. Hao Tai, Haoke Zhang, Haoran Wang, Haoyu Yang. He Liu, He Zhao, Hongwei Liu, Hongxi Yan, Huan Liu, Huilong Chen, Ji Li, Jiajing Zhao, Jiamin Ren, Jian Jiao, Jiani Zhao, Jianyang Yan, Jiaqi Wang, Jiayi Gui, Jiayue Zhao, Jie Liu, Jijie Li, Jing Li. Jing Lu, Jingsen Wang, Jingwei Yuan, Jingxuan Li, Jingzhao Du, Jinhua Du Jinxin Liu, Junkai Zhi, Junli Gao, Ke Wang. Lekang Yang, Liang Xu, Lin Fan, Lindong Wu, Lintao Ding. Lu Wang, Man Zhang, Minghao Li. Minghuan Xu, Mingming Zhao, Mingshu Zhai, Pengfan Du, Qian Dong, Shangde Lei. Shangqing Tu. Shangtong Yang, Shaoyou Lu, Shijie Li, Shuang Li, Shuang-Li, Shuxun Yang. Sibo Yi, Tianshu Yu, Wei Tian, Weihan Wang, Wenbo Yu, Weng Lam Tam, Wenjie Liang, Wentao Liu, Xiao Wang, Xiaohan Jia, Xiaotao Gu, Xiaoying Ling, Xin Wang, Xing Fan, Xingru Pan, Xinyuan Zhang, Xinze Zhang, Xiuqing Fu, Xunkai Zhang, Yabo Xu, Yandong Wu, Yida Lu, Yidong Wang, Yilin Zhou, Yiming Pan, Ying Zhang, Yingli Wang. Yingru Li, Yinpei Su, Yipeng Geng, Yitong Zhu, Yongkun Yang, Yuhang Li, Yuhao Wu, Yujiang Li, Yunan Liu, Yunqing Wang, Yuntao Li, Yuxuan Zhang, Zezhen Liu, Zhen Yang, Zhengda Zhou, Zhongpei Qiao, Zhuoer Feng. Zhuorui Liu, Zichen Zhang Zihan Wang, Zijun Yao, Zikang Wang, Ziqiang Liu, Ziwei Chai, Zixuan Li, Zuodong Zhao, Wenguang
14

---

Published as a conference paper at ICLR 2026
Chen, Jidong Zhai, Bin Xu, Minlie Huang, Hongning Wang, Juanzi Li, Yuxiao Dong, and Jie
Tang. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models, 2025. URL https:
Zihan Zheng, Zerui Cheng, Zeyu Shen, Shang Zhou, Kaiyuan Liu, Hansen He, Dongruixuan Li, Stanley Wei, Hangyi Hao, Jianzhu Yao, Peiyao Sheng, Zixuan Wang, Wenhao Chai, Aleksandra Korolova, Peter Henderson, Sanjeev Arora, Pramod Viswanath. Jingbo Shang, and Saining Xie. Livecodebench pro: How do olympiad medalists judge llms in competitive programming?, 2025. URL https: http:///wwwwo.oaab///oooooso wttporggg/bb///// 11928
15

---

Published as a conference paper at ICLR 2026
A EXPERIMENT SETTINGS
The experimental platform is equipped with 3.8 GHz Intel CPUs and is isolated into several pods using Docker. The container environment SandboxFusion (Cheng et al. (2025) with Ubuntu 20.04, where each container instance is exclusively allocated 2 cores and GB of memory. Each container instance runs only one piece code at time. The gac version used for compiling the code 9.4.0, with the C++17 standard and 02 optimization enabled.
We use the following user prompt for the evaluation.
Please solve the following programming problem using [LANGUAGE]. Please place your final answer in a markdown code block. (STATEMENT)
B CATEGORY DETAILS
The distribution of problem difficulty for each primary category presented in Table 5 and the complete list of the primary categories and secondary categories (tags) is presented in Table 0
C COMPLETE PROCEDURE OF TEST CASE GENERATION
The complete test case generation process is follows:
1. Writing validator. validator a program used to verify whether test case input adheres the problem's constraints. We first utilize the validator agent to generate the validator program, followed by manual correction of any errors.
2. Writing generator. We employ the generator agent to create generator program. This program then used produce test case inputs, which are passed the ground truth solution to obtain the corresponding test case outputs.
3. Writing checker and interactor. We utilize checker and interactor agents to generate the respective programs. checker program is essential for problems that accept multiple valid solutions, while an interactor program is required for interactive problems. Subsequently, these undergo manual review and error correction.
4. Human expert augmentation. Human experts supplement the machine machine-generated test cases, adding new cases until a 100% TNR is achieved on the collected solution set.
5. Elite team audit. Finally, our elite team conducts comprehensive review of each problem. This process includes adding corner cases, rejecting unqualified samples, and specifically authoring incorrect or inefficient solutions re-verify the coverage of the test cases.
D SOURCE AND COPYRIGHT DETAILS
The complete list of the contest sources of AetherCode is presented in Table
Some of the problems have clear copyright holders and licenses. including:
IOI. Copyright held by the IOI General Assembly; released under the CC BY License.
JOL. Copyright held by The Japanese Committee of International Olympiad in Informaties; released under the CC BY-SA 4.0 License.
USACO. Copyright held by USACO; released under the CC BY-NC-SA 4.0 License.
NOI (China). Copyright held by the China Computer Federation; released under the CC BY-NC 4.0 License.
For some problems, the authorization or copyright status currently unverifiable. We remain com- mitted to removing any potentially infringing problems upon the request of the copyright holders.
16

---

Published as a conference paper at ICLR 2026
E DETAILS OF FAILURE ANALYSIS
The distribution of failure reasons (by judging verdicts) across evaluated models is presented in Table g
We performed qualitative analysis of the causes of errors in 04 co4-mini-hie The primary erro categories and their corresponding cases listed below. The problems and responses for these cases are provided in the supplementary material.
aases/acknww// weikknwwlddge i Inability The model admits its inability to complete the problem, outputting: "I'm sorry, but I can't get to a working solution the time have.
cases/corner.can The model identified key properties but failed properly handle all corner cases.
cases/implementenn error: The model encountered an error during code im- plementation, specifially failing close parentheses.
cases/Incorrecttlogg The model failed to employ the correct algorithmic logic.
cases/inefficient The model used an inefficient algorithm. In this example, the model correctly calculated the algorithm's time complexity but failed to realize that an O(n") algorithm is typically unable handle a data scale of n 5000 within one second.
F LLM USAGE
In this work, utilized LLMs facilitate writing on tasks such text refinement, translation, and searching for related literature. Furthermore, Lision-Languag Models (VLMs) were employed for the generation of illustrations.
Table 5: Distribution of problem difficulty for each Category in AetherCode v1 (2401-2505).
Category
Easy
Medium
Hard
Extreme
Basic Algorithms
43.11%
27.11%
25.78%
4.00%
Common Techniques
30.61%
42.18%
21.77%
5.44%
Computational Geometry
16.67%
30.56%
47.22%
5.56%
Data Structures
23.33%
33.33%
37.50%
5.83%
Dynamic Programming
20.91%
30.91%
45.45%
2.73%
Graph Theory
21.88%
31.25%
37.50%
9.38%
Mathematics
25.00%
32.29%
36.46%
6.25%
Search
28.00%
32.00%
36.00%
4.00%
Strings
38.46%
15.38%
38.46%
7.69%
Tree Problems
16.67%
25.00%
50.00%
8.33%
17

---

Published as a conference paper at ICLR 2026
Table 6: Category division and detailed tag distribution of AetherCode.
Category
Tags
Algorithm Basics
Enumeration, Simulation, Recursion, Greedy, Sorting, Divide and Con- quer, Binary Search, Doubling, Recurrence
Search
DFS, BFS, Bidirectional Search, Heuristic Search, A Iterative Deep- ening Search, IDA", Dancing Links
Dynamic Programming
Basic DP, Memorization Search, Knapsack DP, Range DP, DP on DAGs, Tree DP, Bitmask DP, Digit DP, Plug DP, Counting DP, Dy- namic DP, Probability DP. DP Optimization
Strings
String Matching, String Hashing, Trie, Palindrome Automation, Pre- fix Function, Z-function, Automation, AC Automation, Suflix Aray, Suffix Automation, Suffix Balanced Tree, Generalized Suflix Automa- tion, Suffix Tree, Manacher's 's Algorithm, KMP Algorithm, Sequence Automation, Minimal Representation, Lyndon Factorization, Main- Lorentz Algorithm
Mathematics
Number Theory, Linear Algebra, Linear Programming, Abstract Al- gebra, Probability Theory. Game Theory, Young Matrix, Inclusion- Exclusion Principle, Combinatories, Polynomials
Data Structures
Stack. Queue, Linked List, Hash Table, Disjoint Set Union, Heap, Block Structure, Monotonic Queue, ST Table, Binary Indexed Tree, Segment Tree, Balanced Tree, Binary Tree & Balanced Tree, Block Decomposi- tion, Persistent Data Structures, Tree-in-Tree, K-D Tree, Cartesian Tree, Huffman Tree, STL-based Data Structure
Graph Theory
Matrix-Tree Theorem, Directed Acyclic Graph, Topological Sort, Min- imum Spanning Trec, Minimum Diameter Spanning Tree, Minimum Tree Spanning, Connectivity, Shortest Path, 2-SAT, Difference Con- straints, Hamiltonian Graph, Modular Shortest Path, Graph Coloring, Eulerian Graph, Dominating Tree, Bipartite Graph, Prüfer Sequence, Planar Graph, Chordal Graph, Network Flow, Graph Matching, Ran- dom Walk Graphs, LGV Lemma, Strongly Connected Components
Computational Geometry
Fuclidean Distance, Manhattan Distance, Chebyshev Distance, Pick's Theorem, Triangulation, Convex Hull, Sweep Line, Rotating Calipers, Half-Plane Intersection, Closest Pair of Points, Random Increment Method, Reflection Transformation, Misc. CG
Common Techniques
Discretization, Two Pointer Technique, Prefix Sum Difference, Frac- tional Programming. Randomization, Hanging Line Method, Binary Thinking, Pattern Recognition. Gray Code. Expression Evaluation, Construction, Properties of Bitwise Operations, Conjecture of Conclu- sions, Interactive Problems, Meet in Middle, Ad-hoe, Uncertainty Al- gorithms, Square Root Decomposition
Problems on Trees
LCA, DSU on Tree, Divide and Conquer on Points, Block Decomposi- tion on Tree, Heavy-Light Decomposition, Chain Decomposition, Tree Dianeter and Centroid, LCT
18

---

Published as a conference paper at ICLR 2026
Table 7: Curated Contest Source of AetherCode v1 (2401-2505).
Competition Name
Category
Date
Croatian Open Competition in Informatics 2023/2024 Contest #3
Ol
2024/1/13
USACO 2024 Jamuary Contest (Platinum)
USACO Platinum
2024/1/26
The 2023-2024 ICPC Southwestem Europe Regional Contest
ICPC Regional Contests
2024/1/28
Croatian Open Competition in Informatics 2023/2024 Contest #4
Croation O1
2024/2/10
USACO 2024 February Contest (Platinum)
USACO Platinum
2024/2/16
USACO 2024 US Open Contest (Platinum)
USACO Platinum
2024/3/15
Singapore National Olympiad in Informatics 2024 Final Contest Croadaa Open Competition in Informatics 202232024 Contest N5
NOI (SG)
2024/3/16
Of
2024/3/16
The 2024 ICPC atin America Championship
ICPC Regional Championchips/iini
2024/3/17
The 2024 ICPC Europe Championsbip
ICPC Regional Championshipssinals
2024/3/24
The 2024 British Informatics Olympiad Final
British OI
2024/4/6
Baltic Olympiad in Informatics 2024 Day! 1
Baltic Ol
2024/5/4
Baltic
in Informatics 2024 Day
Balric Of
2024/5/5
Asia-Pacific Informatics Olympiad 2024 (APIO 2024)
APIO
2024/5/18
The 2024 ICPC North America Championship
ICPC Regional Championshigs/Fina
2024/5/27
Central Furopean Olympind in Informatics 2024 Day (CEO1 2024 Day I)
Central European Of
2024/6/25
Contral Europcan Olympiad Informatics 2024 Day (CEO) 2024 Day 2)
Central European Ol
2024/6/27
China National Olympiad in Informatics 2024 Day
NOI
2024/7/18
China National Olympiad in Informaties 2024 Day
NOI
2024/7/20
Furopean Girls' Olympiad in Informatics 2024 Day
European Girl's Of
2024/7/23
European Girls' Olympiad in Informatics 2024 Day 2 International Olympiad in Informatics
Europcan Girl's Of
2024/7/25
Day
101
2024/9/3
International Olympind in Informatics 2024 Day
10I
2024/9/5
The 2024 ICPC World Finals Astana
ICPC World Finals
2024/919
The 2024 ICPC Kumming Invitational Contest
ICPC Regional Contests
202499/88
The 2024 Nordic Collegiate: Programming Contest
NCPC
2024/10/5
Croatian Open Competition in
2024/2025 Contest #1
Croation Of
2024/10/5
CCPC 2024 Harbin Site
OCPC
2024/10//26
The 2024 ICPC Asia Chengdu Regional Contest
ICPC Regional Contests
2024/10/27
The 2024 ICPC Asia Nanjing Regional Contest
ICPC Regional Contests
2024/11/3
Croatian Open Competition in Informatics 2U24/2025 Contest #2
Croatian O1
2024/11/9
2024 2025 ICPC Latin American Regional Programming Contest
ICPC Reyional Championships/Fnni
2024/11/9
2024 Rocky Mountain Regional Contest
ICPC Regional Contests
2024/1 1/9
2024 North Central NA Regional Contest
ICPC Regional Contests
2024/11/9
2024 Mid-Central USA Programming Contest
ICPC Regional Contests
2024/11/9
CCPC 2024 Chongging Site
CCPC
2024/1 1/10
The 2024 ICPC Greater NY Regional Contest
ICPC Regional Contests
2024/11/10
The 2024 ICPC Asia Hangzhou Regional Contest
ICPC Regional Contests
2024/11/10
CCPC 2024 Jinan Site
CCPC
3024/11/16
The 2024 ICPC Pacific Northwest Regional Contest (Div. 1)
ICPC Regional Contests
2024/1 1/16
The 2024 ICPC Pacific Northwest Regional Contest (Div. 2)
ICPC Regional Contests
2024/11/16
ICPC NA South Division 2024
Division 2
ICPC Regional Contests
2024/11/166
ICPC South Division 2024 Division 1
ICPC Regional Contests
2024/11/16
The 2024 JCPC Southern Califomia Regional Contest
ICPC Regional Contests
2024/11/16
The 2024 ICPC Southeastern Europe Regional Contest (SEERC 2024)
ICPC Regional Contests
2024/11/17
The 2024 ICPC. Asia Shanghai Regional Contest
ICPC Regional Contests
2024/11/1
The 2024 ICPC Asia Scoul Regional Contest
ICPC Regional Contests
2024/1 1/23
The ICPC Northwestern Europe Regional Contest (NWERC 2024)
ICPC Regional Contests
2024//1/24
The 2024 ICPC Asia Shenyang Regional Contest
ICPC Regional Contests
2024/1 1/24
Romanian Master of Informatics 2024 Day
Romanian
202-4/1 1/28
Romanian Muster of Informatics 2024 Duy
Romanian
2024/11/29
The 2024 ICPC Asia Kunming Regional Contest
ICPC Regional Contests
2024/12/1 1/1
Croatian Open Competition in Informatics 2024/2025 Contest #3
Croation Of
2024/12/1
USACO 2024 December Contest (Platinum)
USACO Platinum
2024/1 1/13
The 2024 ICPC Northern Eurasia Finals
ICPC Regional Championntiips/innll
2024//22/15
The 2024 ICPC Central Europe Regional Contest
ICPC Regional Contests
2024/1 2/15
CCPC 2024 Zhengzhon Site
CCPC
2024/1
The 2024 ICPC Asia Yokohuma Regional Contest
ICPC Regional Contests
2024/12/22
The 2024 ICPC Asia Hong Kong Regional Contest
ICPC Regional Contests
2024/1 0024/12/22 1/22
The 2024 ICPC Asia East Continent Final Contest
ICPC Regional ChampionshipsFinals
2024/12/20
USACO 2025 January Contest (Platinum)
USACO Platinum
2025/1/24
Croatian Open Competition in Informatics 2024/2055 Contest 84
Cruatian Ol
2025/1/25
The 24th Japanese Olympiad in Informatics Final Round (JOT 2024/2025)
Japanese Ol
2025/2/2
Croatian Open Competition in Informatics 2024/2025 Contest #5
Croatian Of
2025/2/15
USACO 2025 Fehruary Contest (Platinum)
USACO Platinum
2025/2/21
The 2025 ICPC Europe Championship
ICPC Regional Championships
2025/3/2
2025 ICPC Asia West Finals
ICPC Regional Championshigs/Fin
2025/3/7
The 2025 ICPC Latin America Championship
ICPC Regional Championshipsttinn
2025/3/16
USACO 2025 US Open Contest (Platinum)
USACO Platinum
2025/3/21
Singapore National Olympiad in Informatics 2025 Final Contest
NOI (SG)
2025/3/22
The 2025 British Informatics Olympiad Final
British
2025/4/12
Baltic Olympiad in Informatics 2025 Day
Baltic Of
2025/4/26
Baltic Olympiad in Informatics 2025 Day 2
Baltic Of
2025/4/27
The 2025 ICPC China Zhejiang Province Programming Contest (22nd)
ICPC Regional Contests
2025/5/10
CCPC Final 2024
CCPC Final
2025/5/11
Asia Pacific Informatics Olympiad 2025 (APIO 2025)
APIO
2025/5/17
19

---

Published as a conference paper at ICLR 2026
Competition Name
Category
Date
The 2025 ICPC. Asia Wihhnn Invitational Contest
ICPC Regional Contests
2025/5/17
The 2025 ICPC North America Championship
ICPC Regional Championships/Fnuul
2025/3/26
Table 8: Distribution of failure reasons across evaluated models (% of total errors).
Model
Wrong Answer
Time Limit
Runtime Error
Compile Error
Reasoning Models
04-mini-high
86.0
6.1
0.3
7.6
Gemini-2.5-Pru
76.3
18.1
0.1
5,4
Seed-1.6-thin 1aat1.6-hiinnin--07155
79.1
15.2
0.1
5.6
DeepSeck-R1-0528
77.1
11.1
0.1
11.7
Qwwnm--22BB-A222--nnnnnnnn
81.3
12.3
0.0
6.4
Gemini-2.5-Flash
79.7
11.4
0.1
8.9
GLM-4.5
71.0
10.5
0.0
18.5
Qwen-3-235B-A223
77.8
12.0
0.1
10.1
Qwen-3-32B
77.7
13.8
0.1
8.5
45.8
0.0
2.5
Claude-4-Opun Oppss-iiinnn
48.2
48.3
0.0
3.5
Clande--Sommee-thinninn
50.8
45.8
0.0
3.4
Qwen-3-8B
69.2
9.1
0.1
21.7
Non-Reasoning Models
GPT-4.1
79.3
12.5
0.1
8.1
Kimi-K2 K2
77.0
7.2
0.0
15.7
DeepSeek-V3
82.8
9.2
0.1
7.8
78.9
15.3
0.1
5.8
Claude-4-Sonnct
65.2
30.7
0.0
4.0
GPT-4o
72.1
8.5
0.1
19.3
G EXAMPLE PROBLEMS
Example 1
Source: The 2024 ICPC World Finals Astana
Title: The Silk Road with Robots!
Time limit: 5 seconds
Parts of the ancient silk road passed through southern Kazakhstan. You' ve been fantasizing about a modern silk road, which has its own special features. Along your fantasy road are robots as well as stores holding stashes of tenges (the national currency of Kazakhstan). If robot moves to a location with a store, the robot collects all that store's tenges for you.
The cost of moving robot is tenge for every meter moved, the amount of profit from moving a robot to store is the number of tenges held by the store minus the number of meters the robot has moved to reach the store.
Consider this scenario, which stretches over several days. Initially, the road is empty. with robots stores. Every day, either location along the road. Immediately before that, each existing store on the road restocked with tenges so that its total amount the same as it when was first placed on the road, and each robot is returned to its original starting location.
robot
store is placed
unoccupied
For each day, you need to determine the maximum amount of profit that could be gained by moving robots collect tenges from the stores. Note that no two robots start the same location, but they may occupy the same location as they move. Each store can be emptied of
20

---

Published as a conference paper at ICLR 2026
its tenges only once during a single day.
Input
The first line contains an integer (1 s n = 2- 10²). the number of days. This is followed by n lines, where the i-th line starts with integer 4, which is equal to 1 if new robot is added on day i, or is equal to 2 if new store is added that day.
If 4 1, the line contains another integer robot.
(0 $ 24 = 10"), denoting the location of the new
If t; 2, the line contains another integer 2; (0 2; 10²) denoting the location of the new store, followed by another integer Oj (0 C 10") denoting the number of tenges the store.
All the given locations are distinct.
Output
Output integers, the maximum profit you can make after each day.
# Sample Input:
6
1 20
2 15 15
2 40 50 1 50
2 88 20
2070 30
# Sample Output:
0
10
35
50
50
60
Example 2
Source: The 2024 ICPC Asia East Continent Final Contest
Title: Boolean Function Reconstruction
Input file: standard input
Output file: standard output
Time limit: 4 seconds
Memory limit: 1024 megabyte
Given the truth table of boolean function with n boolean variables as input, construct an expression that satisfies this function. In the expression, you only allowed to the logical and (&) and logical or () operators. Specifically, truth table of a boolean function with boolean variables gives all the 2" outputs corresponding to the possible values of input variables. A boolean expression (expr) has the following forms:
T. F: Represents True and False.
a, b.... z: Represents one of the variables. The i-th variable is represented by the i-th lowercase letter in alphabetical order.
((expr))(exxrr): Represents the logical and operation applied to the results of two expressions.
((expr) (expr)): Represents the logical or operation applied to the results two expressions.
21

---

Published as a conference paper at ICLR 2026
The logical and operation and the logical or operation are defined as two boolean functions below that take two boolean values.
31
22
&C222
T1
32
0
0
1
1
0
0
1
1
Determine whether an expression exists that satisfies the conditions. If such an expression exists, ensure that the number binary operators (& and }) does not exceed 2n + 10, and the depth of parentheses nesting does not exceed 100 layers. It can be proven that if a solution exists, there is always one that meets the constraints of the problem. Input
The input consists of multiple test cases. The first line contains integer T (1 T 2 the number of test cases. For each test case, there are two lines:
The first line contains an integer (1 15).
The second line contains a binary string with length 2", indicating the truth table of the given function.
To interpret the input binary string, suppose the i-th variable has value of I₁. Then, the corresponding function value, [(1 , RT)) the string 8.
is equal to the (Σi-13) 24 2 1 bit of
It is guaranteed that the sum of 2 over all test cases will not exceed 2
Output
For each test case:
Output Yes No the first line to indicate whether expression satisfying the conditions exists.
If an expression exists, output the expression on the second line. The expression must strictly adhere to the format given in the problem description, without adding omitting parentheses, and without adding extra spaces.
Example
standard. input
standard output
Yes
(a&b)
0001
Yes
2
(a/b)
0111
Yes
2
T
1111
Yes
3
((a&(b/c))
00010111
No
Yes
10
a
Yes
0101
(a&(b((c(((&e)))
00000000000000000000000000000000
Note
Below is the truth table interpretation for the fourth sample.
22

---

Published as a conference paper at ICLR 2026
J3
2F2
1 203) 263
0
0
1
0
1000
1
0
1111-
1
23


Reviews:
[{"review_id": "2LZ5Qn88ni", "reviewer": "Reviewer_dXGB", "summary": "This paper introduces AetherCode, a benchmark designed to evaluate LLMs’ coding capabilities in premier programming competitions. The benchmark sources problems from top-tier contests such as IOI and ICPC, and constructs rigorous test cases through a hybrid approach combining an automated Generator–Validator (G-V) agent and professional human experts.", "questions": "1. The paper states that using the CodeForces judging service poses “compliance risks” since crawlers are explicitly prohibited by CodeForces. Could the authors provide an official citation or link to this policy?\n2. The G-V Agent system is briefly described but remains under-specified. Could the authors elaborate on the exact mechanism? Also, how effective is the G-V agent alone, without expert correction?\n3. How does AetherCode handle language diversity? Is it possible to extend the benchmark to multi-language evaluation?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. A harder and more discriminative coding benchmark is very timely and beneficial to the community, especially as existing datasets are approaching saturation for frontier reasoning models.\n2. The inclusion of multiple ICPC/IOI medalists for test case construction is impressive and likely ensures very high evaluation quality. This is the main contribution of the paper from my opinion.", "weaknesses": "1. While the benchmark’s data source (premier contests) is appealing, the distinction from datasets such as LiveCodeBench Pro is not fully convincing. Those benchmarks already cover similar contest problems, so the contribution may appear incremental without a clearer articulation of what new research insights AetherCode enables.\n2. The evaluation section mainly presents aggregate Pass@N scores but lacks qualitative or failure-case analyses. For instance, what types of reasoning or algorithmic errors are most common? The findings seem simply descriptive without such insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1askwLcqfg", "reviewer": "Reviewer_u8So", "summary": "This paper introduces a coding benchmark based on top high-school and college programming contests (mainly IOI and ICPC). The problems are hand annotated for difficulty and categories of problems allowing a more finessed analysis. The test cases are tested against many contest submissions (both correct and incorrect) and additional test cases are generated by hand and automated systems and reviewed by experts. This level of programming contests is the natural next step for programming contest based benchmarks and based on what is stated in the paper a lot of care has gone into it's quality.", "questions": "- I think the IOI is aimed more at high-school students than middle-school L145\n\n- \"The performance of o4-mini-high and\nGemini-2.5-Pro is exceptional, establishing a significant performance gap that places them\nin a tier of their own above other models. Furthermore, they are the only three models capable of\nsuccessfully solving problems at the “Extremely Difficult” level.\" you mention two models and then say they are the only *three* that solve extreme problems. Reword for clarity.\n\n- If the authors have time, I would like to see the performance of Claude 4.5 Sonnet\n\n- It would be nice to mention other efforts to improve benchmark quality and verify answers or test-cases. Two examples are:\nPlatnumBench http://platinum-bench.csail.mit.edu/\nSWE-bench Verified https://openai.com/index/introducing-swe-bench-verified/", "limitations": "", "rating": 10, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper collects a substantial corpus of high-quality programming contest problems and formats them as a benchmark.\n- The benchmark seems well annotated to allow for more detailed study.\n- According to the paper much care was taken to ensure problem and especially test-case quality.", "weaknesses": "- The paper fails to detail where the example solutions came from and whether the creators of the problems or sample solutions consent to their work being used as an ML benchmark.\n- It would be useful to know the difficulty breakdown in each category for assessing whether e.g. the models are really bad at Tree type problems or if the small number of tree problems all happen to be exceptionally difficult.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KNgT91MOlz", "reviewer": "Reviewer_Lxkp", "summary": "## Paper Summary\n\nThis paper presents AetherCode, a new benchmark designed to thoroughly evaluate the reasoning and programming capabilities of large language models (LLMs) using problems sourced exclusively from premier competitive programming contests such as IOI and ICPC. The authors identify two major deficiencies in existing benchmarks: (1) they lack sufficient difficulty and problem diversity, often consisting of relatively simple tasks (e.g., LeetCode-style problems), and (2) their evaluation suites frequently contain weak or incorrect test cases, resulting in inflated model performance. AetherCode addresses these issues by offering: (1) 456 carefully curated problems spanning multiple difficulty levels and algorithmic domains, (2) high-quality and rigorously validated test cases constructed through a Generator–Validator agent system augmented by expert audits, and (3) a comprehensive evaluation of 17 leading reasoning and non-reasoning LLMs, revealing a significant performance gap. The results clearly demonstrate that even top-tier LLMs can solve only a small fraction of the benchmark, confirming that substantial progress is still needed before LLMs can perform at the level of world-class human competitors.", "questions": "1. How is the \"difficult\" determined in Table 1?\n\n2. Do you have insights to scale the problem generation?\n\n3. What is the coverage of the test cases?\n\n4. Any ablation study of the effectiveness of the agent-based test case generation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "## Strengths\n\n1. Benchmark novelty and rigor. AetherCode is the first benchmark to systematically incorporate premier programming contest tasks, establishing a far more challenging and realistic evaluation setting.\n\n2. Strong evidence of current reasoning limitations in LLMs. The steep decline in Pass@1 performance as problem difficulty increases highlights persistent weaknesses in deep algorithmic reasoning.\n\n3. Clear and well-structured presentation. The paper is clearly written, logically organized, and easy to follow. The motivation, methodology, and findings are presented coherently, making the work accessible even for those not deeply familiar with competitive programming-style evaluation.", "weaknesses": "## Weaknesses\n\n[1] Limited dataset scale. While the problems are high-quality, a total of 456 is still smaller than some widely adopted benchmarks, potentially limiting coverage in long-tail algorithmic domains.\n\n[2] Clarification needed on contamination guarantees. Although timestamps are included, it remains unclear whether contamination checks confirm that evaluated LLMs were not trained on these problems prior to their incorporation into the dataset. Stronger contamination analysis or preventive measures would improve confidence in the benchmark’s integrity.\n\n[3] Difficulty labeling procedure requires further explanation. The paper states that difficulty categories are based on human competition outcomes, but it would be helpful to describe the exact criteria or thresholds used, particularly for distinguishing “Hard” versus “Extreme.”\n\n[4] Limited reporting on coverage of test cases. While the benchmark claims 100% TPR/TNR on a collected set of human solutions, metrics such as input space coverage or alignment with typical competitive programming corner-case patterns would further validate robustness.\n\n[5] Missing related work. The paper would benefit from acknowledging and comparing against recent benchmarks that also focus on code LLMs.\n\n1. Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination (ICML 2025)\n\n2. DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generation (ACL 2025 Finding)\n\n3. Codereval: A benchmark of pragmatic code generation with generative pre-trained models (ICSE 2024)\n\n4. EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories (Neurips 2025)\n\n5. Evaluating and Improving LLM-based Competitive Program Generation\n\n6. PPM: Automated Generation of Diverse Programming Problems for Benchmarking Code Generation Models (FSE 2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CUVYkzgn1P", "reviewer": "Reviewer_eg8w", "summary": "This paper introduces AetherCode, a new benchmark for evaluating the code reasoning capabilities of Large Language Models (LLMs). The authors argue that existing benchmarks are insufficient due to limited difficulty and scope, as well as evaluation biases from low-quality test cases. To address this, AetherCode is constructed using problems from premier programming competitions like the International Olympiad in Informatics (OI) and the International Collegiate Programming Contest (ICPC).\n\nSpecifically, the authors collect 456 challenging problems from world-class competitions, processed into a unified Markdown+LaTeX format. A novel hybrid approach for test case generation is proposed, combining an automated Generator-Validator (G-V) agent with intensive human expert annotation. The quality is assessed using a True Positive Rate (TPR) and True Negative Rate (TNR) metric against a large set of human solutions, reportedly achieving 100% on both.", "questions": "I wonder why the conclusion claimed by this paper is against the breaking news, like OpenAI and Google Gemini achieving golden medals in IOI competitions? What is the difference between AetherCode and the newest IOI contest?\n\nhttps://www.reddit.com/r/MachineLearning/comments/1mnpqu7/n_openai_delivers_goldmedal_performance_at_the/", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper's core strength lies in sourcing problems from truly top-tier competitions (IOI, ICPC World Finals, etc.). This is a commendable step up in difficulty and complexity compared to benchmarks sourced primarily from online judges like LeetCode or CodeForces.\n2. The most significant contribution is the meticulous attention paid to test case quality. The proposed methodology of evaluating the entire test suite as a binary classifier using TPR and TNR on a large corpus of human submissions is a novel and valuable idea for the community. Achieving 100% TPR/TNR on their collected set is an impressive feat of engineering.\n3. The detailed curation pipeline, from PDF conversion to multi-dimensional expert categorization (difficulty, algorithm tags), is thorough and sets a high standard for future benchmark development (Figure 1)", "weaknesses": "1. The paper claims AetherCode is the first benchmark to systematically collect problems from premier programming competitions worldwide. This claim is overstated. The paper itself cites several recent works with similar goals, such as USACO Bench, OJBench, and ICPC-Eval, but fails to adequately differentiate itself. Merging the data sources from several existing benchmarks and stating the comprehensiveness should not be a significant contribution to me. A more nuanced discussion is needed to pinpoint AetherCode's unique value proposition beyond just scale. \n\n2. While the TPR/TNR metric is a great idea, the claim of achieving a 100% True Negative Rate (TNR) is only as strong as the set of incorrect solutions it was tested against.\n- 2.1 The paper provides little detail on the nature of the 30,000+ incorrect human submissions. Do they include solutions with subtle, hard-to-find bugs (e.g., off-by-one errors, mishandled edge cases, time-limit-exceeded on specific inputs), or are they mostly solutions that fail on simple cases?\n- 2.2 A 100% TNR on a potentially limited set of known failure modes does not guarantee robustness against novel incorrect logic that LLMs might produce. The claim, while technically true for their specific dataset, may imply a higher level of generalizability than is warranted.\n\n3. The evaluation in Section 3 is convincing and insightful. Beyond showing that stronger models perform better, the paper offers little insight into why they succeed or fail.\n- 3.1 The performance breakdown by algorithm category (Table 4) is a good start, but the analysis is shallow. For instance, why do models struggle with \"Computational Geometry\" or \"Problems on Trees\"?  What specific reasoning patterns are they failing to capture?\n- 3.2 The paper could have included a qualitative analysis of model failures. Are the errors syntactic, logical, algorithmic (e.g., choosing a brute-force approach instead of dynamic programming), or implementation-related? This would provide far more value to researchers looking to improve these models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)