AetherCode: Evaluating LLMs' Ability to Win In Premier Programming
  Competitions
Authors: name: Zihan Wang; name: Jiaze Chen; name: Zhicheng Liu; name: Markus Mak; name: Yidi Du; name: Geonsik Moon; name: Luoqi Xu; name: Aaron Tua; name: Kunshuo Peng; name: Jiayi Lu; name: Mingfei Xia; name: Boqian Zou; name: Chenyang Ran; name: Guang Tian; name: Shoutai Zhu; name: Yeheng Duan; name: Zhenghui Kang; name: Zhenxing Lin; name: Shangshu Li; name: Qiang Luo; name: Qingshen Long; name: Zhiyong Chen; name: Yihan Xiao; name: Yurong Wu; name: Daoguang Zan; name: Yuyi Fu; name: Mingxuan Wang; name: Ming Ding
Published: 2025-08-22 14:04:55

Abstract: Competitive programming has emerged as a critical benchmark for evaluating
the reasoning and coding capabilities of Large Language Models (LLMs). Despite
impressive progress on existing benchmarks, we argue that current evaluations
overstate model proficiency, masking a substantial gap between LLMs and elite
human programmers. This gap arises from two key limitations: insufficient
difficulty and scope of benchmark problems, and evaluation bias from
low-quality test cases. To address these shortcomings, we present AetherCode, a
new benchmark that draws problems from premier programming competitions such as
IOI and ICPC, offering broader coverage and higher difficulty. AetherCode
further incorporates comprehensive, expert-validated test suites built through
a hybrid of automated generation and human curation, ensuring rigorous and
reliable assessment. By combining challenging problem design with robust
evaluation, AetherCode provides a more faithful measure of LLM capabilities and
sets a new standard for future research in code reasoning.