ICML 2025

Leveraging Online Olympiad-Level Math Problems for LLMs
Training and Contamination-Resistant Evaluation
Sadegh Mahdavi * 1 Muchen Li * 1 Kaiwen Liu 1 Christos Thrampoulidis 1 Leonid Sigal 1 2 3 4 Renjie Liao 1 2 3
Abstract
Advances in Large Language Models (LLMs)
have sparked interest in their ability to solve
Olympiad-level math problems. However, the
training and evaluation of these models are con-
strained by the limited size and quality of avail-
able datasets, as creating large-scale data for
such advanced problems requires extensive ef-
fort from human experts. In addition, current
benchmarks are prone to contamination, lead-
ing to unreliable evaluations. In this paper, we
present an automated pipeline that leverages the
rich resources of the Art of Problem Solving
(AoPS) forum, which predominantly features
Olympiad-level problems and community-driven
solutions.
Using open-source LLMs, we de-
velop a method to extract question-answer pairs
from the forum, resulting in AoPS-Instruct, a
dataset of more than 600,000 high-quality QA
pairs.
Our experiments demonstrate that fine-
tuning LLMs on AoPS-Instruct improves their
reasoning abilities across various benchmarks.
Moreover, we build an automatic pipeline that
introduces LiveAoPSBench, an evolving evalua-
tion set with timestamps, derived from the latest
forum data, providing a contamination-resistant
benchmark for assessing LLM performance. No-
tably, we observe a significant decline in LLM
performance over time, suggesting their success
on older examples may stem from pre-training
exposure rather than true reasoning ability. Our
work presents a scalable approach to creating
and maintaining large-scale, high-quality datasets
for advanced math reasoning, offering valuable
insights into the capabilities and limitations of
* Equal Contribution, order decided by a coin flip 1University
of British Columbia, Vancouver, Canada 2Vector Institute for AI,
Toronto, Canada 3Canada CIFAR AI Chair 4NSERC CRC Chair.
Correspo...