MathNet: A Global Multimodal Benchmark for Mathematical Reasoning and Retrieval
Authors: Shaden Alshammari
Kevin Wen
Abrar Zainal
Mark Hamilton
Navid Safaei
Sultan Albarakati
William T. Freeman
Antonio Torralba
Venue: ICLR 2026 Poster (2026)
Keywords: Mathematical retrieval
Mathematical comprehension
Large language models
Primary area: datasets and benchmarks
TL;DR: A large-scale, multimodal, multilingual dataset of math problems for evaluating LLMs on equivalence retrieval and reasoning

Abstract: Mathematical problem solving remains a challenging test of reasoning for large language and multimodal models, yet existing benchmarks are limited in size, language coverage, and task diversity. We introduce **MathNet**, a high-quality, large-scale, multimodal, and multilingual dataset of Olympiad-level math problems together with a benchmark for evaluating mathematical reasoning in generative models and mathematical retrieval in embedding-based systems. **MathNet** spans 47 countries, 16 languages, and two decades of competitions, comprising **30,676 expert-authored problems with solutions** across diverse domains. In addition to the core dataset, we construct a retrieval benchmark consisting of mathematically equivalent and structurally similar problem pairs curated by human experts.

**MathNet** supports three tasks: (i) mathematical problem solving, (ii) problem retrieval, and (iii) retrieval-augmented problem solving (math RAG). Experimental results show that even state-of-the-art reasoning models (**78.4% for `Gemini-3.1-Pro` and 69.3% for `GPT-5`**) remain challenged, while embedding models struggle to retrieve equivalent problems. We further show that RAG performance is highly sensitive to retrieval quality; for example, `DeepSeek-V3.2-Speciale` achieves gains of up to **12%**, obtaining the highest scores on the benchmark. **MathNet** provides the largest high-quality Olympiad dataset together with the first benchmark for evaluating mathematical problem retrieval, and we publicly release both the dataset and benchmark at [https://mathnet.mit.edu](https://mathnet.csail.mit.edu).

Published as a conference paper at ICLR 2026
MATHNET: A GLOBAL MULTIMODAL BENCHMARK FOR MATHEMATICAL REASONING AND RETRIEVAL
Shaden Alshammari" Kevin Wen
Abrar Zainal" Mark Hamilton
Navid Safaei
Sultan Albarakati
William T. Freeman't Antonio Torralba
MIT
KAUST
HUMAIN
Individual Rescarcher
* Equal Contribution
Website: mathnet.mit.edu mit.edu
Dataset: shadena/mathnel
Large Scale Multilingual Data
High-Quality Solutions
Extensive Benchmarks
Detailed Multimodal Solutions written by Human Experts
Three Evalustion Tasks
Olympied
MathNer Net 30,676
1) Math Comprehension
Problems
PMctarch
Problem Retrieval
Clyy MA4te
1,40
Devi
Cumppedd 6,14
Tarus Proclum
Mathurattaaly
Probiom
Corpus
40- Countries over Decades es
MathNet
Math RAG
Turget Pooharm
Fallerant Protilem
Model
Diverse Mathematical Topics
Probiam
Corpus
Stlution to Pallezant Photilem
Sounion
LLMs, MLLMss, and Embedding Models Leaderboard
Languages
Fnglish st3t, Strrugale, Italano, Paparinl franpiis Polski, Doutsoh, Rondan, 9-+9) Nodorlands, Pyccrse Morrus, Siovaniiia, Masspowee,, Oprizce Magaer
Ontology of Problem Types
Figure 1: Overview of MathNet: dataset of 30K+ Olympiad-level problems across 47 countries, 17 languages, and 143 competitions over 40 years with expert-authored solutions. We evaluate several lcading LLMs on both solution generation and math aware retricval.
ABSTRACT
Mathematical problem solving remains challenging test of reasoning for large language and multimodal models, yet existing benchmarks are limited size, language coverage; and task diversity. We introduce MATHNET, a high-quality, large-scale, multimodal, and multilingual dataset of Olympiad- level math problems together with a benchmark for evaluating mathematical reasoning in generative models and mat mathematical retrieval in embedding- based systems. MATHNET spans 47 countries, 17 languages, and two decades of competitions, comprising 30,676 expert-authored problems with solutions across diverse domains. In addition to the core dataset, we construct retrieval benchmark consisting of mathematically equivalent and structurally similar problem pairs curated by human experts.
МAтнNЕт supports three tasks: (i) mathematical problem solving, (ii) problem retrieval, and (iii)
auggeettee problem solving (math RAG). Experimental results show that even state-of-the-art reasoning models (78.4% for Gemini-3.8 1-Pro and 69.3% for GPT-5) remain challenged, while embedding models struggle to retrieve equivalent problems. We further show that RAG performance is highly sensitive to retrieval quality; for example, DeepSeek-V3. 2--Secciale achieves gains of up 12%, obtaining the highest scores the benchmark. MATHNET provides the largest high- quality Olympiad dataset together with the first benchmark for evaluating mathematical problem retrieval, and we publicly release both the dataset and benchmark at https://mnn https://mattet..... mit.edu.

---

Published as a conference paper at ICLR 2026
1 INTRODUCTION
Recent large language models (LLMs) and large multimodal models (LMMs) have made rapid improvements on mathematical reasoning benchmarks, from grade-school problems to competition mathematics (Cobbe et al., 2021; Hendrycks et al., 2021b; Achiam et al., 2023). Recently, public reports claimed unprecedented gold-medal-level performance at the International Mathematical Olympiad (IMO) by advanced multiple models (Luong et al., 2025; Shao et al., 2025). Moreover, there have been multiple incidents of AI systems reportedly solving open mathematical problems (Nie et al., 2025; Feldman & Karbasi, 2025).
Despite these advances, the lack of open, high quality, and diverse benchmarks constrains research progress, Existing Olympiad-level datasets are typically drawn from community platforms such as AoPS and cover only handful of competitions in the U.S and China (see Table 1). To address this gap, we present MATHNET: a large-scale, multimodal, and multilingual dataset of Olympiad-level mathematics problems together with benchmark for evaluating mathematical reasoning and retrieval, sourced from 47 countries across four decades. MATUNET contains 30K+ problems with official solutions written by experts across a wide range of mathematical domains. Its scale, diversity, and expert quality provide an unprecedented foundation for exploring mathematical generalization and analogical reasoning.
We use the MATHNET dataset to evaluate two model classes on two main areas: math comprehension, or the ability to solve mathematical problems, math retrival, or the ability to recognize and retrieve "mathematically" equivalent or related problems. Unlike existing semantic retricval (Izacard et al., 2021; Khattab & Zaharia, 2020; Formal al., 2021), our problem retrieval lask must be aware of symbolic structure, invariances, and transformations. For example, the problem of solving x² + y² = 1 is equivalent to v + 6² = 1, and is also equivalent to the set of 2D vectors with unit norm |u| 2 1. Crucially, these are not equivalent to solving + y 1. Current retrieval models fail to make this distinction: due to superficial lexical overlap (Das et al., 2025), they often rank a problem containing x + y = 1 as closer to x² + y²² 1 than to the truly equivalent formulations. Despite the foundational importance of retrioval, we note that this task remains largely uncxplored in recent literature.
These challenges arise even in expert workflows such as the annual IMO exam selection process. During shortlist construction, problems may sometimes resemble problems that already exist in books, problem collections, online sources, illustrating how difficult it can be to recognize mathematical equivalence across different notations, formats, and languages. Similar issues arise in mathematical research. For example, a mathematician studying bounds on gaps between consecutive primes may search for phrases such as "upper bounds on prime gaps" rather than a specific formula like Pn+1 Pn < C(log p,.² (where Pn. is the n-th prime and C' is a constant). However, existing retrieval systems are often sensitive to superficial features such as variable naming or textual phrasing, making it difficult to connect mataematically equivalent statements expressed in different forms.
To make progress on these challenges, we introduce MATHNET, a large benchmark of mathematics problems coupled with evaluations of math reasoning in addition to math-aware retrieval and its downstream impact on mathematical reasoning. Our contributions are: 1. MathNet-Core A 30K-problem corpus of Olympiad-style math with aligned LaTeX and natural-language statements, expert solutions, and metadata. MathNet-Con spans 47 countries, 17 languages, and 65+ mathematical domains.
2. Datascts for Retricval Tasks. Mathliet-Retrieve, a retricval dataset comprised of 40K additional synthetic problems, derived from 10K anchor problems each paired with 1 equivalent positive and 3 hard negatives. In addition to MathNet-RAG, 70 IMO level expert-curated structurally similar problems from real competitions.
3. Benchmark Evaluation. Benchmarking across 27 statc-of-tho-art models on three primary tasks: MathNet-Solve, Mathlett-Rttriee, and MathNet--AG, using both auto- matic grading and human expert grading.
4. Analysis:
Solving vs. Retrieving. We demonstrate a sharp divergence between problem solving and retrieval: even state-of-the-art models struggle with mathematical retrieval. Moreover, retrieval-augmente generation (RAG) improves reasoning only when retrievers surface structure-aligned, mathematically relevant neighbors.
2

---

Published as a conference paper at ICLR 2026
Deschanark
Size
Langtages
Draluaton Type
Source
Difiicalty
GSM& Cobbe es al. (2021)
5,500
EN
Numeric Answer
Crondooured probiems
Grade School
MATH Hendryck et al, 12021b)
12,500
EN
Numeric Answer
Cocsettitons texthooks
High School
MATU Video Wing al. (2024)
3,040
EN
Expression Proof
Math Cocspetitions
[lgh School
CMMLU Li al. (209ts)
11,528
ZI1
MQQ
Chincen exom materials
Figh School College
MMLU Henddyddas et al. (2021a)
15,008 EN
MOQ
College profossional
College Level
C-Eoal Hunug et al. (2023)
13,31s
ZH
XOQQ
Chinese colloge camms
Cullege Eutraace
MMMU Yie (2024)
11,500 EN
MOQ/
Expecsiion
Multimocal academic
College Level
AGIlol Zhong et al. (2024)
3,300
EN ZH
MOQQ
Expressiaa
Collage calranex esuss
College Entraance
[[EDDroch Aroca al. (2025)
515
EN
MOQ Numette Anowe
Indian JLE Advanced
JEC Advaneed Exama
ObyppiddDecc 11e ct al. (2024)
6,142
EN ZII
Proof Expeeséoon
Official Webettes
Olympiod Leve]
OvappicAreu Hunne al. (2024)
3,233
EN ZH
Proof Proces
Official Websttes
Olyuplad Level
Onni Math Gao al. (2024b)
4.428
EN
Proof Process
Aol'S Forum Contest Pagee
Olympiod Level
LsegMath Shong
(2525)
1,552
EN
Proo| Amalytioal Tools
Ourated Iusqualities Problcus
Olympled Level
OlymMATII Sun es al. (2025)
200 EN ZII
Numoric Answer
AoI'S Forum,/Otiin Offirial Websttes
Olymplod Level
LiveAoPS Mlalidadi al. (2ua5)
EN
Numerie/ Espression
Aod'S Forum (rolinng snapshot)
Olymplad Level
MathArent Balnmonté et al. (2025)
162
EN
Tiza Ansmer / Proof
Newly releazed competitions
Olympiad Level
DMO1dench Loong al. (2n25)
EN
Numerie: Proot
IMO national archives
Olyupled Level
MathNet (ours)
EN, XII. ES
Official Country Booklets International and National
30,767
RU, FR, BO
Expression Proof
Obympiod Level
and 11 other
Coztests
Table 1: Comparison of mathematical reasoning benchmarks across different sizes, languages, evaluation types, and difficulty levels. We include both unimodal and multimodal datasets, spanning grade-school to Olympiad-level mathematics. Our Proposed MATINET expands coverage to 17 languages and focuses on proof- and process-base evaluation.
RELATED WORK
Mathematical problem solving has long been a core benchmark for evaluating Al reasoning capabilitics. Early efforts focused on text- based arithmetic problems, while recent research has expanded to competition-level reasoning, theorem proving, and multimodal problem- solving. Existing datasets can be broadly categorized into lext-only benchmarks, multimodal benchmarks, and large-scale aggregates.
Text-Only Mathematical Benchmarks. Several datasets evaluate LLMs' mathematical reasoning using text-only problems. Cobbe et al. (2021) introduced GSM8K, grade-school level problems for elementary arithmetic reasoning. Hendrycks et al. (2021b) proposed MATH, which consists of problems spanning high school to competitive mathematies. Gao et al. (2024b) presented Omni-MATH, with 4,428 Olympiad-level problems. He et al. (2024) and Wang et al. (2024) further extend coverage with bilingual and competition-level datasets, though most are limited in scale, language diversity, or structured similarity annotations.
Multimodal Mathematical Benchmarks. Multimodal benchmarks integrate visual information with textual descriptions, primarily for geometry or diagram-basod reasoning. Datasets such as MATH-Vision (Wang al., 2024) and MathVista (Lu al., 2024) incorporate broad visual contexts, including charts and diagrams. Despite this added modality, these datasets remain comparatively easy and do not capture the full difficulty of Olympiad-level problem solving.
Large-Scale Aggregates. Large datasets aggregate problems from multiple sources such as NuminaMath (Li et al., 2024b) and (Albalak et al., 2025). Although valuable for large- scale training and evaluation, these datasots typically lack curated multimodal content, multi-lingual coverage, and fine-grained annotations.
Math-Aware Retrieval. There has been work on formula-aware indexing (Zanibbi et al., 2025; Das et al., 2025), but such systems predate LLMs and typically operate at the formula. level, missing broader conceptual and structural similarities expressed natural language. Meanwhile, modern IR excels at semantic paraphrase but is often blind to symbolic equivalence and cross-modal cues.
Limitations of Prior Work and Motivation for MATINET. Despite these advances, current benchmarks exhibit three main limitations: (i) limited expert solutions, (ii) lack of visual multilingual content, especially for high-diffcully problems, and (tii) limited analysis of mathematical problem retrieval. MATHNET addresses these gaps by offering a large-scale, mult multilingual, multimodal dataset of Olympiad-level problems. It includes expert-validated
3

---

Published as a conference paper at ICLR 2026
problem pairs and fine-grained taxonomy of mathematical similarity, enabling rigorous study of analogical problem solving in generative models, retrieval quality in embedding based systems, and retzieval-anngmenteed reasoning across languages and modalities.
3 DATASET AND BENCHMARK DESIGN
MATHNET consists of Olympiad-level problems logether with a benchmark of three lasks that evaluates mathematical reasoning in generative models, mathematical retrieval in embedding-baal based systems, and analogical reasoning with retrieval augmented generation.
3.1 DATASET SPLITS AND TASKS
MATUNET comprises core dataset, MathNet-Core, and three data-splits MathNet-Solve. Mathliet-Retrieve and MathNet-RAG. Each split has a corresponding task (see Section 4.1).
MathNet-Core: contains 30,676 expert-written Olympiad problems with solutions, span- ning 47 countries, 17 languages, and 143 competitions; this corpus serves as the source collection for all benchmark tasks. Figure 2 summarizes main statistics of the dataset.
2. MathNet-Soovve a 5,500-problem test subset MathNet- Core used to measure mathe- matical problem solving via generated solutions graded against reference solutions.
3. Mathhlet-Retrieve: retrieval benchmark built. from 9,000 anchor problems from MathNet-Core. For each anchor, we construct 1 equivalent positive and 3 hard negatives, yielding 36,000 derived problems in total; it used to evaluate embedding models on retrieving mathematically equivalent problems.
4. MathNet-RAG: a benchmark of 35 anchor problems and 35 expert-paired real prob- lems all drawn entirely from MathNet-Core; used to evaluate retrieval-augmeeneed mathematical problem solving under both retrieved and oracle contexts.
a) Problems by Type Contest
d) Problem Topics and Sub-Topics Distribution
Natiral Conpetiine
Angle Charzz
Algstraic NT NT Faadions
Trikagess
Quadelatniss
Besiees eai Plice Roots
30 066666
Chida
Dlephurtin Laver Corn, Mid,
Analytc Melasla
Chease Canes Dis
PLPS
Plane
Prine Narbhors
Cerr Inc,
Divibbbly
Number Theery 20%
"iachnagaor
b) Solution Length Distribution
Carrgg
Geamelry
Concurcecy
Carb. Gars
Eistemale
Sold Goammryy
Invattats
Abstoot Alghhoo
Couning
Obyrptattecch (2ed
Liner Alpehn
Two Waye
Coust
Figocbele Prixsiple
Corptokeraattn (eolumnntt
Bijertio
IMD Beals tondin
Equaions and long
Alpehnic Expe
Esareiakas Syssadtl
Sugarces Series
Dinorrs Confficiem
Theary
Pelyouriiee
Algortbene
200
160
600
100
1000
Equatioes
Lagic
Solutios longth (wond))
c) Problems per Year
c) Language Distribution (Hnglish 74%, None English 26%)
Nunber buttem sole!
Lo00-
Problers
Non-Euglish Languages Breakdown (7775 Problems)
500-
Romance
Slavii
Germanic
CJK
Other
10-
1665
1950
1565
2000
2006
2010
2015
2000
Figure Overview of MathNet-Core Corpus. The dataset. spans problems from national, regional, TST, and international competitions, with solution lengths varying substantially across repositories. The corpus has grown significantly since the early 2000s and includes both textual and diagram-based problems. The dataset also exhibits broad multilingual and wide topics coverage.
4

---

Published as a conference paper at ICLR 2026
Add Text Butters
Froblomm
Extraction with DoteEEE
Decument
Sturdia
Format Normaltzntion GPT4,1
Semantic Metadae Est rection
PDF
Inpus Booklata
Eduacted
Prostem Solution
Structural Parsing
Boundary Dafection
Problem Soluuion
Human Vertication
Abggrmrnc
Source Consictency Chack (Rula Bassdl)
Gursted Datanal
Variication Dedupfiction
Screenshot
Figure 3: MATHNET's Problem-Solee Solution Extraction and Verification Pipeline.
3.2 DATA COLLECTION, EXTRACTION AND ANNOTATION
Data sources. Each year, participating IMO countries contribute original problems for use in their national contests and team selection examinations. To construct the dataset underlying the benchmark, we collect official problem booklets from 47 countrics spanning 1985-2025 comprising 1595 PDF volumes and more than 25,000 pages in total. Unlike prior math benchmarks that often rely on community-sourced platforms such as AoPS, MathNet is built exclusively from officially published national materials. All included problems and solutions are authored and disseminated by national teams themselves, ensuring expert-level quality, consistency in style, and immunity from noisy or informal annotations. For more details, refer to section A.
Problems Extraction. We first convert all contest booklets into Markdown format using the multilingual document parsing framework dots-ocr Li et al. (2025). This step establishes uniform input format for downstream processing. The underlying source material spans wide range of formats including both digital typeset documents and scanned copies, with some documents spanning several languages. We leverage the multilingual recognition and layout analysis capabilities of dots-ocr to robustly handle this variation and ensure consistent and faithful text extraction across diverse document types.
Problem-Solutien Matching and Annotation In general, it is difficult problem to extract aligned problem-solution pairs from large corpora of heterogencous mathematical documents. In dataset, some documents present problems and solutions in separate sections, while others interleave them. Numbering schemes and naming conventions vary not only across countries but often within single document. These inconsistencies render traditional parsing techniques (e.g., regex-based heuristios) brittle and difficult to scale. To address this, we designed novel LLM-based pipeline for problem solution alignment (illustrated in Figure 3). Our approach operates in three stages:
Stage 1: Document Ingestion and Problem Segmentation. To process each contest booklet, we first extract section-level level segments, then pass these pages to 1.iinni2.5-FF1.. to identify problem and solution corresponding segments by outputting only their line numbers, We also record the problem authors, hints and remarks (if present), as well as the and source file and page number to maintain provenance information for each problem.
Stage 2: Problem and Solution Extraction. Given the line segments from Stage extract the text within these segments, along with additional buffer text before and after them case any content missing. We then use the prompt GPT-4. extract the corresponding problem and solution in a LMTEX- ffiiendildll format. This strategy addresses the issue of problems and solutions appearing in different sections of the booklet, which may exceed the extraction model's context window.
Stage 3: Extraction Verification. We validate the extracted problem-solution pairs through threc-stage verification process of (1) rule-based analytical checker, (2) GPT-4. and (3) human annotators. First, we compute text similarity between the extracted content and the original OCR output to ensure that the LLM editing is restricted to formatting
5

---

Published as a conference paper at ICLR 2026
adjustments and introduces no new hallucinated content. Second, we prompt GPT-4. to compare screenshots of the source pages with the extracted problem-solute solution pairs, acting as judge (see Appendix G) to detect OCR errors, incorrect figure associations, or mismatches between problems and solutions, and to determine whether the solution is complete. Finally, manually review cases with low confidence scores. A problem-solttion only if all three verification mechanisms reach unanimous agreement.
solution pair is retained
3.3
MATH-AWARE RETRIVAL
3.3.1 UNDERSTANDING DIFFERENT TYPES OF MATHEMATICAL SIMILARFTY
Mathematical progress often depends on recognizing when different problems share common structure. These structural commonalities can lake several forms, from strict equivalence to looser thematic connections. We distinguish three modes of similarity: Invoriance, Resonance, and Affinity (see Table 2).
Invariance refers to strict equivalence under transformation. Two problems are invariant when they differ only in representation but share the same underlying structure. Examples include syntactic renaming, algebraic reformulation, geometric re-charactorization, or cross- domain isomorphism.
Resonance refers to partial similarity. Problems are not identical, but they can be addressed using the same idea, proof strategy, or structural analogy. Resonance highlights opportunities to transfer tools or insights across contexts.
A Affinity refers to broad sense of relatedness without structural equivalence. Problems may belong to the same conceptual or disciplinary area (e.g., number theory, geometry) even if they do not share a method or solution strategy. Affinity provides way group problems by theme, context, or historical development.
Mode
Problem A
Problem
Invarlancs
Synsactic Eeuivalence
Pund FRR 3 such that f(x? - =²) - (e X00 flrl). o Prove Yosi
Find gi B R suth that (glo) scbiff 6) - PP).
Resemudation
o, Prove 2o ataaieeee 2
Taunforsatinal
Fial sudi
Find Rouch that (2/3)" (3/2)"
Structurol Resonance
Grneraioation
For 2 I, prove that A disiios (2) for all
Show that (2) [II (2)) (mod wher Engl in Emp!
Common Lemma
If + ifo? show that steal is a perfect square. seve
If + 87 + Roob i en), show that & (1,2,)).
Stroctund Reduction
Prove that 4
number.
Prove that 22t 2" i divisble
Affiaity (Thematie)
Show that lurgest prine
(2e) is groater than 2
every a 1, there primee such that a
Table 2: Taxonomy of mathematical similarity with Olympiad-style examples. Invariance captures strict equivalence under reformulation, Structural Resonance reflects shared lemmas reductions, and Affinity denotes looser thematic clustering.
3.3.2 BUILDING MATHNET-RETRIEWS
To assess the abilities of current embedding models, we designed a problem retrieval task that aims to distinguish between surface-level lexical overlap and deep mathematical equivalence, MathNeet-etrieee
built from 10,000 anchor problems drawn from MathNet- Core. For each anchor, we use GPT-5 (prompt details in the Appendix) to generate exactly 1 cquivalent positive and hard negatives, yielding 40,000 syuthetically derived problems in total.
Equivalent Positives. For each anchor problem, generate one mat hetmeiaailll equivalent variant via variable renaming (e.g., x - a), algebraic manipulation, and paraphrasing using GPT-5 (prompt details in the Appendix). For example, the functional equation f(x) + f1) = f(r + y) can be rewritten as an algebraically equivalent variant such as g(a) g(a + b) -g()..
Hard Negatives. For each anchor, we also generate three adversarial hard negatives that preserve much of the surface form while changing the underlying mathematics (e.g., f(*2)+ + f(y) = f(x poee-y),. v)). These near-miss distractors make it difficult for models to succeed by relying only lexical similarity.

---

Published as a conference paper at ICLR 2026
3.3.3 BUILDING MATHNET-RAG
We also construct MathNet-RAG as a non-synthetic benchmark expert-paired real Olympiad problems. The dataset is organized 35 anchors paired with 35 expert-curated pairs, 70 problems total drawn from real Olympiad competitions. Each pair lies in the Struc- bural Resonance category of our laxonomy and exhibits similarities such Genernliaaion, Common Lemma, or Structural Reduction. These pairs capture a human expert notion of similarity that goes substantially beyond simple algebraic transformation.
Example Conceptual Problem Pair from MathNct-RAG
Problem A. [Chinese TST 2014] Show that there are no 2-Luples (æ. y) of positive integers salisfying
1x 2) -). 2014) (y 1)(x +
4028).
Problem B. [Russia 2009] lireza multiplied a billion consecutive natural numbers, and Matin multiplied two million consecutive natural numbers. Prove that these two got different results one of them has made a mistake.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
We evaluate models on MATHNET under three benchmarks: (a) Math Comprchension, (b) Math Retrieval, and (c) Math RAG on the corresponding data splits: Mathlet-Sollve, Mathlett-Retrieve, and MathNet-RAG.
For Math Comprehension, evaluate two types of models: (i) LLMs and LMMs, including got-4o, that accept images, provide both the text and image as input; otherwise, supply text- only description of the image. (ii) LLMs and LMMs with deliberate reasoning, inchuding gemini-3.1-pro-prees prero-previee, gemini-3-flasie flw.las---reeiee.. gemini-2.. --5--fshh. claude-opus- 4. 6, 5 gpt-5-mini, gpt-5-nano, and DeepSeek-R1.
and Grok-3. For models
For Math Retrieval, we evaluate retrieval performance using embeddings derived from a diverse set of state-of-the-art models, including all-mpnet-base-12. multi-qa-mpnet- base-dot-vi, cohere- embedding-ada-O0 text eettt--eeeddn---nieee and text- mext-embedding---leee pute similarities between problem statements using cosine similarity over the embedding representations.
gxen3-embedding-22 gemini gemini-embedddig--01.. text-
We com-
For Math RAG, we limit the evaluaitons seven state-of-the-art open-source and propri- etary models, as this benchmark requires human grading. The models are gemini-3-pel preview, gpt-5, claude-opus-4.5 DeepSeek-V3. V..eeek-v3.--Secilleee 2-ppecilee, oLMO-3-Think, Grok-4.1 Fast, and Phi Phi-4-reasoning-Po -plu....
4.2 EVALUATION PROTOCOL
Math Comprehension. Similar to the protocol introduced in IMO-Bench (Luong et al., 2025), we adopt score based model grading procedure using GPT-5. For each problem, the judge model is provided with the problem statement, the reference solution, and the model-generated solution, and asked to assess whether the output consistent with the correct answer using numeric score from 0-7. We binarize the score by marking outputs with score 2 6 as correct (fully correct or containing only minor errors) and scores incorrect. This evaluation protocol helps distinguish between models that arrive at the correct answer by coincidence and those that produce solutions consistent with the reference reasoning.
Math Retrieval. For the retrieval benchmark, we evaluate whether embedding models can identify structurally equivalent problems. The primary metric is Recall@k, which

---

Published as a conference paper at ICLR 2026
measures whether any of the top-k retrieved problems correspond to correct match from the equivalent problem variants in MATHNET. We report Recall@1, Recall@5, and Recall@10.
To better characterize embedding behavior, also analyze cosine similarity distributions for three types of pairs: equivalent problems, unrelated problems, and hard negatives (near-miss problems with similar surface structure but different solutions). This analysis highlights cases where models struggle to capture fine-grained mathematical distinctions.
Generation. To assess the impact of retrioval down- stream problem solving, we evaluate model performance under three conditions: (a) zero-shol, (b) RAG using standard off-the-shelf embedding model, and (c) oracle RAG, where the retriever supplies ground-truti related problem from MATHNET.
Math Retrieval-Augmennt
This controlled setup allows us to quantify how retrieval quality affects solution accuracy. In particular, we report the performance gap between standard RAG and oracle RAG to measure the potential gains from improved math-award retricval. We also analyze failure cases where irrelevant retrieved examples degrade solution quality, illustrating the importance of structurally aligned retrieval.
4.3 MAIN RESULTS
4.3.1
MATHNET-SOLVE RESULTS
Table 3 summarizes accuracy across four mathematical domains. Baseline LLMs such as M-aari---aaerr and DeepSeek-V3. V3 achieve modest macro-averages in the mid-40s, indicating that direct pattern matching and shallow heuristics are insufficient for Olympiad- level problem solving.
Reasoning-angmmented models (e.g., Gemini 3.1 Pro and GPT-5) substantially improve performance, achieving macro-average scores above 67%. However, their accuracy remains uneven across domains. While Algebra shows steady gains, Geometry and Discrete Mathematics remain the most challenging categories, reflecting the difficulty of abstract reasoning, non-obvious solution paths, and combinatorial structures. For more detailed breakdown of language, multimodality and fine- fine-grained topies sensitivity, see Table 10 and Table 9 in Appendix D).
Zero-Shot Math Comprehension Results Test-Set-Large (6,400 samples)
Algebra
Number Theory
Geometry
Discrete Math
Macro Avg
Micro Avg
LLMs (Text-only)
Ministral-36
6.4% = 1.0
2.9% + 0.9
4.3% + 0.8
1.7% = 0.6
4.4% + 0.5
4.4% 0.5
DeepSeck-V3.2
51.6% 2.0
45.3%
4: 2.6
32.2% +: 1.8
32.7% 2.1
40.1% :: 1.2
40.1% 1 1.2
Grok-3
37.7% 1.9
33.0%
+ 2.3
21.7% +
24.2% 1.9
28.5%
28.5% +
LVLMs (Vision-enabled)
1lama-4-Havericke
22.5% 1.7
14.4% :: 1.8
10.7% 1.2
8.6% i: 1.3
14.7% 1: 0.9
14.7%
4 0.9
GPT-4.1
29.4% +: 1.8
24.0%
+: 2.2
15.7% + 14
16.6% 1.7
21.1%
+ 1.0
21.4%
4 1.0
QPT-4a
10.9% + 1.2
7.0% + 1.3
4.5%
0.8
4.2% + 0.9
6.8%
0.6
6.8%
0.6
L.MSs + Reasoning (Text-only)
DeepSeek-R1
46.1% 2.0
39.5% +: 2.5
31.2%
27.3%
2.0
36.3%
36.3% +
LVLMs + Reasoning (Vision + deliberate reasoning)
Genin1-3.1-80 1-Pro
83.7% + 1.5
82.2% + 2.0
74.6%
1.7
75.6%
+ 2.0
78.4% + 1.0
78.4% + 1.0
Genini-3-Flas
77.7% 1 1.7
73.3% +: 2.3
67.0%
+: 1.8
64.0%
2.2
70.4% :: 1.1
70.4% 4 1.1
Geniai-2.5-flash
50.5% + 2.0
42.6%
+ 2.5
36.8%
+ 1.8
31.0%
2.1
11.1% - 1.2
41.1% + 1.2
GPT-5
80.3% 1.6
73.6%
2.3
61.1%
65.3%
2.2
69.3%
69.3% +
GPT-6-mini
67.6%
61.3%
50.2%
57.0%
57.0%
GPT-S-nano
53.9% 2.0
19.6%
+ 2.6
32.4%
31.6%
2.1
12.2% + 1.2
42.2%
+ 1.2
21aude-4.6-0pus
53.2%
2.0
44.6% 4 2.5
44.3%
1.9
36.4%
2.2
45.7% +
1.2
45.7%
+ 1.2
Results are reported as mean + standard error. The performefonnng model is shown in bold, and the second-best model underlined. Macro Avg and Micro Avg are highlighted with light red and light blue backgrounds, respectively.
Table 3: Experimental results on MathNet-
8

---

Published as a conference paper at ICLR 2026
4.3.2 WATHNET-RETRIES RESULIS
As shown in Table 4, retrieval on МATINEТ remains highly challenging at the top-1 level, with even the strongest models (oxen3-embedding and Gemini Srriini-embeddingOO00 achieving only ~5% Recall@1. Performance improves markodly at higher cutoffs, with Recall@10 exceeding 80% in several domains. Among all models, Gemini-embecddingg- provides the most consistent gains, delivering the highest Recall@5 and Recall@10 across domains and the strongest aggregate performance (68.88% and 33.79% respectively). In contrast, legacy embedding models such as text seext-mmbddiig-adee ada-000022 and eeee-embedding- 3-small perform substantially worse across all settings.
These results suggest that current general-purpose embedding models fail to capture the deep structural and symbolic relationships that define mathematical equivalence. A critical failure mode is that both LLMs and LMMs often rely on superficial textual overlap (c.g., matching on key keywords such as "triangle" or "polynomial") rather than reasoning over the underlying mathematical concepts. The weak top-1 retrieval performance highlights that these models lack robust internal representation of mathematical knowledge that would support analogical reasoning across problem variants. This gap underscores the need for emboddings explicitly trained encode mathematical structure, rather than depending on incidental surfaco-level cues.
To further illustrate this issue, Figure 6 shows the distribution of cosine similarities between equivalent and equivvlll problems. Surprisingly, non-equivalent pairs often exhibit higher similarity scores than equivalent, ones. This counterintuitive trend highlights that embeddings frequently capture superficial lexical or symbolic overlap rather than true structural relationships, leading models to mis-rank distinct problems as closer than gemuinely equivalent ones. This explains the weak Recall@1 performance observed in Table 4.
Retrieval Results on Test-Set-Retrieval (10,000 equivalent pairs)
Model
Algebra
Number Theory
Geometry
Discrete Mach
All
Ro31
R
Roil
R135
Real
R
Re31
R85
R81
R
all-mpnet-base-v2
4.54%
73.06%
4.67%
82.54%
4.37%
74.76%
4.25%
75.38%
3.78%
57.7%
mild--genppe---a---o--
4.0%
69.4%
3.73%
80.76%
3.88%
71.735
3.98%
73.4%
3.27%
55.08%
cohere-enbed-vd.00
2.73%
59.85%
2.67%
68.85%
2.35%
59.87%
2.78%
63.4%
2.24%
44.81%
geen3-enbedding-4B
5.24%
78.74%
4.62%
86.43%
5.6%
79.05%
5.96%
81.5%
4.96%
64.95%
genini-embeddin--001
5.5%
91.62%
4.95%
87.43%
5.19%
81.86%
5.35%
82.8%
4. 8.83%
68.88%
text-emboddiig--dda-002
2.05%
54..94%%
2.22%
63.35%
2.16%
55.0 00%
2.71%
57.51%
1.94%
42 22.02%
text-eebedding---aaall
2.1%
47 17%
1.89%
54.622
2.1%
47. 61%
2.81%
50%12%
1.98%
35.19%
baxt-enbedding---large
3.19%
68.1 68%
2.73%
75.25%
3.2%
68. 18%
3.33%
69.52%
2.74%
54.23%
Table 4: Experimental results on and Recall@5. The highest score in each setting is underlined, and the highest overall scores are bolded.
expressed as percentages for 1ecall@1
4.3.3 MATUNET-RAG RESULTS
As shown in Table 5, providing these ground-truth pairs as retrieval context (ezpert-RAG) yields consistent gains over zero-shot settings under both human and LLM grading, Im- provements are largest for lower- and mid-tier solvers, indicating that math-aware retrieval supplies structure-aligned hints that current models do not reliably surface on their own. For the strongest systems, we observe occasional small dips (c.g., Gemini-3-Pre with human grading; GPT-5 with LLM grading), which we attribute to over-conditioning on partially relevant context. Tn the embed-RAG setting, we see high variance across results: when it retrieves structure-aligned neighbors it helps, but near-miss distractors often degrade performance.
Together, these results show that retrieval can meaningfully boost Olympiad problem solving but only when the retrieved context is truly structurally similar. Progress in retrieval- enhanced systems will hinge retrievers attuned to mathematical structure rather than surface lexical overlap. MATHNET with expert-aligned pairs and hard negatives offers a controlled setting to develop and rigorously evaluate such math-aware retrieval for RAG.
9

---

Published as a conference paper at ICLR 2026
Math RAG Results on Teat-Set-RAG (70 problems)
Model
RD
Human Grading
LLM Grading
(2025)
shot
ember RAG
expert RAG
shot
embed RAG
expert RAG
DeepSell-V.2-Speccale
01 Dec
84.8%
89.5%
97.3%
82.23%
87.87%
89.03%
Claude-4. 5-Dus
24 Nov
16.8%
55.5%
52.4%
15.97%
50.31%
56.43%
oLM0-3-Tbinx
20 Nov
45.2%
51.6%
47.6%
49.49%
45.56%
51.07%
Gr-a-t.1-Faat
19 Nov
75.4%
83.8%
83.2%
73.00%
67.00%
69.11%
Gemini-3-Pro
18 Nov
89.1%
92.9%
87.5%
73.16%
70.54%
76.43%
GPT-5 Phi-4-Reasoning Plus
07 Aug
76.8%
75.2%
86.6%
87 .9.0
81.8%%
85.70%
30 Apr 15.1%
14.3%
16.7%
24.06%
19.64%
30.04%
Table 5: Performance of evaluated language models on the Math Comprehension Test-Set- RAG (70 problems). The table reports human and average LLM grading accuracy under three prompting and retrieval configurations: zero-shot, Labedd-AGG, and expert-RAG. RD=release date.
5 DISCUSSTON AND LIMITATION
Results MATINET reveal clear gap between strong generative performance math- ematical problem solving and weak retrieval performance on mathematical equivalence. While frontier LLMs and LMMs achieve impressive scores answer-generation benchmarks, our retrieval results show that current embedding-basee systems still struggle to capture mathematical structure. The limited gains from visual augmentation further suggest that multimodal integration for symbolic tasks remains underdeveloped.
The strong performance of the formula-aware baseline indicates that structured, non-textual representatio inpeeettaiion tions are crucial for retrieval. Progress in true mathematical reasoning may require moving beyond next-token prediction toward architectures that explicitly integrate symbolic reasoning.
CONCLUSION
In this work, we introduced MATHNET, the first large-scale, multilingual, multimodal bench- mark for mathematical reasoning and retrieval. By providing a rich dataset of 30,676 problems with a fine-grained taxonomy of equivalence, we enabled a rigorous study of math- ematical generalization and analogical rcasoning. To ensure reliability, we complemented automated extraction with systematic human validation: expert annotators reviewed prob- lem similarity labels, and student evaluators assessed the aligument and completeness of extracted problem coobllem-olution solution pairs. These humaan contributions establish strong ground-truth foundation, ensuring that MATHNET captures deep mathematical structure rather than superficial overlap.
Our comprehensive evaluations show that while frontier generative models can solve complex problems, current retrieval systems still struggle with a fundamental yet overlooked task: retrieving mathematically equivalent structurally related problems from large corpora. This deficiency in retrieval highlights a key limitation in current learned eepresentations of mathematical knowledge. We hope MATHNET will the community, paving the way for research into improved retrieval-augmenter reasoning, symbolic AI, and ultimately, more capable and reliable problem-solving models.
as valuable resource for
BIBLIOGRAPHY
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et. al. Gpt-4 technical report. orXiv preprinl Niw:23333088711 2023.
Alon Albalak, Duy Phung, Nathan Lile, Rafael Rafailov, Kanishk Gandhi, Louis Castricato, Anikait Singh, Chase Blagden, Violet Xiang, Dakota Mahan, et al. Big-math: A large-scale,
10

---

Published as a conference paper at ICLR 2026
high-quality math dataset for reinforcement learning in language models. arXie preprint MrXic:2502.19955 2025.
Daman Arora, Himanshu Singh, et al. Have Iims advanced enough? a challenging problem solving benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7527-7543, 2023.
Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. Matharcna: Evaluating llms on uncontaminated math competitions. arXiv preprint XXtt:2255.23281.. 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint orXiv:2100.14 14168, 2021.
Debrup Das, Sam O' Nuallain, and Razich Rahimi. Rader: Reasoning-aware dense retrieval models. arXiv preprint arXiv:2505.1440 2025.
Moran Feldman and Amin Karbasi. Godel test: Can large language models solve easy conjectures? arXiv preprint arXiv:2509.188 18383, 2025.
Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. Splade v2: Sparse lexical and expansion model for information retrieval. arXiv preprint arXiv:2100 100.00 2021.
Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni-math A universal olympiad level mathematic 10. benchmark for large language models. arXiv preprint 24x:v:24 10. 07985, 2024a.
Bofei Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, et al. Omni- A universal olympiad level mathematic benchmark for large language models. arXiv preprint 24 0..07985 2024b.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint or arxiv: 2402240222 14008, 2024.
Dan Hendrycks, Collin Burns, Sleven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), 2021a.
Dan Hendryeks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021b.
Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, et al. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, (8-62991-63011 2023.
Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Ze Fan, Lyumanshan Ye, Ethan Chern, Yixin Ye, et al. Olympicarena: Benchmarking multi- discipline cognitive reasoning for superintelligent AT. arXiv preprint arXim:2406. 12753, 2024.
Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXi preprint
2021.
Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In SIGIR, 2020.
11

---

Published as a conference paper at ICLR 2026
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang. Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. CMMLU: Measuring massive multitask language understanding in chinese. In Findings of the Association for Computational Linguisties: ACL 2024, 2024a.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9): 2024b.
Yumeng Li, Guang Yang, Hao Liu, Bowen Wang, and Colin Zhang. dots.oer: Multilingual document layout parsing in a single vision-language model, 2025. URL https://arxiis org/abs/ ht//abb//2122 22122.222222 02498.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannanch Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathe- matical reasoning of foundation models in visual contexts. ar preprinl or 0xic:2310002255 2024.
Minh-Thaa Thang Luong, Dawsen Hwang, Hoang H Nguyen, Golnaz Ghiasi, Yuri Chervonyi, Insuk Seo, Junsu Kim, Garrett Bingham, Jonathan Lee, Swaroop Mishra, et al. Towards robust mathematical reasoning. In Proceedings of the 2025 Conference Empirical Methods in Natural Language Processing, pp. 35406-300 -34400,,,,,, 2025.
Sadegh Mahdavi, Muchen Li, Kaiwen Liu, Christos Thrampoulidis, Leonid Sigal, and Renjic Liao. Leveraging online olympiad-level math problems for Ilms training and contamination- resistant evaluation. Xie preprinl orXiv:2501. 14275, 2025.
Fan Nie, Ken Ziyu Liu, Zihao Wang, Rui Sun, Wei Liu, Weijia Shi, Huaxiu Yao, Linjun Zhang, Andrew Y Ng, James Zou, et al. Uq: Asscssing languago models on unsolved questions. arXiv preprint Xiv:2200.. 1 7580, 2025.
Zhihong Shao, Yuxiang Luo, Chengda Lu, ZZ Ren, Jiewen Hu, Tian Ye, Zhibin Gou, Shirong Ma, and Xiaokang Zhang. Decpsockmath-v2: Towards self-vcrifiable mathematical reasoning. arXiv preprint arXiv:2511.225 2025.
Jiayi Sheng, Luna Lyu, Jikai Jin, Tony Xia, Alex Gu, James Zou, and Pan Lu. Solving inequality proofs with large language models. arXiv preprint Xrx:v:2506.072 07927, 2025.
Haoxiang Sun, Yingqian Min, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, Zheng Liu, Zhongyuan Wang, and Ji-Rong Wen. Challenging the boundaries of reason- ing: An olympiad-level math benchmark for large language models. arXiv preprint ar 0rXic:2503321336 2025.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with MATH-Vision dataset. In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGT. In IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Richard Zanibbi, Behrooz Mansouri, Amurag Agarwal, et al. Mathematical information retrieval: Search and question answering. Foundations and Trendso in Information Retrieval, 19(--2):1-190, 2025.
Wanjun Zhong, Ruixiang Cui, Sai Liang, et al. AGIEval: A human-centric benchmark for evaluating foundation models. In Findings of the Association for Computational Linguistics: NAACL 2024, 2024.
12

---

Published as a conference paper at ICLR 2026
APPENDIX
The appendix provides additional tables, figures, prompts, and implementation details to support reproducibility and further analysis.
A Overview of Competitions Covered by MathNet
14
Taxonomy of Topics Commonly used in Math Olympiad
C Dataset Examples and Additional Statistics
20
D Performance Sensitivity to Language, Image Presence and Topics
26
Benchmarking LLM Graders vs Human Expert Graders
26
LLMs Usage and Prompts
28
13

---

Published as a conference paper at ICLR 2026
A OVERVIEW OF COMPETITIONS COVERED BY MATHNET
This section summarizes the competitions represented in MATHNET. The dataset includes problems drawn from wide range of mathematical olympiads, together with related training and team selection materials.
Table lists all sources included in the dataset and the years covered. The first part of the table contains international and multi-country compelitions (e.g., the Asia Pacific Mathematies Olympiad, the Balkan Mathematical Olympiad, and Baltic Way). The second part summarizes national sources grouped by country, including national mathematical olympiads, team selection tests for international competitions (e.g., IMO or EGMO team selection tests), and related national contests and training materials.
Source
Years
Competitions
National competitions and team selection tests by country
Argentina
2003-2024
Argentina Mathematical Olympiad (Argentine MO); Cono Sur Mathematical Olympiad: Rioplatense Mathematical Olympiad: Iberoamerican Mathematical Olympiad: May
Mathematical Olympiad
Australia
2010-2024
Australian Mathematical Olympiad: Australian Intermediate Mathematics Olympiad (AIMO); Australian Mathematical Olympiad Commitlee Senior Contest (AMOC Senior Con- test): APMO; EGMO; IMO; Mathematics Ashes; Mathemat- ics Challenge for Young Australians (MCYA)
Austria
2010-2024 Austrian Mathematical Olympiad (Regional Round; Prelimi- nary Round; Final Round; Beginners' Competition)
Belarus
2010-2022 Belarusian Mathematical Olympiad; IMO TST; Selection and Training Session
Brazil
2006 2022
Brazilian Mathematical Olympiad (OBM), Galois-Noether Noether University Mathematics Competition
Bulgaria
2003 2024
Bulgarian Mathematical Olympiad; Bulgarian Autumn Tour- nament; Bulgarian Spring Tournament; Bulgarian Winter Tournament; BMO TST; IMO TST; JBMO TST
Canada
1969 2025
Canadian Mathematical Olympiad (CMO)
China
2007-2025
China Mathematical Olympiad (CMO); China Mathemat- ical Competition: China TST; China Girls' Mathematical Olympiad (CGMO); China Western Mathenatical Olympiad; China Southeastern Mathematical Olympiad; China Western Invitational Mathematical Competition
Croatia
2011 2011-2019 Croatian Mathernatical Olympiad; Croatian Junior Mathe- matical Olympiad:; Croatia National Mathematical Compe- tition (City Round, County Round, Final Round); MEMO TST; IMO TST
Czech Republic
2000-2025
Czech Mathematical Olympiad: Czech and Slovak Mathemat- ical Olympiad: Czech Czech-Polish Czech- Slovakk Match
Mathematical Match;
Estonia
2010-2025
Estonian Mathematical Olympiad; Estonian Open Contests; IMO TST
France
2011 1024
French Mathematical Olympiad; EGMO TST; French Math- ematical Olympiad Preparation
Germany
2000 2022
German Mathernatical Olympiad; IMO TST
Greece
2007 2024
Hellenic Mathematical Olympiad (Archimedes); Hellenic Mathematical Olympiad; BMO; JBMO; Mediterranean Math- ematical Competition; IMO TST
Continued next page
14

---

Published as a conference paper at ICLR 2026
Source
Years
Competitions
Hong Kong
1997 2024
Hong Kong (China) Mathematical Olympiad (HKMO); Pre- liminary Selection Contest; Pre-IMO Mock Exam; IMO TST
India
2000 2024
Indian National Mathematical Olympiad (INMO); Indian Regional Mathematical Olympiad (RMO); EGMO TST; IMO
TST; IMO Training Camp Practice Test
Iran
2010-2025
Iranian Mathematical Olympiad, IMO TST
Ireland
2007-2025
Irish Mathematical Olympiad Italian Mathematical Olympiad: Italian Mathematical
Italy
1998-200 2024
Olympiad Project
Japan
2006-2024
Japanese Mathematical Olympiad (JMO); Japanese Junior
Mathematical Olympiad (JJMO)
Kazakhstan
2014-2001
International Zhautykov Olympiad
Moldova
2023
Moldova Mathematical Olympiad
Mongolia
2010-200 2025
Mongolian Mathematical Olympiad; IMO TST; EGMO TST
Netherlands
2006 2025
Dutch Mathematical Olympiad: Dutch Junior Mathematical Olympiad; BxMO; BxMO/EGMO TST; IMO TST
New Zealand
2019
2025
New Zealand Mathematical Olympiad
Macedonia
2008 2023
Macedonian Mathematical Olympiad; Junior Macedonian
Mathematical Olympiad: BMO; JBMO; Mediterranean Math- ematical Olympiad: European Mathematical Cup; IMO TST
Philippines
2018
Philippine Mathematical Olympiad
Poland
2023 2025
Polish Mathematical Olympiad
Portugal
2016 2017
Olympic Revenge
Romania
2004 2025
Romanian Mathematical Olympiad: RMM; IMAR Mathemat- ical Competition; Stars of Mathematics Competition; Danube Mathematical Competition: Clock-Tower School Competition; Selection Tests for BMO, JBMO, and IMO; Romanian Math- ematical Olympiad Shortlist; Gazeta Matematică National Olympiad
Russia
2009 2025
Russian Mathematical Olympiad; All-Russian Mathematical Olympiad; Euler Olympiad; EGMO TST
Saudi Arabia
2010 2025
Saudi Arabian Mathematical Competitions; BMÔ TST; EGMO TST; Gulf Mathematical Olympiad TST (GMO TST); IMO TST; JBMO TST; Camp Tests; Presclection Test
Serbia
2007-20023
Serbian Mathematical Olympiad; District Mathematics Com- petition; Municipal Mathematics Competition; IMO TST Singapore Mathematical Olympiad (SMO); Singapore Math- ematical Olympiad Open (SMO Open); Singapore Interna- tional Math Olympiad Challenge (SIMOC); IMO Training Camp
Singapore
2010
2025
Slovenia
2001 2023
Slovenian Mathematical Olympiad; Slovenian Matherattcal Olympiad for Technical Schools; IMO TST
South Africa
2010 2024
South African Mathematical Olympiad (SAMO); Rhodes University Camp; University of Stellenbosch Camp; South African Talent Search; South African Monthly Problem Sets;
IMO TST
South Korea
9006-2024
Korean Mathematical Olympiad (KMO)
Soviet Union
1961 1991
All-Union Mathernttiaa Olympiad; CIS Mathematical Olympiad
Continued on next page
15

---

Published as a conference paper at ICLR 2026
Source
Years
Competitions
Spain
2001 2023
Spanish Mathematical Olympiad: Iberoamerican Mathe- matical Olympiad; Mediterranean Mathematical Olympiad; Barcelona Contest Preparation; Barcelona Tech Mathcontest; IMO
Switzerland
1999 2023
Swiss Mathematical Olympiad; IMO TST
Taiwan
2011-2024
Taiwan Mathematical Olympiad: APMO Taiwan Preliminary Round; Taiwan IMO Team Selection Training Camp; Taiwan Mathematical Olympiad Training Camp
Thailand
2007-2017
Thailand Mathematical Olympiad: Thailand Third Mathe- matical Olympiad (T3MO); Thailand Training Camp
Turkey
2000-2022 Turkish Mathematical Olympiad; Turkish Junior Mathemati- cal Olympiad: EGMO TST; JBMO TST; IMO TST
Ukraine
2005 2022
Ukrainian Mathematical Olympiad; Ukrainian National Mathematical Olympiad
United Kingdom
2006 2022
British Mathematical Olympiad (BMO)
United States
1985 1985-2025
American Mathematics Competitions 10 and 12 (AMC 10/12); American Invitational Mathematics Examination (AIME); United States of America Mathematical Olympiad (USAMO); United States of America Junior Mathematical Olympiad (USAJMO); IMO TST; Harvard MIT Mathemat- ics Tournament (HMMT); Bay Area Mathematical Olympiad (BAMO); Berkeley Math Circle; William Lowell Putnam
Mathematical Competition
Vietnam
2001 2024
Vietnamese Mathematical Olympiad: Mock Test; Preparation
Test; IMO TST
International and Multi-country Competitions
АРMO
1989 2025
Asia Pacific Mathematics Olympiad
BMO (SL)
Balkan Mathematical Olympiad & Shortlist
BW (SL)
1990
Baltic Way & Shortlist
BxMO
2010-2022
Benelux Mathematical Olympiad
CPS
2005-2025
Paech-Polish-llowee Mathematical Match
EGMO
2012-2025
European Girls' Mathematical Olympiad
IMO (SL)
2006-2024
International Mathematical Olympiad & Shortlist
JBMO
2003 2023
Junior Balkan Mathematical Olympiad
MEMO
2008-2024
Middle European Mathematical Olympiad
NMC
1987 2024
Nordic Mathematical Contest
OIM
1985-2019
Ibero-American Mathematical Olympiad
RMM
2010 2021
Romanian Master of Mathematics
SRMC
4002-2025
Silk Road Mathematics Competition
IZhO
2014 2021
International Zhautykov Olympiad
Table Competitions and Countries included in MATHNET.
16

---

Published as a conference paper at ICLR 2026
B TAXONOMY OF TOPICS COMMONLY USED IN MATH OLYMPIAD
We provide the curated taxonomy used for labeling domains, subjects, topics, and subtopics. These labels ground our analyses and enable consistent cross-competition comparisons.
Sub-subtopic
Key Concepts
Geometry
Plane Geometry
Triangles
Centroid, incenter, circumcenter, orthocenter, ex-ceeners, Euler line, nine-point circle; geometric inequalities; trigonometry (metric relations)
Quadrilaterals
Cyclic, diagonals perpendicular
Complete quadrangle,
Circles
Angles, coaxal, tangents, radical axis, metric relations, Apollonius circle
Concurrency / Collinearity
Theorems of Ceva, Menelaus, Pappus, Desargues Translation, rotation, homothety, spiral similarity, inver- sion, the method of moving points
Transformations
Advanced Configurations
Simson line, Miquel, Napoleon / Fermat Brocard points, symmedians, polar triangles, harmonie/isoge isogo nal/isotomic conjugates, barycentric coordinates
Geometric Inequalities Combinatorial Geometry
Classical and advanced
Helly, Sylvester, convex hulls, Pick theorem, Minkowski theorem, convex figures
Analytic/ / Coordinate Methods
Complex numbers, Cartesian coordinates, vectors, trigonometric relations
Miscellaneous
Angle/distance distance chasing, constructions, loci
Solid Geometry
3D Shapes
Polyhedra, prisms, pyramids, spheres, cylinders, cones Cavalieri's principle, Formulae and problem-solving Formulae and applications
Volume
Surface Area
Other 3D problems
Mixed problems, reducing the problem into a plane ge- ometry problem
Differential Geometry
Curvature
Gaussian, mean
Manifolds
Surfaces, parametric
Geodesics
Shortest paths, great circles
Non-Euclidean Geometry
Spherical Geometry
Spherical triangles, angles, area Lines, models, inequalities
Hyperbolic Geometry
Algebra
Prealgebra / Basic Algebra
Integers
Sets of integers, Divisibility, primes, the Greatest Com- mon Divisor (GCD), the Least Common Multiplier (L.M)
Fractions
Operations, simplification, comparison Conversion, operations, rounding Linear equations, word problems Number properties, prime factorization, divisors
Decimals
Simple Equations
Other
Algebraic Expressions
Continued on next page
17

---

Published as a conference paper at ICLR 2026
Sub-subtopic
Key Concepts
Polynomials
Operations, factorization, Algebraic identities, symmet- ric functions, Vieta's formula, interpolation formulae, complex numbers, roots of unity, Chebyshev polynomi- als and other trigonometric polynomials, irreducibility of polynomials, Descartes rule of signs, rootso polyno- mials, Intermediate Value Theorem (TVT)
Soquences / Scries
Recurrences, Charachteristic equations, monotonocity, boundedness, periodicity, convergence and divergence, floor//eilliggs, cumispproducts, telescoping sums, Abel
summation
Functional Equations
Substitution, defining a new function, Cauchy's equa- tions, Injectivity/sur/e /surjectivity, Periodicity, application of Calculus and Mathematical Analysis, iterations
Inequalities Functional considerations Classical inequalities
Linear Sinear/Quadie Quaddatiic solving techniques
Cauchy-Schwarz, QM-AM-GM-HM,, Power Mean, Jensen's Inequality, smoothing, Muirhead, Chebyshev's inequality, majorization, combinatorial optimization
Discrete Mathematics
Graph Theory
Basic concepts
Vertices, edges, path, connected graphs, cycles, Hamilto- nian cycle and path, trees
Matchings
Marriage Lemma, Tutte's theorem
Connectivity
Menger, max-flow min-cut
Extremal
Turán
Euler characteristic
E-E+F
Combinatorics
Emumeration
Symmetry, basic counting techniques, recursion, bijec- tion, inchssin--ccllusion, double counting
Probability
Expected values, probabilistice methods, partitions, gen- crating functions
Binomial coefficients
Algebraic properties
Pigconhole principle
Applications
Invariants / Monovariants
Problem-solving
Coloring / Extremal
Graph problems
Induction
Standard and smoothing
Games Greedy
Strategies, combinatorial games
Logic Algorithms Other
Logic
Propositionaa//reediaat logic, truth tables
Algorithms
Sorting, searching, Dynamic Programming (DP), greedy Miscellaneous problems, strategy development, problems, inter-deciplinary problems
Other
Number Theory
Divisibility
Factorization
Primes
Properties, sieves, prime numbers tests
GCD
Euclidean algorithm; linear combinations; Bezout's 's iden- lity
LCM
Computation; relation with GCD
Factorization
Trial, Fermal, Pollard
Continued on next page
18

---

Published as a conference paper at ICLR 2026
Sub-subtopic
Key Concepts
Modular Arithmetic
Basic operations (mod n), in-
Existence (when ged(a, = 1); computation (extended
verses (mod n)
Euclidean algorithm)
Chinese Remainder Theorem
Solving systems of congruences; applications number
(CRT)
theory and cryptography
Fermat / Euler Wilson
Theorems; proofs; problem-solving applications Roots, factorization; applications to number theory prob- lems
Polynomials mod
Residues / Primitive Roots
Primitive roots
Existence modulo primes; modulo p"; computation
Quadratic residues
Properties; Legendre symbol; Euler's criterion
Quadratic reciprocity
Law of quadratic reciprocity; applications Definition; computation; relation with primitive roots and cyclic groups
Multiplicative order (mod n)
Diophantine Equations
Factorization Methods
Difference of squares, Sophie Germain identity, special factorizationss Unique Factorization Domains (Gaussian, Fisenstein integers); Norms algebraic number fields; Vieta jumping
Modular Arithmetic & Congru- ences
Reductions modulo primes or powers; Quadratic residues, Legendre symbol; Multiplicative order & primilive roots; Hensel lifting; Local- global principles (solvability mod p)
Parametrization of Solutions
Pythagorean triples; Rational parametrization of con- ics (general quadraties); Higher-degree parametrizations (elliptic curves, quartics)
Inequalities Size Arguments
Bounding arguments; Infinite descent; Minimal solutions (no smaller solution possible)
Special Equations
Pell's equation: continued fractions, fundamental solu- tion, recurrence; Fermat-type: x² + +y² =2
Descent & Structural Methods
Infinite descent; Descent elliptic curves; Geometry of numbers
Arithmetic Functions
Euler's totient's function
Properties, applications Computation, properties Basic properties Definition, applications
Number Sum of divisors
Sum of digits
Möbius inversion
Algebraic Number Theory
Algebraic numbers
Minimal polynomials, field extensions, solving Diophan-
tine equations
19

---

Published as a conference paper at ICLR 2026
C DATASET EXAMPLES AND ADDITIONAL STATISTICS
Example from MathNet-Core
Given positive integers m and n, find the smallest integer N (2 m) with the following property: if an N-element set of integers contains complete residue system modulo m, then has a nonemply subset whose sum divisible by n.
Problem.
Answer.
N max m, m - n) + +11
Solution. Let d (m, n), and write m dmi, dnj. We first prove the lower bound
2 max m, m + n 2 mld +m(d+11]
Assume
m d + 1).
2
Choose a complete residue system modulo m, say Ii, ....... Im, whose residues modulo n consist of exactly my copies of each of ..2...., d. For instance, one may take
+ dnij,
1,2 d, j = 1,2...., my
Now let
k k=
2 m m(d 1)
and choose l further integers U1 ......: Mks all congruent to modulo Then
A = fm U1 , Me
contains complete residue system modulo m. However, no nonemply subset of A has divisible by n: indeed, the sum of the least nonnegalive residues modulo of all elements of A is at least 1 and at most
m1(1
.... + + +
-m + +)+ =
Hence no nonempty subset sum can be congruent to modulo n. Therefore
N2 m m 1),
which gives the claimed lower bound.
Next we show that
1 N max (mn m, m + mmd + 11}
2
does have the required property.
We need the following standard fact: among any k integers, there exists nonemply subset. whose sum is divisible by k. Indeed, for integers 01 01,...,,5. 4g. let
S. = 01 + ... +
(1)is
If some S, (mod k), we are done. Otherwise, two of the St have the same residue modulo k, say S, = Sy (mod k) with < j, and then
a+1 + ... + a, = S S =
(mod k).
A useful corollary is: among any & integers each divisible by a, one can find nonempty subset whose sum is divisible by ka.
(see next page)
20

---

Published as a conference paper at ICLR 2026
Continue; Example from MathNet-CCree
We now split into two cases.
Case 1: n S m(d + 1), so = m. Let 21...... Zm be complete residue system modulo m. Since dm1, these may be partitioned into groups, each of which is a complete residue system modulo d. Call finite set of integers d-set the sum of its elements is divisible by If d is odd, write complete residue system modulo as M...... Id with yi = (mod d).
Then each such group can be partitioned into d+1 d-sets, for example
2
{ L Mejs }. {ya).
{ui. vd-13; {32, va-2):
Thus altogether we obtain
m (d + 1)
d-sets. Since
m S 5 (d 1),
the corollary implies that some collection of these d-sets has total sum divisible by ni d = n. If d is even, similarly each complcte residuc system modulo yields d-scts, with one element left over; pairing leftovers from two such groups gives one more d-set. In total, from Tl,
Um we oblain
mi 2
mid 2
+
d-sets. Since
1
n1 S
2 m (d + mnd + m1.
2
we again have enough d-sets apply the corollary and obtain a nonempty subset whose sum is divisible by n.
Case 2: n > fmnld + 1), s0
N m + m(d 1).
Let be an N-element set containing a complete residue system modulo m, say #1 #1,... Tmy logether with
1
71
2 m(d + 1)
additional elements. If d is odd, then as in Case 1 the elements 21.... Zm can be partitioned into
} m (d++) 1)
d-sets. Partition the remaining elements arbitrarily into
11 7 (d++))
groups of size d. From each such group, by the corollary, one can extract a d-set. Hence altogether we obtain exactly nn d-sets. Applying the corollary once more the sums of these an d-sets, we find nonempty union of them whose total sum is divisible by (sce next page)
n,d = n.
21

---

Published as a conference paper at ICLR 2026
Continue; Example from MathNet-COre
If d is even, then from F1,.., 4m we obtain
1
M1
mid+ +
2
2
d-sets, as in Case 1. If my is even, the remaining elements can be grouped so as to produce another
n1 m2 (d +
d-sets, giving again a total of mn d-sets. If m1 odd, one element 24 remains with
d F33 2
Partition the other remaining elements into
2m - mi (d +
groups of size d Each such group contains a subset whose sum is divisible by d 2: and any two such ş-scts combine to form d-set. Together with {æ,), this again yields enough d-sets to obtain ny of them in total. Applying the corollary as above gives a nonemply subset with sum divisible by Therefore, in all cases,
2.
N max m, m + n m + +1)
Metadata. Country: China, Competition: China Math Olympiad, Year: 2013
Example from MathNet-Solve with figures
Problem. Andrew and Olesya take turns cutting out either 2 x 2 square or 1 x square from a x 2n rectangle, always along the grid lines, in such way that after every move the remaining figure stays connected. The player who cannot make move loses. Determine who wins if both play optimally and Olesya moves first.
Solution. We proceed by induction
For = 1, the board is 2 x 2, and Olesya wins immediately by removing the whole 2 x 2 square. For
2, the board is 2 x 4, and Andrew wins by direct case analysis. Now suppose = 2m + 1 is odd. Then Olesya can remove 2 x 2 square from one end of the strip, reducing the game the case of even n. Thus it remains prove that for even n, the second player has a winning strategy.
Consider therefore a board of size 2 x 2k. Until and including the first time the first player cuts x 1 square from bordering column of the active strip, the second player follows a symmetric strategy. This always possible, and the first such move must occur when the active playing field has size 2 with é even.
From that point onward, suffices for the second player to prevent the first player from removing
x 2 square. Indeed, once this is impossible, all remaining moves consist of removing 1 x 1 squares, so the game ends after an even number of further moves, which is favorable to the second player.
The only positions from which 2 x 2 square can be removed are the two ends of the active strip. Hence the second player simply ensures that no end becomes a removable 2 x 2 block. Concretely, whenever there is a "free" end, he removes one 1 x 1 square from that bordering 2 2 block. There can never more than one free end at time: Olesya (see next page)
cannot create two in single move, and initially there are none.
22

---

Published as a conference paper at ICLR 2026
Example from Matlvet-Solve
If there no free end, then one of the following applics at a bordering column:
the bordering column has no removed square, Andrew removes one unit square from it.
the bordering column has exactly one removed square:
a) if the second-to-bordee column also has a removed square, Andrew removes the second square from the bordering columm; b) if the second-to-border column has removed square, then:
i. if the third-to-border column is empty, or has a removed square in the same row, Andrew removes unit. square from the second-to-border column in that row;
ii. if the third-to-border column has a removed square in the other row, Andrew removes unit square from the bordering column, shrinking the active strip without creating a free end.
square appears at an end. Therefore the parity of the remaining x moves is controlled in his favor, and the second player wins for even n. Combining this with the odd case, conclude:
every case, Andrew can respond so that no removable
Olesya wins when n is odd, and Andrew wins when is even.
Mctadata. Author: Bogdan Rublyov.
23

---

Published as a conference paper at ICLR 2026
Example from MathNet-Solve with multiple solutions
Problem. Let a and be positive integers, and let and be finite sets of integers satislying:
(i) A and B are disjoint;
(it) an integer belongs either to A or to B, then either + a belongs to A or belongs to B.
Prove that a|A| = b/B).
Solution 1. Define f AUB + AUBB by
i a a f(1) b otherwise.
If f(t) = f()) and both values lie in A, then i + a = j + a so i = j. If both lie in B, then
i =b= so i j. Thus f is injective and hence bijective.
Σ i= E (i+a)+ (i+a)
Σ bi Σ
i i+a|A| b|
YEAUB
SUDE
J(D)B
HEAJB
Therefore a/.A| b|Bl
Solution 2. Construct a directed graph on AU B with edges -i+ a if and i i b E B. From (i) each vertex has in-degree I 1, and from (ti) out-degree 2 1. Since total in-degree equals total out-degree, every vertex has degree exactly 1, so the graph is union of directed cycles.
If a cycle contains Og elements of A and by of B, then the net change along the cycle is
aak bby- = 0.
Thus oag = 6bg. Summing over all cycles gives a|AA| B|
Metadata. Country: Australia. Competition: Asian Pacific Mathematics Olympiad. Year: 2012.
Author Notes: Solution 2 adapted from Leo Jiang.
Other Benchmarks Sources OmiGi-MTTH (ICLR 2025)
MathNer Sources (official problemsisolutions)
Postonn Cayy le surhi and anger Peon
AoPS Online
DNSEECHOOL 00090 COMAARITY SESOUCEES
Algetra
Solatlune
mever appesmiing fert
Prreove fir Olayre ded
High School Dlynzadds
Adtomio lagart wnster crama oe
Figure 4: MathNet is a collection of official Olympiad documents sourced directly from national problem booklets. This example shows BMO 2023 problem that appears in both MathNet and Omni-MATH Gao et al. (2024a) While Omni-MATH relies on the AoPS discussion shown on the left. MathNet provides the official problem and solution the right.
24

---

Published as a conference paper at ICLR 2026
PDFs Examples
Ulrsststen Mathonatical Olyrepied
heirs, Decanden Peeaaless
OM
remeone Aurge sade deignt dny Marrge
with songh
Thac Sround Jenerry 2015
Fire Tour
menug uneel leme dhough adye turelle farel
weth de
This cediia lalrot atlly ststed patt Be Seding ford vether giv shrmshiv conernction ot sertined park
Soluie by Joakee faos, Sdheel Coiserite offii
life who oole das
dis role-
parderrr sembere.
cadl the
rensing copele C.EEE
Sive so are aduere to oll the sont odlec, rore nextt
of E00 with owd 10
ctoiireet plon perteck, thregh
The
200M
Pron Mccomnsssrr IOUN 215 3015 notheec tas
AO Perepyee sumte 2400- 5 2030
Makic
Prodiom moad Upen otlaee
The
Oduarmas vages oppcees coper sporonuees ADC, en maso
Taes Mil
CAld opoonaa spaay collk CDCA by
180"
secaaac spaatpotans mporonseens API some. crig Chepenmenn JigN - 2007 20000 in bo raose capee O
MNN
atV.
www
speronaanes BMN
ZPAC en MU MPPOO PMM 2R2
wss Theeers 2MAY 2000 Phoseass very wes IV www Mis anace in Laoe apatick vou cpetne do-
Prskonr and
All Di 1i
2ED NATIA LENGE SOLUTION URIEARRRAAATT
I10301T
our baggr them The viica opribie For com alg
by eddetio
lckool wad
apes drabe'y iy U.S
werld su meggina O.
Figure 5: Examples of scanned pages from national mathematics Olympiad booklets across different countries and years. The examples shown are from Ukraine, Argentina, Russia, Saudi Arabia, India, Japan, and Australia.
25

---

Published as a conference paper at ICLR 2026
Language
English 22909
Portuguese 1320
Spanish 900
French 868
Italian 703
Serbian
Count
650
Language
Slovemian
German 577
Chinese 536
Romanian
Korean 296
Dutch
Count
646
452
278
Language
Russian
Mongolian
Macedonian
Polish
Hungarian
Count
211
112
90
Table 8: Distribution of problems by language in the dataset.
D PERFORMANCE SENSITIVITY TO LANGUAGE, IMAGE PRESENCE AND TOPICS
We analyze how model accuracy varies with factors: (1) the presence of figures in the test sample and (2) the language of the sample.
Model
Overall
With images
Text only
gemini-3. 1-pro-preview
78.4% 1.0
85.1% + 1.9
76.7% + 1.2
gemini-3-flash-
70.4% + 1.1
80.4% + 2.1
67.8% 1.3
gpt-5
69.3% + 1.1
72.0% 2.4
68.6% 1.3
gpt-5-mini
57.0% + 1.2
62.5% + 2.6
55.6% + 1.4
claude-opus-4-6
43.7% 1.2
52.9%
43.8% 1.4
gpt-5-nano
42.2% + 1.2
30.9% + 2.5
45.1% 1.4
gemini-2.5-flash
41.1% + 1.2
53.3% + 2.7
38.0% 1.3
DeepSeex-V3.2
40.1% +:2 1.2
38.7% 2.7
40.4% 1.4
DeepSeek-R1
36.3% + 1.2
39.7% + 2.7
35.4% 1.3
grok-3
28.5% 1.1
34.4% 2.5
27.0%
gpt-4.1
21.4% + 1.0
23.6% + 2.2
20.8% + 1.1
14.7% = 0.9
17.2% + 2.1
14.1% 1.0
gpt-4o
6.8% 0.6
10.0% 1.6
6.0% = 0.6
Ministral-3B
4.4% + 0.5
10.0% + 1.6
3.0% + 0.5
Table 9: Average accuracy by model for image-containing and text-only problems (best in bold, second best underlined).
Model
Overall
Chinese
English
French
Italion
Portuguese
Slovenian
Spantshh
gentnt-3. I-pro
78.4%
1.0
64.2%
8.1
77.4%
1.1
80.6%
8.3
9N6.65
4.3
91.8%
2.7
84.0%
77.0%
10.7
genini -i-ahhh
20.15
1.1
18.15 + 5.5
60.5% +
72.0%
12.5
90.6%
4.3
89.3%
70.8% +.9.5
12.1%
11.5
69.3%
45.3%
09.0%
00.4%
14.5
77.0%
82.0%
75.3%
9.3
65.6%
11.5
gpt-E-aiai
C
45.7%
16.3%
44.7%
43.8%
14.5
02.15
12.1
64.7%
50.67
11.1
49.2%
13.1
Spt-Srnano
12.2% 1.2
15.45
43.0% +:
43.8%
14.6
87.956
12.1
86.7%
85.8% 4 10.5
36.1%
12.3
gentai-2.5-r
DespSoci-V3.2
30.1%
13.8% 6.1
30.99%
41.7%
14.5
55.476
30.7%
44.4%
11.1
41.0%
12.3
Despdons-RI
36.15 1 1.2
1.35
4.5
35, 895
45.8%
14.6
30.0/%
12.1
50.3%
39.6%
1: 10.5
31.4%
11.5
grok-3
28.3%
3.7
18.8%
50.0%
gpt-4.1
16.7%
28.6%
Thaza---Kvvrickk
14.7% 2: 0.0
8.855
14.1% 4 0.9
8.1%%
7.8
22.4%
10.8
26.6%
25.99%
9.8% 7.4
4pt-4a
0.855
0.6
1.655
2.0
6.2%
0.6
6.2%
7.3
0.9%
6.0
15.7%
19.8%
3.3%
Rintetrs1-38
Table 10: Average accuracy by model and language (best in bold, second best underlined]
E BENCHMARKING LLM GRADERS VS HUMAN EXPERT GRADERS
We benchmark the accuracy of a wide range LLM graders and compare their judgments to human expert grading on MathNet-RAG. This evaluation quantifies how reliably current models can act as automatic graders for Olympiad-level mathematical reasoning. For each model, report performance under three settings: zero-shot, embed-RAG, and expert-RAG. This measure both cross-model grading consistency and alignment with human scoring.
26

---

Published as a conference paper at ICLR 2026
nopeenal-tex--S-lrrg
b) openal-text-ade-
c) openelltext--3emmell
d) Beeiinl-mbeeddingge..
Migenal seguiaaleess origlia
3.
08
Coann lanty
Coults ainrr
Coane aimaartty
6) muut--ge-mpnet-bssee bese-v1
all-mpnet--aseee basen-22
al cochrre-mmbed-vv.
h) g-wen-3-embeddingg---
Looalne siiv
Coolin similrity
Cosine siilrit
Ci5ine milartyy
Figure Cosine similarity distributions for equivalent (green) and near- haard negatives (orange) problem pairs across different embedding models. Higher separation between the two distributions indicates a model's ability to distinguish structurally identical problems from those with small but critical alterations.
Cross-Model Grading + Human Scores
Model
LLaMA-4 DeepSeek- V3 GPT-4.1
GPT-4o
Average
Human Expert
Zero Shot
claude-opus-4.5
72.243
41.557
31.371
38.70 38.700
45.971
46.8%
despseek-v3.2-9pp --ppeccaale
96.186
74.286
85.457
73.014
82.229
84.8%
comini-3-pro-previor
94.700
72.357
71.686
53.500
73.057
89.1%
gpt-5
98.057
85.014
83.157
82.143
87.086
76.8%
1rok-4.1-fast
92.657
63.357
76.529
59.700
73.060
75.4%
olmo-3-32b-thank
70.243
44.286
35.200
48.214
49.486
45.2%
phi-4-reasoning-phee
46.186
23.529
6.629
19.900
24.057
15.1%
embed- RAG
claude-opus-4.5
64.93%
59.69%
40.30%
36.49%
50.34%
55.5%
(eepseek-v8.2-ppecaaa
94.21%
78.37%
92.21%
86.73%
87.89%
89.5%
Smmin--3-pro-preiiee
95.91%
71.43%
68.63%
46.17%
70.54%
92.9%
gpt-5
93.23%
73.94%
80.96%
79.11%
81.81%
75.2%
1rok-4.1-fast
88.57%
61.13%
72.23%
48.67%
67.66%
83.8%
olmo-3-32b-think
66.89%
38.57%
31.49%
45.29%
45.56%
54.6%
ph1-4-reasoning-Pe
36.64%
12.70%
16%
21.04%
19.64%
14.3%
expert- R.AG
claude-opus-4.5
77.06%
56.51%
53.57%
39.54%
56.43%
52.4%
deepseel-13.2-speccee
97.14%
83.41%
89.29%
86.24%
89.03%
97.3%
genini-3-pro-previorr
99.63%
70.03%
72.70%
63.37%
76.43%
87.3%
gpt-5
97.29%
77.93%
82.60%
85.20%
85.76%
86.6%
grok-4.1-fast 92.50%
55.66%
74.23%
54.09%
69.11%
83.20%
olmo-3-32b-think 74.67%
48.99%
33.16%%
47.44%
51.07%
47.6%
phi-4-reasoning-plee 48.57%
31.26%
09.70%
30.64%
30.04%
16.7%
Table 11: Breakdown of cross-model grading performance under Sero-Shot, embed-RAG,
and expert-RAG configurations, augmented with human evaluation scores for each model.
27

---

Published as a conference paper at ICLR 2026
F LLMs USAGE AND PROMPTS
We include the core prompts used for extraction, evaluation, and metadata classification. These are the exact versions used in experiments.
Listing 1: System prompt for solution generation
SYSTEM PROMPT =
SYSTEM_PROMPT>
"Solve the following math problem. Write out your full reasoning.
"At the very end, place your complete final response inside LaTeX Wboxed().
"- If the problem asks for a numerical or closed-form answer, put only that final expression in the box.
n- If the problem asks for proof argument, then enclose the entire proof (
not just concluding sentence) inside the box."
Listing 2: User prompt for grading
GRADING PROMPT_TEMEE TEMPLATE """)
You are an expert grader for the International Mathematics Olympiad (IMO).
Your task is to strictly and rigorously evaluate a proposed solution using official
IMO standards.
Only fully justified, logically sound arguments may receive credit.
GENERAL SCORING RUBRIC -077)
points:
Correct
A complete, correct, fully rigorous solution. Earlier incorrect attempts do not reduce the score if the final solution is fully correct.
-6 points:
Almost Correct
The main idea is correct, and only minor gaps small fixable
remain.
Missing major steps or relying on unjustified claims does not qualify.
1 point: Partial Progress
The student demonstrates substantial progress toward the solution.
Reformulations, trivial observations, irrelevant partial results do qualify.
points: Incorrect
The solution contains fundamental logical flaws or makes no significant progress.
Fake proofs, circular arguments, or unjustified assumptions receive 0.
PROBLEM TYPE: (problem_tyye
PROBLEM STATEMENT
(problem pproblem_statementy
GROUND-TRUTH SOLUTION
(solution)
GROUND-TRUTH FINAL ANSWER "rinal_answen
STUDENT SOLUTION
{student_anael answer)
28

---

Published as a conference paper at ICLR 2026
EVALUATION PROCEDURE
1. Study the official solution to understand the essential steps and criteria for partial credit.
2. Analyze the student's solution line-by-line, identifying every gap, unjuatified inference, incorrect claim, or flaw in logic.
3. Determine whether the student makes any nontrivial progress toward the solution.
4. Assign the score strictly according to the IMO rubric.
CLARIFICATION: Meaning of 'final aataal_asswer_eeo correcct
'final_ answer correct' refers only to whether the student's final stated answer matches the ground-truth final structurally).
(numerically, algebraically,
The student's final answer is typically enclosed in Wbbxed((O) use that as the answer to compare.
It does NOT evaluate the correctness, completeness, rigor of the student's reasoning.
Use 'doean't apply' in either of these cases:
the problem type is 'proof only' (no final answer expected), OR the student gives \lbozed(O) answer.
Examples:
Correct Wboxed(() ansver but no valid argument + 'yes'
Incorrect Wbboxe(([}) answer but mostly correct reasoning 'no' 'doesn't apply' Proof-only problem 'doean't apply'
No W[boxed(O)
OUTPUT FORMAT (MUST FOLLOW EXACTLY)
Output exactly one JSON object:
{
'score': {{
'points': 0,
'label': : '0 out of 7' }},
'analysis': {{
[ettalle__reaoonnng "
Srentified_errorst 0,
:
}},
'meta': {{
'final_answer answer wwwwer_correct correct': 'yes no doesn't t apply',
"contains "contains_logic_ereee errors'; 'yes no"
}}
}}
RULES
You must output nothing outside the JSON.
'points' must be one of: 0, 1, 6, 7.
'label' match exactly: 'X out of 7',
All fields required.
Listing 3: System prompt for topics, final answer, and metadata extraction
SYSTEM_PRONEE r""" You are rigorous math-olyzpiad content analyzer.
You will be given "problem package" containing:
29

---

Published as a conference paper at ICLR 2026
problem statement
One or more official solutions (labeled "Solution "Solution 2", ...) Optional final answers
Your tasks are:
1. TOPIC EXTRACTION
Assign the problem its most specific topics from the taxonomy.
Each topic path must be array of strings from general specific.
Include ALL paths relevant to the problem or solutions.
Every topic path must be verbatim copy of a path from the taxonomy.
No paraphrasing, renaming, reordering, combining nodes. Every topic must begin with "Topics".
2. MAIN IDEAS TRICKS / TOOLS
Produce bullet list the key structural insights tools used.
Examples:
Techniques used
Classical lemmas or theorems applied
Core inequality strategies
Key constructions or combinatorial ideas
Do NOT retell the whole solution; extract the essential tools.
**...*********
3. NATURAL-LANGUAGE PROBLEX DESCRIPTION
Summarize the core taak of the problem in normal English.
NO mathenatical symbols at all (no variables, no equations, no angle notation, etc
high-level, intuitive, short description.
4. PROBLEM TYPE CLASSIFICATION
Classify the problem into exactly one the following:
"proof only": explicit final mmeriic/cloeed-forz answer is required.
"final answer only": problem only asks for value/choice with proof required. "proof answer": requires both reasoning final aalue/statement.
"MCQ": problem requires choosing from given options.
FINAL AMSWER EXTRACTION
If the problem requires a final aumeic/closed-foon expression, value, or choice, extract it.
If the problem's nature does NOT require final ansver (e.g., proof-only), output "mull'.
Specific rules:
If multiple solutions exist, the final answer must match the official ansver section present.
Accept integers, expressions, ranges, choices, constructed forms, etc. For MCQ, return the *selected option* if identifiable; othervise null.
TAXONOMY BLOCK
Use this taxonony for the "topics" field. Each topic path must follow the hierarchy strictly.
30

---

Published as a conference paper at ICLR 2026
Topics
|- Geometry
|-- Plane Geometry
--- Triangles
--
Triangle centers: centroid, incenter,
circumcenter, orthocenter, Euler line,
nine-point circle
1--
Triangle inequalities
--
Triangle trigonemetry
Quadrilaterals
Cyclic quadrilaterals
Inecribed/circumserii quadrilaterals
-- Quadrilaterals with perpendicular diagonals
- Circles
Coaral circles
Tangents
Radical axis theorem
--
Circle of Apollonius
Concurrency and Collinearity
Ceva's 8 theorem
Menelaus' theorem
(more topics here)
OUTPUT FORMAT (STRICT JSON)
Return ONLY JSON object:
"topics":
["Topica", ....", "..."], ["Topics", "...", ".
}.
"main ideas": [
"key idea 1",
"key idea 2",
"key idea 3"
"natural,
Siatural_Lnnguge_ccm searguaggeeeeerpptiioon "..." "final_ane answer": "... or mull", "problem_typp"" "proof only | final answer only proof and answer MCQ",
"confidence": 0.0-1.0
Rules:
NO text outside the JSON.
NO markdown in the output.
Language_description description must contain zero mathematical symbols. natural natural__anguage
Confidence reflects how certain you are about the classification.
LLMs Usage in the Paper The authors made of large language models (L.LMs) primarily to support the writing process, including polishing the text for clarity and readability. In addition, LLMs were employed to assist in refining the design of the project website as well as the interface used by annotators.
31


Reviews:
[{"review_id": "Ub1XJLYhjy", "reviewer": "Reviewer_Rdfs", "summary": "This work proposes a new dataset of 13k multimodal olympic-level math reasoning problems over 10 languages, as well as a new math retrieval task for locating the similar problems over shallow lexical overlap. The problems are crowd-sourced from textbook and websites and normalized through a data processing pipeline with human-AI incorporation. The evaluation of several powerful LLMs show that they can still struggle with retrieving equivalent math problems.", "questions": "- How are the golden answer of math retrieval determined?\n- Given the result in Recall@5 and Recall@10 (which are significantly greater than Recall@1), how well would model perform if the embedding-based retrieval is used only for a coarse-grained result before another model (perhaps smaller) is used to further identify the answer? And what would be the satisfactory threshold (i.e. is the Recall@10 good enough)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The task of math retrieval can be used to test the reasoning ability of LLMs in a different but daily-use dimension. The community can benefit from the support of MathNet in such functionality.\n- The data collection process is introduced with rich detail, with deep human-AI cooperation in cross-validation and cross-reinforcement.\n- The definition of problem similarity is detailed discussed and categorized.", "weaknesses": "- Only light analysis of the experimental results are provided, enhancing which could better inspire following studies in this line.\n- I still have a few questions which are listed below, and I am happy to adjust the estimation if more convincing details are provided to them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aIzKjZtS1z", "reviewer": "Reviewer_jSKk", "summary": "The paper introduces MathNet, a large-scale multilingual, multimodal benchmark of Olympiad‑level math problems (13,026 problems; 40 countries; 10 languages) with expert solutions, plus a new equivalence-based retrieval task (39,078 synthetic pairs) and a taxonomy of similarity (Invariance/Resonance/Affinity). It details an OCR→LLM pipeline for aligned problem–solution extraction, human/LLM validation, dataset statistics, and evaluates both comprehension (solution generation) and retrieval (Recall@k) across a range of LLMs/LMMs and embedding models. Main findings: frontier LLMs remain challenged on Olympiad problems (e.g., GPT‑5 macro‑avg 72.25%); equivalence retrieval is hard at top‑1 (≈5% R@1).", "questions": "1. **How are the 39k equivalence pairs generated and validated at scale?**\n\nCould you detail pair generation (templates/transformations, LLM rewriting, symbolic rewrites) and filtering? If models assisted, how did you prevent leakage (Sec. 3.5 p. 6 mentions similarity modes; contributions p. 2).\n\n2. **Could you report multilingual and multimodal breakdowns?**\n\nAdd per‑language comprehension/retrieval metrics and cross‑lingual retrieval (query language A → target language B), given Table 7’s imbalance (p. 17).\n\n3. **Can you strengthen statistical rigor and reproducibility?**\n\nReport CIs/variances for Tables 3–4 via repeated runs or bootstrap (p. 8).\n\n4. **Could you tighten mathematical correctness/notation?**\n\nDefine symbols in Intro examples (e.g., clarify Πᵢ)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Scale, Authenticity, and Provenance:** 13,026 problems across 40 countries and two decades; sourced from official national booklets rather than community forums. This supports representativeness and reduces annotation noise. Appendix A.1 lists dozens of national/regional exams enabling wide topical coverage and cross‑year comparisons. Page numbers and source files are recorded during extraction, aiding auditability and reproducibility. \n2. **Novel Equivalence‑Focused Retrieval & Taxonomy:** 9,078 paired problems labeled for equivalence and relatedness (contributions, item 2; p. 2), enabling evaluation beyond lexical similarity. This is novel for Olympiad math IR. Invariance/Resonance/Affinity with concrete examples (Table 2 p. 6) clarifies what “sameness” means in math problems—important for designing retrievers. Cosine similarity distributions show overlap between equivalents and hard negatives (Fig. 3 p. 7), illustrating the task’s difficulty. \n3. **Clear, Multi‑Stage Extraction/Validation Pipeline:** OCR→problem extraction→solution retrieval→cross‑model validation is diagrammed (Fig. 2 p. 5) and explained (Sec. 3.3 p. 4–5), demonstrating a principled approach to noisy PDFs. 20 annotators on 100 pairs plus LLM distractor stress tests and expert consensus checks (Sec. 3.4 p. 5) bolster data quality. Full prompts for extraction/evaluation/metadata are given (A.5 pp. 18–20), supporting reproducibility. \n4. **Comprehensive Baselines and Findings:** Eight LLM/LMM systems compared across four domains (Table 3 p. 8), showing domain‑wise strengths/weaknesses and leaving headroom (e.g., Discrete/NT hardest). This is informative for future modeling. Eight embedding models with Recall@{1,5,10} per domain and overall (Table 4 p. 8) provide a solid baseline landscape. Misinterpretation/logical gaps/context over‑reliance are articulated (Sec. 4.4 p. 9), guiding where methods fail.", "weaknesses": "1. **Equivalence Pair Construction/Validation Under‑specified**\n- Generation details missing: The paper states 39,078 synthetic equivalence pairs (p. 2) but does not detail how they were generated, filtered, or formally validated beyond taxonomy definitions—No direct evidence found in the manuscript. This limits trust in ground truth. \n- Limited expert audit: Only a 100‑problem subset received expert review (Sec. 3.4 p. 5), which may be small relative to 39k pairs. This constrains external validity.\n- Agreement metrics absent: No inter‑annotator agreement or judge‑consensus statistics are reported—No direct evidence found in the manuscript—reducing clarity on label reliability.\n2. **Multilingual & Multimodal Claims vs. Reporting**\n- Language imbalance: English dominates (89.88%; Table 7 p. 17), potentially limiting the strength of “multilingual” claims for training/evaluation comparability.\n- No per‑language scores: Comprehension/retrieval are not broken down by language—No direct evidence found in the manuscript—so cross‑lingual generalization remains unclear.\n- Multimodal share unquantified: The proportion of image‑interleaved problems is not reported—No direct evidence found in the manuscript—yet conclusions note marginal gains for image inputs\n3. **Mathematical Formulation & Notation Clarity (correctness/consistency)**\n- Ambiguous notation/examples: Intro uses expressions like “pi+1 − pi ≤ Πi” without defining Πi (p. 2,line087), which may confuse readers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o1Sz4hibuo", "reviewer": "Reviewer_qJLT", "summary": "This paper introduces MathNet, a new large-scale, multilingual, and multimodal benchmark for advanced mathematical reasoning. The dataset consists of 13,026 Olympiad-level problems sourced from official competition booklets from 40 countries over two decades, covering 10 languages. The paper's primary contributions are twofold: (1) the dataset itself, which is meticulously curated, expert-authored, and annotated with a rich taxonomy of mathematical topics and similarity types; and (2) a novel benchmark task focused on \"math-aware retrieval,\" which evaluates a model's ability to identify mathematically equivalent problems.", "questions": "1.  **Regarding RAG Experiments:** Have the authors considered running a controlled RAG experiment? For example, by comparing a solver's performance under three conditions: (a) zero-shot, (b) RAG with a standard off-the-shelf retriever, and (c) RAG with an \"oracle\" retriever using the ground-truth equivalent problems from MathNet. Such an experiment would provide a direct and powerful demonstration of the value of math-aware retrieval.\n\n2.  **Regarding the Retrieval Benchmark Construction:** Could you provide more detail on the process used to generate the 39K equivalent problem pairs? Were these created by human experts, generated by an LLM and then verified, or through some other procedure? Furthermore, how were the \"near-miss/hard negatives\" used in Figure 3 selected or defined? Understanding this is key to interpreting the retrieval results.\n\n3.  **Regarding Multimodality:** Could you provide more statistics on the multimodal component of MathNet? Specifically, what fraction of the 13K problems include diagrams that are essential for solving? Could you also include a few examples in the appendix showing where current LMMs fail on the visual reasoning component, and what a correct interpretation of the diagram would entail?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "*   **Originality:** The most significant original contribution is the formulation and operationalization of the \"math-aware retrieval\" task. While other benchmarks focus on problem-solving, MathNet pioneers the evaluation of a model's ability to recognize structural equivalence, a cornerstone of mathematical thinking and analogical reasoning. \n\n*   **Quality:** \n    *   **Data Sourcing:** By curating problems from official national Olympiad booklets, the authors ensure expert-level quality, authority, and consistency, avoiding the noise often present in community-sourced platforms like AoPS.\n    *   **Curation Pipeline:** The multi-stage pipeline for extraction and validation is state-of-the-art. Using a specialized OCR model (`dots-ocr`), followed by LLM-based problem-solution alignment, and then cross-model verification (using both GPT-4.1 and Claude 4 Opus) is a robust and scalable approach that minimizes bias and error. This is further strengthened by human validation.\n\n*   **Significance:** \n    *   **Resource for the Community:** MathNet provides the largest multilingual Olympiad-level dataset, a critical resource that will fuel research in mathematical reasoning, cross-lingual transfer, and RAG for years to come. \n    *   **Practical Implications:** The work has clear connections to real-world applications, from improving tools for mathematicians and contest organizers to building more robust and generalizable AI reasoning systems.", "weaknesses": "*   **Underdeveloped RAG Experiments:** The paper compellingly argues that math-aware retrieval is essential for effective RAG in this domain. However, it does not present any experiments to directly validate this claim. This is a significant missed opportunity. A controlled experiment showing that a solver's performance is significantly boosted when provided with ground-truth equivalent problems (an \"oracle\" retriever) versus a standard retriever (e.g., one based on `gemini-embedding-001`) would provide powerful, direct evidence for the benchmark's utility and the paper's core thesis.\n\n*   **Limited Analysis of Multimodality:** The paper positions MathNet as a \"multimodal\" benchmark, but this aspect feels underdeveloped in the analysis. The main text briefly mentions that augmenting models with diagrams yields \"only marginal improvements\" but provides little detail. The paper would be stronger with more statistics (e.g., what percentage of problems are inherently visual?), qualitative analysis (e.g., examples of diagrams that models fail to interpret), and a deeper discussion of the specific challenges posed by mathematical diagrams (e.g., geometric constructions vs. data plots).\n\n*   **Detail on Equivalence Pair Construction:** The methodology for creating the 39,078 \"synthetic problem pairs with labeled equivalence classes\" is not fully detailed in the main paper. While the taxonomy of similarity is explained, the process of generating these pairs—which are central to the novel retrieval task—could be described more transparently. Understanding how these pairs were generated is crucial for assessing the difficulty and validity of the retrieval benchmark.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pxoM5AQDDh", "reviewer": "Reviewer_dueB", "summary": "The paper introduces MathNet, a large-scale, multilingual, and multimodal benchmark for mathematical reasoning and retrieval, specifically targeting Olympiad-level problems. MathNet comprises 13,026 expert-authored problems drawn from 40 countries, spanning 10 languages and two decades of national and international competitions. The dataset includes aligned LaTeX and natural-language problem statements, official solutions, and rich metadata.\n\nThe key contributions are:\n1. Dataset: A curated corpus of high-quality Olympiad problems and solutions from official national sources (not community platforms like AoPS), ensuring expert-level authenticity and diversity.\n2. Novel Retrieval Task: The paper defines and implements a mathematical retrieval task based on equivalence, constructing 39,078 synthetic problem pairs labeled by mathematical similarity.\n3. Taxonomy of Similarity: A fine-grained classification of mathematical relatedness into three modes.\n4. Comprehensive Evaluation: Benchmarking of 16 models across two tasks.\n\nThe paper makes a compelling case that solving ≠ retrieving: models can generate correct answers without understanding underlying mathematical structure well enough to retrieve equivalent formulations.", "questions": "1. Equivalence Pair Generation:\nHow were the 39,078 mathematically equivalent problem pairs constructed? Were they generated via symbolic transformations, human rewriting, or LLM paraphrasing? If synthetic, how was the mathematical correctness of equivalence verified? Could you provide examples of non-trivial equivalences that required deep insight?\n\n2. Multimodal Content Details:\nWhat fraction of MathNet problems include diagrams or images? Can you share statistics on image types (e.g., geometric figures, graphs, tables)? Did you experiment with vision-specific embeddings (e.g., CLIP on diagrams) for retrieval, or only text embeddings?\n\n3. Cross-Lingual Performance:\nDo model performances vary significantly across languages? For instance, is retrieval harder for problems originally in Arabic or Russian vs. English? Could you include a small table or analysis of language-wise accuracy?\n\n4. Retrieval-Augmented Reasoning (RAG) Experiments:\nThe paper mentions that RAG only helps when retrievers surface structure-aligned neighbors. Did you conduct downstream RAG experiments (e.g., using retrieved equivalents to aid problem solving)? If so, what were the gains? If not, is this planned for future work?\n\n5. Human Evaluation of Solutions:\nBeyond validating extraction, did you conduct human grading of model-generated solutions (e.g., on a 100-problem subset)? How does GPT-4o’s judgment correlate with human graders for non-English or proof-based answers?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is highly original in both problem formulation and dataset construction. While mathematical reasoning benchmarks exist (e.g., MATH, GSM8K, OlympiadBench), MathNet is the first to explicitly focus on mathematical retrieval based on equivalence, a critical but underexplored capability. The taxonomy of similarity (Invariance/Resonance/Affinity) provides a principled framework for evaluating analogical reasoning in mathematics—a novel conceptual contribution.\n\n2. The dataset construction pipeline is rigorous:\nSources are official national contest booklets, not crowd-sourced.\nProblem–solution alignment uses an LLM-assisted extraction pipeline with cross-model validation (GPT-4.1 + Claude 4 Opus).\nHuman validation by domain experts ensures high fidelity.\nThe retrieval benchmark includes hard negatives and near-miss distractors, making evaluation meaningful.\nExperimental design is thorough: multiple models, domains (Algebra, Geometry, Number Theory, Discrete Math), and metrics (Recall@k, cosine similarity distributions) are reported.\n\n3. The paper is exceptionally well-written and structured. Figures (e.g., Figure 1 overview, Figure 3 similarity distributions) and tables (e.g., Table 2 similarity taxonomy, Tables 3–4 results) are clear and informative. The distinction between comprehension and retrieval is articulated precisely, and the limitations of current models are illustrated with concrete examples.\n\n4. MathNet addresses a real gap in AI for mathematics: the inability to recognize structural equivalence across notations, languages, or modalities. This has implications beyond benchmarks—it affects retrieval-augmented reasoning, automated theorem proving, and mathematical search engines. By releasing the dataset and benchmark publicly, the authors provide a foundational resource for the community to develop models with a deeper mathematical understanding.", "weaknesses": "1. Limited Analysis of Multimodality:\nWhile MathNet is described as multimodal, the paper reports only marginal gains from visual inputs and does not deeply analyze why vision-language models underperform. Are the diagrams too complex? Is OCR/captioning inadequate? A more detailed error analysis of multimodal failures (e.g., types of diagrams that confuse models) would strengthen the contribution.\n\n2. Synthetic Equivalence Pairs May Lack Real-World Validity:\nThe 39K equivalent pairs are described as “synthetic.” The paper does not clarify whether these were algorithmically generated (e.g., via symbolic rewriting) or human-curated. If synthetic, they may not reflect the kinds of equivalence that arise organically in contest design or research. This risks overestimating or mischaracterizing retrieval difficulty.\n\n3. Evaluation Protocol for Comprehension Relies on GPT-4o as Judge:\nWhile prior work (Omni-MATH) validates this approach, using a single LLM—even a strong one—as the sole evaluator introduces potential bias or blind spots, especially for non-English or non-standard solution styles. Including human evaluation on a subset (beyond extraction validation) would bolster credibility.\n\n4. Under-Specified Cross-Lingual Challenges:\nThe paper highlights multilingual coverage but does not report per-language performance or analyze whether retrieval/comprehension degrades in low-resource languages (e.g., Persian, Ukrainian). Without this, the claim of \"multilingual benchmark\" feels partially substantiated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)