One Example Shown, Many Concepts Known! Counterexample-Driven Conceptual Reasoning in Mathematical LLMs
Authors: Yinghui Li
Jiayi Kuang
Haojing Huang
Zhikun Xu
Xinnian Liang
Yi Yu
Wenlian Lu
Yangning Li
Xiaoyu Tan
Chao Qu
Ying Shen
Hai-Tao Zheng
Philip S. Yu
Venue: ICML 2025 poster (2025)
Keywords: Large Language Models
Mathematical Reasoning
Primary area: deep_learning->large_language_models
TL;DR: The first work aims to enable LLMs to perform mathematical reasoning by giving counterexamples.

Abstract: Leveraging mathematical Large Language Models (LLMs) for proof generation is a fundamental topic in LLMs research. We argue that the ability of current LLMs to prove statements largely depends on whether they have encountered the relevant proof process during training. This reliance limits their deeper understanding of mathematical theorems and related concepts. Inspired by the pedagogical method of "proof by counterexamples" commonly used in human mathematics education, our work aims to enhance LLMs’ ability to conduct mathematical reasoning and proof through counterexamples. Specifically, we manually create a high-quality, university-level mathematical benchmark, COUNTERMATH, which requires LLMs to prove mathematical statements by providing counterexamples, thereby assessing their grasp of mathematical concepts. Additionally, we develop a data engineering framework to automatically obtain training data for further model improvement. Extensive experiments and detailed analyses demonstrate that COUNTERMATH is challenging, indicating that LLMs, such as OpenAI o1, have insufficient counterexample-driven proof capabilities. Moreover, our exploration into model training reveals that strengthening LLMs' counterexample-driven conceptual reasoning abilities is crucial for improving their overall mathematical capabilities. We believe that our work offers new perspectives on the community of mathematical LLMs.

One Example Shown, Many Concepts Known! Counterexampl<-Driven Conceptual Reasoning in Mathematical LLMs
Yinghui Li
Jiayi Kuang" 2 Haojing Huang" 1 Zhikun Xu 133 Xinnian Liang' 5 Yi Yu Wenlian Lu
Yangning Li Xiaoyu Tan 17 Chao Qu? Ying Shen 128 Hai-Tao Zheng 16 Philip S. Yu
Abstract
We believe that our work offers new perspectives on the community of mathematical LLMs.
Leveraging mathematical Large Language Mod- els (LLMs) for proof generation is fundamental topic in LLMs research. We argue that the abil- ity of current LLMs to prove statements largely depends on whether they have encountered the rel- evant proof process during training. This reliance limits their deeper understanding of mathemati- cal theorems and related concepts. Inspired by the pedagogical method of "proof by counterex- amples" commonly used in human mathematics education, our work aims to enhance LLMs' abil- ity to conduct mathematical reasoning and proof through counterexamples. Specifically, we manu- ally create a high-quality, university-level math- ematical benchmark, COUNTERMATH, which requires LLMs to prove mathematical statements by providing counterexamples, thereby assess- ing their grasp of mathematical concepts. Ad- ditionally, we develop a data engineering frame- work to automatically obtain training data for further model improvement. Extensive exper- iments and detailed analyses demonstrate that COUNTERMATH is challenging, indicating that LLMs, such OpenAI o1, have insufficient counterexample-drives proof capabilities. More- over, our exploration into model training reveals that strengthening LLMs' counterexample-drvenn conceptual reasoning abilities is crucial for im- proving their overall mathematical capabilities.
1. Introduction
Mathematics, as fundamental aspect of reasoning, has garnered significant research interest. Recent studies have demonstrated that Large Language Models (LLMs) exhibit strong mathematical reasoning abilities (OpenAI, 2023; Google, 2024; Yang et al., 2024; Shao et al., 2024; Ying et al., 2024; Chern et al., 2023; Luo et al., 2023; Yu et al., 2024a). Enhancing the mathematical reasoning capabilities of LLMS has become a prominent and fundamental topic within the LLMs research community.
Currently, there are two main paradigms for enhancing the mathematical reasoning capabilities of LLMs. The first involves synthetic generation based on seed math ques- tions (Yu et al., 2023; Li et al., 2024a). For example, Wiz- ardMath (Luo et al., 2023) introduces a variety of math instructions generate math questions of different com- plexities using GPT-3.5. The second approach leverages formal mathematical languages to train LLM-based theo- rem provers, such as Lean 4 (Moura & Ullrich, 2021). For instance, Safft-Sketch-Bet
(Jiang et al., 2023), Hunyuan- Prover (Li et al., 2024c), and Lean-STaR (Lin et al., 2024a) interact with formal languages through informal proofs, au- tomatic formalization, and natural language thoughts for theorem proving.
The two methods above enable LLMs to develop problem- solving skills either by training on massive similar problems, or by gaining proficiency through exposure to similar proof processes (Mirzadeh et al., 2024; Yu et al., 2024b). In both cases, these approaches enhance LLMs' mathematical reasoning abilities through training, where proficiency is achieved through familiarity, akin "drill-based" learning in human mathematics learning. However, relying solely on intensive-prann -rrattcce by inundating LLMS with math problems is neither sufficient nor essential for true mathematics learn- ing. In other words, drill-based learning alone does not foster a deep understanding of mathematical concepts in either humans or L.MSs.
Equal contribution.
Tsinghua University. E-mail: lyyinghu20@ wwwaailsssinguu.... Sun Yat Sen University School of Mathematical Science, Fu- dan University + ARCC Lab, Arizona State University 'Bytedance Inc. 'Peng Cheng Laboratory "INFLY TECH (Shanghai) Co., Ltd. 'Guangdong Provincial Key Laboratory of Fire Science and Intelligent Emergency Technology, University of Illinois Chicago. Correspondence Ying Shen, Xiaoyu Tan, Zhikun Xu 76 7ar sys..cu............. txywilliam 1993 @ outlook.comm. zhikunxu asu.edu>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

COUNTERMATH: Counteresamlle-riiee. Conceptual Reasoning in Mathematical LLMs
Drill-based Learning
Example-based Learning
Seed math questions
Formal mathematical languages
Counterexample proof paradigm
geometric shapes
Math Concept:
arithmatic
set, space, measure,
Assume Opposite:
application probloms
function
Formal Proof- I Bl
Prncipla atee
Conclude:
Derive Contradiction:
Must
thon
Generate questions to
Trained as Theorem provers
Learn from examples
Solve: 2x 3 Subtract 2x 2. Divide Answer:
By definition
All continuous functions are differentable.
only ((x)
Consider the example: function fixi
Figure Comparison between
learning and example-basee learning. The first two math LLMs fail when confronted with advanced mathematics, and "Proving by examples" is a highly creative and concept- intennsvtte mathematical skill.
As illustrated in Figure 1, for human mathematics learning, "example-based" learning is more important strategy than drill-based learning. In particular, for mathematical proofs, "proof by counterexamples" is an indispensable approach. Inspired by the idea that counterexample- driven proofs bet- ter reflect deep understanding of mathematical concepts, we propose COUNTERMATH a counterexample-Dasse mathematical reasoning benchmark. COUNTERMATH is designed to evaluate LLMs' ability to distinguish subtle differences between mathematical terms and properties at university-level by providing examples. Specifically, we collect 1,216 statement-rationaal pairs from mathematical textbooks, focusing on disproving certain statements under unusual conditions using counterexamples. In terms of dif- ficulty, COUNTERM ATH covers advanced mathematical knowledge similar to PutnamBench (Tsoukalas et al.) and Putnam-AXIOM (Gulati et al., 2024), both of which assess the depth of mathematical understanding in LLMs.
Lower performance is observed in topology and real analysis during our fine-grained evaluation, which in- dicates promising future research directions. Further studies on mathematical LLMs should explore these underrepresented areas of higher mathematics.
Our fine-tuned model, trained with only 1,025 training samples, demonstrates strong performance on both our benchmark and OOD benchmarks. This confirms that learning counterexample-biiee
reasoning is not only effective for our task but also has general significance for improving mathematical reasoning.
2. Related Work
Math Benchmarks. Recently, the number of math-related benchmarks has increased drastically (Amini et al., 2019; Yang & Deng, 2019; Zheng al., 2022; Hendrycks et al., 2021; Cobbe et al., 2021; Frieder et al., 2023; Liu et al., 2024; He et al., 2024; Lu et al., 2024). The most influen- tial ones are MATH (Hendrycks et al., 2021) and GSM8K (Cobbe et al., 2021), which focus on arithmetic reasoning at the high school competition level and grade school level, re- spectively. Moreover, other benchmarks such as MathBench (Liu al., 2024) and OlympiadBench (He al., 2024) are also blends of problems sets from various competitions and standard examinations, which are used to test human stu- dents' abilities of utilizing the math knowledge and certain tricks to solve complex application- based problems. How- ever, mathematicians are more expecting LLMs to help them in literature review, idea generation, proof-checking and collaborative writing as they focus on broader spec- trum of mathematical activities rather (Frieder et al., 2024). To better accommodate the true need for math research, some formal theorem proving benchmarks like Putnam- Bench (Tsoukalas et al.), CoqGym (Yang & Deng, 2019) and MiniF2F (Zheng et al., 2022) are also proposed re- cently in a combination of formal mathematical languages
In addition to extensively evaluating various mainstream mathematics LLMs on COUNTERMATH, we also develop framework for automatically acquiring counterexample- based mathematical reasoning data enable further model training. Detailed analyses of both the evaluated LLMs and our trained LLMS reveal that:
The contemporary LLMs including OpenAl ol exhibit limited performance in determining whether a state- ment in COUNTERMATH is true or false, indicating significant room for improvement in higher-level math- ematical conceptual reasoning.
When analyzing the reasoning process of LLMs, many models still struggle with example- exaeple-based reasoning. This demonstrates the limitations of drill-based learn- ing and underscores the potential value of COUNTER- MATH in advancing mathematical LLMs.
hhttps://gi thub. com.bbccom///// www.comm///oo/oo http://tKkaaa//bb COUNTRRAAARRARRR
2

---

COUNTERMATH: Counterexample-Drien Conceptual Reasoning in Mathematical LLMs
Supervision
Annotators
Experts
Translation
Validation
OCR Tool
Photocopied Textbooks
Raw Data
CounterMathh
Figure 2. Overview the construction process of COUNTER COUNTERMATH was first extracted from photocopied mathematical textbooks by crowd-source labelers with the OCR tool. For the next step, authors with bachelor degrees in applicd mathematics as annotation experts would filter and correct improper statement-rational pairs, Finally, GPT-4o was prompted to translate the validated data into English under experts' supervision.
compilers (e.g. Coq. Lean), which could be viewed as the important math benchmarks for developing Mathematics Mechanization (Wu, 2001; Wu & Gao, 2007).
Conceptual Reasoning. Conceptual reasoning is an abil- ity to reason in abstract and high-level perspectives (Wang et al., 2024b; Huang et al., 2024; Li et al., 2024d; 2025). Recently, there are numerous studies where LLMs are rea- soning on abstracted and conceptualized structures by anal- ogy, deduction, induction, etc (Saparov et al., 2023; Li et al., 2023; Yasunaga et al., 2024; Xu al., 2024; Wang et al., 2024a; Cheng al., 2024; Zhou et al., 2024). Specifically in math, conceptual reasoning requires people to reason around math concepts and axioms at the play of math hypothesis, statements and problems (Simon, 2011). An example of this ConceptMath (Wu et al., 2024), a math word prob- lem benchmark in elementary school and middle school level, but the reasoning in solving these problems remains superficial as it just requires models extract the correct variables and do basic arithmetic operations and it is also sat- urated with GPT models, which diminishes it from showing whether LLMs are truly mastering mathematics.
On the contrary, our benchmark COUNTERMATH focuses on conceptual reasoning among mathematical concepts and theorems. Specifically, we research certain math reason- ing technique: counterexamples in mathematics, check whether the models fully and correctly understand math con- cepts and theorems, which should be one of atomic abilities for what mathematician are expecting from LLMs compared to independently solving some simple math word problems.
Math Augmented L.Mss. In contrast to general-purpose models such as GPT-4 (OpenAl, 2023) and Gemini (Google, 2024), several mathematics augmented LLMs have been developed using methods like data augmentation, pretrain- ing, fine-tuning, and reinforcement learning with extensive mathematical corpora. For instance, WizardMath (Luo et al., 2023) employed tailored math prompts generate seed data and then underwent RLHF and process supervision for training. Abel (Chern et al., 2023) utilized supervised fine-tuning with meticulous data processing, referred to as Parental Oversight. IntemLLM--Matt (Ying et al., 2024) en- hanced mathematical reasoning with thain-of-thougge (Wei al., 2022), code interpreters, and Lean4 translation and theorem proving. NuminaMath (Li et al., 2024b), which re- cently secured first place in the Kaggle AIMO competition?, leveraged tool tool-integrated reasoning (TIR) to generate math questions with fine-grained solutions. Qwen2.5-Math ( (Yang et al., 2024), initialized with general-purposs Qwen2.5 mod- els, was trained on the undisclosed large-scale and high- quality mathematics-speciin corpus. Deepseck-Man Mathh (Shao et al., 2024) focuses data engineering during pretraining and efficient RL training.
3. COUNTERMATH
3.1. Data Construction
Our dataset is constructed from a series of math textbooks focusing counterexamples in different fields such Al- gebra (Hu, 1983), Topology (Wang & Yang, 2000), Real Analysis (Wang, 1989) and Functional Analysis (Wang, 1994). We have obtained the authors' consent to use their publications solely for academic research purposes. As the raw data sources are in Chinese, we also translate our dataset into English, creating a mathematical conceptual reasoning benchmark based on counterexamples, named as COUNTERMATH. Each data point includes stat rat i onale, judgement (i.e., whether the statement is True or False by its rationale), and field.
As illustrated in Figure 2, we first recruited several Chi- nese annotators from specific vendors to extract statement- rationale pairs from the aforementioned textbooks using an OCR tool, which yielded 1, 1,274
httpps: / /www. kaggle. com/ competitiiiinntnooeeccco si-matther mathhmaticcaaato ool wawaadddppeeeeaaieeeeeeee
pairs.
3

---

COUNTERMATH: Counterexample-rrive. Conceptual Reasoning in Mathematical LLMs
Next, the experts, among the authors, who have the bach- elor's degrees in applied mathematics manually checked all the data points from the previous stage, annotated each statement as True or False based on its rationale, and filtered out ambiguous pairs. This resulted in 1,216 data samples as the final version of our dataset. Additionally, since the data source was originally in Chinese, we prompted GPT-4o to translate the dataset into English. The authors then further validated the translated dataset ensure its correctness and appropriateness. More details about the annotation process are provided in Appendix A.
4. Benchmark Settings
We are testing following large language mod- els with COUNTERMATH. For open-weight models, we use Deepseck-Math-TBBU (Shao et al., 2024), Eurus- 2-7B-PRIME (Cui et al., 2025), Owen2.55Matth77/77B5 Instruct (Yang et al., 2024). Numina (Li
Baselines
et al., 2024b), Abel-7B/133/700 (Chern et al., 2023), (Ying et al.. 2024), WizardMathh7BB708B (Luo et al., 2023), Mathstral-7B', MetaMath-Mistral-PP (Yu et al., 2023), Xwin-Math- 7B/13B/70B (Li et al., 2024a), Rho-math-7h-inn tb-ittreeeeeerrrreeer (Lin et al., 2024b), MAmmoTTH--7B8x77B-
(Yue et al., For proprietary models, use GPT-4o, OpenAl o1 Qwen-Max', Deepseek-R1', Claude3.7-sonnee" and Gemini2.5-pro" The selection of baseline models tries to cover as many representative models as possible, which considers various perspectives, including data processing, training paradigms, base models, and developers (i.e., academia industry). The detailed summary of open-weight baseline models is in Appendix B.
3.2. Data Analysis
2024),
Overview Since the dataset has been constructed from textbooks in four different fields, the distribution of statineet-rationaal pairs is shown in Figure 3a. Morcover, the distribution of judgements is presented in Figure 3b. We observe that most statements are labeled as True, This may be because the data is sourced from mathematical text- books, where most statements are phrased correctly to avoid misleading readers, especially novices in mathematics. In general, as shown in Figure 8, the statements often involve several college-level mathematical concepts or properties, focusing on nuanced understandings of mathematics. In real-world applications, due to the concise formulations, these statements are frequently used as interview questions in math-related graduate programs. This also contributes to the diversity of mathematical testbeds for contemporary LLMs, fostering research in conceptual mathematical reasoning (Klymchuk, 2010).
Prompts To maximize the performance of LLMs on COUNTERMATH, we use the default CoT prompts for each LLM, which are typically mentioned in their Huggingface model cards. These prompts include completion prompts, Alpaca-like prompts, and chat-template- based prompts. Ad- ditionally, we adopt Hint prompt, designed to encour- age LLMs to provide examples when attempting solve problems. A summary of our used prompts is provided in Appendix B. Note that it is observed that using In-Context Learning (ICL) prompts do not significantly affect the per- formance of LLMs on COUNTERMATH. We believe this is due to the nature of counterexample-diiee -driiee conceptual rea- soning, where LLMs struggle to learn the ability to provide examples from a small number of demonstrations. ticular, mathematical subfields exhibit substantial variation in concepts and terminology. Therefore, we do not use ICL prompts in our experiments.
Fobse
12.20%
18.95
35.20%
Alabbra
par-
87.80%
Trum
Roal Arahpis
(a) Different fields.
(b) Judgement types.
Evaluation Metries Our evaluation metrics two-fold. For efficiency, we use lexical matching such as F/ to match
Figure 3. Data Distribution of COUNTERMATH.
"uttps:/ggeeenn..ttttu..bb
Data Validation As mentioned in Section 3.1, the reten- tion rate between the two annotation stages is 95.4%, demon- strating that most high-quality statement-rationale pairs were extracted from the textbooks. The filtered statement- rationale pairs often suffer from issues such as irrelevance between statement and rationale, excessive typos, and trivial rationales, such as simply quoting conclusions from other statement-rational pairs or even papers.
qwq wbgg33bb-reeeew///
Pataps:/cca..ppenai.com
ol --yyatttn-car--2222220555
int.ps://www.elbbabccll
what-is-quen-11mn
news250120
>https://openai. com http:/hhheloopptt--eee
gemini/pro/

---

COUNTERMATH: Counterexample-Drive Conceptual Reasoning in Mathematical LLMs
Final Training Data
Available Math Datasets
Raw Counterexample Data
Refinement
Filter
Align with benchmark
o
Identify examples
use example?
counterproof?
Check
Figure 4. The overview of training data engineering framework.
the judgements of the statements. The reason of not using accuracy is the imbalance of data distributions mentioned in Section 3.2. To assess whether the model has acquired the capability of solving mathematical problems through exemplification, we conducted targeted evaluation in this section. We designed a systematic evaluation framework leveraging GPT-4o as an automated judge to perform the following tasks: Example Extraction automatically identi- fies and extracts instances where the model explicitly uses exemplification (i.e.. generating or referencing specific coun- terexamples justify the statement) during its reasoning process. Alignment Assessment evaluates whether each extracted example aligns with the reasoning approach of a predefined Reference Example in terms of logical structure, problem decomposition, and goal relevance. The evaluation prompts are detailed in Appendix B. We also designed three evaluation metries to assess the models' ability to solve problems by providing examples:
evaluation process and human evaluators, confirming the reliability and validity of our methodology for assessing the use of exemplification in mathematical problem-solving.
5. Conceptual Finetuning
5.1. Training Data Engineering Framework
Filter-based Data Collection To validate our approach, we conduct supervised fine-tuning to enhance the model's ability to provide examples for conceptual reasoning and incentivize its higher-level mathematical understanding. As shown in Figure 4, we propose an automatic training data engineering framework to obtain training data. Since most LLMs currently cannot provide satisfactory mathematical proofs based on examples, for data collection, we filter data from existing human-writter datasets rather than directly generating counterexample data. Specifically, we apply a data filtering and refinement strategy based on strictly la- heled datasets of propositional statements and proofs. We collect several high-quality mathematical proof datasets, such as ProofNet (Azerbayev et al., 2023) and NaturalProof (Welleck et al., 2021), ensuring that there is no overlap be- tween these datasets and our COUNTERMATH. We use GPT-4o to filter data specifically involving proofs using counterexamples. Our designed data filtering prompt is pre- sented Appendix C. Inspired by AutoRace (Hao al., 2024), which applies LLM as judge, we design several cri- teria to evaluate whether the data utilize counter-reasoning, counter-example for proof, or special examples. To max- imize data retention, we employ loose filtering criterion, retaining data at least one criterion is met. Subsequently, we double-check the filtered data by assessing the com- pleteness of statements, and the rigor and validity of proofs, while discarding incomplete or inconsistent entries. We ultimately obtain 1,025 filtered samples from initial pool of over 30,000 data.
Proportion of Examples: This metric calculates the proportion of problem-solving cases in which the model employs examples as part of its solution.
Strict Align: This measures the percentage of the model's provided examples that are fully consistent with the reference.
This evaluates the proportion of instances where at least one example provided by the model aligns with the reference.
Loose Align:
To further validate the reliability of the proposed LLM-based metrics, we conducted a comprehensive human evaluation. This evaluation focused primarily on two key aspects: (1) the accuracy of example extraction, and (2) the consistency of the evaluation results with human judgement. We selected 100 sample cases and manually reviewed them. First, the accuracy of the example extraction process, as performed by the model, was found to be 97%, demonstrating the robust- ness of our automatic extraction mechanism. Second, the examples-assessed assessed for align- ment with reference examples- achieved an accuracy of 93.5% when compared to human judgements. These results indicate a strong alignment between the model's s automatic
Training Data Refinement We further refine the rationale of the collected SFT data to improve the model's ability to ef- fectively provide examples during training. Compared with our COUNTERMATH, we observe that the SFT data feature longer proof processes and lack explicit illustrations of the
evaluation of the extracted examples-
Saappee-achheeed
5

---

COUNTERMATHL, Counlerexample-rrive Conceptual Reasoning in Mathematical LLMs
Table Main evaluation results of various mainstream mathematical L.L.s with the default CoT prompts COUNTERMATH. The Examples, Strict, and Loose represent the three of our designed example-related evaluation metrics.
Judgement (macro)
Rationale Reasoning
Models
Examples (%)
Striet (%)
Loose (%)
Open source models
Deepseek-Math-7B-l
32.2
65.9
18.9
20.6
Burus-2-BB-PRIME
37.5
64.8
32.0
NuminaMath-77B -B--tthh--BB-RRR--
30.4
54.1
13.0
13.7
MiemLLM2-MathhPlus-77
33.9
36.6
9.0
9.5
Abel-7B-002
34.4
66.1
16.0
17.9
WizardMath-7B-v1..
27.9
43.2
6.4
7.2
size = 7B
Mathstral-7B-v0l
28.2
38,9
MetaMath-Wfistral-75
31.0
26.5
0.4
0.7
7B-V1.0
28.1
31.3
1.2
1.7
tho-matth-7--interrreee
22.3
18.3
1.9
TH2
32.3
54.2
10.7
12.1
Qwen2.5-Math-7B-nnssn
38.3
74.2
30.2
33.2
Ahel-13B-001
22.4
24.4
0.8
0.8
Xwin-Math-1338 -3B-VVL V10
30.2
31.3
1.2
1.7
7B 7B<size <70B
InternLM2-Me
18.4
28.8
8.4
9.5
MAmmoTH22-k<7B-PP
28.8
51.4
14.1
15.5
QwQ-32B QwQ-32B-Preview
39.9
70.0
38.6
43.8
IntemL
LittMMat-PPPutt Plin--Pls--aaee Muu--MixtraB8x22B
37.3
63.2
21.5
23.1
Xwin-Math-70B-VV
25.5
25.2
1.4
1.7
size >=70B
Abel-70B
31.0
48.4
5.3
6.1
WizardMath-70B-v1..
24.2
52.9
6.3
7.4
Owen2.5-Math- -2B-Ma---------ruuu
41.8
76.6
38.9
41.6
Commercial models
GPT-4o
59.0
44.7
19.7
21.3
OpenAl al -reevvvw
60.1
55.8
39.8
40.9
Qwen-max
58.9
61.8
30.4
33,9
Deepseck-RI
80.7
86.8
54.2
65.3
Claude3.7-sonnee
64.8
78.0
52.5
Gemini2.5-pro
77.0
90.8
65.1
75.7
provided examples. To address this, employ GPT-4o to refine the rationales better align with the feature distribu- randomly select
using supervised LoRA training 2xI.20 48GB GPUs, with learning rate of 1.0e-5. After training, evaluate the model on both ATU and several out-of-distribution (OOD) benchmarks, such as MATH(Hendrycks et al., 2021) and GSM8K (Cobbe et al., 2021), OlympaidBench (He et al., 2024), MMLU (Wang al., 2024c), AIME 2024. We aim to demonstrate not only improved performance on the COUN- TERMATH but also enhanced generalization to OOD bench- marks, thereby validating the effectiveness of our insight in improving the model's 's overall conceptual reasoning capa- bilities by examples.
tune 10an2.55-Mah--nsiiie
tion of COUNTERMATE, Specifically, one example from each of the four fields in our COUNTER- MATH as a reference, and provide three manually rewritten aedore-and--fter comparisons to guide the model. Our de- signed data refinement prompt is presented in Appendix C. emphasize that while reducing proof length, the rewrit- ing process should preserve the original reasoning structure to maintain the integrity of the data. This refinement sures that the SFT data closely aligns with the characteristics of COUNTERMATH and explicitly incorporates example usage to facilitate training.
6. Analysis and Discussions
5.2. Training and Evaluation Details
6.1. Evaluation Results without Finetuning
For model training, we select Qwen-2.5 2.5-Math-788 7B-Instruct an open-source model known for its strong mathematical reasoning capabilities and general applicability. we fine-
We selected range of advanced mathematical LLMs with varying parameter sizes to evaluate their conceptual reason-
6

---

COUNTERMATH: Dounerresamlle-Drven Conceptual Reasoning in Mathematical LLMs
olgeôro
ematies reasoning, accurately identifying statements suited for counterexample reasoning and surpassing even the com- mercial ol model. However, many open-source models fail to generate meaningful examples. For instance, Meta- Math and rho produce counterexamples in only 26.5% and 18.3% of cases, respectively, with consistency rates as low as 0.4% and 1. 1.9%. This limitation likely originates from the "Practice lots of math problems" training strategies em- ployed, where most data are derived from elementary and high school mathematics. Such training leaves these mod- els ill-equipped to handle the abstraction and conceptual reasoning demands of higher-level mathematics.
everal
fanctiona ondlyss
topology
reoi, ontlls's
previos
MAmmollte THi
Owen 2.5 mrtth 78 Instruct
DWD 328 prokiew
Quen- Semmt-7i--mmtmet
Figure 5. Fine-grained evaluation results of different fields in COUNTERMATH.
Fine-grained Analysis Across Fields We further con- duct a fine-grained analysis of four mathematical fields in COUNTERMATH, evaluating six advanced models of varying sizes. As shown in Figure 5, the ol model consis- tently outperforms others across all fields, with relatively balanced performance in each. Among open-source models, we observe stronger performance in algebra and functional analysis but weaker results in topology and real analysis. This suggests that topology and real analysis pose greater challenges for LLMs, possibly due to their lower occurrence in training data. These findings highlight the need for future research focus on developing LLMs capable of handling these underexplored fields of mathematical reasoning.
ing abilities our benchmark. Table I summarizes their performance across various metrics. From the results, we derive the following findings and insights:
Judgement Performance The performance on the auto- matic evaluation metric Fl reflects the models' fundamental conceptual reasoning abilities, specifically their capacity to correctly determine the truth or falsehood of a given state- ment. While open-source models exhibit some performance, their overall performance is relatively low around 30. Even achieves only 41.8, falling behind commercial models. Notably, the math- maasoning-ssecifii Deepseck-R1 model outperforms others, achieving an F1 score of 80.7. However, compared to bench- marks focused on elementary and high school mathematies, the overall performance of commercial models is relatively lower. This disparity underscores the challenges posed by our higher-level mathematics benchmark, aligning with our objective of exploring deeper conceptual reasoning. When comparing the performance of models of different sizes, we find that some small 7B models outperform the larger outper- We helieve this phenomenon
the advanced
6.2. Results with Finetuning
Evaluation on Our Benchmark We evaluate the ap- proach on our benchmark to verify whether counterexample reasoning effectively enhances a model's conceptual reason- ing capabilities. In addition to the base model, we include comparisons with the hint prompt, where explicit hints en- couraging LLM to reason by example were provided. For a fair comparison, we use identical prompts to generate out- puts during the experiments. The results indicate that, with just 1,025 training samples, the fine-tuned model outper- forms base models in F1 score. Furthermore, the trained model demonstrates superior example- based conceptual rea- soning abilities, showing improvements in both the quantity and quality of examples compared to base models. On the other hand, although the constructed training data are refined to align closely with our COUNTER COUNTERMATH distri- bution, some discrepancies remain. Since our exploration involves only the simple SFT strategy the model performs slightly worse some metrics, which can be considered an acceptable limitation.
72B model. For example, Deepseek-Math-1 forms WizardMath-70B-- is related to the training methods behind these models. As explained in Section 1, driel-based training in WizardMath- 70B-v1.0 is not very effective in improving the ability to understand deep and complex mathematical concepts. In contrast, the superior performance of Deepseek-Mane -athe--B-r suggests that reinforcement learning may play a key role in enhancing counterexample-driee driven conceptual reasoning.
limited dataset,
Conceptual Reasoning Ability with Examples Beyond assessing binary classification F1 in judgements, we also evaluate the reasoning process of the models. We hope that LLMs provide logical proofs or counterexamples rather than relying on random guessing or copying statements. The Qwen-series models demonstrate superior higher math-
Evaluation on Dut-of-Distribution Benchmarks To as- sess the generalizability of the fine-tuned model, we fur- ther evaluate its performance on out-of-distribution (OOD) benchmarks. This aims to verify whether the model's 's counterexample reasoning capability, which is valid on our
7

---

COUNTERMATH: Counterexample-Drive Conceptual Reasoning in Mathematical LLMs
Table 2. The evaluation results COUNTERMATH.
Models
F1 (macro)
Examples( %)
Strict( %
Loose( %
Base models
Owen2.5-Math- Baach-BB-Instruce
38.3
74.2
30.2
33.2
Qwen2.5-Mand Hint prompt
39.4
79.0
33.1
36.4
Our training model
Qwen2.5-Mait 1eee225-Maahh77--nssrreee
39.7
75.2
31.4
34.7
Qwen2.5-Matt 1aa.----aat----Inssttuttttt TB--Instruct-FFF + Hint prompt
41.1
79.4
31.1
34.7
Table 3. The Out-of-distribution Evaluation Results.
Models
GSM8K
MATH
Olympaidhench-Math
MML.U-collee-Mathh
AIME 2024
Qwen2.5-math-77 In-structtttt
95.1
80.5
41.6
74.0
20.0
Qwen2.5-math: Seaath-72BInnn 12s-Inst
95.4
84.9
49.2
78.3
26.7
95.6
87.9
46.4
80.0
30.0
+Countermath-FFT
COUNTERMATHE can transfer to other benchmarks and deliver broader performance improvements. Using identical prompts and configurations for faimess, compare the base models and fine-tuned models OOD benchmarks. The results reveal that the fine-tuned model outperforms the base models on both benchmarks, even surpassing larger models like the 72B 2a-parrmeter model. This aligns well with our hypothesis: equipping the model with counterexample reasoning ability can enhance conceptual reasoning across the general mathematical domain.
6.3. Used Tokens Analysis
The relationship between model performance and token usage efficiency critical factor in understanding the trade-offs inherent in model design. As depicted in Figure 6, analyzed the connection between the mean token ratios (%), which represent the efficiency of token usage relative to ground truth, and the Fl (macro) score, which reflects the predictive performance of the model. The Mean Token Ratio (%) is calculated by dividing the number of tokens ac- tually used by the model during inference by the number of tokens in the ground truth answer. Reasoning models such as "23=ol-previes" and
Efficiency Figure
utilize significantly large number of tokens during inference, but this extensive token consumption does not lead to corre- sponding improvement in F1 (macro) scores. This suggests that the benchmark task is highly difficult, where simply increasing the length or detail of token reasoning does not necessarily enhance performance. Models like "24=Qwen- max" and "22=GPT-44" demonstrate commendable bal- ance between token usage and performance. These models achieve relatively high F1 (macro) scores while maintaining token consumption close to the ground truth token ratio, This indicates their ability to perform accurate reasoning efficiently, without overly relying on additional token usage.
OeDespoodk 78 Sabang 78
Math
Swhicl 382
F1(macro)
www.weeeellllll wollib
-B-DwC-332 320 Frenlww
23wkbbe 308-280
24-Dwen-ee ever
23wit coewies
Meen Token Ratios(%)
6.4. Case Study
Figure 6. The relationship between Mean Token Ratios (5e and Fl (macro) scores for various models. The red dashed line represents the Ground Truth Token Ratio (100%), ), serving as efficiency benchmark. Models closer to this line are more token- eiken-efficient, while those farther the right consume significantly more tokens.
In our case study, we observe that while the LLM correctly judges mathematical statements, it sometimes provides in- correct examples as shown in Figure Case 1. In addition, verify the validity of our evaluation metric of EXAM- PLE. The LLM generates valid counterexamples that dif-
8

---

COUNTERMATH: Counterexample-rrive. Conceptual Reasoning in Mathematical LLMs
Case of model judge the statement correcly but give wrong examples
sma smalleett number ring containing |sgrt(3) \ is real number ring."
Answer:
False
(Correct)
Model Output
"T determing whether If 11/n0000 17 and I Mipp IJ are linerrly kemorpphic I/ p Unfty Ineg 2 we need to explore the properties of these Banach spaces... The key difference lies properbles like the INe which is deep aspect of their structure and connot be bridged by linear somerphism.
X
Example: The minimal ring containing AI (egrt(3) ) is V (mathab/22[(sentta]] \/m + n)sert(3) \mid m, in VmnatibaiJ)) V, not the real numbers.
Model's Issue: model correctly answered "False" but hallucinated irrelevant concepts (e.g. Banach spaces, RNP) unrelated ring theory.
LLM-produced counterexamples which do not appear in reference, and our evaluation identify the validity
Statement: For fixed element Ma') ring MRM, VN Yra Vin RVAI forms an of (RU) but does not necessarily contain Mal)
Reference Example:
(RR) as the ring oven numbers, VIN- M[textfall) )r /cdot 1mmdd r Vin RX/ is an ideal; however, \/4 'nottn M) because (RR) lacks, an identity element.
LLM Output (Simglified]: The model verilies that (NV) is an ideal and argues that Via Inotin MI by considering a trivial ring VR- MOVA).
Evaluation: The LLM's counteroxample differs from the reforence but is marked as "consistent" because it satisfles the evaluation criteria. Our judge model succesafully extracts and assesses these counteresapplcs, determining their consistency with the intended logic.
Case of how SFT model improve in OOD bechmark
Question: Sallie grade 583 WSS, 596 (1E5, 581 (18S, $821 What sum the percent the two missing slades?
marking
equa by weighed
test pepers she find have grades
Answer:
Base Model,
SFT Model:
208
198
Key Steps: Sets up equation: V/ (raa + 96 + 81 82 yH6) - 90
347 e y 540 V) y 540. 342 208 V. producing ginigg an incorrict sum,
Kny the answer.
Steps: Same setup but correctly computes \\ 540 342 = 198 M, aligning
subtraction (540- 342-100 2-199, not 208), leading
Amalysis: Demonstrates precision in arithmetic operations logical
critical error.
Figure 7. Our case studies of the performance of LLM include: (1) correct judgment with incorrect examples, (2) validation of novel but effective counterexamples, and (3) performance improvement on MATH dataset after counterexample training
fers from reference ones, yet are correctly validated by our judge model, highlighting the effectiveness of our evaluation method. Finally, we compare the improvement of MATH benchmark between base and SFT model, demonstrating the benefit of incorporating counterexample traning enhance mathematical reasoning capabilities.
Acknowledgements
This research is supported by National Natural Sci- ence Foundation of China (Grant No. 62276154), Re- search Center for Computer Network (Shenzhen) Min- istry of Education, the Natural Science Foundation of Guangdong Province (Grant No. 1023A15150120
and 44030024103310000011 JCYI2021032412000 2033,
Basic Research Fund of
7. Conclusion
Shenzhen City (Grant No.
ICYJ2024081311200000 and 01172002202181113000000 the Major Key Project of PCL (NO. PCL2024A08). the Key- Area Research and Development Program of Guangdong Province No.2024B1111 part by NSF under grants II1-2106758, and POSE-2346158,
In this work, we address the limitations of drill-based learning in mathematical LLMs by introducing COUNTER- MATH, a counterexample-benee based reasoning benchmark. Un- like existing datasets, COUNTERMATH evaluates models on their ability to distinguish nuanced mathematical con- cepts through example-driven reasoning. Our key contri- butions include constructing a high-quality dataset with 1,216 university-level counterexample-basee proofs, bench- marking otate-of-the-ar art mathematical LLMs to reveal their conceptual reasoning gaps, and developing an automated framework for counterexample data generation and train- ing. Experimental results show that current LLMs strug- reasoning, particularly in
This work is supported in
Impact Statement
This paper introduces conceptual mathematical benchmark aimed at advancing research on Large Language Models (L.LMs) in the realm of genuine mathematical reasoning. The dataset utilized in this work derived from publications with copyrights reserved and authors' permission only for academic research purposes. It is important to acknowledge that experiments and evaluations rely heavily on LLMs, but this study does not fully explore or mitigate potential biases inherent in their outputs. Addressing these biases and ensuring model alignment with social values remain critical challenges. This underscores the importance of conducting comprehensive evaluations that consider diverse dimensions of human society and their implications.
gle with topology and real analysis, highlighting areas for future re- search. Furthermore, our fine-tuned model, trained on only 1,025 examples, significantly outperforms baseline mod- els, demonstrating the effectiveness and generalizability of counterexample-drivee learning in mathematical reasoning.
9

---

COUNTERMATH:
Counterexample-rrive Conceptual Reasoning in Mathematical LLMs
References
Hao, S., Gu, Y., Luo, H., Liu, T., Shao, X., Wang, X., Xie, S., Ma, H., Samavedhi, A., Gao, Q- et al. Llm reasoners: New evaluation, library, and analysis of step- by-step reasoning with large language models. arXiv preprint arXiv:2404.0522, 2024.
Amini, A., Gabriel, S., Lin, P., Y., and Hajishirzi, H. Mathqa: Towards interpretable math word problem solving with operation based formalisms. arXiv preprint Xrviv: / 13319, 2019.
R., Choi,
Azerbayev, Z., Piotrowski, B., Schoelkopf, H., Ayers, E. W., Radev, D.; and Avigad, J. Proofnet: Autoformalizing and formally proving undergraduate-leved mathematics. doi: 10.48550/ARXIV,
He, C., Luo, R., Bai, Y., Hu. S., Thai, Z. L., Shen, J., Hu, J., Han, X., Huang, Y., Zhang, Y., Liu, J., Qi, L., Liu, Z., and Sun, M. Olympiadbench: challenging benchmark for promoting agi with olympiad-leve bilingual multimodal scientific problems, 2024.
CoRR, abs/2302. 12433, 2023. 2302.12433.
Chen, W., Ma, X., Wang, X., and Cohen, W. W. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Transactions on Machine Learning Research, 2023.
Hendrycks, D., Burns, C.,, Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Mcasuring math- ematical problem solving with the math dataset. NeurlPS, 2021.
Cheng, K., Yang, J., Jiang, H., Wang. 7.,, Huang. B., Li,
Hu, C. Counterexamples in Algebra. Shaanxi Science and Technology Press, 1983.
R., Li, S., Li, Z., Gao, Y., Li, X., et al. Inductive or deductive? rethinking the fundamental reasoning abilities of llms. arXiv preprint arXiv:2408.000 001144 2024.
Huang, S., Ma, S., Li, Y., Huang, M., Zou, W., Zhang, W., and Zheng, H. Lateval: An interactive Ilms evalua- tion benchmark with incomplete information from lateral thinking puzzles. In Calzolari, N., Kan, M.. Hoste, V., Lenci, A., Sakti, S., and Xue, N. (eds.), Proceedings of the 2024 Joint International Conference on Computa- tional Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy, pp. 10186-10197. ELRA and ICCL, 2024. URL https: / agy . org/ 2024. prec-mate inn 8.9.
Chern, E., Zou, H., Li, X., Hu, J., Feng, K., Li, J., and Liu, P. Generative ai for math: Abel. https: github. com/GA TR-NLP / a 2023.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, I.., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers solve math word problems. arXiv preprint arXiv:2110. 2021.
Cui, G., Yuan, L., Wang, Z., Wang, H., Li, W., He, B., Fan, Y., Yu, T., Xu, Q., Chen, W., Yuan, J., Chen, H., Zhang, K., Lv, X., Wang, S., Yao, Y., Peng, H., Cheng, Y., Liu, Z., Sun, M., Zhou, B., and Ding, N. Process reinforcement through implicit rewards, 2025.
Jiang. A. Q.. Welleck, S., Zhou, P., Lacroix, T., Liu, J., Li, W., Jamnik, M., Lample, G., and Wu, Y. Draft, sketch, and prove: Guiding formal theorem provers with informal proofs. In The Eleventh International Conference Learning Representations, 2023.
Frieder, S., Pinchetti, L.,, Chevalier, A., Griffiths, R.-R., Salvatori, T., Lukasiewicz, T., Petersen, C., and Berner, J. Mathematical capabilities of chatgpt, 2023.
Klymehuk, S. Counterexamples in Calculus. 01 2010. ISBN 9780883857656. doi: 10. 10.1090/clm/034.
Frieder, S., Berner, J., Petersen, P., and Lukasiewicz, T.
Li, C., Wang. W., Hu, J., Wei, Y., Zheng, N., Hu, H., Zhang, Z. and Peng, H. Common 7b language models already possess strong math capabilities. arXiv preprint arXiv:2403.04706 2024a.
Large language models for mathematicians, 2024.
Google, G. T. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of contexts, 2024.
Gou, Z., Shao, Z., Gong, Y., Yang, Y., Huang. M., Duan, N., Chen, W., et al. Tora: A tool-integrated reasoning agent for mathematical problem solving. In The Twelfth International Conference Learning Representations, 2024.
Li, J., Beeching, E., Tunstall, L., Lipkin, B., Soletskyi, R., Huang, S., Rasul, K., Yu, L., Jiang, A. Q., Shen, Z., et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9, 2024b.
Gulati, A., Miranda, B., Chen, E., Xia, E., Fronsdal, K., de Moraes Dumont, B., and Koyejo, S. Putnam-AXIOO AXIOM: A functional and static benchmark for measuring higher level mathematical reasoning. In The 4th Workshop on Mathematical Reasoning and Al at NeurlPS 24, 2024.
Li, Y., Xu, Z.,, Chen, S., Huang, H., Li, Y., Jiang, Y., Li, Z., Zhou, Q., Zheng, H.-T., and Shen, Y. Towards real- world writing assistance: A chinese character checking benchmark with faked and misspelled characters. arXiv preprint arXiv:231 1.1 2023.
10

---

COUNTERMATH: Countersxaple--Drven Conceptual Reasoning in Mathematical LLMs
Li, Y., Du, D., Song, L., Li, C., Wang. W. Yang, T., and Mi, H. Hunyuanprover: A scalable data synthesis framework and guided tree search for automated theorem proving. arXiv preprint arXiv:2412.20733 2024c.
deductive reasoning capacity of large language models using OOD examples. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Shao, Z., Wang. P., Zhu, Q. Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y., Wu, Y., al. Deepseekmath: Push- ing the limits of mathematical reasoning in open language models. arXiv preprint arXivv2402.03300, 2024.
Li, Y., Zhou, Q. Luo, Y., Ma, S., Li, Y., Zheng, H.-T., Hu, X., and Philip, S. Y. When Ilms meet cunning texts: A fallacy understanding benchmark for large language models. In The Thirt Conference on Neural In- formation Processing Systems Datasets and Benchmarks Track, 2024d.
Simon, M. A. Studying mathematics conceptual learn- ing: Student learning through their mathematical activity. North American Chapter of the International Group for the Psychology of Mathematics Education, 2011.
Li, Y., Ma, S., Chen, S., Huang, H., Huang, S., Li. Y., Zheng, H.-T., and Shen, Y. Correct like humans: Progres- sive learning framework for chinese text error correction. Expert Systems with Applications, 265: 126039, 2025.
Tsoukalas, G., Lee, J., Jennings, J., Xin, J., Ding, M., Jen- nings, M., Thakur, A., and Chaudhuri, S. Putnambench: Evaluating neural theorem-provers on the putnam math- ematical competition. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
Lin, H., Sun, Z., Yang. Y., and Welleck, S. Lean-star: Learn- ing to interleave thinking and proving. arXiv preprint 2rXiv:2407.10040 2024a.
Lin, Z., Gou, Z., Gong, Y., Liu, X., Shen, Y., Xu, R., Lin, C., Yang, Y., Jiao, J., Duan, N., et al. Rho-1: I: Not all tokens are what you need. arXiv preprint arXiv:2404.0777 07965, 2024b.
Wang, L. Counterexamples in Real Analysis. Higher Educa- tion Press, 1989.
Wang, L. Counterexamples in Functional Analysis. Higher Education Press, 1994.
Liu, H., Zheng, Z.. Qiao, Y., Duan, H., Fei, Z., Zhou, F., Zhang, W., Zhang, S., Lin, D., and Chen, K. Mathbench: Evaluating the theory and application proficiency of Ilms with hierarchical mathematics benchmark, 2024.
Wang, L. and Yang, F. Counterexamples in Topology. Sci- ence Press, 2000.
Wang, R., Zelikman, E., Poesia, G., Pu, Y., Haber, N., and Goodman, N. Hypothesis search: Inductive reasoning with language models. In The Twelfth International Con- ference on Learning Representations, 2024a.
Lu, P., Bansal, H., Xia, T., Liu, J.. Li, C., Hajishirzi, H., Cheng, H., Chang, K.-W., Galley, M., and Gao, J. Math- vista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024.
Wang, W., Fang, T., Shi, H., Xu, B., Ding, W., Zhang, L., Fan, W., Bai, Li, H., Liu, X., et al. On the role of entity and event level conceptualization in generalizable reason- ing: A survey of tasks, methods, applications, and future directions. arXiv preprint arXiv:2406. 10885, 2024b.
Luo, H., Sun, Q. Xu, C., Zhao, P., Lou, J., Tao, C.,, Geng, X., Lin, Q., Chen, S., and Zhang, D. Wizard- math: Empowering mathematical reasoning for large lan- guage models via reinforced evol-instruct. arXiv preprint wxXiv:2308.09583 2023.
Wang, Y., Ma, X.,, Zhang, Ni, Y., Chandra, A., Guo, S., Ren, W., Arulraj. A., He, X., Jiang, Z., et al. Mmlu-pro: A more robust and challenging multi-task language un- derstanding benchmark. Advances in Neural Information Processing Systems, 37:95266 -92220, 2024c.
Mirzadeh, I,, Alizadeh, K., Shahrokhi, H., Tuzel, O.,, Bengio, S., and Farajtabar, M. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint arXiv:2410.05222 2024.
Wei, J., Wang. X., Schuurmans, D., Bosma, M., Xia, F., Chi, F., Le, Q. V., Zhou, D., et al. Chain-of-thouge prompting elicits reasoning in large language models. Advances in neural information processing systems, 01144244-44831 2022.
Moura, L. d. and Ullrich, S.
The lean 4 theorem prover and programming language. In Platzer, A. and Sutcliffe, G. (eds.), Automated Deduction - CADE 28, pp. 625-635, Cham, 2021. Springer International Publishing. ISBN
978-3-030-79876-5.
Welleck, S., Liu, J., Bras, R. L.,, Hajishirzi, H., Choi, Y., and Cho, K. Naturalproofs: Mathematical theorem proving in natural language. In Vanschoren, J. and Yeung, S. (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks I, NeurlPS
OpenAI. Gpt-4 technical report, 2023.
Saparov, A., Pang, R. Y., Padmakumar, V., Joshi, N., Kazemi, M., Kim, N., and He, H. Testing the general

---

COUNTERMATHL, Coumterexample-rrkee. Conceptual Reasoning in Mathematical LLMs
Dalasets and Benchmarks 2021, December 2021, virtual, 2021.
Yue, X., Zheng, T., Zhang, G., and Chen, W. Mam- moth2: Scaling instructions from the web. arXiv preprint urXiv:2405.03548 2024.
Wu, W. and Gao, X.-S. Mathematics mechanization and applications after thirty years. Frontiers of Com-
Zheng, K., Han, J. M., and Polu, S. minif2f: cross-system benchmark for formal olympiad-level mathematics. In International Conference on Learning Representations, 2022.
puter Science in China, 1:1-8, 02 2007. doi: 10.1007/
s11704-07-00001-8
Wu, W.t. Mathematics Mechanization: Mechanical Geom- eiry Theorem-Proving Mechanical Geometry Problem- Solving and Polynomial Equatiins-Solving Mathematics and Its Applications. Springer Netherlands, 2001. ISBN 9780792358350.
Thou, B., Zhang, H., Chen, S., Yu, D., Wang, H.,, Peng, B., Roth, D., and Yu, D. Conceptual and unbiased reasoning in language models. arXiv preprint 2024.
Wu, Y., Liu, J., Bu, X., Liu, J., Zhou, Z.., Zhang, Y., Zhang, C., Bai, Z., Chen, H., Ge, T., et al. Conceptmath: A bilin- gual concept-wise benchmark for measuring mathemati- cal reasoning of large language models. arXiv preprint 0rXiv:2442.14660 2024.
Xu, Z., Li, Y., Ding, R., Wang, X., Chen, B., Jiang, Y., Deng, X., Ma, J., Zheng, H.-T., Lu, W., et al. Let lims take on the latest challenges! a chinese dynamic question answering benchmark. arXiv preprint arXiv:2402.19248, 2024.
Yang. A., Yang, B., Hui, B., Zheng. B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 technical report. arXiv preprini arXiv:2407. 10671, 2024.
Yang, K. and Deng, J. Learning to prove theorems via inter- acting with proof assistants. In International Conference Machine Learning, pp. 6984-6994, PMLR, 2019.
Yasunaga, M., Chen, X., Li, Y., Pasupat, P., Leskovec, J., Liang, P.. Chi, E. H., and Zhou, D. Large language models as analogical reasoners. In The Twelfth International Conference on Learning Representations, 2024.
Ying, H., Zhang, S., Li, I.., Zhou, Z., Shao, Y., Fei, Z.., Ma, Y., Hong, J., Liu, K., Wang, Z., et al. Internim- math: Open math large language models toward verifiable reasoning. arXiv preprint arXiv:2402.063322 2024.
Yu, L., Jiang, W., Shi, H., Yu, J., Liu, Z., Zhang, Y., Kwok, T., Li, Z.., Weller, A., and Liu, W. Metamath: Boot- strap your own mathematical questions for large language models. arXiv preprint 2.KIv:23009122288. 2023.
Yu, L.. Jiang, W., Shi, H., Jincheng, Y. Liu, Z., Zhang, Y., Kwok, J., Li, Z., Weller, A., and Liu, W. Metamath: Bootstrap your own mathematical questions for large lan- guage models. In The Twelfth International Conference on Learning Representations, 2024a.
Yu, X., Zhou, B., Cheng, H., and Roth, D. Reasonagain: Using extractable symbolic programs to evaluate math- ematical reasoning. arXiv preprint arXiv:2410.19056. 2024b.
12

---

COUNTERMATHI, Counteresample-Drivee Conceptual Reasoning in Mathematical LLMs
A. Details for Data Curation
Annotation Cost In the first-stage annotation, there were three annotators recruited who have at least bachelor's degrees in engineering or science-related majors. On average, each of them has annotated about 416 statement-rational pairs from the given textbooks. They were working through the vendor's provided platform with the provided OCR tool and annotation examples. The cost for the first-stage crowd-sourced annotation was about S1306.
Annotation Process The annotation process consists of two stages. In the first stage, the recruited annotator were asked to annotate statement, rat ratiel (i.e. the response for supporting the statement, typically examples or counterexamples), field (i.e. algebra, topology, real analysis or functional analysis), and txt (i.e. mappings original annotations for the validation use). During annotation, they were provided with several annotation examples created by authors, demonstrating the annotation targets. Moreover, we also asked annotators to focus on statements related to proving or disproving and existence of certain mathematical objects and ignore those statements without clear or complete answers such as definitions for certain advanced and complicated concepts. For the second stage, authors with at least bachelor's degrees in applied mathematics were checking the LaTeX formats and typos in statements and rationales, keeping the rationales that correct and clearly support or deny the statements with examples or counterexamples. Besides, we also added another element in annotated data points, judgement, to show whether the statement is true false by its rationale. One full annotation example is shown in Figure 8. It should be noted that we have modified some statements (less than 5 5%), which are easily revised to be the reverse, to make them False for diversity because nearly all statements are stated as True in the original textbooks. In all, we tried to make sure that all statements and rationales were concise and related.
field: real analysis
statement: A differentiable function f on (0,1), if $\\lim fx Wrightaarrw 0+} f(x)=\linfty $, then $1llim (x Wrightarrow 0+) fNMNprimm) Winftys [holds)
25.
(0, 1) 1: 1
f00) (B linf (N)
# ft
i8t
((x) 11/s-11/ (1)))))
M1
lim f(x)- oo.
modification (optional): "does not hold" (original) to "holds"
X
(2(x) (1//2 O<FKL
# s.- 1/ (2 1),,, (. HE
(e) 00 *
AR a.
judgement: False
rationale: Let $f(x)=1 / x+\\cos /1/ x), 0cx=1,$ then $A _x Wrightarrow 0+) f(x)=+\linftys and $\\quad f (xxxprrmee(()=(((xs))) (1 xx-11/ / x*(2), Viquad 0<x<1 $. If we let x_(n]=1 Wleft(( nN\fractuH2A/right Wpis, then $ P<MWprime)\ (xteNleetxx___etxl (eft(( Milm fx Wrrightarrw 0+} (x) $ does not hold.
$. Therefore, $
Figure 8, An annotation example from Counterexamples in Real Analysis. (Wang, 1989).
B. Details for Experimental Settings
Summary for Open-weight Baselines The summary of the open-weight baseline models is shown in the following Table 4. From Table 4, it is evident that current math-focused LLMs are built on variety of base models, with Mistral being the most commonly used, followed by Llama2, Qwen2-Math and Deepseek-Math. Notably, most models utilize data generation and augmentation strategies centered around MATH and GSM8K, which has accelerated the saturation of these benchmarks and highlighted the limitations of current "mathematical reasoning" capabilities in LLMs.
Furthermore, most academia-developed models are often constrained supervised fine-tuning (SFT) only due to limited computation resources. Recent efforts, such as Eurus-2-PRIME and Rho-Math, have begun explore advanced pre- training and post-training techniques, which have been (implicitly) validated by companies as effective in enhancing mathematical reasoning. However, the opacity surrounding engineering details in technical reports (OpenAI, 2023; Shao et al., 2024; Yang et al., 2024) hinders progress toward genuine mathematical reasoning in LLMs, which is critical for helping math researchers with true mathematical research.
Consequently, our proposed conceptual mathematical benchmark, COUNTERMATH, is timely and significant
13

---

COUNTERMATHE: Counterexamlle-rriee Conceptual Reasoning in Mathematical LLMs
contribution to fostering advancements in genuine mathematical reasoning of LLMs.
Details Prompts
The used prompts are summarized as follows. We follow the corresponding tokenizers for chat- template-based maasette-baeddaee prompts with the instruction as system prompt and statement as user input.
Completion Prompt
{statement]
Please reason step by step about whether above statement is or False, and put your final answer within [boxed(].
Alpaca Prompt
Below is instruction that describes task. Write response that appropriately completes the request.
#### Instruction:
{statement)
Please reason step by step about whether the above statement is True or False, and put final answer within \/boxed
#### Response:
Chat Template- -aase Prompt
<BOS >ssystem
Please reason step by step about whether the statement is True or False, and put your final answer within [boxee() <EOS> <BOS >user
(statement) (oatemen)><EOSS <EOS>
Prompt with Hint
(statement)
Please reason by giving examples about whether the above statement is True or False, and put your final answer within \
!boxed.
Details of Evaluation Prompt The evaluation prompts used are as follows:
14

---

COUNTERMATH: Coumeereaample-rivver Conceptual Reasoning in Mathematical LLMs
Evaluation Prompt
Role
You are professional evalustor responsible for usessing the vorreciness of students" use of proof by contradiction solve mabeeaatical proof problems.
#* Objective
Extract examples in the reference solution.
Extract all examples presented the student's prof.
Compare and determine whether the examples provided by the student's s proof are semantically equivalent to those in the reference solution.
MA Workllow
understand given mathematical statement
2. Extract all examples from the reforence solution.
Extract all examples from the student's proof. the student does not provide any examples during the proof process, please output "None" the Ath (Examples Student's Proof ).
4. Compare each etample provided by the student's s proof with the examples the reference solution, determining whether they convey the same meaning. If they are ouiput ENCUNNISTENCY.
semantically equivalent, ouiput CONSISTENCY:
Pocus only on counterexamples used within the proof contradiction. ignoring other proof methods.
Ensure the accuracy of the comparison to avoid misinterpreting the student's s intended meaning.
## Oulput Format
When student provides examples proof, stating consider example to follows may lead presentation example:
44 <Examples in ReCerence>
Excamlee ...... Example
Example 3: ......
4*# <Examples in Student's a Proofy Example ...... Exemple 2:
44d <Cach Result in Examples in Student's Proof> Example 1: CONSISTENCY
Example UNCONSISTENCY
When the student does not provide any examples:
**# <Examplss in Reference>
Example ...... Example
Example 3:
44d <Examples in Student's Proot>
Result Exemples Student's
Nore
#6 Input
44d Question:
Please judge whecher the following statement True False: (statement
44 Reference Anawer: (answer)
4+# Reference Proof: [rationsle:
Student Proof: (nodel output)
15

---

COUNTERMATH: Couneereaample-rriver Conceptual Reasoning in Mathematical LLMs
Models
Scale
Base Models
Training Data
Training Paradigms
Deepseek-Mand -MthhRR
Deepseek-Matte MaateeekMMtt--Baee
from Common Crawl,
Math corpus
CP, SFT, GRPO
CoT/POT/TIR math-
ematical instruction data, CoT data related to MATH and GSM8K
Owen2.5-Math--Inse
7B/72B
Qwen2.5-Math Base
Math corpus including Common Crawl and syn- thetic data, synthetic CoT/TIR math data
CP, SFT, GRPO
IntemL.MD-Math-PPM
7B/20B/8x22B
Math corpus from Com- Crawl, in-house data and synthetic data, Open math-related in- struction data
CP.
WizardMath
7B/70B
Mistral/Llama2
Data augmentation based on MATH and GSM8K
SFT, PPO
Abel
7B/13B/70B
Llama-2
Data augmentation by SFT Parental Oversight
Xwin-Math
7B/13B/70B
Llama-2
Data augmentation based on MATH and GSM8K
SFT
MAmmoTH2-Phus
7B/8x7B
Mistral/Mixtral
WebInstruet + addi- SFT tional instruction tuning datasets, including Math- Plus, Code-Feedback, etc.
Eurus-2-PRING
7B
Owen2.5-Math-Bant -ase
Reasoning-related in- struction data from open datasets, Curated RL data from NuminaMath. Codeforces, etc.
SFT, PRIME
Numina-Maie Math-TIR
7B
Deepseek-Mate Base
NuminaMath
SFT
Mathstral
7B
Mistral
NuminaMath
SFT
MetaMath-Mistral
7B
Mistral
Data augmentation based
SFT
on MATH and GSM8K
Rho-math-interpreeer
Mistral
OpenWebMath& general corpora. data)
SLM, SFT
ToRA(TIR
QwQ-Preview
32B
Qwen2.5'
Table Summary of open-weigh baseline models. CP stands for Continue Pretrain. SFT stands for Supervised Fine-Tuning. GRPO refers to variant of PPO, which replaces the value network with the group average (Shao et al., 2024). PoT (Chen et al., 2023) and TIR (Gou et al., 2024) stand for Program-of-Thougee and Inte-Ented Reasoning, respectively. PRIME stands for using ORM PRM by DPO-like rewards (Cui et al., 2025). SLM stands for Selective Language Modeling (Lin et al., 2024b), only stands for the same model architecture.
16

---

COUNTERMATHE: Counteresample-rrivee Conceptual Reasoning in Mathematical LLMs
C. Details for Training Data Engineering Framework
In constructing our training dataset, we utilized GPT to filter and refine the data. Below, we outline the specific prompt designs used in this process:
Data Filtering We developed multiple criteria for the LLM to evaluate whether the original data qualified as the counterexample data required for our task. These criteria ensured that the selected data met the conceptual and structural requirements of counterexample reasoning. The specific criteria are as follows:
Data Filtering Prompt
Instruction
Please assess the following mathematical proof based these criteria determine if it employs counterexamples.
# Criterion
## Criterion
Does the proof involve assuming the negation of the conclusion and deriving contradiction from it? This indicates the of proof by contradiction.
## Criterion 2:
there specific instance provided that directly refutes general assertion? This suggests the of counterexample.
## Criterion
Does the explanation include specially chosen cases whose characteristics are essential for illustrating concepts or validating arguments? This points towards the use of special cases.
at one criterion is satisfied, consider as meeting condition mark True. Remember, even only one criterion applies, please retain this data.
# Input
The rationale you need to judge:
}
(rationale) }
Data Refinement We design prompts for the LLM to rewrite the data, making the training dataset more closely aligned with the benchmark distribution. This refinement process also highlights examples more explicitly. enabling the model to better learn counterexample-base conceptual reasoning. The details of our prompt design are provided below:
17

---

COUNTERMATH: Counterexample-Drive Conceptual Reasoning in Mathematical LLMs
Data Refinement Prompt
Task:
You are tasked with modifying the proof processes in given rationales.
# Instructions:
## Annotate Examples:
Identify and annotate any specific examples used within the proof process.
#i Follow Reference Examples:
Use the reference examples and modified hefore-and-after examples provided below guideline for how modify the text.
## Preserve Proof Content:
Focus reducing the length of the proof without altering its core content or logical flow.
Example References Field algebra: [Example A]
Field real analysis: [Example B]
Field functional analysis: [Example C]
Field topology: [Example D]
Modified Before-and-Afic Examples:
## Example 1:
Before:
After:
## Example 2: Before: After:
## Example
Before:
After:
# Input: (rationale}
18


Reviews:
[{"review_id": "y5A178TSL3", "reviewer": "Reviewer_H7Ry", "summary": "The paper studies the capability of LLMs in providing counterexamples for mathematical proofs. Specifically, the paper proposes a benchmark of university level natural language theorem statements along with their rationale and correctness. The authors evaluate a wide range of LLMs on the F1 score and the counterexample usage frequency. Experimental results suggest the benchmark is quite challenging and finetuning on counterexample proof samples enhance reasoning capability on standard benchmarks like GSM8K and MATH.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer H7Ry  \nThank you for your efforts in the review, and hope our response will address your concerns.\n\n> On Line 200-202, Evaluation Metrics. \"We use lexical matching such as F1 to match the judgments of the statements.\" What is lexical matching? Also, how exactly is F1 calculated here?  \n\nWe apologize for the ambiguity. **\"Lexical matching\" refers to extracting the model's final judgment (True or False) using regular expressions.** Specifically, we parse the model's output to identify the final decision and map \"False,\" \"True,\" and empty responses to numerical values (0, 1, and 2, respectively). The F1-score is then computed as a macro-averaged multiclass F1, where each class's F1-score is calculated as:  F1 = 2 * (precision * recall) / (precision + recall). We implement this using the `evaluate` library's F1 function, treating the empty response as a separate class. **The final reported score is the macro-averaged F1 across all classes.**  \n\n> Why are strict and loose align meaningful metrics to evaluate? I imagine there could exist many valid counterexamples for a given statement. Both strict and loose align seem to suggest that the model should really provide counterexamples given by textbooks. Is this too restrictive?\n\nOur evaluation aims to assess whether the model can generate examples that support its reasoning. During evaluation, we use GPT-4o to verify whether the model-generated examples lead to the same final conclusion as reference examples. **This approach allows for flexibility: the generated examples do not need to match the reference ones exactly but should be logically consistent in leading to the conclusion.** Thus, our metric captures the model’s reasoning ability rather than enforcing rigid textbook-style counterexamples.  \n\n> Can the authors provide more details on which subject or what type of questions the SFT model improves the most on MATH? \n\nWe conduct a case study on 50 instances where the base model failed but the SFT model succeeded. Since the MATH dataset lacks explicit problem type annotations, we manually annotate and categorize them. The most significant improvements are observed in:  \n\n1. **Geometry:** Problems requiring length and area calculations. The base model struggles with triangle proportions and spatial reasoning.  \n2. **Number Theory:** Tasks like repeating decimal periods and divisibility checks, where errors stem from incorrect cyclic period calculations or overlooked edge cases.  \n3. **Algebra:** Problems involving averages, fractions, and arithmetic, where the base model frequently miscalculates.  \n\nThe SFT model demonstrates a better understanding of mathematical concepts, more effective application of formulas, and improves accuracy in calculations. Here is an example: \n\n**Question:** Sallie earned a grade of exactly 90% based on six equally weighted tests. The four test papers she found have grades of 83%, 96%, 81%, and 82%. What is the sum of the two missing grades?  \n**Correct Answer:** 198  \n\n- **Base Model:** Predicted **208**  \n  - Step: \\( \\frac{83 + 96 + 81 + 82 + x + y}{6} = 90 \\) → \\( 342 + x + y = 540 \\)  \n  - Error: Miscomputed \\( 540 - 342 = 208 \\) (correct value is **198**).  \n- **SFT Model:** Predicted **198**  \n  - Step: Same, but correctly computed \\( 540 - 342 = 198 \\).", "claims_and_evidence": "The claims are supported by experimental results.", "methods_and_evaluation_criteria": "The proposed benchmark is interesting and tackles a unique angle in mathematical reasoning. The evaluation metric of F1 score makes sense for a highly unbalanced dataset. However, the strict and loose matching criteria seem to be not appropriate especially because there might be many correct counterexamples that are different from the counterexamples provided by the textbooks.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "The experimental design is reasonable and the authors evaluate a wide range of open-source and proprietary LLMs.", "relation_to_broader_scientific_literature": "The contribution of the paper is relevant and timely.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "One strong experimental result is that the authors show that by finetuning on counterexample training data, the model improves significantly on MATH. which is very encouraging.", "other_comments_or_suggestions": "Please see questions below.", "questions_for_authors": "1. On Line 200-202, Evaluation Metrics. \"We use lexical matching such as F1 to match the judgments of the statements\". What is lexical matching? Also how exactly is F1 calculated here?\n\n2. Why are strict and loose align meaningful metrics to evaluate? I imagine there could exist many valid counterexamples for a given statement. Both strict and loose align seems to suggest that the model should really provide counterexamples given by textbooks. Is this too restrictive?\n\n3. Can the authors provide more details on which subject or what type of questions the SFT model improves the most on MATH?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mGSUM8VFHE", "reviewer": "Reviewer_KrNF", "summary": "The paper introduces a benchmark called COUNTERMATH. It is designed to assess the ability of LLMs to reason about mathematical statements and justify them using counterexamples. The dataset comprises statement-rationale pairs, sourced from math textbooks, and undergoes manual filtering to ensure quality. The evaluation framework includes F1 for assessing the correctness of model judgments on math statements and introduces three new metrics: (a) proportion of examples, (b) strict alignment, (c) loose alignment. The three metrics are used to measure how effectively models use counterexamples in their reasoning.\n\nThe paper presents a data engineering framework for obtaining counterexample-based math proofs for fine-tuning. This framework leverages GPT-4o to filter and refine proofs to ensure that the training data aligns with the conceptual reasoning requirements of COUNTERMATH.\n\nThe paper evaluates a range of open-weight and proprietary models on COUNTERMATH, revealing that open-weight models have significantly lower performance than proprietary models. The proprietary models still perform poorly compared to benchmarks focused on simpler mathematical problems. Furthermore, the results demonstrate that fine-tuning on counterexample-based math proofs, combined with hint-based prompting, leads to an improvement in F1 in COUNTERMATH, outperforming the base model.\n\n## update after rebuttal:\nI will maintain my score of 2. The model fine-tuned on counterexample based math proofs shows only marginal improvements over model fine-tuned on non-counterexample proofs, and this doesn't sufficiently demonstrate that learning from counterexamples is valuable for improving model's math abilities.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer KrNF\n\n> No comparison with model fine-tuned on dataset of non-counterexample-based math proofs.\n\nWe agree that such a comparison would further validate our approach. Since our primary motivation was to explore how LLMs use counterexamples in proofs, we initially evaluated only counterexample-based fine-tuning. To address your suggestion, we randomly selected 1,025 non-counterexample samples from the NaturalProofs dataset (the same source as our counterexample data) and fine-tuned Qwen2.5-Math-Instruct under identical settings. Results are as follows:\n\n| Model                              | Overall F-1 | Examples (%) |\n|------------------------------------|-------------|--------------|\n| Base Model                         | 38.3        | 74.2         |\n| Counterexample SFT Model           | **39.7**    | **75.2**     |\n| Non-counterexample SFT Model       | 37.9        | 74.6         |\n| Base Model (Hint prompt)           | 39.4        | 79.0         |\n| Counterexample SFT Model (Hint)    | **41.1**    | **79.4**     |\n| Non-counterexample SFT Model (Hint)| 39.9        | 78.4         |\n\n**The superior performance of counterexample-based models supports our claim that improvements stem from learning counterexamples rather than increased exposure to advanced data.**\n\n> The current evaluation prompt may overly penalize valid and mathematically sound counterexamples.\n\n**Our evaluation prioritizes whether generated examples lead to the same final conclusion as the reference.** While the examples need not match exactly, they are marked as “consistent” if they function equivalently in proving the statement.\n\n> Concerns about GPT-4o's evaluation.\n\nGiven the current challenges of human annotation in terms of cost and efficiency, **our approach aligns with prior works about GPT-4o evaluation** (e.g., DPO[1], SimPO[2]) and established benchmarks like MT-Bench[3] and Alpaca-Eval[4]. To enhance accuracy, we provided reference information for GPT-4o and **our human-validated results further confirm the approach's reliability**.\n\n[1] Direct Preference Optimization: NeurIPS 2023  \n[2] SimPO: Simple Preference Optimization: NeurIPS 2024  \n[3] MT-Bench: NeurIPS 2023  \n[4] AlpacaEval-LC: COLM 2024  \n\n> Concerns about the generalizability of SFT model to more challenging OOD datasets.\n\nWe extended evaluations to **competition-level** (OlympiadBench-Math, AIME 2024) and **college-level** (MMLU-College-Math) datasets: \n| Model | GSM8K | MATH  | OlympiadBench-Math | MMLU-College-Math | AIME 2024 |\n|-|-|-|-|-|-|\n| Qwen2.5-Math-Instruct-7B  | 95.1  | 80.5  | 41.6   | 74.0 | 20.0 |\n| **Our SFT Model** | 95.6  | 87.9  | 46.4   | 80.0  | 30.0  |\n\n**Our SFT model shows significant improvements across both simpler and more complex OOD datasets**, supporting our hypothesis that learning from counterexamples enhances performance over a range of difficulty levels.\n\n> Lack of training details.\n\nWe apologize for the oversight. our training used Qwen2.5-Math-Instruct on two L20 48G GPUs with LoRA (rank 16). The learning rate was 1.0e-4 for one epoch. Full details will be clarified in the final version.\n\n> Suggestion to add performance of o1 model on simpler benchmarks.\n\nBelow is a comparison using official o1-preview results (due to resource constraints):\n\n| Model | Countermath | AIME 2024 | MMLU | MATH-500 |\n|-|-|-|-|-|\n| F-1 Example | 60.1 | 39.8| 74.4 | 90.8 |\n| Strict  | 55.8 | 39.8| 74.4 | 85.5|\n\nThe gap between COUNTERMATH and simpler benchmarks underscores **the challenge of counterexample-based reasoning.**\n\n> Hint prompt evaluations limited to Qwen2.5-Math-7B-Instruct.\n\nWe have also tested the Hint prompt with the Deepseek-Math-7B-rl model. Although space constraints prevented us from including these results in the paper, the summary is as follows:\n\n| Model                         | Overall F-1 | Examples (%) | Strict (%) | Loose (%) |\n|-------------------------------|-------------|--------------|------------|-----------|\n| Deepseek-Math-7B-rl           | 79.1        | 65.9         | 18.9       | 20.6      |\n| Deepseek-Math-7B-rl + Hint    | 78.4        | 83.5         | 25.6       | 27.7      |\n\n**The Hint prompt notably increases the \"Examples (%)\" metric, demonstrating its positive impact on example generation.** Due to consistent trends observed between Qwen2.5-Math-7B-Instruct and Deepseek-Math-7B-rl, we limited further evaluations to these two models.\n\n> Evaluation \"consistency\" of examples when facing multiple examples.\n\n**A generated example is considered \"consistent\" if it effectively serves as a valid proof for the statement**—matching the intended meaning and function of at least one reference example. So even if the example is different from the ones we provided, it’s still considered consistent as long as it has the same meaning.\n\n> Clarification on Table 3 metrics.\n\nWe apologize for the confusion. The values represent accuracy, not F1 scores. This will be clarified in the final version.", "claims_and_evidence": "While the paper claims that counterexample-driven learning improved performance of the fine-tuned model on COUNTERMATH, my concern is that the observed gains might not be attributable to the use of counterexample-based datasets. The fine-tuning process involves exposing the model to 1025 high-quality proofs using counterexamples. Without experiments using a similar number of math proofs with similar difficulties that are not structured around counterexamples, it is difficult to determine whether the performance boost arises specifically from counterexample-driven learning or from increased exposure to more material.", "methods_and_evaluation_criteria": "Overall the proposed benchmark makes sense. However, in the metrics for example alignment of COUNTERMATH, if a model generates valid counterexamples that differ from the provided reference, the evaluation might mark them as inconsistent based on the current evaluation prompt. This could penalize correct reasoning simply because the output diverges from the reference form, even though the counterexamples are mathematically sound. Moreover, the reliability of this automated evaluation of example alignment remains questionable as the reported accuracy of 93.5% for using GPT-4o at evaluating alignment compared to human judgments.", "theoretical_claims": "The paper is primarily empirical and does not present formal proofs.", "experimental_designs_or_analyses": "The experiments are thoughtfully constructed. Various open-weights LLMs and proprietary models are evaluated on COUNTERMATH with CoT prompts. Qwen-2.5-Math-7B-Instruct is fine-tuned on the counterexample-based math proofs, and is evaluated on both COUNTERMATH dataset and out-of-distribution datasets like MATH from (Hendrycks et al., 2021) and GSM8K from (Cobbe et al., 2021). Both the open-weights models and proprietary models achieve low F1, showing the difficulty of this dataset. Although the fine-tuned model achieves higher F1 on COUNTERMATH, the original base model outperforms the fine-tuned model on Strict/Loose Align metrics. This might suggest that by fine-tuning on the new counterexample-based dataset, counterexamples generated don’t align well with the counterexamples in the reference solution. The authors explain this drop in performance with the discrepancies of the dataset used for fine-tuning and COUNTERMATH. More detailed analysis should be provided to determine if the drop in alignment with the reference counterexamples causes an issue in the reasoning process or not.", "relation_to_broader_scientific_literature": "This paper focuses on counterexample based benchmark, which is claimed to better reflect understanding of math concepts than previous benchmarks that focus on solving simple math problems. By fine-tuning on counterexample-based math proofs, the model shows better performance on the proposed benchmark and out-of-distribution tasks compared to the base model. The results also show that the current open-weights LLMs struggle with utilizing counterexamples to reason, and perform badly in topics like topology and real analysis.", "essential_references_not_discussed": "The paper offers insights into counterexample based reasoning through natural language proofs, but it overlooks formal theorem provers like LeanDojo from (Yang et al., 2023) that employs programming languages like Lean4 (Moura & Ullrich, 2021). Given that formal proofs benefit from more objective evaluation, a discussion on the choice to focus on natural language proofs, and the challenges associated with their evaluation would help contextualize the paper’s contributions.", "other_strengths_and_weaknesses": "The counterexample based reasoning benchmark is a resource for evaluating the ability of LLMs to understand complex math concepts. The paper shows improvements on F1 score for proving statements in COUNTERMATH and improvements on the ability to use counterexamples in proofs by fine-tuning on counterexample-based datasets. \n\nSome weaknesses:\n1. No comparison with model fine-tuned on dataset of non-counterexample-based math proofs makes it unclear whether improvements are due to counterexample-based fine-tuning or simply increased exposure to advanced material.\n2. The current evaluation prompt may overly penalize valid and mathematically sound counterexamples that differ from the reference, and mark them as inconsistent.\n3. Using GPT-4o for evaluating alignment with a reported 93.5% accuracy relative to human judgment raises concerns about potential misclassifications.\n4. Although the paper shows there is a gain in performance on the out-of-distribution datasets by fine-tuning on counterexample proofs, the OOD datasets used are described as high school competition level and grade school level. I am concerned whether this improvement generalizes to more challenging OOD datasets.\n5. Training settings are not described(even in the Appendix).", "other_comments_or_suggestions": "Some additional comments:\n1. The paper claims that although o1 model achieves the best performance in COUNTERMATH F1 score, the performance is still significantly lower than what it achieved on elementary and high school mathematics benchmarks. It is helpful to add the performance of o1 model on simpler benchmarks for readers to see the difference. \n\n2. Hint prompt is only used with Qwen2.5-Math-7B-Instruct but not any other models. It is helpful to see how hint prompt can influence the performance of different models on COUNTERMATH dataset and metrics.", "questions_for_authors": "1. When evaluating consistency of generated examples by models and reference examples, will a generated example be marked as consistent if it conveys the same meaning with at least one or all of the reference examples if there are multiple reference examples? \n2. In table 3, are the numbers showing the F1 scores?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k7ee7y820t", "reviewer": "Reviewer_bVLs", "summary": "The paper presents new mathematical benchmark which tests the counterexample-based proof generation ability of LLMs across several sub-areas of math. The overall experimental results show that today's LLMs generally have low scores when trying to solve these problems with counter-example based reasoning. Further, the authors experiment with fine tuning a model to improve the scores, however, the results are borderline.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer bVLs\n\nWe sincerely appreciate your efforts in the review and hope our response will address your concerns.\n\n> Did the authors look at EN-based textbooks for such proofs?\n\nYes. While our annotators are native Chinese speakers and thus did not annotate directly from English textbooks, **our core authors aligned the English version of COUNTERMATH with standard EN-based textbooks during quality control**. Additionally, **many counterexamples in our Chinese sources originate from English works** like *Counter-Examples in Topology* (Steen & Seebach) and *Counterexamples in Topological Vector Spaces* (Khaleelulla). We are confident that our dataset maintains cross-language representativeness.\n\n> Did the authors consider generating a counterexample-based dataset using formal tools?\n\n**We chose not to use formal tools because the resulting data would conflict with the natural language rationale annotations in COUNTERMATH.** Using formal tools risks misalignment between generated counterexamples and the human-provided explanations, and **their quality depends heavily on the autoformalization model—which, given current performance on datasets like GSM8K and MATH, remains inconsistent.** However, we agree that formal generation of counterexamples is an intriguing direction for future work.\n\n> It was not clear if the authors considered simpler problems that can be solved with examples, which would be more amenable to formal provers (e.g., SAT/SMT encodable instances).\n\n**We have indeed considered simpler math datasets such as GSM8K and MATH.** Our experimental results (Table 3) show that counterexample-driven reasoning yields significant improvements on these elementary math scenarios—**demonstrating that even 7B-scale models can outperform 72B-scale models when leveraging examples effectively**.\n\n> Do you think that using RL instead of standard SFT training would yield better results?\n\nWe share your view on the potential of RL. Unfortunately, **our current GPU resources (only 2 L20 48G GPUs) limited our ability to conduct effective RL experiments for LLMs.** Moreover, our primary goal was to emphasize counterexample-driven reasoning and introduce COUNTERMATH. We believe that further exploration—including RL-based training—remains a promising direction for future research.\n\n> It would have been nice to see a bit more of an evaluation of where the models fail, e.g., due to reasoning errors, miscalculations, or deeper issues?\n\nThis is a very interesting and important research direction. We did attempt to evaluate the reasoning process of the model in order to identify the precise step where errors begin. **However, the complexity of the prompts led to instability in the output of the judge model.** For example:\n\n**Statement**: There exists a divergent series, which can be rearranged to arbitrarily slow down its divergence.\n**Model Output**: Correct understanding of divergent series, but mistakenly applies properties of conditionally convergent series.  *\"First, let's recall that a divergent series is a series whose sequence of partial sums does not converge to a finite limit...it is possible to construct a divergent series whose divergence can be arbitrarily slowed down by rearranging its terms... Therefore, it is indeed possible to rearrange a divergent series...\"*\n\n**Human Judgment**: False  \n**LLM Judgment**: True  \n\nGiven these inconsistencies, we used the Example evaluation approach. **Future research will explore stepwise error analysis**, like using reward models to more systematically analyze the causes of the model's failures.\n\n> Provide an end-to-end example to clarify benchmark components and model performance.\n\nWe appreciate the reviewer’s suggestion and have refined the examples to better illustrate our benchmark’s utility and model performance, highlighting model successes and failures:\n\n**Example 1**\n- **Statement:** \"The smallest number ring containing \\( \\sqrt{3} \\) is the real number ring.\"\n- **Answer:** False\n- **Model Prediction:** False (Correct)\n- **Model Output:** *\"To determine whether \\( L^{p}[0,1] \\) and \\( l^{p} \\) are linearly isomorphic for \\( p \\neq 2 \\)...\"*\n- **Correct Rationale:** *\"The minimal ring containing \\( \\sqrt{3} \\) is \\( \\mathbb{Z}[\\sqrt{3}] \\).\"*\n- **Model Issue:** The model correctly answered \"False\" but **hallucinated irrelevant concepts** (e.g., Banach spaces) unrelated to ring theory.\n\n**Example 2**\n- **Statement:** \"There exist rings without identity elements, and there are cases with left or right identity but not both.\"\n- **Answer:** True\n- **Model Prediction:** False (Incorrect)\n- **Model Output:** *\"We aim to determine whether there exists a bilinear functional \\( \\varphi(x, y) \\)...\"*\n- **Correct Rationale:** *\"A finite 4-element ring can have left identities without right identities.\"*\n- **Model Issue:** The model entirely **misinterpreted the problem type**, discussing bilinear functionals instead of ring-theoretic identities.", "claims_and_evidence": "The claims are well supported by the paper.", "methods_and_evaluation_criteria": "Yes, the proposed benchmark makes sense. However, from the paper, it would have been good if the authors had shown few examples of exactly how the data points in the benchmark look like. I looked at the supplementary material, and that was lacking there as well.", "theoretical_claims": "This is an experimental paper, no theoretical contribution.", "experimental_designs_or_analyses": "Yes, the experiments overall make sense.", "relation_to_broader_scientific_literature": "Measuring the performance of language models on mathematical tasks is a very active area of research. I have not yet seen a comprehensive benchmark which tests the LLMs ability on example-based proofs, so this is a useful contribution.", "essential_references_not_discussed": "Would have been good if the authors had discussed a bit more on contamination. The problems they select and the reference solutions, perhaps it would be good, if a contamination test on the base models, was done?", "other_strengths_and_weaknesses": "I believe this is a solid contribution, although the technical depth is not high. Couple of questions: \n\n- did the authors look at EN-based textbooks for such proofs?\n- did the authors consider generating counter-example based dataset based on formal tools? These tools, are generally good for producing counter-examples to conjectures when the problem is encoded? \n- it was not entirely clear, especially given the low scores, whether the authors considered simpler competitions / problems that can be solved with examples? They would also be more amenable to generating data with formal provers (simpler instance would be anything SAT/SMT encodable).\n- The results with the fine-tuning are somewhat mixed. Do you think that doing RL instead of standard SFT training would improve the results better? It would have been good to see such an experiment.\n- It would have been nice to see a bit more of an evaluation of where the models fail, e.g. it it due to wrong reasoning steps, or miscalculation errors, or deeper issues, when deriving the example?", "other_comments_or_suggestions": "I think it would be very useful if the authors, early on, provide an end to end example (simplified) for a data point in the benchmark dataset and an illustration of the model succeeding and failing, i.e., a reason.  Even when this is explained (rationale, judgement, etc), it needs more elaboration what each item is.", "questions_for_authors": "I already mentioned few questions earlier.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1kyr0BSEC2", "reviewer": "Reviewer_vkWZ", "summary": "The paper targets solving mathematical problems with LLMs, and focuses specifically on proofs by counterexample. The authors introduce a novel benchmark: CounterMATH. It contains 1216 examples of mathematical statements that are (dis)proved via showing a counterexample. The pipeline of creating the dataset is demonstrated. Multiple LLMs are evaluated on the benchmark, showing that it is challenging. Additionally, the authors extract counterexample-based proofs from existing benchmarks to finetune Qwen2.5-Math-7B-Instruct and show improved performance of the finetuned model on MATH and GSM8K datasets.\n\n**Update after rebuttal**\n\nI decided to keep my score unchanged. I think the counterexample-style reasoning is definitely an interesting topic, but I also think that the paper in its current shape should not be accepted. The writing is imprecise and has too much of a sales-pitch aesthetic (for instance: the false dichotomy between the \"drill-based reasoning\" aka direct proofs and supposedly superior counterexample-based reasoning). There are also other problems, like the validity of the \"hint prompting\". I suggest the authors to make the paper more solid and resubmit.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer vkWZ\n\n> **Claim (2) has weak experimental evidence: only one model fine-tuned, evaluated on two math benchmarks.**\n\nOur primary goal is to introduce a benchmark and **explore LLMs’ math capability in providing counterexamples.** We observed that even state-of-the-art LLMs struggle with this task. Therefore, we fine-tune the typical model, Qwen2.5-Math-Instruct, to enhance its counterexample reasoning abilities. To strengthen our claim, we expanded our evaluation to more challenging datasets, including **Competition-level** (OlympiadBench-Math, AIME 2024) and **University-level** (MMLU-College-Math). \n\n| Model | GSM8K | MATH | OlympiadBench-Math | MMLU-College-Math | AIME 2024 |\n|-|-|-|-|-|-|\n| Qwen2.5-Math-Instruct-7B | 95.1 | 80.5 | 41.6 | 74.0 | 20.0 |\n| Our SFT Model | **95.6** | **87.9** | **46.4** | **80.0** | **30.0** |\n\n**These results demonstrate significant improvements on both simpler and complex datasets**, supporting our hypothesis that learning from counterexamples enhances performance.\n\n> **Concern about \"Hint prompt\".**\n\nWe observe that directly prompting LLMs for true/false judgments led to poor performance and random guessing. Therefore, we hope to use the Hint prompt to inspire LLMs to \"give examples\", instead of assuming the truth. **It guides LLM to prove in a more reasonable way of thinking, which aims to let the model reason by giving examples**.\n\n> **Code documentation and dataset quality.**\n\nThank you for your meticulous review. Regarding the second statement you mentioned, we classify True/False based on the existence of examples: False if no such examples exist, and True if at least one example exists, and framing these with binary labels enables more efficient evaluation. As for dataset quality, COUNTERMATH was initially from textbooks and verified by experts to ensure all statement-rationale pairs are unambiguous and reasonably categorizable, as detailed in Sections 3.1, 3.2, and the Appendix.\n\n> **Lack of benchmark disclosure for quality assessment.**\n\nSorry for only providing the sampled dataset during the anonymous period. **The main reason is that we plan to set up a public Leaderboard, so it is not appropriate to disclose all data.** In addition, providing sampled datasets during the anonymous review stage does not violate any rules (in fact, this is a common practice, which can protect from being leaked prematurely and facilitate reviewers to review samples).\n\n> **Writing clarity issues.**\n\nWe apologize for any confusion, and we will ensure clearer phrasing in the final version. Key clarifications:\n- **\"Conceptual Finetuning\"** refers to fine-tuning on annotated, complex reasoning data.\n- **\"Statement is True or False by its rationale\"** means determining validity based on supporting reasoning.\n\n> **Claim that counterexample-based reasoning is more important is unjustified.**\n\nSorry for the confusion caused here. We aim to emphasize that **drill-based learning alone is insufficient for developing deep mathematical understanding, and fostering deeper mathematical understanding.** While drills transfer knowledge, counterexample-based reasoning enhances problem-solving and concept validation. We appreciate your observation and will revise Line 51 to avoid overclaims.\n\n> higher F1 but lower Examples (%)\n\n1. Sometimes the judgment is correct, but the reasoning process is not rigorous.\n\n**Example 1:**\n- *Statement:* The smallest number ring containing \\(\\sqrt{3}\\) is the real number ring.\n- *Answer:* False\n- *Model Prediction:* False (Correct)\n- *Model Output:* Incorrectly discusses Banach spaces and the RNP instead of relevant number theory concepts.\n- *Explanation:* The model guessed the correct answer but provided an irrelevant rationale, resulting in a correct judgment but a poor Examples score.\n\n2. Constructing effective counterexamples requires a comprehensive master of mathematical knowledge.\n\n**Example 2:**\n- *Statement:* All countably compact spaces are compact.\n- *Model Output:* Proposes an invalid counterexample based on Hilbert cube subsets.\n- *Explanation*: The Hilbert cube itself is compact, and its closed subsets inherit compactness. Thus, this counterexample is self-defeating.\n\n> **Inspection of model-generated counterexamples.**\n\nHere is an LLM-generated counterexample:\n\n- *Statement:* For a fixed element \\(a\\) of the ring \\(R\\), \\(N = \\{ra \\mid r \\in R\\}\\) forms an ideal of \\(R\\) but does not necessarily contain \\(a\\).\n- *Reference Example:*  \\(R\\) as the ring of even numbers; \\(N = \\{r \\cdot 4 \\mid r \\in R\\}\\) is an ideal, but \\(4 \\notin N\\).\n- *LLM Output:* The model analyzed a trivial ring \\(R = \\{0\\}\\) where the counterexample was consistent with evaluation criteria.\n\nThe LLM’s counterexample differs from the reference but is marked as \"consistent\" because it satisfies the evaluation criteria. Our judge model successfully extracts and assesses these counterexamples, determining their consistency with the intended logic.", "claims_and_evidence": "I find two main claims in the paper: (1) proving with counterexamples is difficult for LLMs, (2) LLMs fine-tuned on counterexample-based proofs is helpful for improving \"reasoning skills\" of LLMs in general. \n\nClaim (1) is supported by strong evidence with experiments with multiple LLMs evaluated on CounterMATH.\n\nClaim (2) has rather weak experimental evidence: the authors fine-tune only one model and evaluate it on just two math benchmarks. Additionally, on one of them, GSM8K, the improvement is small (95.1% vs 95.6%).", "methods_and_evaluation_criteria": "In general, the methodology seems to be sound. However, I'm not sure about this aspect of the pipeline:\n\nWhen you evaluate LLMs on CounterMATH, you say that \"we adopt a Hint prompt, designed to encourage LLMs to provide examples when attempting to solve problems\". But when you ask about whether a math statement is true or false and at the same time say that a proof by counterexample is expected, this means that in many (all?) cases one can infer the truth value without any reasoning.\n\nIf we have a math statement X, typically only one of the two alternatives, X, neg(X), can be proved by showing a counterexample. \n\nFor instance, if one asks me whether this in true\n```\n\"Every subgroup $H$ of an abelian group $G$ is a normal subgroup.\"\n```\nand I know that I should use a proof by counterexample, I know that the statement must be false. \n\nCould you comment on that?", "theoretical_claims": "There are no theoretical claims in the paper.", "experimental_designs_or_analyses": "See above (the issue with the Hint prompt).", "relation_to_broader_scientific_literature": "The paper proposes a new mathematical benchmark / dataset and therefore is related to multiple other recent paper proposing similar math benchmarks. The benchmark proposed by the authors has a clear distinguishing feature, namely it focuses on proofs by counterexample, and I'm not aware of another benchmark focusing on this aspect.", "essential_references_not_discussed": "The authors in general discuss relevant related literature.", "other_strengths_and_weaknesses": "### Strengths\n\n1. The authors target an interesting and challenging problem: counterexample-based proving. \n2. The authors evaluate many (25) different LLMs on their benchmark.\n3. The authors provide some interesting analyses beyond assessing accuracy, like number of tokens produced. \n\n### Weaknesses\n\n1. The authors did not disclose the CounterMATH benchmark which would allow to asses its quality.\n2. The writing is quite poor or imprecise in some places. For example: \"Conceptual Finetuning\" -- why such a phrase for finetuning on counterexample-based proofs? Or \"statement is True or False by its rationale\" -- what does it mean?\n3. The authors present a narrative that the counterexample-based reasoning constitutes more important, deeper form of reasoning; they write that learning from non-counterexample proofs is \"drill-based\" learning, and that \"example-based learning is a more important strategy than drill-based learning.\" Although I agree that counterexample-based proving is important and challenging, such claims are completely unjustified and false.", "other_comments_or_suggestions": "Figure 5 is difficult to read -- a simple table or a bar plot would be better in my opinion.", "questions_for_authors": "1. In Table 1, some models have higher F1 but lower Examples (%) -- for instance, Qwen2.5-Math-7B-Instruct vs OpenAI o1-preview. As all the examples in the benchmark require a proof by counterexample, I find it strange. Could you explain it? Could you show some concrete examples where the model provided good answer but didn't use counterexamples?\n2. Did you perform inspection of the quality of the counterexamples produced by some LLMs? Could you show some LLM-produced counterexamples which do not appear in CounterMATH? Or counterexamples which are incorrect?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x0C1njuKBW", "reviewer": "Reviewer_tdTp", "summary": "The paper presents CounterMath, a novel benchmark to assess and enhance LLM's ability to reason through counterexamples in mathematical proofs. Inspired by the pedagogical method of learning by counterexamples, the work introduces a dataset of university-level mathematical statements requiring counterexample-driven disproving. The authors also propose a data engineering pipeline to generate counterexample-based training data and demonstrate through empirical evaluation that LLMs, including OpenAI o1, struggle with counterexample-driven conceptual reasoning. Overall, this paper highlights the importance of counterexample reasoning in improving LLMs' overall mathematical capabilities and provides insights into potential future research directions in modern/research-level reasoning.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer tdTp\n\nThank you for your efforts in the review, and hope our response will address your concerns!\n\n> Supported by out-of-distribution (OOD) evaluations, but additional benchmarks from formal theorem proving could further validate this claim.\n\n**To further validate our approach, we expanded our evaluation to include higher-difficulty datasets** at the competition level (OlympiadBench-Math, AIME 2024) and university level (MMLU-College-Math). The accuracy results are summarized below:\n\n| Model                         | GSM8K | MATH | OlympiadBench-Math | MMLU-College-Math | AIME 2024 |\n|-------------------------------|-------|------|--------------------|-------------------|-----------|\n| Qwen2.5-Math-Instruct-7B      | 95.1  | 80.5 | 41.6               | 74.0              | 20.0      |\n| **Our SFT Model**             | 95.6  | 87.9 | 46.4               | 80.0              | 30.0      |\n\nThese results show significant improvements on both simpler and more complex OOD datasets, supporting our hypothesis that current LLMs lack the ability to effectively leverage counterexamples for mathematical reasoning.\n\n> While the results indicate that counterexample-based reasoning is crucial, the experiments lack an ablation study comparing different types of proof strategies (e.g., direct proof, proof by contradiction, proof by construction). Including such an analysis could strengthen the claim that counterexample-driven reasoning is uniquely impactful.\n\nWe appreciate this insightful suggestion. In our work, **we conducted a preliminary analysis using different prompting strategies (CoT, ICL, and Hint) to assess their impact on example generation**. Due to space limitations, we provided only a brief overview. We plan to conduct a more detailed ablation study in future work and hope our current analysis will encourage further research in this area.\n\n> Have you considered applying COUNTERMATH to a Lean-based proof assistant (or Isabelle, Coq, etc.) to test whether counterexample-driven learning aids in formal symbolic reasoning?\n\n**Our decision not to use formal tools stems from a mismatch with COUNTERMATH’s natural language rationale annotations.** Formal data generated by automated tools rely heavily on autoformalization models—which currently show inconsistent performance on datasets like GSM8K and MATH. However, we agree that exploring formal generation of counterexamples is a promising direction for future research.\n\n> How does the model perform on longer proofs (like theorems in research-level math papers) that require lots of counterexamples? Is reasoning depth a bottleneck?\n\nOur benchmark is derived from university-level mathematics textbooks, covering key definitions, theorems, properties, and axioms. In many cases, the model incrementally builds its reasoning by adding constraints and refining assumptions. While this demonstrates some reasoning depth, **research-level theorems—requiring extensive counterexamples and multi-step proofs—pose a significant challenge.** We hypothesize that tackling such complex proofs requires the LLM to possess multiple capabilities, and reasoning depth does emerge as a bottleneck for LLMs in these scenarios.\n\n**Moreover, as seen in models like o1 and Deepseek-R1, an excessively prolonged reasoning process can lead to a phenomenon where the model \"overthinks\" and becomes trapped in a loop of iterative self-correction.** This results in reasoning fallacies, where the model repeatedly attempts to refine its initial claim but ultimately diverges further from a correct solution. We encountered this specific issue during our evaluation of Deepseek-R1 on the COUNTERMATH benchmark, resulting in more than 500s thinking time per question.\n\n**Addressing these research-level reasoning challenges requires a multi-faceted approach.** Further model optimization like SFT and Reinforcement Learning can improve the model’s understanding of mathematical concepts, its proficiency in using formal languages for structured reasoning, and its ability to generate logically consistent counterexamples. We believe that continued advancements in these areas will significantly enhance LLMs' performance on complex mathematical proofs.\n\nThank you once again for your valuable question, which really encourage us to further improve our research.", "claims_and_evidence": "This paper claims that LLMs rely heavily on exposure to existing proof patterns, which limits their generalization to novel mathematical statements. \nThis is supported by the evaluation of LLMs on the CounterMath dataset which emphasizes counterexample-driven proofs. Even the SOTA models struggle with the benchmark and validating their hypothesis. Additionally, they provide fine-grained analysis across different mathematical fields, demonstrating that real analysis and topology pose particular challenges. The evidence is well-supported through rigorous experimentation, statistical evaluation, and comparisons with related mathematical benchmarks (e.g., PutnamBench, MATH, GSM8K).", "methods_and_evaluation_criteria": "Overall, the methodology for dataset curation is well-structured and clearly articulated. The problems are selected from university-level textbooks and validated through expert review, ensuring high-quality data. The evaluation criteria include:\n1. F1 lexical matching score, which measures correctness in determining the truth value of statements.\n2. Frequency that models generate counterexamples.\n3. Strict and Loose Alignment, assessing whether generated examples align with reference solutions.\n4. Testing performance on MATH and GSM8K to assess OOD generalization.", "theoretical_claims": "This paper does not make any theoretical claim.", "experimental_designs_or_analyses": "The experiments include baseline comparisons across various mathematical LLMs, including proprietary (gpt4o) and open sourced models (e.g. Deepseek-Math-7B-RL). The performance is breakdowned by mathematical fields (e.g. Algebra, Functional Analysis, Real Analysis, Topology). Furthermore, this paper conducts fine-tuning experiments on Qwen2.5-Math-7B-Instruct to demonstrate the effectiveness of counterexample training.\n\nOn the other hand, while the results indicate that counterexample-based reasoning is crucial, the experiments lack an ablation study comparing different types of proof strategies. Including such an analysis could strengthen the claim that counterexample-driven reasoning is uniquely impactful.\nThe claim that improving counterexample reasoning enhances general mathematical reasoning is supported by out-of-distribution (OOD) evaluations, but additional benchmarks from formal theorem proving could further validate this claim.", "relation_to_broader_scientific_literature": "This paper is closely related to formal theorem proving and provides a promising approach towards automating modern mathematical solving", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strength:\n1. This paper introduces a novel benchmark that addresses a crucial gap in counter-examples in mathematical reasoning. The dataset is rigorously validated with valuable insights into mathematical LLMs' limitations.\n2. The training methodology shows that counterexample-driven learning is effective in improving reasoning.\n\nWeakness: \nWhile the results indicate that counterexample-based reasoning is crucial, the experiments lack an ablation study comparing different types of proof strategies (e.g. direct proof, proof by contradiction, proof by construction). Including such an analysis could strengthen the claim that counterexample-driven reasoning is uniquely impactful.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Have you considered applying COUNTERMATH to a Lean-based proof assistant (or Isabelle, Coq, etc) to test whether counterexample-driven learning aids in formal symbolic reasoning?\n2. How does the model perform on longer proofs (like theorems in research-level math papers) that require lots of counterexamples? Is reasoning depth a bottleneck?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper was reviewed by five reviewers and received mixed scores. The reviewers generally agree that the paper tackles an interesting and under-explored problem (using LLMs to find counterexamples) and propose a potentially useful benchmark. However, opinions are mixed due to concerns about the evaluation and ablation. The authors addressed many of these issues in the rebuttal with new experiments and clarifications, though some concerns remain. The AC recommends a weak accept.
Decision: Accept (poster)