Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate
Authors: Yubo Wang
Xiang Yue
Wenhu Chen
Venue: COLM 2025 (2025)
Keywords: Reasoning
Large Language Model
Fine-Tuning
TL;DR: We introduce Critique Fine-Tuning, a training method that teaches LM to critique responses, achieving better performance than SFT with fewer training samples and comparable results to RL methods.

Abstract: Supervised Fine-Tuning (SFT) is commonly used to train language models to imitate annotated responses for given instructions. In this paper, we propose Critique Fine-Tuning (CFT), a method more effective than SFT for reasoning tasks. Instead of simply imitating correct responses, CFT trains models to critique noisy responses, inspired by human learning processes that emphasize critical thinking, deeper analysis, and nuanced understanding--traits often overlooked by standard SFT. To validate the effectiveness of CFT, we construct multiple critique datasets (e.g., WebInstruct, MetaMath, NuminaMath), where GPT-4o serves as the teacher to generate critiques in the form of ([query; noisy response], critique). Experiments on these datasets demonstrate that CFT consistently outperforms SFT by 4--10% across six mathematical reasoning benchmarks, and is effective across different base models including Qwen2.5, Qwen2.5-Math, and DeepSeek-Math. Notably, our model Qwen2.5-Math-CFT only requires 1 hour of training on 8xH100 over the 50K examples, yet matches or outperforms strong competitors like Qwen2.5-Math-Instruct on most benchmarks, which use over 2M samples. Moreover, it matches the performance of SimpleRL, which is a DeepSeek-r1 replication trained with 140x more compute. Experiments on IF_Eval and MT-Bench further demonstrate that CFT can significantly enhance the model's general generation and instruction-following capabilities, outperforming the Qwen2.5-Math-Instruct by a large margin. Ablation studies show that CFT is robust to noisy response sources and teacher critique models. These findings highlight that CFT offers a more effective alternative to advance the reasoning of language models.

Published as a conference paper at COLM 2025
Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate
Yubo Wang!,, Xiang Yue², Wenhu Chen
!Department of Computer Science, University of Waterloo
CCrnneiie Mellon University, Pittsburgh Veccor Institute, Toronto
https://tiger-aat gi thub. wtt.:iiitiiueeeeentnniiii
Supervised Fine-Tuning (SFT)
Critique Fine-Tuning (CFT)
Your airline is considering replacing its McDonnell Douglas MD-80s with either the Airbus 320ceo or the Airbus 320neo. Both models.. How would you conduct a financial analysis to determine which model is the
Q: Your airline considering
The financial
A: To determine the better financial between Airbus 320ceo and Airbus 320neo suggest that the initial expected lives.
replacing its McDonneil Douglas MD-80s with either the Airbus 320ceo or the Airbus 320neo. Both
analysis you propose is thorough and well- structured.. reliable basis decision- making...
determine better financial choice between
effective choice in the long run?
Aitbus 320ceo and Airbus 320...
Conclusion: right
MATH
OlympiadBench
100
Base
SFT-GPT4o SFT-verified
CFT
Base
SFT-GPT4o SFT-verified
CFT
41.6
80
71.1
73.2
80.2
30.1
35.7
37.6
61.5
62
30
28.3
60
42,2
49.8
19.7
16.1
35.8
12.4
31.7
9.3
8.9
4.5
20
0
DeepSeek
Qwen2.5
Qwen2.5-Math
DeepSeek
Qwen2.5
Qwen2.5-Man
Figure 1: Comparison between CFT and SFT on 50K samples from WebInstruct (Yue et al., 2024b), SFT-verified means SFT training on the responses validated by GPTT-4o, SFT-GPT40 means SFT training on the responses from GIT--o.. CFT is our approach, which trains on the critique provided by GPT-4o.
Abstract
Supervised Fine- Tuning (SFT) is commonly used to train language models imitate annotated responses for given instructions. In this paper, we propose Critique Fine-Tuning (CFT), method more effective than SFT for reasoning tasks. Instead of simply imitating correct responses, CFT trains models to critique noisy responses, inspired by human learning processes that emphasize critical thinking, deeper analysis, and nuanced understanding traits often overlooked by standard SFT. To validate the effectiveness of CFT, we construct multiple critique datasets (e.g., Webln- struct, MetaMath, NuminaMath), where GPT-4o serves the teacher to generate critiques in the form of (lquery; noisy response), critique). Experi- ments these datasets demonstrate that CFT consistently outperforms SFT by 4-10% across six mathematical reasoning benchmarks, and is effec- tive across different base models including Qwen2.5, Owen2.5-Math, and DeepSeek-Mate Notably, our model (wwn2.5-Math-CFTT only requires 1 hour of training on 8xH100 over the 50K examples, yet matches or outper- forms strong competitors like -wwen..--Math-Instree on most benchmarks, which use over 2M samples. Moreover, matches the performance of SimpleRI. which is DeepSeek-rl replication trained with 140x more com- pute. Experiments on Eval and MT-Bench further demonstrate that CFT can significantly enhance the model's 's general generation and instruction- following capabilities, outperforming the @wwen2.--Math-Instruct by a large margin. Ablation studies show that CFT is robust to noisy response sources and teacher critique models. These findings highlight that CFT offers a more effective alternative to advance the reasoning of language models.

---

Published as a conference paper at COLM 2025
1 Introduction
Recently, large language models (LLMs) (Achiam et al., 2023; Team et al., 2023; Dubey et al., 2024) have shown unprecedented performance real-world problems. One core techniques is supervised fine-tuning (SFT), which trains LLMs to follow natural language instructions (Wei et al., 2022; Ouyang et al., 2022; Sanh et al., 2022). In the process of SFT, LLMs are forced to imitate the annotated responses. Numerous efforts have been made to build high-quality SFT datasets using approaches like Self-Instruct (Wang et al., 2023b) and Evol-Instruct (Xu et al., 2024) to enhance LLMs' general instruction-following capabilities. More recently, works such MA (Yue et al., 2024a;b), MetaMath (Yu et al., 2024), and WizardCoder (Luo et al., 2024) have employed SFT to improve the targeted capabilities of LLMs in areas like mathematical reasoning, coding, and more. While these approaches have shown significant gains on weaker base models such as Mistral (fiang et al., 2023) or LLaMA3 (Dubey et al., 2024), diminishing returns become evident as SFT dataset size and quality scale up. This limitation is particularyy pronounced for already www.sppaariiiiaaae..
base (Yang
models ned,,, such as Qwen2.5-base (Yang et al., 2024a), DWwn2.5-Math-bn. etal., or DeepSeek-Codees V2-base (Guo et al., 2024), which have undergone extensive domain-adaptive pretraining on reasoning-focuset corpora comprising hundreds of billions of tokens. Our experiments in section 3 reveal that applying SFT to these models can even degrade performance without stringent quality control.
In this paper, we propose new learning framework called Critique Fine-Tuning (CFT) where critical thinking and constructive feedback are vital for improvement-we shift the focus from imitation to critique-based learning. When humans learn, they do not merely replicate provided answers but analyze, critique, and refine them. Similarly, in CFT, the model learns to provide critiques for noisy responses, identify flaws, suggest improvements, and verify correctness. Formally, CFT involves training the model to critique a given query-response pair, maximizing the likelihood P(c| [[)], where c is the annotated critique for query- response pair y) A detailed visualization of CFT is presented in Figure 1.
as an alternative to traditional SFT. Inspired by human learning-where
To validate CFT's effectiveness, we designed series of experiments using multiple critique datasets, including WebInstruct (Yue et al., 2024b), MetaMathQA (Yu et al., 2024), and MatinaMath (Li et al., 2024b), with critiques synthesized by advanced models such as GPT-4o (Achiam et al., 2023). We applied CFT to strong 7B base language models (non- instruction-tunedd including DeepSeekMath-baat 2024a), and Qwen2.5-Math (Yang et al., 2024b). These models were compared against strong SFT-trained variants, such Weblnstruct-verrfiidd (SFT on -Pr-fo-vrrfieed responses) and WebInstruct-GITA. (SFT directly on GPT-to-generated responses). When evaluated on six math benchmarks, including MATH and AIME24, CFT-trained models consistently outperform the best SFT-trained models by an average of 4-10 absolute points.
(Shao et al., 2024), Qwen2.5 (Yang et al.,
We further expanded the evaluation to broader STEM benchmarks, including GPQA (Rein et al., 2023), TheoremQA (Chen et al., 2023), and MMLU-Pro (Wang et al., 2024b). Our results show that the best CFT-trained model, Qwen2.5-Math-CFT trained on 50K exam- ples, outperformed strong competitors like AceMath (Liu et al., 2024) and Qwen2.5-Math- Instruct (Yang et al., 2024b), which were trained on over 2M examples. We also compare Owen2.5-Math-CFF with SimpleRL (Zeng et al., 2025), open replication of DeepSeek- R1 (Guo et al., 2025) trained with 140x more compute (1152 vs 8 H100 hours). Results show that (0wen2.5-Mat-CCF) reaches the same average performance across 5 math benchmarks, highlighting the efficiency and effectiveness of CFT for reasoning-focused tasks.
To understand the models' general instruction following abilities, we also evaluate our Swen2..--Mah--CFF on MT-Bench (Zheng et al., 2023) and IF Eval (Zhou et al., 2023). We show that our method also enhances general Instruction-folooiin abilities. It outperforms our SFTed version and the official
by notable margin.
To better understand different factors of CFT, we conducted comprehensive ablation studies:
1. Robustness to dataset sources: Comparing WebInstruct (Yue et al., 2024b) against Meta- MathQA (Yu et al., 2024) and NuminaMath (Li et al., 2024b), we observed that WebInstruct
2

---

Published as a conference paper at COLM 2025
provided slight advantage (3%+) due to its diversity and broader topic coverage.
2. Robustness to noisy response sources: Experiments with original noisy responses and
Qwen2.5-base responses critiqued by GPT-4o showed negligible performance differences.
3. Flexibility to the teacher: Using a weaker critique dataset synthesized by GPT-4o-mini
still yielded notable improvements over SFT, despite 4% overall score drop.
4. Controlling for token length: Even controlling token length, shorter critique examples significantly outperformed standard SFT (55.2% 50.4%), confirming improvements stem from critique-based training rather than increased sequence length.
Our approach not only demonstrates strong improvement on cassninng-ocusee tasks, but also exhibits notable improvement over general-purpose instruction following tasks. These evidence has shown great potential of CFT to replace SFT in language model training.
2 Method & Dataset
To validate the effectiveness of CFT, construct several fine-tuning datasets. Most of our experiments are based on WebInstruct, an instruction dataset collected from online educational resources and quiz websites. The dataset undergoes synthetic processing in its pipeline using LLMs to improve solution quality and format consistency.
2.1 WebInstruct
WebInstruct spans wide range of topics, including Mathematics (65%), Physics (8%), Chemistry (4%), Business (10%), Humanities (4%), and more. Unlike other datasets, which are primarily derived from math contests and competitions, WebInstruct offers broader topic coverage. The responses in WebInstruct are extracted and refined by large language models such as Qwen-72B (Bai al., 2023) and Mixtral (Jiang et al., 2024), making them highly prone to noise due to the lack of verification or quality control.
We curate the following subsets from WebInstruct:
Weblnstruct-SFF:: A 50K subset directly sampled from the original WebInstruct dataset.
This subset has a very high error ratio (over 50%).
Weblistrruttverrffee:: We prompt GPT-4o-1120 to judge the correctness of original WebIn- struct answers. We retained the top 50K samples as "verified" SFT data.
WebInstruct- A 50K subset that reuses questions from WebInstruct-SFI but replaces the answers with those generated by GPT-4o-1120.
WebInstruct-CFT (Ours): A 50K subset derived from WebInstruct-SFT, where GPT-4o- 1120 provides detailed critiques of the original responses. Approximately 56% the responses in this subset are judged "correct" while the rest considered "wrong". Despite containing some critique errors introduced by GPT-4o, this dataset is comparable in quality to Weblnstrrut-GPT-4oo
We compare our CFT datasets with existing SFT datasets in Table 1. As shown, our datasets cover broader range of topics while being significantly smaller in size, highlighting their efficiency in boosting LLMs' reasoning abilities.
2.2 MetaMath & NuminaMath
In addition to WebInstruct, synthesized critiques for other datasets, including Meta- MathQA and NuminaMath. From each dataset, we randomly sampled 50K examples and used GPT-4o to critique the original responses. We then applied CFT to these datasets to demonstrate the generalizability of our approach across other datasets.
2.3 Training Objective
Our training objective is straightforward. We concatenate question and noisy response as input, then optimize the model parameters to generate critique c. Formally, the training loss is: argmaxg log [[c[[x:y]] where is the parameters of the language model.
3

---

Published as a conference paper at COLM 2025
Dataset
Size
Source Seed
Discipline
Supervised Fine-Tuning Data
WizardMath (Luo et al., 2023)
96K
CSM8K, MATH
Math
MathInstruct (Yue al., 2024a)
260K
GSM8K, MATH, etc
Math
MetaMathQA (Yu et al., 2024)
395K
GSM8K, MATH
Math
XwinMath (Li et al., 2024a)
1.4M
GSM8K, MATH
Math
OrcaMath (Mitra et al., 2024)
200K
GSM8K
Math
NuminalMath (Li al., 2024b)
860K
GSM8K, MATH, AIME
Math
AceMath (Liu et al., 2024)
1.6M
GSM8K, MATH, AIME
Math
OpenMath-2 (Toshniwal et al., 2024)
14M
GSM8K, MATH
Math
Critique Fine- Tane-Tnnigg Data (Ours)
CFT
50K
WebInstruct
STEM
Table The comparison of CFT and SFT datasets.
Experiments
In this section, we will detail our experiments.
Model
Method
MATH
Minerva-Math
GSMSK
OlympiadBench
AIME24
AMC23
AVG
Rase
33.8
9.2
64.3
4.5
0.0
10.0
20.3
Web
12.1
34.7
6.2
0.0
17.5
16.1
DeepSeck-Math--B
Weblnstruct verified SET
35.8
10.7
67.5
9.3
0.0
7.5
21.8
WeblnstructGGTTo-SFF
70.9
8.9
3.3
17.5
24.0
Weblnstruct-CFT
422
12.5
74.3
12.4
3.3
20.0
27.5
A CFT SPT
6.4
0.4
3.6
3.1
0.0
2.5
3.5
Base
15.1
85.4
26.3
37.5
37.4
WebInstruct SPT
6.6
59.5
5.8
3.3
15.0
20.2
Q-en2.5-78
weblnstruct-verffedd.ee
61.5
16.2
70.8
30.1
13.3
37.5
38.2
Weblnstrud-GPTHooSt
77.4
19.7
Weblnstruct- CFT
71.1
27.9
88.8
35.7
13.3
55.0
48.6
CFTTF-SB SPT
96
9.5
11.4
5.6
0.0
5.0
10.4
Base
WebInstruct-SFT
59.0
13.2
77.4
19.9
3.3
37.5
35.1
Qwen2.5-Mallin
veriifiect--eifieed
62.0
12.5
78.8
22.1
16.7
50.0
40.4
WobInstruct-GPPA SPF-O-S-F
90.0
WeblnstructCFFT
802
42.3
90.9
41.6
20.0
67.5
57.1
CFT- SFT
16.6
0.9
4.0
3.5
5.0
6.7
Table 2: Performance comparison of SFT and CFT on different base models. All the experiments are trained with WebInstruct subset. We select the checkpoint with highest validation score and report their results.
3.1 Experimental Setup
Evaluation Datasets We evaluate method wide range of mathematical reasoning benchmarks. For standard mathematical reasoning, we evaluate MATH (Hendrycks et al., 2021), Minerva-Math (Lewkowycz et al., 2022) and GSM8K (Cobbe et al., 2021). To assess performance on more challenging competition-level mathematics, we incorporate AIME 2024, AMC 2023, and OlympiadBench (He et al., 2024) containing various difficulty levels of Mathematical Olympiad problems. We further extend our evaluation to broader STEM reasoning capabilities through TheoremQA (Chen et al., 2023) for mathematical theorem understanding, MMLU-Pro (Wang et al., 2024b) covering physics, chemistry, mathematics, etc., and GPQA (Rein et al., 2023) for complex problems requiring scientific reasoning.
Training Details
We evaluate three different SFT training settings and one CFT training setting in our experiments. For the SFT training settings, we explore: (1) SFT: direct training on original dataset, (2) SFT-verified: training on responses validated by GPT-4o, and (3) SFT-GPT-40: training responses generated by GPT-4o. For CFT, we train the model using our curated CFT datasets as described in section 2. We use MATH-500 (Lightman et al., 2023b) as our validation set and select the best-performing checkpoint after training the

---

Published as a conference paper at COLM 2025
Model
#Data MATH GPQA TheoremQA MMLU-Pro OlympiadBench AIME24 AMC23 AVG
Frontier Models
GPT-4o (2024-08-06) GPT-ol-mini
81.1
51.6
43.3
47.5
51.7
90.0
60.0
57.2
80.3
65.3
56.7
95.0
72.1
Other Oppen-soureed Reasoning LLMs
Deepseek-Malhh77-nntttue
44.3
31.8
23.7
35.3
13.6
3.3
15.0
23.9
Mathstral-7B-vx..1
32.2
42.5
21.5
42.4
32.9
NuminaMath-78-Col
55.2
30.6
28.6
38.6
19.9
6.7
30.0
29.9
30.0
30.3
1amma-31-700-Instrure
65.7
42.2
51.3
62.8
14.4
16.7
30.0
40.4
NuminaMathh72B-CoT
68.0
35.3
24.9
55.0
35.0
3.3
52.5
39.1
Owen225-Math-72-nnmmmea
85.9
49.0
60.3
49.0
70.0
56.4
Initialized from Owenn.5-Math-7B--e
Qwen2.5-MMi
Diwen_.-MMat--Bae Math-Base
0
55.4
31.0
37.4
39.3
16.1
10.0
40.0
32.7
Eurus-2-SFT
230
62.4
32.1
44.2
29.8
30.1
34.3
IStarMMatbGGreedy
747K
78.4
47.1
26.7
47.5
AceMatt--Wenn255MM..
2.3M
83.1
26.1
24.6
48.1
42.2
16.7
60.0
43.0
Qwem255-Mahh-7--nsstra
2.5M
83.6
31.1
37.0
39.5
41.6
16.7
62.5
44.6
Qwen2.5-Math-TD-e.
50K
80.2
39.4
40.4
47.5
41.6
20.0
67.5
48.1
Table 3: Performance comparison of our models vs. other reasoning iassoing-speciaalzze models. #Data denotes total training set size. We select the checkpoint with highest validation score.
entire dataset for 1 epoch. We maintain consistent hyperparameters across all experiments with learning rate of 5e-6, cosine decay learning schedule with warm-up ratio of 0.1, and global batch size of 512.
3.2 Main Results (CFT vs. SFT)
To evaluate the effectiveness of CFT, we compare it with various SFT methods on three 7B-scale base models using mathematical reasoning benchmarks. Table 2 presents the results across different base models and methods. Our key findings are as follows:
Base Model Selection We experiment with three 7B-scale base models: DeepSeek-Mate serves as
7B, Qwen2.5-7B, and Qwen2.5-Math-7B Results show that stronger foundation, with its base version achieving 37.8% average accuracy across benchmarks. When enhanced with CFT, it achieves the best performance with 57.1% average accuracy.
Performance Gains CFT consistently outperforms all SFT baselines across different models. On M it achieves a 3.5% absolute improvement over the SFT-GPT4o. On Qwen2.5-7B, it demonstrates substantial 10.4% improvement over the SFT-verified. On Owen2.5-Math-7B, it surpasses the strong GPT-4o SFT baseline by 6.7% over SFT-GPT4o.
More Results (CFT Models vs. Existing Models)
In Table 3, we compare our best CFT-models with other competitive models with different scales. We expanded the evaluation benchmarks to cover broader STEM topics.
Our Swwen2.5-Math77B-FFF achieves the highest average performance (48.1%) among 7B-scale models while using significantly less training data (50K samples). Specifically:
It substantially outperforms specialized math models such as Deepseck-Mathh-T-nnsttue (23.9%), Mathstral-7B (32.9%), and NuminaMath-7B-CC (29.9%).
This strong performance achieved with remarkably less training data only 50K samples compared to AceWlath-Qwen2.55MM.. (2.5M samples), demonstrating the superior data efficiency of our approach. In addition, method outperforms Eurus-2-SF (230K) Cui et al. (2025) and (Star-MathGrreedy (747K) Guan et al. (2025) on most tasks.
(2.3M samples) and Sreen..5-Maat-7B-nnsttee
Despite being smaller in scale, our Owen2.5-Math-7B-CFT demonstrates strong performance compared to larger models. With only 7B parameters, it achieves better average performance (48.1%) than Llama-311 1amaa-..1-70BImm
perraggeerfommanee..... (40.4%) and NuminaMath-72B-Com (39.1%). When

---

Published as a conference paper at COLM 2025
compared to Qwwen2.5-Mathh-2---nst several benchmarks (e.g,, 67.5% vs 70.0% on AMC23) despite using only one-tenth of the parameters and less training data. While frontier closed models like GPT-4o still maintain performance lead, our results demonstrate that efficient training strategies can help smaller models achieve strong performance with fewer resources.
(56.4%), our model shows competitive results
3.4
Comparison with RL-based Method
Model
Data Size
GPU Hours
MATH-500
Minerva-Math
OlympiadBench
AIME24
AMC23
AVG
SimpleRL-Zero
8Kx12
1152
77.2
33.5
37.9
62.5
SimpleRL
8K48Kx12
1152
62.4
39.7
43.3
26.7
62.5
50.9
CFT
50K
79.6
42.3
41.6
20,0
67.5
50.2
Table 4: Performance comparison with RL-driven methods.
Recently, researchers have shown that reinforcement learning can significantly boost the reasoning capabilities of large language models. Here, we compare with SimpleRI. (Zeng et al., 2025), which is an open replication of DeepSeek-RI (Guo et al., 2025). We consider pure RL-based training (SimpleRL-Zenu competitors, both require 32xH100 to train for 1.5 days. In contrast, our method only requires 8xH100 to train for 1 hour. Furthermore, our approach does not requirea long decoding length leading to higher efficiency.
and Distill+RL-based training (SimpleRL) as our
As shown in Table 4, CFT can improve Qwen2.5-Matt-7T-bass to the same level as Sim- pleRL. On several benchmarks like AMC23 and Minverva-Math, CFT outperform both SimpleRL significanty The biggest difference is AIME24, which only contains a test set of 30 questions. The accuracy heavily impacted by the randomness.
3.5 Ablation Studies
To understand the impact of different factors in CFT, we conduct several ablation studies:
Task
MetaMathQA
NuminaMath
Weblistruct
Task
Base
Self-generaled
Other-generaled
MATH
55.4
78.2
80.2
MATH
57.5
74.4
708
74.2
99.0
80.2
Minerva-Math
13.6
33.1
42.3
Minerva-Math
23.9
42.3
28.3
32.8
13.2
42.3
91.6
92.4
90.9
GSMSK
OlympiadBench
16.1
42.5
41.6
OlympiadBench
20.0
36.4
36.3
37.2
19.9
41.6
AIME224
AIME24
6.7
23.3
10.0
23.3
3.3
20.0
AMCZ3
40.0
67.5
67.5
AMC23
37.5
57.5
50.0
62.5
37.5
67.5
AVG
37.5
53.3
47.3
53.2
35.1
57.1
AVG
37.8
35.1
57.1
Table 5: Performance comparison of SFT Table 6: Comparison between self-generated (by Qwen2.5-Math- and original solutions (from WebInstruct) for CFT training.
and CFT with different training datasets on Owen2.5-Math-7B.
IF Eval
Task
SPT
GPT mini-CFC
GPT-49-1120-CFT
MT-Bench
MATH
62.0
73.9
80.2
strict
loose
Minerva Math
12.5
36.4
42.3
Qwen2.5-Me 5-ren255-Mtth-7B
4.79
GSM8K
78.8
84.5
Qwen2 5-Matth-7---nnstruc
0.333
0.345
5.49
OlympiadBench
22.1
41.6
Swen2.5-Math-7BB-
0.315
0.330
5.23
AIME24
16.7
20.0
20.0
Owenn255Mathh7B-eeniiddd-.-
0.328
0.341
5.41
AMC23
50.0
62.5
67.5
Qwwen255Maahhh7BBPP7H65555
0.325
0.343
5,38
AVG
40.4
52.0
57.1
Qwen2.55Math-7RR....
0.335
0.362
6.49
Table 7: Performance comparison of CFT us- ing different teacher critique models (GPTT40 and mini) on
Table 8: Performance comparison across different models on Instruction Following (IF Eval, instruction-level) and general in- struction alignment (MT-Bench).

---

Published as a conference paper at COLM 2025
Dataset Source We ablate the impact of different training datasets model performance. As shown in Table 5, when trained with SFT, both MetaMathQA and NuminaMath achieve better performance than WebInstruct (47.3% and 37.5% vs. 35.1% on average), indicating their higher data quality. However, when trained with CFT, WebInstruct surprisingly achieves the best performance (57.1%), outperforming both MetaMathQA and NuminaMath. This suggests that the effectiveness of CFT is not solely determined by the quality of solution data. Instead, by learning identify and critique incorrect solutions, the model can develop stronger mathematical reasoning capabilities even from imperfect demonstrations, highlighting the robustness and effectiveness of our critique-based learning approach.
solutions gener- ated by Owen2.5-Math-77 itself and original solutions from the WebInstruct dataset. Table 6 shows that using original solutions achieves comparable performance (57.1% 1%%1% vs. 55.1% % on average), with some variation across different benchmarks. The improvement is more noticeable on challenging datasets like Minerva-Math (9.2% increase). These results demon- strate that CFT robust to different solution sources and can effectively learn from both model-generated and original solutions from the WebInstruct dataset.
Response Source We compare two sources of solutions for CFT training:
Teacher Critique Model To understand the impact of critique model quality on CFT, we compare the performance when using GPT-4o-mini and GPT-4o-1120 critique models in Table 7. First, we observe that even with a relatively modest critique model GPT-4o-mini, CFT significantly outperforms SFT-verified baseline (52.0% vs. 40.4% average), with substantial improvements on MATH (11.9% increase) and Minerva-Math (23.9% increase). This demonstrates the effectiveness of CFT without requiring an extremely powerful critique model. Furthermore, using stronger critique model GPT-44-1120 leads to even better performance across all benchmarks (57.1% on average), with notable gains on GSM8K (6.4% increase) and OlympiadBench (6.5% increase). These results confirm that while CFT is effective with modest critique models, stronger critique models can provide more accurate and insightful feedback, leading to better mathematical reasoning capabilities. In the future, we plan to leverage ol or even 03 as teacher critique model to understand the potential.
Model
MATH
Minerva-Math
GSMSK
OlympiadBench
AIME24
AMC23
1aamn225-att-7B-CFF
80.2
42.3
90.9
41.6
20.0
67.5
57.1
Mix 50K + 50K AceMath SFT
78.8
40.8
40.3
16.7
62.5
CFT + 50K AceMath SFT (two-stage)
76.2
35.4
88.2
42.5
10.0
57.5
51.6
AceMath SFT (50K)
33.1
35.1
Table 9: Comparison of strategies for combining CFT with high-quality SFT data
Model
MATH Minerva-Math
GSM8K
OlympiadBench
AIME24
AMC23
AVG
WebInstruct-SFT
59.0
13.2
77.4
19.9
3.3
37.5
35.1
Weblnstruut-eerfffedSFFT
62.0
12.5
78.8
22.1
16.7
50.0
40.4
Weblnstruct-GPTloo-FF
73.2
25.7
90.0
37.6
13.3
62.5
50.4
Weblnstruct-CFF
80.2
42.3
90.9
41.6
20.0
67.5
57.1
WeblnstructCFF--S..
78.4
40.4
90.4
40.1
16.7
65.0
55.2
Table 10: Ablation on controlling token length (CFT vs. SFT)
Combination with High-Quality SFT Data We investigate whether combining CFT with high-quality SFT datasets can further improve performance. As shown in Table 9, mixing AceMath SFT data or applying it in a two-stage training setting slightly degrades perfor- mance compared to pure CFT training (54.8% and 51.6% vs. 57.1% on average, respectively). This indicates that incorporating standard supervised instruction data may conflict with critique-based training objectives. Therefore, purely learning to critique incorrect solutions is more effective for reasoning tasks than jointly learning to imitate high-quality solutions.
Controlling for Token Length Since critique-based training typically involves longer se- quences, one might suspect this as the primary reason for its superior performance. To address this concern, select WebInstruct data with token lengths similar to standard SFT training data and perform critique fine-tuning The results in Table 10 show that even when controlling token lengths, shorter-length critique data still significantly outperforms SFT
7

---

Published as a conference paper at COLM 2025
baselines (55.2% vs. 50,4% average). This confirms that the advantage of CFT comes from the critique-based learning mechanism rather than merely increased token length.
General Generation and Instruction-Folllviin Capability
To assess whether CFT affects the general generation and instruction-folloiing abilities, evaluate the models MT-Bench (Zheng et al., 2023), benchmark for general in- struction alignment, and IF Eval (Zhou et al., 2023), comprehensive instruction-folloving evaluation suite. As shown in Table Qwen2.5-Math-7B-CFF achieves achieves score of 6.49 MT- Bench, significantly improving over the base Qwen2.5-Math-77 (4.79) and instruction-tuned Wraen2.5-Mtth-BB-nmme
(5.49). It also outperforms various SFT approaches, including standard SFT (5.23), verified-SFT (5.41), and GPT40-SFT (5.38). On IF _F_Eval, Qwen2.5-Math- 7B-CFT demonstrates superior instruction-followigg ability with the highest scores in both strict mode (0.335) and loose mode (0.362), outperforming all other variants including the instruction-tunet model. These comprehensive improvements across different benchmarks demonstrate that critique-based training not only enhances mathematical reasoning but also improves general sottuuttion-oolooinng capabilities and text generation quality. This indicates that the critique mechanism helps the model develop more robust general capa- bilities even when trained primarily on mathematical content, suggesting positive transfer between specialized and general abilities.
4 Limitations
The Noisy Critique Data Our ablation study indicates that the quality of critique feedback notably influences the effectiveness of CFT. manual inspection of 50 critique samples generated by GPT-4o-1120 on WebInstruct reveals that roughly 20% contain inaccuracies, such as misjudging correct steps, missing errors, or providing imprecise explanations (see Appendix A.3 for examples). This highlights the importance of using higher-quality critique data to further enhance CFT. Future research could explore automated critique verification methods or develop curated datasets with human-verified critiques improve mathematical reasoning capabilities.
Limitations of Self-Critique
We explored incorporating self-critique mechanisms into our framework, where the model critiques and iteratively refines its own outputs. However, we found these approaches consistently underperformed compared to direct inference. In particular, self-critique suf- fered from inconsistent critique standards, where the model either overlooked genuine errors or mistakenly flagged correct solutions as incorrect. Additionally, increasing sam- pling temperatures necessss refinement-introdee necessary to maintain diversity and avoid repetition during iterative instability, further degrading performance. Due to these challenges, our final CFT implementation relies direct inference without self-critique. For detailed experimental results, specific methodological comparisons, and further analysis of observed issues, please refer to Appendix A.4.
5 Related Work
5.1 Instruction Tuning
Instruction tuning one of the most crucial part of aligning pre-trained language models with human expectations. The current instruction-tuniig datasets are either based on (1) human annotation: such as FLAN (Wei et al., 2022), TO (Sanh et al., 2022), SuperNT (Wang et al., 2022), which compiles large instruction-tuning datasets from existing human-labeled datasets; and (2) model synthesis: such as Self-Instruct (Wang et al., 2023b), Wizard (Xu et al., 2024), WildChat (Zhao et al., 2024), which creates instruction-tunite
datasets by synthesizing from powerful LLMs (Achiam et al., 2023). Both types of instruction datasets have shown great performance improvement of LMs general evaluation tasks. More recently, Tulu (Wang et al., 2023a) and Tulu-3 (Lambert et al., 2024) have explored how to combine existing post-training data and algorithms to maximize LMss' performance.
8

---

Published as a conference paper at COLM 2025
5.2 Mathematical Instruction Tuning
Taking this further, math-instructed models have been developed to advance LLM perfor- mance in the mathematical domain (Luo et al., 2023; Yue et al., 2024a; Yu et al., 2024; Li et al., 2024a; Mitra et al., 2024; Li et al., 2024b). Recently, there has been a wave to scale up the math instruction dataset to millions of examples like MAmmoTH2 (Yue al., 2024b), Open-Vathhnntruct (Toshniwal et al., 2024), and AceMath (Liu et al., 2024) and Qwen2.5- Math-Instruct (Yang et al., 2024b). These methods have shown tremendous performance gains on math reasoning datasets. However, we also observe diminishing marginal gain by further scaling the instruction data up, suggesting that a more efficient training algorithm is needed. In this paper, we aim to challenge SFT and propose much more efficient learning algorithm CFT and show similar performance with only 1-10% of SFT data.
5.3 Critique Learning
Teaching AIs to critique has been long standing goal in the pursuit of AGI.
Self-Correction The concept of sseff-correction has been emerged as a promising direction in LLMs since 2023. There has been a line of work (Madaan et al., 2024; Welleck et al., 2023; Shinn et al., 2024; Bai et al., 2022; Ganguli et al., 2023; Gou et al., 2023) aiming to use feedback from the model itself to further improve its performance. However, later work (Huang et al., 2023; Valmeekam et al., 2023) revealed that the self-correction in reasoning is not quite reliable. More recently, with the rise of GPT-ol (Jaech et al., 2024), LLM self-correction has again demonstrated its potential to improve LLMs' own reasoning capabilities.
Critique Model A critique model differs from self-correction in that it employs specialized model to provide feedback to another model during generation. In mathematical reasoning, critique models often take the form of reward models. Recent work has explored both outcome-based reward models (Uesato et al., 2022; Yang et al., 2024b) and process- based reward models (Wang et al., 2024a; Lightman et al., 2023a; Yuan et al., 2024) to enhance reasoning capabilities of language models. However, these approaches typically focus on directly predicting reward scores without explicitly providing intermediate reasoning or explanations. The most similar prior work to ours is critique-out-lout (Ankner et al.), which functions solely as a reward estimator rather than directly guiding the generation process.
In contrast, our proposed approach differs substantially from these existing methods. We leverage critique feedback explicitly as a training objective to encourage deeper understand- ing and reasoning about the given problems. At inference time, the trained model generates responses directly, without involving any explicit critique or iterative refinement steps.
Conclusion
In this paper, we introduced Critique Fine-Tuning (CFT), a novel paradigm that trains lan- guage models to critique and analyze responses rather than imitating them as in traditional SFT. Experiments demonstrated that CFT consistently outperforms SFT by 4-10% on math- ematical reasoning benchmarks, achieves comparable performance to resource-intensive RL methods using significantly fewer training examples (50K vs. 2M+) and compute (8 H100 GPU-hours), and generalizes effectively to broader STEM domains. Interestingly, even without explicit instruction tuning via traditional SFT or RL, CFT-trained models inherently capabilities, challenging conventional assumptions regarding the necessity of imitation-based training for instruction-followingg tasks. These findings suggest that explicitly teaching models to identify and critique incorrect reasoning can significantly enhance their reasoning and generalization capabilities. Future research directions include improving the quality and coverage of critique data, enabling models to perform self-critique for continual self-improvement, combining CFT with complementary training paradigms like RL, extending the approach to multi-modal scenarios, and further investigating its theoretical foundations. Overall, we believe CFT represents a promising step forward in making language model training more efficient, robust, and effective, poten- tially reducing the computational and data requirements for developing bighhperformin models while substantially improving their reasoning and instruction-followinng abilities.
exhibit strong instruction-folowiii

---

Published as a conference paper at COLM 2025
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. ar preprint ar Xf::23003.08 2023.
Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan Daniel Chang, and Prithviraj Ammanabrolu. Critique-out-loud reward models. In Pluralistic Alignment Workshop at NeurlPS 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609 2023.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitu- tional ai: Harmlessness from feedback. arXiv preprint ar Xft::2212.08073, 2022.
Wenhu Chen, Ming Yin, Max Ku, Pan Lu, Yixin Wan, Xueguang Ma, Jianyu Xu, Xinyi Wang, and Tony Xia. Theoremqa: theorem-driven question answering dataset. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7889-7901, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. aiv preprint ar Xri::2110.14166 2021.
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang, Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv, Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process reinforcement through implicit rewards, 2025. URL https://arxiv http:://arxd..org/bbb///Bee 01456.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. arXiv preprint at (tt::2407.217833 2024.
Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamilè Lukošiūté, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. The ca- pacity for moral self-correction in large language models. arXiv preprint arXiv:2302.07459, 2023.
Zhibin Gou, Zhihong Shao, Yeyun Gong, Yujiu Yang, Nan Duan, Weizhu Chen, et al. Critic: Large language models can self-correct with tool-interactive critiquing. In The Twelfth International Conference on Learning Representations, 2023.
Xinyu Guan, Li Lyna Zhang, Yifei Liu, Ning Shang, Youran Sun, Yi Zhu, Fan Yang, and Mao Yang. rstar-math: Small Ilms can master math reasoning with self-evolved deep thinking, 2025. URL htt.s: //arx//arxivvii/ www.o//b/b.///1111111 04519.
Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. Deepseek-coder: When the large language model meets programming- the rise of code intelligence. arviv preprint arXio:2401.14196 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. Xiv preprint ar XrXiv:2501.12944 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging bench- mark for promoting agi with olympiad-leve bilingual multimodal scientific problems. ar Xiv preprint ar Xit::2402 14008, 2024.
10

---

Published as a conference paper at COLM 2025
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations, 2023.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed EI-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. Xrv preprint ar Xixo:24122.1672) 2024.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arviv preprint ar Xrie:22300.06825, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, al. Mixtral of experts. arviv preprint Kic:2401.04088 2024.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T," ulu Pushing frontiers in open language model post-training. arv preprint ar Xi1o:24411.11124 2024.
ShengyiHuangg
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Chen Li, Weiqi Wang, Jingcheng Hu, Yixuan Wei, Nanning Zheng, Han Hu, Zheng Zhang, and Houwen Pengg Common 7b language models already possess strong math capabilities. ar arXiv preprint axX1o:2403.04766, 2024a.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13, 2024b.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.20055 2023a.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. arv preprint ar drXie:2305.20055 2023b.
Zihan Liu, Yang Chen, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Acemath: Advancing frontier math reasoning with post-training and reward modeling. arXiv preprint ar arXie:2412.15084 2024.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang, Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint ar Xiv:2308..00 09583, 2023.
Ziyang Luo, Can Xu, Pu Zhao, Qingfeng Sun, Xiubo Geng, Wenxiang Hu, Chongyang Tao, Jing Ma, Qingwei Lin, and Daxin Jiang. Wizardcoder: Empowering code large language models with evol-instruct. In The Twelfth International Conference on Learning Representations, 2024.
11

---

Published as a conference paper at COLM 2025
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of sims in grade school math. arXiv preprint ar Xroo:22402.18800, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google- proof q&a benchmark. Xrv preprint arXio:2311.12022, 2023.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnau Stiegler, Arun Raja, Manan Dey, et al. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Represenlations, 2022.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, al. Gemini: family of highly capable multimodal models. Xrv preprint ar XrXiv:2312.11805, 2023.
Shubham Toshniwal, Wei Du, Ivan Moshkov, Branislav Kisacanin, Alexan Ayrapetyan, and Igor Gitman. Openmathinstructt::: Accelerating ai for math with massive open-source instruction data. ar Xiv preprint ar Xio::2410.01560, 2024.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXio preprint ar xrXi::2211.14275, 2022.
Karthik Valmeekam, Matthew Marquez, and Subbarao Kambhampati. Can large language models really improve by self-critiquing their own plans? NeurIPS 2023 Foundation Models for Decision Making Workshop, 2023.
Peiyi Wang, Lei Li, Zhihong Shao, Runxin Xu, Damai Dai, Yifei Li, Deli Chen, Yu Wu, and Zhifang Sui. Math-shepherd: Verify and reinforce lims step-by-step without human annotations. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 9426-9439) 2024a.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, et al. hupernaaturalinstuutionss Generalization via declarative instructions on 1600+ nlp tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5085-5109, 2022.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? exploring the state of instruction tuning on open resources. Adoances in Neural Information Processing Systems, 36.74764-74786, 2023a.
12

---

Published as a conference paper at COLM 2025
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self- instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13484-13508, 2023b.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-pro: A more robust and challenging multi-task language understanding benchmark. In The Thirty-eight Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2024b.
URL https: http::///pppeeveeeeeeeee www.:neefforee ottp
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022.
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. In The Eleventh International Conference on Learning Representations, 2023.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. WizardLM: Empowering large pre-trained language models to follow complex instructions. In The Twelffh International Conference on Learning Representations, 2024. URL https://openreviiee http:/wwwwwweeffuteeeeeee nett/forummid-CFX/aaaN
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. ar preprint arXiv:2412.15115, 2024a.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement arXiv preprint ar Xix::249..12122 2024b.
Longhui Yu, Weisen Jiang, Han Shi, YU Jincheng, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathemati- cal questions for large language models. In The Twelfth International Conference on Learning Representations, 2024.
Lifan Yuan, Wendi Li, Huayu Chen, Ganqu Cui, Ning Ding, Kaiyan Zhang, Bowen Zhou, Zhiyuan Liu, and Hao Peng. Free process rewards without process labels. arviv preprint arXio:2412.019811 2024.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang. Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. In The Twelfth International Conference on Learning Representations, 2024a.
Xiang Yue, Tianyu Zheng, Ge Zhang, and Wenhu Chen. MAmmoTH2: Scaling instructions from the web. In The Thirty-eighth Anmual Conference on Neural Information Processing Systems, 2024b. URL https: /ttss///opeeeeieeee httpppparwiiiiieeooooouoroa/aaa met//forumiidyyyusdhPl.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint ar arx::25503.1889) 2025.
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wild- chat: 1m chatgpt interaction logs in the wild. In The Twelfth International Conference on Learning Representations, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595- 46623, 2023.
13

---

Published as a conference paper at COLM 2025
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-followii evaluation for large language models. arXio preprint ar Xit:2231..07911, 2023.
A Appendix
A.1 GPT-4o Case Study for Critique Data Generation
To illustrate our approach for generating critique data, we present example using GPT-4o as the critique generator. The following shows the input prompt, student solution, and the resulting critique generated by GPT-4o.
Input Prompt to GPTT-4o
You are science expert. A student is trying to solve a question, please explain briefly whether his answer is correct or not. Finally, conclude your judgement with right/ hight/wrong wrong [END]'
'Conclusion:
Question:
How does mitosis contribute the maintenance and continuity of life in eukaryotic organisms?
Solution:
Mitosis plays crucial role in maintaining and continuing life in eukaryotic organ- isms by:
Grooth and Development: Mitosis allows for the production of new cells, enabling organisms to increase size and develop from a single cell into a multicellular entity. Tissue Repair and Regeneration: When tissues are damaged, mitosis generates new cells to replace the lost or injured ones, facilitating wound healing and tissue regeneration. Cell Replacement: Mitosis continually replaces old or dying cells with new ones, maintaining the integrity and function of tissues and organs. RRpro- duction: In asexual reproduction, mitosis is the primary mechanism by which new individuals are produced. It also plays role in sexual reproduction by generating gametes (eggs and sperm) that combine form a new organism.
Without mitosis, eukaryotic organisms would be unable to grow, repair themselves, or reproduce, ultimately compromising their survival and the continuity of life.
GPT-4o Generated Critique
The student's answer is mostly correct, but there is a significant error in the ex- planation regarding the role of mitosis in reproduction. While mitosis is indeed essential for growth, tissue repair, and cell replacement, it is not involved in the production of gametes (eggs and sperm) in sexual reproduction. Gametes are pro- duced through a different process called meiosis, not mitosis. Mitosis is involved in asexual reproduction for some organisms, but it does not play a direct role in sexual reproduction.
The rest of the answer correctly highlights the importance of mitosis growth, tissue repair, and cell replacement, which are vital for the maintenance and continuity of life. Conclusion: wrong [END]
This example demonstrates how GPT-4o effectively identifies the conceptual error the student's answer regarding reproduction. The critique correctly points out that meiosis, not mitosis, is responsible for gamete production, while acknowledging the accurate compo- nents of the answer. The structured output concludes with clear judgment, facilitating further computational processing of the critique data. We also used critique data obtained through this instruction to filter and create the WebInstruct-GPT-4owvee SFT dataset.
14

---

Published as a conference paper at COLM 2025
A.2 GPT-4o Case Study for Web dataset Generation
To illustrate how generated the Weblnstruct dataset, present example prompt and the corresponding output from GPT-4o.
Input Prompt to GPT-4o
You are science expert. You need to answer given question clearly and conclude your answer with 'nswer: |YOUR ANSWER] Question: How do you find the derivative of a quadratic function using the Power Rule?
GPT-4o Generated Answer
To find the derivative of quadratic function using the Power Rule, follow these steps. A quadratic function is generally in the form:
f(x) = ax² + bx + where a, b, and are constants.
The Power Rule for differentiation states that if you have function f(x) = x", then its derivative (xx is given by:
fx( = nxx"
Apply the Power Rule to each term of the quadratic function:
Differentiate ax2: Here, 2, using the Power Rule: dx (ax2) -2x2211 2ax
2. Differentiate bx: - Here,
d 1, so using the Power Rule: dx (bx) = 1 bxl b
Differentiate c: Since a constant, its derivative is 0.
Combine these results to find the derivative of the entire quadratic function: f'(x) = 2ax +
Thus, the derivative of the quadratic function f(x) = ax² + bx + is: f((x 2ax +
Answer: The derivative of quadratic function f(x) ax² + bx using the Power Rule is (x) 2ax + b.
This example demonstrates how GPT-4o provides clear, structured mathematical explana- tions with step-by-step reasoning. The response includes proper mathematical notation, methodical breakdown of the differentiation process, and concise final answer. These high-quality responses formed the foundation of our WFb dataset.
A.3
Case Study: Noisy Critique Data
Here we present examples of noisy critique data and analyze their impact on model perfor- mance:
15

---

Published as a conference paper at COLM 2025
A.3.1 Case 1: Incorrect Critique of Valid Solution
Case Incorrect Critique of Valid Solution
Problem
What is flawed in the counter-proof attempt against the statement that two randomly chosen points the surface of an N-dimensional unit sphere with large N are nearly orthogonal? The counter-proof involves selecting random arc with length in the interval (0, st) and placing it the sphere's surface at random orientation, claiming that the ends of the arc don't tt have high probability of being nearly orthogonal.
Original Solution
The flaw in the counter-proof attempt lies in the misunderstanding of the uniform distribution of randomly selected points on the sphere's surface. The original statement assumes that the points are chosen independently and uniformly, whereas the counter-proof proposes specific distribution by creating random arc. This distribution does not represent the uniform selection of two points on the sphere.
The Bertrand Paradox, related concept in probability, demonstrates that different methods of selecting random chord in circle can lead to different probabilities for the chord being longer than half the circle's 's radius. Similarly, in the N-dimensional case, the probability of orthogonality depends on the method of random selection, and the are method does not refute the original statement about two uniformly chosen points.
To address the issue, one should focus on the probability of two points being nearly orthogonal when they are each chosen independently and uniformly from the sphere's 's surface, which is the basis of the original statement.
Incorrect Critique
Your explanation on the right track, but it could be clarified further. The key issue with the counter-proof is that it introduces biased method of selecting points on the sphere. By constructing a random are and using its endpoints, the counter-proof imposes specific correlation between the two points. This violates the assumption of independence and uniformity in the original statement.
In high-dimensional spaces (large N), the geometry of the sphere leads to the phenomenon where two points chosen independently and uniformly are almost always nearly orthogonal. This is result of the concentration of mea- sure phenomenon, which is a well-established property in high-dimensional geometry. The cuuter--proof 's method of selecting points via an arc does not reflect this uniform and independent selection process, and thus cannot serve as a valid refutation of the original statement.
Conclusion: wrong [END]
Analysis
The critique overly harsh, as the original solution already provides a clear and sufficient answer to the core question.
16

---

Published as a conference paper at COLM 2025
A.3.2 Case 2: Missed Errors in Solution
Case 2: Missed Errors in Solution
Problem
43x² Find the roots of the following polynomial: 5 + 5x + 5
Original Solution
To find the roots of the polynomial, we can use the quadratic formula:
x
where a, 6, and are the coefficients of the polynomial.
In this case, we have:
43 5 b 5, 28
Substituting these values into the quadratic formula, we get: x =
-5+ ())
Simplifying this expression, we get:
-5 25
-25 238
51v 213
-5+i/213
2
2
x
x
2(
5tiv 4191
Therefore, the roots of the polynomial are 5:11/4191
86
Inadequate Critique
Let's verify this step by step:
The quadratic formula -bt//2-aa
was correctly applied For the given 28 When we substitute:
2a
equation 43x2
ク + 5x
5:
a
5
5
b2 25 4ac = 4
( (
28
238 12
--acc = 25 238 = -213 Therefore:
x = -51i/213 _)
-411
The denominator 86 comes from 2 43 86,
86
5
and V213. VI9667 Rs /491
Thus, the answer [END]
-5+
1191
must be correct. Crituque Conclusion: correct
Correct Analysis
The critical calculation error occurs in step 4: The transformation from
-5+1v213 to -5+ 4191 is incorrect. 1) While the denominator's 's was properly -
handled (2 43 = )) 2) It ignored that this coefficient 5 must affect the nu-
5
merator as well The correct simplification should be: -5+ 2515 -
A.4 Detailed Analysis of Self-Critique Methods
We experimented with two self-critique approaches during inference:
1. Single-pass self-critique: The model generates and critiques its solution within inference pass, and regenerates if errors detected.
17

---

Published as a conference paper at COLM 2025
2. Two-stage self-critique: The model separately evaluates its solutions in dedicated critique step and iteratively regenerates (up to attempts) until a satisfactory solution is found.
Table 11 shows the comparative performance of these methods across various temperature settings.
Method
Temperature
MATH
Minerva-Math
0.0
80.2
42.3
0.1
78.8
38.9
Direct inference
0.3
77.5
37.7
0.6
75.2
34.1
0.1
77.2
36.7
Single-pass self-critique
0.3
76.1
35.2
0.6
73.5
34.3
0.1
77.9
38.2
Two-stage self-critique
0.3
75.8
35.4
0.6
74.6
34.6
Table 11: Comparison of inference methods across various temperature settings.
Our analysis revealed several issues that limit the effectiveness of self-critique:
1. Inconsistent critique standards: The model often applies inconsistent criteria when evaluating its own work, leading to either missed errors (false negatives) or incorrectly flagging valid solutions as problematic (false positives).
2. Temperature sensitivity: Higher temperatures introduce variability that com- pounds across iterations, making the overall process less stable. The single-pass method drops from 77.2% to 73.5% on MATH as temperature increases from 0.1 to 0.6, with similar trends on Minerva-Math.
3. Regeneration inefficiency: Even when errors are correctly identified, the model often struggles to effectively address the specific issues in subsequent regeneration attempts, sometimes introducing new errors while fixing others.
4. Computational overhead: The iterative nature of self-critiuue significantly increases inference time and computational cost, with diminishing returns performance.
The two-stage method performs slightly better than the single-pass approach but still underperforms relative to direct inference. This suggests that while our model benefits from critique-based training, applying self-critique during inference creates additional complexity that the model struggles to navigate effectively.
A.4.1 Inference Method Prompts
We present the prompt templates used in our different inference approaches: Direct Inference Template
Direct Inference Prompt Template
Please reason step by step, and put your final answer within boxed.
Question: [Problem text here]
Answer: Let's solve this step by step: [Solution steps] Therefore, the final answer is ANSWER
Single-pass Self-critique Template
18

---

Published as a conference paper at COLM 2025
Single-pass Self-critique Prompt Template
Please reason step by step to solve this problem and then critique your solution. If any errors are found, provide corrected solution. Please put your final answer within O
Question: [Problem text here]
Answer: Let's solve this first: [Initial solution steps] Therefore, my initial answer is
ANSWER
Critique: [Critique points]
[If errors found: Based on my critique, let me provide a corrected solution: Corrected solution: ...]
Two-stage Self-critique Template
Two-stage Self-critique Process
Stage 1 (Solution Generation):
Please reason step by step, and put your final answer within boxed.
Question:
[Problem text here]
Answer: Let's solve this step by step: [Solution steps] Therefore, the final
answer ANSWER
Stage 2 (Critique):
Please critique whether the following solution to the question is correct. [Problem text here] Solution: [Previous solution] Critique: 1. [Critique point 1] 2. [Critique point
Question:
Critique Conclusion:
Correct/Incorrect
If the conclusion is "Incorrect" the process returns to Stage for new solution attempt. This iterative process continues until either:
The critique conclusion becomes "Correct", indicating satisfactory solution has been found, or
The maximum number iterations (8) is reached, in which case the last generated solution is used the final answer.
The complete process can be represented as:
For in range(1, 9): 1. Generate solution (Stage 1) 2. Critique solution (Stage 2) If critique conclusion is "Correct": Return current solution and exit If == 8: Return current solution 5, Continue to next iteration
19

---

Published as a conference paper at COLM 2025
A.4.2 Obseroed Issues in Self-Critique Methods
Common Issues with Self-Critique Methods
Our analysis revealed several systematic issues with self-critique approaches:
Inconsistent Self-Evaluation
We observed the following patterns of inconsistency in self-evaluation:
Over-lenient Evaluation: Cases where the model failed identify clear logical errors in its own reasoning
False Negatives: Instances where correct solutions were incorrectly flagged erroneous
Specific Failure Cases
Here we document specific examples where self-critique methods failed:
Example 1: Over-lenient Evaluation
Problem
Chandra has four bowls. Each one is a different color (red, blue, yellow, green). She also has exactly one glass the same color as each bowl. If she chooses bowl and glass from the cupboard, how many pairings are possible? One such pairing is a blue bowl and yellow glass.
Original Solution
To solve this, we need to determine the total number of possible pairings of one bowl and one glass when there are four bowls and four glasses, each uniquely paired by color.
Reasoning: 1. Chandra has **4 bowls** (red, blue, yellow, green) and **4 glasses** (red, blue, yellow, green). 2. Each bowl matches exactly one glass by color. For example: The red bowl matches with the red glass The blue bowl matches with the blue glass The yellow bowl matches with the yellow glass The green bowl matches with the green glass 3. Since each bowl must pair with its corresponding glass, there is only #*1 valid pairing** (a red bowl with red glass, etc.).
Self-Critique
The question asks for the number of possible pairings, but based the problem setup, only one valid pairing exists. The reasoning above is correct. *CConclusion: Correct [END]**
20

---

Published as a conference paper at COLM 2025
Example 2: False Error Detection
Problem
Chandra has four bowls. Each one is a different color (red, blue, yellow, green). She also has exactly bowl and glass from the cupboard, how many pairings are possible? One such pairing is a blue bowl and yellow glass.
glass the same color each bowl. If she chooses
Original Solution
To solve this problem, we need to calculate the total number of possible pairings of bowls and glasses. Each bowl can be paired with any of the glasses.
Here are the steps:
Chandra has 4 bowls and 4 glasses. 2. Each bowl can be paired with any one of the 4 glasses. 3. Therefore, the total number of pairings is calculated as the product of the number of bowls and the number of glasses:
4 16
Self-Critique
The reasoning provided incorrect. While the calculation of the total number of pairings as 4 x 4 = 16 is mathematically correct, the explanation is flawed. The problem does not state that each bowl must be paired with a glass different color. Therefore, the assumption that each bowl can only be paired with glass the same color is invalid. Conclusion: *Critique Conclusion: Incorrect**
21


Reviews:
[{"review_id": "ues2C2Q1zh", "reviewer": "Reviewer_XEXb", "summary": "This paper proposes Critique Fine-Tuning (CFT) as a compelling alternative to standard Supervised Fine-Tuning (SFT) for reasoning tasks in language models. Instead of training models to imitate correct answers, CFT trains them to critique noisy responses, thereby fostering deeper reasoning and analytical capabilities. The authors construct multiple critique datasets (e.g., WebInstruct, MetaMath, NuminaMath) where critiques are generated by GPT-4o. Empirical results show CFT outperforms strong SFT baselines by 4–10% on various math reasoning benchmarks and even competes with compute-intensive RL-based approaches like SimpleRL. Ablations demonstrate robustness across data sources and critique models. The work is well-motivated, clearly presented, and makes a strong case for critique-based learning as a scalable, data-efficient training paradigm.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper introduces a learning paradigm in which language models are trained to critique rather than imitate, drawing a clear and compelling motivation from human learning processes.\n- The proposed Critique-Fine-Tuning (CFT) method achieves strong empirical performance, significantly outperforming supervised fine-tuning (SFT) on multiple benchmarks—for instance, achieving a +6.7% gain on Qwen2.5-Math-7B—and performing comparably to reinforcement learning approaches like SimpleRL while being 140 times more compute-efficient.\n- The method demonstrates remarkable data efficiency, with models trained on only 50K critique examples matching or even surpassing the performance of models trained with millions of supervised examples.\n- The approach shows robust generalization capabilities across different datasets, teacher models, and reasoning domains. It improves both reasoning accuracy and general instruction-following ability, achieving, for example, a +1.0 improvement on MT-Bench over the best SFT baseline.\n- The paper is clearly written and well-organized, featuring thorough ablation studies, strong conceptual motivation, and effective visualizations that together enhance the clarity and persuasiveness of its core contributions.", "reasons_to_reject": "- The paper demonstrates strong empirical performance, but it does not provide a rigorous theoretical explanation for why critique-based learning enhances reasoning capabilities. This absence of formal justification weakens the conceptual grounding of the work.\n- The model relies on critiques generated by GPT-4o, which introduces a dependency on potentially biased or inaccurate supervision. Manual inspection reveals that around 20% of the critiques are flawed, raising concerns about the reliability of the training signal.\n- The evaluation focuses predominantly on math-related benchmarks, making it unclear whether the proposed method generalizes well to non-STEM or open-ended domains. Additional experiments are needed to establish the broader applicability of the approach.\n- Although the paper briefly mentions self-critique mechanisms, it does not explore in depth why these methods fail or how they might be improved. As a result, the potential for self-improving critique-based training remains insufficiently addressed.\n- More related work and baselines should be discussed, e.g., [1][2].\n\n[1] Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision\n[2] Critic-CoT: Boosting the reasoning abilities of large language model via Chain-of-thoughts Critic", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qh4Hcur89q", "reviewer": "Reviewer_Yors", "summary": "This paper proposes a novel method for training large language models (LLMs) using synthetic data derived from critiques of instruction-tuning datasets, rather than imitating high-quality responses. The proposed Critique Fine-Tuning (CFT) framework is shown to be both data- and compute-efficient, requiring significantly fewer resources than traditional supervised fine-tuning (SFT) while consistently outperforming SFT-based models across a wide range of benchmarks.", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is clearly written and well-organized. The appendix provides detailed insights into experimental settings and analyses.\n2. The method is straightforward and easy to implement, yet novel and conceptually elegant.\n3. The experiments are thorough and diverse. Ablation studies further support the robustness and generality of the approach.\n4. The proposed method is highly efficient: it achieves strong performance using only 50K training examples and 1 hour of training on 8×H100 GPUs, far less than competing methods.", "reasons_to_reject": "Most experiments are conducted on English datasets. Apart from MMLU-Pro, the method’s effectiveness in multilingual or non-English settings remains unclear.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bHQkeqTmHP", "reviewer": "Reviewer_tGLF", "summary": "This paper introduces Critique Fine-Tuning (CFT), a novel approach to training language models that departs from traditional Supervised Fine-Tuning (SFT). Instead of directly imitating responses, CFT trains models to critique and analyze generated responses, focusing on identifying and correcting errors in reasoning. The authors demonstrate that this approach consistently outperforms SFT by 4-10% on mathematical reasoning benchmarks, achieves comparable performance to resource-intensive reinforcement learning (RL) methods using significantly fewer training examples and compute resources, and generalizes effectively to broader STEM domains. Notably, CFT-trained models exhibit strong instruction-following capabilities without explicit instruction tuning, challenging conventional assumptions about the necessity of imitation-based training. The core idea is that by explicitly teaching models to identify and critique incorrect reasoning, their reasoning and generalization capabilities can be significantly enhanced. The authors achieve this by fine-tuning models on a dataset of question-response pairs, where the response is a critique of the model's initial answer. This critique is generated by a powerful language model, GPT-4o, and includes an assessment of the answer's correctness, identification of errors, and a corrected solution. The experiments are thorough, comparing CFT with various SFT methods and RL-based approaches, and the results provide strong empirical support for the effectiveness of CFT. The paper also explores the limitations of CFT, including the impact of noisy critique data and the challenges of self-critique mechanisms.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a compelling case for Critique Fine-Tuning (CFT) as an alternative to traditional Supervised Fine-Tuning (SFT), backed by strong empirical evidence and thoughtful analysis. The experimental evaluation is comprehensive, covering a wide range of open- and closed-source LLMs across multiple mathematical reasoning benchmarks. Notably, the authors demonstrate that models trained with CFT on relatively small datasets can match or surpass the performance of models trained with significantly larger datasets or more compute-intensive methods such as reinforcement learning. This indicates the method's high data efficiency and practical relevance. Additionally, the work challenges conventional views on instruction tuning by showing that CFT-trained models achieve strong instruction-following capabilities without explicit imitation-based training, highlighting the broader applicability and impact of critique-based learning.", "reasons_to_reject": "1. **Unclear Inference Procedure**: While the training setup is reasonably well described, using question and noisy response pairs as inputs and critiques as outputs, the testing procedure remains ambiguous. Specifically, it is unclear how the model is expected to generate final answers at inference time. If the critique merely assesses an initial (possibly incorrect) response without producing a corrected answer, it raises the question of how the model ultimately arrives at a valid final output. The paper does not explain whether critique outputs are further utilized to refine or replace the initial response, leaving a critical gap in understanding the method’s practical utility.\n\n2. **Insufficient Positioning Against Prior Work**: The idea of self-critique or language-model-generated feedback has been explored in several recent works [1,2,3], including multi-agent critique pipelines and feedback-based self-improvement. However, the paper lacks an adequate comparison to these methods, either theoretically or empirically. Without situating CFT more clearly in the context of this growing literature, it is difficult to assess its novelty or advantages over existing critique-based training frameworks.\n\n3. **Lack of Analysis in High-Quality SFT Regimes**: One of the motivating claims is that CFT can be more efficient or effective than conventional SFT, particularly when training data is limited or noisy. However, the paper does not sufficiently explore how CFT compares to high-quality, clean SFT scenarios. If critiques are generated from potentially flawed or noisy model outputs, it is unclear whether GPT-4o feedback is beneficial or harmful when the baseline data already contains correct answers. A deeper analysis or controlled experiments in such settings would help clarify the robustness and boundaries of CFT’s applicability.\n\n\n\n[1] Lan, T., Zhang, W., Lyu, C., Li, S., Xu, C., Huang, H., ... & Chen, K. (2024). Training language models to critique with multi-agent feedback. arXiv preprint arXiv:2410.15287.\n[2] Hu, C., Hu, Y., Cao, H., Xiao, T., & Zhu, J. (2024). Teaching Language Models to Self-Improve by Learning from Language Feedback. arXiv preprint arXiv:2406.07168.\n[3] Feng, T., Wang, Z., & Sun, J. (2023). Citing: Large language models create curriculum for instruction tuning. arXiv preprint arXiv:2310.02527.", "questions_to_authors": "Please check the \"reasons to reject\".", "extra_scores": {}}, {"review_id": "bpIvWQye8v", "reviewer": "Reviewer_G4No", "summary": "This paper introduces Critique Fine-Tuning (CFT), a novel supervised fine-tuning strategy for large language models (LLMs) aimed at enhancing their reasoning and analytical capabilities. CFT trains models to generate critiques given a query and a noisy response, using GPT-4o-generated critique data of the form ([query, noisy response], critique). Through extensive experiments, the authors demonstrate that CFT significantly improves performance across multiple reasoning benchmarks. The method shows robustness to imperfect data, different solution sources, and teacher models. The paper is clearly written and provides ablation studies to examine the contributions of various components.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper presents a simple but effective fine-tuning strategy that can be readily applied to existing LLMs.\n\n2. Empirical results show consistent improvements over baselines across diverse settings.\n\n3. The proposed CFT approach is robust to noisy data and model variations, making it broadly applicable.\n\n4. The authors provide meaningful ablations and experimental validations to support their claims.", "reasons_to_reject": "1. The core idea—training with critique data—may be perceived as intuitive or incremental, potentially lacking strong novelty.\n\n2. The connection between the CFT objective and actual model behavior post-fine-tuning is underexplored; more discussion or analysis is needed to justify why this approach leads to better reasoning.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposed a novel fine-tuning method called Critique Fine-Tuning (CFT), which is more effective than the popular SFT for reasoning tasks as demonstrated in the experiments. Reviewers show unanimous support to the acceptance expect for XEXb. The authors have prepared careful responses to the questions raised by XEXb. I believe it is ready for publication after the authors incorporate the rebuttal responses into the final version. Therefore, I recommend Accept for this work.
Decision: Accept