Offline Learning and Forgetting for Reasoning with Large Language Models
Authors: Tianwei Ni
Allen Nie
Sapana Chaudhary
Yao Liu
Huzefa Rangwala
Rasool Fakoor
Venue: Accepted by TMLR ()

Abstract: Leveraging inference-time search in large language models has proven effective in further enhancing a trained model's capability to solve complex mathematical and reasoning problems. However, this approach significantly increases computational costs and inference time, as the model must generate and evaluate multiple candidate solutions to identify a viable reasoning path. To address this, we propose an effective approach that integrates search capabilities directly into the model by fine-tuning it on unpaired successful (learning) and failed reasoning paths (forgetting) derived from diverse search methods. A key challenge we identify is that naive fine-tuning can degrade the model’s search capability; we show this can be mitigated with a smaller learning rate. Extensive experiments on the challenging Game-of-24 and Countdown arithmetic puzzles show that, replacing CoT-generated data with search-generated data for offline fine-tuning improves success rates by around 23% over inference-time search baselines, while reducing inference time by 180$\times$. On top of this, our learning and forgetting objective consistently outperforms both supervised fine-tuning and preference-based methods.

Published in Transactions on Machine Learning Research (10/2025)
Offline Learning and Forgetting for Reasoning with Large Language Models
Tianwei ; Allen Nie², Sapana Chaudhary?, Yao Liu', Huzefa Rangwala', Rasool Fakoor2
Mila Quebec Al Institute 8 Unáversité de Montréol, Amazoon Web Services
Reviewed on OpenReview: https: openreview. net/ forum? id= RFGraBUATe
Abstract
Leveraging inference-time search in large language models has proven effective in further en- hancing a trained model's capability to solve complex mathematical and reasoning problems. However, this approach significantly increases computational costs and inference time, the model must generate and evaluate multiple candidate solutions to identify viable reasoning path. To address this, we propose an elfective approach that integrates search capabilities directly into the model by fine-tuning it on unpaired successful (learning) and failed reasoning paths (forgetting) derived from diverse scarch methods. A key challenge we identify that naive fine-tuning can degrade the model's search capability; we show this can be mitigated with a smaller learning rate. Extensive experiments on the challenging Game-of-24 24 and Count- down arithmetic puzzles show that, replacing CoT-ggnerated data with search-generated data for offline fine-tuning improves success rates by around 23% over inference-ting search base- lines, while reducing inference time by 180x. On top of this, our learning and forgetting ob- jective consistently outperforms both supervised fine-tuning and preference-baie based methods."
1 Introduction
Countdown Test Set
Base LLM
The last few years has seen the rapid development of large language models (LLMs) (Vaswani et al., 2017; Achiam et al., 2023) and their applications to diverse set of lasks. In particular, these LLMs have been tested on challenging benchmarks requiring high-level reasoning across domains such as mathematics (Glazer al., 2024), abstract reasoning (Chollet al., 2024), code generation (Zhuo et al., 2024), and science (Mitchener et al., 2025).
Success Rate (%)
50
SFT
CPO-SimPO
UFT (ours)
40
CoT (greedy)
CoT (pass@1)
30
6 ToT
RAP
20
10
Chain-of-Thoughl (CoT) and inference-time search (search) have been proposed to enhance LLM generalization to reasoning problems at test time. Specifically, given a task description and input x, CoT (Wei et al., 2022) outputs single reasoning path via greedy decoding, or samples multiple paths compute average performance. In contrast, scarch-based methods (Lightman et al., 2023; Yao et al., 2023a; Snell et al., 2024) explicitly generate intermediate reasoning (possibly with multiple passes) before extracting ý via a reward model or self-evaluation. Despite inference- timeence-timeeee search's s effectiveness, it comes with greater computational costs (e.g., Fig. and Chen et al. (2024cbb))
2
2
27
Inference Time (GPU minutes)
Figure 1: Trade-off between inference time and success rate in challenging arithmetic puzzles, Countdown. We find that fine-tuning on CoT-style data from diverse reasoners substantially enhances CoT inference over the base LLM (Qwen2.5-Matte 7B) while preserving inference-time search (ToT (Yao et al., 2023a), RAP (Hao et al., 2023)) using smaller learning rate. Among fine-tuning methods including SFT and preference optimization, achieves the best CoT and search performance.
method (UFT)
* "Wrk primarily done while Tiauwei Ni intern Amazon. Correspondence: Tianwei Ni (tmm120160gmil. com)..ccmm.......mm Allen Nie (anielcs. stanford. edu), Rasool Fakoor (rascol. faw.orggmail..coooooooaakkll Code open-source https://gitte http:/gituub.oo.oooo httt://ccoottmnn/////eoo rew.//nnnn..ssnn-uf......

---

Published in Transactions on Machine Learning Research (10/2025)
A beneficial outcome of utilizing inferenco-time sampling strategies, such as CoT and search- methods, is the generation of rich datasets. These datasets can subsequently be fed back into the model for further fine-tuning, potentially enhancing the base model's overall performance. Prior work has primarily focused on distilling the knowledge from successful reasoning paths, drawing from CoT-gencrated data (Zelikman et al., 2022; Yuan et al., 2023b; Singh el al., 2023), paths from search trees (Feng et al.; 2023; Tian et al., 2024; Zhang et al., 2024a), or full search traces (Gandhi et al., 2024; Lehnert et al., 2024). However, in many complex problems, failed (reasoning) paths far outnumber successful ones, representing an underutilized resource. While potentially valuable, these failed paths are often umpaired, making them difficult to use in preference optimization (Rafailov et al., 2023). Diverse search algorithms further enrich data coverage and reasoning quality but produce heterogeneous structures (e.g., trees) unlike linear CoT traces. This raises a natural question: can we make full use of both successful and failed reasoning palhs, from both CoT and search-based methods, to improve LLM reasoning?
In this paper, we propose a straightforward method consisting of three phases designed to enhance the reasoning capabilities of large language models: data generation, fine-tuning, and evaluation. First, we collect heterogeneous reasoning data from both CoT and the search- based methods, and convert the data into a unified CoT-style path format. Next, we fine-tune the base LLM by learning successful paths with the supervised fine-tuning (SFT) and forgetting unpaired failed paths with the unlikelihood loss. We refer to this objective as unlikelihood fine-tuning (UFT). Finally, we evaluate the fine-tuned model on test sets using both CoT and inference- tiference-timeeeeee search, with a primary focus on efficient CoT inference. We demonstrate the effectiveness of our three-phase pipeline on two arithmetic puzzle domains: Game-of-24 24 and Countdown, which are widely used as testbeds for explicit search (Yao et al., 2023a; Gandhi et al., 2024). We summarize our key findings below, using Qwen2.5-Math 1.5B and 7B (Yang et al., 2024) as base LLMs:
1. Data quality drives performance: On Countdown, replacing CoT data with high-quality classic search data boosts CoT performance from 33.5% to 57.1%, significantly outperforming the inference- time search bascline (25%) with a 180x reduction in inference cost.
2. Unpaired forgetting reliably improves performance:
Incorporating failed paths using UFT improves CoT performance by 1 1-2% on average, with up to 7% gains in the best case. In contrast, preference- based methods like CPO-SimPO (Xu, 2024) often underperform SFT due to the paired format.
3. Controlled learning rate helps maintain search capability: Naive fine-tuning can quickly degrade model's search ability. A substantially reduced learning rate helps prevent this, making it a simple but crucial step when fine-tuning with CoT-style data.
2 Related Work
LLM inference-time search. Inference-time search has shown remarkable success across reasoning (Wang et al., 2022; Stechly et al., 2024; Hao et al., 2024; Snell et al., 2024) and planning tasks (Valmeekam et al., 2023; Zheng et al., 2024; Bohnet et al., 2024). Approaches include best-of-n sampling (Nakano et al., 2021; Wang et al., 2022), multi-agent debates (Du et al., 2023), and iterative correction via predefined rules (Bai et al., 2022) or self-refinement (Shinn et al., 2023; Madaan et al., 2024). A key direction, which this paper focuses on, integrates classic search algorithms with LLMs, employing BFS (Yao et al., 2023a; Xie et al., 2023), DFS (Qin et al., 2023), A* (Zhuang et al., 2023; Meng et al., 2024a; Koh et al., 2024) and MCTS (Hao et al., 2023; Zhou et al., 2023; Zhao et al., 2024; Xie el al., 2024; Gao et al., 2024). While effective, inference-time search methods remain computationally expensive: self-refinement requires multiple expensive passes, multi-agent approaches demand substantial GPU memory, and search trees grow exponentially with depth (Chen et al., 2024c).
LLM policy distillation from synthetic data. Synthetic data, generated by classic algorithms or LLMs, is widely used for SFT in reasoning tasks to mitigate the scarcity of human-annotated datasets (Liu et al., 2024). However, despite its abundance, synthetic responses may fail to solve the task. To address this, most prior work distills reasoning knowledge to policy (Hinton et al., 2015) by selecting only correct responses. Policy distillation strategies can be categorized based on whether the source and target data originate from CoT or search algorithms. CoT-to-CoT distillation learns CoT reasoning from correct CoT paths filtered by rewards, widely used in practice (Zelikman et al., 2022; 2024; Uesato et al., 2022; Yuan et al., 2023b; Gulcchre et al., 2023; Singh et al., 2023). Bearch-to-seardh distillation either refines the proposal policy in tree search by learning
2

---

Published in Transactions on Machine Learning Research (10/2025)
from expert behavior (Feng et al., 2023; Tian et al., 2024; Zhang et al., 2024a) and preference pairs (Chen et al., 2024a), or learns meta-policy by imitating entire search traces (Yang et al., 2022; Gandhi et al., 2024; Lehnert et al., 2024). Notably, Zhang et al. (2024c) performs search-to-CoT distillation using preference pairs from Troe-of-Though without a verifier. In comparison, we focus on {CoT, scarch]-to-CoT distillation from diverse reasoners and distill unpaired positive and negalive data to the policy, labeled by a rule-based verifier.
Offline fine-tuning with negative data. Negative data has been extensively studied in LLM safely, where the goal is to unlearn harmful content using objectives such as Gradient Ascent and unlikelihood train- ing (Welleck et al., 2019; Keskar et al., 2019; Yao et al., 2023b; Zhang et al., 2024b). In preference optimization, negative examples are used to increase the relative likelihood of preferred responses in paired data yx.yt.y commonly used in alignment (Rafailov et al., 2023; Yuan et al., 2023a; Zhao et al., 2023; Hong et al., 2024; Xu et al., 2024; Meng et al., 2024b) and reasoning (Pal al., 2024; Pang al., 2024; Zhang al., 2024c; Setlur et al., 2024; Chen et al., 2024a). However, this paired format discards unpaired positives negatives, reducing data efficiency. KTO (Ftthayaajh et al., 2024) addresses the limitation of relative likelihood (Tuаn & Wang, 2024) by learning from unpaired data, although it requires a reference model. In LLM reasoning, negative data is also used to train a reward model (Cobbe et al., 2021; Uesato et al., 2022; Lightman et al., 2023; Feng et al., 2023; Zhang et al., 2024a; Hosseini et al., 2024) for response re-ranking and search curing inference. However, this does not inherently improve the basc model as a policy. Instead of requiring preference pairs or reference models, we adopt unlikelihood training ( (Welleck et al., 2019) to directly forget failed paths as an auxiliary loss. Finally, concurrent work (Wang et al., 2025) highlights the challenges of unlearning in reasoning tasks, namely forgetting both incorrect reasoning traces and final answers while retaining overall reasoning ability.
3 Preliminaries
Reasoning task as an MDP. We formalize reasoning task as a token-level Markov decision process (MDP) (Sution et al., 1998) (N, R, T). The initial state 80 = x E X' consists of a tokenized sequence representing the task description and input. An action Yt E y is chosen based on the current state 8t, which is the soquence of all previous tokens: s, = (so. V0:t-1). A terminal state ST is reached upon generating a special ond-of-sequence token. The ground-truthe reward function R is a rule-based verifier that checks the correctness of 8T, yielding r = R(sr) E (0. 1), where O means failure and means success. The goal is to find policy ı that generates an action sequence V0:T-1 (denoted as y) which maximizes the reward: ma R(sT)= = maxy y). We refer to the terminal state (x. as a (reasoning) path.
LLM reasoners. We use an LLM policy ne, parameterized by 0, along with a search algorithm / as the reasoner. We denote this LLM reasoner (TTO) and focus on three popular LLM reasoners (Hao et al., 2024): (I) CoT (Wei et al., 2022) that uses greedy search, (2) Theeggf-Thought (ToT) (Yao et al., 2023a) that uses beam search, (3) Reasoning-via- vian-laniig (RAP) (Hao et al., 2023) that uses MCTS (Kocsis & Szepesvári, 2006).
CoT directly outputs one (via greedy decoding) or multiple reasoning paths (under non-zero temperature) without intermediate search process". On the contrary, ToT and RAP are inference-time search methods, first constructing a search tree of reasoning paths before selecting a final path (x.y) as the answer. Their performance is evaluated based on the success rate of the selected reasoning path. Please see Appendix C for more details.
4 Fine-tuning on Unpaired Correct and Failed Paths from Diverse Reasoners
In this section, we describe our approach to solving a reasoning task, as outlined in Fig. 2. Our approach involves three stages: (1) generating reasoning data from diverse reasoners, (2) offline fine-tuning the base LLM from unpaired correct and failed paths, and (3) evaluating it with reasoning algorithms.
2o'''s performance is often measured by the average success rate over generated reasoning paths, also known as the pass@1 metric (Chen al., 2021; Guo al., 2025), as used in this work.
3

---

Published in Transactions on Machine Learning Research (10/2025)
Data gen
CoT
Reasoners
Failed Cs paths
Base LLM
To'
BFS
RAP
DFS
Correct Co T-style paths
Eval
Improved LLM
Learning: NLL loss
Forgetting: Unlikelihood loss
Figure 2: Our method for reasoning tasks. We first generate synthetic reasoning data using multiple LLM reasoners (e.g, CoT, ToT, RAP) and classic algorithms (e.g., BFS, DFS). This data is converted to a unified CoT-style format and labeled as correct or failed by a ground-trutt verifier to form an unpaired dataset. We then fine-tune the base LLM with negative log-likelihood (NLL) loss correct paths and unlikelihood loss on failed paths, which we refer to unlikelihood fine-tuning (UFT). Finally, evaluate the improved LLM with multiple LLM reasoners on a test set.
4.1 -sy-l Data Generation Using LLM and Classic Reasoners
For a given reasoning task, we generate synthetic reasoning data from multiple reasoners. LLM reasoners f(m)) share a common base policy Te but differ in their search algorithms f. which include CoT and inference-time search methods such as ToT and RAP. Optionally, reasoning paths can also be generated using classical search algorithms (classic reasomers) such as DFS and BFS, which rely on an external verifier rather than LLMs.
For search-based methods, we extract all root-to-leaf paths from the search tree rather than only the final selected path to follow the format. of Co T-Tyle paths. This enables (1) unifying heterogeneous search traces into a common format training data, and (2) efficient inference, as CoT inference is fastest. We aggregate CoT-style paths from all reasoners to compose the training dataset D:
D =
R(x,y)).
(1)
where F is the set of considered reasoners, and f(TT)((x) is the set of all reasoning paths produced by reasoner (TT). Note that classic reasoners ff do not depend on Tg. Each path is labeled by a verifier as correct (r = 1) or failed (r = 0), splitting D into two unpaired datasels: the correct dataset D+ = {(x,y) (x,y, 1) E D) and the failed dataset D {xxxx) y) (x. E D).
4.2 Unlikelihood Fine-Tuning on Unpaired Correct and Failed Paths
While prior work lypically fine-tunes LLMs on correct CoT-generated paths (Yuan et al., 2023b; Singh el al., 2023) or on preference pairs correct and failed responses (Pang et al., 2024; Zhang et al., 2024c), we extend these approaches in two directions: (1) incorporating diverse reasoners beyond CoT to augment the training data, (2) fine-tuning the model to avoid and forget failed reasoning patterns using only failed examples. Unlike most preference-basel methods, our approach leverages unpaired positive negative data, without. requiring success-failus pairs for the same question.
Learning to follow correct reasoning paths. First, we consider the negative log-likelihood (NLL) loss, also known supervised fine-tuning (SFT) (Ouyang et al., 2022; Cen et al., 2025), correct reasoning paths collected from multiple reasoners:
min - -x..y.,)) 1) log To x)] -E(x.t) [log To (y+ |x)] (0;
[log
(2)
Examining the gradient of the NLL objective (Eq. 2). -EIx.y,)> 1) log To xx]] reveals its connection to REINFORCE (Williams, 1992) with a binary (indicator) reward function, as noted in prior work (Zelikman et al., 2022; Gulcehre al., 2023; Singh et al., 2023)3.
Thhis resemblance holds when close on-pollcy data (Fakoor et al., 2020), egg., -enerrtddd but may break down as model weights shift during fine-tuning or if includes search-derived data.
4

---

Published in Transactions on Machine Learning Research (10/2025)
Learning to avoid and forget incorrect reasoning paths. While NLL encourages the model to learn correct reasoning paths, it can only leverage the limited number of available successful trajectories and discards a large number of failed ones which often vastly outnumber the correct ones (e.g., |D=1/1D* ranges from 5 to 400 in our experiments). To address the limitation of NLL, we propose leveraging failed trajectories by incorporating the unlikelihood (UL.) loss (Welleck et al., 2019) as an additional objective. The UL loss enables the model to learn to avoid (i.c., forget) failed reasoning paths, effectively utilizing the large set of failed trajectories.
min Jon,CLLODD)) -E m jND [log(1- - To(y x))].
This objective explicitly reduces the probability of incorrect paths under the model. Unlike Gradient Ascent (GA) unlearning (Yao et al., 2023b) 4, UL objective has a smooth optimization landscape, with gradient:
Ju) (xy------- Ty
xx)
x) Vlog (o (x)
To
The gradient implies convergence to the stationary point where Toly x) = 0, E D= From a REINFORCE view, Eq. 4 imposes adaptive penalty of moly [x] more strongly and thus efficiently suppressing wrong paths.
1 1x). penalizing higher-probability failures
Combining learning and forgetting: unlikelihood fine-tuning (UFT). Our final objective UFT combines NLL (Eq. 2) and UL. losses (Fq. 3) with a coefficient E (0,
min (0; = (1 (0) (0; + (0;
55)
9
Since correct paths provide explicit solutions while wrong paths only rule out alternatives, we treat UL as an auriliary loss by setting a close to O. Moreover, as correct and failed paths may be drawn from the same search trees, they often share prefixes, introducing conflicts when a is large, as also noted in preference-based methods (Zhang et al., 2024c). For example, suppose a prefix sequence (x, Wo:r) (t < T - appears in both D+ and D with equal frequency. The gradicnt of Eq. 5 w.r.t. this prefix is (1- a ( - - V Vlg To x), which has stationary point at (o x) = 1 a. Thus, 1 1- sets the dcsired probability of the shared prefix, further motivating the need for small a to avoid conflicting objectives. Unlike prior work (Zhang et al., 2024c; Setlur et al., 2024; Chen et al., 2024a) that identify high-credit steps with more computation, we sidestep this with an auxiliary loss without preference pairs.
4.3 Practical Algorithm
We summarize our method in Algo. 1. For a given reasoning task, we partition the initial state set X' into training, validation and test sets, namely, Xrain Xvalid, Mest- Each subset has distinct input cases with follows the same instruction template. Our approach assumes access to ground-truth verifier in Xraain (for generating training data) and Xealia (for checkpoint selection), but not in Neatt on which the final evaluation is conducted.
Algorithm 1 Fine-tuning on unpaired correct and failed paths from diverse reasoners
Require: Reasoning task (x R, T), base LLM Ta, set of reasoners F, number of epochs E, batch size B, learning rate v,, UL coefficient a (close to 0)
1: Generate CoTT-style training data from various reasoners: D = ((x,y.r) |/ F.x E Xrmin, y E ((Ta)(x))
y), and split D into correct data D+ and failed data D=
2: Fine-tume the LLM with UFT (Eq. 5) for E|D+1/R steps. In each step, sample batches B+ ~ D+ and B D
of size B each, and update:
0 - 0 - n111 a) V
(6)
3: Evaluate the fine-tuned LLM we test cases by collecting D' {(x,y,)) 1f E F,x Xest, f(x))(x), y)x
4The GA objective ming (xxyy)~> J-D [logmo(y") |x] is shown to be unstable to optimize (Zhang et al., 2024b;d), as it has an unbounded optinum (-oo) and divergent gradient near the optimum. We provide results for GA Sec. B.3 for completeness.
5

---

Published in Transactions on Machine Learning Research (10/2025)
Table 1: Performance on Countdown and Game-of-24 using Qwen2.5-Math 7B the base model fine-tuned on Countdown. Each cell shows (success rate / inferrnce time) averaged with 8 seeds, where the time is measured in minutes (m) on an A100 GPU. We compare the base LLM with three fine-tuning methods (SFT (Ouyang et al., 2022), CPO-SimPO (Xu, 2024), UFT (ours)) under two learning regimes (varying learning rate and training data). SimPO (Meng et al., 2024b) is omitted due near-zero success rate. Bolded numbers indicate the best success rate per row. Fig. 1 visualizes the stronger result per fine-tuning method across learning regimes.
Test set & Inference
1r=5e-6, CoT+BBS+ training data
Ir=1e-6, Co'I' training data
Base LLM
SET (a=0)
CPO-SimPO UFT (a=le-3)
SFT (a=0)
CPO-SimPO UFT (a=1e=4)
Countdown
(1000 cases)
CoT (greedy)
6.2% / 1.0m
55.4%
0.6m
50.6%
0.7m
57.4%
0.5m
24.7%
0.5m
23.4%
/ 008m
24.7%
/ 0..5m
CoT (passtit)
5.6% / 3.4m
51.8%
3.5m
45.7%
4.7m
53.5%
3.4m
24.1%
2.9m
22.1%
5..7m
24.7%
3.3m
search: ToT
25.0%
/ 90m
2.4%
104m
4.1%
61m
0.0%
1.9m
22.2%
/
96m
23.3%
179m
28.6%
153m
search: RAP
10.2% 278m
7.3%
155m
7.0%
/ 137m
0.0%
/ 341m
17.5%
141m
17.7%
190m
28.9%
439m
Came-of-24 22 (100 cascs)
CoT (groedy)
6.0% / 0.1m
35.0%
0.1m
24.7%
/ 0.1m
42.0%
0.1m
8.0% / 0.1m
6.7%
0.1m
8.3%
0.1m
Coll (pass@1)
6.0% / 0.6m
28.3%
0.7m
23.0%
1.3m
29.7%
0.8m
7.3% / 0.6m
7.0%
1.5m
6.7%
0.8m
search: ToT
28.0%
6.5m
2.3%
8.2m
1.0%
4.3m
0.3%
/ 0.4m
30.7% / 8m
22.7%
13m
24.0%
/ 13m
search: RAP
27.0%
8.3m
1.3%
/ 13m
3.0%
19m
3.7%
29m
27.7% / 16m
25.7%
42m
33.7%
/ 75m
5 Experiments
We evaluate our approach on two arithmetic puzzle tasks.
Game-of-24. In of the goal is to use basic arithmetic operations (+-+/) and parentheses to combine four input numbers to obtain the number 24. Each input number can only be used once, which requires exactly three high-level reasoning steps. We follow the step-by-step response format from ToT (Yao et al., 2023a). For example, given numbers 2 9 10 12, a correct reasoning path is 12 2 24 (left: 10 24), 10 9 1 (left: 24), 24 1 24 (left: 24), Answer: (12 2) (10 9) 24. Each high-level step must include the remaining numbers after the operation. We implement a process- based verifier (Uesato et al., 2022) that checks both the intermediate steps and the final answer. For evaluation, we use the same test set from Yao et al. (2023a), consisting of 100 cases. For training and validation, we use 900 distinct cases, considered easier than the lest set based on the human performance.
Countdown. The Countdown game (Gandhi et al., 2024) extends Game-of-24 by requiring the use of four integers to obtain a specified target number beyond 24. All other rules remain the same including the step-by-step format. Following Gandhi et al. (2024), randomly generate 500k training cases, excluding 24 target number. For evaluation, we randomly generate 1,000 test cases with distinct target numbers from training set. This setup allows us to test generalization to Game-of-24 when trained on Countdown data.
5.1 Setup
Base LLMs. We use Qwen2.5-Math 1.5B (Q1.5) and Qwen2.5-Math 7B (Q7B) (Yang ct al., 2024) base LLMs Qwen2.5-Math series were state-of-the-ar open-weight mathematioul LLMs as of December 2024.
Step 1: Data generation setup. We mainly build upon the LLM-reasoners repository (Hao et al., 2024), which provides a unified library for all the LLM reasoners studied here. We significantly accelerate inference by batching inputs using vLLM (Kwon et al., 2023). All reported inference time in our results is based on our optimized codebase. For each LLM reasoner algorithm, we vary its hyperparameters to gather data from Xrrain For CoT (Wei et al., 2022), we vary the temperature between (0.5, 1.0) and top-p between (0.7, 0.8, 0.9) to have 9 variants. For each variant, we use fixed 5-shot prompt template and collect 100 paths for each case in Game-of-24 and 3 paths for Countdown. For Game-of-24, 24, we follow LLM-reasoners to implement ToT (Yao et al., 2023a) (varying the beam size from 5 to 16) and RAP (Hao et al., 2023) (varying the exploration parameter from 1.0 to 10.0). Since Countdown is implemented in Stream-of-Search (SoS) (Gandhi et al., 2024), we use SoS code to implement classic BFS and DFS algorithuss, which do not use LLMs but rely on the oracle verifier. We merge the reasoning paths generated by all the variants of the same reasoner, and then remove duplicates by exact matching, to serve as our training dataset. See Appendix C for all details.
Snttps://en http:/:///eenvviieeii...c (pxrs///22__puzeee) Onttps://github. wwwp//sitt/oooorreeeeoeccoe http:///hthuuu..commmttrir-eeeee
6

---

Published in Transactions on Machine Learning Research (10/2025)
Step 2: Fine-tuning setup and baselines. We follow the alignment handbook (Tunstall et al., 2024) to implement all fine-tuning methods on an instance with 8 A100 GPUs. All methods share the same datasets and use common hyperparameters when applicable. We fix the batch size B to 128 and train E' = 10 epochs on each dataset for Game-of-24 and E = 2 epochs for Countdown. We use a cosine schedule and sweep the peak learning rate y) over (Ie-5, 5e-6, 2e-6) for Q1.5B and over (5e-6, 26-6, 1e-6) for Q7B. We set a = 0 to have the SFT baseline and sweep a over (Ie-3, 1o-4, 1o-5, 1c-6) for our UFT. We also implement two preference-based baselines: SimPO (Meng et al., 2024b), which optimizes over paired preference data constructed from successful and failed reasoning paths, and CPO-SimPO (Xu, 2024),, which augments SimPO with NLI term to better support reasoning tasks. Both methods are reference-free and leverage negative data as UFT. Notably, CPO-SimPO closely resembles the non-iterative variant of RPO (Pang et al., 2024). See Appendix A for details and Table 4 for method comparison.
Step 3: Evaluation on CoT and scarch-based reasoning. Since our training dataset are all CoT--tyycc reasoning paths, fine-tuning on these data should directly improve CoT reasoning capability. This is measured by performing zero-shot CoT (via greedy decoding or sampling) on Restt with fine-tuned LLM. On the other hand, we also aim to retain search-based reasoning capability after fine-tuning. This mcasured by performing ToT or RAP on Xeest-
Inference-time baselines on CoT and search-based reasoning. Inference-time baselines refer to reasoners that use base LLMs without fine-tuning. For CoT reasoning, we use the few-shot CoT (via greedy-decoding or sampling) on the base LLMss For search-based reasoning, we report the best variants of ToT and RAP with the base LLM. The results for these baselines are shown in Table for Q7B and Table 6 for QI.5B. We find that search-based reasoning (especially ToT) significantly improves success rates over CoT reasoning for Q7B (6.2% + 25% in Countdown, 6% + 28% in Game-of-24),, but not for Q1.5 (2.2% + 2.2% in Countdown, 2% to 8% in Game-of-24). However, search-based reasoning substantially increases inference time compared to CoT; for example, in Countdown, ToT and RAP are 90x and 278x slower than CoT (greedy decoding).
5.2 Boosting Chain-of- Chain-of-Thouuht Reasoning
We present CoT reasoning results al inference time for LLMs fine-tuned on the Countdown (Table 2) and the Game-of-24 datasets (Table 3). Similar to Ye et al. (2024), we also evaluate the models trained on Countdown on the Game-of-24 test set, because conceptually Game-of-24 is a subset of Countdown. For both tables, we use the highest learning rate (1e-5 for Q1.5B and 5e-6 for Q7B) that yields best performance in CoT reasoning, leaving further discussion on the learning rate for Sec. 5.3. For the UFT objective, we fix the coefficient a for the same model across different datascts, selected by the average success rate on validation set. Xvalid- Our key findings are summarized below.
NLL loss is essential for offline fine-tuning on reasoning tasks. The most salient obscrvation is that SimPO, while successful in alignment lasks using only a preference loss (Meng el al., 2024b), yields near-zero success across model sizes and datasets. In contrast, CPO-SimPO (Xu, 2024), which simply adds an NLL term to SimPO, achieves substantially better results. This highlights the critical role of NLL loss, shared by SFT, CPO-SimPO, and UFT, which explicitly encourages models to follow correct reasoning paths. These findings align with prior results in Pang et al. (2024); Pal et al. (2024).
Incorporating high-quality training data is decisive for CoT reasoning. Second, among methods that use NLL loss, performance is primarily determined by the quality of the training data, outweighing the effect of forgetting objective. We mcasure data quality by the best-of-n success rate: the fraction of unique solved cases out of all the training cases.7 The best performances in Table 2 and Table 3 correspond to the datasets with the highest quality (BFS+DFS data in Countdown, CoT data in Game-of-24). In addition, when the two datasets have similar quality (BFS+DES vs CoT+BFS +DFS in Countdown, CoT vs CoT+ToT+RAP in Game-of-24), their performances are also alike. Remarkably, fine-tuning on the high-quality classic search (BFS+DFS) data from Countdown yields significantly better performance on
7 Although ToT and RAP greatly outperform CoT as inferenco-time methods (Table 1, Table 6) Came-of-24, they produce lower-quality training data than CoT (Table 3), because CoT's best-of-n (n=100) success rate far exceeds average (93% versus 6% in Q7B, 82% versus 5% in QI.5B).
7

---

Published in Transactions on Machine Learning Research (10/2025)
Table 2: CoT reasoning performance of LLMss fine-tuned on Countdown. Each row shows the averaged result (3 seeds) an LLM fine-tuned on data from specified sources. Models are evaluated on Countdown (1000 cases) and Game-of-24 (100 cases) test sets using zero-shot CoT via greedy decoding (greedy) sampling (pass@1) 1). BFS+DFS data are generated by oracle reasoners without LLMs. We highlight our two contributions (incorporating search-derived data and the fine-tuning method UFT) and the best cell for each (test set, CoT inference) pair. Bold indicates the highest mean for each row. A superscript marks results that are statistically significant according to Welch's t-test (p 0.05).
(a) Base LLM: Qwen2.5 M-th 7B (best rate with search: 25% Counldown, 28% Game-of-24)
SET
SimPO
CPO-SimPO
UFT
CoT data (quality: 37.6%)
Countdown succ (greedy)
32.5%
0.0% 0.0%
27 07..%%.....59
33.5% 1.0%
Countdown succ [pass(11
32.2% 9..6%
0.0% 0.0%
25.9%
32.5% 1.7%%
Game-of-24 suce (greedy)
16.0%:11.00
0.0% 0%
11.7%23.
16.7%%
Game-of-24 succ (pass&1)
12.3%:12.3%
0.0%21.000 0%
10.3%;24.0%
14.7%
BFS+DFS data (quality: 86.2%)
Countdown succ (greedy)
56.1%.10.
0.0% 0%
51.4%
57.1%
1:0.3%
Countdown succ (passf)1
53. 53,1%.15 16..1.2.00
0.0% 0.0%%:0 0%
188.6%..1... 13%
64.4%
Game-of-24 suce (greedy)
37. 97.0%%-41.00 1..0%
0.0% 0.0%%=i..om 0.0%
31.7%
39.7% +..5%
Game-of-24 suce (pass31)
28.3% 440.%
0.0%
28.3 4.0%
24.7%++5..7%
COT+BFS+DFS data (quality: 86.9%)
Countdown suce (greedy)
55.4%+11.9%
0.0%==.0%%
50.6%
57.4%+00.4%
Countdown suce (pass@11) Game-of-24 sucr (greedy)
51.8%++..4%
0.0%=0.07%
45.7%=#..40
53.5%++0.5%
35.0%=11.09
0.0%==..0%
24.7%#3.111 1%
42.0% 11.7%
Game-of-24 succ (pass31)
28.3%%446.7
0.0%
23.0%
29.7% d: 4.0%
(b) Base LLM: Qwen2.5-Math 1.5B (best succ rate with search: 2.2% on Countdown, 8% on Game-of-24)
SET
SimPO
CPO-SimPO
UFT
Co'T' data (quality: 20.0%)
Countdown suce (greedy)
+0.3%
0.6%
21.6%
26.1% 1.06%
Countdown suce (pass@1)
24.4%
0.6% 5%
21.3%
26.1% 20.4%
Game-of-24 suce (greedy)
03.4%
0.0% 0%
8.7% 1.5%
14.3%:10.%%
Game-of-24 succ (pass81)
9.3%
0.0%=20.0% 0.0%
7.3%:12.19 1%
14.0% 12.0%
BFS+DFS data (quality: 86.2%)
Countdown suce (groody)
50.8%.10.79
0.0%
50.5% 1.4% 1.4%
53.7%% 1.2%
Countdown suce (pass011
4..5%
0.0%
47.4% 1. 6.6%
51.2% +.11%
Game-of-24 suce (greedy)
23.2%
0.0%++0.08
27.3% +..8%
32.0% 4.66%
Game-of-24 suce 1p
23.0%47.0%
0.0%
23.0%++..2%%
CFT+BFS+DFS data (quality: 86.3%)
Countdown suce (greedy)
51.1%++..9%
0.0%=0.00%
48.9%+0.8%
51.5%+0.7%
Countdown suce (passQ1) Game-of-24 suce (greedy)
47.1 11.1%
0.0%=+..0%
45.3%+ 1.2%
46.5%.11.55
26.3%
0.0%
24.7%+2.00
26. 26.0%
Game-of-24 suce (passB1)
24.0%
0.0%
20.7% +2.1%
25.0% +..3%
Game-of-24 than models fine-tuned directly Game-of-24 CoT data (37% vs 28.2% for Q7B and 29.7% 14.5% for Q1.5B). This highlights the importance of incorporating high-quality training data, rather than relying solely on CoT data, as is common practice.
Unpaired learning and forgetting is more robust than paired one. Lastly, we analyze the effect of forgetting loss to SFT. UFT achieves the highest mean (ignoring variance) in 26 out of 36 evaluation settings, and is statistically significantly better than baselines in 7 cases. Given the limited number of seeds (3-4), these results may underestimate the true significance, and we expect the evidence strengthen with additional runs. In terms of magnitude, UFT yields an average improvement of 1.3% (with maximum of 4%) in Table 3, and an average of 1.5% (with maximum of 7%) in Table
In contrast, PP shows mixed results. While it improves over SFT with average gain of 1.2% and maximum of 5% in Table 3, it underperforms SFT in most scenarios in Table 2, with an average drop of 3.5%. We believe several factors contribute lo the contrasting behavior of CPO-SimPO relative to SFT (and UFT) when fine-tuned on Countdown versus Game-of-24. First, as shown in Appendix Table 5, the paired preference requirement in CPO-SimPO results in the exclusion of 0.1% to 5.8% of correct data on Countdown, while no such filtering occurs for Game-of-24. Since correct data determines data quality, this reduction likely contributes to CPO-SimPO's weaker performance on Countdown. Second, the exclusion
8

---

Published in Transactions on Machine Learning Research (10/2025)
Table 3: CoT reasoning performance of LLMs fine-tuned and evaluated on 2f.. Each row shows the averaged result (4 seeds) of an LLM fine-tuned on data from specified sources. We highlight our two contributions (incorporating search-derived data and the fine-tuning method UFT) and the best cell for each (test set, CoT inference) pair. Bold indicates the highest mean for each row. superscript marks results that are statistically significant according to Welch's t-test (p 0.05).
(a) Base LLM: Qwen2.5 Swen2..5MMaie M-Mth 7B (best succ rate with search: 28%)
SPT
SimPO
CPO-SimPO
UFT
CoT data (quality: 92.9%)
succ (greedy)
28 28.2%=1..7
2.2%
28.5%22.99
28. 2.66%%
succ (pass@1)
22.5%
2.2%
27.5% 4.2.4%
23.2%%+44.4%
ToT+RAP data (quality; 69.3%)
succ (greedy)
13. 13.2%=23.68
0.0%220.0%
16.5%.12.999
15.2%612.2%
succ 1)
13.2%:22.88%
0.0%
15.0%614418
17.2%11000
CoT+TTTT+ RAP data (quality: 95.3%)
succ (greedy)
27.5%
4..8%
26.8% 4..7%
30.2% +.11%
succ (pess@1)
24.2%
0.2%%
22.8% +1.7%
26.2% 22.2%++2.. +2.2%
(b) Base LLM: Qwen2.5 5-Math L.5B (best suce rate with search: 8%)
SFT
SimPO
CPO-SImPO
UFT
CoT data (quality: 82.1%)
succ (greedy)
14.5%f13.8%
0.0% 0.0%1100%
18.5%=11
18. 28.2%
succ (pass&1)
17.8%613.4%
17.3%22.55%
18.5%13.5%
ToT+RAI data (quality: 44.7%)
succ (greedy)
9.5%+t0.6%
0.0%
10.3%=1.95%
1.0%
succ (pass@1)
10.0%11.4%
0.0% 0.00%
1.3%%
9.5%.12.66
CoT+TOT+H RAP data (quality: 87.6%)
suoc (greedyy
20.0%f12.7%
0.0%
21.8%=1.7%
20.5%
suce (pass@1)
17.0%:12.2%% 2.%%
0.00%10.0%
18.0%+12.6%
18.8%
of unpaired failed data, along with the challenge of balancing preference and NLL objectives, may have a greater impact on the more difficult Countdown task.
5.3 Mitigating Forgetting in Search-based Reasoning
We investigate whether fine-tuned LLMs can retain search-based reasoning abilities (e.g., ToT, RAP), question that remains undcrexplored in the absence of continued pretraining. Evaluating this retention is analogous to measuring backward transfer (Lopez-Paz & Ranzalo, 2017) in continual learning, i.e., whether performance on a prior task (search- based reasoning) degrades after data). We focus on Q7B, which exhibits strong search-based reasoning performance and a clear gap over CoT baselines (Table 1). For completeness, the results of Q1.5B is shown in Appendix B.
on a new one (CoT-style
Although learning rate decay is employed following the alignment handbook, find that using a peak learning rate of 2e-5, as in the original Qwen2.5-Math setup (Yang et al., 2024), leads to catastrophic forgetting (McCloskey & Cohen, 1989), with near-zero performance in inference-time search. We hypothesize that this degradation stems from the lack of reward modeling in CoT-style supervision, which would be essential for search-based reasoning (see Sec. C.4 for details).
A very low (peak) learning rate is crucial to retaining search capability. Inspired by continual learning (Mirzadeh et al., 2020), we adopt a much smaller learning rate to keep parameters within the "basic capability basin" " (Chen et al., 2025), preserving pretrained knowledge. Fig. 3 and Fig. 5 shows the effects of (peak) learning rate in standard SFT on Q7B. As we reduce the learning rate from 5e-6 to 1e-6 (an exceptionally low value for SFT), we observe a clear trade-off: CoT reasoning performance declines rapidly (although it remains above the baseline), while inference- time search improves significantly (although often remains below the baseline). This behavior is similar to learning rate effects seen in reference-free preference optimization (Meng et al., 2024b).
CoT training data is more effective for preserving search capability. An interesting obscrvation from Fig. 3 to Fig. 5 is that in both Countdown and Game-of-24 24 training settings, CoT data leads to better inference-time search than search-derived data, although CoT data has much lower quality compared to
9

---

Published in Transactions on Machine Learning Research (10/2025)
DFS+DFS
CoT data
CFT+BFS+DDS data
Qwen2.5-Math-78
(a) Trained Countdown and Tested Countdown (pass@1)
(greedy)
Success Rate
0.2
0.15
0.4
0.1
0.2
(b) Trained Countdown and Tasted Game-of-24
Success Rate
0.3
0.3
0.3
0.2
0.2
0.2
0.1
0.1
0.1
0.0
0.0
n
Peak 1e-6
Peak 1e-6
Impact of (peak) learning rate and data sources on CoT vs. search-based reasoning for standard SFT (a=0) using Qwen2.5- 7B the base model. Learning rate mediates trade-off between Coll and search performance; CoT data better preserves search capability in most cases.
Figure 3:
BFS+DFS data. Both CoT and search-derived data fall under CoT-style paths, but the latter generated by search algorithm rather than directly by the LLM. This suggests that CoT data, being more aligned with the LLM's on-policy distribution, induces less distribution shift during SFT, thereby reducing forgetting.
BFS+DFS
CoT CoT+BFS
Qwen2.5-Math-78
(a) Trained Countdown and Tested on Countdown (pass@1)
(greedy)
Search:
0.5
0.30
success Rate
0.4
0.25
0.2
0.3
0.2
0.1
0.15
0.1
0.10
(b) Trained Countdown and Tested on Game-of-24
0.3
Success Rate
0.25
0.3
0.2
0.20
0.2
0.2
0.1
0.1
0.1
0.10
0.05
0.0
10-4 10-3 10-4 10-3
10-6 10-3 10-4 10
10-4 10-3 10-4 Unlikelihood
10-3
10-4 10-3 10-4 10-3
Unlikellhood
Unlikelihood
Unlikelihood
Figure 4: Effects of unlikelihood loss and data sources CoT search-based reasoning using Qwen2.5- Math 7B as the base model, with the peak learning rate as 1e-6. Unlikelihood loss leads to greater improvements in search performance than in CoT inference in most cases.
Unlikelihood auxiliary loss can significantly improve search capability. Building on the findings in Fig. 3, we observe that the effect of a low learning rate extends to UFT and CPO-SimPO as well. Here, we examine the impact the unlikelihood loss specifically the lowest learning rate of 1c-6. Consistent with Sec. 5.2, UFT provides marginal improvements over SFT in CoT reasoning, regardless of the choice of a. However, with a=1e-4, inference-time timennee-iieeeeee search improves considerably compared to SFT (a=0) in most cases. For example, with the same CoT data, ToT achieves 28.6% vs. 22.2% under SFT, and RAP reaches 28.9% vs. 17.5% far exceeding the RAP baseline of 10.2%. This indicates that learning to forget incorrect CoT-style paths could generalize to forgetting incorrect intermediate search steps.
6 Conclusion and Future Work
We propose a simple and effective method for improving reasoning on arithmetic puzzle domains, addressing three underexplored challenges: (1) augmenting training data by unifying mixed data formats from diverse
10

---

Published in Transactions on Machine Learning Research (10/2025)
reasoners into CoT-style format, (2) leveraging unpaired correct and failed paths via unlikelihood fine-tuning, and (3) mitigating loss of search capability using small learning rate. Experiments on Countdown and Game-of-24 show that data quality, unlikelihood loss, and learning rate are key to balancing CoT efficiency and search capability. Future work can explore more advanced unlcarning objectives (Tamirisa et al., 2024; Huang et al., 2024), incorporate stabilization techniques (Meng et al., 2024b), and extend the framework to other reasoning-oriented L.Mss.
References
Josh Achiam, Steven Adler, Sandhiui Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv::303. 088774, 2023. 1
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. Xiv preprint arXio:221 0000073, 2022. 2
Bernd Bohnet, Azade Nova, Aaron T Parisi, Kevin Swersky, Katayoon Goshvadi, Hanjum Dai, Dale Schuurmans, Noah Fiedel, and Hanie Sedghi. Exploring and benchmarking the planning capabilities of large language models, 2024. 2
Zhepeng Cen, Yao Liu, Siliang Zeng, Pratik Chaudhari, Huzefa Rangwala, George Karypis, and Rasool Fakoor. Bridging the training-inference gap in LLMs by leveraging sell-generated tokens. Trensactions on Machine Learning Research, 2025. ISSN 2835-8856.
Huanran Chen, Yinpeng Dong, Zeming Wei, Yao Huang, Yichi Zhang, Hang Su, and Jun Zhu. Understanding pre-training and fine-tuning from loss landscape perspeclives. Xiv preprint arXio:2505.1760 17646, 2025. 9
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint ar orXix:2107.055 07.0337,, 2021. 3
Sijia Chen, Yibo Wang, Yi-Feng Wi, Qingguo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Lijun Zhang. Advancing tool-augmented large language models: Integrating insights from errors in inference trees. Advonces in Neural Information Processing Systems, 57:106555-106581 2024a. 3, 5
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of ol-like llms. ar preprint arXiv:2412.21187, 2024b. 1
Ziru Chen, Michael White, Raymond Mooney, Ali Payani, Yu Su, and Huan Sun. When is tree search useful for Ilm planning? it depends on the discriminator. Xix preprint arXi:2202. 10890, 2024c. 1,2
Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. Are prize 2024: Technical report. arXio preprint arXiv:24 12. 046044 2024. 1
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.m 14168, 2021. 3
Yilun Du, Shnang Lı, Antonio Torralba, Joshua B Tenenbaum, and Tgor Mordatch. Improving factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.1 14325, 2023. 2
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment prospect theoretic optimization. arXiv preprint arxiv:2402. 01306, 2024. 3, 17
Rasool Fakoor, Pratik Chaudhari, and Alexander J. Smola. P3o: Policy-on policy-off policy optimization. In Proceedings of The 35th Uncertainty Artificial Intelligence Conference, volume 115 of Proceedings of Machine Learning Research, pp. 1017-1027. PMLR, 22-25 Jul 2020. 4
Xidong Feng, Ziyu Wan, Muning Wen, Stephen Mareus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero- like tree-search can guide large language model decoding and training. arXiv preprint arXiv:2309. 17179, 2023. 2,
11

---

Published in Transactions on Machine Learning Research (10/2025)
Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah D Goodman. Stream of search (sus): Learning to search in language. arXie preprint arXie:2404.0080 2024. 2, 3, 6, 23, 26
Zitian Gao, Boye NIu, Xuzheng He, Haotian Xu, Hongzhang Liu, Aiwei Liu, Xuming Hu, and Lijie Wen. Interpretable contrastive monte carlo tree search reasoning. arXiv preprint or orXiv:24 0a0:2410.000 01707 2024. 2
Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, et al. Frontiermath: benchmark for evaluating advanced mathematical reasoning in ai. arXiv preprinl or urXiv:2411.0487 2024. 1
Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis, Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reasoning models. arXiv preprint arXi::2006.00 04178, 2025. 27
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, et al. Reinforced self-training (rest) for language modeling. arXiv preprint ar arXiv:2308.0800
2023. 2,
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llus via reinforcement learning. Xix preprint ar Xio: 12948, 2025. 3, 27
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu, Reasoning with language model is planning with world model. arXiv preprint arxiv:2305 14992, 2023. 1, 2, 3, 6, 25, 26
Shibo Hao, Yi Gu, Haotian Luo, Tianyang Liu, Xiyan Shao, Xinyuan Wang, Shuhua Xie, Haodi Ma, Adithya Samavedhi, Qiyue Gao, et al. Lim reasoners: New evaluation, library, and analysis of step-by-step reasoning with large language models. Xie preprint or orXiv:2404.0520 2024. 2, 3, 6, 25
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math datasel. NeurIPS, 2021. 27
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arxic preprint orXiv:1508.0253 02531, 2015.
Jiwoo Hong, Noah Lee, and James Thorne Orpo: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.0769, 2024. 3
Arian Hosseini, Xingdl Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners. arXio preprint arXin:2402.06457 2024. 3
Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Booster: Tackling harmful fine ane-tuning for large language models via attenuating harmful perturbation. arXiv preprint arXiv:2409.015: 0150086 2024. 11
AQ Jiang, A Sahlayrolles, A Mensch, C Bamford, DS Chaplot, D) de las Casas, F Bressand, G Lengyel, Lample, L Saulnier, et al. Mistral 7h (2023). arXiv preprint orXiv:2310.000 0.0825, 2023. 26
Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. Ctrl: A conditional transformer language model for controllable generation. arXiv preprint Xxiv:1909.05854 2019. 3
Levente Kocsis and Csaba Szepesvári. Bandit based monte-carlo planning. In European conference on machine learning, pp. 282-293. Springer, 2006. 3, 26
Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakluldinov. Tree search for language model agents.
ar preprint 0rXiv22407.01477 2024. 2
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operaling Systems Principles, pp. 611-626, 2023. 6, 25
Lucas Lehnert, Sainbayar Sukhbaatar, DiJia Su, Qinqing Zheng, Paul Mevay, Michael Rabbat, and Yuandong Tian. Beyond a#: Better planning with transformers via search dynamics bootstrapping. arXio preprint arXio:2402.14083 2024. 2,
12

---

Published in Transactions on Machine Learning Research (10/2025)
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, el al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):9 2024. 27
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, llya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2355.2008 200500 2023. 1, 3
Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, et al. Best practices and lessons learned on synthetic data. arXiv preprint arXiv:2404.0155 2021. 2
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neurol Information Processing Systems, 36, 2024. 2
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist. networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989. 9
Silin Meng, Yiwei Wang, Cheng-Fu Yang, Nanyun Peng, and Kai-Wei Chang, Lim-a*: Large language model enhanced incremental heuristic search on path planning. arXiv preprint arXiv:2407.02511 2024a.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference- free reward.
Advances in Newral Information Processing Systems, 37:124198 124235, 202/b. 3, 6, 7, 9, 11, 17, 18
Seyed Iman Mirzadeh, Mehrdad Farajtahar, Razvan Pascanu, and Hassan Ghasemzadeh. Understanding the role of training regimes in continual learning. Advances in Neural Information Processing Systems, 33:7308- 2020. 9
Ludovico Mitchener, Jon M Laurent, Benjamin Temmann, Siddharth Narayanan, Geemi P Wellawatte, Andrew White, Lorenzo Sani, and Samuel G Rodriques. Bixbench: a comprehensive benchmark for Ilm-based agents in computational biology. arXiv preprint ar arXi:2508.00096 Feb 2025. doi: 2..86500/7Xxi.2258300000 URL
http:/abb/220000000000 1
Reichhiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint ar arxiv:2 12. 09332, 2021. 2
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow
instructions with human feedback, 2022. 4, 6
Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with dpo-positive. arXiv preprint 240: 13228, 2024. 3, 7, 18
Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. Advances in Nearal Information Processing Systems, 37:116617-11637, 2024. 3, 4, 7, 17, 18
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint 1177:23807.16789 2023. 2
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 14453728-537411 2023. 2, 3, 17
Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszeze, Robie Gonzales, Subhabrata Majumdar, Hassan Sajjad, Frank Rudzicz, et al. Representation noising: A defence mechanism against harmful finetuning. Advances in Newral Information Processing Systems, 1212636-126700 2021. 20, 21
Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. Rl on incorrect syuthetic data scales the efficiency of ul math reasoning by eight-fold. Advances in Neural Information Processing Systems,
2024. 3, 5
13

---

Published in Transactions on Machine Learning Research (10/2025)
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634 0686634-6622, 2023. 2
Avi Singh, John D Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Xavier Garcia, Peter J Liu, James Harrison, Jachoon Lee, Kelvin Xu, et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint or Xrx:2:2312.00 16588888, 2023. 2,
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408. 03314, 2024. 1, 2
Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. Chain of thoughtlessnes:: An analysis of cot in planning, arXio preprint 3rXiw:24050047 2024. 2
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998. 3
Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang, Ron Arel, al. Tamper-rest resistaatt safeguards for open weight llms. arXiv preprint arXiv:2408.0000 00761, 2024. 11
Ye Tian, Baolin Peng, Linfeng Song, Lifeng Jin, Dian Yu, Lei Han, Haitao Mi, and Dong Yu. Toward self-improvement of Ilms via imagination, searching, and criticizing. Advances in Neural Information Processing Systems, 37: 52723-52748, 2024.
Yi-Lin Than and William Yang Wang. A gradient analysis framework for rewarding good and penalizing bad examples in language models. erViv preprint arXiv:2408.107 16751, 2024. 3
Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Shengyi Huang, Kashif Rasul, Alvaro Bartolome, Alexander M. Rush, and Thomas Wolf. The Alignment Handbook, 2024. URL https://github: com/huggingface/ alignment-handeee handbook. 7, 17
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome- based feedback. Xr preprint arXiv:2211.14997 2022. 2, 3, 6
Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models-a critical investigation. Advances in Neural Information. Processing Systems, 36:75993 76005, 2023. 2
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakoh Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in newral information processing systems, 30, 2017.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning, https: /ttps://github com/ tuggingface/trl, 2020. 17
Changsheng Wang, Chongyu Fan, Yihua Zhang, Jinghan Jia, Dennis Wei, Parikshit Ram, Nathalie Baracaldo, and Sijia Tiu. Reasoning model unlearning: Forgetting traces, not just answers, while preserving reasoning skills. arXip preprint arXiv:2506.1200 2025. 3
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXie preprint orXiv:2203.11 111711 2022. 2
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Adnances in neural information processing systems, 35:24824-2455 248377 2022. 1, 3, 6, 23
Sean Welleck, Ili Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training. arXiv preprint arXiv:1908. 04319, 2019.
Ronald J Williams. Simple statistical sraddeet-followingg algorithms for connectionist reinforeement learning. Machine learning, 8:229 256 1992.
14

---

Published in Transactions on Machine Learning Research (10/2025)
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, James Xu Zhao, Min- Yen Kan, Junxian He, and Michael Xie. Self-evaluation guided beam search for reasoning. Advances in Neural Information Processing Systems, 36:41618 111-1,18-41600, 2023. 2
Yuxi Xie, Anirudh Goyal, Wenyue Zheng, Min-Yen Kan, Timothy P Lillicrap, Kenji Kawaguchi, and Michael Shieh. Monte carlo free search boosts reasoning via iterative preference learning. arXie preprint oiv:22405000,51 2024. 2
Haoran Xu. The joint of contrastive preference optimization (cpo) & simple preference optimization (simpo), 2024. wwwetttww.....ftteeeeeeeeeeeeeee wtw com/fetiixw//PO_SSS.. SIMPD. 2, 6, 7, 17, 18
URL https:/ / hgtggttthbbb.a
Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of Imm performance in machine translation. arXiv preprint 00100477 2024. 3, 17, 18
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, el al. Qwen2. 5-math technical report: Toward mathematical expert model via sell-immrovenent. ar preprind Xiv:2440 Stt:2409.12122 12122, 2024. 2, 6, 9, 17
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXie preprint arxi2:2505. 09388, 2025. 27
Mengjiao Sherry Yang, Dale Schuurmans, Pieter Abbeel, and Ofir Nachum. Chain thought imitation with procedure cloning. Advances in Neural Information Processing Systems, 35:36366 36381, 2022. 3
Shunyu Yao, Dian Yu, Jeffrey Zhao, Tzhak Shafran, Tom Griffitss, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2023a. 1, 2, 3, 23, 25
Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. arXiv preprint arXiv:2310. 10683, 2023b. 3, 5, 20
Jiacheng Ye, Jiahui Gao, Shansan Gong, Lin Zheng, Xin Jiang, Zhenguo Li, and Lingpeng Kong. Beyond autoregression: 10. Discrete diffusion for complex reasoning and planning. arXiv preprint 100.:2440.14157, 2024.
Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses align language models with human feedback. Advances in Neural Information Processing Systems, 36:10935 10950, 2023a.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXie preprint arXiv:2908.00 018825, 2023b. 2,
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang, Does reinforcement learning really incentivize reasoning capacity in Ilmss beyond the base model? arXiv preprint arXiu:2504. 2025. 27
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Nearal Information Processing Systems, 35:15476 15488, 2022. 2,
Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D Goodman. Quiet-star: Language models can teach themselves to think before speaking. arXiu preprint or orXi::203..00 09629, 2021. 2
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets": Llm self-training via process reward guided tree search. arXiv preprint arXim:24660088 03816, 16, 2024a. 2,3
Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. Negative preference optimization: From catastrophic collapse to effective unlearning. Xix preprint arXiu:2404 00.00644, 2024b. 3, 5, 21
Xuan Zhang, Chao Du, Tianyu Pang, Qian Liu, Wei Gao, and Min Lin. Chain of preference optimization: Improving of reasoning in 1lms. Advances in Neural Information Processing Systems, 37:333-30 3566, 2024c. 3, 4, 5
Zhexin Zhang, Junxiao Yang, Pei Ke, Shiyao Cui, Chujie Zheng, Hongning Wang, and Minlie Huang, Safe unlearning: A surprisingly effective and generalizable solution to defend against jailbreak attacks. arXiv preprint ar 200.005007 2024d.
Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter Liu. Slic-hi: Sequence likelihood calibration with human feedback. arXiv preprint Xiv:230. 10425, 2023. 3
15

---

Published in Transactions on Machine Learning Research (10/2025)
Zirui Zhao, Wee Sun Lee, and David Hsu. Large language models commonsensse knowledge for large-scale task planning, Advances in Newrul Information Processing Systems, 36, 2024.
Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng- Tze Cheng, Quoc Le, Ed H Chi, al. Natural plan: Benchmarking llms on natural language planning. arXiv preprint arXix:2466.00 04520,, 2024. 2
Andy Zhou, Kai Yan, Michal Shlapentoki-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning language models. arXiv preprint arXi::2310004500 2023.
Yuchen Zhuang, Xiang Chen, Tong Yu, Saayan Mitra, Victor Bursztyn, Ryan A Rossi, Somdeb Sarkhel, and Chao Zhang. Toolchain*: Efficient action space navigation in large language models with a* search. arXie preprint arXi::2310. 13227, 2023. 2
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406. 208877706,11877, 2024. 1
16

---

Published in Transactions on Machine Learning Research (10/2025)
A Fine-tuning Details
We follow the alignment handbook (Tunstall et al., 2024) to fine-tune two base LLMs with full parameters, Qwen2.5-Math 1.5B (Q1.5B) and Qwen2.5-Math 7B (Q7B) (Yang et al., 2024). Below are common setups for all the methods (i.e., SFT, UFT, and preference- based methods).
We disable sequence packing to prevent crossconttamination" and set a context length of 256 tokens. which is sufficient for our lasks. Training performed in bfloat16 precision, and model checkpoints are saved every 5% epochs for later evaluation.
The learning rate follows cosine _ateee_t________eeeeeeeeeeeeeeee frith_min___ ir schedule 2: it lincarly warms up from 0 to 1 over the first 10% epochs, then decays to a minimum of 7e-8 via cosine function for the rest epochs. In Sec. 5.2, use a peak learning rate of le-5 for Q1.B and 5e-6 for Q7B, while in Sec. 5.3, we decrease 7) to much lower values, as discussed in the main paper.
A.1 SFT Baseline and UFT
Following Sec. 4.2, in our implementation, Unlikelihood Fine-Tuning (UFT) and Supervised Fine- Tuning (SFT) differ only by the unlikelihood loss coefficient. A batch size B 128 is used for both successful and failed data batches (after gradient accumulation) in UFT.
A.2 sefference-bssed (Reference-free) Baselines: SimPO and CPO-SimPO
Background. In addition to SFT, we compare UFT against offline preference-based methods: SimPO (Meng et al., 2024b) and CPO-SimPO (Xu, 2024), both designed to align LLMs with human preferences. While UFT is not preference- based method per se, our reasoning task can be interpreted through the lens of preference aligmment, where successful reasoning paths are preferred over failed ones. Like preference-based approaches, CFT learns from both successful and failed paths, making these methods natural baselines for comparison. The implementations of these methods follow the TRL library (von Werra et al., 2020).
SImPO and CPO-SimPO build DPO (Rafailov et al., 2023) and CPO (Xu et al., 2024), requiring preference pairs but eliminating the need for reference model used in KL regularization (i.e., reference-free). This results in improved training efficiency and reduced GPU memory usage. By contrast, while KTO (Ethayarajh et al., 2024) like UFT operates on unpaired data, it still requires reference model during training. In our preliminary experiments, running KTO using the same global batch size as UFT necessitated precomputing reference model logits and offloading the optimizer to the CPU, resulting in a 3x increase in training time on the Game-of-24 dataset and out-of-memory errors on the Countdown dataset. Consequently, we exclude KTO from our baselines and instead focus on reference-free approaches such as SimPO.
Table summarizes the comparison between UFT and several offline preference-based methods. Notably, CFT is both reference-free and does not require preference pairs.
Table Brief comparison of selected offline preference-based methods and UFT.
Offline fine-tuning method
Preference data
Reference- free??
DPO (Rafailov et al., 2023), (non-iterative) RPO (Pang el al., 2024) CPO (Xu al., 2024), SimPO (Meng al., 2024b), CPO-SimPO (Xu, 2024) KTO (Ethayarajh et al., 2024)
Paired
No
Paired
Yes
Unpaired
No
UFT (Ours)
Unpaired
Yes
http:://gthtbbbooooooggggggg///a hhttps::/hugggiggaac. co/quen/Qweei 2aaam2255Katti 5-Kath-1.5B
whattps://hugiingfaac
www//uee//vee2..55eeee 5-Kahh-7B
httppp:://nggginffcc.../////t/// trainer
17

---

Published in Transactions on Machine Learning Research (10/2025)
Simple Preference Optimization with Reference- Free Reward (SimPO) (Meng et al., 2024b). SimPO relies on paired preference data, denoted as Daaired ((x,yt, ytx yy"},, where yy,y" are successful and failed reasoning paths for the same imput x, respectively. To construct paired from our original dataset D, we first filter for imputs that have both successful and failed paths. For each successful path (x,y*), we generate E prelerence pairs by randomly sampling E failed paths (x,y") corresponding to the same input, treating them as rejected responses. This sampling strategy ensures that the resulting dataset similar size to that used for UFT and captures much diversity as possible within the constraint of pairing.
The training objective is defined as:
B lyt|
B y
1 xx) -7) (7)
min JsimPo Daaired) -
E [xxy y-) P
log
log co x)
log To (y
where is the sigmoid function, 8 = 0.1 is a scaling factor, and 7 = 0.5 is the target reward margin, following default values in TRL. The length normalization terms (ly+), ly=l) help reduce bias toward longer responses. This normalization could also be explored in UFT future work. The margin y enforces minimum separation between the preferred and rejected log-probabilities.
CPO-SimPO (Xu, 2024). Although pure preference-based methods (e.g., DPO, SimPO) perform well in alignment tasks, they have been found to underperform reasoning tasks that require explicit maximization of preferred responses (Meng et al., 2024b; Pal et al., 2024; Pang et al., 2024). While CPO-SimPO was not originally motivaled by reasoning lasks, we adopt it because CPO-SimPO addresses this issue by augmenting SimPO with the NLL term from CPO (Xu al., 2024):
min Jopo-simpsoPP (0; Ppaired)
-- - y-) [logor (xt||
B ly
log To |x) -7) + Aogg ro 1x)
(8)
log (y xx)
where the NLL coefficient À = 1.0 following the TRL.
Connection with RPO (Pang et al., 2024). CPO-SimPO (Xu, 2024) resembles a non-iterative version of lterative Reasoning Preference Optimization (TRPO) (Pang et al., 2024), which we call RPO:
min JRPO (0; Dpaired)
--Ex y )n Dpatred log a [Bog
T x) Tref y x)
33 log
We (y
x)
+
log a xx).
Traf
x)
y
where Fee is the reference model (i.e., the base LLM). CPO-SimPO becomes equivalent to RPO when a reforence model is introduced, the length normalization and reward margin in the preference term are removed, and length normalization is applied the NLL term instead. However, since RPO, like DPO and KTO, depends on reference model, we exclude it from our baseline comparisons.
B Additional Results on Countdown and Game-of-24
B.1 Training Dataset Statistics
For Countdown, train for 2 epochs due to the large dataset size, shown in Table 5. On instance with A100 GPUs (40GB), total training time on COT+BFS+DFS data is under 5 hours for Q1.5B and under 30 hours for Q7B using UFT. For Game-of-24, we train for 10 epochs. Total training time is under 0.5 hours for Q1.5B and under 2.5 hours for Q7B using UFT. SimPO and CPO-SimPO only require a single dataloader over paired data, resulting in roughly 2x shorter training time compared to UFT.
We emphasize that our goal to reduce inference time in reasoning. While training may be relatively expensive, it is performed offline and does not impact deployment efficiency user-facing lateney.
18

---

Published in Transactions on Machine Learning Research (10/2025)
Table 5: Statistics of training datasets: total correct paths are used by SFT and UFT, while paired correct paths are used by SimPO and CPO-SimPO (shown as a percentage of lotal). Data quality refers to the best-of-n success rate reported in Table 2 and Table 3.
Training data (task, base, reasoner)
Total correct paths
#
Paired correct paths
Quality
Countdown, Q7B, CoT Countdown, N/A, BFS+DFS Countdown, Q7B, +oT+BBS+DDS
428.0k
97.50%
37.6%
614.2k
97.55%
86.2%
985.3k
99.88%
86.9%
Countdown, Q1.BB CoT
147.9k
94.22%
20.0%
Countdown, Q1.5B, CoT+BES+DFS
734.3k
99.86%
86.3%
Came-of-24, Q7B, CoT
9.1k
100%
92.9%
Game-of-24, Q7B, ToT+RAP
5.1k
100%
69.3%
Game-of-24, Q7B, CoT+
13.7k
100%
95.3%
Game-of-21,, Q1.5B, CoT
4.4k
100%
82.1%
Game-of-24, Q1.5, ToT+RAP
1.7k
100%
44.7%
Game-of-24, Q1.5B, CoT+TOT+RAP
6.0k
100%
87.6%
B.2
Additional Results with SFT and UFT
ToT data
CoT data
COT+TOT+RAP RAP data
@wen2.5-Math-78
Trained Game-of-24 and Tested Game-of-24
CoT (greedy)
CoT (pass@1)
Search: ToT
Search: RAP
0.30
0.3
Success Rate
0.25
0.20
0.15
0.15
0.10
0.1
0.10
0.05
Peak Ir
1e-6
Peak Ir
1e-6
Peak Ir 1e-6
Peak Ir 1e-6
Figure Results of standard SFT (a=0) using Qwen2.5-Math 7B the base model for Game-of-24.
The same conclusion in Fig. 3 holds.
For completeness, provide fine-tuning results based on Qwen2.5-Math 1.5B in Fig. 6 and Fig. 7. Given its weak scarch-based reasoning performance as a base model (c.g., success rate 22% in Countdown) shown in Table 6, the effects of learning rate, CoT data, and unlikelihood loss are minimal, likely due to its poor initial capability. This suggests that the observed benefits depend on the model capability.
Table 6: Performance Countdown and Game-of-24 test sets, using Qwen2.5-Math 1.5B the base model. Due to its weak search capability, we omit fine-tuning results here (please see Table 2 and Table for fine-tuiig results on CoT inference).
Test set & Inference
Base LLM
Countdown (1000 cases)
CoT (greedy)
2.2%
0.6m
CoT (pass@1)
2.1%
3.2m
search: ToT
2.2%
68m
search: RAP
1.4%
143m
Game-of-24 (100 cases)
CoT (greedy)
2.0%
<0.1m
CoT (pass@1)
3.0%
0.5m
search: ToT
8.0%
5.4m
search: RAP
4.0%
11.3m
19

---

Published in Transactions on Machine Learning Research (10/2025)
data
C data
Sieen.5-MMat--1.
Trained Countdown and Testad Countdown (pass@1)
(greedy)
0.100
0.4
0.4
0.015
0.075
Success
0.2
0.2
0.010
0.050
0.005
0.025
0.0
(b) Trained on Countdown and Tested on Game-of-24
0.3
0.3
0.06
Success
0.2
0.2
0.1
0.1
0.02
0.00
0.25
0.50
0.75
1.00
0.25
0.50
0.75
1.00
0.75
0.25
0.75
1.00
7
1e-5
7 1e-5
? 1c-5
1e-5
TRT+RAP data
CoT data
CoT+ TRP+ data
Qwen2.5-Manl 5-Math-1.50
Trained on Game-of-24 24 and Tested on Game-of-24 Search: TOT
CoT (greedy)
COT (pass@1)
Search: RAP
0.08
Success Rate
0.06
0.15
0.04
0.10
0.10
0.02
0.02
0.05
0.05
0.00
0.25
0.25
1.00
0.25
1.00
1.00
Peak Ir 1
Peak Ir n
Peak Ir
Peak Ir
Figure 6: Results of standard SET (a=0) using Qwen2.5-Math 1.5B as the base model for Countdown (top) and Game-of-24 (bottom). Due to the base model's weak search ability, learning rate and data source have little observable effect.
BFS+DFS data
CoT data
CoT+BFS+DFS data
Qwen2.5-Math-1.5.
(a) Trained on Countdown and Tested on Countdown
CoT (greedy)
CoT (pass@1)
Search: ToT
Search: RAP
Success Rate
0.02
0.4
0.05
0.01
0.2
0.00
0.00
0.0
0.0
(b) Trained Countdown and Game-of-24
0.08
Success Rate
0.3
0.06
0.04
0.2
0.2
0.04
0.02
0.00
0,00
10- 10-3 10-4 10-3
10 10-3 10 10-3
10-* 10 10-4 10-3 10 10 10-4 10-3
Unlikellhood
Unlikellhood
Unlikellhood
Unlikelihood
Figure 7: Results of UFT using Qwen2.5-Math 1.5B as the base model for Countdown, with the peak learning rate 1e-5. Due to the base model's weak search ability, unlikelihood loss has little observable effect.
B.3 Additional Results with Alternative Unlearning Methods
Gradient Ascent and RepNoise. To further evaluate our framework, we conduct additional experiments by replacing the unlikelihood (UL) loss JuL in Eq. with other established unlearning objectives, while keeping all other settings identical. In particular, we examine Gradient Ascent (GA) (Yao al., 2023b) and RepNoise (Rosati et al., 2024).
20

---

Published in Transactions on Machine Learning Research (10/2025)
GA is simple baseline for unlearning that aims to minimize the probability of failed paths:
JGA(0;D := E J-D [log (o( xx]]
(10)
Although GA is shown to be unstable during optimization (Zhang al., 2024b), we mitigate this issue by using a small loss coefficient of = le-5, which is on similar magnitude that used for UL. This choice substantially improves numerical stability in our experiments.
Representation Noising (RepNoise) (Rosati et al., 2024), on the other hand, is a recent advanced unlearning method designed for enhancing LLM safety. It combines Gradient Ascent with removing harmful representa- tions, encouraging the model's representations (per-layer hidden states) for negative inputs to align with random noise. Within our framework, its objective written as:
JRepNoise (0:
EEe WD
L log Te hhead (y h1p
1)
+ B - EXND
Σ (x)
(11)
:= :=log
post (x))
The first term acts as a generalized gradient- ascent objective (with an additional log- transformation), reducing the likelihood of undesired outputs based on the per-layer post-residual hidden states hi (x)) passed into the model's output head Thhet where 28 denotes the total number of layers. The second term enforces that the pre-residual hidden states h1 (x) to be close to Gaussian noise according to the maximum mean discrepancy (MMD). Following the original implementation, we set the RepNoise loss coefficient to 0.5
and its coefficient for the MMD term 3 1c-3.
CoT reasoning results for Gradient Ascent and RepNoise. Table 7 shows results for CoT reasoning with the same learning rate for all unlearning hodh on Game-of-24. GA achieves performance comparable to SFT and UFT, with slightly lower averages than UFT. RepNoise, which places strong emphasis on the unlcarning objective (via equal weighting and representation-love forgetting). performs consistently worse than the other methods. These observations suggest that directly applying unlearning methods originally designed for LLM safety may require additional design considerations.
Across all unlearning methods, our results consistently support the claim that higher training data quality, regardless of whether the data originate from CoT or search-based algorithms, leads to better fine-ture fine-tunin performance.
Search-based reasoning results for Gradient Ascent and RepNoise. Table 8 further examines the impact of learning rate and dataset source on CoT and search-based reasoning. Following the previous setup, we sweep learning rates over {1e-6, 2e-6, 5e-63 for the 7B base model.
For GA, our key findings for SFT and UFT (Sec. 5.3) remain valid. A smaller learning rate slightly reduces CoT reasoning performance but can substantially preserve, or even improve, the base model's ' search capability (e.g., 31% in ToT and 28% in RAP). Moreover, CoT training is more effective for maintaining search a ability compared to mixed data (CoT+ToT+RAR suggesting that using more on-policy data reduces distribution shift and mitigates forgetting, as discussed in Sec. 5.3.
In contrast, Ropee exhibits a drastic drop in search-based performance (ToT and RAP), reaching 0.0% regardless of the learning rate, even though its CoT reasoning improves over the base LLM. We hypothesize that this degradation arises from the large default coefficient (a = 0.5) in the RepNoise loss, which amplifies the effective learning rate for unlearning far beyond those used in other methods such as UFT and GA (< 10-3). This aggressive unlearning, coupled with representation-levell regularization toward random noise, likely disrupts internal representations crucial for search behaviors (e.g., hypothesis branching and reward estimation), even when the training objective (CoT reasoning) itself remains stable. These results underscore the importance of carefully tuning the learning rate and loss coefficient when extending unlearning objectives to LLM reasoning domains.
21

---

Published in Transactions on Machine Learning Research (10/2025)
Table 7: Additional results with alternative unlearning methods: CoT reasoning performance of LLMs fine-tuned and evaluated Game-of-24. Each cell shows the averaged result (4 seeds) of LLM
fine-tuned on data from specified sources. We highlight our two contributions (incorporating search-derived data and the fine-tuning method UFT).
(a) Base LLM: Qwen2.5-Math 7B (best suce with search: 28%)
SFT
UFT
GA
RepNoise
CoT data (quality: 92.9%)
suce (greedy)
28.2%
28.2%
26.8%
20.5%=a..7%
suce (pass@11
23.5%
23.2%+3.4%%
25.0%
17.2%=110%%
RRA+ data (quality: 69.3%)
suce (greedy)
13.2%
15.2%=2.2%
13.5%
12.8%=3.00%
suce (pass@01)
13.2%.+2.8%
17.2% 1.0%
15.8% 19
13.2%6=5.1%
CoT4TOT+RA RAP data (quality: 95.3%)
suce (greedy)
27.5%
30.2%
28.0%+1.77%
20.8%=2.11
succ (pas=01)
24.2%:15.22% 2%
26.2%.12.29
25. 25.5% 1%
19.0%.2.22
(b) Base LLM: Qwen2.5- Mathh 1.5B (best suce rate with search: 8%)
RepNoise
CoT data (quality: 82.1%)
succ (greody)
14.5%.3.88%
18.2%13.29
15.8%12.88
12.2%
(pass01)
17.8%13.4%%
18.3%.13.58
17.0%.12.2%
12.0%.12.29
ToT+RAP data (quality: 14.7%
suce (greedy)
9.6%++0.6%
8.0%
9.2%%+33.8%
5.5%%
suce (pass@1)
10.0%++1.4%
9.5%++1..6
9.0%++1.8%
6.2%
CoT+TOT+RP data (quality: 87.6%)
suce (greedy)
20.0%++2.7%
20.5%+3.77%
20.8%
10.3%=3.08
suce (pass@11)
17.0%
18.8%*113%%
20.0%
12.8%
Table 8: Additional results with alternative unlearning methods: CoT and search-based reasoning performance of LLMs fine-tuned and evaluated on Game-of-24. Each cell shows the averaged result (4 seeds) of LLM fine-tuned on data from specified sources. We highlight the best results for each pair training data and inference method.
(a) Gradient Ascent
Learning rate
Co'l' (greedy)
ToT
RAP
Base LLM (Qwen2.5-Math 7B)
N/A
6.0%
28.0%
27.0%
CoT data (quality: 92.9%)
5e-6
26.8%=2.1%
15.5%=4..1%
11.2%%41.7%
2e-6
23.0%
31.0%:t..85
22.8 22.8%+4.66%
20.2%:11.00
24.2%
28.0%
ToT+RAP data (quality: 69.3%)
5e-6
13.5%=a.19 %
6.8%=2299%
6.2%+22.%
2e-6
9.8%
20.0%
18.8%=7.6%
le-6
11.8%21.7%%
20.8%.1..99
18.5%22..3%
CoT+ToT+RAP data (quality: 95.3%)
5u-6
28.5%
14.8%=2..5%
12.0% 1.8%
20-6
20.2%
19.5%%13.35
21.8%=2.555
1e-6
17.5%
23.8 23.8%
13.5%%133.1
(b) RepNoise
Learning rate
Coll (greedy)
ToT
RAP
Base LLM (Qwen2.5-Math 7B)
N/A
6.0%
28.0%
27.0%
CoT data (quality: 92.9%)
5o-6
20.5%.3.199
0.0%.10
0.0%:10.00
2c-6
13.5%
0.0%
0.0%
le-6
8.0%
0.0%++..06
0.0%++o.00
ToT+RAP data (quality: 69.3%)
5e-6
12.8%=3. 0%
0.0% 00%
0.0%:2o000%
2e-6
8.2%+11.4%
0.0%++o.0%
0.0%=+..0%
le-6
6.0%+11.4%
0.0%=+0.0%
0.0%++0.0%
CoT+TOT+RAP data (quality: 95.3%)
5c-6
20.8%.22.15 19%
0.0% 0%
0.0%
26-6
17.8%
0.0%%+4..0%
0.0%+++..0%
le-6
14.5%+4..4%
0.0%
0.0%=40.0%
22

---

Published in Transactions on Machine Learning Research (10/2025)
C Reasoner Details
C.1
The Problems
This section provides details on the two math games not covered the main paper.
2fme-of-24. We randomly split the first 900 cases (rank #1-900 by human average performance in the dataset 1 into Xersin (720 cases) and Xealid (180 cases). The Neett are the next 100 cases (rank #901-1000) following the setup in ToT (Yao et al., 2023a). As a result, the validation set evaluates in-distribution (ID) generalization, while the test. set evaluates out-of-distributtion (OOD) generalization based on human difficulty levels. Each guaranteed be solvable using the four input numbers, which range from 1 13.
Countdown. We follow SoS codebase 15 (Gandhi et al., 2024) to randomly generate 500k training cases, 1k validation cases, and 1k test cases. The training and validation sets share the same distribution of target numbers, while the test set includes distinct target numbers (inchuding the number 24). Each problem consists of four input numbers sampled from 99, with target number between 10 and 100. All generated problems are guaranteed to have least one valid solution path using only integer intermediate results.
C.2 CoT Reasoner
Listing 1: Few-shot CoT template on Countdown (5-shot prompt; only shown for brevity), used in data generation. Here, <input> denotes a placeholder for the input numbers, <target> for the target number, and <response> LLM's output.
Use aumbers and basic arithmetic operations (+ -+- /) to obtain the target number. Each step you only alloved to choose two of the remaining numbers to obtain new number.
Input: 25 5 33 Target: 27
Stepa:
(left: 5 33
(left: 33
6 27 (left: 27)
Answer
((25 5) 5)
Input: 45 10 11 70 Target: 94
Steps:
10 11 = 21 (left: 45 70 21)
45 70 = 115 (left: 21 115)
115 21 - 94 (left: 94)
Ansuer: (45 70) (10 + 11) - 94
Input: <input> Target: <target>
Steps: <response>
Listing 2: Zero-shot CoT template Countdown, used in fine-tuning and evaluation.
Use aumbers and basic arithmetic operations (+ /) to obtain the target number.
Input: <input> Target: <target> Steps: <response>
Chain-of-Thought (Wei et al., 2022) uses the few-shot prompting lemplate (see Listing 1 for Countdown: Game-of-24 template follows the same format) for data generation and the zero-shot template (see Listing 2) for finc-tuning and evaluation. The "groedy-decoding" evaluation uses zero temperature. The "pass@1" evaluation uses temperature of 0.7 and top_p of 0.8 sample paths (n=8) for Countdown and 20 paths (n=20) for Game-of-24, then calculates the average success rate.
23

---

Published in Transactions on Machine Learning Research (10/2025)
C.3 ToT and RAP Reasoners
Listing 3: Proposed prompt template used for search-based reasoners on Countdown (6-shol prompt; only 3 shown for brevity).
Perform basic arithmetic operation (+, /) two of the given numbers, replacing them with the result. Your goal is to explore combinations that may lead to final result of the target number.
Input: 25 5 Target:
Possible stepa:
30 (left: 5 33 30)
125 (left:
125)
(left:
5)
58 (left:
(left: 25 33
33 38
(left:
38)
8
(left: 5 8)
5 28
(left: 5 28)
Iaput: 9 25 2 Target: 43
Possible next stepa:
9 + 25 34 (left: 34 2)
9 * 2 18 (left: 25 18)
9 + 2 11 (left: 25 11)
25 * = 50
(left: 9 50)
27 (left: 9 27)
Input: 21 115 Target:
Possible steps:
(left:
(left: 136)
Input: <input> Target: <target> Possible next steps: <response>
Listing 4: Intermediate reward prompt template used for search-based reasoners on Countdown (11-shot prompt; only shown for brevity).
Evaluate if given number(s) can reach the target number (sure/liieely/mpooabbbl
Input: 27 Target: 27
sure
Input: 25 18 Target: 43 25
18 43
Input: 31 10 Target: 52
10 3.1
impossible
Input: 45 70 21 Target: 94
45 70 + 21
115 + 21 136
-45 + 70 21 25 + 21 = 46
45 70 21
115
21 94
sure
Input: 15 16 16 Target: 43
15 16 16 = 47
(16 15) * 16
16 16
cannot obtain now, but numbers vithin reasonable range likely
Input:
Target:
24

---

Published in Transactions on Machine Learning Research (10/2025)
108 97 all big impossible
Input: <input> Target: <target> <response>
Listing 5: Terminal reward prompt template used for search-based reasoners on Countdown (6-shot prompt; only 4 shown for brevity). Here, <answer> is a given equation.
Use numbers and basic arithmetic operations (+ /) obtain the target number. Given input and answer, give judgement (sure/ (urre/impassibl) if the is correct, 1.e. it each input exactly once and no other numbers, reach the target number.
Input: 25
Target: ((25 5)
Ansuer:
Judge: sure
Input: 92 91 23 54 Target: 78
Ansuer: 92 91 (23 54) 106
Judge: imposaible
Input: 45 13 11 70 Target: 94
Ansuer: 13 + 11 + 70 94
Judge: impossible
Input: 25 5 5 33 Target: 27
Ansver: (33 5) (25 5) 27
Judge: impossible
Input: <input> Target: <target>
Ansuer: <ansver>
Judge: <response>
The search-based reasoners (ToT (Yao et al., 2023a) and RAP (Hao et al., 2023)) are implemented using the LLM-reasoners codebase (Hao et al., 2024). Both use the same prompt templates but differ in their underlying search algorithm.
Tree-of- Thought (Yao et al., 2023a). ToT employs a beam search strategy. It begins with an empty set of nodes (the beam). At each search step, the LLM is prompted using the propose prompt (Listing 3) to generate plausible next steps for each node in the beam. Each proposed step is then evaluated with the intermediate reward prompl (Listing 4), which labels responses as "sure", "likely", or "impossible". Following the original ToT setup, these labels are mapped heuristic scores of 1, 0.1, and 0.0001, respectively. To roduce variance, sample three responses for cach reward step, and compute the average heuristic reward.
The beam is updated by selecting the top candidate nodes based on these heuristic scores, maintaining the predefined beam size. Since both Countdown and Game-of-24 tasks involve four steps in thain-of-thought reasoning, this process is repeated for four iterations. At the final search step, the terminal reward prompt (Listing 5) is used to assign a terminal reward to each node in the final beam. The path with the highest terminal reward is selected as the final reasoning path for evaluation, while all paths the final beam are retained for training data.
To improve efficiency, optimize the codebase using vLLM (Kwon al., 2023), which enables batched inference across =1000 cases at the same time. In total, the ToT process requires 4 x 2 = 8 passes (i.c., 8 calls to 11m. mim.geeeeaae((() () in vLLM) per batch. This batched approach significantly accelerates the search process, achieving over 100x speedup compared to the original for-loop execution in the L.M-Reasoners codebase.
In practice, we vary the beam size between and 16 while keeping other search parameters fixed during training data generation. Empirically, we find that beam size of 5 for Countdown and for Game-of-24 yields the best performance on Qwen2.5-Math 7B, while beam sizes of 8 and 10 perform best for the respective
25

---

Published in Transactions on Machine Learning Research (10/2025)
tasks Qwen2.5-Math 1.5B. Accordingly, we adopt these optimal beam sizes for evaluating both the base LLMs and the improved models fine-tuned from them.
Deasoning-via-Plannnne (Hao et al., 2023). RAP employs Monte Carlo Tree Search (MCTS) strategy, maintaining search tree initialized as empty. At each iteration, a node is selected for expansion using the Upper Confidence Bound applied to Trees (UCT) algorithm (Kocsis & Szepesvári, 2006), where the exploration parameter balances exploitation and exploration. The selected node is then expanded by simulating full rollout: the propose prompt generates plausible next steps, and reward prompts are used to assign scores each step along the way. Steps sampled proportionally to their rewards throughout the rollout. After completing the rollout, the terminal reward is backpropagated through the tree to update the value estimates of the visited nodes.
We run this process for 100 iterations, resulting in up to 100 x x = 800 passes. This is a loose upper bound, 88 many selected nodes are leaf nodes or near-terminal, leading to shorter rollouts practice. While the original -LM-Reasoners implementation uses only 10 iterations, we increase it to 100 to improve performance, made feasible by our use of batched inference. At the end of the search, the path with the highest terminal reward is selected for evaluation, and all explored paths are extracted for training data.
In practice, vary the exploration parameter over (1.0, 2.0, 4.0, 6.0, 8.0, 10.0) while keeping all other search parameters fixed during training data generation. Empirically, we find that an exploration parameter of 1.0 yields the best performance for both Coundown and Game-of-24 on 2aathh5-Math 7B, whereas a value of 2.0 performs best for both tasks on Qwen2.5- 1.5B. Accordingly, we adopt these optimal exploration parameters for evaluating both the base LLMs and the improved models fine-tuned from them.
C.4 Loss of Search Capability after CoT Fine- Tuninnn Planning Perspective
An important insight emerges when comparing the prompting strategies used in CoT reasoning versus search-based methods like ToT and RAP: although they aim solve the same lask, they frame the role of the LLM in fundamentally different ways.
CoT treats the LLM as open-loop controller, generating complete reasoning paths in a single pass (Listing 1) without any feedback or evaluation. In rast, search-based methods integrates the LLM into closed-loop system, where the LLM serves two interconnected roles: policy that proposes next steps (Listing 3) and a reward model that evaluates them (Listing 4, Listing 5). This interaction between proposal and intermediate feedback is central to structured planning and search.
This difference has meaningful consequences for fine-tuning, When an LLM is fine-tuned on CoT--syyl paths that strip away reward signals, may weaken its ability to perform reward modeling, which is critical in search-based reasoning. This observation offers theoreic explanation for why models fine-tuned on -oy-style paths may struggle with inference-time search.
C.5 Classic BFS and DFS Reasoners
We also employ classic search algorithms, Breadth-First Search (BFS) and Depth-First Search (DFS), to generate training data for the Countdown task. We adopt the implementation from SoS (Gandhi et al., 2024) without modification. These symbolic reasoners are significantly faster than LLM reasoners and achieve high success rates of 65.4% (BFS) and 81.5% (DFS) on the training set (due to the use of pruming heuristics, they do not guarantec a solution in all cases). However, unlike LLM reasoners, these methods rely on external tools such as calculators and goal checkers to guide the search process.
Limitation
This paper focuses on improving reasoning language models. Preliminary experiments on one generic language model, Mistral 7B (Jiang et al., 2023), show that while diverse reasoners still provide benefits, small learning rate and the forgetting objective offer limited improvement. We suspect these effects depend more on the base model's reasoning ability than size, similar to observations in RL that performance is bounded by base
26

---

Published in Transactions on Machine Learning Research (10/2025)
model capability (Yue et al., 2025). Extending method to other reasoning- -oriente LLMs is valuable direction for future work.
E Results on Standardized Math Benchmarks
For completeness, we also report results standardized math benchmarks. These evaluations not the main focus of our work, which centers on math puzzles such as Countdown and Game-of-24.
The training problems for our standardized math experiments are drawn from the Open'Thoughts- Sinittss-i 14k-mmtth dataset 16 (Guha et al., 2025), which itself is extracted from NuminaMath-Cof1 (Li et al., 2024). This corpus integrates problems from many high-school math competitions, including the Olympiad, MATH (Hendrycks et al., 2021), AoPS forums, and AMC/ATME competitions. From this collection, we curated a subset of 38k problems with numerical gold answers, which we used as our training problems.
The training CoTss are constructed from three sources: (1) human- CoTs containing gold answers in (2) self-generated CoTT whose final answers match the gold answers, and (3) self-generated CoTIs whose final answers do not match the gold answers. Since Open l'houghts, like many other public datasets, does not include negative CoTs, we generate them ourselves. To verify correctness, we employ math-verify to check the answers wrapped by [booxed[} in CoTIs following common practice. The instruction used for both training and testing is: Please reason step by step, and put your final answer within (boxed SFT is trained only on CoTss (1) and (2), while the other methods are trained with CoTs (1) (3). It is worth noting that although CoTIs in category (2) contain gold answers, their reasoning steps may still be incorrect, since process-basse verifier exists for general math problems.
Due to computational resource limits, we train each method for 2 epochs with a context length of 2048, using random seeds. The training time for the 7B models is around 15 hours on 8 A100 GPUs (40GB each).
We evaluate each fino-tuned LLM on the held-out test datasets using real math competitions AMC-12 (2023 and 2024)'8. We do not include more advanced contests in our evaluation, as they typically require substantially longer context lengths (exceeding 4k tokens (Yang et al., 2025)) and large-scale online RL (Guo et al., 2025), which go beyond the scope of offline fine-tuning considered in this work.
Table 9: CoT reasoning performance of LLMs fine-tuned on 38k high-school math problems and evaluated on AMC-12 datascts. Each mothod is trained with sceds. The evaluation metrics are pass@1 success rate, estimated from 32 sampled paths for each problem.
(a) Base LLM: 2wenh.5-Math 7B
Base
SFT
SimPO CPO-SimPO
UFT
AMC 2023
56.3%
61.5%10.9%
0%10%
59.3%.12.
61.6%110.%
AMC 2024
29.5%
30.1%=0.7%
0%400
30.6%+2.2%
29.8%=4..%
(b) Base LLM: Qwen2.5-Math 1.5B
Base
SFT
SimPO CPO-SimPO
UFT
AMC 2023
38.3%
50.0%+222%
0%-20
50.9%+3.2%
50.2%++1.2%
AMC 2024
18.6%
28.8%=
0%-0%
28.6%+40.% 28.7%%40.5%
Table 9 shows the results. Except for SimPO, all other methods are not statistically distinguishable. Morcover, the improvements over the base LLM are minor for all 7B models. This likely reflects the noisier nature of human-written written general math training data compared to the synthetic, structured data in our math puzzles. In math competitions, test problems differ substantially in style and content from traïning problems across years, making it harder to generalize from failed reasoning paths. By contrast, in our math puzzles, training and test problems share the same synthetic structure, which facilitates generalization. The weaker performance
10https:/hhugginggaac http:///www..tttaaaaaa/a///////t 17 lFnttpe://githhbb.m http:://hhhh.cccccooooogggggaaaa pMp/AMC ww_.lololsssaddd Bttpollessaad_Souuttieee Solutions ohttpe::/artofrroo.mmovvvvggg.ee
27

---

Published in Transactions on Machine Learning Research (10/2025)
may also stem from our limited training setup (38k problems, 2048 context length). Nevertheless, unlike SimPO, our method UFT, which also leverages negative paths, remains as stable as SFT, underscoring its robustness under limited training conditions.
28


Reviews:
[{"review_id": "yvpa8Upiy5", "reviewer": "Reviewer_n7cM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper propose a better method to utlize the unsuccessful reasoning traces.  The idea is to extend the unlearning methods from LLM safety research to unlearn the unsuccessful reasoning traces.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. Extensive experiements are done. \n\n2. However, I require the authors to conduct the following experiment to validate the advantage of  \"unlikelihood (UL) loss\" compared to several alternatives:\n\n **Ascend loss**. This is the most standard unlearning loss, which is basically the negative NLL (or cross entropy) loss. Basically optimizing this loss is taking one step of standard gradient ascend over the unlearning dataset. \n\n**Reprensetation noise loss**[1]. The representation loss (See Eq. (5) in [1]) basically make the representation of the unlearning dataset to be similar the pure random Gaussian noise. In your case, you basically make the representation of  failed reasoning traces to be random. \n\n**Tamper resistant loss[2]**. The tamper resistent loss basically make the unlearning loss to be high  and **hard to recover** even though taking extra recover steps.    See the first term in Eq. (1) of [2]\n\n**Smooth loss[3]** Similar to tamper resistent losss, smooth loss make the unlearning loss hard to learn. In order to achieve that, smooth loss (See the second term in Eq.(1) of [3]) make the learning curve of the failed reasoning traces to be smooth.  \n\n[1] Representation Noising: A Defence Mechanism Against Harmful Finetuning\n\n[2] Tamper-Resistant Safeguards for Open-Weight LLMs\n\n[3] Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbation\n\n\n**All these alternatives** should be tried and performed by the authors. Also, the authors should also combines the above methods with inference time reasoning methods (ToT and RAP) to evaluate.  Otherwise, I can't agree that the contribution of proposing to use this  \"unlikelihood (UL) loss\" is beneficial against other unlearning loss.  Feel free to leave me a comment if you have difficulty in reproducing these four unleanring losses. I am very familar with all these unlearning losses I mention.", "audience_yn": "Yes", "audience_explanation": "Yes, at least I am very interested  and excited in this study.", "requested_changes": "1. Conduct experiments to validate the advantage of unlikelihood (UL) loss against other unlearning loss. \n\nI am okay with acceptance if unlikelihood indeed outperform the other four losses. Please also consider to give me the anonymous code for your implementation such that I can verify by myself.", "broader_impact_concerns": "No concerns.", "additional_comments": "I personally don't know whether there are existing works that already explore the idea of unlearning failed reasoning traces. But the idea looks standard and straightforward and there might be other existing (or concurrent) work. \n\nI suggest the authors to actively search and  honestly disclose whether there are existing or concurrent work explore the same idea of  unlearning failed reasoning traces.   If this is not done sufficient, I think accepting this paper will be unfair for other authors who explore the same idea and be detrimental to the research community.  I will look at other reviewers' comments and the authors response to see whether this concern is addressed.", "extra_scores": {}, "extra_text": {}}, {"review_id": "5pAIVLbt3t", "reviewer": "Reviewer_xHGN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents Unlikelihood Fine-Tuning (UFT), a procedure that extracts CoT-style paths from different reasoners (CoT, ToT, RAP, classic BFS/DFS) and fine-tunes an LLM by maximizing likelihood on successful paths while jointly applying an unlikelihood loss to unpaired failed paths to “forget” bad trajectories.\nUFT outperforms other methods for CoT distillation on Game-of-24 and Countdown problems with Qwen2.5-Math base models. The work further discusses that low learning rates are needed to preserve the base models search capabilities.\n\n### Strengths\n* The problem of distilling reasoning traces into LLMs is important.\n* The experiments demonstrate empirical improvements over prior work.\n* The insights on the benefits of using high-quality search-generated data and low learning rates is practically useful.\n\n### Weaknesses\n* The novelty of UFT is incremental as it combines two existing ideas (NLL and UL loss) in a straight-forward manner.\n* The experimental scope is confined to two closely related arithmetic reasoning tasks. The paper does not demonstrate that the findings generalize to more diverse and complex reasoning domains.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Despite the limited evaluation scope, I do think the core findings are technically correct.", "audience_yn": "Yes", "audience_explanation": "The problem of distilling unpaired positive and negative reasoning traces into LLMs remains an important problem. I do think the core observations are of interest.", "requested_changes": "I think the following two changes would be critical for improving the work:\n* Experimental results on more diverse reasoning problems. The evaluation would be significantly more convincing if it included some tasks that are not arithmetic puzzles like Countdown and Game-of-24.\n* Evaluation data for different base models. It is unclear whether the empirical observations are specific to the Qwen2.5-Math model family or if they are more broadly applicable.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0Rwg95Vqet", "reviewer": "Reviewer_xuWZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a method to improve reasoning in LLMs. The method consists of the following steps: generate CoT data with LLMs and search-based methods (DFS, BFS), unify the format, assign reward with a process-based verifier (correct or wrong). This gives unpaired failed and correct reasoning traces. Fine-tune on these with likelihood (for correct) and unlikelihood (for incorrect) training. The authors experiment with two tasks (countdown and game-of-24) and two LLMs (qwen2.5-math 1.5B and 7B). Baselines compared to are inference-team methods on the base models as well as other finetuning methods using paired correct/failed reasoning paths. They show gains of baselines in most cases, and thoroughly investigate the factors driving this, finding that high-quality data is important. Additionally, the authors find that finetuning in these ways can lead to reduced inference-time search performance, which can be mitigated both by training on \"on-policy\" CoT data as well as a lower learning rate.", "claims_and_evidence_yn": "No", "claims_explanation": "The experimental setup is sound and the authors use the right baselines and ablations, but theres a few issues:\n- countdown and game-of-24 are not two separate tasks, as the authors also note themselves, one is a subset of the other. It's difficult to make claims about \"mathematical reasoning\" when experiments are only done on such a narrow domain.\n- Gains are mostly modest or within confidence intervals, and sometimes baselines perform methods better. Can you test significance of the difference between normal SFT and UFT for the 14/18 and 16/18 times you claim your method improves? For example, I count 10 where the confidence intervals seem overlapping in table 2.", "audience_yn": "Yes", "audience_explanation": "The authors propose a simple method for training on unpaired correct and failed reasoning traces, which is potentially beneficial for mathematical reasoning tasks that explicitly require search (like countdown). Additionally, the authors make some interesting observations, like that CoT data leads to better retainment of search abilities after fine-tuning.", "requested_changes": "- (critical) Add another mathematical reasoning task that is different from countdown, or change the wording to make it less about mathematical reasoning and more about countdown / an investigation into what factors influence successful fine-tuning (data quality, learning rate, etc).\n- (critical) Change wording about the method improving over baselines to only include cases where the method statistically significantly outperforms SFT\n- (would strengthen the work) Recurring spelling mistakes (e.g. \"while prior work typically fine-tune LLMs ..\"), consider correcting the text with an LLM.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)