Dualformer: Controllable Fast and Slow Thinking by Learning with Randomized Reasoning Traces
Authors: DiJia Su
Sainbayar Sukhbaatar
Michael Rabbat
Yuandong Tian
Qinqing Zheng
Venue: ICLR 2025 Poster (2025)
Keywords: planning
reasoning
Sequential Decision Making
Primary area: foundation or frontier models, including LLMs

Abstract: In cognition theory, human thinking is governed by two systems: the fast and intuitive System 1 and the slower but more deliberative System 2. Analogously, Large Language Models (LLMs) can operate in two reasoning modes: outputting only the solutions (\emph{fast mode}) or both the reasoning chain and the final solution (\emph{slow mode}).  We present \dualformer, a single Transformer model that seamlessly integrates both the fast and slow reasoning modes by training on randomized reasoning traces, where different parts of the traces are strategically dropped during training. At inference time, \dualformer can be easily configured to execute in either fast or slow mode, or automatically decide which mode to engage (\emph{auto mode}).  It outperforms baselines in both performance and computational efficiency across all three modes: \textbf{(1)} in slow mode, \dualformer achieves $97.6\%$ optimal rate on unseen $30 \times 30$ maze tasks, surpassing the \searchformer baseline (93.3\%) trained on data with complete reasoning traces, with $45.5\%$ fewer reasoning steps; \textbf{(2)} in fast mode, \dualformer achieves $80\%$ optimal rate,  significantly outperforming the Solution-Only model trained on solution-only data,  which has an optimal rate of only 30\%;  \textbf{(3)} in auto mode, \dualformer achieves $96.6\%$ optimal rate with $59.9\%$ fewer steps than \searchformer.  For math reasoning problems, our techniques have also achieved improved performance with LLM fine-tuning, demonstrating its generalization beyond task-specific models. We open source our code at https://github.com/facebookresearch/dualformer.

Published as a conference paper at ICLR 2025
DUALFORMER: CONTROLLABLE FAST AND SLOW THINKING BY LEARNING WITH RANDOMIZED REASON- ING TRACES
DiJia Su
Sainbayar Sukhbaatar
Michael Rabbat Yuandong Tian Qinqing Zheng
Meta AI
ABSTRACT
In cognition theory, human thinking is governed by two systems: the fast and intuitive System I and the slower but more deliberative System 2. Analogously, Large Language Models (LLMs) only the solutions (fast mode) or both the reasoning chain and the final solution (slow mode). We present Dual Dualformen former, seamlessly integrates both the fast and slow reasoning modes by training on randomized reasoning traces, where different parts of the traces are strategically dropped during training. At inference time, Dual I former can be easily configured to execute in either fast slow mode, or automatically decide which mode to engage (auto mode). It outperforms baselines in both performance and computational efficiency across all three modes: (1) in slow mode, Dual former achieves 97.6% optimal rate on unseen 30 x 30 maze tasks, surpassing the Search rmer baseline (93.3%) trained on data with complete reasoning traces, with 45.5% fewer reasoning steps; (2) in fast mode, Dual former achieves 80% optimal rate, significantly outperforming the Solution-Only model trained solution-only data, which has optimal rate of only 30%; (3) in auto mode, Dualformer achieves 96.6% optimal rate with 59.9% fewer steps than produces more diverse reasoning
operate in two reasoning modes: outputting
single Transformer model that
Sea tearchforme: Moreover, Dual fo traces than Sea rch meaacchorrme former. For math reasoning problems, our techniques have also achieved improved performance with LLM fine-tuning, demonstrating its generalization beyond task-specific models. We open source our code at https: / //gg/gitthub/ com www.//faceboooeeeeereeeeeeoeeeee
INTRODUCTION
Transformers (Vaswani et al., 2017), the sequence modeling tool that serves the cornerstone of foundation models various domains including Large Language models (L.LMs) (Dosovitskiy, 2020; Baevski al., 2020; Radford al., 2021: Touvron al., 2021: Hsu et al., 2021; Touvron et al., 2023; Dubey et al., 2024), have been widely used in many works to approach reasoning and planning problems, see e.g., Zhou et al. (2022); Kojima et al. (2022); Pallagani et al. (2022); Valmeekam et al. (2023a): Chen et al. (2024); Gundawar et al. (2024); Wang & Zhou (2024), Specifically, we can categorize the reasoning modes of Transformers into fast and slow. In fast mode, a Transformer will output a final solution without any reasoning steps, whereas the intermediate steps of thinking, such as search trace for finding a short path, will be generated along with the plan in slow mode.
The two inference modes share lot of similarities with the two thinking systems inherent in us (Wason Evans, 1974; Kahneman, 2017): an automatic and unconscious System and a controlled and conscious System 2. More importantly, they come with analogous pros and cons. As discussed in previous works (Wei et al., 2022; Valmeckam et al., 2023b; Lehnert et al., 2024; Gandhi et al., 2024; Saha et al., 2024), Transformer models operating in fast mode have lower computational cost and allow for a quicker response, yet they fall short in accuracy and optimality compared with slow mode models (see Figure B.1 for concrete example). This raises interesting question:

---

Published as a conference paper at ICLR 2025
Can we integrate both the fast and the slow modes into Transformer-add based reason- ing models, similar to how humans possess two distinct thinking systems, and let them complement each other?
Multiple approaches have been proposed. A popular paradigm is to start with a pure System 2, and then fine-tune to make it more efficient like System 1, e.g., to distill System 2 output into System 1 (Yu et al., 2024; Deng et al., 2024), to improve the reasoning efficiency of existing system 2 by distillation (Wang et al., 2023; Shridhar et al., 2023) or bootstrapping from symbolic systems (c.g.. Sea racchhhe fo rmmrr (Lehnert al., 2024), Stream of Search (Gandhi al., 2024)). However, in these cases, further fine-tuning is needed, which is computationally expensive, and it is non-trivial to adapt the resulting system to be more like System or System on the fly. To address this problem, Saha et al. (2024) design an explicit meta-controller to switch between two different systems.
In this work, we demonstrate surprising finding: a simple data recipe suffices achieve on-the-fly System I and System 2 configuration in solving reasoming tasks. The resulting model, Dual former, r, can be easily configured to execute in either fast or slow mode during inference, and determines which mode to use by itself if not specified. More specifically, to imitate the System 2 reasoning process, our Transformer is trained on data that contains both the reasoning trace and the final solution. Leveraging the structure of reasoning steps, we design specific trace dropping strategies such that the resulting traces resemble the shortcuts taken by System in the thinking process. In the extreme case, we drop the entire trace and encourage the Transformer to output a final solution directly, bypassing all the intermediate steps. We randomly choose those structured trace-dropping strategies training time. See Section for the details.
We first apply our framework to train an encoder-decoded Transformer model to solve pathfinding problems, where the trace is generated by the A search algorithm. We consider two domains: the Maze navigation and the Sokoban game as in Lehnert et al. (2024), where we use the same tokenization scheme. Interestingly,
have found that these problems are challenging for sate-of-thhearr LLMs like ol -prevviw and ol-mini, where the output path often breaks into the walls (See Appendix for an example). In each reasoning mode, Dual f former r outperforms the established baselines, achieving stronger results in both the solved rate and optimal rate. Moreover, Dual former significantly enhances the diversity of the generated plans by identifying greater variety of unique paths that reach the goal. Notably, Dualf former is also efficient even when working in slow mode, generating much shorter reasoning traces than the baseline model. Next, we apply our framework to fine-tune LLMS for answering math questions. Following Yu et al. (2023), the training examples are taken from the MATH dataset (Hendrycks et al., 2021) where answers are rewritten by Llama-3. 1--1---B-nnnnrie model to include detailed intermediate steps. Likewise, the obtained LLMs demonstrate enhanced efficacy and efficiency.
RELATED WORK
Learning to Plan and Reason Tremendous efforts have been made to enhance the capability of Transformer based models to plan and reason over a long horizon. Two main types of approaches have been developed. The first type leverages existing LLMs. For instance, researchers taught L.MMs to call external existing symbolic solvers, such as those found in Ahn et al. (2022); Besta et al. (2024);
Sel al. (2023); He- YueYu et al. (2023); Liu al. (2023); Silver et al. (2024). pallagani al. (2022; 2024) investigate the fine-tuning of LLMS to use a symbolic solver, (Schick et al.; Hao et al., 2024) fine-tune LLMs use external tools, and (Hao et al.) propose to fine-tune LLMS to do reasoning and planning within a world model. Among them, several works try to integrate System-1 I and System-2 thinking into LLM reasoning. Weston & Sukhbaatar (2023) proposed the System Attention (S2A). more deliberate attention mechanism aimed at improving the reasoning capabilities of large language models by reducing their susceptibility to spurious correlations in the context. Yu et al. (2024) distill System 2 output into System model, which aims to compile higher-quality outputs from System 2 techniques back into LLM generations without intermediate reasoning token sequences.
The second type aims train Transformers from scratch to plan and reason independently (Lehnert et al., 2024; Saha et al., 2024; Gandhi et al., 2024), using task-specific language. Both Lehnert et al. (2024) and Gandhi et al. (2024) teach LLMs to search by representing the search process in language, Lehnert et al. (2024) developed Searchforma for pathfinding problems. Gandhi et al. (2024) applied their model to the Countdown game. In the
that is trained to mimic the A search process
2

---

Published as a conference paper at ICLR 2025
concurrent work Saha al. (2024), the authors trained two separate models and manage them by an external contrcootrrlller. There, one model generates rapid traceless response, while another one responds slower but with trace. Similarly, (Lin et al., 2023) leverages multiple models to implement slow and fast thinking using an agent style workflow. Our research is closely related to the above work, but has some key differences. Unlike Lehnert et al. (2024); Gandhi al. (2024) which do not modify the reasoning trace in the training data, our approach involves randomizing the reasoning trace. Unlike Saha al. (2024); Lin et al. (2023), we do not use any explicit controller nor use two networks for each mode. Instead, we integrate both the fast and the slow mode functionalities into a single model.
Synthetic Data Generation using LLM Large language models (LLMs) have been utilized for synthetic data generation in various domains. For instance, Wei et (2021); Longpre et al. (2023); Taori et al. (2023) introduced synthetic instruction dataset by sampling from diverse templates containing natural language instructions that outline specific tasks. This approach has also been applied to the visual domain (Liu et al., 2024b;a; Zhu al., 2023; Brooks et al., 2023; Peng et al., 2023). To improve the performance of LLM to answer math questions, Yu et al. (2023) developed method to rewrite, verify, and augment the original MATH dataset (Hendrycks et al., 2021) using specialized prompts. Similar methodologies have been further explored in other studies, including those by Yuan et al. (2023); Luo et al. (2023); Lee et al. (2023); Yue al. (2023); Tong et al. (2025).
3 STRUCTURED TRACE DROPPING AND RANDOMIZED TRAINING
Our work builds upon the Search former work of Lehnert et al. (2024). To perform planning, train Transformer to model sequence of tokens that sequentially represents the planning task (prompt), the computation of A# algorithm (trace), and the optimal path (solution) derived. See Appendix A to the details of our tokenization scheme and example input for Maze navigation task.
Search forchfoorrer has proven efficacy in addressing a variety of complex decision-making tasks. However, it still suffers from two important limitations. Firstly, the model only operates in slow mode and outputs lengthy reasoning chains, which significantly increases the inference time. While this can be reduced by bootstrapping (Lehnert et al., 2024), an iterative refining technique that consists of cycles of rollouts followed by fine-tuning, such procedure incurs significant extra demand on computational resources. Secondly, Search formmrrrrerr struggles to generate diverse solutions, where identical rollouts are frequently sampled', For example, across 1000 30 30 maze problems we have tested, Search foree's 's reasoning chain contains more than 1500 tokens on average, and can only find 7.6 unique feasible paths out of 64 responses (see Section 4.1.
To address these challenges, we propose a training framework that utilizes randomized reasoning traces. Our approach is inspired by two lines of work. First, have noticed that even though Searchformez is trained on complete A search traces, it generates shorter traces that are sketching the search process. Second, research has shown that humans often rely on shorteuts and patterns when making decisions, a concept known as System thinking (Kahneman, 2017). These observations, combined with the success of dropout technique (Hinton, 2012; Srivastava et al., 2014) that randomly drop units from the neural network during training, motivated us to explore the use of randomized reasoning traces in our framework, and we aim to simplify the A search trace by exploiting its structured elements and selectively dropping certain parts for each training example.
As shown in Figure 3. the A* search trace contains both the create e and the close clauses, and each clause includes the node's coordinates and its (estimated) cost to reach the start and the goal locations. To derive Dual former, we exploit the structure of the search trace and drop certain parts of it for each training example. There are three natural types of dropping:
DI: drop close clause
D2: drop the cost tokens in clause
D3: drop create clause
We note that the lack of diversity is not due to the limitation of training data, For all our experiments. the search traces in the training datasets are generated by non-deterministic * algorithm that randomly breaks cost. ties.
3

---

Published as a conference paper at ICLR 2025
Structured Drogping Strategies
Edtop clow clunsi
14t 1 D1
I/ drop all the cless clauses
create
Close
14t. 01
// drop all the close clauses the COST takens
A seerch
create
0
62 81
(deep cost Sobena
create
51
close
0
c1
exspled Lot dzop create claures
88
Ideop coeate cissre)
1vl - drop the entice cruce
Figure 3.1: An illustration of the structured trace dropping strategies, which consists of four levels. Each level employs a progressively more aggressive dropping approach than the previous onc. For an introduction of trace format, see Appendix A.
Based on them, we develop four levels of dropping strategies, each building upon the previous one. Specifically,
Level
strategy eliminates all the c ose clauses from search trace.
Level
strategy goes further by additionally dropping all the cost tokens.
Level clauses.
strategy is even more aggressive. It further randomly drops 30% of the create
Level strategy drops the entire trace.
Figure 3.1 illustrates our strategies using the previously mentioned Maze task. Intuitively, the Level dropping instructs Dualformer to effectively bypass the close-set computation of A search, the Level 2 dropping promotes Dualforme
to bypass both the close-set and the cost computation. The Level and Level 4 dropping encourage Dual formes r to omit certain or all of the search steps. As we will show in Section 4, these strategies effectively guide Dual former in learning a more concise and efficient search and reasoning process.
To promote diversity in the training data, do not perform dropping data preprocessing step. Instead, at training time, for each training example within a batch, we randomly sample the dropping strategy from categorical distribution Cat (pu, P1, P2,P3, P4), where p1.....PP P4 the probabilities of performing Level 1-4 droppings, and Po is the probability of maintaining a complete trace. This training framework enables Dual 1 fo r to learn from multiple reduced traces even for single training example, as the same example might appear in multiple batches.
Comparison with Token Masking Curious readers might already be wondering that whether our training framework resembles the token masking techniques used by famous LLMs including BERT (Devlin, 2018; Liu, 2019; Song, 2019; Gauthier & Levy, 2019; Sinha et al., 2021; Kitouni et al., 2024). However, there are significant differences that distinguish our approach from those masking techniques. First, standard masking techniques usually mask the input tokens of sequence uniformly in random. In contrast, our dropping strategies only apply to the search trace. Second, while masked LLMs generally employ bidirectional attention layers and predict the masked tokens, Dual 1 former uses causal attention layers, and our training objective solely focuses on the next token prediction with the overall goal of improving its reasoning and planning capability. Computationally. training procedure is also more efficient. Dropping tokens shortens the input sequence and saves computation. For example, it takes 30 hours to train Dual 1 for former r for the 30 x 30 maze task on Tesla V100 32GB GPUs, while it requires 36 hours if the full reasoning trace. We defer the training details to Section 4 and Appendix C.
3.1 CONTROLLABLE GENERATION
One appealing property of Dual former is that it can be casily prompted to operate in either fast or slow generation mode inference time. The control mechanism extremely simple: we append bos and a control token to the standard prompt (which includes environment and task description), where the control token either plan or create. If we use plan, Dualformes will operate in fast mode and directly output the plan, bypassing the reasoning steps. On the other hand, if we inject create after bos, D Dual former will work in slow mode and generate both the reasoning trace and the final plan. See Appendix D for concrete examples. If we only the standard prompt, Dua former z will mimic the dual process of human decision making making-depenane -depending on the situation, it generates either types of responses, which correspond System and System 2 reasoning.
4

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
Our experiments are designed to answer the following questions:
1. Does Dual fo roreer outperform corresponding baselines in fast, slow and auto mode? Does it generate more diverse plans?
2. In slow and auto model, does Dualformes lead to faster reasoning, ie., output a shorter trace?
3. Does the structured trace dropping technique generalize to LLMs trained on natural language datasets?
We answer questions 1 and 2 in Section 4.1, where train Transformers solve Maze navigation tasks and Sokoban games, similar to rearch fo Former (Lehnert et al., 2024). To answer questions 3, we fine-tune LLama-3.1-8B and Mistral-7B models solve math problems in Section 4.2.
4.1 NAVIGATION TASKS: MAZE SOKOBAN
Following Lehnert et al. (2024), we consider the Maze and the Sokoban tasks and use the same dataset. Both datasets contain 10a training examples with complete A search trace. The A implementation is non-deterministic where it breaks cost ties randomly and randomizes the order in which child nodes expanded. The size of Maze varies from 15 x 15 to 30 x 30. For all the Maze tasks, randomly generate 30% 50% percentage of wall cells as obstacles, and randomly sample the goal and start locations. The Sokoban map is of size 7 x 7 with randomly placed two docks, two boxes, and the worker location. We also randomly add two additional wall cells to the interior of the map. For the detailed map generation procedure and example figures, we refer the readers to Appendix C.I.
We first demonstrate that Dualf f former can be explicitly controlled to operate in either fast or slow mode in Section 3.1. It will only output the final plan in fast mode, while reasoning trace will be generated in slow mode. In Section 4. 1.111....2, we compare Dual former with the corresponding baselines in each mode, respectively. variety of metries, as listed below, are used to systematically evaluate the performance, including the correctness, optimality and diversity of generated plans, length of the reasoning trace, etc. Last, we ablate our design choices Section 4.1.
Hyperparameters We instantiate Dual former using the same encoder- architecture as in Lehnert al. (2024). The encoder is an adaptation of the T5 architecture (Raffel et al., 2020) with rotary embeddings. while the decoder is a GPT style architecture. We employ model size of 15M and 46M for the Maze and the Sokoban environments, respectively. All models are trained on 100k training examples. We trained the model for 4 x 10y iterations for the Maze environment and 8 x 10a iterations for the Sokoban environment. We defer the details of model architectures and the other hyperparameters to Appendix C.
We train Dual former using the structured trace dropping strategies described in Section 3. For choosing the dropping strategies for each training example, we sweep 3 sets of probabilities (1)
(po 0.45, p1 = P2 = Pa 1/6, P4 = 0.05). (2) (po = 0.8, p1 = p2 (po 0.7, Pl 0.05, p2 P3 0.1,p4 0.05) and choose the set that yields the lowest validation error. The final choices are (1) for Maze and (3) for Sokoban.
= P4 = 0.05],
Baselines For the fast mode, our baseline is the Solution- Solution-Only model, which uses the same architecture as Dualforme r but trained on sequence data that only include the optimal final solution, without any reasoning trace. The slow mode baseline is the Complete-Tree -Trace model trained on data with complete A search traces. It is referred to as the search augmented model in Lehnert et al. (2024), and is also the base Searchforme roee model without search dynamics bootstrapping. We will also compare Dual 1 Former with bootstrapped models in Section 4.1. All these models have the same amount of parameters, 15M for Maze problems and 46M for Sokoban problems.
Metrics We evaluate whether model generates correct and optimal plans using two metrics: /- Solved-64 and 1-Optimal-6-. Namely, for each evaluation task (e.g. one maze or one Sokoban game), we randomly sample 64 responses from a trained model. Each response is parsed and evaluated regardless of the generated trace part. If any of the 64 plans is correct, i.e. is feasible and reaches the goal location, this task is labelled as success for the 1-Solved-66 64 metric. If any of the 64 plans is optimal, this task is labelled as success for the I 1-Optimal 64 metric. We repeat such process for 1000 unseen evaluation tasks and report the average success rate. To investigate the robustness
5

---

Published as a conference paper at ICLR 2025
of each method, also report metrics 3-Solved-64 and 3- 3-Optimal-6e where task is labelled as success if at least plans are correct or optimal. Additionally, we consider the Success Weighted by Cost (SWC) (Wu et al., 2019) that measures the quality of the resulting plans in terms of their costs, aggregated over individual responses. More precisely: SWC 61 Di=1 I(plan is correct) a. where I is the indicator function, ct is the cost of an optimal plan, and c; is the cost of the i-th plan. Clearly, the higher the SWC score is, the more optimal the resulting plans are. If all the generated plans are optimal, the SWC score reaches its maximum value Last, to quantify the diversity of the generated plans, we check the number of unique correct plans out of the 64 responses for each task, and report the average number across 1000 evaluation tasks.
4.1.1 FAST MODE
Table 4.1 reports the performance of Dual former and the baseline Solution-Only model on Maze and Sokoban tasks, respectively. In terms of generating correct and optimal plans, Dual former significantly outperforms the baseline in both 1-Solved-644 and 1 1-0ptimal-64 criteria. It also notably surpasses the baseline in terms of the 3-Solved-64 and 3-0ptimal-64 rates, which demonstrates the robustness of Duall 1 former in plan generation. In particular, the performance gap increases as the task difficulty increases. For the largest 30 x 30 Maze, the 1-Optimal-64 rate of Dua 1 former is 2.8x that of Solution-Only model, and the 3-Optimal-64 rate is 2.97x. Dual former also achieves much higher SWC score than the baseline, which is above 0.9 for every environment. This demonstrates that each individual plan close to the optimal plan.
generated is of high quality, whose cost is very
Dua 1 former also consistently generates more diverse plans for all the considered problems. In Ap- pendix E, we show one Maze example and plot the unique correct plans generated by Dual 1 former and the baseline model. One interesting observation is that the diversity score of Dual former, ie.. the average number of unique correct plans out of 64 responses, increases as the Maze size goes up. Intuitively, the Maze becomes larger, there are more possible routes to reach single goal location. This suggests that Dualf rmer r learns the Maze structure, while the Solution-Onie -ntlyo model is potentially memorizing the optimal plans its diversity score is close to for all the Maze sizes.
4.1.2 SLOW MODE
Table 4.2 reports the results when Dual former operates in slow mode. The corresponding baseline is the Complete-Trace model, which uses the same architecture and is trained on data with complete A search traces. In addition to the metrics reported before, we report the average length of reasoning traces across the 64 responses, aggregated over all the 1000 evaluation tasks. The results show that Dualformer achieves both enhanced planning power and reasoning speed. It outperforms the Complete-Trace model for all the correctness and optimality metrics: solved rates, optimal rates, and SWC. Moreover, the reasoning trace yielded by Dua 1 former notably shorter than the baseline model. On average, before, Dual former also generates more diverse plans compared with the baseline. We refer the readers to Appendix E for concrete examples.
reduces the trace length by 49.4% across the five tasks. As
Comparison with Search Dynamics Bootstrapping The Complete- -racee model is the base Searchformer model in Lehnert et al. (2024), which has also proposed search dynamics bootstrapping method to enhance its performance on the Sokoban task, similar to those in Anthony model, we fine-tune it
al. (2017); Zelikman al. (2022). After training the Searchforme on a newly created seff-bootstrapped dataset. For each Sokoban game in the original dataset, we generate 32 responses and include the shortest optimal response into the new dataset. We can repeat this process multiple times. In this way, the Searchformer learns to generate shorter responses. Table 4.4 compares Dual former with Searchformer models fine-tuned up steps. Dualforme former is comparable or better than bootstrapped models in most of the metrics, while only using fewer than 45.1% % reasoning steps. We note that each bootstrapping step requires rollouting 3.2 10 total responses and extra fine-tuning of 10a iterations, This means, including the 8 x 10s pretraining iterations, Searchformed step requires a total of 8.3 x 10 training iterations and 9.6 10th rollouts, which is expensive in computation. In comparison, Dual folfforeee only needs a single training stage that consists of x 10 iterations, with no additional rollout requirements.

---

Published as a conference paper at ICLR 2025
Method
1-Optimal-64/ 3-0p4///--Opp6-66
1-Solved-64 3--ooleed6//Solledd---
SWC
Diversity
Dualf fnalformer: (fast)
91.8 887.6
97.1 94.8
0.960
9.05
Maze 15x15
Solution-Only
72.0 68.9
82.7 80.1
0.610
1.52
Dual former (fast)
90.9 / 88.0
97.0 94.0
0.960
17.27
Maze 20x20
Solution-Only
56.3 / 52.0
71.91 / 67.5
0.690
1.52
Dualfo turero (fast) Solution-
83.9 72.9
95.5 90.6
0.940
21.23
Maze 25x25
39.7 / 34.7
60.3 / 55.4
0.570
1,9
Dualformer former (fast)
80.0 / 66.0
91.8 85.7
0.906
18.2
Maze 30x30
Solution-Only
30.0/ 26.0
54.1
0.500
1.86
Dual former (fast)
97.3 / 194.4
94.8 99.0
0.970
4.92
Sokoban
Sulution-Only
86.8 / 83.4
92.8 / 90.0
0.919
1.24
Table 4.1: Evaluation performance of fast mode Dual former. The baseline model is the same architecture trained on the solution only data.
Method
Avg Truce Length
1-Optimal-64/ 3-0ptimul-64
1-Salved-644 / 33Solved--64
SWC
Diversity
Dual Duel_ormer Torner (slow)
278
99.6 99.2
99.9 / 99.9
0.999
12.54
Maze 15 15
Complete- Trace
90.1
96.7
0.964
Dual Mozllooner (slow) Complete-Trice
98.9 97.8
99.77
0.998
18.86
20
851
98.3 95.5
98.8 8 93.00
0.987
14,53
Maze 25 x 25
Dual Sorner (slow)
589
99.9 19.22
99.77 / 999.3
0.997
25.05
Complete-Trace
1208
95.2 85.7
97.0/ 90.4
0.968
18.85
Dual Horner (slow)
854
97.6 19322
99.5 / 998.2
0.993
25.77
Maze 30 30
Complete-Trace
88.1
0.964
7.60
Dualforner (slow)
1482
94.5 [87.6
97.4 [94.]
0.970
4.66
Soloban
Complete-Trace
3600
92.97 84.4
94.7 / /8.00
0.944
2.91
Table 4.2: Evaluation performance of slow mode Dual former. The baseline model is the same architecture trained on the complete-trace data (Searchformer).
Method
Avg Trace Length
1-Opptinald44/3-ppthnnaaaa
1-Solved-64/
2oolvad-64/3-Sored---- 3-Solved-64
SWC
Diversity
(auto)
99.4
/ 99.8
0.999
12.52
Muze 15
Complete/Tmace
495
94.6/ 90.1
96.7 /9.0
0.964
7.60
Solution-Onl
72.0/ 68.9
82.7/80.1
0.610
1.52
Dual forner (auto)
381
99.5 98.6
99.9 / 99.3
0.997
20.28
Cumplete Solution-Only
56.3 52.0
71.9 67.5
0.690
1.52
Dual moliforer (auto)
427
98.6 96.9
99.8 / 199.0
0.998
24.81
Maze 25
Complete-Tracee Solutiot-Only
1208
95, 85.2
97.0/ 90.4
0.968
18.85
39.7
1.9
former (auto)
96.6 992.1
/ 97.77
0.989
24.42
Maze 30
Complete-Trai Solution-Only
1538
93.3 //.44
95.9/
0.964
7.60
30.0/ 26.0
54.1 /4.88
0.500
1.86
Dualforner (autt)
294
94.0 90.0
97.4 / 994.7
0.979
4.97
Sokohan
Complete-Trice
3600
92.9 84.4
94.7 /
0.944
2.91
Solution-Onl Only
86.8 83.4
92.8/ 90.0
0.919
1.24
Table 4.3: Evaluation performance of auto mode Dua 1 former. The baselines are the Solution-Only and Complete-Trace (Searchforme models.
Method
Avg Trace Length
1-0ptimal-64 64/ 3-0ptimal-64
1-Solved-64/ 3-Solved-64
SWC
Diversity
Dualformez (slow)
1482
94.5 94.5/87.6
97.4 /94,1
0.970
4,66
Searchformer Step
3785
94.4/91.2
95.9 92.4
0.957
2.19
Searchformer Step
3507
94.9 91.5
96.7 92.9
0.965
2.27
Searchformet Step 3
3283
94.5 91.4
96.6/ 94.4
0.964
2.48
Table 4.4: Performance for the Sokoban game. Searchformer is fine-tuned up to 3 steps using the search dynamics bootstrapping method.
4.1.3 AUTO MODE: DUAL PROCESS
Instead of controlling the inference mode of Dual former by injecting control token after bos, we can also sample from it directly, allowing it to freely determine the mode of operation, similar to the dual process of human decision making. We call this auto mode for Dual former. Table 4.3

---

Published as a conference paper at ICLR 2025
reported the results. The auto mode Dual rormer also outperforms both the Complete-Tran and Solution-Only model for all the tasks we consider.
An interesting question to ask is whether Dual 1 former can automatically adjust its operation mode in response to problem difficulty. To investigate this, we generated 64 auto-mode responses for 1000 unseen mazes with varying wall densities (between 0.3 and 0.5). We then analyzed the percentage of slow-mode paths among all feasible solutions. The results are plotted in Figure 4.1. As the wall density increases, indicating a (likely) more challenging maze, the proportion of slow-mode paths also increases. This could be due to two factors: (1) the inherent need for slower thinking solve more difficult Figure 4.1: problems, and (2) Dua former is selecting the slow paths (among all the feasible paths it gen- mode more frequently. Similarly, we observe that erates) that Dualforne the maze size increases, where the problem becomes tivates in. As the wall density and maze harder, formr consistently employs more slow size, Dua 1 former engages in slow think- thinking.
The percentage of slow mode
(auto mode) ac-
ing more often.
4.1.4 GENERALIZATION PERFORMANCE
For all the experiments have presented far, test mazes of the same size and wall density. is intriguing to inspect the OOD generalization performance of Dual former. this section, we consider Dualfoss fo rmer trained mazes of size 20 x 20, where wall density is uniform randomly sampled between 0.3 and 0.5. At test time, we vary the wall density from 0.1 to 0.6. Table 4.5 presents the slow mode results. We test 50 unseen examples for each case. As the wall density increases, we expect the maze to become more challenging and the prompt also becomes longer. Therefore,
for in-distribution test cases (wall density 0.3, 0.4, 0.5), Dual I former (slow mode) achieves approximately 100% optimal rate. Yet when the wall density increases to 0.6. which OOD, the performance drops. Interestingly, Dualforme maze for lower wall densities. Our intuition is that the prompt becomes too short for Dua 1 fo rmerr to generalize. In addition to varying wall densities, we also check the slow mode performance of on rectangular mazes (e.g., height-20, width=10). As before, our model is trained on 20 x 20 mazes. Table 4.6 shows the results. Surprisingly, Dualformes generalizes for the cases we consider here.
(slow mode) does not solve a single
Wall Density
1-Optimal-64
1-Solved-64
Maze Size
1-Optimal-64
1-Solved-64
0.1 (OOD)
0.0
0.0
20 x 10
100
100
0.2 (OOD)
0.0
0.0
20 x 12
94
95
0.3
97
100
20 14
100
100
0.4
100
20
73
0.5
100
100
20 x 18
99
100
0.6 (OOD)
68
72
20 x 19
100
100
Table 4,5: Dualformer former (slow mode, trained on
Table 4.6: Dual former (slow mode,
20 20 mazes) achieves nearly-perfec, optimal rate when the wall density is in distribution, while the performance drops for OOD values.
trained on 20 20 mazes) generalizes to rect- angular mazes.
4.1.5 ABLATION STUDY
As discussed in Section 3, the randomized traces for training Dualformes results from different trace dropping strategies, and there are numerous ways combine them. We hereby ablate the design choices we made. First, to enable execution in both fast and slow modes, naïve alternative approach to train Dualformer
using mixture of solution-only and complete- Pi = P2 = F3 0 in our randomization strategy. We refer to such variants as Mix-p models, where p is the fraction of solution-only data in the training dataset. Note that the Solution-Only model essentially a Mix-1 model, and the Complete-Trai Trace model is Mix-0 model. Below, we compare Dual former to Mix-p models for both inference modes. Second, our dropping strategies are
data, i.c.
8

---

Published as a conference paper at ICLR 2025
Method
Avg Trace Length
1-Optimal-64 3.00iimaa-644
1-Sokved-64/
3-Sulved-6d
SWC
Diversity
Dnopping Level
396
99,5798,4
9% 99,9/99.4
0.995
11.54
Maze 15 15
Drupping Level 1-2
134
96.9/98.2
99.7/9011
0.955
11.57
Doopping 2+2+3
99.8
/99.9
DualTormrr (slow)
276
99.6/9992
99,9/99.9
0.999
12.54
Dasping Level]
563
09.1 /96.7
99.9/97.3
0.996
15:79
Maze 20
Dapping Level 1 2-2
549
98.7 /96.7
99.2/98.5
0.991
17.51
Drspping Level 1-2++
542
89.5 (90.9)
100 / |99.7
1.00
19.51
Jualfoprer (sloo)
98.9/97.8
18.56
Dupping
97.9/99.4
99.0/97.1
19.46
Mozu
Dropping Level 1
750
99.2 996.5
[9:88 9991
(099)
23, 18
Drupping Level 1-2+3
98.2/995.0
99.5798.7
0.994
23.1
Tuwl forser (skow)
589
99.0 907.2)
90.77 (99.3
0.997
25.05
Dripping Level
1181
99,5/
95.979974
0.963
22.61
Maze 30x 30
Dupping 1-2
98.01 [98.4
99.6/99.7
Dropping Level 1-2+3
96.8
[98.4
(0.99
24.54
DualIorerr (slow)
554
97.6/99.2
99,5/96.2
0.993
25.77
Doopping Level
3000
94.2 8666
05.67 9019
0,950
3.00
Sokoban
Dropping Level ++2+3
Dripiing Lavel 1-2
2273
16.3
98.3 [9007]
09.9//66.7
90.3 [95.0)
96.57
01970)
0.980
4.39
4.41
Juall former (slow)
94.5787.6
07.4/94.1
0.970
4.66
Table 4.7: Comparison of different combinations of trace randomization strategies.
structured in a hierarchical manner. For instance, Level 2 dropping is developed based on the Level 1 strategy. We investigate how the performance changes when we halt the process at a specific level.
Comparison with Mix-p Models Figure F.1 compares the 1-0ptimal-64 rate of Dua 1 fo rmer and Mix-p models. We test 8 values of p: 0 (equivalent to the Complete- Trace model), 0.05, 0.1, 0.2, 0.4, 0.6, 0.8, and 1.0 (equivalent the Solution-Only model). In both inference modes, Oualforme beats all the Mix-p models for all the five tasks we consider. In fact, Dual former also outperforms Mix-p models the other metrics we consider too, see Appendix F. particular, it is the "fastest" one when operating in slow mode: it generates the shortest reasoning trace.
Combination Randomization Strategies We compare Dual moreforeer with its variants where we halt the dropping strategies at specific levels. For the Maze environments, we fix the probability of using the complete reasoning trace for training example to be po = 0.5 for all the variants and vary the other probabilities. For the Sokoban environments, Po = 0.05. Table G. lists the probabilities of different dropping strategies we use for all the models, and Table 4.7 shows the results. It should be noted that the variants cannot operate in fast mode, since they are not trained on any Level 4 data. Therefore, we only report the slow mode performance. As we increase the strategy level, the length of the reasoning trace decreases. In regard to the other metrics, the performance of Level 1+2+3 model and Dual 1 former r are comparable. However, Dua 1 former enjoys the advantage of shorter reasoning trace and the capability function in fast mode.
fix the probability of Level I dropping be
4.2 APPLICATION TO LLM TRAINING: MATH REASONING
In this section, we show the efficacy of structured trace dropping techniques for training large-scale LLMs to solve math problems. Particularly, we finetune Llama-3-8B and Mistral-7B models using a dataset that contains a variety of math questions and answers with detailed reasoning steps, where we utilize trace dropping technique that leverages the specific structure of the reasoning trace for math problems too. We benchmark the resulting models against corresponding base models finetuned on the dataset directly.
Dataset We evaluate all the models using a dataset named Aug-MATH, which is derived from the MATH dataset (Hendrycks et al., 2021) that contains 7500 training examples of math questions and so- lutions, and 5000 testing examples. Following Yu et al. (2023), we query the Llama-3. 15.1--00--nsmmuee model rewrite the solutions to include more detailed intermediate steps, following given format. To encourage the diversity of reasoning traces, we sample 4 LLM responses for each problem using a temperature of 0.7 and top-p = 0.9. The resulting dataset then contains 30000 training examples and 5000 testing examples. In Appendix H. we show the prompt template that we use for solution rewriting, and a concrete training example before and after rewriting.
Structured Trace Dropping and Randomization The answer of the math questions rewritten by Llama-3.1- 1-110---Insttinn contains 6-10 intermediate reasoning steps on average, where every step might contain multiple sentences. We use randomized training procedure similar to the framework proposed in Section 3. For each the training examples sampled batch, we randomly drop each intermediate reasoning step with probability p. Below, we report the results when varying

---

Published as a conference paper at ICLR 2025
Hyperparameters We finetune two base models Mistral-7B and LLama-3-8B for two epochs using a batch size of 32. We use the AdamW optimizer (Loshchilov & Hutter, 2019) with a learning rate of 8e-6 for the Mistral models, and 5 for the Llama-3 models. Full training details, including how we select the learning rates and other hyperparameters, are deferred to Appendix C.2. For fine-tuning the models, we use the same prompt as in Yu al. (2023), which is displayed in Appendix H.2.
Similar to Dual former, we evaluate the models in both the fast and the slow mode, where the LLM is required output the final solution directly to solve the problem step by step. Following Yu et al. (2023), we use zero-shot prompting when evaluating the models. See Appendix H.2 for the prompts we use in each mode. We consider the Greedy @ metric (Dubey et al., 2024; Yu et al., 2023), where for each question we generate response using a temperature of 0 and verify the correctness. We also report the pass@ 20 metric (Chen al., 2021), where we randomly sample 20 responses using temperature of 0.5. For reference, we also report the results of fine-tuning those models on the original MATH dataset.
Evaluation
Results The results are presented in Table 4.8. We test four values of p: 0.1, 0.3 and 0.4. Our results show that the proposed training strategy makes both LLMs more effective and efficient. We first inspect the results of the Mistral-7B models. For the slow mode inference, fine-tuning the model with trace dropping and randomized training improves upon the baseline model that is directly finetuned on Aug-MAIl dataset. The absolute Greedy @ I metric improved by 1.7% when p = 0.1 (which amounts to 10% relative performance improvement), and 0.9% with = 0.2 and 0.3, and 0.1 1% when p 0.4. Our models are also outperforming the baseline model for the Pass @ 20 metric when evaluation schemes, the average length of the reasoning trace goes down as p goes up. Similarly, for inference in the fast mode, our models also achieve higher correct rate. A similar trend of performance improvement also holds for the Llama-3-8B models. Finally, for reader's 's reference, we include the results of both the Mistral-7B and the Llama-3-800 8B models fine-tuned on the original MATH dataset, which clearly lags behind?
0.1, 0.2 and 0.3, where the absolute correct rate increases to 61.9%. Under both
Model
Dataset & Dropping Prob
Greedy @1(%) (slow / fast)
Trace Length
Pass 20X 5) (slow / [ast)
Trace Length
Aug- ISRT (baseline)
16.9
527/-
29.6/
521
AUT-MATH (p-0.1)
18.6 11.3
61.67 32.0
479/-
Mistral-7B
Aug- (p-0.2)
17.8
11.2
477 /-
61.4/3199
470
Aug-MATH (p=0.3)
17.8/ 11.8
497
61.9 /31.7
166/-
Aug-MeTll (pm(.44
17,0/
434/-
56.4/ 28.9
397
MATE
13.1/8.5
290/-
53.0/ 29.4
Aug-MATTH (baseline)
19.7/3.11
S48
62.7 35.6
535/-
Aug-MeTe METH (p=0.1)
13.3
544/-
63.4/ 36,2
522
3-88
MMIT (p-0.2)
20. 138
525
63.9 / 36.7
497 1-
A (p=0.3)
20.5 13.5
515/-
474/-
A (p=0.4)
20.1/
490/-
63.1/37.2 52.8/33.5
450
13.3/12.6
432/-
332/-
Table 4.8: The performance of Llama-3 and Mistral models finetuned to solve math problems.
CONCLUSION
We present a simple and easy-to-implement framework for training Transformers to solve reasoning and planning tasks. We carefully probe the structure of the reasoning traces and design corresponding dropping strategies that imitate the shortcuts in human thinking process. By randomly applying the dropping strategies to training examples, the resulting model, Dualformer, can be controlled to execute in either fast or slow reasoning mode, or in auto mode where it decides the mode to engage. Dual former r achieves enhanced performance for the maze navigation tasks and the Sokoban game, while reducing the number of reasoning steps required. Remarkably, training task-specific models from scratch. We apply techniques in the same spirit to fine-tune LLMs to answer math questions and obtain improved performance. Last, the proposed framework also reduces computation consumption as the input sequences are shortened after trace dropping. Future work could investigate whether our approach helps models scale, and explore methodologies such as curriculum learning and hierarchical planning to adapt Dual I former for more complex tasks.
approach is not limited to
"The trace generated by models trained MA is much shorter, because the answers in MATH do not. contain as many intermediate steps as those in Aug-MATH.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as can, not i say: 2022.
Grounding language in robotic affordances. arXiv preprint arXiv:2204.0106
Thomas Anthony, Zheng Tian, and David Barber. Thinking fast and slow with deep learning and tree search. Advances in neural information processing systems, 30, 2017.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for sulf-supe learning of speech representations. Advances in neural information processing systems, 33: 13:1:4:: 2249-1246600 2020.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 17682-17990, 2024.
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18392-18402, June 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et Evaluating large language models trained code. arXiv preprint arXiv:2107.03374 2021.
Weizhe Chen, Sven Koenig, and Bistra Dilkina. Why solving multi-agent path finding with large language model has not succeeded yet. arXiv preprint arXiv:2401.03630, 2024.
Yuntian Deng, Yejin Choi, and Stuart Shieber. From explicit cot to implicit cot: Learning to internalize cot step by step. arXiv preprint arXiv:2405.14888, 2024.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 181000080, 2018.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 /929, 2020.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 1783, 2024.
Kanishk Gandhi, Denise Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah D Goodman. Stream of search (sos): Learning to search in language. arXiv preprint arXiv:2404.0300 036633 2024.
Jon Gauthier and Roger Levy, Linking artificial and human neural representations of language. arXiv preprint 1990.0144,, 2019.
Atharva Gundawar, Mudit Verma, Lin Guan, Karthik Valmeekam, Siddhant Bhambri, and Subbarao Kambhampati. Robust planning with Ilm-modulo framework: Case study travel planning. arXiv preprint arXiv:2405.2000 20625, 2024.
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model planning with world model. In The 2023 Conference on Empirical Methods in Natural Language Processing.
Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings. Advances in neural information processing systems, 36, 2024.
Joy He- Yueya, Gabriel Poesia, Rose E. Wang, and Noah D. Goodman. Solving math word problems by combining language models with symbolic solvers, 2023. URL https:/ / /wwxiv.vvoo org/ 09102.
11

---

Published as a conference paper at ICLR 2025
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021.
GE Hinton. Improving neural networks by preventing co-adaptation of feature detectors. arXiv preprint arXiv: 2012.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:3451-348 2021.
Daniel Kahneman. Thinking, fast and slow. 2017.
Ouail Kitouni, Niklas Nolte, Diane Bouchacourt, Adina Williams, Mike Rabbat, and Mark Ibrahim. The factorization curse: Which tokens you predict underlie the reversal curse and more. arXiv preprint ar M4X1::2406.01183,, 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199-22213, 2022.
Ariel N Lee, Cole Hunter, and Nataniel Ruiz. Platypus: Quick, cheap, and powerful refinement of 1lms. arXiv preprint arXiv:2308.07311 2023.
Lucas Lehnert, Sainbayar Sukhbaatar, DiJia Su, Qinqing Zheng, Paul Mevay, Michael Rabbat, and Yuandong Tian. Beyond a*: Better planning with transformers via search dynamics bootstrapping. COLM, 2024.
Bill Yuchen Lin, Yicheng Fu, Karina Yang, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Prithviraj Ammanabrolu, Yejin Choi, and Xiang Ren. Swiftsage: generative agent with fast and slow thinking for complex interactive tasks. Advances in Neural Information Processing Systems, 36:23813 22-222 2023.
Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Lim+p: Empowering large language models with optimal planning proficiency, 2023. URL
https: /ttp:////axivv. abs/ 2304 11477.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 26296-26306, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in neural information processing systems, 36, 2024b.
Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 2019.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: designing data and methods for effective instruction tuning. In Proceedings of the 40th International Conference Machine Learning, pp. 22631-22648, 2023.
Ilya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with restarts. CoRR, wwwpeiivww.rrr.eoooooooooooo abs/ 03983.
abs/1608.03998 2016. URL
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. ICLR, 2019.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint axXiv:2388.09583 2023.
12

---

Published as a conference paper at ICLR 2025
Vishal Pallagani, Bharath Muppasani, Keerthiram Murugesan, Francesca Rossi, Lior Horesh, Biplav Srivastava, Francesco Fabiano, and Andrea Loreggia. Plansformer: Generating symbolic plans using transformers, 2022.
Vishal Pallagani, Bharath Chandra Muppasani, Kaushik Roy, Francesco Fabiano, Andrea Loreggia, Keerthiram Murugesan, Biplav Srivastava, Francesca Rossi, Lior Horesh, and Amit Sheth. On the prospects of incorporating large language models (Ilms) in automated planning and scheduling (aps). In Proceedings of the International Conference on Automated Planning and Scheduling. volume 34, pp. 432 444, 2024.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, 2023. URL https: /ttp:://aaxiii org/ /ass /304 03277.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 1148-8763. PMLR, 18-24 Jul 2021. URL https: /ttps://prrceee pttpp/eccceeeeesececcepe/// mall.presse / 21399//aaford2/22 html.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL
http://jmlr wwtp://.r.ppppppppeeaeee http://aapr///11//0// /v21/2022222-/////// 074
Stuart J. Russell and Peter Norvig. Artificial Intelligence: A Modern Approach. Pearson Education, edition, 2021.
Swarnadeep Saha, Archiki Prasad, Justin Chih-Yao Chen, Peter Hase, Elias Stengel-Eskin, and Mohit Bansal. System-1. I. x: Learning to balance fast and slow planning with language models. arXiv preprint arXiv.2407.14414 2024.
Timo Schick, Jane Dwivedi-Yo Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Thirty-seventh Conference on Neural Information Processing Systems.
Bilgehan Sel, Ahmad Al-Tawaha, Vanshaj Khattar, Ruoxi Jia, and Ming Jin. Algorithm of thoughts: Enhancing exploration of ideas in large language models. arXiv preprint arXiv:2308. 10379, 2023.
Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. Distilling reasoning capabilities into smaller language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 7059-7073, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. (0.1833/v1/2023 441. URL https: aaclllllleee wallanooooooyyy org/ 2023..222222222.... findi ngs acl 441.
Tom Silver, Soham Dan. Kavitha Srinivas, Joshua B. Tenenbaum, Leslic Kaelbling, and Michael Katz. Generalized planning in pádl domains with pretrained large language models. Proceedings of the AAAI Conference on Artificial Intelligence, 38(18):20256-00) 20264, Mar. 2024. doi: 10. 1609/aaai. w38i18. 30000. URL https: //t/s///s/....... aaai orr. index,ndxxxx.///// php/ AAAI/artiinn view/3000
Koustuv Sinha, Robin Jia, Dieuwke Hupkes, Joelle Pineau, Adina Williams, and Douwe Kiela. Masked language modeling and the distributional hypothesis: Order word matters pre-training for little. arXiv preprint arXiv:2104. 06644, 2021.
Kaitao Song. Mass: Masked sequence to sequence pre-training for language generation. arXiv preprint arXiv: 100.00055502 2019.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199 2014.
13

---

Published as a conference paper at ICLR 2025
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsumori B. Hashimoto. Stanford alpaca: An enstructionfollooing llama model. https: / //gittutb//b/ www//tttouitteecttteeecce tatsuaaaa/bbaaaaaa wwtanfordaaapaeeeeeeeeaaeeeeeeae 2023.
Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. Dart-math: Difficulty-aware
rejection tuning for mathematical problem-solving. Advances in Neural Information Processing Systems, 37:7821-7846, 2025.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention. International conference on machine learning, pp. 10347-10357. PMLR, 2021.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cris- tian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnie, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models a critical investigation, 2023a.
Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Large language models still can't plan (a benchmark for iims on planning and reasoning about change), 2023b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. CoRR, (ebs//706.0366,, 2017. URL http:// / rxiv. atgg/abs/700/ 03762.
Peifeng Wang, Zhengyang Wang, Zheng Li, Yifan Gao, Bing Yin, and Xiang Ren. SCOTT: Self-consistent ofain-of-thought distillation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5546-5558, Toronto, Canada, July 2023. Asso- ciation for Computational Linguistics. doi: 10. 10.18653/v1/2223.aall acl-long.334. URL https:
//aclanthologe org/ 2023/223...2 acl-long 304.
Xuezhi Wang and Denny Zhou. Chain-of-thought reasoning without prompting. arXiv preprint arXiv:2402.10200, 2024.
Peter C Wason and J St BT Evans. Dual processes in reasoning? Cognition, 3(2):141-144, 1974.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652. 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thowe thottttugtt prompting elicits reasoning in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 24824 24837. Curran Associates, Inc., 2022.
Jason Weston and Sainbayar Sukhbaatar. System 2 attention (is something you might need too).
arXiv preprint arXiv:2311.1188 2023.
Yi WU, Yuxin Wu, Aviv Tamar, Stuart Russell, Georgia Gkioxari, and Yuandong Tian. Bayesian
relational memory for semantic visual navigation, 2019.
14

---

Published as a conference paper at ICLR 2025
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint 1rXXi::330911224., 2023.
Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. Distilling system 2 into system 1. arXiv preprint arXiv:2407.06023, 2024.
Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.0188 01825, 2023.
Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang. Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.0565: 056533, 2023.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 15476-15488. Curran Asso- ciates, Inc., 2022. URL
neurips ec paper htlplapaaaaeaa//aelalallaeeaeeee slles/paper/ coneeceen ppf.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205. 10625, 2022.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing linion-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
15

---

Published as a conference paper at ICLR 2025
Orey
Figure B.1: The left panel plots a path generated by fast-thinking, which is suboptimal. The right panel plots optimal path generated by tlow-thnnking. Cells highlighted in red the positions visited by the reasoning trace of slow-thinking mode. Green circle shows critical cell that needs to identified for optimal path yet missed by the fast-thinking.
A PRELIMINARIES
Our work builds upon the work of Lehnert et al. (2024). To perform planning, we train Transformer to model a sequence of tokens that sequentially represents the planning task, the computation of A algorithm, and the optimal solution derived from the A search. The tokenization method is illustrated in Figure A.1. As toy example, consider navigation task in 3 x Maze where the goal is to find one shortest path from the start cell to goal cell, without hitting a wall cell. The A algorithm has successfully determined optimal plan. We both the task and the Maze structure, which also serves as the prompt for Dual forme.. The solution is depicted by the plan token sequence that describes the path using coordinates. The A* algorithm generates search trace sequence that records the search dynamics performed, displayed in Figure 3.1. Recall that the A algorithm is pathfinding algorithm on a weighted graph. The create clause adds the node (represented by the subsequent coordinates) into the search frontier, ose,
sequence of tokens express
and the c ose clause adds the node to the closed set. Each clause, either create followed by the tokens y, cO, and c1, representing the node's coordinates, cost-since-start value, and heuristic values, respectively. For details of A and the tokenization approach, we refer the readers to Russell & Norvig (2021) and Lehnert et al. (2024), respectively.
bos
plan
0 2
wall cell
start 2
plan
0
start cell
Prompt
goal 1
plan
0
gnal cell
wall 1 2
plan 1 0
wall 2
0
2
plan step
eos
Figure A.I: An example Maze navigation task and one optimal plan obtained by the A* algorithm. The task and the plan are expressed as a prompt token sequence and plan token sequence.
B COMPARISON OF FAST THINKING AND SLOW THINKING
C NETWORK ARCHITECTURE AND HYPERPARAMETERS
We use the same encoder-decoder Transformer architecture as in Lehnert al. (2024) for Dua forner. first converts each token into a one-hot vector, which is then transformed into set of vectors through the embedding layer. The embedding vectors then go through the subsequent layers shown Lehnert et al. (2024, Figure 4). We use RoPE embeddings for positional encoding, and dropout is used in our architecture.
For the model size, architecture parameters, batch size, we use the same setup as as (Lehnert et al., 2024). Specifically, for the Maze tasks, heads and layers with hidden size 64. We optimize the model by the Adam W (Loshchilov & Hutter, 2019) optimizer with learning rate 2.5e-4 and batch size 16, where Bn and B1 set 0.9 and 0.99,
15M A.M-aaraeeter model which consists of 3 attention
16

---

Published as a conference paper at ICLR 2025
respectively. A linear warm-up strategy employed for the first 2000 gradient updates. Afterward, we use the cosine learning rate scheduler (Loshchilov Hutter, 2016).
For the Sokoban tasks, with hidden size 96. We use batch size and learning rate 7.5e-5, while the other hyperparameters the same above.
46M--60rameee model which consists of 4 attention heads and layers
C.I ENVIRONMENT & DATASET
We used the same dataset as Lehnert al. (2024), which available https:///itt com/ wacebookreseaze wearchhhaacccfommeee The Maze dataset contains 10M examples and the Sokoban dataset contains 10M examples and we use the first 100k example sorted in accordance to their "id" field in mongodb (following same approach as (Lehnert et al., 2024)). For reader's reference, the dataset generated as follows. For the Maze tasks, 30-50% of the cells were first randomly designated as walls. Next, start and goal location were chosen randomly, Then, the A" algorithm was applied to generate an optimal plan. For the Sokoban tasks, we use 7 x 7 grid map where two wall cells worker locations were placed randomly. Once a game is generated, it is only added to the dataset if could be solved by the A algorithm.
randomly inserted obstacles. Moreover, two docks, two boxes, and two
Figure C.I: Map of an example Sokoban task in a 7 x 7 grid world, with two boxes and two docks (figure taken from Lehnert al. (2024)).The The worker needs push (and not pull) the boxes from the start locations to the destinations.
bas
bos
plan 2
worker
create worker box box 4 c0
c3
Trace (2583 tokens)
box 4
plan
bex 4
plan 1
dock
3
create worker 5 box
e10 c3
p' 2
Prompt
Jock
close worker
e12
plan
plan
wall
0
Plan
pian 2
wall 1
plan 2
plan
plan
8os
plan 1
plan 2
eos
Figure C.2: Example prompt and response token sequences for the Sokoban task depicted in
Figure C.I (example taken from (Lehnert et al., 2024)).
C.2 MATH REASONING
We
the implementation provided https::/ggtthb..come -lama-recipen for fine-tuning the models.
We train all the models for 2 epochs, using batch size of 32. We use the AdamW optimizer with learning rate of 5 x 10 for the Llama model and x 10 for the Mistral models. The learning
rate is selected as follows. We sweep over values 2 x 10-6 5 10-6 28 10-6 and choose the
17

---

Published as a conference paper at ICLR 2025
learning rate that yields the lowest validation loss. We then retrain the models using the selected learning rates on the full training dataset and report the results. We use the default values for the other do not use linear rate warmup. weight decay, multistep
hyperparameters. More specifically, gradient accumulation. We use betas=(0.9, 0.999). eps=le - 8, ? 0.85 (multiplicative step-wise learning rate decay) for Adam W and "packing" as the batching strategy.
18

---

Published as a conference paper at ICLR 2025
D CONTROLLABLE GENERATION OF DUALFORMER
Dua former r offers flexible output options for users to choose. In this section, we demonstrate this by example navigation task in 15 x 15 maze, see Figure D.I. The start location is (9, 10) and the goal location (3, 6). The location coordinates and the maze structure is encoded in the original prompt below.
Original Prompt
bos, start, 9, 10, goal, 3, 6, wall, 0, 0, wall, 4, 0, wall,
7, 0, wall, 10, 0, wall, 12, 0, wall, 13, 0, wall, 3, 1, wall, 7, 1, wall, 11, 1, wall, 12, 1, wall, 13, 1, wa wall, 14, 1, wall, 0, 2, wall, 3, 2, wall, 4, 2, wall, 6, 2, wall, 7,
2, wall, 8, 2, wall, 10, 2, wall, 11, 2, wall, 14, 2, wall,
1, 3, wall, 2, 3, wall, 3, 3, wall, 11, 3, wall, 13, 3, wall,
2, 4, wall, 8, 4, wall, 10, 4, wall, 11, 4, wall, 12, 4,
wall, 14, 4, wall, 3, 5, wall, 4, 5, wall, 5, 5, wall, 7, 5, wall, 9, 5, wall, 11,
wall, 12, 5, wall, 14, 5, wall, 0,
6, wall, 4, 6, wall, 6, 6, wall, 8, 6, wall, 1, 9, 6, wall, 14,
6, wall, 2, 7, wall, 4, 7, wall, 7,
wall, 9, 7, wall, 10,
7, wall, 13, 7, wall, 2, 8, wall, 3, 8, wall, 5, 8, wall, 6,
8, wall, 8, 8, wall, 10, 8, wall, 11, 8, wall, 12, 8, wall,
1, 9, wall, 5, 9, wall, 6, 9, wall, 9, 9, wall, 11, 9, wall,
14, 9, wall, 10, 10, wall, 12, 10, wall, 13, 10, wall, 14,
10, wall, 1, 11, wall, 8, 11, wall, 9, 11, wall, 12, 11,
wall, 3, 12, wall, 5, 12, wall, 6, 12, wall, 8, 12, wall,
10, 12, wall, 12, 12, wall, 14, 12, wall, 0, 13, wall, 1, 13,
wall, 3, 13, wall, 8, 13, wall, 12, 13, wall, 13, 13, wall, 14, 13, wall, 0, 14, wall, 1, 14, wall, 2, 14, wall, 6, 14, wall, 14, 14, e0s
To configure Dual 1 former's operation mode, we only need to append bos and control token to the original prompt. To output solution only (fast mode), we inject p plan. On the other hand, we inject create to let Dual former output both trace and solution.
D.1
FAST MODE
The modified prompt for fast mode displayed below. The extra tokens are highlighted in purple.
19

---

Published as a conference paper at ICLR 2025
Fast Mode Prompt (two extra tokens added to the original)
bos, start, 9, 10, goal, 3, 6, wall, 0, 0, wall, 4, 0, wall, 7, 0, wall, 10, 0, wall, 12, 0, wall, 13, 0, wall, 3, 1,
wall, 7, 1, wall, 11, 1, wall, 12, 1, wall, 13, 1, wall, 14,
1, wall, C, 2, wall, 3, 2, wall, 4, 2, wall, 6, 2, wall, 7,
2, wall, 8, 2, wall, 10, 2, wall, 11, 2, wall, 14, 2, wall,
1, 3, wall, 2, 3, wall, 3, 3, wall, 11, 3, wall, 13, 3, wall,
2, 4, wall, 8, 4, wall, 10, 4, wall, 11, 4, wall, 12, 4,
wall, 14, 4, wall, 3, 5, wall, 4, 5, wall, 5, 5, wall, 7, 5, wall, 9, 5, wall, 11, 5, wall, 12, 5, wall, 14, 5, wall, 0,
wall, 9, 6, wall, 14, 6, wall, 2, 7, wall, 4, 7, wall, 7, 7, wall, 9, 7, wall, 10,
6, wall, 4, 6, wall, 6, 6, wall, 8,
7, wall, 13, 7, wall, 2, 8, wall, 3,
wall, 5,
wall, 6,
8, wall, 8, 8, wall, 10, 8, wall, 11, 8, wall, 12, 8, wall,
1, 9, wall, 5, 9, wall, 6, 9, wall, 9, 9, wall, 11, 9, wall,
14, 9, wall, 10, 10, wall, 12, 10, wall, 13, 10, wall, 14,
10, wall, 1, 11, wall, 8, 11, wall, 9, 11, wall, 12, 11,
wall, 3, 12, wall, , 5, 12, wall, 6, 12, wall, 8, 12, wall,
10, 12, wall, 12, 12, wall, 14, 12, wall, 0, 13, wall, 1, 13,
wall, 3, 13, wall, 8, 13, wall, 12, 13, wall, 13, 13, wall,
14, 13, wall, 0, 14, wall, 1, 14, wall, 2, 14, wall, 6, 14, wall, 14, 14, ecs, bos, plan
The following box shows the generated tokens, and Figure D. plot the corresponding path.
Fast Mode Output
9, 10, plan, 8, 10, plan, 7, 10, plan, 10, plan, 5, 10, plan, 4, 10, plan, 3, 10, plan, 2, 10, plan, 1, 10, plan, 0, 10, plan, 0, 9, plan, 0, 8, plan, o, 7, plan, 1, 7, plan, 1,
6, plan, 2, 6, plan, 3, 6, eos
(a) Dual Dualformer former operates in fast-mode, directly outputting the final plan.
(b) Dual former operates in slow-mode, gener- ating search traces hefore the final solution.
Figure D.I: Controllable generation of former...r. Paths correspond to the output displayed Appendix D. The wall cells are depicted in dark. while the unoccupied cells are shown gray. The starting point is marked in green, and the destination point is in red. The final path determined by the trained model highlighted in yellow. When Dual former operates in slow mode, the corresponding cells explored by the search trace are highlighted in red, visually representing each step of the search process. The color intensity indicates the total cost value.
20

---

Published as a conference paper at ICLR 2025
D.2 SLOW MODE
Similarly, present the modified prompt for slow mode below. Comparing with the fast mode prompt, the only change the control token becomes create.
Slow Mode Prompt (two extra tokens added to the original)
bos, start, 9, 10, goal, 3, 6, wall, 11, 0, 0, wall, 4, 0, wall, 7, 0, wall, 10, 0, wall, 12, 0, wall, 13, 0, wall, 3, 1,
wall, 7, 1, wall, 11, 1, wall, 12, 1, wall, 13, 1, wall, 14, 1, wall, C, 2, wall, 3, 2, wall, 4, 2, wall, 6, 2, wall, 7,
2, wall, 8, 2, wall, 10, 2, wall, 11, 2, wall, 14, 2, wall, 1, 3, wall, 2, 3, wall, 3, 3, wall, 11, 3, wall, 13, 3, wall,
2, 4, wall, 8, 4, wall, 10, 4, wall, 11, 4, wall, 12, 4,
wall, 14, 4, wall, 3, 5, wall, 4, 5, wall, 5, 5, wall, 7, 5, wall, 9, 5, wall, 11, 5, wall, 12, 5, wall, 14, 5, wall wall, 0,
6, wall, 4, 6, wall, 6, 6, wall, 8, 6, wall, 9, 6, wall, 14,
6, wall, 2, 7, wall, 4, 7, wall, 7, 7, wall, 9, 7, wall, 10,
7, wall, 13, 7, wall, 2, 8, wall, 3, 8, wall, 5, 8, wall, 1, 6, 8, wall, 8, 8, wall, 10, 8, wall, 11, 8, wall, 12, 8, wall,
1, 9, wall, 5, 9, wall, 6, 9, wall, 9, 9, wall, 11, 9, wall,
14, 9, wall, 10, 10, wall, 12, 10, wall, 13, 10, wall, 14,
10, wall, 1, 11, wall, 8, 11, wall, 9, 11, wall, 12, 11,
wall, 3, 12, wall, 5, 12, wall, 5, 12, wall, 8, 12, wall, 10, 12, wall, 12, 12, wall, 14, 12, wall, 0, 13, wall, 1, 13, wall, 3, 13, wall, 8, 13, wall, 12, 13, wall, 13, 13, wall, 14, 13, wall, 0, 14, wall, 1, 14, wall, 2, 14, wall, 6, 14, wall, 14, 14, ecs, bos, create
The following box shows the generated output, and Figure D. plots the corresponding trace and final path.
Slow Mode Output
9, 10, c0, c10, create, 8, 10, cl, c9, create, 8, 9, c2, c8, create, 7, 10, c2, c8, create, 7, 9, c3, c7, create, 7, 8, c4, c6, create, 6, 10, e3, c7, create, 7, 11, e3, c9, create, 5, 10, c4, c6, create, 6, 11, c4, c8, create, 5, 11, c5, c7, create, 4, 10, c5, c5, create, 4, 11, c6, c6, create, 3, 10, c6, c4, create, 4, 9, c6, c4, create, 3, 9, c7, c3, create, 4, 8, 27, c3, create, 2, 10, c7, c5, create, 3, 11, c7, c5, create, 2, 9, c8, ct, create, 2, 11, c8, c6, create, 1, 10, c8, c6, create, 4, 12, c7, c7, create, 7, 12, c4, c10, create, 7, 13, c5, cll, create, 0, 10, c9, c7, create, 4, 13, c8, ca, create, 2, 12, cs, c7, create, 7, 14, c6, c12, create, 6, 13, c6, el0, create, 0, 9, c10, c6, create, 0, 11, c10, c8, create, 0, 8, ell, c5, create, 1, 12, c10, co, create, 2, 13, cl0, c8, create, 4, 14, c9, c9, create, 5, 13, c9, c9, create, 0, 7, c12, c4, create, 1, 8, cl2, c4, create, 1, 7, c13, c3, create, 5, 13, c7, cs, create, 1, 6, c14, c2, create, 5, 14, c8, cl0, create, 1, 5, c15, c3, create, 2, 6, c15, cl, create, 2, 5, c16, c2, create, 3, 6, c16, cO, plan, 9, 10, plan, 8, 10, plan, 7, 10, plan, 6, 10, plan, 5, 10, plan, 4, 10, plan, 3, 10, plan, 2, 10, plan, 1, 10, plan, 0, 10, plan, 0, 9, plan, 0, 8, plan, 0, 7, plan, 1, 7, plan, 1, 6, plan, 2, 6, plan, 3, 6, eos
21

---

Published as a conference paper at ICLR 2025
E DIVERSITY OF GENERATED PLANS
The Dual I fo rmer outperforms baseline models in discovering unique feasible solutions. To illus- trate this visually, select one example maze task and generate 64 responses using Dormrrrmer in fast mode. Figure E. plots all the unique feasible paths discovered by fast mode DualE forlformer alongside those found by the Polution-Only baseline (64 responses). Dual former (fast mode) identified 42 unique feasible paths, while the Solution-Onle model only found 3. Similarly, Fig- E.2 compares slow mode Dualformer and the Complete- Trace Trace (Searchformer) baseline. Dual 1 former r (slow mode) discovered 39 unique feasible paths, whereas the Complete- Trace model only found 17.
22

---

Published as a conference paper at ICLR 2025
(a) Unique fcasible paths found by the Solution-Only Model
Cost
Explored Step
(b) Unique feasible paths found by Dual former (fast mode)
Figure E.1: Fast mode Dualfouuooooooo hoalformer finds more feasible paths than the Solution-Only model. wall cells are depicted dark, while unoccupied cells are shown gray. The starting point is marked in green, and the destination point is in red. The final path determined by the trained model is highlighted in yellow.
23

---

Published as a conference paper at ICLR 2025
(a) Unique feasible paths found by the Complete- Trace Model (Searchformer)
Enioty
Exstored Slep
(b) Unique feasible paths found by Dual former r (slow mode)
Figure E.2: Slow mode Dual 1 former finds more feasible paths than the Complete- Trace model Seearchformer The wall cells are depicted in dark, while the unoccupied cells are shown in gray. The starting point is marked in green, and the destination point is in red. The final path determined by trained model is highlighted yellow. The cells explored by the search trace are highlighted in red, where the color intensity indicates the total cost value. If the generated search trace does not contain cost values (used Level 2, 3, dropping, Section 3), the explored cells are indicated in blue.
24

---

Published as a conference paper at ICLR 2025
F COMPARISON WITH MIX-p MODELS
(a) Slow mode comparison with Mix-p models.
23:2
2lo25
2010
(b) Fast mode comparison with Mix-p models.
Figure F.1: The 1-0ptimal-64 rate of Dual former and Mix-p models with varying values of p. where p is the fraction solution-only data in the corresponding training dataset. The top and bottom panels plot the results in fast and slow mode, respectively. Dual formerr outperforms all the Mix-p models in both inference modes. The probabilities of different dropping strategies used for Dual 1 fo rmer is described in the Hyperparameter paragraph, see Section 4.1.
25

---

Published as a conference paper at ICLR 2025
Method
Avg Trace Length
1-0ptimal-64//--0ttnnnad 5-0ptimma-64
1-Sehre-6445--000
SWC
Diversity
Complete- -eopleto-Trace
495
94.6/90.1
96.7 /9300
0.964
7.60
Dual Tormer (slow)
278
99.6799.2
12:54
Mix-5%
506
97.1/94.8
97.6/
0.98
10.25
Mix-10%
542
99.0/97.4
99.4/98.7
Maze 15 x
Mix-20%
525
99.1797.7
99.7/99.
1.00
12.10
Mix
556
98.4/996
99.5 988.8
11.77
96.7
10.75
Mix-60%
577
88.1 85.6 85.6
94.2 92.
0.93
6.48
Complete- Trace
851
98.3/955
98.8 18.8/931 93.0
0,987
14.53
Dua formar (slow)
439
/97.8
99.9/99.7
0.998
18.86
Mix-5%
789
98.67 98.1
99.5 99.
0.99
17.71
Mix-10%
866
98.9/99
99.8 99.3
1.00
18.52
Muze
Mix-10%
900
93.5 90.4
97.4 95.1
0.97
16.51
Mix-50%
905
89.1
96.3
16.30
Mix-60%
891
72.6/ 67.9
86.5 82.7
0.84
3.92
Mix-80%
943
62.4 / 56.4
80.9 75.4
0.77
2.62
Complete-Trace
1208
95.27
97.0/ 90.4
0.968
18.85
Diaa former (slow)
589
99.9 197.2
99.7 / 99.3
0.997
25.05
99.0/97.2
Mix-10%
1264
98.7 1963
99.5 98.0
0.99
22.33
Maze 25 25
Mix-20%
1283
97.9/95.0
99.1 /97.9
23.68
Mix-40%
1278
90.8/87.4
96.3/933 93.9
0.96
22.30
Mix-50%
1334
81.0/75.5
93.3 88.9
0.91
18.02
Mix-80%
1501
$7.2 50,3
78.5 70.3
0.75
3.76
Complete-Trae
Dua formar (slow)
854
97.67933.
99.5 98.2
0.993
25.77
Mix-5%
2022
96.3/92.0
98.4/96.
24.50
Mix-10%
1720
99.7
99.2/ 99.22
27.17
Maze 30 x 30
Mix-20%
1851
95.0/91.4
98.3 / 96.2
0.98
25,02
Mix-40%
64.3/56.2
83.1
Mix-50%
1652
$8.0 50.8
78.6/ 71.9
0.76
9.66
Mix-60%
1983
43.0/ 35.4
68.4 58.1
0.65
3.21
Mix-80%
1648
38.4
71.9 61.9
3.98
Complete-Trace
3600
92.9/ 84.4
94.7 / 89.0
0.944
2.91
Dua former (slow)
Mix-5%
3278
85.3 72.7
91.0/ 80.9
(1.90
3.18
Mix-10%
3402
88.6/77.2
94.1 /7.9
4.07
Sokoban
Mix-20%
3331
89.0 81.3
95.7 89.1
0.95
4.22
Mix-40%
3294
92.7 86,
97.1/93.1
4.14
Mix-60%
2594
91.17 83.2
96.4/91.0
0.96
4.48
Table F.1: Comparison of Dualformer and Mix-p models in the slow mode.
26

---

Published as a conference paper at ICLR 2025
Method
1-0ptimal-64 Batppmaaal-a3/-t-iiiiiii 3-0ptimal-64
3eabbved-64//Solveeeee
SWC
Diversity
Dua 1 Former (fast)
87.6
97.1 / 94.8
0.960
9.05
Mix-5%
89.8/ 83.2 83.2
92.7/ 89.0
0,92
11.72
Mix-10%
80.3 75.6
90.5 / 86.9
0.88
7.01
Maze 15x15
Mix-20%
7
87.8 86.4
0.86
1.97
Mix-40%
81.2/78.8
88.6 86.9
0.87
1.73
Mix-50%
79.0/ 76.4
87.7 85.8
0.86
Mix-60%
81.3/ 78.0
89.3 / 87.3
0.88
1.95
Mix-80%
79.8 76.6
88.6 86.0
0.87
2.12
Solution-Only
72.0 68.9
82.7 / 80.1
0.80
1.52
Dua Former (fast)
90.9/ 84,0
97.0/ 94.0
0.960
17.27
Mix-5%
86.2 74.8
94.0/ 87.7
0.93
15.92
Mix-10%
88.6/81.4 81.4
0.94
16.04
Mix-20%
65.2
81.7 / 77.1
0.79
2.72
Maze 20x20
Mix-40%
53.2
80.4/ 77.2
0.78
2.65
Mix-50%
57.3
79.6 76.0
0.77
2.39
Mix-60%
66.8/ 63.6
82.4/ 80.1
0.80
2.50
Mix-80%
56.5 51.8
74.5 / 70.2
0.71
2.13
Solution-Only
56.3 / 52.0
71.9/
0.69
1.52
Dua former (fast)
72.9
95.5 / 90.6
0.940
21.23
Mix-5%
73.4 56.9
87.0/ 77.3
0.85
13,37
Mix-10%
80.2 / 68.0
90.4 / 83.1
0.89
16.19
Mix-20%
57.8
85.0/ 78.0
0.81
10.51
Maze 25x25
Mix-40%
50.7/46.3
73.8
0.69
3.60
Mix-50%
56.5 50.8
77.9 / 71.9
0.74
3.15
Mix-60%
53.6/ 48.9
75.9 / 69.8
0.72
3.07
Mix-80%
48.2 / 44,2
69.6 / 64.4
0.66
2.88
Solution-Only
39.7 / 34.7
60.3 /55.4
0.57
1.9
forlfrrmer (fast)
80.0 66.0
91.8 / 85.7
0.906
18,23
Mix-5%
66.3
83.6 / 72.0
0.82
11.85
Mix-10%
76.9/655.
90.8 / 84.7
0.89
19.35
Mix-20%
/3.5
78.3 67.4
0.75
9.97
Maze 30x30
Mix-40%
33.3
69.7 / 63.2
0.66
3.79
Mix-50%
43.0/ 38.7
68.1 / 62.9
0.64
3.23
Mix-60%
39.2 33,8
65.2 59.0
0.61
3.09
Mix-80%
38.0 32.3
62.2 / 55.5
0.58
2.75
Solution-Only
30.0 / 26.0
54.1 / 47.8
0.50
1.86
Dualformes (fast)
97.3 94.4
94.8 / 90.0
0.970
4.92
Mix-5%
84.0 73.0
91.0/ 84,0
0.90
4,30
Mix-10%
94.0/ 87.0
97.0/ 94.0
0.97
5.58
Mix-20%
94.0 89.0
97.0/ 95.0
0.97
4.10
Sokoban
Mix-40%
/
98.0 97.0
0.98
4.15
Mix-50%
97.0/
99.0 98.0
0.99
3.38
Mix-60%
98.0/ 97.0
99.0 99.0
0.99
3.25
solution-only
86.8/ 83.4
92.8 90.0
0.92
1.24
Table F.2: Comparison of Dual former and Mix-p models in the fast mode.
27

---

Published as a conference paper at ICLR 2025
G COMPARISON WITH DIFFERENT RANDOMIZATION STRATEGIES
In Table G.I below, we show the probabilities of different dropping strategies we use for comparing different randomization strategies.
Model
Probabilities of Dropping Strategies
Dropping Level 1 Po 0.5, 0.5, Ps P4 0
Dropping Level
Po 0.5.P1 -P2 022-0.25pp P4
Maze
Dropping Level 1 + +
Po 0.5. =P= = 1/6, =
Dual Former
0.45.p: = = = 1/6, 0.05
Dropping Level
Po 0.95, 0.05. == ===
Dropping Level
Po 0.85, P1 0.05, -11.p==-4-- -
Sokoban
Dropping Level 1 + 2 Dual former
Jo 0.75. 0.05.
=0 = 0.7,P1 0.05,
= 00..0.,.44 = 0.05
Table G.1: The probabilities of the dropping strategies used for each model.
28

---

Published as a conference paper at ICLR 2025
H DETAILS OF THE MATH REASONING EXPERIMENTS
H.1 SOLUTION REWRITING
H.1.1 SOLUTION REWRITING PROMPT
We use the following prompt template for rewriting the solutions via LLama-3. 1.mma-3.-777 3.1-70B-Imm Instruct.
Solution Rewriting Prompt
I have math problem and an imitial chain thought reasoning that needs elaboration. Please provide a more detailed step-by-step explanation for each part of the reasoning, including intermediate steps, calculations, and rationales behind each decision. Also, suggest any additional insights that might be relevant. Problem: [Question] ? Current Chain of Thought: {CoT} The answer is (Final-answer). Keep the total response less than 2048 tokens. Please expand the above reasoning and give detailed reasoning chain and explanation. Write your answer in this format: "Here a detailed step-by-step explanation of the reasoning: Reasons: <rraaoon step eg.Step 1..S Step St.. N... The answer is: <answer >." Remember, always conclude with "In) answer is: <answer >."
H.1.2 SOLUTION REWRITING EXAMPLES
Below, we provide an example of math question along with four generated solutions that have been rewritten by the Llama-3.1 1----..----------- Instruct model. evident from this example that the original math dataset lacks sufficiently detailed solution steps required to solve the question, such as the formula for geometric series.
Original Training Example from MATH
Question
The first term of geometric sequence is 729, and the 7th term is 64. What is the positive, real value for the 5th term?
Answer
The only positive, real common ratio for this sequence is 3. Thus, a is the 5th term, then (2) x 64, so x
144
29

---

Published as a conference paper at ICLR 2025
Rewritten Answers in Aug-MATH
Llama Allmma-.1-7700 3.1-70B Instruct Rewriting
Here a detailed step-by-step explanation the reasoning: |n' [n/nStep 1: Recall the formula for the nth term of geometric sequence. The nth term of geometric sequence is given by the formula: a, ar(x-1), where is the first term, r is the common ratio, and n is the term number.
up equation. We are given that the first term (a) 729 and the 7th term (a>) is 64. We can use this information to set up an equation using the formula from Step 1: 64 = 729r
\ntep 2: Use the given information
\n'nStep 3: Simplify the equation. Simplifying the equation from Step 2, get: 64 = 7
\n'nntep-4: Solve for the common ratio (r). To solve for r, we can divide both sides
of the equation by 729 and take the 6th root of both sides:
\n nStep 5: Calculate the value of Evaluating the expression from Step 4, get:
64 ( =
729 (n nStep 6: Use the value of r to find the 5th term. Now that we have the value of
we can use the formula from Step to find the 5th term (as): as = 729 (3) (6-1)
nStept 7: Simplify the expression for ag. Simplifying the expression from Step get: a5 729 (3) 729 87 144.
in n'The answer is: 144.
Llama-3. 3-1---00----see Rewriting
Here detailed step-by-step explanation the reasoning:
'n' SStep I: Understand the problem The problem presents geometric sequence with given first term (729) and given 7th term (64). We need to find the positive, real value of the 5th term.
(n' nStep Identify the common ratio Since it's geometric sequence, we know that each term is obtained by multiplying the previous term by fixed constant, called the common ratio (r). need find this common ratio.
\n' nntep 3: Use the given terms to set up equation We know that the 7th term 64, and the first term is 729. Using the formula for the nth term geometric sequence: a, = 01 nn1 (n' nStep 4: Solve for the common ratio Divide both sides by 729: 729 = G. Take
up the equation: 64 729 6
the 6th root of both sides: in inStep 5: Verify the positive, real common ratio We are told that the only positive, real common ratio for this sequence is 3. This is indeed true, as the 6th root of 64 is
64 729
729
SStep 6: Use the common ratio to find the 5th term Now that we have the common ratio, we can find the 5th term. Let z be the 5th term. Using the formula for the nth term: 729 (2)*.
SSte 7: Simplify the expression
729 = 144.
81
nStep 8: Verify the answer We can verify that (2)2 64 indeed holds true for
144. The answer is: 144
30

---

'n' nStep 2: Recall the formula for a geometric sequence geometric sequence is defined by the formula: On ai -"-1, where an is the nth term, di is the first term, r the common ratio, and n is the term number. 'n'ept 3: Use the given Published as a conference paper at ICLR 2025
Rewritten Answers in Aug-MATH
Alama-3.1-708 3.1-70B Instruct Rewriting 3
Here is a detailed step-by-step explanation of the reasoning: \n\nStep 1: Understand the problem We are given a geometric sequence with a first term of 729 and 7th term of 64. We need to find the positive, real value of the 5th term.
information to up an equation We know the first term (a1 729) and the 7th term (a7 = 64). We can use this information to set up an equation: 64 729 7 which simplifies to 64 729 ,
in' nStep 4: Solve for the common ratio (r) To find the common ratio, we can divide both sides of the equation by 729, resulting in 729 ,. Taking the 6th root both
sides, we get = 729
(n' SStep 5: Simplify the common ratio Evaluating the expression, we find that
'n't inSttep Use the common ratio to find the 5th term Now that we have the common ratio, we can use it to find the 5th term. We know that as - 729 (3)
An' nSSepte 7: Simplify the expression for the 5th term Evaluating the expression, we get 05 729 in answer is: 144.
144.
1aama-3.1-70B-Inaee Rewriting 4
Here detailed step-by-step explanation the reasoning:
Reasons:
(n' SStep 1: The problem provides us with the first term of geometric sequence, which is 729, and the 7th term, which is 64. We are asked to find the positive, real value for the 5th term.
nStep 2: To solve this problem, we need to find the common ratio of the geometric sequence. We know that the common ratio term. Let's denote the common ratio in nntep 3: Using the given information, we can set up an equation to find the common ratio. Since the 7th term is 64,
ratio of any term to previous
write: 729 ,6 64.
(n' [Step 4: Now, we can solve for r by taking the 6th root of both sides of the equation: r = (72))
(ntenSe Simplifying the expression, we get: r (2) = However, this not the only possible value for the common ratio. We need check if there are other positive, real values of r that satisfy the equation.
in' \ntep 6: We can do this by factoring 729 and 64 to see if there any factors that could lead to other possible values of r. Factoring 729, we get: 729 = 36. Factoring 64, get: 64 26
(n' (SStp 7: Since 729 and 64 have no common factors other than 1, we can conclude
that the only positive, real common ratio for this sequence is indeed (3)² \n' (nStep 8: Now that have found the common ratio, term of the sequence. Let's denote the 5th term as x. We know that the 5th term is obtained by multiplying the 3rd term by the common ratio twice. Since the 3rd term 729 (2)²= 144, we can write: x = 144 (2)²
use it to find the 5th
nStep Simplifying the expression, we get: 144. nhe answer is: 144.
31

---

Published as a conference paper at ICLR 2025
H.2 FINE-TUNING AND EVALUATION PROMPTS
We the following prompt for fine-tuning both the Mistral and the Llama model, following (Yu et al., 2023).
Fintuning Prompt
<start-header-token: Below is an instruction that describes task. Write a response that appropriately completes the request. (n'n ### Instruction: n Question) \n)n ### Response: Let's think step by step. (CoT steps final solution]
We the following prompt for the slow mode evalaution.
Slow Mode Evaluation Prompt
<start seatt-hheeer--tkoeess Below is an instruction that describes task. Write response that appropriately completes the request. in ### Instruction: n (Question) \n),n #### Response: Let's think step by step.
For fast mode evaluation, we force the generation to directly output the final answer by adding the phrase "The answer is:
Fast Mode Evaluation Prompt
sttart-header-token--BBBBBe is an instruction that describes a task. Write a response that
appropriately completes the request. (n ### Instruction: n (uuestion] \n)) in ### Response: nn'nT answer is:
H.3 EXAMPLE OUTPUTS
We provide few example math questions and answers randomly sampled from our trained model, alongside the answers output by the baseline model. Within each answer box, on the left is the baseline model output (long and lengthy) and on the right is our method output (effeettve and efficient, colored in blue).
Question
In Pascal's Triangle, the third row has four elements, and the of these elements is 8. What is the sum of the elements in the tenth row? (Note that we are counting the top row with single I as the Oth row).
32

---

Published as a conference paper at ICLR 2025
Answers to Q1
Llama-3-Aug (baseline) Here a detailed step-by-step explanation of the reasoning:
Llama-3-Aug (p=0.4, ours) Here is de- tailed step-by-step explanation of the reason- ing:
Step 1: Understand the problem and the given information. We are asked find the sum of the elements in the tenth row of Pascal's Triangle, given that the third row has four elements and their sum is 8.
Step 1: Understand the problem and the given information. We are asked to find the sum of the elements in the tenth row of Pascal's 's Triangle, given that the third row has four elements and their sum is 8.
Step 2: Recall the formula for the sum of the elements in row of Pascal's Triangle. The sum of the elements in the nth row is given by the formula:
Step 2: Recall the properties of Pascal's Triangle. The elements in each row of Pascal's 's Triangle are the binomial coeffi- cients, which represent the number of ways choose k items from sum of the elements in each row is equal to 2", where the row number.
(%)
= 2".
items. The
Step Identify the relationship between the sum of the elements in a row and the sum of the elements in the previous row. Since the sum of the elements in the nth
Step Identify the relationship between the sum of the elements in the third row and the sum of the elements in the tenth row. Since the sum of the elements in each row equal to 2", we can write the sum of the elements in the third row as 2 8. Similarly, the sum the elements in the tenth row is 20
row 2", the sum of the elements in the (n-1) row is 2n-1
Step 4: Recognize that the sum of the ele- ments in the third row 8, which is equal
to 23. This is because the sum of the ele- ments in the third row is (9) +)+ (+) + ( 1 + 3 + 7, and the extra comes from the top row, which counted as the Oth row.
Step 4: Calculate the sum of the elements the tenth row. 210 1024.
Step 5: Use the relationship between the sums the elements in consecutive rows to find the sum of the elements in the tenth row. Since the sum of the elements the third row is 23, and the sum of the elements the (n-1)th row 2n-1 we can set up the equation: 23 = 210-1
The answer is: 1024.
Step 6: Solve the equation for n. Tak- ing the logarithm
both sides, we get: 3 log 2 (10 1) log Simplifying, we get: log = log 2, which implies that log 0. This is contradiction, since log is positive number.
Step 7: Conclude that the equation 2 210-1 has no solution, and therefore, the sum of the elements in the tenth row is not equal to 10-1
Step 8: Recognize that the sum of the ele- ments the tenth row is still given by the formula: (10) (19) + (20) + (10) 210
The answer is: 1024.
33

---

Published as a conference paper at ICLR 2025
Question
juice company sells its product in either 48-ounce size or 32-ounce size. charges $3.90 for the 48-ounce size. How much should it charge for the smaller size if it wants the price per ounce be 25% more than the price per ounce of the larger size?
Answers to Q2
Llama-3-Aug (baseline) Here a detailed step-by-step step explanation of the reasoning:
-lama-3-Aug(==0.4, ours) Here is de- tailed step-by-step explanation of the reason- ing:
Define the problem and the goal. The juice company wants to charge a price per ounce for the smaller size that is 25% more than the price per ounce of the larger size. This means that we need to find the price of the smaller size such that the price per ounce of the smaller size is % more than the price per ounce of the larger size.
Step 1:
Step I: Understand the problem and the given information. The juice company sells two sizes of its prod- uct: 48-ounce size and 32-ounce size. The price per ounce for the larger size is $3.90/48 $0.08125. The company wants the price per ounce for the smaller size be $25% more than the price per ounce of the larger size.
Calculate the price per ounce of the larger size. The price the larger size is given as $3.90, and the size is 48 ounces. To find the price per ounce, we divide the price by the size: 3.90 0.08125.
Step 2:
Step 2: Calculate the price per ounce for the smaller size. The price per ounce for the smaller size should be 25% more than the price per ounce of the larger size. There- fore, we need to multiply the price per ounce of the larger size by 1 + 0.25 = 1.25. So, the price per ounce for the smaller size $0.08125 1.25 $0.1015625.
48
Calculate the price per ounce of the smaller size. Since the price per
Step 3:
of the smaller size is 25% more than the price per ounce of the larger size, we multiply the price per ounce of the larger size by 1.25: 0.08125 1.25 0.1015625.
Step 3: Calculate the cost of the smaller size. The cost of the smaller size is the price per ounce mul- tiplied by the number of ounces. Therefore, the cost of the smaller size is $0.1015625 32 = $3.25.
Step 4: Calculate the price of the smaller size. To find the price of the smaller size, multiply the price per ounce by the size of the smaller size, which 32 ounces: 0.1015625 x 32 = 3.245.
The answer is: 3.25.
Step 5: Round the price to the near- est cent. Since the price is given in cents, we round the price to the nearest cent, which $3.25.
The answer is: $3.25.
34

---

Published as a conference paper at ICLR 2025
I LLM GENERATED SOLUTIONS FOR MAZE
Readers might wonder whether modern state-of-the-ard L.MSs can effectively solve our maze problem. To test this, randomly selected a 30 30 maze problem and ask the ol -preview model to find the shortest path. O1-preview is the latest reasoning model by OpenAI which operates in slow mode: spends more time thinking before they respond. It turns out that these problems very challenging for LLMs. As illustrated in Figure 1.1, the path suggested by ol preevieeww incorrectly traverses through the maze walls. In contrast, Dual former correctly identifies one optimal path that follows through the maze without any errors.
Cnet
ol ol-preview
ol-mini
1X 240 20
(c) Dualformer (slow mode)
Figure I.1: Example 30x30 maze problem. The wall cells are depicted in dark, while the unoccupied cells shown in gray. The starting point is marked in green, and the destination point is in red. The output path is highlighted in yellow. (a) (b) The generated paths provided by ol ol-preview and ol-mini incorrectly traverse through walls. (c) Dual former (slow mode) identifies one optimal path without any errors. The cells explored by the Dua 1 former reasoning trace are highlighted in red, where the color intensity indicates the total cost value.
Below, also provide the exact prompt used for ol models, and their responses.
35

---

Published as a conference paper at ICLR 2025
Prompt for Ol
Here''s is the penerated 30x30 mase along with Locations of Walls: (1.
01
64,
0)
(5. 03
(7. 0), 09 03 110 O0
(13 01
(15. D3
(17.
01
(18
01
(24, 01 (25. 05
0J, 1. 31, (10,
(28,
(13,3 5).
5)
5%,
51.
5),
6
(10. 6) (1) 60 (12, 60 177
63
18 60
(24 61 (25
61
63
65
(11, 7/, (13, TV (14, 7%, (39,
Th
(22.
7h,
125.
TV,
(27, Th
64, 80 18. 81,
(0.
81.
(1A.
81
(14, 81
(16, 81
115 81 19 80
(3 99.
64 96. 96
(10,
96.
112.
96, 113.
(22,
9),
(13,
(15,
123,
(25.
(11. (13,
113,
111,
13,,
11X
12
12,
131, 13), (4, 13), 15. 13%
(15. 13L (19. 131, (20, 13), (24. 13x, (25. 13 12R. 131,
13
(1L 14) (12
14) 11 181,
224
14
629
14, 15
(2, (13, (17, 15), (20, 15), (22. 15), (23, (25, 15), 126, 15), 127. 15 64. 161. (10. 16) (18 150, 120.
16), (
17),
64
17
65
17.
10.
(1.
(19. (20. (22.
(12, (14,
(25,
(26
(28.
28).0, 21). (9, (10, 21), (11. (13, 21). (15. 213.(17 (17, (20, 213 213.
21) (). 23) (3 22
22
(10.
220 (11, 22, (12. 22).
(13, 22) (16, 22), (20, (29, 22), 16, (10, (11, (12, 25), 118, 25), (19. 23), (21,
251,
25
00.
240,
65.
241,
24
(17.
240, (20 24),
243 253,
25), 251, 25), 25), 25),
261.
261.
(19, (20,
(25, (36, (7),
(1), (1),))),,, (15, (15, 27).116
(20,
(27. 27).
293, 15, 253, (8, 283, (12, 28), 281, 28%, LE7, 28), 28), (29, 28), (0. 29), C2. 29), (3, 29), 29) 17. 290,
29),
(10.
290,
(13,
(15, (90,
(19, 29), (21, 29) Start Location: (29.7). End Location: (27.29).
Could
optional
(X.y)
Excample (8. 08, 5), (9, D), Start Locatiio:: 0). End Location: 19, Optimal Puth: (0, 01, 10, 1X (0, 2), (1, 2% 12. (2) 12 41, (2. (2, 60, (3.
43, 51, (3, (1, 63,
(4,
73, (5, 73, 7%, 8t, 80, (8, 83, (9, (9. 9)
Example Locations Walls:
(1.2))
(9.
Location: (). Location: (9, Opeinall Puth: (0. 0) (0. 1) 2) (1, (2. 3) (3, 5) (5. (6.
(7, 5) (S.
(8.) (8. (9,9)
36

---

Published as a conference paper at ICLR 2025
01 Response
Thought for 81 seconds
Mapping out the I'm unalyzing the maze dimensions, idemtifyng wall locations, start, and points. The goal is to determine the shortest puth, considering the mare's constraints
Navigating the grid
I'm mupping 30c30 grid to represent the maze. Using for open spaces and for walls, I'll utiliz BFS find the shortest path (27.29). Mapping wall positions
I'm idcatifying wall coordinates for 30x30 grid, setting gridly [Ix] to at each specified location. This helps visualizing the grid and understadinng wall placements.
Setting up the grid
I'm setting up 50c30 grid for the algorithm to find the shonest path. The walls and startiend locutions marked, and I'm initializing grid and quese for BPS exploration.
Mapping valid moves
initialize the queue with the start position and set tracking for visited positions and predecessurs. Then. BTS to explore and validase neighboring positions, noting predecessirs for potential path roconstruction.
Assessing potential moves
I'm vendying the possible moves from (29.7) by checking (29.6), S1, and (28,7) for boundaries, walls, and visit starus, then addiing them the queue. Novigating the queme
I'm working thoough neughbors of (29,6), (6, checking bounds Adding viable positions the gueue, upcating peodecessors needed. Progressing one step at time.
Evalnating quesee options
I'm digging into gnd evaliattinn, validating neighbors, adding them the queue, and marking them visited. This thorough approach helps assessing potential moves effectively.
Assessing poeshillttes
Starting with (29,5), none of its neighbors lead to further progress dae to walls out-of bounds Incations. Inspeeting polential movensents
Checking (27.6) 6) for walls. Marking visited updating prodecessors, Moving to (29,10) and (28.9) next.
Assessling udjacent spaces I'm minking through the neightors of each grid cell, distinguishing berween walls and open spaces. Adding new valid coordiantes the quene for further exploration.
Roaluating options I'm evaluaring neighbrrs of current coondarates to see any of them are walls, and if they aren't, add thes the quene and mark them as visited. Novigating the grid
position examining neightoring cells. Walls block paths, while open cells are queued for visiting This approsch heelp avoid cevisiting cells.
I'm tncking
Charting the path
Novigsting the quene. I identity adjacent spots, checking for walls, Fach marks neightors is visited and records prodecesior. Progross is steady
path unfolds.
Navigating walls
I'm updating neighhors and checking the queue for walls, If elear, I Ludd the nest step. This process allows movement and exploration, gaduullly revealing progress.
Mapping the journey
I'm planning tocree a puth from (29.7) (27,29) in mare. Avoiding walls and siming for shonest rourc, the RPS algorithim will meed.
Mapping the path
I'm noting that the shontsst path the maze is from (29. 7) to 29,29), then (27.29), while avoiding walls ym20 CHUNK: Assessing obstseles along x-29 Naticing obstmctions coordinates (29.0), (29,2), (20,4), (29,14), (29,20)) (29,24), and (20,29), with walls al (27,29) or (28.29), Noticing walls block movement at y=29 on x=29. Identifying wall positions
Mappiing out wall locations berween y and 2-29 along x-28. Key walls identified at y-1. y-5 y-7. y-9 y-16 y-13, y-20, y-23, and y-24. Plansing
OK, me see. Nomad is thinking of moving along x-78, navigating aronnd elogged scctions, and expanding to x=27 x-26 to the optimal poth. Sitmated in diffieult spot
OK, let mo see. The puth forwand blocked by walls at (29, 14) and (28.13), movement restricted to (29, 13) and back (29.12).
Assessing path constraints
I'm noting that movement is hindered by walls at various coundinates, such as (28, 12) and (27, Prugress is restricted specific divections.
Charting the course
Stnck at (28. 11), occupynng (28. 10) and (28, 12), Moving left scems advissble, so ascending through (27. and (26, 10) might he heneficial.
Sure. I'd happy
maze provide the optimal puth the sun location (29. 7x29, 7x29.7i
location (27,29)27, 29,27,299)
solve this mare, we'll perform brendth-fisst seach (BF5) algorifhm, which is ideal for finding the shortest puth an unweighted grid like a mune.
Hose's is how we can proczed:
Represent the Maze: Create 30x30 grid where each cell comesponds coundinate (uy),. Kiag), Mark the walls the grid based the provided wall locations.
Initialize BFS:
Start Point: 2927, 7)229,7 7) End Point: (27, 22,29y27, 291127.29) Queue: Initialize with the stant point. Visited Set: Keep track of visited cells to avoid revisiting them. Purent Map: Keep trick of the path by recomling the porent of each visited cell.
Perform
While the queue is not empty:
Dequeee the fins cell. it's the and point, reconstrnct the puth using the porent maps Otherwise, explore its neighboring oells (og, down, right if they wall, and visited.
within bounds,
Recomstruct Path:
Surting from and paint, backimek neing the parent map to reconstmet the aptimall path to the start point.
After performing the DFS algorithm, we find the following optimal puth:
Optimal Path:
(29. 7) (27. 7) (26.
83 (26. (26, 10) (26. 11) (26, 12) (26, (26. 14: (26, (26, 16) (26, 17) (26. 18) (26. 19) (26, 20) 126, 21) (26, 22) (26, (26, 24) (26. 25) (26, 26) (26, 27) (26, 28) (27, 28) (27, 29)
Explanation:
From the start point (29.7X29, 7x29,77,
left (28.7V28. 7W28.7) and continus (27.7H27, 7Y27.7) and (26,7x26, 7X26,7), then down from 7x26. 7126.7) 7) to (26,28X26, 28x26.25), avoding and obstacls. Finally, move right to (27.28x27, 28)27.28) and up and point (27 277.7))27, 29.27.29).
This path avoids all walls and provides the shortest coute from the start the end location. there
Note: This push
walls blocking the ruute along the cells specified. If there walls that prevent this direct puth, adjusttentts would need made novigaating around obstacles, BFS ensures shortest path found gaven the maze contiguratioo. If you have any questions or need further clarification, feel free to ask!
37

---

Published as a conference paper at ICLR 2025
O1-mini's Response
Thought for few secunds Scopiing the task beginning Crafting the BPS appreach
Using
path
I'm working through the optinal path in 20x30 gad with walls, ensuring accuracy and recommending puthfinding alguridim for efticiency.
sulve the 30c30
provided, we' ll **rredtth First Search (BFS)** algorihm find the shoriest puth from the *Sart Location (27,29),
729,77 ""add
step-by-stg step
Step I: Represent the Maze
Fins, we represent the maze 30c30 grid where each cell can be either
wall. The walls are provided list of coordinates.
Initialize
'12977" explore possible moves tup, down, right) avoiding already
Step 3: Parform BPS to Find the Shortest Path
systematically explocing the maze, BPS ensures that the first time reach the ""ind Location",, we' taken the shartest possible pash. optimal path
Thace Back the Path
Optimal Patn from (29,77 '(27,99)
After processing the maze, the optimal path follows: ""
(26.)
(25,17) (25.18) (25,19) (25,20) (25.20) (25,22
(25,24) (25,25) (25,26) (25,27) (26.27) (27,27)
(27.28) (27,29)
Detailed Path Coordiiattes
complese list conrdinttes
optimal
((2977) 128,7)3 (27.) 1126.7 (25,7) "(2.8) (25,0) (25.100 10y0 (25,1YY 10, (25.122 (25.13) (25.14)
13, "25,155" 14, (25.16) 15, 125,177 16. 125.189" 17, ((2,11)) 18. ((25,20)" 19, (25,211 20. (225,22) 21. (25.23) 22. "25.24)" (25.25) 24.
(25,26) 25. (25.22) 26. 126.27y 2 (21,271 28. (27.28" 29, ""End:" '127299)
Visualization understanding,
simplitied
path grid:
Stari (29,7) -V (28.7) - (27.7)
(27,29) End
Coaclasion
efficentll finds more,
optinal, you any further questions
38


Reviews:
[{"review_id": "blPvtj2ygF", "reviewer": "Reviewer_MUZn", "summary": "This paper aims to endow a transformer model with both slow and fast reasoning modes using only a simple data recipe. The proposed model -- Dualformer -- is trained using randomized trace dropping, which enables the model to switch between fast (solution-only) and slow (step-by-step reasoning) processing without any architectural modifications. Dualformer is evaluated on maze and Sokoban pathfinding tasks. The model is evaluated on maze and Sokoban pathfinding tasks and is also applied to fine-tune large language models (LLMs) for solving math problems. Experimental results demonstrate that Dualformer achieves superior performance compared to baseline methods.", "questions": "- What are the specific failure modes in fast mode?\n- Does fast mode performance degrade as maze complexity increases (e.g., with a higher density of wall cells or more structured wall arrangements)?\n- How would Dualformer perform on a rectangular maze instead of a square one at test time?\n- In the example mazes shown, it appears relatively easy for a human to find the optimal path. How does fast mode perform on more challenging mazes where a human might need more time to find a path?\n- For the dropping strategies, how were the initial three sets of probabilities selected?\n- The experiment with OpenAI o1 models seems to use them in a zero-shot manner. Have you tried using few-shot approaches and testing on smaller mazes?\n- Could you elaborate on how the randomized dropping strategies mimic shortcuts in human cognitive processes?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "In this work, the authors leverage a purely data-driven approach (randomized trace dropping) to enable controllable dual-mode reasoning in a Transformer model.  This approach is computationally efficient and reduces the need for complex architectural changes. In the pathfinding tasks, Dualformer outperforms baseline models in both fast and slow modes, both in terms of the number of optimal solutions found and the diversity of these solutions.. Additionally, the approach proves effective in improving performance on math problem-solving tasks. The methodology, experiments, and results are clearly presented, making the paper easy to follow and understand.", "weaknesses": "It is unclear whether the chosen tasks (mazes, Sokoban) encapsulate fast versus slow thinking as humans would apply them. In particular, simply outputting a solution without explicitly listing the reasoning steps does not necessarily capture fast cognitive processing.\n\nIt would be valuable to have more extensive analysis of where and why Dualformer’s fast mode fails. Specifically, examining how task complexity—such as increased wall density or structured obstacle layouts—impacts performance could reveal important limitations.\n\nMore investigation into the auto mode, specifically on how the model determines which mode to activate under varying maze difficulties, would add depth. Understanding the criteria or patterns that lead to automatic mode-switching would be very useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mMUel8XwBV", "reviewer": "Reviewer_Rqgz", "summary": "The paper proposes **Dualformer**, a new Transformer model designed to integrate both fast and slow reasoning modes, inspired by the dual process theory in human cognition. The model is trained on data with randomized reasoning traces, allowing it to dynamically choose between quick, solution-only outputs (fast mode) and detailed, step-by-step reasoning (slow mode).", "questions": "1. **Code Availability**: Will the authors release the code to ensure reproducibility? If not, can they provide more detailed pseudocode or implementation guidelines?\n\n2. **Comparison with SwiftSage**: The authors should include a comparison with SwiftSage and discuss the differences and similarities. How does Dualformer improve upon or differ from SwiftSage?\n\n3. **Complex Scenarios**: Can the authors extend their experiments to more complex tasks such as strategic games (e.g., Werewolf) or combinatorial optimization problems? This would better demonstrate the model's versatility and robustness.\n\n4. **Theoretical Justification**: While the empirical results are promising, a theoretical analysis of why and how the randomization of reasoning traces improves performance would strengthen the paper. Can the authors provide more theoretical insights or hypotheses?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The idea of combining fast and slow reasoning modes within a single model is novel and inspired by human cognition, which is a strong conceptual foundation.", "weaknesses": "1. **Reproducibility**: The paper does not provide code or sufficient details about the implementation, which raises concerns about reproducibility.\n\n2. **Comparison**: There is no comparison with similar methods such as SwiftSage[1], which should be cited and discussed to contextualize the contributions of this work.\n\n3. **Experimental Scope**: The experimental tasks (maze navigation and Sokoban) are relatively simple and may not fully demonstrate the model's potential in more complex, real-world scenarios. More challenging and diverse benchmarks would strengthen the paper.\n\n### Reference:\n\n- [1] Lin, Bill Yuchen, et al. \"Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks.\" Advances in Neural Information Processing Systems 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wY6q8ZXr7W", "reviewer": "Reviewer_ui7f", "summary": "Inspired by slow and fast thinking modes in human cognition theory, this paper introduces Dualformer by training models on reasoning traces with randomized dropping, which allows the model to learn efficient reasoning shortcuts. The performance outperforms baselines on a maze environment and MATH dataset.", "questions": "1. In section 5.2 (Structured Trace Dropping and Randomization), the MATH dataset is constructed by randomly dropping intermediate reasoning steps from the solution. However, in the examples shown in Answers to Q1 and Q2, the model doesn't skip steps but rather presents solutions more concisely. Does this truly demonstrate learned slow and fast thinking strategies, or is it simply a more compact problem-solving approach? And where does the mismatch between the training demo and inference response come from?\n2. Following Limitations 3 and 4, the toy example's limitations seems also stem from the generalizability in approach. In the maze example, the procedure first creates a search path using A*, and the model attempts to mimic this search procedure with random dropout. However, in math problems, the baseline provides a direct path with multiple steps, which is then subjected to random step dropping. This raises questions about where the equivalent of the A* algorithm is in MATH problems and how Structured Trace Dropping is implemented on MATH or other general tasks. The methodology appears to be using different approaches for maze and math problems without clear justification for their equivalence.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written and easy to follow. \n2. The idea of integrating fast and slow reasoning modes from human cognition theory is novel.", "weaknesses": "1. The motivation is not super convince to me, why LLM needs to integrate both the fast and slow reasoning modes are not clear, it’s not convinced that human takes benefit from the these two modes then LLM should also follow these two modes given the fact that human brain and LLM are running with two distinct systems.\n2. The mathematical foundation of the methodology requires more rigorous discussion. Specifically, it's important to explain why dropping random intermediate steps can lead to better performance. This contrasts with findings in recent reasoning papers, for example [1, 2, 3] where learning longer reasoning steps involving (1) different reasoning strategies; (2) self-refinement; (3) backtrack, etc, led to better performance since it provides the model with more flexible function mapping, which is easier to fit compared to direct query-to-solution mapping. Importantly, a section discussing the foundation of the proposed method is necessary.\n- [1] Qin, Yiwei, et al. \"O1 Replication Journey: A Strategic Progress Report--Part 1.\" arXiv preprint arXiv:2410.18982 (2024).\n- [2] Kumar, Aviral, et al. \"Training language models to self-correct via reinforcement learning.\" arXiv preprint arXiv:2409.12917 (2024).\n- [3] Gandhi, Kanishk, et al. \"Stream of Search (SoS): Learning to Search in Language.\" arXiv preprint arXiv:2404.03683 (2024).\n3. A major limitation is that the toy example is too simplistic for understanding the method's effectiveness in real reasoning tasks, though it serves to illustrate the problem scope, and there is one table showing results on MATH. To demonstrate that the model truly learns better solution strategies rather than overfitting on synthetic data, one idea is evaluating in out-of-distribution environments. For example, the model could be finetuned on the maze dataset but evaluated on MATH or GSM8K - problems that could benefit from better thinking/reasoning strategies. \n4. Moreover, in Table 5.6, the differences between the baseline and the method proposed in this paper are marginal. This raises questions about whether the results stem from hyperparameter selection, especially since finetuning in the MATH domain uses the same hyperparameters despite different response lengths. Additionally, the evaluation is conducted with only one seed, and the model uses temperature=0.7 and top-p=0.9 for better diversity. While this approach is valuable for studying whether the model can generate diverse solutions, it introduces uncertainty when evaluating the method's performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bDgKZ1LEif", "reviewer": "Reviewer_pvvZ", "summary": "- The paper proposes the question: Can we integrate both the fast and the slow modes into Transformer-based reasoning agents, similar to how humans possess two distinct thinking systems, and let them complement each other?\n- Main contributions: A framework for training Dualfromer, a Transformer that solves reasoning and planning tasks inspired by the “System 1 & System 2” thesis by Kahneman in _Thinking, Fast and Slow_. \n- One of the tasks is a maze-like environment where the model is trained to find the shortest path between two cells. The model can either output the solutions and reasoning chain or only the final solution. In “auto mode,” it can decide which of both strategies to engage in. \n- Being able to engage in these different modes is an improvement over current models in the cluster, noticeably Searchformer, which is only able to engage in outputting reasoning chain & solution, potentially requiring lots of inference time. The new model improves here by being trained on randomly sampled reasoning traces and dropping traces with various strategies. \n- They also continue evaluating their new framework by training a Dualformer to solve the Aug-MATH benchmark, where the model outperforms finetuned LLMs.", "questions": "- Why did you not test auto mode on the MATH benchmark?\n- How did the training differ between Dualformer and finetuned MATH models?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Considering system 1 & 2 while working with transformers isn’t particularly novel, I thought the way it was included while training the transformer on reasoning traces interesting. Beyond that, the paper provided good motivation for why the auto mode strategy could be more efficient than current models. \n- I found the results on the MATH dataset fascinating and was surprised to see a “more real-world” application of this strategy.", "weaknesses": "- While I could understand the performance gains of the new model based on some results being higher in Table 5.1-5.3, I found it very hard to make sense of which models exceed where and why. E.g. I understood that “Auto mode” should, in theory, be less exploratory than “Slow mode”, why is the SWC for 30x30 mazes higher? (0.993>0.989). This is, of course, a nitpick, and I don’t expect the reason here to be super relevant, but I would have, in general, appreciated a better overview of the results. Every score improvement has a blue box, which ultimately doesn’t carry a lot of information value (_if everything is highlighted, nothing is_). It could easily be that I don’t understand a fundamental consideration of this yet for them to be all evaluation indistinction, so I am excited to hear your response. \n- I have a similar, though slightly less confusing, nitpick with the results in the MATH benchmark. I appreciate this inclusion of a finetuned Mistral and Llama, though I would have wished to better understand how good this is compared to other sota models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)