Can Language Models Learn to Skip Steps?
Authors: Tengxiao Liu
Qipeng Guo
Xiangkun Hu
Cheng Jiayang
Yue Zhang
Xipeng Qiu
Zheng Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Models
Natural Language Processing
Reasoning
Human-like Abilities
Primary area: natural_language_processing

Abstract: Trained on vast corpora of human language, language models demonstrate emergent human-like reasoning abilities. Yet they are still far from true intelligence, which opens up intriguing opportunities to explore the parallels of humans and model behaviors. In this work, we study the ability to skip steps in reasoning—a hallmark of human expertise developed through practice. Unlike humans, who may skip steps to enhance efficiency or to reduce cognitive load, models do not inherently possess such motivations to minimize reasoning steps. To address this, we introduce a controlled framework that stimulates step-skipping behavior by iteratively refining models to generate shorter and accurate reasoning paths. Empirical results indicate that models can develop the step skipping ability under our guidance. Moreover, after fine-tuning on expanded datasets that include both complete and skipped reasoning sequences, the models can not only resolve tasks with increased efficiency without sacrificing accuracy, but also exhibit comparable and even enhanced generalization capabilities in out-of-domain scenarios. Our work presents the first exploration into human-like step-skipping ability and provides fresh perspectives on how such cognitive abilities can benefit AI models.

Can Language Models Learn to Skip Steps?
Tengxiao Liu
Qipeng Guo
Xiangkun Hu
Cheng Jiayang
Yue Z Xipeng Qiu
Zheng Zhang
Fudan University
UC Santa Barbara
Shanghai AI Laboratory
Westlake University O Amazon AWS AI
tengxiao@ucsb. edu zhangrue@vestlake edu.ed.
xpqiuOfudan. edu. en zhaz0amazon.com
Abstract
Trained on vast corpora of human language, language models demonstrate emer- gent human-like reasoning abilities. Yet they are still far from true intelligence, which opens up intriguing opportunities to explore the parallels of humans and model behaviors. In this work, we study the ability to skip steps in reasoning hallmark of human expertise developed through practice. Unlike humans, who may skip steps to enhance efficiency to reduce cognitive load, models do not inherently possess such motivations to minimize reasoning steps. To address this, we introduce controlled framework that stimulates step-skipping behavior by iteratively refining models to generate shorter and accurate reasoning paths. Em- pirical results indicate that models can develop the step skipping ability under our guidance. Moreover, after fine-tuning on expanded datasets that include both com- plete and skipped reasoning sequences, the models not only resolve tasks with increased efficiency without sacrificing accuracy, but also exhibit comparable and even enhanced generalization capabilities in presents the first exploration into human-like step-skipping ability and provides fresh perspectives how such cognitive abilities can benefit Al models.
scenarios. Our work
1 Introduction
The pursuit of Artificial General Intelligence (AGI) is profoundly influenced and inspired by human intelligence [35] 6). Trained extensively on human language, language models not only excel in various tasks, but also begin to exhibit emergent human- like abilities that are not explicitly engineered
into them (24]. Among these, reasoning stands out as a core human-like cognitive ability, and has
demonstrated great potential in a wide range of problem solving scenarios [47 [1 60
37 28 34)
Despite their advances in displaying human-like like cognitive activities, huge gaps remain in how models
and humans actually behave (22 46 20). These differences bring up interesting questions regarding the exploration and development of similar capabilities between models and humans.
We aim to investigate whether the models exhibit any reasoning abilities unique to human experts, and whether they can evolve from beginners to reasoning experts. When humans learn to reason, beginners typically start with detailed, step-by-step solutions to imitate the gradual process of problem solving. As practice makes perfect, human experts not only solve problems more swiftly but also utilize shorter mental pathways, often skipping steps in their reasoning process [36]. This particular ability helps them speed up the reasoning and saves cognitive load for more challenging steps [44] As demonstrated in Figure [] the hep-skkiping behavior illustrated on the right side is commonly adopted by human experts during equation simplification.
Work done during internship at AWS Shanghai Lah. Cooresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Q: *
D F
Q: A B x C- D
F
Solve it in steps.
Solve it in steps.
Step1: x
C
F / F / C /
D
Step1: * x (Skip)
D C A A
2
Step2:
B X
=
D
X =
D
/
F C
Step3:
D
/
F
C / A
A
(Skip) x
=
D
F
Step4:
Step2:
/ F
C
Figure 1: Step skipping equation simplification. We use the specified number of steps in the input as stimulation to induce the model to perform skipping by using fewer steps.
In this work, we are curious whether models exhibit mature human-like reasoning a skipping steps, and how such abilities can influence the model's reasoning behaviors. Unlike humans, models do not inherently possess the intrinsic motivation like time limit or skill maturity that naturally drives efficiency cognitive tasks. To induce the skipping step behavior in models, we introduce controlled training environment where models are instructed to generate reasoning sequences within a specified number of steps. Our method includes two phases: initialization and iteration. We begin with dataset that contains complete stepwise reasoning processes for the questions. In initialization, models are first trained solve the tasks comprehensively. adhering to the full sequence of reasoning steps. In Figure D the illustration on the left demonstrates how models are trained to follow specified number of steps. Then in the iteration phase, the models are prompted to produce shorter answers based on the original training data (Figure[rrigh) We then select the shorter reasoning paths that still achieve correct answers and mix them with the full-step reasoning paths. This expanded dataset is used to train new model to have advanced step-skipping capabilities. Each iteration refines the model's s ability to identify how steps can be skipped without sacrificing accuracy. Finally, fine-tune the models using these iteratively generated datasets, including data instances that demonstrate successful step-skipping during each iteration.
We conduct experiments with three different reasoning datasets, each characterized by clear internal reasoning steps, to evaluate model behaviors. Empirical results demonstrate that models exhibit and develop the ability of skipping steps in our framework not only solving tasks effectively but also actively omitting steps to enhance efficiency. Further analysis of model behaviors indicate that these skipped reasoning paths act as beneficial enhancements rather than mere biased shortcuts, as evidenced by their maintenance or even improvement of out-t--distribution (OOD) performance across various tasks. To the best of our knowledge, this work is the first investigation into the human- like ability of step-skipping in language models, providing empirical evidence that models can indeed skip steps. These preliminary findings provide fresh perspective easy-to-hard generalization training models on simpler data comprising both comprehensive and skipped reasoning steps can enhance their ability to generalize to more complex scenarios.
2 Related Work
Human-like Abilities in Language Models Many of the capabilities widely used in current models are inspired by human intelligence. For instance, in-context learning enables models to address problems by mimicking the patterns demonstrated in examples
[5]. In reasoning tasks, models benefit from progressively answer derivations and step-by-step chain-of-though processes [47] and their humanlike enhancements, such as planning (18), task decomposition (50), and refinement [32 38
Another series of studies explore from the perspectives of cognitive science and psychology 10
12 9.. Kosinski 24 reveal that current large language models have demonstrated certain level of Thfory-of-Mind (ToM) abilities by testing their performance to impute another's 's mental states and perspectives. Further studies [21] provide preliminary evidence of correlation between the embeddings in LLMs and human brain neurons during ToM tasks, while Ma et al. [31] highlights the limitation of current ToM evaluations they target narrow and inadequate aspects of ToM. Apart
from these cognitive abilities, our work draws inspiration from human problem solving (3 42 3 44 and evaluates language models these unique step skipping behaviors. Additionally, aligns with an expanding field exploring the correlation between System I and System 2 reasoning
work
"Code and data publicly available at: httppp//ggttttoo.co//////gsaaaas

---

Initialization
Iteration
finetuning
Mo Matandard
Mytandard
Mstandard
inference
standard finetuning
Dinit
D1 Du + Do
D2 Dg + Dj'
Solve in steps
Do
D2'
Solve it in n steps
Mo
1
M₂
Figure 2: Overall framework. The initialization phase aims to equip the model with the ability to reason according to a specified number of steps. During iterations, each cycle produces a mixed dataset D, which is used to train standard model to evaluate the model's step-skipping capabilities.
mechanisms 14 15 49.. Rather than removing all reasoning trajectories, our work explores gradual shortening to provide smoother transition that mirrors natural cognitive processing.
Compositional Generalization Challenges Transformers have shown limitations in complex compositional generalization scenarios [17 [9]. Previous work also indicates that models may
develop biased shorteut, negatively impacting their OOD performance 22 25 16] A growing body
of research focuses on easy-to-hary tord generalization 4 7 19
41 48), where models improve their generalization ability by learning from easy tasks, without requiring intensive supervision on harder ones. Following this line, our work encourages the model to learn from self generated skipping paths, which has been empirically shown to maintain and even enhance OOD) generalization capabilities.
3 Method
Humans develop the ability to skip steps for several reasons. With practice in specific tasks, they evolve from novices to experts, optimizing lengthy thought processes into quicker, more efficient reasoning. Additionally, factors such as time constraints or the desire to conserve cognitive resources can also prompt humans to skip steps [13 In contrast, models lack an inherent cognitive signal that would drive them to minimize reasoning steps. Rather than attempting to replicate these human-like like signals, design training approach to directly control the number of steps in their reasoning processes. By restricting the steps in model responses, we can guide the model to self-generate data including skipped steps. Our framework has two phases: initialization and iteration.
3.1 Initialization
We begin with training dataset Do. which contains detailed full-step reasoning answers to the questions. Our goal to train model that can generate answers by following the specified number of steps in the input question. Depending the characteristics of different tasks, there are two design choices to initialize the framework: cold start and warm start.
Cold start In the cold start approach, we directly fine-tune the model the original full-step training data, i.e., Danst = Do. The trained model is expected to not only learn to solve the problems, but also adhere to the specified number of steps in the input instructions.
Warm start Training exclusively with full steps does not always guarantee the ability of controlling the number of steps, especially for the challenging tasks. Therefore, we manually create answers that contain skipped steps based on human expertise. Optionally, we can also randomly merge adjacent steps or simply omit random steps within the rationales to create such skipped-step data. In either way, we can expand the original training set with additional data Dekip that can better help models
3

---

learn how to solve the problems with fewer steps. Thus, the data for warm start initialization can be describes Dinit Do Dakip-
Using the prepared data, we fine-tune the model to generate the answers with the given number of steps. For each QA pair in Dinit, the question q is concatenated with the instruction I, which
indicates that the reasoning process a! should be completed in steps. Therefore, the resulting model in the initialization phase, Mo, is described as:
Mo II
P(a q In);
(1)
(g.ain))EDo
where the instruction In stands for "Solve it in steps".
3.2 Iteration
After the initialization, the model expected to have learned to solve the problems with detailed steps using the specified number of steps in the input. Leveraging this particular ability, we can encourage the model to actively engage in step skipping behavior. At the beginming of each iteration k, the model M is prompted to solve the same problems in the training set using fewer steps than the full number. Responses that are both correct and meet the reduced step criterion are filtered and composed into new dataset D/. These reasoning answers generated solely by the model itself, reflecting its understanding after training on the initialized data and demonstrating its active preferences when reducing steps.
We define the dataset used for current iteration as D3 = DoUD where the original training set Do includes full reasoning steps and the filtered dataset D/ contains new responses that successfully utilized fewer steps. This ensures that the model has access to both the original complete reasoning processes and examples of effective step-skipping generated by the model itself. To finalize current iteration, the model Mg is trained on Dg: M2- II [l(ga(())). P(a(")|g, 4n).
The iterative training process described above requires specifying the number of steps in the input, which is impractical in real-world applications because it can be difficult to determine the exact number of steps needed for given question. To be more applicable, we aim understand how models learn from the generated skipped data and what benefits they can derive from it. Therefore, for each intermediate resulting dataset DR. we train a new model using a standard QA finetuning setting without specifying the number of steps in the input:
Matandard
II
P(a!")
k
(g.a()))ED,
This phase aims to solidify the model's skipping behavior, simulating more advanced form of cognitive processing akin to human reasoning.
4
Experiments
4.1
Datasets
We design three tasks to investigate the model's step skipping behavior (Figure 3). In each task, the intermediate steps needed to solve these problems are explicitly detailed and defledd, facilitating a clear analysis of the model's predictions. When creating skipped data for warm start, we either omit certain steps or heuristically merge two adjacent steps. Details on data creation can be found in Appendix B.
Analog of Algebra Following Blessing and Anderson we create an analog of algebra by replacing the variables and operators with different symbols. As shown in Figure each variable and standard operator mapped
unique, unrelated symbol. The desired result is isolate the symbol v (i.e., I)) on the left side of the symbol ++ (i.e., =). This task is entirely new for the model, making it an ideal scenario to understand how models develop problem-solving abilities from scratch. We use heuristic script to generate the questions along with the stepwise solutions. After generating the QA pairs, we filter the data based on the number of variables involved in the question and the steps required to solve it. The training and in-domain test data contains questions with up to

---

Analog Algebra
Multi-digit Addition
Directional Reasoning
Q: 23 67
Initial orientation facing
(A x)
North. ... execute actions: Right, Around, Left
Answer:
Answer:
1. 7 (plus carry 0) = 10
1.
write down O, carry over 1.
Answer:
[C
2. 2 + 6 (plus carry 1) =
1. East. Around, Left.
2.
M
write down carry over.
2. West. Left.
A< B
Thus, the answer is 90.
South.
Figure 3: Illustrations of three different tasks. Each question is accompanied by a comprehensive detailed step-by-step solution.
variables and requiring no more than 5 steps. addition, we create two out-of-domain datasets of varying difficulties to evaluate generalization performance: OOD-easy includes variables unseen during training, with and variables, no limit on steps. OOD-hard is the most challenging setting, including 10 14 variables and > 9 steps solve. Both OOD sets contain unseen variables.
As basic arithmetic task, multi-digit addition naturally involves detailed stepwise reasoning processes, serving as a suitable task for studying model behaviors in composi- We utilize ste--bystee reasoning processes to perform addition operation digit by digit, as illustrated in Figure 3 For training and in-domain test data, we only consider additions involving numbers with up digits. We introduce two out-of-domain datasets depending on the number of digits involved in the addition: OOD-easy includes one number with up digits and another with 4-7 digits. OOD-hard contains two numbers, both with 4-7 digits.
Multi-digit Addition
tionality generalization 45
Directional Reasoning We additionally consider long-form symbolic directional reasoning, which poses challenge for direct solution and necessitates continuous reasoning steps to arrive the answer. This task provides an initial direction and list of turning actions. The desired answer is the final facing direction. For training and in-domain test set, we consider questions that contain < 10 actions. OOD-easy includes questions with 11-20 actions and OOD-hard includes questions with 21-30 actions. The detailed statistics of three datasets can be found in Table
Table 1: Dataset statistics.
Task
Train
In-domain test
OOD-easy
OOD-hard
Analog of Algebra 5,770
1,000
2,000
420
Multi-digit Addition
2,885
1,000
1,200
1.600
Directional Reasoning
2,080
1,000
500
500
4.2 Experiment Setting
For all our experiments, use Llama 2 (7B parameters) 43 and phi-3-mini (3.8B parameters, with context length of 4K) [T] as our base model. We train the model using learning rate 5e-6 for epochs with the Adam optimizer [29]. During inference, we employ greedy decoding. We run our experiments with three different random seeds and report the average and standard deviation. All experiments conducted on eight V100 GPUs each with 32GB memory. The total training time required to complete one full cycle five iterations is under six hours.
Results
5.1
Can models learn to skip steps?
To make sure our framework can proceed to iterations smoothly, one crucial factor is the initialized model's 's ability to adhere to the specified number of steps in the input. In the cold start setting,
5

---

Table 2: Step number following ability of the initialized Llama2 models across different tasks. "# Skipping" represents the number of instances where n 2 >> O. "Step Consistency" quantifies the match between the actual number of steps taken and the number indicated in the input. "Answer Accuracy" calculates the percentage of correct final answers out of the "# Skipping" cases. "Average Step" reflects the mean number of steps across all predictions within the dataset.
Analogy of Algebra
Multi-digit Addition
Directional Reasoning
i=
i= 2
i
1
i= 2
i
1
i= 2
# Skipping
5,308
4,159
2,844
2,175
2,071
2,049
Stop Consistency
100.00
99.19
100.00
100,00
86.24
39.19
Answer Accuracy
8.14
2.77
98.35
82.58
85.47
29.62
Average Step
2.33
1.81
1.90
1.38
6.14
6.66
train the model exclusively using the full step training data. We then run inference on the training set, instructing the model use i steps to solve the question, where n denotes the original full step number and i E [1. 2]. If n - -is S O, we do not ask the model to try skipping on such cases and instruct the model to use n steps instead.
As shown in Table 2 the results demonstrate that the fine-tuned model exhibits good step-number following ability on the Analog of Algebra over 99 % of the answers follow the given number of steps. Additionally, when prompted to generate condensed answers with fewer steps, the model can produce some correct answers in the specified number of steps, achieving accuracies of 8. 8.14% and 2.77% respectively. Despite this relatively low accuracy, these small amount of correct data can still assist the model in gradually developing step skipping ability through iterations. Ultimately, the model manages to produce over 90% of correct skipping data. The trend of the data quantity change can be found in Appendix B.2
However, this ability varies across different tasks. For the other two tasks, models do not naturally develop the capability for active step skipping, leading to near zero step consistency when required to provide answers in fewer steps. To address this issue, we employ the warm start setting for these tasks. Tablee[present the results of Multi-digit Addition and Directional Reasoning under the warm start setting, indicating that this approach enhances the models' proficiency with step skipping.
Ideally, we aim for models to be initialized through cold start. The benefits of this approach are obvious- - allows the model to spontaneously develop step skipping behavior, giving it sufficient freedom to decide and control which steps to skip. However, our experiments have revealed that can be challenging for models to develop such capability in all scenarios. In contrast, the warm start offers alternative design choice by providing human- intuitive and valid skipping steps derived from human expertise, making it more natural and helping models develop
skipped data. This data includes
understandabll behaviors. However, it might also introduce human biases that constrain the model's independent exploration of step skipping. This influence can be mitigated in the subsequent iteration phase, where the model is given full freedom to develop and amplify its own step-skipping behavior.
5.2
What do models learn from skipping steps?
Based on this new mixed data including both complete and skipped answers at each iteration, we train the standard models to analyze the change of model's performance what models can learn from the behavior of skipping steps.
Models learn to solve problems more effectively with fewer steps. We evaluate the standard models on both in-domain and OOD data, with the results presented in Table Detailed results from each iteration of the evaluation can be found in Appendix B.3 Given the simplicity of the tasks, the model is able to overfit on in-domain data, achieving nearly perfect performance. Further iterations of skipping steps manage to guide the model to use fewer steps while maintaining the performance. In two OOD scenarios, we find that the model trained with mixed data performs comparably to the model trained with complete steps on the OOD test sets, and even exhibits superior generalization abilities. Specifically, in Analog of Algebra, Llama2 models of iteration 5 achieves 4.765 4.76% gain on OOD-easy, while phi-3-mini achieves 7.08% gain on OOD-hard set. In the Multi-digit Addition task, the Llama2 model demonstrates a 13.91 1 % improvement in OOD-casy performance

---

Table Performance comparison of models from different phases. Avg steps denotes the average number of steps taken in the prediction. With the skipped step data, models achieve even better generalization performance with fewer steps.
In-domain
OOD-easy
OOD-hard Avg steps
Task
Iteration
Acc
Avg steps
Ace
Avg steps
Acc
Llama2-7B
Analog of Algebra
Cold start Iter 5
99.87
3.19
85.91
4.79
7.94
11.57
99.80
2.43
90.67
4.05
8.10
10.92
Cold start
100.0
2.86
0.06
3,25
0.00
3.69
Multi-digit Addition
Warm start
99.53
2.72
0.14
3.02
0.11
3.49
Iter
99.17
1.46
13.97
1.49
4.75
2.06
Cold start
100.0
7.01
90.00
15.77
42.00
19.39
Directional Reasoning
Warm start
99.97
6.28
87.20
14.65
42.33
18.02
Iter
100.0
6.45
89.33
14.87
51.80
19.49
phi-3-mini
Analog of Algebra
Cold start Iter 5
99.60
3.19
98.04
6.16
4.05
10.01
99.90
2.75
98.95
5.60
11.13
7.98
Cold start
99.92
2.86
35.93
5,03
5.39
5.44
Multi-digit Addition
Warm start Iter
99.97
2.62
39.08
3.80
5.11
4.06
99.93
2.08
46.61
2.31
14.98
2.59
Cold start
99.83
7.01
91.47
15.46
62.67
24.85
Directional Reasoning
Warm start
99.80
6.82
93.67
15.19
71.80
24.61
Iter 99.70
6.12
93.73
14.44
73.87
23.77
(a) Analog of Algebra
(b) Directional Reasoning
18
cart
Panantage
Step ength of compante answers
Action length questions
Figure 4: Comparison of models across different phases relative to question length and complexity. Models achieve near perfect performance on in-domain data but diverge lengthy OOD data.
and 4.75% increase in OOD-hard performance. the OOD-hard dataset for Directional Reasoning, Llama2's performance improvs by 9.2%. These results suggest that not only is the model unaffected by potential shortcut bias from the skipping steps, but it actually benefits from the mixed training data to gain enhanced task solving abilities. The ablation analysis various data-mixing approaches are provided in Appendix B.5) Furthermore, we observe that the model uses fewer steps, thereby increasing problem-solving efficiecc.
5.3
Model Behavior Analysis
5.3.1 Analog of Algebra
Figure aa) presents the performance of Llama2 models across various iterations in the Analog of Algebra task, differentiated by the number steps required represent the accuracy of final answers. We perform uniform evaluation on the union of all in-domain and OOD test sets. Initially, all models maintain high accuracy for in-domain problems with up to five steps, after which significant drop is observed the complexity increases. As the model undergoes
complete answers. The solid lines
7

---

iterations, there is a noticeable improvement in its ability to handle longer step lengths (green solid line), particularly in the range of 6 to 10 steps where other models show significant weaknesses. The dashed lines illustrate the proportion of data exhibiting step-skipping in model predictions. The blue dashed line indicates models initially adopt step-skipping as problems extend in length. After iterations, the green dashed line indicates the models consistently employ step skipping in shorter questions, thereby improving the reasoning efficiency.
5.3.2 Directional Reasoning
Figure 4 (b) illustrates the comparison of Llama2 model's performance across different question lengths on Directional Reasoning task. We observe that the artificial skipped data has minimal impact on the model, with negligible differences between the cold start and warm start phases. Upon entering the iterative phase, the model's performance notably declines during the first iteration, particularly in handling longer problems. This downturn may reflect the model's ' adjustment from manually injected skipped data to its own step skipping ability. Subsequent iterations show that the model benefits more significantly from data generated during the iteration process, as evidenced by the results in Iteration 5. The model maintains consistency with the baseline in both in-domain and off-oand performances, and exhibits slight advantage in solving longer problems. Similar to the previous task, the Iteration 5 Ratio curve (dashed green line) also shows significant increase in step-skipping behavior, suggesting an evolved efficiency in reasoning as the model opts to bypass steps while maintaining or even improving accuracy.
5.3.3 Multi-digit Addition
we show a finer-grained evaluation of multi-digit addition tasks on Llama2. The horizontal and vertical axes of the matrices represent the number of digits in the two addends for each question in the test set (both in-domain and OOD test data). We utilize the following three metries: Question-level accuracy assesses whether the final answer is correct for additions involving different numbers of digits. Step-level distribution illustrates the distribution of the digit lengths used in each individual step of the model's 's stepwise reasoning process. Step-level accuracy measures the accuracy of the single step calculations involving different numbers of digits.
In Figure 5
In Figure 5 (),, iterations progress, the model demonstrates improved generalization performance across all test datasets. When initialized with a cold start, the model can only learn from the training data involving single-digit addition steps, resulting in overfitting in-domain test data (digit < 3). When augmented with manually created skipped data for a warm start, the model begins to incorporate multi-digit additions with skipped steps. However, the inconsistency between the manually injected data and the model's inherent behavior does not significantly enhance the question-level accuracy. As the model is encouraged explore during the iteration phase, it undertakes broader and bolder attempts-often combining additions across more digits in skipped steps. With the integration of these data, the model trained on this expanded iterative dataset also shows a more pronounced ability to solve OOD problems. As seen in Figure b), the model increasingly employs multi-digit additions in single-step operations. Furthermore, as illustrated in Figure FigureBjc). there is an improvement in the accuracy of these skipped single-step operations. We believe this may be due to the model generated data during self-iterations, which are more conducive to enhancing its capability to skip steps, thereby benefiting from this process.
5.4 Accuracy of Step-Skipping Answers
Figure [ggure//shower shoss the step skipping behavior and accuracy of the standard models at each iteration on the Analog of Algebra task using Llama-2. The Skipping Ratio measures how often the model skips steps in the test set, while Accuracy reflects the correctness of these skipping answers.
We observe that in the beginning models inherently struggle in OOD scenarios. often producing reasoning steps that are incomplete or shorter than the problem complexity requires. In "cold start" settings, where the model trained solely with complete steps. it performs well with in-domain questions but fails to maintain complete reasoning steps and tends to generate shorter responses on OOD sets. Due to its limited generalizability, these skipping or missing steps negatively impact the performance. However, as the model progressively adapts to step skipping over iterations, the accuracy of the shorter responses improves, suggesting it gradually develops a more reliable ability to skip steps when appropriate. Analysis across all tasks can be found in Appendix B.4
8

---

Cold start
Warm start
Iteration
Iteration
1:23
2.04
C.0
1:2
X:9
4:00
5:04
523
1-4
1:0
0:09
3:0
0.00
0:99
6:30
020
0.00
0.00
[a] Question-level accuracy
DCs
SJB
3:00
3C2
8:20 S4N
0.00
1:04
2:2
1AT
1M
K
0:2
0:49
9%
144
(b) Step-level distribution
S442
309
0:00
3C2
0:00
3:00
0.0
3:0
3:00
ACB
5:11
1.0
12
373
6:30
171
8:3
0:00
3:18
0.00
2
1:08
0.23
0.00
(c) Step-level accuracy
Figure 5: Model behavior analysis on the test of multi-digit addition task. Initially constrained to single-digit additions. the model progressively incorporates multi-digit calculations with skipped steps through iterative learning, showing an enhancement in solving out-of-distribution problems and executing more complex calculations with higher accuracy.
40
Figure 6: Skipping ratio and the accuracy of the skipping responses on Analog of Algebra.
5.5 Analysis on the Influence of Training Steps
Throughout the iterations, as the model progressively generates more successful step skipping data, the size and the quality of the resulting dataset also gradually increases. This can be considered as a special form of augmentation for answer diversity. To investigate whether the performance improvements are primarily due to the model learning from more training steps, we increase the number of training epochs during the initialization phase to match the data volume after iterations. The comparison results shown in Table reveal that increasing the number of training epochs does not always lead to performance enhancements; instead, it may cause a performance decline due to overfitting. In contrast, mixing skip-step data from the iterative process not only maintains or improves performance in in-domain and OOD-easy -eay tasks but also achieves consistent gains in OOD- hard setting. When the total number of training steps is similar, the integration of skipping data yields better performance.
9

---

Table 4: Performance comparison across different tasks with varying training steps.
Task
Iteration
steps
In-domain
OOD-easy
OOD-hard
Cold start ep4
2.9K
99.9
89.7
4.5
Analog of Algehra
Cold start ep5
3.6K
100
84.9
2.4
Iteration 5 ep2
3.3K
99.8
90.5
14.3
Cold start cp5
1.8K
100
Cold start ep6
2.2K
100
0
Multi-digit Addition
Warm start ep2
1.4K
99.9
0
0.1
Warm start ep3
2.1K
100
0.1
Iteration 5 ep2
2.0K
99.5
13.5
5.8
Cold start ep3
0.8K
100
91.2
43.2
Cold start ep4
1.0K
100
91.0
34.8
Directional Reasoning
Warm start ep2
1.0K
43.4
Warm start ep3
1.5K
100
84.6
34.4
Iteration 5 ep2
1.0K
100
90.4
56.2
5.6 Extended Iterative Training
In this section, extend the iterative process allow the model to skip up steps, rather than restricting it to less than 2 steps on Analog of Algebra. The process is continued for total of 9 iterations, and the results shown in Figure 7 The model continues benefit from additional iterations beyond Iteration 5, which serves as the default cutoff in our main results. Specifically, the accuracy on the OOD-hard set improves steadily, reaching over 18% by the ninth iteration. This increase suggests that even with a greater allowance for step-skipping, the model's 's ability to generalize to harder out-of-domain samples enhanced with continued training.
Simultaneously, the average number of steps taken decreases across all test sets as iterations progress, suggesting that the model is converging towards fewer steps and becoming increasingly efficient. By the ninth iteration, the step count appears to plateau, indicating that the model has likely reached stable balance between accuracy and efficiency. We hope our work provides fresh perspective on exploring the connection between System 2 slow reasoning and System fast thinking, and on facilitating their transformation, paving the way for future research this direction.
1.0
AdC-
333
325
3
Figure 7: Performance of phi-3-mini across 9 iterations with relaxed deipskipping constraints (up to 4 steps) on Analog Algebra. The figures show the changes average steps taken (left y-axis) and accuracy (right y-axis). Continuous iteration improves OOD-hard accuracy and reduces the average number steps, converging towards stability.
Conclusion
In this work, we explore the human-like ability of step skipping in language models, providing initial empirical evidence that models can skip steps and benefit from such cognitive behaviors. Addressing the absence intrinsic motivation for step skipping in models, we design an approach that not only enables models to spontaneously develop the ability but also iteratively encourages models to actively adopt and enhance this behavior. Through experiments on three tasks, we demonstrate that models equipped with step-skipping capabilities can solve tasks more efficiently in fewer steps, without sacrificing accuracy. Further empirical results suggest that training on easy data containing both full steps and skipped reasoning steps can potentially help models generalize to harder scenarios. We hope this work offers insights into the relationship and transition between System and System thinking and contributes to advancing casy-to-hard generalization in language model reasoning.
10

---

References
[1] M. I. Abdin, S. A. Jacobs, A. A. Awan, J. Aneja, A. Awadallah, H. Awadalla, N. Bach, A. Bahree, A. Bakhtiari, H. S. Behl, A. Benhaim, M. Bilenko, J. Bjorek, S. Bubeck, M. Cai, C. C. T. Mendes, W. Chen, V. Chaudhary, P. Chopra, A. D. Giorno, G. de Rosa, M. Dixon, R. Eldan, D. Iter, A. Garg, A. Goswami, S. Gunasekar, E. Haider, J. Hao, R. J. Hewett, Huynh, M. Javaheripi, X. Jin, P. Kauffmann, N. Karampatziakis, D. Kim, M. Khademi, L. Kurilenko,
J. R. Lee, Y. T. Lee, Y. Li, C. Liang, W. Liu, E. Lin, Z. Lin, P. Madan, A. Mitra, H. Modi, A. Nguyen, B. Norick, B. Patra, D. Perez-Becker, T. Portet, R. Pryzant, H. Qin, M. Radmilac, Roy, O. Ruwase, O. Saarikivi, A. Saied, A. Salim, M. Santacroce, S. Shah, N. Shang, H. Sharma, X. Song, M. Tanaka, X. Wang. R. Ward, G. Wang. P. Witte, M. Wyatt, C. Xu, J. Xu, S. Yadav, F. Yang, Yang, D. Yu, C. Zhang, C. Zhang. J. Zhang, L. L. Zhang, Y. Zhang, Y. Zhang, Y. Zhang, and X. Zhou. Phi-3 technical report: A highly capable language model locally on your phone. CoRR, abs/2404. 14219, 2024, doi: 10.48550/ARXTV.24400 14219. //doi. org//o///// 1...48550//r.1i 1.940.142111
C. Rosset,
[2] M. Binz and E. Schulz. Using cognitive psychology to understand GPT-3. CoRR, abs/2206. 145776, 2022. doi: 1048550/ARXIV2200.445577 URL https: /ttp::///oooor/1000
@6550/arXiv 2206. 14576
[3] S. Blessing and J. R. Anderson. How people learn to skip steps. Journal of Experimental Psychology: Learning. Memory and Cognition, 22:576-596 598, 1996. URL https: /saps://api
semanticscholar Trg/CorpusID:: 5558448811
[4] S. R. Bowman, J. Hyun, E. Perez, E. Chen, C. Pettit, S. Heiner, K. Lukosiute, A. Askell, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, C. Olah, D. Amodei, D. Amodei, D. Drain, D. Li, E. Tran-Johnson, J. Kernion, J. Kerr, J. Mueller, J. Ladish, J. Landau, K. Ndousse,
L. Lovitt, N. Elhage. N. Schiefer, N. Joseph, N. Mercado, N. DasSarma, R. Larson, S. McCan- dlish, S. Kundu, S. Johnston, S. Kravec, S. E. Showk, S. Fort, T. Telleen-Lawton, T. Brown, T. Henighan, T. Hume, Y. Bai, Z. Hatfield-Dodds, B. Mann, and J. Kaplan. Measuring progress on scalable oversight for large language models. CoRR, abs/2211.033555 10.48550/ARXIV222) 1. .03540. URL https:///0/. org 100gg/1..oo 48550/ 48550/arXiv arwiv. 22112222200011
2022. doi: 2211..350000 .3540
[5] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, Agarwal, A. Herbert- Voss, G. Krueger, Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. Bal- can, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020. URL /ttp:////// nwwrips httpps.ccppaee
[6] S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. M. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4 CoRR, Abs/2303.11713 2023. doi: 0rg/10.4440 111111....0//1111 arXiv. 2303. 12712
10.44550//RXXI.23300. 12722. httpss/:::///do
[7] C. Burns, P. Izmailov, J. H. Kirchner, B. Baker, I.. Gao, I.. Aschenbrenner, Y. Chen, A. Ecoffet, M. Joglekar, J. Leike, I. Sutskever, and J. Wu. Weak-to-strone generalization: Eliciting strong doi: 10.48550/ARXIV.2312.
capabilities with weak supervision. CoRR, abs/2312.093900, 2023.
09390. URL httpp://///o111 org/ 48550/ 2312.
[8] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman. Training verifiers to solve math word problems. CORR, abs/21 10. 14168, 2021. URL https:t/atsxiiiii httpp/////aara..o//aa/////////// wwtp:////////b/./oo/// 14168
[9] Coda-Forno, M. Binz, X. Wang, and E. Schulz. Cogbench: a large language model walks into a psychology lab. CoRR, abs/2402. 18225, 2024. doi: 1044850/ARXIV22402..88
URL
https://áoi.org:: org/10. 00wg//00.8550////
arxiv. 2402 18225
[10] K. M. Collins, C. Wong, Feng, M. Wei, and J. B. Tenenbaum. Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks. ArXiv, abs/2205.05718 2022. URL, hhtpps://pp
semanticscholar. wtw.:ooaaw.rrorrou....ii:0... 2487221155

---

[11] A. Creswell, M. Shanahan, and I. Higgins, Selection-inferenn Exploiting large language models for interpretable logical reasoning. CoRR, abw.22as.099//11210 2022. doi: 10.48550/arXiv.
2205.09712. URI. http:httts::/tt //tdps::/ddd. org/ /0/ 48540//55Xa1 2205.09712
[12] I. Dasgupta, A. K. Lampinen, S. C. Y. Chan, A. Creswell, D. Kumaran, J. L. McClelland, and
Hill. Language models show human-likk content effects on reasoning. ArXiv, abs/2207.07051,
2022. URL URIhttps:: //atps::/ap.... http:/pp...maaiiiiiiieeammmaa www.ntiischoolsttie.....ii
org/ Saapg/CoppusID- 2D: 250526626
[13] w. De Neys. 223CI The Cognitive Unconscious and Dual Process Theories of Reasoning, In
The Cognitive Unconscious: The First Half Century, Oxford University Press, 08 2022. ISBN
9780197501573 573, doi: 10. 0009//ss//78011000
URL http::h/tooo. /tttts:://doooor/100
000.0003..000 0011
[14] Y. Deng, K. Prasad, R. Fernandez, Smolensky, V. Chaudhary, and S. M. Shieber. Implicit chain of thought reasoning via knowledge distillation. CoRR, abs/231 1,01460, 2023, doi: httpss:t http///iiiooro//////// 48550/ 118850/arXivv 201...11.1111
10.48550/ARXI.23111..
[15] Y. Deng, Y. Choi, and S. M. Shieber. From explicit cot implicit cot: Learning internalize cot step by step. CoRR, abs/2405. 14838, 2024. doi: 10.48550/AXXI.22005.60
URL
https://dof wrg/10.///0 48550/ 119650/arXivv arxiv. 2405 14838
[16] M. Du, F. He, N. Zou, D. Tao, and X. Hu. Shortcut learning of large language models in natural language understanding: A survey. CoRR, abs/2208. 11857, 2022. doi: 10.48550/ARXIV.208.. 11857. httpss/ts::////t
org/10 10. 18850/arXivy 2208 11857
[17] N. Dziri, X. Lu, M. Sclar, X. L. Li, L. Jiang, B. Y. Lin, S. Welleck, P. West, C. Bha- gavatula, R. L. Bras, J. D. Hwang, S. Sanyal, X. Ren, A. Ettinger, Z. Harchaoui, and Y. Choi. Faith and fate: Limits of transformers
compositionality. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances Neural Informa-
in Neural Information Processing Systems 36: Annual Conference tion Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16. 2023, 2023. URL http:// nips wwtpcccpppppeeeeeeeeeeeee www.ffftweeeee/e/eeeeeeeaaaaeeee pwwppppppp///aaaapppppppp www.3315cbb663c11e6
[18] S. Hao, Y. Gu, H. Ma, J. Hong. Z. Wang. D. Z. Wang, and Z. Hu. Reasoning with language model is planning with world model. In H. Bouamor, J. Pino, and K. Bali, editors. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 8154-8173, Association for Computational Linguistics. 2023. doi: 10.18653/V1/2023 EMNLP-MAIN.507, URL 2023. smnlp-main.55 507
[19] Hase, M. Bansal, P. Clark, and S. Wiegreffe. The unreasonable effectiveness of easy training URL
data for hard tasks. CORR, abs//401 abs/2401.06751, 2024. doi: 10.48550/ARX112240000677
https: //dps://doi.. org/10. 48550/ arXiv. 2401 2401.06751
[20] I.. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng. X. Feng. B. Qin, and
T. Liu. A survey on hallucination large language models: Principles, taxonomy, challenges, and open questions. CoRR, abs/231 1.0232,, 2023. doi: 10.48550/AXXV.2311.0555 https://doi httpp::///oooo///0..../ 48550/ 111650/arXivv arxiv 2311 05232
URL
[21] M. Jamali, 2. M. Williams, and J. Cai. Unveiling theory of mind in large language models: A parallel single neurons in the human brain. CoRR, abs/2309.01660, 2023. doi: 10.48550/
ARXIV.2309.0 0666. URL htttt::////aaat hwweddi.......geooeeee org/ /01 008444555/aii1 230.ii....3.....0 01660
[22] Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. Bang, A. Madotto, and P. Fung. Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12):248:1-248:
2023. doi: 10. 14533571173
https::////oi
1145/3571730
[23] K. R. Koedinger and J. R. Anderson. Abstract planning and perceptual chunks: Elements of expertise in geometry. Cogn. Sci., 14(4):511-550, 1990. doi: 10.1207/1151167990001442 URL (RLLhhtps:
10. 110..1007/1155117700001/44////
[24] M. Kosinski. Theory of mind may have spontanecussly emerged in large language models. https://://doi
CoRR, abs/2302. 2023. doi: 10.48550ARRIV.220022000 10. 48550/arXiv, www.xii.......... 02083
[25] Y. Lai, C. Zhang, Y. Feng, Q. Huang, and D. Zhao. Why machine reading comprehension models leamn shortcuts? In C. Zong, F. Xia, W. Li, and R. Navigli, editors, Findings of
12

---

the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 2021 of Findings of ACL. pages 989-1002. Association 10L FINDINGGS-ALL.... URI 18653/ w122021. htt.//222...innnngsse.ee www.iiinnnnnc....cc
1-6, 2021, volume ACL/IJCNLP
for Computational Linguistics, 2021. doi: 10. 18613//11/2221
https://doi..og
[26] N. Lee, K. Sreenivasan, J. D. Lee, K. Lee, and D. Papailiopoulos. Teaching arithmetic to small transformers. CoRR, 2ttp22377.0331111 2023. doi: 10.48350/ARXIV22307.000 URL
https:///oi. org/10. 48550/ 48550/arXiv arXiv 2307. 03381
[27] B. Liu, T. Ash, S. Goel, A. Krishnamurthy. and C. Zhang. Transformers learn shortcuts to automata. The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL Stttps://openrevieet net.
pdf?id-De4FYgjFue
(28) T. Liu, Q. Guo, Y. Yang, X. Hu, Y. Zhang, X. Qiu, and 7. Zhang. Plan, verify and switch: Integrated reasoning with diverse x-of-thoughts. In H. Bouamor, J. Pino, and K. Bali, edi- tors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, EMNLP 2023, Singapore, December 6-10, 2023, pages 2807-2822. Association URL
for Computational Linguistics, 2023. doi: 10.18653/V 12W.:///222220MM.AAMIAAAA https://doi org/10. 18653/v1/2022 emnlp-mamee main. 169
[29] I. LLoscchilov and F. Hutter. Decoupled weight decay regularization. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL BRLLitttss//opeerrvve
net/forumiid-
[30] Lu, B. Peng, H. Cheng, M. Galley, K. Chang. Y. N. Wu, S. Zhu, and J. Gao. Chameleon: Plug- and-play compositional reasoning with large language models. CoRR, abs/2304. 099402 2023, doi: 10.48550//r2iv.2344..9999 290.42.2.. URL httpp:://osti http://iiooor/////////
org/ooo.//../... 10. 48550/ ar 2304. 09842
[31] Z. Ma. J. Sansom, R. Peng, and J. Chai. Towards A holistic landscape of situated theory of mind in large language models. In H. Bouamor, J. Pino, and K. Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 1011-1031. Association for Computational Linguistics, 2023. doi: 10.18653/V1/20233 FINDINGS-EMNIAN P.72. URL httpp::/d/o/ org 10. 18653/166vvvv1 v1/2////v/222222 f...3.findii findinigs-ennnnn
72
[32] A. Madaan, N. Tandon, P. Gupta, S. Hallinan, L. Gao, Wiegreffe, U. Alon, N. Dziri,
S. Prabhumoye, Y. Yang. S. Welleck, B. P. Majumder, S. Gupta, A. Yazdanbakhsh, and P. Clark. Self-refine: Iterative refinement with self-feedback. CORR, abs/2303. 17651, 2023. doi: 10. 48550/arXiv.2303. 17651. URL htt.tttss://do.. org/ 10. 2.0.44850//rr// arxiv. 2303 17651
[33] A. Meurer, C. P. Smith, M. Paprocki, O. Čertik, S. B. Kirpichev, M. Rocklin, A. Kumar, Ivanov, J. K. Moore, S. Singh, T. Rathnayake, Vig, B. E. Granger, R. Muller, F. Bonazzi, H. Gupta, S. Vats, F. Johansson, F. Pedregosa, M. J. Curry, A. R. Terrel, Roučka, A. Saboo, I. Fernando, S. Kulal, R. Cimrman, and A. Scopatz. Sympy: symbolic computing in python. PeerJ Computer Science, 3:e103, Jan. 2017. ISSN 2376-5992. doi: 10.7717/peerj-cc... 1003. URL
https://doi. otw.:o....gg/.... 10. www.:077777/pperrj-ej j- 1003
[34] Mondorf and B. Plank. Beyond accuracy: Evaluating the reasoning behavior of large language models - A survey. CoRR, abs/2404.01869, 2024. doi: 10448550/ARXX..24440000
URL
htttps://oi. 4855://0000000000/////0/// att////Xiii../ 2404.400400 01869
[35] M. R. Morris, J. Sohl-Dickstein, N. Fiedel, T. Warkentin, A. Dafoe, A. Faust, C. Farabet, and
S. Legg. Levels of AGI: operationatizing progress on the path to AGI. CoRR, abs/2331.002461 2023. doi: 10.48550/ARXIV.333 1.02462. htttt::st/:/o..ooe 48550//rrx.555 2311 02462
[36] A. Newell, H. A. Simon, et al. Human problem solving, volume 104, Prentice-hall Englewood Cliffs, NJ, 1972.
[37] L. Pan, A. Albalak, X. Wang, and W. Y. Wang. Logic-Im: Empowering large language models with symbolic solvers for faithful logical reasoning. CoRR, abs/2305. 122./ 2023. doi: 10.48550/ARXXI.223055 12295. U httt:::t//:/aoo
org/10.0o 48550/arXttt 2305. 12295
[38] L. Pan, M. Saxon, W. Xu, D. Nathani, X. Wang, and W. Wang. Automatically correcting large language models: Surveying the landscape of diverse self self-correction strategies. CoRR,
abs/2308. 03 188 2023. doi: 10.48550/ARXII.22000 200.23300003883883 03188 URL /https:: .ttp//0...g/0000
48550/ 18550/arXiv.. 2308. 03188
13

---

[39] A. Saparov and H. He. Language models are greedy reasoners: A systematic formal analysis of chain-of- of-thought. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OvenReview.net, 2023. /tt. net/pdf 2idFFFWBZBWBW2
[40] R. Shen, S. Bubeck, R. Eldan, T. Lee, Y. Li, and Y. Zhang. Positional description matters for transformers arithmetic. CoRR. abs/2311. 2023. doi: 1048550AARXXV3311..47777
URL httpps/:
org/11 10. 48550/ arrvvv 2311 14737
[41] Z. Sun, L. Yu, Y. Shen, W. Liu, Y. Yang. S. Welleck, and C. Gan. Easy-to-hard generalization: Scalable alignment beyond human supervision. CoRR, abs/2403.09472, 2024. doi: 10.48550/
ARXIV.2403.0947 .99472.. URL htRpshtps::. :ttp:s///doo /tt.///o.i.oo www.aii..oggg..eooooeeeeeooooeee org/orgg//oo 48550/arXiv www.v. 09472
[42] J. Sweller, R. F. Mawer, and M. R. Ward. Development of expertise in mathematical problem solving. Journal of Experimental Psychology: General, 112:639-661, 1983. URL https
//api. wwrp..ssmaaiicscco.....
org/ Strg/Corpssliae 201296611
[43] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaci, N. Bashlykov, S. Batra, Bhargava, Bhosale, D. Bikel, Blecher, C. Canton-Ferrer, M. Chen, G. Cucurull, D. Es- iobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Ko- renev, P. S. Koura, M. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao. X. Martinet, Mihaylov, Mishra, I. Molybog, Y. Nie, A. Poulton, Reizenstein, adi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open foundation and fine- tuned chat models. CoRR, abs/2307.09286 2023. doi: 10-8550/ARXX12237..022888 URL https://doi.. org/10..//oooo.ooo 48550/ arxiv 2307. 09288
Rungta, K. Sal-
[44] J. Van Merrienboer and J. Sweller. Cognitive load theory and complex learning: Recent developments and future directions. Educational psychology review, 17:147-177, 2005.
[45] C. Wang, B. Zheng. Y. Niu, and Y. Zhang. Exploring generalization ability of pretrained language models on arithmetic and logical reasoning. In L. Wang. Y. Feng. Y. Hong, and R. He, editors, Natural Language Processing and Chinese Computing 10th CCF International Conference, NLPCC 2021, Qingdao, China, October 13-17, 2021, Proceedings, Part I, volume 13028 of Lecture Notes in Computer Science.pages pages 759.769.665 Springer, 2021. doi: 1007/ 978-3-030 3-030-884800-22-1 88480-21 2\ 61. URL https://aoi...... http:::///oo.or//////// 10. 10000000077000 030 88480-888-222 -2640022_611
[46] C. Wang, X. Liu, Y. Yue, X. Tang, T. Zhang, C. Jiayang, Y. Yao, W. Gao, X. Hu, Z. Qi, Y. Wang, L. Yang, J. Wang, X. Xie, Z. Zhang, and Y. Zhang. Survey factuality in large language
models: Knowledge, retrieval and domain-specificcty. CoRR, abs/2310.07522 2023. doi: httpps::::///tt wrg/o0.../....ooooooo... 48550/ 205500arXivv arviv
10.45550/ARXV.22310000
[47] J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, and D. Zhou. Chain-of-thougll prompting clicits reasoning in large language models. In NeurIPS, 2022. URL http::/ nips. cc cttpppscccppppp__epeee paapr_ffllee pappr/ haperr20022hash. htt.://sseeccttaaaooo.oo.aaaaaaa
html
[48] Y. Yang, Y. Ma, and Liu. Weak reasoning. CORR, abw/2407.136477, 2024 doi:
10448550/ARXIV.24477 13647. htRps:ttp::://et /r.. 40l.088550//
[49] P. Yu, J. Xu, J. Weston, and I. Kulikov. Distilling system 2 into system Co 2024. doi: (0.48550/ARXIV.24077. URL
06023
[50] D. Zhou, N. Schärli, L. Hou, J. Wei, N. Scales, X. Wang, D. Schuurmans, O. Bousquet, Q. Le, and E. H. Chi. Least-to-most prompting enables complex reasoning in large language models. CoRR, abs/2205. 10625, 2022, doi: (0.48850/aaxxiv22005.006 https::t//// //do1. org/10
48550/ @8550/arXiv 2205. 10625
14

---

A Limitations
Our work serves preliminary exploration of human-like step skipping capabilities in models, focusing solely on the expansion of problem types in terms length and compositional complexity. without extending to advanced problem difficulty generalization. We also recognize that ideally there should clear criterion for determining when to terminate iterations. We observe that the model can also perform better in intermediate rounds, which suggests the need for further adjustment of this hyperparameter. Additionally, for the convenience in evaluation, our investigations were confined to three simple yet representative tasks. While our designed method can be applied to practical tasks, we leave the exploration of scalability to complex reasoning scenarios as future work.
B Appendix
B.1 Details of data creation
B.1.1
Training data creation
For the Analog of Algebra task, we ensure the quality of the tutto-generaed dataset by creating full-step reasoning data using standard algebraic rules applied to operators. To further verify the validity and consistency of the intermediate steps, we utilize the SymPy (33 library. Specifically, we perform SymPy simplification for each intermediate step and ensure that the resulting equation remains algebraically equivalent to the final simplified answer.
For the Multi-digit Addition task, the internal results generated using Python's built-in calculation modules, ensuring accurate computations.
For the Directional Reasoning task, the clarity of the question formulation guarantees that all intermediate steps are 100% correct. Each step is derived through rule-based decomposition, ensuring the correctness of the intermediate steps.
B.1.2 Manual skipping data for warm start
We define several heuristic rules to create skipping data for warm start initialization. For the multi- digit addition task, we randomly merge two single-digit addition steps form two-digit addition step. For the directional reasoning task, we incorporate more human expertise by skipping steps that involve two adjacent directions that result in no change. For example, adjacent actions such as "right-left", "left-right", and "around-around" will not alter the final facing direction, so we manually skip these steps. We only manually create one skipped step within a single data.
B.2 Skipping data accuracy trend in cold start
Skipping Accurscy oerotion
sepppng Pourroy 750
Iraitin
Figure 8: Skipping data accuracy change during cold start in Analog of Algebra.
From Figure 8 the number of correct skipping data keeps increasing as the iterations progress. Higher accuracy results in more valid data involved in the mixed dataset. This iterative approach allows the model to gradually develop the step skipping ability and produce more valid data with fewer steps.
15

---

B.3 Detailed results of each iteration
Table and Table show the detailed performance of standard finetuned models from each iteration on Llama2-7B and phi-3-mini respectively. We report the average performance and the standard deviation across three rums with different random seeds.
Analyzing the results from each iteration, we find that the final iteration does not consistently yield the best performance, highlighting the importance of identifying an optimal stopping point as direction for future work. Additionally, significant fluctuations are observed in the test results, particularly in the OOD settings. Therefore, developing a more stable approach for OOD generalization tasks is another potential area for further exploration.
Table 5: Performance comparison of models from different iterations on Llama-7B. "Avg steps" denotes the average number of steps taken in the prediction.
In-domain
OOD-easy
OOD-hard
Task
Iteration
Acc
Avg steps
Acc
Avg steps
Acc
Avg steps
Cold start
99.870.12
3.190.00
85.91
4.790.04
7.944.1
11.57
Ieer
99.77a.15
3.1300t
86.721.60
4.630.06
8.653.81
I1.051.31
Analog of Algebra
Iter 2
99.770.06
3.040.z2
88.932.16
4.690.11
5,881.58
16.441.29
Iter
99. 99.90n.17
2.890.05
88.47 1.22
4.500.09
6.032.78
12.322.38
Iter 4
99. 99.930.12
2.530.07
90.770.30
4.190.12
8.575.15
11.39
Iter 5
99.80,.10
2.430.13
90.67 1.88
4.05g.17
8.10
10.92a.89
Cold start
100.00.00
2.860.00
0.06g.10
3,250.04
0.000.00
3.690.06
Warm start
99.530.32
2.72024
0.14g.13
3.02a.38
0.1
3.490.37
Iter
99.070.33
1.750.11
14.362.75
1.850.08
4,060.89
2.180.18
Multi-digit Addition
Iter 2
98.87o.12
1.45o.07
14.11
1.54p.07
1.44
2.05a.08
Iter 3
99.130.2
1.46004
16.811.70
1.53v.08
4.061.00
2.000.13
Iter 4
98. 98.770.06
1.41004
16.084.01
1.490.05
5.131.17
2.08a.11
Iter
99.170.35
1.460.04
13.97a.42
1.490.20
4.75g.7
2.06u.26
Cold start
100.0..00
7.01
90.00g.53
15.770.46
42.006.24
19.39
Warm start
99.97m.6
6.28p.04
87. 37.20
14.65o.43
42.339.25
18,02 18,0224
Iter
100.0a.00
6.466.04
83.00s.57
14.69u.14
29.476.59
14.242.86
Directional Reasoning
Iter 2
99.97%.06
6.440.6
86.473.93
14.950.44
40.67
17.423.23
Iter 3
100.0..00
6.490.13
88 88.601.64
14.93u.44
41.537.30
17.60p.68
Iter 4
99. 99.90n.10
6.36m.06
89. 89.202.03
14.666.30
44.336.99
17.791.31
Iter 5
100.0p.00
6.45n.06
89.331.36
14.87o.12
51.804.21
19.49
Cold start vs. warm start In the Multi-digit Addition task, observe that phi-3-mini achieves satisfactory results with cold start training alone, allowing the model to enter the iteration phase without relying manually provided skipping data. Table the model's performance when initialized with a cold start in Multi-digit Addition. Compared to the results in Table where the model begins with a warm start, the cold start approach enables the model independently explore and develop its skipping behaviors. This leads to a more pronounced improvement in the OOD settings, with accuracy of 25.06% versus 14.98% Iteration on OOD-hard. Additionally, we observe that while warm start enables more immediate reduction in steps, cold start shows a more gradual decrease in the number of steps taken.
B.4 Accuracy of step-skipping answers
In this section, we provide the ratio and the accuracy of the skipping responses across three tasks using both base models. The results are shown in Figure Plnd Figure TO In general, the models demonstrate a progressively enhanced step skipping capability across various test settings for all tasks. In most cases, the model increasingly favors adopting more skipped reasoning steps over iterations, with the accuracy of skipped responses also improving correspondingly. However, we observe that the proportion of skipped responses fluctuates across different stages of iteration, rather than following a strictly monotonic trend. Given that the model autonomously decides whether to employ skipping, this pattern may indicate the model's 's attempt find a balance between using step
16

---

Table 6: Performance comparison of models from different iterations on phi-3-mini. "Avg steps" denotes the average number of steps taken in the prediction.
In-domain
OOD-easy
OOD-hard
Task
Iteration
Acc
Avg steps
Acc
Avg steps
Acc
Avg steps
Cold start
99.60g.10
3.190.1
98,041.09
6.16p.00
4.052.11
10.01632
Iter I
99.77a.06
3.180.00
99.020.34
6.10p.02
3.173.64
9.82a.69
Analog of Algebra
Iter 2
99.83a.12
3.130.02
98.89 1.08
6.07p.1
5,402.74
9.00p.36
Iter 3
99.90u.10
2.950.05
99.54o.11
5.890.09
9.923.47
7.670.39
Iter 4
99.97a.66
2.710.06
99.4
5.62
10.160.96
7.34o.
Iter 5
99.900.17
2.750.28
98.950.23
5.60p.33
11.131.50
7.98p.44
Cold start
99.92a.13
2.860.00
32.93
5.03p.22
5.39
5.440.17
Warm start
99.97a.06
2.620.07
39.08s.87
3.80p.35
5.112.62
4.060.44
[ter I
99.87a.15
2.21o00
45.036.98
2.43p.30
12.360.66
2.350.34
Multi-digit Addition
Iter 2
99.93a.06
2.02o.13
49.45s.18
2.22a.15
13.883.84
2.420.06
Iter 3
99.93a.12
2.130.08
43.08s.00
2.30p
13.54
2.570.m
Iter 4
99.87a.15
2.01o.05
45.259.93
2.28p.11
12.84
2.520:24
Iter 5
99.93a.06
2.08o.12
12.61
2.31p.11
14.983.19
2.590.12
Cold start
99.83g.36
7.01o.00
91.473.68
15.460.25
62.67
24.850.43
Warm start
99.80o.17
6.82o.17
93.67
15.19p.07
71.80s.30
24.61o.15
Iter I
99.93a.12
6.480.15
94.401.51
14.94a.13
73.136.93
24.430.30
Directional Reasoning
Iter 2
99.97a.6
6.36p.10
95.332.42
14.72p.11
74.80%.77
24.266.63
Iter 3
99.670.35
6.40p.13
94.471.70
14.83o.13
75.406.39
24.240.60
Iter 4
99.60g.35
6.230.12
95.130.95
14.72a.29
12.87
24.20p.59
Iter 5
99.70a.17
6. 6.120.06
93.73a.70
14.44p.04
73.874.17
23.770.18
Table 7: Performance across iterations in the Multi-digit Addition task with the phi-3-mini model, initialized from a cold start rather than warm start.
In-domain
OOD-easy
OOD-hard
Task
Iteration
Acc
Avg steps
Acc
Avg steps
Acc
Avg steps
Cold start
99.92o.13
2.860.00
35.93
5.03a.22
5.39
5.44o.17
Warm start
99.97 99.970.6
2.62a.07
39,081.77
3.80
5.11262
4.06p.44
Ierr
100.0..00
2.830.05
37.441273
5.03a.18
5.2
5.28o.17
Multi-digit Addition
Iter 2
100.0p.00
2.780.15
38.5028.57
4.770.s7
4.83
5.00
Iter 3
99, 99,90p.1
2.780.07
58.78p:73
5.030.20
9.04p.66
5.270.13
Iter
99.93
2.380.28
49.1916.52
4.180.78
25.35
4.95u.27
Iter 5
99.83o.15
2.54g.27
55.47
4.5
25.06679
5.290.13
skipping and providing full-step solutions. Exclusively relying on skipping would not necessarily be the optimal answering strategy. We also find that warm start significantly boosts the model's skipping behavior. Consequently, in models with warm start, the changes across iterations are less pronounced, though overall accuracy still improves.
B.5
Data mixing choices for standard model training
Table 8: Ablation of different data mixing choices on Analog of Algebra.
In-domain
OOD-easy
OOD-hard
Training data
Acc
Avg steps
Acc
Avg steps
Acc
Avg steps
Skipping
98.70
1.94
93.66
4.97
7.86
7.44
Skipping w/ Cold start
99.90
2.75
98.95
5.60
11.13
7.98
In this section, we analyze the role of data mixture in iterative training and its effect on the performance of standard models Metannard Specifically, we examine how the inclusion of both cold-start data and generated skipping data enhances the model's generalization ability and comprehension of complex
17

---

Analog of Algebra
044
SHOW
Multi digit
CUTM
C20 may
020 word
Directional Reasoning
020mm
X
Accurtt
Figure 9: Skipping ratio and accuracy at each iteration on Llama2-7B.
reasoning paths. Table 8 presents an ablation study comparing different data mixing strategies with phi-3-mini model the Analog of Algebra task. The "Skipping"" setting utilizes only the generated skipping data D/-2 for training the standard model Mg. while "w/ Cold Start" incorporates both the original cold-start data and the skipping data, which serves the default configuration in experiments. The analysis is based on data from Iteration 5, and we report average performance across three runs with different random seeds. Our findings suggest that relying solely on skipping data may limit the model's capacity to address OOD scenarios. Although skipping data provides shorter average steps, lacks the complete reasoning steps essential for comprehensive understanding of the task, potentially leading the model to depend on shortcuts that harm generalization. By incorporating mixture of cold-start and skipping data, the model able learn from both complete and skipped reasoning chains, which enables more robust understanding, supporting stronger generalization capabilities.
B.6 Cross-Domain Generalization of Step-Skipping Ability
Table 9: Cross-domain generalization of step-skipping capability in the phi-3-mini model. In the specified "Withheld Task" setting, step-skipping data excluded from specific task, while the "All" setting includes only full-step data across three tasks.
In-domain
OOD-easy Acc
OOD-hard Avg steps
Evaluation Task
Withheld Task
Acc
Avg steps
Avg steps
Acc
All
51.3
2.65
44.5
5.58
1,9
10.68
Analog Algebra
Analog of Algebra
53.9
2.71
56.9
5.74
7.1
10.97
All
100.0
2.86
22.4
4.71
4.2
5.39
Multi-digit Addition
dilti-digit Addition
95.7
2.59
34.3
4.75
2.4
5.35
All
100.0 97.8
7.01
96.0
15.46
75..8
25.03
Directional Reasoning
Directional Reasoning
96.2
15.42
80.0
24.92
18

---

Analog of Algebra
100
0
Multi digit
0:00 wao
coo hard
Directional Reasoning
020-BB
050-trt
Figure 10: Skipping ratio and accuracy at each iteration on phi-3-mini On Multi-digit Addition, we illustrate the analysis of the model that is initialized from Cold start.
To investigate the cross-domain generalization of step-skipping capabilities, we conduct a controlled experiment to assess the impact of step-skipping training data from one task the model's per- formance in others. Specifically, we sampled 2,000 training examples per dataset, including 1,600 step-skipping answers in which exactly step was successfully skipped from these samples, all from Iteration 5. This setup ensures an equal balance of full-step and dtep-skppping data across all three tasks.
We use the phi-3-mini model across three tasks, with the "withheld task" representing the task that lacks step-skipping data during training. The "All" setting contains only full-step answers for all tasks, with no step-skipping data included. The configurations are follows:
All setting: task1-full + task2-full + task3-full
Withheld setting: task1-full + task1 -skip task2-full task2-skip + task3-full
Table 9 summarizes the model's performance on each evaluation task. The withheld task's results compared to those from the "All" setting, where all tasks trained with only full-step answers. Our findings reveal that step-skipping data in one or more tasks positively affects the performance of the withheld task. In most cases, models trained with step-skipping data from other tasks exhibit improved accuracy and step-skipping performance across datasets, maintaining a comparable number of steps the "All" setting. For example, in the Analog of Algebra task, the average steps remain similar, yet accuracy improvements are observed in OOD settings, indicating that training with skippik data promotes transferable ability reason efficiently across domains. The overall accuracy increase suggests that inclusion of step-skipping data in some tasks enables the model to generalize this ability, even when explicit step-skipping examples are unavailable in the target task. These results suggest that the step-skipping capability learned in one domain can generalize across different tasks, underscoring the potential for enhancing model efficiency through strategic data composition.
19

---

B.7 Experiments on GSM8K
In addition the synthetic datasets analyzed in the main body of the paper, we conduct experiments on GSM8K [8] to evaluate the applicability of our method to more complicated tasks. To create controlled experimental setting, we classify data requiring no more than 4 steps in the annotated answers as in-domain data and the remaining out-of-domaie data. Table TO provides an overview of the dataset splits.
Table 10: Dataset split for GSM8K.
Splits
In-domain
Out-of-domain
Total
Train
6,235
1,238
7,473
Test
1,094
225
1,319
The results across different iterations is presented in Table [1] We observe that while the average number of reasoning steps per iteration progressively declines, the accuracy remains stable across iterations. Several factors may explain the limited improvement in accuracy. Analysis of the model's step-skipping behavior reveals that intermediate steps frequently contain errors, indicating limitations in the model's 's ability for effective step reduction. Throughout the iterations, the model struggles to generate responses in fewer steps, as the complexity reasoning chain to reach a solution. This aligns with findings by Yu et al. [49], which suggest that CoT reasoning is difficult distill into System 1. We consider further exploration of the gradual transition between System and System 2 thinking, particularly for complex tasks, promising direction future research.
the questions often necessitates a complete
Table 11: Performance comparison across different iterations. The table shows accuracy and average steps for various test and training datasets.
Test-ID
Test-OOD
Train-OOD
Iteration
Acc
Steps
Acc
Steps
Acc
Steps
Cold start
79.89
4.23
61.33
6.5
63.33
5.99
Iterl
78.06
4.24
59.56
5.9
64.62
5.96
Iter2
78.52
4.15
57.78
5.84
63.33
6.02
Iter3
79.16
4.19
52.44
5.86
63.57
5.90
Iter4
75.69
4.16
56.44
5.78
63.97
5.88
Iter5
78.43
4.08
60.44
5.77
61.55
5.72
B.8
Case study
C
B D
x +
)
B ((C D) + A) B B (C D)
Figure 11: Frequent skipping pattern in Analog of Algebra (translated into standard algebra).
Figure II shows two frequent cases where models spontaneously develop to skip certain steps in iterations. Note that in Analog of Algebra, we employ the cold start setting in initialization. The behaviors shown the figure emerge solely from the model itself, which indicates the model has developed its step skipping preference. In addition, of Algebra and Multi-dight digit Addition tasks. In these examples, the full step answers exhibit errors in their reasoning processes. In contrast, the skipped step answers choose to skip steps and arrive the correct answer in fewer steps.
show two cases in Figure 12 from Analog
20

---

Analog of ALgebra
C /
Full-step answer:
A C F / D
B B
C H
/ G
* F D
/ C
/ C B A/C
/ X
G
H
/ C
/CCH A/C/H)
= F
D / G H
I
C/
Skipped- step answer:
C + F D
I
H x / G
B A
F D H
/ H
B /
H
A A H
C
6
H +
Multi-digts Addition
Q: 3909 641
Full Full-step answer:
1. 0 1 (plus carry 8) 1 (write down 1, no carry over).
2. 4 (plus carry 0) 13 (write down 3, carry over 1).
3. 3 6 (plus carry 1) 10 (write down 0, carry over 1).
4. Carry the final carry over to the next place.
Thus, the answer is 1031.
Skipped-step answer:
1. 9 + 1 (plus carry 0) 10 (write down 0, carry over 1).
2. 90 64 (plus carry 1) 155 (write down 55, carry over 1).
3. 3 0 (plus carry 1) 4 (write down 4, no carry over).
Thus, the answer is 4558.
Figure 12: Case study of skipped-step answers in Analog of Algebra and Multi-digit Addition.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We made clear descriptions and clarifications of our contributions. They well supported by the experiment results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We explicitly discuss the limitations of our paper in Section A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
This work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We describe the necessary implementation details in Section 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [No]
Justification: While currently we have not released our code and data, we will make them publicly available in online repository upon acceptance of this work.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cr ec/
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https http://oooouuuisss.ooeee
//nips //nips.cc/pil cc/public/guu
SttSSSabmisiioooiiiioi.oooia for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We describe all the details of datasets and implementations in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report the average results and the standard deviation in our experiments
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe the compute resources for our experiments in Section 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: bttppp////baaroo..cc///////////a
Answer: [Yes]
Justification:
This work conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work investigates the human-like like behaviors of language models. There is no necessary societal impact of this work that should specified.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: We mainly evaluate model behaviors on simple and well developed reasoning tasks. This work does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We use Llama2 models: https://ai.nonn meta. watiaaacoommmaaaaaaeeeeee
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "oziVAmlX9d", "reviewer": "Reviewer_rzDm", "summary": "The paper proposes a method for training an LLM to solve reasoning problems using fewer verbalized reasoning steps than it is naturally encouraged to by a fixed training dataset. The resulting model is shown to maintian or improve performance on in-distribution data and OOD data testing extrapolation w.r.t. length or compositionality, while using fewer reasoning steps at inference time. Analysis shows that performance gains are concentrated around problems requiring an intermediate number of reasoning steps, rather than very few reasoning steps. Experiments are conducted with Llama-2-7b on three synthetic datasets. The method itself works by using warm-start data with mixed length reasoning demonstrations, followed by bootstrapped training data created by controlling model generations with control codes (instructions) combined with filtering model generations for correctness to create new gold data.", "questions": "- Why keep the cold start data in the training data if the bootstrapped data is good or better? Do you have ablations that suggest what mixture of the data is best?\n- Suggested experiment: if you could have two models that are similar except for one being better at long-context reasoning, it would be interesting to see how your method affects each model. The reason for this is that compressing reasoning length could be beneficial by virtue of reducing the context length, rather than some other inherent benefit like allowing the model to spend more computation on harder steps. Such an experiment would help disambiguate if the improvement comes from shortening context length or from using fewer steps.\n- Note L.68-69 is heavily disputed by follow work on ToM, e.g. https://arxiv.org/pdf/2310.19619\n- Just so you’re aware, some highly related work has appeared contemporaneously: (1) https://arxiv.org/pdf/2405.14838, (2) https://arxiv.org/pdf/2407.06023\n- L.34: use an em-dash rather than single dash here\n- L.221: Fig7(a) should read Fig4(a)", "limitations": "Discussion was adequate, but it could be worth mentioning that experiments were conducted with only one model and three synthetic datasets.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Very important: The idea of shortening reasoning steps, particularly to mimic human reasoning that is variable in its verbalized length, is a very interesting and practical direction.\n- Very important: Results are positive and promising for model generalization at an increased level of efficiency. Particularly interesting are results suggesting that model performance can increase on difficult OOD data by virtue of skipping some reasoning steps. Initially, CoT was found to improve OOD generalization, but it seems that this iteration on CoT could improve OOD performance even more in some situations.\n- Important: The paper is overall straightforward to read and understand, with only a few exceptions.\n- Of some importance: The connection to easy-to-hard generalization was interesting to me. That this method could improve OOD performance, specifically length/compositional generalization, was very interesting.", "weaknesses": "- Important: What are the instructions at inference time? Do you require a ground-truth number of reasoning steps to run the model at inference time? If so, this important detail is missing from the paper and could make the method difficult to use in practice for problems if it is not know how difficult they are in advance. Would the method be robust to misspecified instructions at inference time?\n- Important: I find it a little confusing to reconcile the results of Sec. 5.1 with Sec. 5.2. Sec. 5.1 makes it look like using fewer steps greatly hurts model performance, while Sec. 5.2 makes it seem like using fewer steps does not hurt performance (specifically the Warm start rows, relative to Cold start baselines).\n- Of some importance: The data is a little artifical. There are existing reasoning and compositional reasoning benchmarks that could be appropriate for this work (though they could require stronger models), including SCAN (https://arxiv.org/abs/1711.00350), GSM8k, StrategyQA, and MATH datasets. However, this is not a major weakness as using clean, controlled datasets is advantageous for studying these kinds of phenomena and they enable automatic construction of warm start data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful and encouraging feedback. We are pleased that you found our approach and results promising and will continue to refine and expand upon these ideas in future revisions.\n\n**[W1]**\n\nWe only require the step number as input when generating the skipped data for the training set during the iteration phase. At inference time, we train an additional standard model (described as Eq. 2), which only takes the question as input, without the need for a number of reasoning steps. The instruction used is:\n\n\"Transform the expression to isolate ❤ on the left side of the ↔. \\\nQuestion: [[QUESTION]] \\\nAnswer:\"\n\nThis approach ensures the model is robust and practical for real-world applications, as it does not rely on the ground-truth number of reasoning steps during inference.\n\n**[W2]**\n\nWe apologize for the confusion regarding the results in Sections 5.1 and 5.2. We will provide more detailed captions to clarify this. In Section 5.1, we analyze the initialized model $M_0$, which is trained with full step data only and requires the step number as input. When asked to generate answers in fewer steps, the model struggles to maintain accuracy because it has not been trained on the skipping-step data. The primary purpose of $M_0$ is to generate skipped step data during the iteration phase, and it is not expected to achieve high accuracy initially—only sufficient accuracy to generate usable data.\n\nIn Section 5.2, we evaluate the resulting standard model $M^{standard}$, which is trained on both the full step and the generated skipped data, and does not require the step number as input. This training method allows $M^{standard}$ to maintain accuracy while using fewer steps, demonstrating the effectiveness of our approach.\n\n**[W3]**\n\nThank you for your valuable suggestion! We acknowledge the importance of using established benchmarks. We are actively working on additional experiments using these datasets and will provide an update on the results once the experiments are completed.\n\n\n**[Q1]**\n\nMixing the data allows us to gradually process the iterations. At the beginning of the iterations, the valid skipping data may not be enough and they cannot help the model fully understand the task. For example, on Analog of Algebra, Iter 1 only has less than 150 valid skipping data to use. We are also working on additional ablation analysis to provide empirical support following your advice.\n\n**[Q2]**\n\nThank you for this insightful suggestion! It would indeed be valuable to investigate where the improvements come from. We will consider incorporating this experiment in future revisions to better understand the underlying reasons.\n\n**[Q3]**\n\nThank you for pointing this out. We will include this in the related work.\n\n**[Q4]**\n\nThank you for providing the references. We are excited to see related work in the community that shares similar ideas and interests! We will incorporate these references in the revised version.\n\n**[Q5, Q6]**\n\nThank you for pointing out these typos. We will revise them accordingly.\n\n**[Limitation]**\n\nWe additionally conduct experiments with Phi-3-mini on the Analog of Algebra task. The results confirm that our method is generalizable to other model series. We will include the results across all tasks in the revision.\n\n\n|          | In-domain    |            | OOD-easy   |            | OOD-hard   |            |\n|----------|--------------|------------|------------|------------|------------|------------|\n|          | Acc          | Avg steps  | Acc        | Avg steps  | Acc        | Avg steps  |\n| Cold Start | 99.5       | 3.18       | 96.86      | 6.16       | 1.67       | 9.83       |\n| Iter 1   | 99.7         | 3.18       | 99.22      | 6.13       | 0.95       | 10.61      |\n| Iter 2   | 99.9         | 3.11       | 100.0        | 6.06       | 5.48       | 9.06       |\n| Iter 3   | 99.9         | 2.90        | 99.6       | 5.79       | 6.67       | 7.74       |\n| Iter 4   | 99.9         | 2.64       | 99.41      | 5.36       | 9.05       | 7.46       |\n| Iter 5   | 99.7         | 2.43       | 98.82      | 5.23       | 10.24      | 7.77       |\n\nTable: Phi-3-mini on Analog of Algebra.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KDZXiDtT1B", "reviewer": "Reviewer_vxGY", "summary": "This paper proposes to teach LLMs to deliberately skip steps when doing complex tasks involving multi-step reasoning. The authors use self-generated inference paths with fewer steps to fine-tune the models, which is similar to self-distillation. The authors conduct experiments on a few controlled tasks show that the proposed approach can effectively reduces the reasoning steps while maintain performance.", "questions": "N/A", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of teaching LLMs to skip steps following the human reasoning process is intuitive and makes sense.\n2. The proposed method is overall technically sound and well described.\n3. The paper is in general well-written and easy to follow.\n4. Experimental results confirm the effectiveness of the proposed approach, at least on these \"artificial\" tasks.", "weaknesses": "1. The experiments are not solid because the tasks considered in the experiments are very artificial and not representative for real-world reasoning tasks. The paper could be made much stronger by conducting tasks/datasets such as GSM8K/MATH or coding tasks, instead of simple reasoning tasks. Without the empirical study on realistic tasks, it is hard to confirm the contribution and usefulness of the proposed metric.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and valuable suggestion! We acknowledge the importance of using established practical benchmarks. We are currently conducting additional experiments with these datasets and will provide an update on the results once the experiments are completed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6SVQFuC9zW", "reviewer": "Reviewer_DBKK", "summary": "This paper proposes an iterative training method that helps sequence models learn to skip steps. The method starts from a training set with full-length solutions or mixed with some skipped-length solutions. At each stage a model learns these solutions with the instruction “Solve it in n steps” and is prompted to generate shorter answers. Correct shorter answers are added to the training set. The effect of this approach is tested using LlaMa-7B on three tasks, including algebraic evaluations, multi-digit addition, and a direction inference task.", "questions": "The accuracy metric measures final answer accuracy, are intermediate steps correct?\n\nI'm unsure if it makes sense to read too much into the average step metric when accuracy is low.\n\nIn figure 4, what does the accuracy look like for only problems where the model skipped steps?\n\nWhat do you think make multi-digit addition and directional inference more difficult than the algebra task? Especially that accuracy and average step for OOD problems are still pretty bad for multi-digit addition even with the warm start and a few iterations in.\n\nWhat is the change in the ratio between D_init and D_skipped over the iteration process?\n\nWhat’s the range of i (in n-i) of the added skip-step instances in D_skipped under warm start?\n\nLine 221 has incorrect figure reference.", "limitations": "Noted in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed skip reasoning pipeline is interesting and was evaluated against a diverse set of tasks with different levels of OOD generalization tests.\n\nThe authors conducted detailed analyses to understand the effect of the training pipeline, e.g., Figure 5 with the multi-digit addition is very informative.\n\nThe overall presentation is very clear and easy to follow.", "weaknesses": "My main concern is the generalizability of this method. As shown in the paper, the model largely benefited from the warm start setup that includes some skipped problems in the first training set, and has trouble generalizing to problems requiring more steps. One interesting generalization test would be to train on a mixture of all three tasks, but withhold adding skipped step instances for one task, and see if the model can generalize skipping steps on the withheld task.\n\nThe shorter answers generated during the iterative process also don't seem quite “generated by the model itself”, as filtering out correct answers would require oracle knowledge. Is it assumed that correct answers are any exact subset of the full-length solution?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable and constructive feedback. We provide specific responses and clarifications as follows.\n\n\n**[W1]**\n\nThank you for your valuable suggestion! We acknowledge the importance of evaluating the generalizability of our method. We are actively working on additional experiments and will provide an update on the results once the experiments are completed.\n\n\n**[W2]**\n\nIn the iteration phase, we query the model to generate shorter answers on the training set and filter based on the correctness of the final answers. The data used are the labels required for training any model, so our filtering process does not rely on any additional information beyond the training data.\n\nWhen generating answers with fewer steps, the model determines which steps to skip or omit. Thus, the skipping behaviors are entirely generated by the model itself, and the filtering serves as guidance on how to strengthen such behaviors.\n\nCorrect answers can indeed be subsets of the full-length solution. The models may actively omit certain steps from the full-length answer to generate the shorter output.\n\n**[Q1]**\n\nThank you for this insightful question. Our accuracy metric does not measure the correctness of intermediate steps. Automatically evaluating the correctness of the internal reasoning process is a common issue and can be challenging [1]. Previous work [2] has shown that training with incorrect reasoning traces can still lead to improvements. From our manual analysis of Analog of Algebra on In-domain test set, we observe 50 skipped answers and find that 96% of them are correct in its reasoning process. We'll provide the analysis across all tasks in the revision.\n\n[1]Solving math word problems with process- and outcome-based feedback. \\\n[2]MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models.\n\n\n**[Q2]**\n\nThe primary motivation of this metric is to measure whether the model has learned the ability to skip steps. Even when the accuracy is not high, the model still uses fewer steps overall while maintaining its overall accuracy. This consistent performance, both when accuracy is low and high, indicates that the model is developing this step-skipping ability and demonstrating it on both in-domain and out-of-domain data.\n\n**[Q3]**\n\nThank you for your insightful question. We will include the analysis across all tasks in the revision. Here we present the results for the Analog of Algebra task. The Skipping Ratio indicates the proportion of answers where steps were skipped in the entire test set. The Acc measures the accuracy of these skipping answers. The results show that as iterations progress, the model tends to skip steps in more problems, and the accuracy increases accordingly. In the OOD-hard setting, due to the lengthy and difficult nature of the problems, the model initially tends to use shorter answers with lower accuracy. However, the overall accuracy increases with each iteration.\n\n\n\n|         | in-domain   |                   | OOD-easy   |                   | OOD-hard    |                   |\n|---------|--------------------|-------------------|-------------------|-------------------|-------------------|-------------------|\n|         | Skipping Ratio          | Acc         | Skipping Ratio           | Acc         | Skipping Ratio           | Acc         |\n| Iter 1  | 6.0                  | 100               | 15.4               | 63.3             | 69.8              | 4.1              |\n| Iter 2  | 16.8               | 99.4              | 21.5               | 73.7             | 34.8              | 5.5              |\n| Iter 3  | 32.7               | 100               | 29.9               | 79.1              | 52.4              | 6.4              |\n| Iter 4  | 69.3               | 99.7              | 58.9              | 88.4             | 63.6              | 15.0             |\n| Iter 5  | 68.8               | 99.7              | 63.2               | 90.0             | 61.9               | 7.7              |\n\n*Table: Skipping ratio across the iterations and the accuracy of problems with skipping-step answers.*\n\n**[Q4]**\n\nThese two tasks involve more steps and actions in the answers. From our manual observation, we noticed that the model tends to overfit on the multi-digit addition task. As the training data only contains numbers with up to three digits, the model overfits to three-digit additions. Here is an incorrect answer from the model's prediction:\n\nQ: 4423 + 684\\\nPrediction:\n1. Add the 10^0's place digits: 2 + 4 (plus carry 0) = 6 (write down 6, no carry over).\n2. Add the 10^1's place digits: 4 + 8 (plus carry 1) = 12 (write down 2, carry over 1). \n3. Add the 10^2's place digits: 4 + 6 (plus carry 1) = 11 (write down 1, carry over 1). \n4. Carry the final carry over 1 to the next place. \\\nThus, the answer is 11266. \n\nIn this example, the model actually performs the addition for 442 + 684, which indicates a biased overfitting from the initialization phase. During the iteration phase, the skipping data helps to mitigate this overfitting, thereby improving accuracy.\n\n**[Q5]**\n\nOn the task of Analog of Algebra, the original training set contains 5770 data. As the iteration proceeds, the number of mixed data points (Total) increases significantly, nearly tripling by the fifth iteration. Here, $i$ indicates the number of valid answers that skip $i$ step in the reasoning process.\n\n|         | Total | i=1 | i=2 |\n|---------|-------|--------|--------|\n|Initialization   | 5770  | 0      | 0      |\n|Iter 1   | 5945  | 153    | 22     |\n|Iter 2   | 7078  | 1050   | 258    |\n|Iter 3   | 10010 | 2895   | 1345   |\n|Iter 4   | 13744 | 4753   | 3221   |\n|Iter 5   | 15048 | 5255   | 4023   |\n\n*Table: The change in data volume across iterations.*\n\n**[Q6]**\n\nIn the warm start phase, we only use i=1 in the manual skipping data.\n\n**[Q7]**\n\nThank you for pointing it out. We will revise the figure reference accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wO9SteHJN4", "reviewer": "Reviewer_E6Qj", "summary": "The paper explores the ability of language models to skip steps in their reasoning processes. The authors introduce a controlled framework to stimulate step-skipping behavior by iteratively refining models to generate shorter and accurate reasoning paths. The study demonstrates that models can develop this ability under guidance, leading to increased efficiency without sacrificing accuracy. The paper presents empirical results showing enhanced generalization capabilities in out-of-domain scenarios after fine-tuning on expanded datasets that include both complete and skipped reasoning sequences.", "questions": "see weakness", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The empirical results are robust in three domains, showing benefits in efficiency of the proposed method.\n- The paper is clearly written and well-organized, making it easy to follow the authors' methodology and findings.", "weaknesses": "- Only one backbone model is considered. Experiments across model families and model sizes should be considered to show the generalization ability of the proposed methods.\n- The OOD test is actually the harder in-domain test. I am curious about the across domain effect of the proposed method For example, how's the effect of training on \"Analog of Algebra\" and test on  \"Multi-digit Addition\", given that the skip ability should be a general ability across different domains?\n- In methodology: \"We begin with a training dataset D0, which contains detailed full-step reasoning answers to the questions.\" -> How the full-step reasoning data created?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable and constructive feedback. We provide specific responses and clarifications as follows.\n\n\n**[W1]**\n\nThank you for the suggestion. We acknowledge the importance of testing our method across different models. Following your advice, we performed additional experiments on Phi-3-mini for the Analog of Algebra task. These experiments confirm that our method is generalizable to other model series. We will include the results across all tasks in the revision.\n\n\n|          | In-domain    |            | OOD-easy   |            | OOD-hard   |            |\n|----------|--------------|------------|------------|------------|------------|------------|\n|          | Acc          | Avg steps  | Acc        | Avg steps  | Acc        | Avg steps  |\n| Cold Start | 99.5       | 3.18       | 96.86      | 6.16       | 1.67       | 9.83       |\n| Iter 1   | 99.7         | 3.18       | 99.22      | 6.13       | 0.95       | 10.61      |\n| Iter 2   | 99.9         | 3.11       | 100        | 6.06       | 5.48       | 9.06       |\n| Iter 3   | 99.9         | 2.9        | 99.6       | 5.79       | 6.67       | 7.74       |\n| Iter 4   | 99.9         | 2.64       | 99.41      | 5.36       | 9.05       | 7.46       |\n| Iter 5   | 99.7         | 2.43       | 98.82      | 5.23       | 10.24      | 7.77       |\n\n*Table: Phi-3-mini on Analog of Algebra.*\n\n**[W2]**\n\nThank you for your valuable suggestion! We acknowledge the importance of evaluating the generalizability of our method. We are actively working on additional experiments and will provide an update on the results once the experiments are completed.\n\n\n**[W3]**\n\nFor the three datasets used in our work, we use heuristic rules to automatically generate the reasoning processes. For example, in the Analog of Algebra dataset, we first create full-step reasoning data using standard algebra rules on operators. Then, we replace the variables with analog symbols. We also plan to release the code for creating the dataset in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this work, the authors explore how to guide LLM's training so they can skip unnecessary reasoning steps like humans do. Concretely, they propose a framework that iteratively refining models to generate shorter reasoning paths, at every iteration, they keep the generated shorter reasoning paths that lead to the correct answers to fine-tune the model in the next iteration. By doing so, they show that on three dataset, a llama2-based model could solve tasks with less steps but not sacrificing accuracy. 

This work has many merits. All reviewers agree that the idea is interesting and intuitive, shortening reasoning steps could be very useful practically; the proposed method is technically sound; the paper is clearly written, the structure is well organized; the three tasks, despite synthetic, are well designed to show how and why the proposed method could work, the experimental results support well the authors' claim; the authors provide detailed analyses that helped readers to understand the training pipeline; the idea is generally well connected to the paradigm of easy-to-hard generalization.

On the negative side, reviewers' concerns were mainly around 1) clarity on some details; 2) the relatively simple datasets/tasks; and 3) the authors failed to demonstrate the method with more than one LLM backbones. 

To address the concerns, during the author-reviewer discussion period, the authors provided the following additional information:
- The authors added experimental results with Phi-3-mini, in addition to the Llama2-7B in their submission. Results provide more evidence on generalizability of the method on other backbone models.
- The authors provided an analysis on the skipping ratio (the proportion of answers where steps were skipped in the entire test set) at every iteration. This shows that the model indeed learns to skip more steps (while accuracy keeps unchanged or even improved) with more training iterations.
- The authors provided an additional experiment testing the method's generalizability across different domains. They show that training the model with such step-skipping behavior could generalize to unseen tasks.
- The authors clarified some points that were unclear before.

The authors agreed to conduct experiments on more realistic reasoning datasets such as GSM8K, MATH, or coding. However, due to the limited time, they fail to finish that set of experiments (which is understandable). Because of this, Reviewer vxGY remains their negative score of 4, which makes sense, I also understand. 

In general, the authors did a great job addressing the reviewers' concerns and answering their questions. Reviewer DBKK increased their score from 4 to 6. During the reviewer-AC discussion period, both Reviewer DBKK and rzDm involved in the discussion, they both express that while acknowledging that the work could be much stronger if they could include the more realistic datasets, they believe the current version (initial submission + authors' response) gives them enough confidence that the work could be well received by the NeurIPS community. 

Personally, I agree with all reviewers, I think this work provides some interesting insights on how models could be guided to evolve from system 1 to system 2 thinking. At the same time, I also agree with Reviewer vxGY that more complex / non-synthetic datasets could be crucial in better justifying the method. I tentatively suggest an accept.
Decision: Accept (poster)