First-Step Advantage: Importance of Starting Right in Multi-Step Math Reasoning

## Abstract

Abstract Language models can solve complex reasoning tasks better by learning to generate rationales for their predictions. Often these models know how to solve a task but their auto-regressive decoding nature leads to incorrect results if they start incorrectly.
We observe that smaller models in particular when corrected, can solve a task that they would have otherwise struggled with. We demonstrate this phenomenon by using a larger model to guide smaller models, which leads to significantly improved performance (up to +24 points on the GSM8K dataset by 7B models). To assist smaller models in initiating the starting step, we propose QuestCoT , where a smaller model first asks itself how to start , before proceeding with a chain of reasoning.
On various multistep mathematical reasoning datasets over multiple smaller models, we show that getting the right start can lead to significant performance gains across all models (gains of up to +6 points on GSM8K, +9 on SVAMP, +5 on ASDiv, and +7 on MultiArith).

## 1 Introduction

Over the years, large language models (LLMs) have improved their reasoning abilities by explaining their intermediate thoughts Wei et al. (). This trend has been extended to smaller models (^1^11we use smaller models in a relative sense and most of our experiments are carried out on models smaller or equal to 7B parameters), either through pre-training Jiang et al. (); Magnusson et al. (), fine-tuning Yu et al. (); Shao et al. (), or knowledge distillation Shridhar et al. (); Yuan et al. (); Magister et al. (); Hsieh et al. ().
While it is commonly assumed that smaller models acquire new knowledge through fine-tuning or distillation, recent research by Gekhman et al. () suggests that the acquisition of new knowledge is quite slow. Instead, models often improve in the areas they are already familiar with. This suggests that while models may have the knowledge to solve a given task, they struggle to understand how to apply it effectively.

Wang et al. () demonstrates that model accuracy improves significantly when multiple reasoning chains are generated, indicating that the model understands how to answer the given problem. However, models often struggle to select the correct initial chain, and if they start on an incorrect reasoning path, it becomes difficult to fix it due to the autoregressive nature of decoding. Similarly, in our work, we observed that if a smaller model initiates an incorrect reasoning chain, it will continue down that incorrect path. Conversely, if the initial step is correctly determined, the model can successfully complete tasks that it would otherwise find challenging.

Figure: Figure 1: Comparison between Chain-of-Thought (CoT) approach and QuestCoT. The CoT approach enables a Language Model (LM) to generate accurate answers through multiple samplings, yet it frequently struggles to confidently select the correct one. Conversely, QuestCoT utilizes self-question-guided generation, which facilitates the model’s ability to choose the appropriate reasoning chain with higher confidence.
Refer to caption: x1.png

In this work, we first investigate whether providing initial guidance can improve the reasoning capabilities of smaller language models. We then investigate whether the quality of this initial guidance varies depending on the expertise of different large language models (LLMs). In particular, we investigate whether smaller models can use this guidance without fine-tuning or additional training, and whether models of different sizes benefit equally. Finally, we investigate whether the benefits of initial guidance extend beyond simple two-step problems to tasks requiring four to eight steps of reasoning.

Once the critical role of initial step guidance in reasoning is established, we focus on enabling smaller models to learn *how to start correctly*. To this end, we introduce QuestCoT, a self-questioning guidance mechanism designed to teach models how to start. With QuestCoT, the model first generates a sub-question that initiates the reasoning chain, and then follows that path. Essentially, it identifies the most effective reasoning chains needed to answer the given question. A comparison of our proposed methodology, QuestCoT  and Chain-of-Thought (CoT) is demonstrated in Figure 1.

We demonstrate the importance of self-questioning for initializing reasoning chains (QuestCoT) on several mathematical datasets involving multi-step word problems. Consistent performance improvements were observed for all smaller models (all within 7B parameters). Moreover, QuestCoT performs similarly to expert LLM guidance improving the quality of reasoning and outperforms the standard reasoning techniques of chain-of-thought (Wei et al., , CoT) and sub-question decomposition approaches (Shridhar et al., ; Zhou et al., , Subques).

## 2 Related Work

It is possible to elicit reasoning abilities from LLMs through in-context learning, either by providing the model with intermediate steps Wei et al. (); Kojima et al. (); Yang et al. (); Wang et al. (), or by decomposing the problem into smaller sub-problems Shridhar et al. (); Zhou et al. () and solving them to reach the final answer. However, if the problem is misinterpreted, it can lead to a cascade of errors in subsequent steps.

To counter this, several techniques have been proposed to intervene and correct intermediate steps by providing feedback on their own generations, and eventually “self-correcting” their own generations Welleck et al. (); Madaan et al. (); Shridhar et al. (). While the LLM’s ability to revise its own generations may prove helpful in many cases, it sometimes leads to worse results in refinement, requiring a “rollback” to the previous output Shridhar et al. (). To address this, Yao et al. () introduces the Tree of Thoughts (ToT), which plans subsequent steps to solve a reasoning task Huang et al. (); Wang et al. (, ). ToT conceptualizes the decision-making process as a series of heuristically based decisions. Through deliberate search, ToT explores different reasoning paths and self-reflects on its decision at each step. We, on the other hand, propose to get the first step right, thus reducing the cost of “finding” and “fixing” errors.

Previous work has also focused on understanding when to intervene and correct the errors.
Saha et al. () presented an approach based on Theory of Mind Kosinski (); Kadavath et al. (), where a teacher model intervenes in a student model only for harder questions by creating an implicit mental model of the student’s understanding. In contrast, an alternative that avoids the need to backtrack and correct mistakes, thus saving time and effort, is to *start right*.

Figure: Figure 2: Accuracy (if an answer exists in one of the output chains) comparison on GSM8K data set between different sized models: Mistral 7B, LLaMA-70B, and GPT-4.
Refer to caption: x2.png

## 3 First-Step Advantage

In this section, we address three research questions: 1) the ability of smaller models to solve a reasoning task, 2) the importance of taking the correct *first step* in reasoning, and 3) how smaller models can learn to take the correct first step.

### 3.1 Are smaller models capable of solving a reasoning task?

#### Hypothesis

*Smaller models can solve a given task but are not confident enough to choose the correct reasoning chain.*

For multi-step reasoning tasks, performance generally improves with increasing model size Wei et al. (). While this trend is generally observed, we hypothesize that smaller models can solve reasoning tasks (beyond what their maj@1 accuracy indicates), but often fail to choose the correct initial chain.

#### Experimental Design

We investigate the ability of smaller models to solve reasoning tasks by sampling their output chains multiple times [1, 3, 5, 10, 15, 20, 25, 30, 35].
A temperature setting of 0.7 is used to generate diverse multiple samples.
We compared the performance of the smaller model (Mistral-7B Jiang et al. ()) with the larger ones (LLaMA2-70B Touvron et al. (), and GPT-4 OpenAI ()) on the GSM8K dataset Cobbe et al. () for mathematical reasoning. Our analyses were conducted on a test set of 1,319 samples using a 4-shot Chain-of-Thought (CoT) reasoning chain. The prompts used are listed in the Appendix (Figure 10).

#### Our approach

To assess whether smaller models understand how to solve a problem but fail to select the correct reasoning chain on their first attempt, we generate multiple samples from the model and check whether a correct answer appears in any of them. This method will show that the model understands how to solve the problem, but has difficulty selecting the correct chain at first.

#### Results

Figure 2 illustrates the accuracy of whether an answer exists in one of the sampled outputs for different model sizes (ranging from Mistral 7B to LLaMA 70B to GPT-4, where the true size of GPT-4 is unknown but presumably large) when sampled multiple times. In a single sample, the performance gap between GPT-4 and Mistral 7B is nearly 50 points, indicating GPT-4’s superior ability to select the correct reasoning chain in its initial sample. However, with 35 samples, this gap narrows to less than 10 points, suggesting that smaller models can answer correctly but struggle to consistently select the right chain in their first few attempts. Conversely, GPT-4’s accuracy does not increase significantly with additional samples, suggesting greater consistency in generating and selecting the correct inference chain. This finding is consistent with Lyu et al. (), which suggests that scaling improves output consistency.

#### Key Findings

We observed that smaller models can answer a reasoning question when sampled multiple times, but fail to select the correct reasoning chain on the first attempt.

Figure: Figure 3: LLM-based first step guidance is highlighted in yellow followed by model generation.

**Table 1: Accuracy comparison when the first step is provided by a larger LLM versus the baseline (no first step provided) for a smaller model. The best results are shown in bold. Note that when a weaker model provides guidance (LLaMA2-70B performance is worse than Phi3-mini), it hurts the performance (underlined).**
|  | GSM8K | SVAMP |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Model | CoT | LLM Guidance | CoT | LLM Guidance |  |  |  |  |
|  | No guidance | LLaMA2-70B | GPT-3.5 | GPT-4 | No guidance | LLaMA2-70B | GPT-3.5 | GPT-4 |
| Gemma-2B | 7.50 | 12.81 | 16.23 | 17.84 | 34.60 | 36.30 | 46.70 | 49.20 |
| Phi3-Mini-3.8B | 76.95 | 75.10 | 77.39 | 80.27 | 86.30 | 84.20 | 86.10 | 87.80 |
| LLaMA2-7B | 10.53 | 19.48 | 21.00 | 23.27 | 38.00 | 40.10 | 41.40 | 48.20 |
| OlMo-7B | 13.64 | 28.20 | 36.54 | 37.90 | 18.60 | 40.90 | 46.50 | 49.90 |
| Mistral-7B | 40.25 | 46.17 | 48.82 | 49.50 | 62.00 | 65.60 | 66.80 | 73.40 |
| Gemma-7B | 46.55 | 52.23 | 59.43 | 63.45 | 70.30 | 72.10 | 74.10 | 78.30 |

### 3.2 Importance of starting right

#### Hypothesis

*Smaller models can solve a given task if they get the first step right*

To evaluate the impact of providing a hint in the form of the first step, we generate this first step using a large language model (LLM) and provide it as guidance to the smaller model. This approach allows us to test the importance of getting the *first step* right.

#### Experimental setup

We investigate whether providing *first-step* guidance can help smaller models get better results. We evaluate smaller models in the 2B - 7B range, namely Gemma-2B Team et al. (), Phi3-mini 3.8B Abdin et al. (), LLaMA2-7B Touvron et al. (), OlMo-7B Magnusson et al. (), Mistral-7B Jiang et al. (), and Gemma-7B Team et al. (). All the models are instruction-tuned versions except LLaMA2 and Mistral. For guidance coming from LLMs, we use LLaMA2-70B Touvron et al. (), GPT-3.5, and GPT-4 OpenAI (). We test our hypothesis on the test set of two datasets: GSM8K with 1319 samples and SVAMP Patel et al. () with 1000 samples. Greedy sampling (temperature=0) was used for sampling and acc@1 accuracy is reported.

#### Our approach

We use large language models (LLMs) to generate the first step of the solution to a given problem by providing specific instructions (details in the Appendix Figure 9). Although the first-step guidance varies by task, for mathematical reasoning tasks, we provide the first step until we encounter a mathematical equation. We perform sanity checks to ensure that no answer is revealed in this step (detailed analysis in section 4) and limit the solution to a maximum of one equation. Since the problem requires at least two to eight equations to solve, the first-step guidance does not lead directly to the answer but provides a solid starting point for the model. Smaller models then decode the answer by following this first-step guidance, and their final accuracy is compared to the baseline without first-step guidance. Figure 3 shows an example of LLM-based first-step guidance (highlighted in yellow).

#### Results

Table 1 demonstrates the usefulness of the first-step guidance provided by LLMs. The performance of the pre-trained models increases by more than 2-3X when a larger model such as GPT-4 is used for first-step guidance. For example, the performance of Gemma-2B Team et al. () and LLaMA2-7B model Touvron et al. () goes from 7.5 $\rightarrow$ 17.8 and 10.5 $\rightarrow$ 23.2, respectively, while for OlMo-7B it goes from 13.6 $\rightarrow$ 37.9 (an almost 3X jump). Performance increases monotonically with larger and more expert models providing first-step guidance (for Gemma-2B, performance increases from 7.5 $\rightarrow$ 12.8 with LLaMA-70B first-step guidance and to 16.2 with GPT-3.5). For the more expert models on the GSM8K task, Mistral-7B Jiang et al. () gains almost +10 points (40.25 $\rightarrow$ 49.50), Gemma-7B gains +17 points (46.5 $\rightarrow$ 63.4), and Phi3-Mini Abdin et al. () gains almost +4 points (76.9 $\rightarrow$ 80.2). The trend is similar for the SVAMP dataset, where accuracy goes from 34.6 $\rightarrow$ 49.2 for Gemma and 38.0 $\rightarrow$ 48.2 for LLaMA2-7B with GPT-4 first-step guidance, and from 18.6 $\rightarrow$ almost 50 for OlMo-7B. This shows the importance of the first-step guidance in arriving at the correct final answer.

#### Key Findings

We observed that when the first step of the reasoning chain is provided to the smaller models as a starting point, the performance of the smaller models improves significantly.

### 3.3 QuestCoT : Teaching smaller models to get the first step right

Figure: Figure 4: Example of a comparison between CoT reasoning and QuestCoT. QuestCoT first asks a question that helps to decide the first step and is highlighted in pink.

**Table 2: Accuracy comparison between the chain of thought (CoT) and QuestCoT. QuestCoT achieves the best results across all model sizes for various multi-step mathematical reasoning datasets.**
| Model | Dataset |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | GSM8K | SVAMP | ASDiv | MultiArith |  |  |  |  |
|  | CoT | QuestCoT | CoT | QuestCoT | CoT | QuestCoT | CoT | QuestCoT |
| Gemma-2B | 7.50 | 8.76 (<br>$\uparrow$ +1.1<br>) | 34.60 | 35.00 (<br>$\uparrow$ +0.4<br>) | 42.34 | 42.95 (<br>$\uparrow$ +0.6<br>) | 17.77 | 18.88 (<br>$\uparrow$ +1.1<br>) |
| Phi3-Mini-3.8B | 76.95 | 78.92 (<br>$\uparrow$ +2.0<br>) | 86.30 | 88.40 (<br>$\uparrow$ +2.1<br>) | 80.82 | 82.34 (<br>$\uparrow$ +1.5<br>) | 98.83 | 99.44 (<br>$\uparrow$ +0.6<br>) |
| LLaMA2-7B | 10.53 | 15.10 (<br>$\uparrow$ +4.5<br>) | 38.00 | 41.10 (<br>$\uparrow$ +3.1<br>) | 41.43 | 40.90 (<br>$\downarrow$ -0.5<br>) | 25.55 | 28.88 (<br>$\uparrow$ +3.3<br>) |
| OlMo-7B | 13.64 | 19.40 (<br>$\uparrow$ +5.8<br>) | 18.60 | 27.20 (<br>$\uparrow$ +8.6<br>) | 39.37 | 44.40 (<br>$\uparrow$ +5.0<br>) | 20.00 | 27.22 (<br>$\uparrow$ +7.2<br>) |
| Mistral-7B | 40.25 | 45.47 (<br>$\uparrow$ +5.2<br>) | 62.01 | 65.15 (<br>$\uparrow$ +3.1<br>) | 54.18 | 57.26 (<br>$\uparrow$ +3.0<br>) | 61.66 | 65.55 (<br>$\uparrow$ +3.9<br>) |
| Gemma-7B | 46.55 | 48.21 (<br>$\uparrow$ +1.6<br>) | 70.30 | 71.40 (<br>$\uparrow$ +1.1<br>) | 68.59 | 69.84 (<br>$\uparrow$ +1.2<br>) | 79.44 | 78.22 (<br>$\downarrow$ -1.2<br>) |
| LLaMA3-8B | 78.86 | 79.80 (<br>$\uparrow$ +1.0<br>) | 83.70 | 84.89 (<br>$\uparrow$ +1.2<br>) | 73.88 | 74.27 (<br>$\uparrow$ +0.4<br>) | 97.77 | 98.33 (<br>$\uparrow$ +0.5<br>) |

#### Hypothesis

*Can smaller models learn to get the first step right?*

Given that smaller models can get better results if they learn to start right, can we teach smaller models to learn the first step on their own?

#### Experimental setup

We explore the effect of *starting right* on four multi-step mathematical data sets: GSM8K Cobbe et al. (), SVAMP Patel et al. (), ASDiv Miao et al. (), and MultiArith Roy and Roth ().
GSM8K consists of grade-school math word problems with a test set of 1319 samples, requiring between two and eight steps to solve.
SVAMP consists of 1000 samples of math word problems designed to challenge systems that require reasoning beyond shallow approaches.
ASDiv consists of 2,305 test samples of word problems that were constructed to have more lexical diversity than other datasets at the time.
MultiArith is a dataset of 180 test samples published with the algorithmic solver for mathematical word problems.

We tested smaller models ranging from 2B to 8B parameters, starting with Gemma-2B, followed by Phi3-mini with 3.8B parameters, followed by Mistral-7B, LLaMA2-7B, OlMo-7B, and Gemma-7B with 7B parameters, and finally LLaMA3-8B with 8B parameters. We report the top-1 accuracy (maj@1) on the test sentences of both datasets. To compare CoT and QuestCoT, we used 4-shot prompting with prompts randomly selected from the test set. All models were evaluated using a greedy approach (temperature=0, top p=1). A comparison of prompts between CoT and QuestCoT can be found in the Appendix (Figure 11).

#### Our approach

To help the smaller models learn how to start with a correct first step, we propose an initial question-guided strategy called QuestCoT. With QuestCoT, a model first asks the most important question that will help it start the reasoning chain and then continues that chain. The initial question it asks can also be thought of as a search strategy that looks for the right starting chain and, once selected, continues along that path. A comparison with CoT and QuestCoT is presented in Figure 4. Note that the model learns this questioning itself, and the only change from CoT is to add an extra question in the prompt as a demonstration.

Figure: Figure 5: Accuracy comparison between Subques and QuestCoT on the GSM8K and SVAMP datasets. Gemma refers to Gemma-2B, Phi3-Mini is Phi3-mini-3.8B, and LLaMA2, OlMo, and Mistral are all 7B variants, while LLaMA3 is LLaMA3-8B.
Refer to caption: x3.png

#### Results

We test the effectiveness of QuestCoT against one of the most popular reasoning strategies: CoT. QuestCoT outperforms CoT on all four datasets for all models except LLaMA2-7B on ASDiv and Gemma-7B on MultiArith. Smaller models such as Gemma-2B and Phi-mini-3.8B gain between $+0.5$ and $+2$ points on all four datasets. We hypothesize that Gemma-2B’s limited gains are due to its initial weak performance and undertraining, while Phi3-mini is already a very strong model with performance in the 80s and 90s, making further improvement difficult. Nevertheless, improvements are observed in both cases.

Performance improves significantly with the 7B models, with OlMo-7B showing the most gains ($+6$ on GSM8K, $+9$ on SVAMP, $+5$ on ASDiv, and $+7$ on MultiArith). This is followed by LLaMA2-7B and Mistral-7B, which show gains of $+3-5$ points, and Gemma-7B, which shows gains of $+1-2$ points. Similar to Phi3-mini, LLaMA3-8B’s baseline performance is quite high, showing gains of $+0.5-1$ points.

#### Key Takeaways

Smaller models improve their performance by learning to get the first step right by asking themselves how to start. This improvement is achieved with our proposed approach, QuestCoT.

## 4 Analysis

#### Does the first step leak the final answer?

We investigate whether the performance gains from LLM guidance are due to LLMs leaking the answer to the smaller models. To verify this, we created a development set of 1000 samples from the GSM8K training set. By comparing the generated first-step answers with the final answers in the dataset, we found that in 999 out of 1000 samples, the answers did not match. Furthermore, our instructions to the LLMs specified that they could only generate the first step, corresponding to the first step in the inference chain with only the first equation, and could not reveal the final answer. This strategy was applied consistently across all data sets. Since each question required at least 2-8 steps to solve, we are confident that the final answer was not revealed. Furthermore, if the approach relied on revealing the final answer, the QuestCoT approach would not have been effective in the prompt style at all.

Figure: Figure 6: Accuracy comparison between baseline (no guidance) and LLM guidance (GPT-4) for the Mistral-7B model on the GSM8K dataset. 2-8 represents the number of steps required to solve the problem.
Refer to caption: x4.png

#### Can first-step guidance go beyond two-step problems?

Figure 6 illustrates the performance of the Mistral-7B model with and without first-step LLM guidance for different steps in the GSM8K dataset. For all steps (2 to 8), first-step guidance improves performance, suggesting that starting with a solid foundation can help reasoning over a longer context.

Figure: Figure 7: Venn diagram to show when different strategies got the solutions right.
Refer to caption: x5.png

#### What if subquestions are included at each step?

The subquestion that guides the model on *how to start* can be applied to any reasoning step in the chain to guide that specific step. This approach is similar to subquestion decomposition (Shridhar et al., ; Zhou et al., , Subques), where a complex reasoning problem is first broken down into simpler problems that are then solved sequentially. Figure 5 shows a comparison between Subques and QuestCoT over different models on two datasets: GSM8K and SVAMP. QuestCoT shows higher accuracy across all models on both datasets while incurring lower token costs. Through manual inspection of over 100 samples, we found that introducing additional questions increases the likelihood of errors propagating through the inference chain. In contrast, QuestCoT guidance avoids this by focusing solely on *how to start*. These errors can result from incorrect decomposition, incorrect reasoning about the decomposed problem, or inconsistencies throughout the inference chain.

#### Comparing CoT, Subques and QuestCoT

Figure 7 presents a Venn diagram illustrating when different strategies correctly solve problems on the GSM8K dataset using the Phi-3 mini-model. QuestCoT has more overlaps with both CoT and Subques (82 and 95, respectively) than the overlaps between CoT and Subques (71). This suggests that while CoT and Subques each have unique cases where they are successful, QuestCoT captures the strengths of both strategies, resulting in better overall performance.

#### Can QuestCoT work with even smaller models?

We tested our approach on the OlMo-1B model, which has 1 billion parameters, and found that it was not well equipped to understand the instructions or to generate a reasoning chain starting with an initial question (a necessary condition for QuestCoT). On the GSM8K dataset, the CoT performance was 3%, and QuestCoT performed comparably at 3.5%, with the outputs not looking significantly different. As a result, we did not observe any statistically significant improvements. We suspect that because the OlMo-1B model’s CoT abilities are quite limited in mathematical reasoning tasks, it cannot leverage any advantages from QuestCoT.

## 5 Deeper exploration of why QuestCoT works

Figure: Figure 8: Example cases associated with different error types and how QuestCoT assists in getting the correct reasoning chain.

Understanding why one technique outperforms another can be quite challenging. To address this, we examined instances where QuestCoT was successful and conducted a detailed analysis. We identified situations where CoT failed and broadly categorized these errors. Here are some error types where QuestCoT was beneficial:

#### Unnecessary calculations

A common mistake CoT makes is performing unnecessary calculations on the numbers in the statement. These numbers may be completely irrelevant to the problem, or they may need to be used in a different way than the model uses them. QuestCoT helps to correct these errors by initiating the reasoning process with an appropriate question. An example of an unnecessary calculation is given in Figure 8 with the main error shown in red, where there was no need to calculate the total cost of coffee for the entire week at the old price.

#### Real-world knowledge

Often, the first step in CoT seems somewhat "rushed," focusing on quickly manipulating numbers without considering real-world facts or knowledge. In these cases, the model demonstrates its understanding of these facts and knowledge in the subsequent steps but cannot elicit it immediately in the first step. This suggests that encouraging the model to think more deliberately in the first step (e.g., by allowing it to consider what needs to be done before it starts reasoning) may remedy this problem. These scenarios illustrate the effectiveness of QuestCoT. An example is shown in Figure 8, where the model fails to convert "half a dozen" to 6, and instead continues its calculations with 0.5 (as shown in blue). Although the model demonstrated its understanding of "dozen" later in the problem, since it started incorrectly, it was unable to correct the chain later.

#### Context understanding

With CoT, the model often confuses or misses the context in the problem statement and makes incorrect initial assumptions that are difficult to recover from in later steps. For example, in Figure 8 we can see that despite following a fairly elaborate reasoning template of variable assumptions, the CoT reasoning misses the fact that the price of the Super Ticket already includes the price of the popcorn. The incorrect assumption is highlighted in the response.

#### Other errors

Other errors we have observed include that QuestCoT may be better at handling direct numeric computations and understanding the simple arithmetic required by the problems. In contrast, CoT may deviate or fail to capture the essential computational aspects of the query. In addition, CoT sometimes takes more steps than necessary, resulting in an incorrect final solution.

## 6 Conclusion

We find that smaller models sometimes struggle with taking the correct first step, but their performance increases significantly once this step is corrected. We demonstrated this by using LLMs to guide smaller models to take the correct first step, helping them to establish the correct reasoning chain. To facilitate this for smaller models, we propose QuestCoT, which uses initial question-based guidance to improve their reasoning without any guidance. We demonstrate the effectiveness of our approach on four multi-step mathematical reasoning datasets using different open-source small models.

## 7 Limitations

Our experiments focus only on English datasets, and we have not tested the performance of our methods in other languages. We acknowledge that including a sub-question to initiate the chain of reasoning may incur some additional cost compared to the chain-of-thought approach. However, it is significantly less costly than the sub-question decomposition approach and yields superior performance compared to both methods.

## 8 Ethical Considerations

The initial guidance provided by expert LLMs or the self-questioning mechanism could introduce or perpetuate bias due to the unknown training process of the large LLMs (especially the closed-source LLMs such as GPT-4). It’s crucial to evaluate and mitigate any biases in the generated output of the LLMs.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: I wonder why the Gemma 2B and Mistral 7B GSM8K performance is significantly below the officially reported numbers. This may need more clarification to make the results more credible
soundness: 3
strengths: The authors clearly articulated why the first step matters, and give a detailed performance analysis of Quest CoT

- Clear articulation of why the first step matters: the authors use experimental results and concrete examples to show that the first step correctness is important to final answer accuracy.
- Additional analysis: the authors use fair followup analysis to discuss more details like data leakage, comparisons between different types of CoT, and if QuestCoT work with smaller models.
summary: This paper introduces QuestCoT, where the model is instructed to first ask itself how to start, then start the chain-of-thought reasoning by its self-generated guidance question. The QuestCoT approach is motivated by the observation that the first step correctness is important for the model to achieve the correct final answer. The authors further show that if there is a stronger larger model to guide the first step, the small model would consequently reason better. The authors also provide a list of analyses about why QuestCoT works.
weaknesses: While the authors take inspiring steps towards strengthening CoT, I generally feel the experiments are insufficient to support an ICLR quality paper. Specifically:

- The GSM8K performance of Gemma 2B in this paper Table 2 is only 7.5, while on its official report is 17.7, the gap is 10.2. The Mistral 7B performance in Table 2 is 45.4, while in Mistral blog is 52.1, the gap is 6.9. Such large gap is in generally questionable. Given that Gemma and Mistral are well-established models whose performance is validated by a large portion of the community, I wonder why the authors’ numbers are so different than their official reports.
- This paper only considers GSM8K, SVAMP, ASDiv and MultiArith, all of them are quite old. The field have moved on, and the authors need to use stronger models (like Qwen 2, DeepSeek) and harder and broader datasets (MMLU-Pro, GPQA .etc) to demonstrate the effectiveness of their methods — simply use GSM8K-sytle benchmarks is insufficient.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: My main concerns have already been expressed in the "weakness" section.
soundness: 2
strengths: - The paper is well-written with clear and fluent prose. Additionally, the figures and tables are well-designed and effectively present the data.
- The key finding that smaller models can significantly improve their performance when corrected by larger models is insightful and empirically investigated well.
summary: The authors note that smaller models, when corrected, can handle tasks better, which is demonstrated by using larger models to guide them, resulting in improved performance (up to +24 points on the GSM8K dataset for 7B models). They propose QuestCoT, where a smaller model first asks how to start before proceeding with reasoning. The method shows performance gains across various mathematical reasoning datasets (up to +6 points on GSM8K, +9 on SVAMP, +5 on ASDiv, and +7 on MultiArith).
weaknesses: - The novelty of the QuestCoT method is somewhat lacking. It seems more like a new template for chain-of-thought reasoning presented in a question-and-answer-in-first-step format rather than a fundamentally innovative approach.
- The "Why QuestCoT works" part in Section 3.3 is confusing, especially when it is associated with the "What if subquestions are included a teach step" part in Section 4. If the analysis in "Why QuestCoT works" makes sense, we could also draw a conclusion that Subques would be better than QuestCoT.This contradicts the results in "What if subquestions are included a teach step". This paper lacks analysis and explanation for this discrepancy.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: Address the weakness pointed out in the above section
soundness: 3
strengths: 1. The paper proposes a new framework to increase model robustness to solve mathematical word problems.
2. The author clearly demonstrates the problems with current SLMs and shows how providing the first step will boost model performance.
3. Extensive experiments to answer the three questions stated in Section 3
4. Great analysis of the results, where the model talks about the effect of first-step guidance on a longer reasoning chain, does the first step leak the final answer? Etc
summary: Large Language Models (LLMs) have the capability to solve complex reasoning tasks by generating each intermediate reasoning step. Previous works show that model accuracy improves significantly when multiple reasoning chains are generated, indicating that the model understands how to answer the given problem. However, the model often struggles to select the correct initial chain and when it starts on an incorrect path it is difficult to fix it due to its autoregressive nature of decoding. Therefore, the author in this work answers three main questions:
1. Are smaller models capable of solving a reasoning task?

By sampling multiple reasoning chains while keeping a high temperature, they show that smaller models can reach the correct answer but struggle to consistently select the right chain in their first few attempts. 

2. What is the importance of taking the correct first step in reasoning?

Here the author investigates the change in performance when the first step is provided as input along with the question. They observe a significant improvement in the accuracy. Therefore choosing the right first step is very important.

3. Can smaller models learn to take the correct first step on their own?

Here they introduce their novel framework called QuestCOT, where the model self questions about the first step before starting to answer the question. When asking the subquestion it helps the model focus on the correct reasoning path, as the model’s predictions are more concentrated around the correct reasoning path, increasing the likelihood of producing the correct answer.

The author then compared QuestCOT with COT on GSM8K and SVAMP datasets, showing its increased performance.
weaknesses: 1. It is unclear how the questions are generated for each question, the prompt used to generate them, and how the quality of questions asked affects the performance of the model. An ablation study on the subquestions asked to the model can help better understand QuestCOT
2. In Section 3, the author talks about how “introducing additional questions increases the likelihood of errors propagating through the inference chain”. Can the author provide more evidence on why error increases when subquestions are asked at every step vs. only on the first step
3. Running more baselines: Comparison with more existing methods such as Self-consistency[1], Tree of thought [2] etc and more difficult datasets such as MATH, MATHBENCH etc,  will help prove the robustness of QuestCOT.

Reference:

[1] : Wang, Xuezhi, et al. "Self-consistency improves chain of thought reasoning in language models." arXiv preprint arXiv:2203.11171 (2022).

[2] : Yao, Shunyu, et al. "Tree of thoughts: Deliberate problem solving with large language models." Advances in Neural Information Processing Systems 36 (2024).

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. As this work aims to emphasize the importance of the first step, is there any evidence to support that the first step is more important than other intermediate steps?
soundness: 3
strengths: 1. This work validate the importance of the initial step in the multi-step reasoning.
2. The authors provide experimental results to reveal that small language models can achieve the correct final results with multiple sampling, but fail to accurately find out the correct one.
3. This work QuestCoT to improve the reasoning performance of small language models by improving the generation quality of the initial step.
summary: This work focuses on analyzing and improving mathematical reasoning for relatively small language models.
The authors experimentally demonstrate the importance of the initial step in the multi-step reasoning process and also show the limitation of small language models.
Moreover, the paper introduces a novel approach called QuestCoT (Questioning Chain of Thought), which enables smaller models to self-generate a preliminary question that guides the subsequent reasoning chain.
weaknesses: The main limitation lies in the novelty of this work.
For the three key research questions explored in this work, the first two about the capability of small language models have been well studied in [1].

[1] Common 7B Language Models Already Possess Strong Math Capabilities.