Self-Training with Direct Preference Optimization Improves Chain-of-Thought Reasoning (2024)

Abstract: Teaching small-scale language models to perform math reasoning is a valuable yet challenging task. Besides obtaining labeled data from human experts, one of the most common ways to collect high-quality data is by sampling from a larger and more powerful language model. Although previous works have demonstrated the effectiveness of this method, such a knowledge distillation paradigm can be costly and unstable, especially considering that many large language models, such as GPT-4 (OpenAI, 2023), are closed-source, proprietary, and their behaviors are unpredictable. In this work, to avoid relying on outputs from large models, we demonstrate that the reasoning abilities of small-scale language models can be enhanced through self-training, which involves training models with their own outputs. We also show that the conventional self-training can be further augmented by an alignment algorithm called Direct Preference Optimization (DPO) (Rafailov et al., 2023). We empirically found that models trained with the DPO objective are capable of making better generations that largely benefit multi-turn self-training. The experimental results show our models outperform the existing models with comparable sizes on the GSM8K benchmark with minimal resource requirements.1

heading: 1 Introduction
text: Making language models (LMs) perform mathematical reasoning is a valuable, yet challenging research objective (Hendrycks et al., 2021; Cobbe et al., 2021). Recent works focus on enhancing large-scale LMs’ reasoning abilities, e.g., chainof-thought prompting (Wei et al., 2022b; Kojima et al., 2022), continual pretraining (Azerbayev et al., 2024), or adding external verifiers (Li et al., 2023b). However, the research question of how to
1Our code and data are released at https://github.com/ tianduowang/dpo-st.
improve LMs with smaller sizes (e.g., less than 1 billion parameters) is still under-explored.
Recent studies (Fu et al., 2023; Ho et al., 2023; Magister et al., 2023) demonstrate that the reasoning capabilities of smaller LMs can be significantly enhanced through learning from the outputs of larger and more advanced LMs, such as Codex (Chen et al., 2021) and PaLM (Chowdhery et al., 2022). Despite the relative ease of implementation, the associated costs can be substantial. The computational demand, measured in floatingpoint operations (FLOPs), is considerably higher for large LMs. Additionally, the use of proprietary and closed-source large LMs for data annotation can incur significant economic costs. Ho et al. (2023) demonstrated that annotating multiple reasoning chains for a single question with large LMs can markedly enhance the performance of smaller models. Hence, a trade-off between cost and performance exists.
11917
Another line of work focuses on making improvements via self-training (Zelikman et al., 2022; Gulcehre et al., 2023; Singh et al., 2023). Instead of using data generated by larger models, the essence of self-training methods is to make small models learn from their own generations. While these methods demonstrate the effectiveness of utilizing self-generated data, their success largely depends upon the pre-existing abilities of the models. For example, Zelikman et al. (2022) started by few-shot prompting a large model, i.e., GPT-J (Wang and Komatsuzaki, 2021), to self-generate rationales, which is an emergent ability that only comes with sufficiently large models (Wei et al., 2022a). However, it is still unclear whether small-scale LMs can benefit from self-training.
Recently, we have witnessed that reinforcement learning from human feedback (RLHF) has emerged as a prominent method to precisely modify LMs’ behavior towards human preference (Ouyang et al., 2022; Casper et al., 2023). In this work, we propose to augment the self-training algorithm with an RLHF training process, i.e., Direct Preference Optimization (Rafailov et al., 2023), for its better performance and stability. Our experimental results demonstrate that the proposed method can significantly improve LMs’ reasoning capabilities while minimizing the computational costs. We visualize the relationship between the downstream task performance and computational cost over a series of specialized models in Figure 1. The computational costs for each model are estimated following previous practice (Kaplan et al., 2020; Yuan et al., 2023). It can be observed that our method not only achieves the highest accuracy, but also minimizes the computational demand by learning from its own generations. Overall, the main contribution of this work can be summarized as follows:
• We propose a novel extension to the classic self-training framework with Direct Preference Optimization, and we show its effectiveness through standard math problem-solving tasks.
• We demonstrate that this novel extension improves the reasoning abilities of the LMs with minimal computational resource requirements.
• We propose an efficient method for integrating LMs with external tools, significantly improving the performance without sacrificing much inference speed.
Algorithm 1 Self-training for CoT reasoning tasks
Input: pre-trained language model fθ Input: labeled dataset L = {(xi, yi, ai)}li=1 Input: unlabeled dataset U = {(xi, ai)}ui=1 Output: fine-tuned model fθ′
1: Fine-tune fθ on L to get fθ′ 2: repeat 3: Build pseudo-labeled dataset S:
S = {(xi, ŷi, âi)}si=1 where xi ∼ U and ŷi, âi ∼ fθ′(·|xi)
4: Select Sα ⊂ S when âi = ai 5: Update L ← Sα ∪ L 6: Train fθ on L to get a new fθ′ 7: until convergence or max iteration is reached

heading: 2 Background
text: Math word problem solving The math word problem solving task can be formulated as a sequence-to-sequence task where the input x is a question asking for an unknown value and the output y is a rationale leading to the answer a (Cobbe et al., 2021). Normally, the answers can be extracted from the rationales via some rule-based methods, e.g., regular expressions. A generated rationale ŷ is regarded as correct if the extracted answer â matches the gold answer a. Formally, the labeled dataset for a math word problem solving task with l instances can be represented as:
L = {(xi, yi, ai)}li=1. (1)
A common way for specializing a LM fθ towards math reasoning with the labeled dataset L is supervised fine-tuning (SFT). It optimizes fθ by minimizing the negative log likelihood loss LSFT(θ):
E (x,y)∼L
− [ T∑
t=1
log fθ(yt|x, y1:t−1) ] , (2)
where T is the length of the rationale y and we use yt to represent the t-th token in y.
Self-training Self-training is one of the earliest approaches in semi-supervised learning (Scudder, 1965; Fralick, 1967) that has risen in popularity recently (He et al., 2019; Amini et al., 2022). This method first regards a base model trained with a labeled dataset L as teacher, and uses it to build a pseudo-labeled dataset S by annotating an unlabeled dataset U . Then, a student model is trained with the combination of L and S that are expected
to outperform the teacher model. Such a framework has been shown effective in a wide range of natural language processing tasks, e.g., natural language understanding (Vu et al., 2021) and generation (He et al., 2019). A formal description of a self-training algorithm for CoT reasoning tasks is provided in Algorithm 1.
Previous studies have demonstrated that the quality of the pseudo-labels largely impacts the overall performance of the self-training algorithm (He et al., 2019; Amini et al., 2022). For example, Gulcehre et al. (2023) proposed to select high-quality pseudo-labels with a learned reward function. Zelikman et al. (2022) filtered the generated rationales to include only the ones that lead to correct answers. Although many methods are proposed to select pseudo-labels, few works discuss how to improve the fine-tuned model fθ′ so that more high-quality pseudo-labels can be generated. In this paper, we present a method to enhance fθ′ in each iteration so that higher-quality pseudo-labeled data can be generated.
Direct Preference Optimization The Reinforcement Learning from Human Feedback (RLHF) methods align LMs with human preference (Ouyang et al., 2022; Bai et al., 2022). The standard pipeline of RLHF requires to first train a reward model from human preference data. Then, the reward model is used to fine-tune language models via reinforcement learning objective, e.g., Proximal Policy Optimization (Schulman et al., 2017). A recent study propose Direct Preference Optimization (DPO) (Rafailov et al., 2023) to avoid explicitly training a reward model so that language models can be directly tuned with human preference data.
The DPO pipeline can be described as follows. First, given some prompt x, sample several completions from the reference model πref (normally it is the model after supervised fine-tuning):
y1, y2 ∼ πref(· | x). (3)
Next, construct the DPO datasetD from the completions based on the human preference:
D = {( xi, yiw, yil )}Ni=1, (4)
where yiw and y i l represent the winning and losing completions respectively. Then, we optimize the language model πθ to minimize LDPO(πθ;πref)
Algorithm 2 DPO-augmented self-training
Input: pre-trained language model fθ Input: labeled dataset L = {(xi, yi, ai)}li=1 Input: unlabeled dataset U = {(xi, ai)}ui=1 Output: fine-tuned model fθ′
# Warm-up stage 1: Fine-tune fθ on L to get fθ′ 2: repeat # DPO step 3: Generate DPO dataset D:
D = {( xi, yiw, yil )}Ni=1 where xi ∼ U and yiw, yil ∼ fθ′(·|xi)
4: Tune fθ′ with LDPO on D to get fθd # SFT step 5: Build pseudo-labeled dataset S: S = {(xi, ŷi, âi)}si=1 where xi ∼ U and ŷi, âi ∼ fθd(·|xi) 6: Select Sα ⊂ S when âi = ai 7: Update L ← Sα ∪ L 8: Train fθ on L to get a new fθ′ 9: until convergence or max iteration is reached
which can be defined as follows:
E (x,yw,yl)∼D
[ − log σ ( r(yw|x)− r(yl|x) )] , (5)
where r(·|x) = β log πθ(·|x)πref(·|x) and β is a coefficient that controls πθ’s deviation from πref.

heading: 3 Method
text: In this section, we first describe the proposed approach. Then, we demonstrate how we integrate an external calculator into the model’s decoding process which significantly improves LMs’ performance on the downstream tasks.

heading: 3.1 DPO-Augmented Self-Training
text: Our approach starts with a warm-up stage, and then followed by an iterative process, where each iteration is composed of two sub-steps: DPO step and SFT step. The iterative process ends when the model performance converges or reaches the maximum iteration. A formal description of the proposed method is illustrated in Algorithm 2. An illustration of our method is presented in Figure 2.
Warm-up stage Like classic self-training, we start by fine-tuning the base model fθ to optimize LSFT(θ) on the labeled data L to get a new model

heading: Deduplication
text: fθ′ . After this stage, we assume that fθ′ is capable of solving certain math problems. Specifically, given a math question x, fθ′ will generate a rationale ŷ with answer â.
Iterative step 1: DPO step In this step, we first sample rationales ŷ from the fine-tuned model fθ′ given some questions x from U . For each question x, we generate multiple rationales to build the DPO training dataset D. As mentioned, for math problem solving tasks, it is easy to know whether a generated rationale ŷ can be considered as correct. We label rationales with correct answers as winning completions, while consider rationales with incorrect answers as losing completions. Then, we train fθ′ on D to optimize the objective function LDPO and get a DPO model fθd in the end.
Iterative step 2: SFT step After obtaining fθd , we use it to generate a new pseudo-labeled dataset S for the next-round supervised fine-tuning:
S = {(x, ŷ)|x ∼ U , ŷ ∼ fθd(·|x)} (6)
After generation, we clean S by eliminating rationales with incorrect answers and removing duplicates. Therefore, the pseudo-labeled dataset we obtained in the end is a subset of the original one, i.e., Sα ⊂ S . The final training dataset is the combination of the original labeled dataset L and the newly-generated pseudo-labeled dataset Sα.
Notice that during this process, once we collect a new dataset, we train from the original base model fθ instead of continually fine-tuning fθ′ to avoid overfitting, following previous practice (Zelikman et al., 2022; Singh et al., 2023).

heading: Q: James writes a 3-page letter to 2 different friends twice a week. How many pages does he write a year?

heading: A: He writes each friend
text: 3*2=<<3*2=6>>6 pages a week.

heading: So he writes
text: 6*2=<<6*2=12>>12 pages every week.

heading: That means he writes
text: 12*52=<<12*52=624>>624 pages a year. #### 624
Figure 3: An example from the GSM8K dataset. The calculation annotations are highlighted in blue. All calculation steps are wrapped within special tokens <<...>>. During decoding, the calculator will be triggered when such patterns exist and the model’s output tokens will be overridden by the calculator results. Following Cobbe et al. (2021), the calculation is performed with the python eval() function.

heading: 3.2 Batch Decoding with Calculator
text: We empirically observed that, in contrast to large LMs which excel at basic arithmetic calculations (Brown et al., 2020), smaller LMs like FlanT5-Large exhibit poor performance on similar arithmetic tasks. This deficiency significantly impacts their ability to handle math reasoning tasks. To address this, various studies (Parisi et al., 2022; Schick et al., 2023; Kadlčík et al., 2023) have proposed augmenting these smaller models with an external calculator to enhance their math reasoning capabilities. However, many of these existing methods are limited to a batch size of one during decoding. This constraint substantially reduces the inference speed, which hinders their widespread adoption.
To alleviate this issue, we propose a simple yet efficient method to enables the use of a large batch size during inference with an external calculator. We adopt the calculator annotation provided in the original GSM8K dataset (Cobbe et al., 2021). Figure 3 demonstrates an example of this annotation and describes how such annotations can be used during decoding. Our models are built with the Transformers library (Wolf et al., 2020). During generation, we adopt a customized LogitsProcessor2 to override the model’s generation. LogitsProcessor provides an interface to modify the language model’s output tokens during generation.
To demonstrate the efficiency of the proposed solution, We compare the inference speed of our methods (w/ and w/o calculator) based on Flan-T5Large against an open-source tool-using method, Calcformer (Kadlčík et al., 2023) based on T5Large, in Figure 4. We find that when the batch size equals 1, all three methods have a similar inference speed of around 40 tokens per second. However, as the inference batch size increases, the speedup of our methods increases significantly.

heading: 4 Experiments
text: In this section, we first describe our experimental setup. Next, we present the performance of our models across a series of math word problem solving tasks, comparing them against a selection of competitive baselines. Finally, we empirically analyze what makes the proposed method effective.
2https://huggingface.co/docs/transformers/ internal/generation_utils#logitsprocessor

heading: Dataset Split # Data

heading: 4.1 Setup
text: Base models We employ Flan-T5 models (Chung et al., 2024) as the base models in our experiments. Specifically, we consider two models of different sizes from the Flan-T5 model family: Flan-T5Base (250M) and Flan-T5-Large (780M). We select Flan-T5 over the original T5 models (Raffel et al., 2019) as our backbone models based on the evidence from previous research (Chung et al., 2024; Fu et al., 2023), which demonstrated that instruction-tuned models, i.e., Flan-T5, outperform their pre-trained counterparts in math reasoning tasks. Besides Flan-T5 models, we also consider the Llama models (Touvron et al., 2023a,b; Meta, 2024) as our base models.
Datasets The labeled dataset L used in our experiments comes from the training split of the GSM8K dataset. Our unlabeled dataset U is also built upon GSM8K’s training data by removing its annotated rationales y. For evaluation, we consider three other commonly used math reasoning tasks besides GSM8K: MultiArith (Roy and Roth, 2015), ASDiv (Miao et al., 2020), and SVAMP (Patel et al., 2021). Table 1 shows the statistics information of each dataset. Following previous practice (Fu et al., 2023), we only fine-tune the base models on the GSM8K training data while utilizing the rest three datasets to evaluate our models’ out-of-domain performance as they do not have an official in-domain training split.
Evaluation metrics We use accuracy of the greedy decoding results as the main evaluation metric. The questions in the datasets in our experiments ask about the values of the unknown variables. The answers to these questions are real numbers that can be extracted from the model-generated rationales.
Implementation details In every DPO step, we sample rationales from the SFT model fθ′ to build the DPO training data. We sample 5 rationales from fθ′ per question with a temperature of 0.7. We regard generated rationales ŷ as winning ones yw if it contains the correct answer, while regarding the rest as the losing ones yl. For the SFT steps, we generate 3 rationales per question from the DPO-tuned model fθd also with a temperature of 0.7. Only the correct generated rationales ŷ will be selected to build the pseudo-labeled dataset S. For both generated DPO data and SFT data, we make simple deduplication based on the Jaccard similarity scores with a threshold of 0.7.

heading: 4.2 Main Results
text: Baselines We mainly consider two baseline methods to compare with our method: Supervised FineTuning (SFT) and Self-Training (ST). SFT baselines are trained on the original GSM8K annotations with theLSFT(θ) objective. The Self-Training baseline adheres to the procedure outlined in Algorithm 1. Consequently, this makes it the most directly comparable and relevant baseline for evaluating the effectiveness of our approach.
Comparison with baselines Table 2 shows the performance of our method compared with the baselines using two base models over four datasets. It can be observed that both the Self-Training baseline and our proposed method outperform the supervised fine-tuning baseline by a large margin, indicating the effectiveness of the general self-training framework. Although the Self-Training baselines make significant improvements over the SFT baselines, the proposed DPO-augmented Self-Training models consistently outperform them on both indomain and out-of-domain tasks.
Effect of iterative training In Figure 5, we demonstrate how our model improve over mulitple iterations using two base models: Flan-T5-Base and Flan-T5-Large. Both the ST baseline and our method start with a SFT step using the original GSM8K annotation at iteration 0. We can observe that, as iterations progress, our method consistently outperforms ST, which suggests that the proposed method offers a substantial improvement over ST after repeated training cycles. We also notice that the degree of improvement gradually approaches zero as the iterative process proceeds, suggesting that the model has converged in the last iteration.

heading: 4.3 Comparison with Existing Methods
text: In this section, we compare our methods with existing approaches. Additionally, we scale up the proposed method by increasing the number of sampled pseudo-labels per question, denoted by the hyperparameter K as in Yuan et al. (2023).
Table 3 presents a detailed comparison between our method and exisiting methods using a simialr base model size. The base models we considered include GPT-2-Large (Radford et al., 2019), T5-Large (Raffel et al., 2019), and FlanT5-Large (Chung et al., 2024). All these models have approximately 770 million parameters. As shown in Table 3 our approach not only outperforms other methods on the GSM8K benchmark, but also demonstrates remarkable label efficiency by utilizing only the annotations from the original GSM8K dataset.
In Table 4, we further verify the effectiveness of the proposed method with the Llama model family (Touvron et al., 2023a,b; Meta, 2024). comparing them with several state-of-the-art closed-source models as well as open-source models with similar sizes. We observe a substantial performance gap between proprietary and open-source models. Among the open-source models, those utilizing knowledge distillation consistently outperform their counterparts without such enhancement. Notably, our models using Llama-1-7b (Touvron et al., 2023a) and Llama-2-7b (Touvron et al., 2023b) base models surpass other open-source alternatives that do not employ knowledge distillation, achieving accura-
cies of 44.7% and 54.7% respectively. Furthermore, our model employing the latest Llama-3-8b (Meta, 2024) matches or exceeds the performance of earlier models with knowledge distillation, demonstrating a significant accuracy of 68.8%.

heading: Ours

heading: Models w/o knowledge distillation

heading: Models w/ knowledge distillation

heading: Closed-source models

heading: Method Base Model Acc.

heading: 4.4 Effects of the DPO Step
text: As mentioned earlier, the main difference between the proposed method and the classic self-training is the DPO step in every iterative process. We now analyze how the DPO steps improve self-training. Figure 6 compares the performance of models before and after the DPO step in the first iteration on the Pass@K metrics. Pass@K measures the probability that at least one of the K generated solutions for a problem is correct, which serves as a gauge for both the quality and the variety of the modelgenerated solutions. The models we investigate here are fine-tuned from the Flan-T5-Large.
As shown in Figure 6, the DPO step yields only marginal improvements over the SFT model in the Pass@1 performance on the development set. However, the performance significantly improves when multiple rationales, i.e., 10 solutions per question, are sampled with temperature 0.7 (measured with the Pass@10 metric). This indicates that the DPO training objective makes language models inclined to generate rationales of both high quality and diversity. We also compare the number of generated rationales on the training set L for models with and without the DPO step. Figure 6 (right) clearly shows that the model after the DPO step can produce more SFT data for the next iteration.

heading: 4.5 Effects of External Calculator
text: Driven by the observation that small-scale LMs frequently make basic calculation errors, we develop a simple yet efficient method that integrates an external calculator into the models’ decoding
process. To evaluate the impact of this integration, we conduct an ablation study by omitting the calculator and present the findings in Figure 7. Our results indicate that decoding without the calculator markedly reduces accuracy across all iterations. We believe that this is because models will generate large amount of false positive pseudo-labels without calculator, that is, the generated pseudo-labels may have correct final answers but make errors in the intermediate reasoning steps.

heading: 5 Related Work
text: Learning from pseudo-labels Supervised finetuning (SFT) is prevalent technique employed to enhance the performance of pre-trained language models on specific downstream tasks (Ouyang et al., 2022; Chung et al., 2024). However, this method heavily depends on the availability of highquality labeled data, which can be both expensive and labor-intensive to procure (Brown et al., 2020). To address this limitation, various strategies have been developed to generate high-quality pseudolabels using either unlabeled or synthetic data for a wide range of applications, including text classification (Xie et al., 2020), sentence representation learning (Wang and Lu, 2022), instruction tuning (Honovich et al., 2022), and math reasoning (Wang and Lu, 2023). Recent advancements in this area primarily focus on two directions: selftraining and knowledge distillation. The key difference between these methods lies in the source of the pseudo-labels: self-training uses the model’s own predictions on unlabeled data, while knowledge distillation utilizes the insights from larger, more powerful models.
Self-training in language model Recently, we have witnessed a large number of works focusing on self-training algorithms for language models (He et al., 2019; Zelikman et al., 2022; Yuan et al., 2023). Most of such methods are built upon the classic self-training framework (Scudder, 1965). He et al. (2019) empirically studied the effectiveness of self-training in natural language generation tasks, e.g., summarization and translation. Zelikman et al. (2022) proposed selftaught reasoner (STaR), which demonstrated that language models can be iteratively improved from its own generation, even there are no gold rationales provided. Yuan et al. (2023) proposed rejection sampling fine-tuning to improve language models’ math reasoning abilities. This method can be interpreted as only executing one iteration of the self-training algorithm. Singh et al. (2023) proposed ReSTEM , a self-improving algorithm based on expectation-maximization framework. This method demonstrates significant improvements in problem-solving tasks, e.g., math reasoning and code generation.
Knowledge distillation from LLMs Many of the recent research efforts demonstrated large language models (LLMs) are capable of doing math reasoning (Wei et al., 2022b; Gao et al., 2022; OpenAI, 2023; Anil et al., 2023; Anthropic, 2023, 2024). Therefore, a recent line of work focuses on improving smaller language models’ reasoning abilities by distilling chain-of-thought pseudo-labels from LLMs (Ho et al., 2023; Magister et al., 2023; Fu et al., 2023). For example, Luo et al. (2023) proposed Reinforcement Learning from Evol-Instruct Feedback built upon EvolInstruct (Xu et al., 2023), which requires ChatGPT to provide the training signals. An et al. (2023) demonstrated that language models can effectively learn from the mistakes that can be corrected by larger models, e.g., GPT-4 (OpenAI, 2023), during supervised fine-tuning. Yu et al. (2024) proposed a novel question bootstrapping method with the help of larger models to augment the existing training dataset. Although these methods are shown to have promising experimental results, they are costly to implement as large models cost more FLOPs during inference. Our work demonstrates that smallscale language models can also learn from their own generations like the larger ones (Zelikman et al., 2022), which is more resource-efficient compared with the knowledge distillation methods.

heading: 6 Conclusion
text: We present an effective and resource-efficient method called DPO-augmented Self-Training (DPO-ST), which augments the original SelfTraining algorithm with Direct Preference Optimization (Rafailov et al., 2023). Unlike previous studies that improve small-scale language models’ reasoning abilities by distilling a larger and more powerful model, we argue that small models that are trained merely on the limited human-labeled data can improve themselves significantly. We also empirically find that models trained with DPO loss can generate pseudo-labeled data with higher quality and diversity. Our experiments demonstrate that the proposed method not only outperforms existing methods with comparable model sizes on the GSM8K benchmark, but also achieves remarkable resource efficiency in terms of both computational cost and the requirements of human-labeled data.

heading: Limitations
text: Use of unlabeled data Our method is built upon the classic self-training algorithm, which provides an effective semi-supervised learning framework that makes good use of unlabeled data. However, this work doesn’t explore the use of unlabeled data explicitly. Future research efforts can be made to explore how to collect high-quality unlabeled data for math word problem solving. In other words, we need to find an efficient method for collecting unlabeled data U = {(xi, ai)}ui=1 that for each math question xi, there is a corresponding groundtruth answer ai.
Generalization to other tasks One of the limitations of this work is the narrow scope of our experiments, which were exclusively conducted on math reasoning tasks. The primary reason for this limitation is the lack of appropriate training data for other reasoning tasks. As our method requires a set of training data with chain-of-thought labels, many existing reasoning tasks lack such annotations, making it challenging to extend our experiments beyond the current scope. Future research may focus on identifying and developing suitable datasets for a wider range of reasoning tasks to fully evaluate the applicability and effectiveness of our method across different reasoning tasks.

heading: Acknowledgements
text: This work was done when Shichen Li was a visiting student at the StatNLP Research Group of SUTD. We would like to thank the anonymous reviewers, our meta-reviewer, and senior area chairs for their constructive comments and support on this work. This research/project is supported by Ministry of Education, Singapore, under its Academic Research Fund (AcRF) Tier 2 Programme (MOE AcRF Tier 2 Award No: MOET2EP201220011), the National Research Foundation Singapore and DSO National Laboratories under the AI Singapore Program (AISG Award No: AISG2RP-2020-016), and Ministry of Education, Singapore, under its Tier 3 Programme (The Award No.: MOET320200004). Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of the funding agencies.

heading: Future Work
text: Future research efforts can be made to explore how to collect high-quality unlabeled data for math word problem solving. In other words, we need to find an efficient method for collecting unlabeled data U = {(xi, ai)}ui=1 that for each math question xi, there is a corresponding groundtruth answer ai. Generalization to other tasks One of the limitations of this work is the narrow scope of our experiments, which were exclusively conducted on math reasoning tasks. The primary reason for this limitation is the lack of appropriate training data for other reasoning tasks. As our method requires a set of training data with chain-of-thought labels, many existing reasoning tasks lack such annotations, making it challenging to extend our experiments beyond the current scope. Future research may focus on identifying and developing suitable datasets for a wider range of reasoning tasks to fully evaluate the applicability and effectiveness of our method across different reasoning tasks