DialCoT Meets PPO: Decomposing and Exploring Reasoning Paths in Smaller Language Models
Authors: Chengcheng Han
Xiaowei Du
Che Zhang
Yixin Lian
Xiang Li
Ming Gao
Baoyuan Wang
Venue: EMNLP 2023 Main (2023)
Keywords: Chain-of-Thought
PPO
Reasoning

Abstract: Chain-of-Thought (CoT) prompting has successfully enhanced the reasoning capabilities of Large Language Models~(LLMs) with at least 100 billion parameters. However, it is ineffective, or even detrimental, to the performance on reasoning tasks in Smaller Language Models (SLMs) with less than 10 billion parameters. In this paper, we propose Dialogue-guided Chain-of-Thought (DialCoT) to improve the reasoning capabilities of SLMs, with the aim of generating intermediate reasoning steps in a dialogue format to guide the model to the final answer. Furthermore, we optimize the model to choose the optimal reasoning path through the Proximal Policy Optimization (PPO) algorithm, further enhancing its reasoning capabilities. Compared to previous methods, our advantages lie in: 1) We transform the process of solving complex reasoning problems into decomposing problems and solving a series of simpler sub-questions, significantly reducing task difficulty and making it more suitable for SLMs. 2) We optimize the model to choose the optimal reasoning path through the PPO algorithm. Comprehensive experiments on four arithmetic reasoning datasets show that our method can achieve significant performance gains over state-of-the-art competitors.

Di Meets PPO: Decomposing and Exploring Reasoning Paths in Smaller Language Models
Chengcheng Han .
Xiaowei Du Ming G
Che Zhang
Yixin Lian
Xiang Li
2 + Baoyuan Wang*
$ School of Data Science and Engineering, East China Normal University Xiaobing.AAl
School of Software & Microelectronics, Peking University
KLATAADDS-MOE in School of Statistics, East China Normal University chengchenghanestu. ecnu. edu. cn
{duxiaowei, wiw.:iiiiniiiiiiiiiiiiiii wangbaoyuanj mmtēstu. pku. cn {xiangli, Adagaaaeeddeeeeeeeeem .ww..edu.ccrrrrrrr.mmcccmm cn
ai
Abstract
There are 25.0 students trying out for the school 's trivia teams. If 15.0 them did not get picked for the team and the rest were put into 2.0 groups. How many students would be each group
Mhain-of-Thouglle (CoT) prompting has proven to be effective in enhancing the reasoning ca- pabilities of Large Language Models (LLMs) with at least 100 billion parameters. However, it is ineffective or even detrimental when ap- plied reasoning tasks in Smaller Language Models (SL with less than 10 billion param- eters. To address this limitation, we introduce Dialogue-guid -uideed Chain-of-Thought (DialCoT) which employs a dialogue format to generate in- termediate reasoning steps, guiding the model toward the final answer. Additionally, we opti- mize the model's reasoning path selection using the Proximal Policy Optimization (PPO) algo- rithm. further enhancing its reasoning capabil- ities. Our method offers several advantages compared to previous approaches. Firstly, we transform the process of solving complex rea- soning questions by breaking them down into a series of simpler sub-questions, significantly reducing the task difficulty and making more suitable for SLMs. Secondly, optimize the model's reasoning path selection through the PPO algorithm. We conduct comprehensive ex- periments on four arithmetic reasoning datasets, demonstrating that our method achieves signifi- cant performance improvements compared to state-of-the aht competitors.
Chaun-ot-Thought
The 25,0 - 15.0 10.0 students who picked for the trivia teams were put into 2.0 groups. There would be 5.0 students each group. So the answer 5.0.
Dialoque-guided Chain-of- Chain-of-Thought
How many students were picked for the trivia teams?
25.0-154 15,0 - 10.0 students were picked for the trivis teams.
many students would each group?
We know that the 10.0 students who were pleked for the trivia tenms were put into 2.0 groups. So there would he students in each group. So answer is 5.0.
Figure I: A comparison of DialCoT and CoT prompt- ing (Wei et al., 2022). CoT prompting guides the model to output all intermediate reasoning steps at once ob- tain the final answer, while DialCoT leads the model to gradually generate intermediate reasoning steps in a dialogic format to ultimately arrive at the answer.
significant improvement. However, preliminary results of Wei et al. (2022) have demonstrated that CoT prompting only shows significant perfor- mance gains on LLMS (2 100B), such as LaMDA- 137B (Thoppilan et al., 2022), GPT-3 175B (Brown et al., 2020) and PaI.M-540B (Chowdhery et al., 2022). But it is ineffective, even detrimental, to the performance on reasoning tasks in Smaller Language Models (5 10B). This phenomenon is explained by Wei et al. (2022), who attribute it to abilities such as semantic comprehension and symbolic mapping, which only manifest at larger
1 Introduction
With the advent of Chain-of-Thought (CoT) prompting (Wei et al., 2022), which encourages Large Language Models (LLMss) to generate a series of intermediate steps to help get the final answer, reasoning capabilities of LLMs has seen
"Work done during an internship Xia.
Cooreeponndig author.
!Our code and dataset are publicly available at https: //github com/hccngu/DialCo..

---

scales, The massive computational requirements and inference costs of LLMs make them unfeasible for widespread deployment. There a pressing community interest in figuring out how to further enhance the reasoning capabilities within Smaller Language Models (SLMs).
2.
By breaking down intermediate reasoning steps into dialogue-formattee sub-questions and answers, we can use reinforcement learn- ing more effectively to choose the optimal reasoning path from various options.
Recent works (Magister al., 2022; Ho al., 2022; Fu al., 2023) have attempted to enhance the performance of SLMs on reasoning tasks by fine-tuning them with training data generated from LLMs that contain intermediate reasoning steps. However, the results have been less than opti- mal. To further boost the reasoning capabilities of SLMs, we propose Dialogue-guided Chain-of- Thought (DialCoT),, which aims to progressively generate intermediate reasoning steps in the dia- logue format, instead of generating all interme- diate reasoning steps at once (as shown in Fig- ure 1). Specifically, we assign to the model two roles: Decomposer and Solver. The Decomposer is tasked with breaking down the original question into a series of sub-questions. The Solver sequen- tially addresses each sub-question presented by the Decomposer, thereby obtaining the answer to the original question. They utilize different instruc- tions while sharing the same model parameters. We propose three different forms of DialCoT: 1) DialCoT-A (All at once), in which the Decomposer generates all sub-questions at once and the Solver simultancously provides all answers. 2) DialCoT- M (Mixed), where the Decomposer generates all sub-questions at once but the Solver sequentially delivers the answers of the sub-questions gener- ated by the Decomposer. 3) DialCoT-S (Step by step), where both the Decomposer and Solver op- crate sequentially to generate sub-questions and their corresponding answers. We provide de- tailed comparison of the performance of the three different forms of DialCoT in Section 4.4. Fur- thermore, building upon DialCoT-S, we design DialCoT-S-PPO, which leverages the Proximal Policy Optimization algorithm select the opti- mal reasoning path, thereby further enhancing its performance in reasoning tasks. Compared to pre- vious methods (Ho et al., 2022; Fu et al., 2023), our approach has two main advantages:
To validate the effectiveness of our approach, we fine-tune Flan-T5 (Chung et al.. 2022) using 7000 training examples from the GSM8K (Cobbe et al., 2021) dataset that include intermediate questions and answers. The results surpass the latest method, SpecialFT (Fu et al., 2023), by 6.2%. Notably, the amount of training data we use is only 1/20 of that used by SpecialFTT. In addition, to ver- ify the model's generalization capability on out- of-distribution tasks, also test the Multi- Arith (Roy and Roth, 2015), ASDiv (Miao et al., 2020) and SVAMP (Patel et al., 2021) datasets. Our method achieves state-of-the-an performance compared to other baselines.
Related Work
2.1 Chain-of-Thought Prompting
Chain-of- Ohooughf-Thought (CoT), which significantly en- hances the reasoning capacities of large language models, originally pioneered by Wei al. (2022). The approach focuses augmenting few-shot examples with detailed reasoning steps, therehy markedly improving performance on rea- soning tasks. Subsequent works, inspired by Wei et al. (2022), have further refined the CoT method- ology, such as Self-Consistency (Wang et al., 2022), Least-to-Most prompting (Zhou et al., 2022b), Dy- namic Least-to-Most prompting (Drozdov et al., 2022), Self- Self-nrainin (Huang et al., 2022), Veri- fier (Li et al., 2022) and Tree of Thought (Yao et al., 2023). The aforementioned methods pri- marily focus on improving the specific format of CoT prompting to better stimulate the reasoning capabilities of LLMs (2100B). However, they are not tailored to augment the reasoning capabilities of SLMs (S 10B). We propose a novel method specifically designed to enhance the performance of SLMs on reasoning tasks.
2.2 Reasoning Enhancement in SLMs
1. We transform the process of solving a com- plex reasoning question into decomposing the question and solving series of simpler sub- questions, which reduces the task difficulty and more suitable for SLMs.
Chung et al. (2022) observes that training SLMs with data that include intermediate reasoning steps can improve the reasoning capabilities of SL.Ms. Both Magister et al. (2022) and Ho et al. (2022) enhance the reasoning capabilities of SLMs by

---

fine-tuning them with training data, which includes intermediate reasoning sleps generated by LLMs. STaR (Zelikman et al., 2022) enables the model to self-improve through its own generated rationales. SpecialFT (Fu et al., 2023) employs LLMs as teacher models and utilizes distribution matching in knowledge distillation to transfer the reasoning capabilities from LLMs to SLMs. Orca (Mukher- jee al., 2023) learns imitate the reasoning process of LLMs from rich signals generated by LLMs, including explanation traces, step-by-step thought processes and other complex instructions. Differently, DialCoT transforms solving complex reasoning questions into decomposing questions and addressing a series of simpler questions, sig- nificantly reducing the task difficulty. Furthermore, incorporate the PPO algorithm to enable the model choose the optimal reasoning path among multiple options, thereby further enhancing the per- formance in reasoning tasks. Notably, our method does not require generating a large amount of train- ing data with intermediate reasoning steps through LLMs. For example, by simply fine-tuning with only 7,000 examples from the GSM8K dataset, we could achieve a remarkable enhancement in SLM performance reasoning tasks.
decomposition and resolution. Compared the aforementioned methods, we not only decompose the question but also enable the model to choose the optimal reasoning path through reinforcement learning methods, thereby further enhancing the model's capability to solve complex questions.
Dialogue-guided Chain-of-Thought
We propose Stalogue-guided Chain-of-Thoughee (DialCoTT), which aims to decompose complex questions into sub-questions in a dialogue format and gradually guide the model to obtain the fi- nal answer. Specifically, for the model, namely the Decomposer and the Solver, who engage in a dialogue-based interaction. The Decomposer is responsible for breaking down the original question into series of simpler sub- questions, while the Solver sequentially answers these sub-questions. We design distinct instruc- tions for the Decomposer and Solver, followed by performing instruction tuning (Wei et al., 2021) SLMs. We first introduce three different forms of DiaICoT, Subsequently, we describe how we in- corporate the Proximal Policy Optimization (PPO) algorithm into DialCoT to enable the model to se- lect the optimal reasoning path and further enhance its reasoning capabilities.
introduce two roles
2.3 Question Decomposition
3.1 Three Forms of DialCoT
Question decomposition is crucial for understand- ing and solving complex questions. Earlier re- search (Kalyanpur et al., 2012) uses decomposition rules based on lexico-syntactic features to facilitate question decomposition. HSP (Zhang et al., 2019) proposes a hierarchical semantic parsing method based on sequence-to-sequens model, which combines question decomposer and an informa- tion extractor. Patel et al. (2022) designs human- in-the-loop question decomposition method to im- prove model performance. Least-to-Most prompt- ing (Zhou et al., 2022b) improves the format of COT, enhancing the reasoning capabilities of LLMs by decomposing problems. Self-Ask (Press et al., 2022) explicitly asks itself follow-up questions before answering the initial question to perform compositional reasoning tasks. DecomT5 (Zhou et al., 2022a) develops robust decomposition-basee models using distant supervision from comparable texts. Decomposition Distillation (Shridhar al., 2022) learns a semantic decomposition of the orig- inal question into a sequence of sub-questions and uses to train two models designated for question
We propose three different dialogue forms of DialCoT, namely DialCoT-A, DialCoT-M, and DialCoT-S. Specifically, DialCoT-A aims guide SLMs in reasoning through minimal dialogue turns. DialCoT-M refines the Solver based on DialCoT-A, further reducing the task complexity. DialCoT- maximally decomposes intermediate reason- ing steps, allowing it to reference previous sub- questions and their answers when proposing new sub-questions. Figure 2 and Figure 3(1) presents the overall frameworks of them. Next, will introduce each of these forms individually.
DialCoT-A (All once). We first establish in- struction for the Decomposer, enabling it to gener- ate all sub-questions in a single step. Subsequently. we incorporate the generated sub-questions to the original texts and design a new instruction for the Solver, which allows the Solver to answer all sub- questions simultaneously. Figure 2(1) displays
"The specific prompt structures can be found in Table 3 of Appendix A.

---

DisiCoF-A (Ali oncs)
Content: There 25.0 students irvine out for school's in If 15.0 of them did not oct dicktd the seam and the cest mere out Inco 2.0 kouna How mane discents would eacht scsun.
Sub-question How many suaeeu were picked for the aridaa teamss
Pial question: How waity studeuts would group?
beeak
question mulriple sub-qucctions
Decompoer
Content: There are 35 mudents rrine for the scoool urivia seams, If oct nicked for seam
cost oun into
group, How many stadents would each gooup?
25.0- 15.0 10.0 students were picked for the tzivia teams.
Saloquactionn How many stueess were pickest for the ariida teasss? Plaal question: How mary studosts would Please answer the above sub-questions order and provide the final apower.
The 1000 students who were picked for the civia seame groups So would students group. the answer S.O.
group?
Selver
Instrection
Original tetts Decompooser Raspoese Solver Response
DialCoT-M (Nixed)
Content: There are 25 stadents ircine out for school's salvia stamn... 15.0 of chom did not eet stcked for the team and the cost were zut into
Sau question 1: How many mudents were picked for the trinn teams?
rouure Hose stalents would each group?
Final question: mory snudents each group?
Please break down above questio into multiple sub-qucation.
Decomposer
Content: Theme are 25.0 stadens tryine out for school's Ieamsa 15.0 of chem did not ect dicked for the seam and the und ware out Inso potunn How many modents were picked for the trinn leoms?
25.0- students were pickod the trivis toins,
Sulver
Content: Thera aee 25,0 stéents tryine out for the school's 's nivis teams. If of did not dicked for the lcam and the and gut into gouss,
The 10.0 studenrs who were picked the temms were
1100 students picked for trivia How Rong modents
2. groups So there would be stucents in each group.
each group?
Solver
Figure 2: (I) D-A, where the Decomposer generates all sub-questions at once, and the Solver responds to all the sub-questions in a single step. (2) D-alCoT-M, where the Decomposer is the same in DialCoT-A, while the Solver addresses sub-question at a single step, with the response being incorporated into the original texts to aid in solving subsequent sub-questions.
the instructions we designed and examples of in- put/output when the model operates Decom- poser and Solver. DialCoT-A shares similar mo- tivation with Orca (Mukherjee al.. 2023), both striving to improve model reasoning performance by providing explicit reasoning paths. Orca repre- sents the reasoning path through problem-solving steps, whereas our method exhibits the reasoning path via sequence of sub-questions.
"Solver:" ". The history of the dialogue is appended after the original texts, aiding the model in answer- ing subsequent questions and deriving the final an- swer. Figure 3(1) displays the overall framework of DialCoT-S. Compared to the previous two forms of DiaICoT, DialCoT-S can reference previous sub- questions and their answers when generating new sub-questions. Moreover, DialCoT-S is more sim- ilar to the traditional multi-turn dialogue format. Therefore, it can more effectively stimulate the model's 'ss multi-turn dialogue capability to improve the model performance on reasoning tasks.
DiaMCoT-M (Mixed). Upon deriving series of sub-questions via the same Decomposer used in DialCoT-A, sequentially replace the final ques- tion in the original texts with these sub-questions, which allows the Solver to address each sub- question individually. The Solver's response from each sub-question is appended to the original text, providing contextual support for solving subse- quent questions. Figure 2(2) presents an example of DialCoT-M solving math word problem. Com- pared to DialCoT-A, DiaLCoT-M mitigates the task complexity for the Solver by addressing single and simpler question in each step.
3.2 DialCoT-S-PPO
DialCoT-S-PPO aims to enable the model to select the optimal reasoning path by combining DialCoT- with the PPO algorithm, further improving the model's performance on reasoning tasks. Fig- ure 3(2) presents an example of DialCoT-S-PPO solving a math word problem. DialCoT-S-PPO chooses the optimal intermediate questions or an- swers from the model's 's multiple outputs, thus form- ing reasoning path through series of choices. Specifically, we first need to collect some data com- posed of states S, actions A and rewards R for training the policy network Tg. S represents the space of states of the environment, which are the input of the policy network. Let S, E S be a state
DialCoT-S (Step by Step). We design new in- structions to direct the Decomposer generate only single sub-question at step and the Solver to address the sub-question. Responses are prefixed with role identifiers such as "Decomposer: " and

---

DialCoT-8 (Step
( DiaiCoT-S-PPO
leamt, of and nut mea
Incse zue Do sclool livia
Content: Those stndeans
Irying
sسuلن dil picual for bu Jaam nan
Content:
Indruction
Decorposey Response
OTO
mudel
Rxinod lx 1sm.
Solver Roeporne
Ro0U0 - <Dialogue History?
in.aaa
Raaed the dialegue
"SLoioque His Locy>
Decompaser Sub-quezzion E View mary statents are ryying our for the wchoolss triss tvams?
X
generate a reply.
chis quesilon,
Tist de know
Decemptoer:: Safoguection Viow mudents pioked the trivis toums?
Decompoerr:
How
Decemposer: Sub question J. Flor mony sudden did get picked for the ream?"
Gor the critt troms?
X
Solver: 25.0 15.0 10.0
Solver:
10.0 stadent picked the givia teamns
soudents wore picktd for the Iivi teams
Decomsposer
Soh er
Decomposer
Decomposer: Prinal gaestion:: start mualont group?
Selver
Vlow many mudents would br bach gruig?
Decsmposer: Final guestion:
Solver:
know
Sulver: We know that the inD
students who were picked for the irvia were put
pivia
2.0
grnupe So there would be S.O Ausiamte in each griup, Su the guswer is 3.0.
ptudents So
anower 5.O.
Figure 3: (I) DialCoT-S, where the Decomposer presents a sub-question at step and Solver answers it. Their past dialogue information is inserted into <Dialogue History> assist in generating the answer of the final question. (2) An example of DialCoT-S-PPR solving a math word problem. The policy network is used to select response from each step of the SLMM, ultimately forming an optimal reasoning path and arriving the final answer.
at time t defined as
0. We update the policy network through the following objective function:
St [h h2 ...; hg],
(1)
h1, h2 hi] (X),,
(2)
To (at 88))
(()) E. min
As:
THali (au,
where X denotes the input text constructed via DiaICoT-S, LM3(-) represents the SLM after in- struction tuning and h denotes the last hidden state of the model's response. We utilize beam search to generate the top-k responses with the highest probability as candidates and St is the con- catenation of these candidates' last hidden states. A [0, 1, .... k|] represents the action space. At each time t, input s; into me to obtain the prob- ability p of actions:
clip
Woold (ar [8c) :1.11116. (o))
To (at st)
(4)
where mo(+) represents the training policy network and aam denotes the policy network that interacted with the enviromment collect data. Further infor- mation regarding the clip(-) and A, can be found in Schulman et al. (2017). After update the param- eters of тy. the new parameter of is transmitted to Toas- We then repeat data collection and To updates until training completion.
(s:).
(3)
Based on the probability p. πy chooses an action a, E A, which represents choosing the ay-th can- didate. During the exploration phase, my obtains through sampling. In the inference phase, chooses the a, with the highest probability. When the model correctly answers sub-question, it re- ceives a reward rm E [0, 1). T' is a hyperparam- eter that represents the extent which the model focuses on the intermediate steps. When the model correctly answers the final question, it receives a reward ry = 1. all other cases, the reward is
4
Experiments
4.1 Datasets
We consider the following four math word problem datasets: GSM8K (Cobbe et al., 2021), Multi- Arith (Roy and Roth, 2015), ASDiv (Miao et al., 2020) and SVAMP (Patel al., 2021). The GSM8K dataset contains 7,000 training instances with intermediate questions and answers. In con- trast to previous work (Magister et al., 2022; Ho et al., 2022: Fu et al., 2023), we only fine-tune

---

our model using these 7,000 instances, eliminat- ing the need for generating additional training data with intermediate reasoning steps via LLMs. Apart from evaluating our method on the GSM8K test set, we evaluate the model's out-of-distribution per- formance on three other datasets. All datascts com- prise arithmetic reasoning problems at a primary school level, varying by the entities they incorpo- rate. This form of out-of-distiibution generaliza- tion is typically classified as lexical-leved composi- tional generalization (Liu et al., 2021). Following SpecialFT (Fu et al., 2023), for each dataset, we employ 500 instances as the validation set, using the remaining instances (800 for GSM8K, 400 for MultiArith, 18K for ASDiv, 500 for SVAMP) as the test set.
Hyperparameters
Scope
learning rate
[1e 4, J,,e-4, 5c
le 31
batch size
[1024, 2048, 4096)
[0.1, 0.2, 0.3)
(2.3, 4, 6]
I'm
10.1, 0.3, 0.4,
Table I: The searching scope for the hyperparameters of the proximal policy optimization algorithm. We high- light the best settings in bold.
Following Chung et al. (2022), we fine-tune the model for 50 epochs with the batch size 1096 and the learning rate 5e - 4. For the PPO algorithm, use three feed-forward layers as the policy net- work and set the number of hidden units to 1024. Moreover, use the grid search to find the best hyperparameters. The details of grid search are shown in Table 1. As a result, we set the learning rate as 3e - 4 and batch size as 4096 for the policy network. We also set to 0.2, &s to 3 and r'm to 0.3. During the stage of optimizing the policy network, we freeze the backbone parameters. All baseline re- sults except CoT-FT (Wei et al., 2021) are recorded in SpecialFT (Fu al., 2023). For CoT-FT and our method, we keep the experimental setup consistent with other baselines. We run all the experiments on eight NVIDIA Tesla A100 GPU.
4.2 Baselines
In our experiments, we compare method with some competitive baselines which can be grouped into two categories: (I) generic large language models: code-davinci-002 (Chen et al., 2021) pre- sumably with size of 175B or more, LaMDA- 137B (Thoppilan et al., 2022), PaLM-60B (Chowd- hery et al., 2022) and UL2-20B (Tay et al., 2022), each of which exhibits strong reasoning abilities in Chain-of-Thought prompting. (2) concurrent works enhancing SLMs' reasoning capabilities: CoT-FT (Wei al., 2021) directly employs the 7000 CoT training instances from the GSM8K dataset to perform instruction tuning, which is a vanilla approach to enhancing the reasoning capa- bilities of SLMs. DecomDistill (Shridhar et al.. 2022) is eccomposition-baseee method, which learns a semantic decomposition of the original problem into sequence of sub- uub-problems through LLMs. Both Magister et al. (2022) and Ho al. (2022) fine-tune SLMs by generating training data with intermediate reasoning steps through LLMs. SpecialFT (Fu et al., 2023) employs LLMs as teacher models and utilizes distribution matching in knowledge distillation to transfer the reasoning capabilities from LLMs to SLMs. It is noted that SpecialFT uses 130K training instances with inter- mediate reasoning steps generated by LLMs, which is nearly twenty times the size of our training set.
4.4 Results
Table 2 shows the performance of various methods on four arithmetic reasoning datasets. First, we discuss the results for DialCoT-A, DialCoT-M, and DialCoT-S. Then, we compare our method with other baselines to demonstrate its superiority. Fi- nally, we validate the effectiveness of incorporating the PPO algorithm based on DialCoT-S through an ablation study.
Discussion of Three DialCoT Variants. Com- pared to CoT-FT, all three forms of DialCoT out- perform on the four reasoning tasks, demonstrating their effectiveness in enhancing the reasoning ca- pacities of SLMs. More specifically, DialCoT-M performs better than DialCoT-A. This indicates that SLMs lack the capability to decompose a reason- ing problem and answer it all once. DialCoT- M addresses only one sub-question step. which reduces the task difficulty and makes more suitable for SLMs. DialCoT-S, in com- parison to Dia,CoT-M, shows greater performance gains, which can be attributed to two factors: (1)
4.3 Implementation
single
We consider using FlanTS-XL (3B)XXL (1IB) as the backbone of our model. The Decomposer and Solver utilize different instructions (as shown in Figure 2) but share the same model parameters.

---

Methods
Backbone
Params..
GSM8K
MultiArith
ASDiv
SVAMP
Average
Generic Large Language Models
code-davinci-002 (Chen et al., 2021)
175B
63.1
95.8
80.4
76.4
78.9
LaMDA (Thoppilan et al., 2022)
137B
14.8
45.0
46.6
37.5
36.0
PaLM (Chowdhery et al., 2022)
60B
29.9
75.0
61.9
46.7
53.2
UL2 (Tay al., 2022)
20B
4.4
16.9
12.5
Concurrent Works to Boosting SLM Reasoning
DecomDistill' (Shridhar al., 2022)
GPT
7B
21.0
Magister (2022)1
TS-XXL
42.1
Ho et al. (2022)
GPT
6B
6.8
33.3
Cof-FT (Wei at., 2021)
FlanTS-XL
3B
13.5
24.0
20.7
17.7
19.0
SpecialFT' (Fu al., 2023)
FlanTSSXL
3B
22.4
42.3
28.4
23.8
29.3
DialCoT-A (All at once)
FlanTS-XL
3B
20.3
40.3
24.6
21.3
26.6
DialCoT-M (Mixed)
FlanTS-XL.
22.9
27.1
23.2
29.1
DialCoT-S (Step by Step)
FlanT5-XI.
3R
24.3
45.7
29.3
25.5
31.2
BalCoT-S-PPO
FlanT5-XL
3B
25.6
46.9
30.7
27.1
32.6
CTT-FT (Wei 2021)
TX-XXXX
11B
51.7
36.5
39.7
36.0
SpecialFTi (Fu et al., 2023)
Flan
11B
27.1
63.0
37.6
35.6
40.8
DialCoT-A (All once)
TS-nT--XXL
11B
21.7
57.1
32.5
34.2
36.4
DialCoT-M (Mixed)
F-XTTS-XXL
38.2
37.7
DialCoT-S (Step by Step)
11B
35.2
65.7
39.3
40.3
45.1
DisPCoT-S-PPO
FlanT5-XXI.
11B
37.1
68.1
40.9
41.7
47.0
Table 2: Accuracy (%) of various methods on four reasoning tasks. indicates that the method employs additional training data with intermediate reasoning steps generated via L.MSs, where SpecialFT uses nearly 20 times the training data of our method. We highlight the best results SLMs (~ 10B) in bold.
DialCoT-S obtains more intermediate information before generating sub-questions. (2) DialCoT-S more effectively stimulate the model's multi-turn dialogue capabilities to boost its reasoning perfor- mance.
capabilities via appropriate fine-tuning methods." 4
The ablation study is conducted demonstrate the effectiveness of incorporating the PPO algorithm based on DialCoT-S. Compared to DialCoT-S with - DialCoT-S-PPO achieves an improvement of nearly 2%, confirming the effectiveness of employing the PPO algorithm for selecting the optimal reasoning path. Addition- ally, we observe that when using FlanT5-XL. as the backbone, the performance gain brought by the PPO algorithm 1.4%, which is lower than the performance on T5-nT-XXXL This might be at- tributed to the lower diversity in the multiple replics generated by the smaller model.
Ablations.
Comparison between DialCoT and Baselines. From the Table 2, we observe that DialCoT-S-PPO attains statc-of-the-art results on SLMs. Specifi- cally, when using X- as the backbone, DiaICoT-S-PPO improves the average performance across the four datasets by 6.2% compared to Spe- cialFT. Notably, the training data we used is only 1/20 of what SpecialFT used, which clearly demon- strates that our method is very effective in improv- ing reasoning capabilities of SLMs. On the other hand, when compared with LLMs, all variations of DialCoT (i.e., DiaICOT-A/MISIPPO) outperform LaMDA-137B on average across the four datasets, despite the parameters of our approach are merely 1/12 of LaMDA's. This further substantiates the su- periority of our approach. While there is still a no- ticeable gap when compared code-davinci-O00 our experimental results demonstrate that there is potential for SLMs achieve LLM-level reasoning
4.5 Analysis
Different Model Size. We extend method to smaller backbones. including FlanT5-Base (250M) and Flange-Large (760M), on the GSM8K and MultiArith datasets. Our experimental results are illustrated Figure 4. In comparison to the original FlanT5, our approach improves the performance the model on reasoning tasks across different model sizes, affirming the effectiveness of our method for varying model sizes. Notably, we observe that 'A more detailed discussion of the three DialCoT variants can he found in Appendix D.
"The detailed experimental comparison between DialCoT and SclfAsk can be found in Appendix A.

---

CSM8K
28
FlanTS
DialCoT-A
Accuracy (%)
DialCoT-M
DiaCCuT-S
Accuraes (%6
24
20
DiM
22
20
18-
Base (550)
Larger 760M)
(I(3B)
XXL(IIB)
16
GSM8K
CoT-FT
DRIKOTSSDINCOTS-PPS
MultiArith
HlanTSS
80
DiaICoT-A
Accuracy (%)
DialCoT-M
75
40
DiaICoT-S
DiSIOOT-S-PPO
Accurecy (%)
70
65
60
10
55
Basc(250M)
(6rge(760M) XL(3B)
XXL(IIB)
MultiArith
50
COT-FT DialCoT-A
DialCoT-M
DiaKoTSDDalCoTsSPPS
Figure 4: Results under different model sizes on the GSM8K and MultiArith datasets. FlanT5 indicates the results of using Few-shot CoT (Wei et al., 2022) the backbone (Chung et al., 2022). Our methods achieve performance improvements across all model sizes.
Figure 5: Results using LLaMA-7B (Touvron et al., 2023) as the backbone the GSM8K and MultiArith datasets. Our method achieves significant performance gains on decoder-only LMs.
40
method yields larger performance gains larger model sizes, which is similar the results of Chung al. (2022). This could be attributed to the stronger capabilities that larger models ob- tain during pre-training, making them more readily stimulated.
35
Accuracy (%)
FlanTS-XL
30
FlanTS-XXL
25
Different Model Architectures. To evaluate the generalizability of DialCoT across LMs with varying architectures, in addition to the encoder- decoder LM (e.g., FlanT5), we conduct experi- ments using the decoder-only LM (e.g.. LLaMA- 7B (Touvron et al., 2023)) the backbone of our method on the GSM8K and MultiArith datasets. The results are illustrated in Figure 5. As can be seen from the figure, all of our methods achieve significant performance gains compared to CoT-FT, especially DialCoT-S and DialCoT-S-PPa This demonstrates that our approach is applicable to SLMs with various architectures, not merely effec- tive on encoder-decoder LMs. Moreover, we ob- serve that DialCoT-A performs better than DialCoT- M on the MultiArith dataset, which different from the results based Flan'55 (as shown in Fig- ure 4). This suggests that the most suitable form of DialCoT may differ for different SLMs, which could potentially be related to model architecture and pre-training corpora.
20
k-2
k=3
k- GSM8K
k=
k=6
70
65
Accursey (%)
60
FlanTS-XL
55
FlanT5-XXI
50
45
40
k=2
k=
MultiArith
Figure The effect of hyperparameter the model performance on the GSM8K and MultiArith datasets.
Effeet of Hyperparameter k. We set the model output the top-k responses with the highest probability in each step of dialogue through beam search. In other words, represents the size of the action space, indicating that we can select the op- timal response from different responses each

---

step. Figure 6 illustrates the effect of on the model performance on the GSM8K and MultiArith datasets. Specifically, the model can achieve op- timal performance when & is set to 3 or 4. When k is too small, the number of reasoning paths we can choose from is too limited, preventing us from achieving optimal performance. Conversely, when k is too large, the space of available reasoning paths is too large, which may introduce noise and make it difficult for the model to learn select the optimal reasoning path.
capabilittiss of SLMs. Due to constraints on compu- tational resources, we do conduct experiments on larger scale language models (2 20B), thus the applicability of our method for LLMs remains undetermined. We will further explore the perfor- mance of DialCoT on larger scale language models in future research.
Ethics Statement
The proposed method has no obvious potential risks. All the scientific artifacts used/created are properly cited/licensed, and the usage is consis- tent with their intended use. Also, open up our codes and eyper-parameters facilitate future reproduction without repeated energy cost.
5 Conclusion
this paper, we explored strategies to boost the reasoning capabilities of SLMs and proposed Dial- CoT, which aims to generate intermediate reason- ing steps in a dialogue format leading to the final answer. Specifically, we designed two roles for the model, namely Decomposer and Solver. The Decomposer is responsible for breaking down ques- tions into multiple questionss while the Solver is tasked to address the sub-questions. They engage in dialogue to arrive at the final answer. We intro- duced three different dialogue formats: DialCoT- A (All at once), DialCoT-M (Mixed) and DialCoT- S (Step by step). Furthermore, we incorporated the PPO algorithm into DialCoT-S to enable the model choose the optimal reasoning path among mul- tiple options, thereby further improving its perfor- mance on reasoning tasks. We conducted extensive experiments on four arithmetic reasoning datasets and the experimental results demonstrate the ef- fectiveness of our method. Future work primarily involves extending our method to other types of rea- soning tasks, such as commonsense reasoning and symbolic reasoning. In addition, will explore other decomposition methods or other reinforce- ment learning methods optimize the reasoning paths of SI.Ms.
Acknowledgements
This work has been supported by the National Natural Science Foundation of China under Grant No.U1911 1203, the National Natural Science Foun- dation of China under Grant No.62377012 and Fun- damental Research Funds for the Central Universi- ties under grant number YBNLTS2023-015.
References
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu. Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In NeurIPS.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374
Limitations
We conduct experiments on four arithmetic reason- ing tasks, demonstrating the effectiveness of Dial- COT. However, as our reward pattern is specifically designed for arithmetic reasoning, modifications are necessary upply method to commonsense symbolic reasoning. This presents limitation the broader applicability of our method. We plan to extend DialCoT to a wider range of reasoning tasks in the future. On the other hand, DialCoT specifically focuses on enhancing the reasoning
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi. Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay. Noam Shazeer. Vin- odkumar Prabhakaran. Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia,

---

Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivami Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz. Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstem., Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways. CoRR, abs/2204.02311.
advance of making language models better reasoners. CoRR, abs/2206.023333
Linqing Liu, Patrick Lewis, Sebastian Riedel, and Pon- tus Stenetorp. 2021. Challenges in generalization in open domain question answering. arXiv preprint arXiv:2109.01155 156.
Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adámek, Eric Malmi, and Aliaksei Severyn. 2022. Teaching small language models to reason. CoRR. abs/2212.08411
Shen- Yun Miao, Chao-Chun Liang, and Keh- Yih Su. 2020. A diverse corpus for evaluating and developing english math word problem solvers. ACI., pages 975-
Hyung Won Chung. Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai. Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu. Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Lc, and Jason Wei. 2022. Scaling cnrructionffnettnnd language models. CoRR, abs/2210. 11121....11...... 11166.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawa- har, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. Orea: Progressive learning from complex explanation traces of gpt-4, arXiv preprint 27Xiv:2306.02707.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? In NAACL, pages 2080-2094.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word prob- lems. CoRR, abs/21 14168.
Pruthvi Patel, Swaroop Mishra, Mihir Parmar, and Chitta Baral. 2022. Is question decomposition unit all we need? In EMNLP, pages 4553-4569.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.00
Andrew Drozdov, Nathanael Schärli, Ekin Akyürek, Nathan Scales, Xinying Song, Xinyun Chen, Olivier Bousquet, and Denny Zhou. 2022. Compositional semantic parsing with large language models. CoRR, abs/2209. 15003.
Subhro Roy and Dan Roth. 2015. Solving general arith- metic word problems. In EMNLP, pages 1743-1752.
Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot. 2023. Specializing smaller lan- guage models towards multi-step reasoning, CoRR, abs/2301.1272
John Schulman, Filip Wolski, Prafulla Dhariwal, Alee Radford, and Oleg Klimov. 2017. Proxi- mal policy optimization algorithms. arXiv preprint arXiv: 06347.
Namgyu Ho, Laura Schmid, and Se- Young Yun. 2022. Large language models are reasoning teachers. CoRR, ab0/2212.100711.
Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2022. Distilling multi-step reasoning ca- pabilities of large language models into smaller mod- els via semantic decompositions. arXiv preprint arXiv:2212.00193
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu. Xuezhi Wang. Hongkun Yu, and Jiawei Han. 2022. Large language models can self-improve. CoRR, abs/2210.11610.
Yi Tay. Mostafa Dehghani. Vinh Q Tran, Xavier Gar- cia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022. Unify- ing language learning paradigms. arXiv preprint 2rXiv22005.05131.
Aditya Kalyampur, Siddharth Patwardhan, Branimir Boguraev, Adam Lally, and Jennifer Chu-Carroll. 2012. Fact-based question decomposition in deepqa. IBM J. Res. Dev. 56(3):13.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li. Hongrae Lec. Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang. Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Yanqi Zhou, Chung-Chiin Chang. Igor Krivokon, Will Rusch, Marc Pickett, Kathleen S.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. Advances in neural information processing systems, 35:22199- 22213.
Yifei Li, Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Tian-Guang Lou, and Weizhu Chen. 2022. On the

---

Meise-Hellstem. Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny So- raker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Ale- jandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Co- hen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera- Arcas. Claire Cui, Marian Croak, Ed H. Chi, and Quoe Le. 2022. Lamda: Language models for dialog applications. CoRR, abs/2201.082233
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint a7X1v.230..1397.
Xuezhi Wang. Jason Wei, Dale Schuurmans. Quoc Le, Ed H. Chi, and Denny Zhou. 2022. Self- consistency improves chain of thought reasoning in language models. CoRR, abs/2203. 11171.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoc V Le. 2021. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXiv:2109.01652.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoe V Le. Denny Zhou, al. 2022. Chain-of-thought prompting elicits rea- soning in large language models. In NeurIPS.
Shunyu Yao. Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao. and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. CoRR, fbs/2305.10601.
Eric 7elikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. 2022. Star: Bootstrapping reasoning with reasoning. In NewrIPS.
Haoyu Zhang, Jingjing Cai, Jianjun Xu, and Ji Wang. 2019. Complex question decomposition for semantic parsing. ACL, pages 4477 4486.
Ben Zhou, Kyle Richardson, Xiaodong Yu, and Dan Roth. 2022a. Learning to decompose: Hypothetical question decomposition based on comparable texts. EMNLP, pages 2223-2235.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans. Olivier Bousquet, Quoe Le, and Ed H. Chi. 2022b. Least-to-mos most prompting enables complex reasoning in large language models. CoRR, abs/2205. 10625.

---

A Comparison between DialCoT and SelfAsk
setting compared Standard COT. This could be either because Flan-T5 was trained with some training data or due to the weaker instruction followiing capabilities of SLMs, where complex instructions increase the task diffi culty.
We add further discussion regarding the compar- ison with SelfAsk. Self-Ask (Press et al., 2022) explicitly asks itself follow-up questions before answering the initial question to perform composi- tional reasoning tasks. Please refer Table 3 for the comparison between SelfAsk and our methods. From the table, designed for in-context learning method without fine-tuning, whereas DialCoT a based method. (2) Even if fine-tuning can applied to SelfAsk theoretically, how to format the fine-tuning is still un-explored. We offer a novel way to lever- age two tailored tasks (problem decomposition and problem solving) through fine-tuning task-oriented instructions on the same model (SLM). Therefore, we believe the performance gain mainly come from the way of decomposing and solving sub-questions through fine-tuned model with tailored instructions. The additional instructions naturally come along with the proposed solution. They will instruct the model to play dedicated role during the instruction fine-tuning. However, we don't otn't think they are the major factors.
For encoder-decoder structure, i.e., T3, the dif- ference between SelfAsk-A and SelfAsk-S is sig- nificant, due to the fact of bidirectional attention within the input. For decoder-only structure, the difference is indeed very subtle. Table shows the results additional experiments (SelfAsk) on the GSM8K using LLaMA-7B. The SelfAsk-A outputs all intermediate questions, answers and connecting words between them such as "Follow up" and "In- termediate answer", while the sequential finctuning is focused on outputting intermediate question answer, and does not include the loss of connecting words, encouraging the model to focus more on most important part of the learning. We speculate this subtlety brings the improvement.
that: (1) SelfAsk is
B Fine-grained Analysis Each Sub-step
We select samples that require decomposition into three sub-questions from all test sets and report the accuracy for each sub-question in Table 7. As shown in the table, as the number of steps increases, i.e., as the complexity of the questions rises, our method shows a notable performance improvement (7%) compared to CoT-FT. On the other hand, al- though our method's performance does decline as question complexity increases, the rate of decline is significantly slower compared to vanilla COT. When comparing DialCoT-S with DialCoT-S-PPO, It is evident that step-level PPO significantly im- proves the model's performance on reasoning tasks at every step.
Moreover, we conduct additional experiments (SelfAsk) on the GSM8K dataset using Flan-T5- XXL. The results are illustrated in Table 4. From the table, we can draw the following conclusions: Firstly, in the fine-tuning setting, SelfAsk im- proves Standard CoT by 5.5% with all- all-at-once finetuning and by 13.2% with sequential finetuning. We believe these improvements stem from prob- lem decomposition. Furthermore, SelfAsk with sequential finetuning improves by 7.7% compared to SelfAsk with all-at-once finetuning. This once again confirms our previous conclusion that decom- posing problems sequentially is more effective than decomposing them all at once. Morcover, DialCoT- S improves by 5.9% over SelfAsk with sequential finetuning. We attribute this additional improve- ment to fine-tuning with different instructions tai- lored for specific tasks. Compared to SelfAsk with sequential finetuning, DialCoT-S has clearer and more independent instructions for both problem decomposition and problem solving.
C Comparison of Inference Speed
In terms of the time cost for reasoning, the infer- ence speed of our method is comparable to that of SelfAsk (Press et al., 2022), as both need to gen- erate sub-questions and answers. In terms of the number of iterations, DialCoT-A requires two iter- ations, which is comparable to Zero-shot CoT (Ko- jima al., 2022), while DialCoT-M and DialCoT-S require approximately three and six iterations re- spectively. Overall, in our methods, DialCoT-A has the fastest inference speed, followed by DialCoT- M, and finally D-aICoT-S. The trade-off between
Secondly, compared finetune-based methods, methods without fine-tuning perform poorly, in- dicating that fine-tuning is crucial for SLMss. At the same time, we found that both SelfAsk and D-a experience a performance drop in the

---

Methods
Prompt Structure
All-at-once?
ICL FT
SelfAsk
Input: Original question I Output: Sub-question Intermediate
All-at-once
In-context Learning
step +... Final answer
Decomposer: Iopi:: Original question Ouipu:: Sub-questions
DialCoT-A
All-at-once
Finc-tuning
Solver: Input: Original question + Sub-questions Oudput: Inter-
mediate steps Final unswer
Decomposer: Imput: Original question Output: Sub-questions
Solver: Ixput: Original question Sub-question Ondpu: Inter-
Output intermediate
DialCoT-M
mediate step
Fine-tuning
steps sequentially
Solver: Input: Original question + Intermediate step 1 + Sub-
question 21 Output: Intermediate step 2
Solver: Impat: Original question Intermediate steps Final
question Outpud: Final answer
Decomposer: Input: Original question I Outpuz: Sub-question
Solver: Iиpиr: Original question Sub-question Outpuz: Inter-
mediate step
Output sub- sub-questions and intermediate steps alternateyy
Decomposer: Imput: Original question Sub-question Inter-
DialCoT-S
Fine-tuning
mediute step Output: Sub-question 2
Solver: Ixput: Original question Sub-question Internediate step Sub-question 2
Output: Intermediate step 2
Decomposer: Input: Original question Sub-questions Inter- mediate steps Ourput: Final question
Solver: Input: Original question Sub-questions+ + Intermediate
steps + Final question
Output: Final answer
Table 3: Detailed Comparison between DialCoT and SelfAsk. The table omits the specific details of the prompts and instead focuses on the input-output formats of the prompts. DialCoT is fine-tuned using distinct instructions tailored for two specialized tasks (Decomposer and Solver) on a shared model.
Method
Finetune or not finetune
GSM8K
D Detailed Discussion of Three DialCoT Variants
Standard CoT
16.1
SelfAsk
all once, finetune
21.6
SelfAsk
sequencially, finetune
29.3
From Table 3, clearly the differences in the prompt structures of different methods. Com- bined with the experimental results from Table 2, we can draw the following conclusions: (1) In a side comparison between standard CoT and DialCoT-A, it becomes evident that DialCoT-A employs self-generated sub-problems as strate- gic guide for formulating intermediate steps and solutions. The enhanced performance of DialCoT- A over standard CoT implics that self-generated navigation through these sub- problems can signifi- cantly enhance the reasoning capability of Smaller Language Models (SLMs). (2) When comparing DialCoT-M and DialCoT-A, the former opts for a sequential approach to answering sub-questions, opposed to addressing them all-at-once. The supe- rior performance metrics of DialCoT-M in compar- ison DialCoT-A substantiate the claim that a quential methodology for answering sub-questions yields greater efficacy than an all-at-once approach. (3) The primary difference between DialCoT-S and DialCoT-M is that DialCoT-S generates sub-
DialCoT-S
finetune
35.2
Standard CoT
without finetune
12.7
SelfAsk
without finetune
11.3
DialCoT-S
without finetune
10.9
Table 4: Accuracy (%) of various methods on the GSM8K dataset. The fine-tuning dataset used for all experiments is the GSM8K training set. The prompt structures corresponding to the two different fine-tuning methods for SelfAsk are shown in Table
inference time and performance needs to be con- sidered. If faster inference speed is required, one can opt for DialCoT-A, the expense of some per- formance loss. Conversely, if better performance is the priority, DialCoT-S can chosen, although this would come at the cost increased inference time.

---

Method
Fine-tuning Melhed all at once
Prompt Structure
SelfAsk
Iiput: Original question Output: Follow up: Sub-question
Intermediate answer: Intermediate step 1
Follow up: Sub-question 2
Intermediate
Intermediate step Follow up: Final question
Intermediate answer: answer
SelfAsk scquencially
Iiput: Original question
Follow Output: Sub-question
Inpu:: Original question
Follow up: Sub-question
Intermediate answer: Output: Intermediate step 1
Input: Original question
Follow up: Sub Lub--quetion!
Intermediate answer: Intermediate step
Follow up: Output: Sub-question 2
Imput: Original question
Follow up: Sub-question
Intermediate answer: Intermediate step Follow up: Outpar: Final question
Imput: Original question
Follow up: Sub-question
Intermediate answer: Intermediate step +... Follow up: Final question
Intermediate answer: Output: Final answer
Table 5: Prompt structures corresponding the different fine-tuning methods for SelfAsk.
Method
GSM8K
SelfAsk All-at-once finetuning
21.00
SelfAsk Sequential finetuning
22.95
Table 6: Accuracy (%) of SelfAsk using L.L.MA-BB on
the GSMSK dataset.
Method CoT-FT 55.2
first step
second step
final step
41.9
34.7
DialCoT-S 60.7
48.8
43.8
DiaICoT-S-PPO 63.9
51.2
45.3
Table 7: Accuracy (%) of various methods using TS-
XXL on the GSM8K dataset for each sub-step. CoT-FT refers to the results obtained by replacing the original question with different step-based sub-questions.
questions sequentially rather than generating all sub-questions at once. Given DialCoT-S's stronger performance metrics, this indicates that the ap- proach of sequentially decomposing sub-questions is more effective than generating all sub-questions in a single step.


Reviews:
[{"review_id": "i5LeTJeMDh", "reviewer": "Reviewer_Rgam", "summary": "", "questions": "A. How much of the improvement is due to the proposed prompting method vs fine-tuning of the LM? An ablation study that investigates the proposed method for non-fine-tuned LMs would be interesting, and conversely, one using vanilla CoT with a fine-tuned LM.\n\n B. Is the improvement in performance due to PPO statistically significant? 2% doesn't seem like a very substantial difference. But the improvement seems to be larger for decoder-only models.\n\n C. In Figure 4, what is the prompting method used for \"FlanT5\"?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Before reviewing the responses, **please make sure to read the [general response](https://openreview.net/forum?id=VyjNXY2wgi&noteId=sjLPj6ptDw) first.** Thank you very much for your valuable comments. Below are our replies to the questions you have raised and we hope they can address your concerns.\n\n**R1:** Aside from the fine-tuning and PPO, the approach seems very similar to recent question decomposition prompting approaches, such as SelfAsk. And the improvement due to PPO seems marginal relative to the other aspects of the approach. Would something like SelfAsk + fine-tuning work just as well as the proposed approach?\n\n**A:** Firstly, we propose three different forms of DialCoT, among which DialCoT-A is somewhat similar to SelfAsk[1], while **DialCoT-M and DialCoT-S employ entirely different approaches from SelfAsk** for guiding the model to generate intermediate reasoning steps. Secondly, the results of SelfAsk+ fine-tuning can be approximately analogized to the results of DialCoT-A. Compared to DialCoT-A, both DialCoT-M and DialCoT-S achieved a greater performance gain, indicating that **the approach of incrementally generating questions and answers (DialCoT-M/DialCoT-S) is more effective than generating all questions and answers at once (DialCoT-A/SelfAsk).** Finally, our proposed step-level PPO further optimizes the model's reasoning path, which brings more improvement. Based on our experiments shown in Table 2,   the overall method DialCoT-S-PPO is way more effective than DialCoT-A, which indicates SelfAsk+fine-tuning would not work as well as ours. Nevertheless, a more rigorous study should be performed as future research.\n\n**Q1:** How much of the improvement is due to the proposed prompting method vs. fine-tuning of the LM? An ablation study that investigates the proposed method for non-fine-tuned LMs would be interesting, and conversely, one using vanilla CoT with a fine-tuned LM.\n\n**A:** **CoT-FT is the model fine-tuned in the form of vanilla CoT using the same training data as our method.** As seen in Table 2, our method (DialCoT-S) outperforms CoT-FT by **9.1%**, where the improvement can be mainly attributed to the prompting method we propose. On the other hand, since smaller models do not acquire strong reasoning capabilities during the pre-training phase, relying solely on in-context learning without fine-tuning yields poor results in stimulating their reasoning capabilities, as evidenced in [2]. This is also why current research aimed at improving the reasoning capabilities of smaller models[3][4][5] all require fine-tuning. Therefore, we believe that an ablation experiment using few-shot CoT prompting without fine-tuning on smaller models is unnecessary.\n\n**Q2:** Is the improvement in performance due to PPO statistically significant? 2% doesn't seem like a very substantial difference. But the improvement seems to be larger for decoder-only models.\n\n**A:** Our experimental results are the average outcomes across five different seeds, and our test set contains 20,000 questions. A stable increase of 2% means correctly answering an additional 400 questions. Meanwhile, **it's important to emphasize that this improvement is achieved on top of the already substantial performance gains realized by DialCoT-S.** Therefore,  we believe such improvement should NOT be treated as **minor**.\n\n**Q3:** In Figure 4, what is the prompting method used for \"FlanT5\"?\n\n**A:** In Figure 4, Flan-T5 employs Few-shot CoT[2] prompting. Specifically, the prompt used includes 8 exemplars, and an example of one is as follows:\n\n\"Question: Ivan has a bird feeder in his yard that holds two cups of birdseed. Every week, he has to refill the emptied feeder. Each cup of birdseed can feed fourteen birds, but Ivan is constantly chasing away a hungry squirrel that steals half a cup of birdseed from the feeder every week. How many birds does Ivan’s bird feeder feed weekly?\nAnswer: The squirrel steals 1/2 cup of birdseed every week, so the birds eat 2 - 1/2 = 1 1/2 cups of birdseed. Each cup feeds 14 birds, so Ivan’s bird feeder feeds 14 * 1 1/2 = 21 birds weekly. So the answer is 21.\"\n\nWe will add detailed information about all prompts we use to the appendix of the revised version. Thanks again for your valuable comments.\n\n[1] Measuring and narrowing the compositionality gap in language models, 2022\n\n[2] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, NIPS 2022\n\n[3] Large Language Models Are Reasoning Teachers, 2022\n\n[4] Star: Self-taught reasoner bootstrapping reasoning with reasoning, 2022\n\n[5] Specializing Smaller Language Models towards Multi-Step Reasoning, ICML 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed approach is interesting as it leverages the fact that many recent LMs are instruction-tuned on dialogue/chat data, and so it may be a more effective way to elicit reasoning in smaller models.\n - They experiment with Flan-T5, a smaller open LLM, which facilitates reproducibility.\n - The paper is well-written and easy to understand.", "reasons_to_reject": "- Aside from the fine-tuning and PPO, the approach seems very similar to recent question decomposition prompting approaches, such as SelfAsk. And the improvement due to PPO seems marginal relative to the other aspects of the approach. Would something like SelfAsk + fine-tuning work just as well as the proposed approach?", "questions_to_authors": "A. How much of the improvement is due to the proposed prompting method vs fine-tuning of the LM? An ablation study that investigates the proposed method for non-fine-tuned LMs would be interesting, and conversely, one using vanilla CoT with a fine-tuned LM.\n\n B. Is the improvement in performance due to PPO statistically significant? 2% doesn't seem like a very substantial difference. But the improvement seems to be larger for decoder-only models.\n\n C. In Figure 4, what is the prompting method used for \"FlanT5\"?", "paper_topic_and_main_contributions": "The authors propose a new approach to elicit more advanced reasoning in smaller LMs, by breaking down the CoT into a dialogue format between a \"Decomposer\" and \"Solver.\" The authors test whether this format is able to better elicit reasoning in smaller LMs such as Flan-T5.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper is well-written, with only a few minor errors:\n\nEquation 4: This could be made easier to read by increasing the spacing between some of the variables, and increasing the sizes of the brackets to more appropriately surround the fractions, etc.\n\nLine 400: \"We\" -> \"we\""}}, {"review_id": "TgefCqDHxn", "reviewer": "Reviewer_orRw", "summary": "", "questions": "QA) I doubt that this method has limitations due to multi-round dialogue. For example,  the paper does not show how many rounds of dialogue are usually required to solve a problem because this form of dialogue is equivalent to output sub-questions and answers at each step, which is longer than direct step-by-step reasoning. If the problem is more complex and the number of rounds of dialogue increases, will there be a significant decrease?\n\nQB) Some parameters in the RL section are set relatively briefly. For example, why should rm be selected as 0.3 in Table 1 in the reward model? Is it differ from task types? The authors do not prove such a setting, which seems to be summarized from the experimental results of downstream tasks.\n\nQC) Homogenization of task types. As the paper says, \"All datasets comprise arithmetic reasoning problems at a primary school level, varying by the entities they incorporate\", I doubt the generalization of methods on other tasks.\n\nQD) How is the intermediate step data obtained in the training data, such as how to determine whether the solver answers correctly or incorrectly for a sub-problem decomposed from the decomposer?\n\nQE) Should the action space of models of different sizes be different because the models have different abilities to decompose problems correctly?\n\nQF) Is the method still effective on more complex datasets that require more reasoning steps and larger action space?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Before reviewing the responses, **please make sure to read the [general response](https://openreview.net/forum?id=VyjNXY2wgi&noteId=sjLPj6ptDw) first.** Thank you very much for your valuable comments. Below are our replies to the questions you have raised and we hope they can address your concerns.\n\n**R1:** The Intro section explains two advantages, but the first advantage is the idea that CoT already exists, and the innovation is not enough to be the first contribution of the paper. Except for this part, the paper's overall innovation appears insufficient. \n\n**A:** Vanilla CoT still employs LLM to directly answer reasoning questions but provides intermediate reasoning steps before offering the final answer. The first advantage of our method is that **we decompose the challenging task of directly answering reasoning questions into two simpler sub-tasks**: one sub-task is breaking down the original question into simpler sub-questions, and the other sub-task is answering a series of simple questions. **This effectively reduces the complexity of the original task and is more suitable for SLMs**, which is an advantage not present in vanilla CoT. Furthermore, we propose a **step-level PPO**, which is fundamentally different from the token-level PPO in RLHF[1], to further optimize the model's reasoning path. The overall method yields significant gains compared with Vanilla CoT finetuning.\n\n**R2:** The purpose of using PPO is to choose a better reasoning path, but the experimental part only shows the overall effect after using the PPO method. It is recommended to supplement the model accuracy at different stages after dismantling sub-problems for different methods, which can better illustrate the method's effectiveness.\n\n**A:** We select samples that require decomposition into three sub-questions from all test sets and report the accuracy for each sub-question, as shown in the table below. **It is evident that PPO significantly improves the model's performance on reasoning tasks at every step.** Additionally, we found that as the number of steps increases, the performance gain from PPO tends to diminish. We will include the detailed experimental results in the revised version. Thank you for your valuable comments.\n|Method|first step|second step|final step|\n|:-----:|:---:|:----:|:----:|\n|DialCoT-S|60.7|48.8|43.8|\n|DialCoT-S-PPO|63.9|51.2|45.3|\n\n**R3:** Using the dialogue method to reason, the comparison of reasoning speed and time cost compared with the previous method needs to be supplemented.\n\n**A:** In terms of the time cost for reasoning, **the inference speed of our method is comparable to that of SelfAsk[2]**, as both need to generate sub-questions and answers. In terms of the number of iterations, **DialCoT-A requires two iterations, which is comparable to Zero-shot CoT[6]**, while DialCoT-M and DialCoT-S require approximately three and six iterations respectively. Overall, in our methods, DialCoT-A has the fastest inference speed, followed by DialCoT-M, and finally DialCoT-S. The **trade-off between inference time and performance** needs to be considered. If faster inference speed is required, one can opt for DialCoT-A, at the expense of some performance loss. Conversely, if better performance is the priority, DialCoT-S can be chosen, although this would come at the cost of increased inference time. We will include a more detailed report on the inference costs in the revised version. Thank you for your valuable comments. \n\n**R4/5:** Figure 2 is overcrowded, poor readability. Figure 3 is the main method diagram of the paper, which is separated from Figure 2, and it isn't easy to intuitively compare the differences between different methods.\n\n**A:** Thank you for your valuable suggestions. In the revised version, we will place the architecture diagrams of DialCoT-S-PPO and the other methods together to make it easier for readers to compare the differences between them. We will also try our best to reduce the texts in Figure 2 to enhance readability.\n\n**Q1:** If the problem is more complex and the number of rounds of dialogue increases, will there be a significant decrease?\n\n**A:** Firstly, through statistical analysis, we found that on average, a question in our test set needs to be decomposed into 2.7 sub-questions. Secondly, we select samples that require decomposition into three sub-questions from all test sets and report the accuracy for each sub-question, as illustrated in the table below. In this context, CoT-FT refers to the results obtained by replacing the original question with different step-based sub-questions. As shown in the table, as the number of steps increases, i.e., as the complexity of the questions rises, **our method shows a notable performance improvement (~7%) compared to CoT-FT**. On the other hand, although our method's performance does decline as question complexity increases, **the rate of decline is significantly slower compared to vanilla CoT.** We will include the detailed experimental results in the revised version. Thank you for your valuable comments.\n|Method|first step|second step|final step|\n|:-----:|:---:|:----:|:----:|\n|CoT-FT|55.2|41.9|34.7|\n|DialCoT-S|60.7|48.8|43.8|\n\n**Q2:** Some parameters in the RL section are set relatively briefly. For example, why should rm be selected as 0.3 in Table 1 in the reward model? Is it differ from task types?\n\n**A:** We obtained **the best performance on the validation set** when the intermediate reward score was set to 0.3, as determined through **grid search**. Therefore, we chose to set the intermediate reward score at 0.3. **Using the same settings across multiple arithmetic reasoning datasets yielded significant performance gains.** We are in the process of extending our method to common sense reasoning and symbolic reasoning tasks, and we will continue to explore the impact of these hyperparameters on different types of tasks in the future.\n\n**Q3:** Homogenization of task types. As the paper says, \"All datasets comprise arithmetic reasoning problems at a primary school level, varying by the entities they incorporate\", I doubt the generalization of methods on other tasks.\n\n**A:** In this paper, we **focus on arithmetic reasoning tasks**. We conducted experiments on **four arithmetic reasoning datasets** to validate the effectiveness of our method. These four datasets cover **the most commonly used datasets in arithmetic reasoning-related papers**[3][4][5], sufficiently demonstrating the generalizability of our method for arithmetic reasoning tasks. We plan to further extend our approach to commonsense reasoning and symbolic reasoning tasks in the future.\n\n**Q4:** How is the intermediate step data obtained in the training data, such as how to determine whether the solver answers correctly or incorrectly for a sub-problem decomposed from the decomposer?\n\n**A:** The original GSM8K dataset provides sub-questions and answers for each sample. We directly utilized the training set from GSM8K.\n\n**Q5:** Should the action space of models of different sizes be different because the models have different abilities to decompose problems correctly?\n\n**A:** We believe that the action space should differ for models of varying sizes. When the model is larger, the correct response is more likely to appear among the top few candidates, thereby reducing the space that needs to be explored.\n\n**Q6:** Is the method still effective on more complex datasets that require more reasoning steps and larger action space?\n\n**A:** As can be seen from the table above, **our method remains effective for more complex data that requires additional reasoning steps.** On the other hand, as shown in Figure 6, **as the action space increases, PPO becomes more difficult to converge, consequently affecting the model's performance.** Therefore, it's reasonable to speculate that the performance of our method may decline for problems requiring a larger action space. We plan to continue exploring the performance of our approach on more complex tasks in the future.\n\n[1] Training language models to follow instructions with human feedback, NIPS 2022\n\n[2] Measuring and narrowing the compositionality gap in language models, 2022\n\n[3] Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, NIPS 2022\n\n[4] Self-consistency improves chain of thought reasoning in language models, ICLR 2023\n\n[5] Specializing Smaller Language Models towards Multi-Step Reasoning, ICML 2023\n\n[6] Large Language Models are Zero-Shot Reasoners, NIPS 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Introduce PPO to help the model choose a better reasoning path\n2. Compared with the previous method, the performance has been greatly improved\n3. The experimental part illustrates the effectiveness of the method to a certain extent through different settings", "reasons_to_reject": "1. The Intro section explains two advantages, but the first advantage is the idea that CoT already exists, and the innovation is not enough to be the first contribution of the paper. Except for this part, the paper's overall innovation appears insufficient.\n2. The purpose of using PPO is to choose a better reasoning path, but the experimental part only shows the overall effect after using the PPO method. It is recommended to supplement the model accuracy at different stages after dismantling sub-problems for different methods, which can better illustrate the method's effectiveness.\n3. Using the dialogue method to reason, the comparison of reasoning speed and time cost compared with the previous method needs to be supplemented\n4. Figure 2 is overcrowded, poor readability\n5. Figure 3 is the main method diagram of the paper, which is separated from Figure 2, and it isn't easy to intuitively compare the differences between different methods", "questions_to_authors": "QA) I doubt that this method has limitations due to multi-round dialogue. For example,  the paper does not show how many rounds of dialogue are usually required to solve a problem because this form of dialogue is equivalent to output sub-questions and answers at each step, which is longer than direct step-by-step reasoning. If the problem is more complex and the number of rounds of dialogue increases, will there be a significant decrease?\n\nQB) Some parameters in the RL section are set relatively briefly. For example, why should rm be selected as 0.3 in Table 1 in the reward model? Is it differ from task types? The authors do not prove such a setting, which seems to be summarized from the experimental results of downstream tasks.\n\nQC) Homogenization of task types. As the paper says, \"All datasets comprise arithmetic reasoning problems at a primary school level, varying by the entities they incorporate\", I doubt the generalization of methods on other tasks.\n\nQD) How is the intermediate step data obtained in the training data, such as how to determine whether the solver answers correctly or incorrectly for a sub-problem decomposed from the decomposer?\n\nQE) Should the action space of models of different sizes be different because the models have different abilities to decompose problems correctly?\n\nQF) Is the method still effective on more complex datasets that require more reasoning steps and larger action space?", "paper_topic_and_main_contributions": "This paper uses multi-round dialogue to decompose the problem using small models and answer the decomposed sub-problems step by step. At the same time, the RL algorithm is used to select a better sub-problem decomposition/reasoning path, which improves the reasoning ability of small models.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MUAD90vAIX", "reviewer": "Reviewer_v8cq", "summary": "", "questions": "Quite rarely, I don't have any minor questions except from the very important ones in Reasons To Reject. The paper is very well-written in this regard.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Before reviewing the responses, **please make sure to read the [general response](https://openreview.net/forum?id=VyjNXY2wgi&noteId=sjLPj6ptDw) first.** Thank you very much for your valuable comments. Below are our replies to the questions you have raised and we hope they can address your concerns.\n\n**R1:** \"The insufficiency of explaining why or how the dialog format is helpful. Dialog is a format. Regular CoT is also a format.  ... Without stronger evidence that the dialog format is really the secret sauce here (a fact which is not that self-evident for LLMs), it's hard to gain insights from this paper beyond \"this particular data format works well and I only have some speculations of why.\"\"\n\n**A:** Firstly, **CoT-FT and our method use exactly the same training data** (7,000 samples from the GSM8K training set). The results in Table 2 confirm that **the form of DialCoT is superior to the regular form of CoT**. Secondly, by comparing the results of DialCoT-A and DialCoT-M, we find that **sequentially answering individual sub-questions is more effective than answering all sub-questions at once.** Comparing the results of DialCoT-M and DialCoT-S, we find that using a format to **sequentially break down the problem is more effective than decomposing all sub-questions at once.** The above are the insights that can be drawn from the experiments in this paper. For a more fine-grained analysis, such as the specific impact of punctuation, instructions, and other parts on model performance, we will conduct more experiments and analyses in the future. We hope our work could inspire more research efforts in this direction.\n\n**R2:** The usage of PPO seems to be a secondary contribution of the paper, but its effect is not sufficiently disentangled from that of the dialog format. The regular CoT format can also incorporate PPO. Without further experiments, we're left with a half-baked claim that \"PPO works well with the dialog format\" without understanding the whole picture.\n\n**A:** Firstly, we would like to clarify that our step-based PPO is not a secondary contribution but rather **one of the main contributions of this paper.** We are **the only ones to utilize the PPO method to optimize the intermediate reasoning steps of the SLMs.** This resulted in a further improvement on top of the already significant performance gains achieved by DialCoT-S. Specifically, our version of PPO is step-level, which is completely different from the token-level PPO in RLHF[1]. **Regular CoT cannot explicitly separate each reasoning step, so it's not suitable for the step-level PPO we propose.** In contrast, our method has effectively decomposed the reasoning steps, making it more suitable for step-level PPO. \n\n**R3:** The performance gain of DialCoT-A over CoT-FT, as well as that of DialCoT-S-PPO over DialCoT-S is rather small (considering literature that works on GSM8K), but I'll not use this point as a deal-breaker.\n\n**A:** Compared to DialCoT-M and DialCoT-S, DialCoT-A is most similar to regular CoT, and therefore yields smaller performance gains. As argued before, **our step-level PPO achieved an additional 2% improvement on top of the already substantial performance gains realized by DialCoT-S.** Considering that our test set contains 20,000 questions, this means an additional 400 questions were answered correctly, which is a stable and very noticeable performance boost.\n\n[1] Training language models to follow instructions with human feedback, NIPS 2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Assuming the findings of this paper, the dialog format has the potential to compete with or even replace the standard CoT format in certain scenarios.\n2. The paper is well-presented with sound logic behind design choices.", "reasons_to_reject": "My major objection to this paper's acceptance is:\n1. The insufficiency of explaining why or how the dialog format is helpful. Dialog is a format. Regular CoT is also a format. Thus, it would be imperative to rigorously control for this variable by doing a precise side-by-side comparison with two finetuning datasets whose only difference is the format (dialog vs. regular CoT). The citing of CoT-FT (Wei et al., 2021) row does not sufficiently answer this question as there could be differences in the finetuning data. Even were I to be informed that CoT-FT (Wei et al., 2021) is exactly the same as the proposed work except the format, it is quite necessary to show what exactly is it in the format that boosts performance. Is it simply about using a question instead of a statement? Is it because punctuations like question marks (a little silly, but the point is made)?\nExample of a strict side-by-side: \nDialog: Sub-question 1: How many students were picked for the trivia teams? Sover:...\nCoT: First, I need to find out the number of students that were picked for the trivia teams. That is calculated as ...\nWithout stronger evidence that the **dialog format** is really the secret sauce here (a fact which is not that self-evident for LLMs), it's hard to gain insights from this paper beyond \"this particular data format works well and I only have some speculations of why.\"\n\n2. The usage of PPO seems to be a secondary contribution of the paper, but its effect is not sufficiently disentangled from that of the dialog format. The regular CoT format can also incorporate PPO. Without further experiments, we're left with a half-baked claim that \"PPO works well with the dialog format\" without understanding the whole picture.\n\n3. The performance gain of DialCoT-A over CoT-FT, as well as that of DialCoT-S-PPO over DialCoT-S is rather small (considering literature that works on GSM8K), but I'll not use this point as a deal-breaker.", "questions_to_authors": "Quite rarely, I don't have any minor questions except from the very important ones in Reasons To Reject. The paper is very well-written in this regard.", "paper_topic_and_main_contributions": "The paper tackles the task of finetuning SLMs to perform symbolic reasoning tasks such as GSM8K. The proposed method is to format the data as dialog turns instead of regular CoT texts. The paper shows favorable performance of doing so on a variety of datasets. Furthermore, the incorporation of PPO further boosts accuracy by 1-2%.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed methods to decompose CoT with dialog format for smaller models and a step-wise PPO to improve the results further.

**Pros**: The paper is very well written and proposes a very well-motivated method to solve the problem timely with open-source smaller models. (all reviewers)

**Cons**: Some of the figures could be improved (**orRw**)

Overall, the paper presentation is clear, with strong motivations. The proposed method is intuitive, with solid logic behind the design choices. The experiment results are strong compared to baselines using smaller models.
Decision: Accept-Main