Small Language Models Fine-tuned to Coordinate Larger Language Models improve Complex Reasoning
Authors: Gurusha Juneja
Subhabrata Dutta
Soumen Chakrabarti
Sunny Manchanda
Tanmoy Chakraborty
Venue: EMNLP 2023 Main (2023)
Keywords: Language Model Reasoning
Problem decomposition
Multistep reasoning
TL;DR: We propose an automated, model agnostic method that takes (possibly erroneous) chain-of-thought reasoning output from LLMs, breaks the reasoning problem into multiple subproblems, and guide the LLM through those subproblems to correct reasoning.

Abstract: Large Language Models (LLMs) prompted to generate chain-of-thought (CoT) exhibit impressive reasoning capabilities. Recent attempts at prompt decomposition toward solving complex, multi-step reasoning problems depend on the ability of the LLM to simultaneously decompose and solve the problem. A significant disadvantage is that foundational LLMs are typically not available for fine-tuning, making adaptation computationally prohibitive. We believe (and demonstrate) that problem decomposition and solution generation are distinct capabilites, better addressed in separate modules, than by one monolithic LLM. We introduce DaSLaM, which uses a decomposition generator to decompose complex problems into subproblems that require fewer reasoning steps. These subproblems are answered by a solver. 
We use a relatively small (13B parameters) LM as the decomposition generator, which we train using policy gradient optimization to interact with a solver LM (regarded as black-box) and guide it through subproblems, thereby rendering our method solver-agnostic. Evaluation on multiple different reasoning datasets reveal that with our method, a 175 billion parameter LM (text-davinci-003) can produce competitive or even better performance, compared to its orders-of-magnitude larger successor, GPT-4. Additionally, we show that DaSLaM is not limited by the solver's capabilities as a function of scale; e.g., solver LMs with diverse sizes give significant performance improvement with our solver-agnostic decomposition technique. Exhaustive ablation studies evince the superiority of our modular finetuning technique over exorbitantly large decomposer LLMs, based on prompting alone.

Small Language Models Fine-tuned to Coordinate Larger Language Models improve Complex Reasoning
Gurusha Juneja" IIT Delhi, India
Subhabrata Dutta" IIT Delhi, India
Soumen Chakrabarti IIT Bombay, India
Sunny Manchhanda DYSL-AI, India
Tanmoy Chakraborty IIT Delhi, India
Abstract
models (LMs) (Devlin et al., 2019) in essence, fitting continuous feature vectors to tokens and modeling smooth conditional distributions over thousands of token positions with multi-task ob- jectives. The next generation of large LMs (LLMs) such as T5, GPT4 and Bard (Raffel et al., 2020; OpenAL, 2023) developed protean capabilities, ex- tending to mathematical and logical ability, based on prompting and in-context learning. Chain-of- thought (CoT) prompting has been key enabler (Wei et al., 2022; Feng et al.. 2023). LLMs can solve middle-school word problems and equations reasonably well. has also acquired the ability to invoke specialized external tools such as Wolfram Alpha (Wolfram, 2023; Schick et al., 2023).
Large Language Models (LLMs) prompted to generate chain-of-thought (CoT) exhibit im- pressive reasoning capabilities. Recent at- tempts prompt decomposition toward solv- ing complex, multi-step reasoning problems depend on the ability of the LLM to simulta- neously decompose and solve the problem. A significant disadvantage is that foundational LLMs are typically not available for fine- tuning, making adaptation computationally pro- hibitive. We believe (and demonstrate) that problem decomposition and solution genera- tion distinct capabilites, better addressed in separate modules, than by one monolithic LLM. We introduce DaSLaM, which uses decomposi- tion generator decompose complex problems into subproblems that require fewer reasoning steps. These subproblems answered by solver. We use relatively small (13B parame- ters) LM as the decomposition generator, which we train using policy gradient optimization to interact with solver LM (regarded as black- box) and guide through subproblems, thereby rendering our method solver-agnostic. Evalu- ation on multiple different reasoning datasets reveal that with our method, a 175 billion pa- rameter LM (text-da) addvvinni-eee can produce competitive or even better performance, com- pared to its orders-of-matet ma-giniude larger succes- sor, GPT-4. Additionally, we show that DaSLaM is not limited by the solver's capabilities as a function of scale; e.g., solver LMs with di- verse sizes give significant performance im- provement with our solver-agnostic decompo- sition technique. Exhaustive ablation studies evince the superiority of our modular finetuning technique over exorbitantly large decomposer L.MMs, based prompting alone.
Recent advances in LLMs have arisen largely from dleveely-engineered, ever-growing training data, rather than any significant change in network structure, which remains relatively regular, but with rapidly increasing network size and number of pa- rameters. One outcome of such giant monolithic LLMs is unsustainable levels of hardware and en- ergy (Dhar, 2020) to train them. Meanwhile, neu- rologists and brain scientists have known, via fMRI scans, inter alia, that cerebral functions are special- ized and spatially localized (Fedorenko and Varley, 2016; 6; Mahowald et al., 2023).
Many recent complex reasoning challenges thrown at LLMs have a two-level character - the input task needs to be decomposed into subtasks, then the subtasks need to be solved, and finally, subtask solutions have to be consolidated and com- bined to solve the input task. Existing approaches use the same LLM both decompose and solve the task, sometimes in tangled and uninterpretable ways. Because the sharing of an LLM across these functions cannot be closely controlled, very large models are needed for this double ability (decom- pose and solve) to emerge.
1 Introduction
In recent years, astounding variety of text and NLP tasks have been accomplished by language
"Equal contribution author. "Correppondence: tanchak@t i td.ac..ii in
Staying entirely inside the LLM regime, and avoiding the possibility of specialized tools, we

---

Decomposer LM
dag the
X
Wast the qreeztaio d O,F. Q.8.27
Vataalao Ear pesilees
Tae cocebzatoo of lbe peass 0 wr (3.0 9:
The cocrizates of la peass F ws (3.0.1
eoik ea segas and GP and 0R clone the ase und lis y oea repecteede
Copuar pernusd DPQRS locate las lost sctaei (o 2 9. the
Tercernzaaioo the pesss
5,0
3
The bar
jonaadd square with
prin
Theerriiatto of lbe peesi 7 ux (1.5.15.0))
alone
diguual
The seedlratoo of iiu paiza 8 am (1.5.1..33 1.5.5)
(3) wose maal seseec O2 mad GB
(B) the coution the plase conteisng the itieaal ogg is -
(C) lonc the pap dester froe to plane contriting trtkagk
Whst the oquiian of the place costeising ite lningh 0PP
perpendaaden diinnee froe
aongan liac
The poke de yebr diggwal Og Tivedtrr, the ove Slaates of oan be fondd by sakine the serig of conniratto of aut
v4
a- 300 a of Ein () +. d)) (...
Theyine
Thender acceedaatt of
(A) sotte regle bewes OQ and use sanng pridus
Solver LM
of be frodd by odfine
s- d. 2-3-- 2. (..d Et
d2 68 (02)00) cnd u/ 105)
and--
3. They plane costiiiig the trknnge OPg is peppodeede ded to the oate dp
O2, Theeforr, the equalise lho plant
food by ualin ihe de poolect contll vadte od ploar will the poolion wont no pouz
di azt entaaa UGS on vittee 5- H f- j= uge to y-g
equatien of the plae
Let olk the peist Fis the post the plane The soenel vndo of
plase
sccatit plans contaring integl oge i -i ogg be loand peppadaciar
plane is glans S OP Gl) - (6 5.0y (8.2.6 - (03000
Thessoos opactoo toe place glowa be E (0 0.3)
ihmaase seanis, Let the cxedain I is (tu is A.l. The equiin the plane gane y-y y u. Ltemiee thr darazes berwen the pean and plane pinn by:
repution
wonsiong
dlow
22 is
Wass oquation pase mosuansng far tcsspp
peppodicate
plow
mtagke cags 2g
be braud uder the fetatce becrdls Lee the conditates of o be (ey
dittaso five to the cosige the corsaing 85
Tae
Tae courüazzes of lle poist P ice 2,000)
and coofione and tes and to muidd reportieds The expation
crarizzkko the poist ir tasogle GQS
it apg Thed the pryyattonaa dheam hor O die emuight the conaiing RS
the of
Pid (21,
of Ga
tostnge O0S
Figure 1: Working example of DaSLaM on mathematical reasoning question from the JEEBench dataset (Arora et al., 2023). In this example, the solver LM is text-davinci-00. In step D the solver is prompted to answer the question (blue textbox) and it fails to answer correctly (red textbox). A problem decomposing LM generates subproblems (violet textboxes) conditioned on the original question and the initial response of the solver in step ② step 3 the solver answers these subproblems iteratively and appends to the prompt. Finally, the original problem is appended to the prompt in step ④,, and the solver answers it correctly (green textbox).
ask a simple question - is it possible to affload the ability of problem decomposition to a dedicated, relatively smaller scale model, which is specialized and can act in synergy with any solver model of choice? To incorporate flexibility and better gen- cralization, an immediate requirement of such a setup would be to enable a model-agnostic commu- nication between the decomposer and the solver.
mance of OpenAl GPT-3 text-davinci-088 outperform GPT-3.5 and even begins to compete with GPT-4, outperforming other similar methods. DaSLaM boosts text-davinci-099 from an exact match accuracy of 41.6 to 54.5 in zero-shot regime, which is 3.9 points higher than few-shot GPT-4, Similarly, on Physics problems from JEEBench dataset, DaSLaM-augmente scores only 0.58 points short of GPT-4 while out- performing GPT-3.5. The decomposer LM in DaSLaM reduces decomposition errors, and general- izes well across diverse small-scale LMs. It is also more robust in the face of difficult datasets, where the solver gives near-random performance.
text-davinci-003
Our contributions. To study this research ques- tion, we develop DaSLaM (Decomposition And So- lution LAnguage Models), in which we separate the decomposer from the solver, as shown in Fig- ure 1. The solver LM can be conventionally trained or fine-tuned. In the illustration, when it answers a question incorrectly, the decomposer LM takes over to produce sub-questions. The partial solu- tions are appended and resubmitted to the solver LM, which solves the question correctly. The de- composer LM regards the solver as a black box, and uses reinforcement learning (RL) to become a specialized expert at decomposition, informed by the solver's mistakes.
These results support our founding hypothesis that heterogeneous functional specialization im- proves model efficiency and robustness of LLMs. A crucial findings from our experiments is that finetuning the decomposer is much more powerful choice than finetuning the solver. Morcover, a fine- tuned decomposer is largely superior compared an orders of magnitude larger LLM prompted act as a decomposer. Given the prohibitive cost of finetuning LLMs like GPT 3, 3.5, or 4, we hope this method would provide us a promising direction to-
Extensive experiments with three reasoning datasets (MATH, AQuA, and JEEBench) show that the proposed specialization improves the perfor-

---

wards future development of task-expert models."
3 A General Overview of DaSLaM
2
Related Work
Given an LM 0 and a question Q as sequence of tokens, standard zero-shot prompting can be described as,
Eliciting superior reasoning abilities in LM through specially designed prompts has found its popu- larity through CoT prompting (Wei et al., 2022) asking the LM to explain the reasoning steps improves overall performance. Decomposing a complex reasoning requirement into multiple, sim- ple steps results in superior reasoning capabili- ties across modalities other than free-form natural language text as well, e.g., reasoning over tabu- lar data (Ye et al., 2023), visual question answer- ing (Lu et al., 2022), etc. These methods generally solicit a single run of LM inference with no inter- mediate prompting interactions. Consequently, the LM often misses key reasoning steps or halluci- nates irrelevant ones.
Á = arg max Po (AlQ)
AEA
where Å is the inferred answer, and is the set of possible answers (c.g., numerical values, multiple- choice options, True/False, etc.). With a CoT prompting, the LM generates sequence of tokens explaining the steps S to reach the answer given the question. The modified process can be described as,
Å == Po(A||s Q) arg max pe (5(Q))
(1)
When answering Q requires multistep reasoning, one can conceptualize S sequence of smaller steps (S), S22 ... .., such that the LM íteratively answers sequence of subproblems {Q,,... ,nn)
On the other hand, prototypical prompter, se- quentially interacting with the LM, has shown im- pressive performance. Progressive Hint Prompting (Zheng et al., 2023) uses such setting; first, the LM is asked to provide a base answer. The prompt then uses the answer as a hint to the LM that pro- gressively guides it to the final answer. Zhou et al. (2023) followed a similar direction by breaking down the problem itself. Their method, Least-to- most prompting, asks the LM to generate simpler, related problems from complex problem. The final solution to the original question generated by the LM conditioned upon the solution of the sub- problems. major bottleneck then becomes the solver's ability to identify the critical subproblems. Decomposing complex task and then solving each task via multiple LLMs with their own in-context examples have been attempted as well (Dua et al., 2022; Khot al., 2023). Recently, Shridhar et al. (2022) explored subquestion generation from com- plex questions as means of distilling reasoning abil- ities from larger LMs to smaller ones.
to finally reach the desired answer to the original question. Eq. 1 can then be rewritten as,
A = arg max] I ps (A(IS) Q.) arg max (S(IQ)) (2) A.E.AA
St.
where A4 is the answer to the subproblem Q, the usual regime of CoT, the subproblems Q are implicit; the LM discovers them on its own and generates the reasoning steps and the answers. Due to the repeated multiplication in Eq. 2, any error in the initial stages quickly propagates along the chain of steps.
In DaSLaM. we seek to alleviate this problem by offloading the task of inferring the subproblems
{Q} to decomposer LM σ. For a more guided problem decomposition, DaSLaM uses the answer and the steps generated by the solver LM, 0 in the naive CoT regime as described in Eq. I to generate set of subproblems {Qi)i=
follows:
Q. arg max po (Q. : [1. (1.f-11), 1]),
Q, 2a So).
Our proposed method, DaSLaM makes a depar- ture from these mentioned approaches in three par- ticular features: (i) we seek to separate out the de- composer from the solver to get rid of the solver's limitations affecting decomposition, (ii) the decom- poser acts as plug-and-play module that can gen- eralize to any solver, and (iii) the decomposition actuates with complete knowledge of the solver's actions.
for í E [1, where Åg and So are the initial an- swer and reasoning steps, respectively generated by A. The solver LM 0 then solves the subproblem set oye-by-one similar to Eq. 2. However, instead of secking to generate the final answer to the last subproblem Qn. we append the origi- nal question at the end and let 0 answer it directly given the context generated by the subproblems, their answers, and the corresponding CoTs The four stages of workflow with DaSLaM, as described
response
'The codehase is given at: LCS2-IIITD/DaSLaM
com/

---

in Figure 1, can then be summarized follows:
problem within ø. However, it is still blind the actual errors made by the solver LM. In the next stage, we seek to make the decomposer LM work in synergy with any solver LM of choice. This solver agnosticism restrains & to observe the inter- nal representations computed by 0 while solving the problem. To handle this imposed blackbox characteristics of the solver, we resort to policy gra- dient optimization of assuming @ to be part of the environment. We formalize the setup as follows.
Ao arg max AEA
po (A]S, Q) ang max (0
50
Q = arg max Pe
Q So)
Å, ang mAx ALEAA
(o(AA)) (.) arg max po
5;
Å arg max AEA
Po(A|S,.Q)
arg max pa (S,
Elements of the environment: The solver LM 0 constitutes the core component of the environment. Given question Q, the solver-gent -eveeregeeertedd CoT S and the answer A sequences of tokens define the observation of the policy.
4 Learning to Decompose from Feedback
Typical LMs are not suitable to reliably perform the problem decomposition stage in Eq. 3. We seek to finetune the decomposer LM, & for this task. Specifically,
State and action space: We define the state space S. The initial state, so E S, is defined by the original question Q and the initial response from the solver, So, Ao. that are provided the decomposer LMM Φ as input. A single timestep is defined on generation of a single token, i.e., the action ag at time t. Given the autoregressive na- ture of L.M, we define the state at f-th timestep as S; = ( i.e., the token generated at I t- 1 appended to the tokens generated till / - 1. Trivially, the action space is the same as V. the vocabulary of db
use the LLaMA 13 billion model. Instead of full LM-tuning, we use LoRA adapters (Hu et al., 2022) for parameter-efficient training. For brevity, we will denote the adapter- augmented LM as é, although only the adapter weights are being updated while training. The whole process of teaching the LM to perform the problem decomposition comprises two successive stages: (i) subproblem construction from the origi- nal question and CoT, and (ii) policy optimization in conjunction with the solver LM. Details of the su- pervised data curation required for these three steps are described in Section Each data sample in the supervised dataset D can be conceptualized as a sequence of triplets (Q Sguld: Agold, Q' g'glaa. where Qgoid denotes the original question, Sgota de- notes reasoning steps in form of CoT, Agold denotes the answer to the question, and Q' O'gold is sequence of subproblems generated by decomposing Qgold (see Section 5 and Appendix A for further details on supervised data curation process).
Policy: The decomposer LM is conceptualized as a policy network Tа : S -> V, i.e., it generates token given the inputs and the token generated hitherto, till the end of episode T. Inspired by the recent success in Reinforcement Learning from Hu- man Feedback (RLHF) with autoregressive LMs, we choose the Proximal Policy Optimization (PPO) algorithm to train the policy TT(a)s). In a typical PPO setup, we define the advantage function as follows:
The first stage is straightforward with the decom- poser LM being finetuned using language modeling objective. In the first stage, we seck to optimize the following objective:
G. Σ (xx) freee (erirttttttn (V((014)1) ((+++))] (6)
where Ti is the reward at step I, V(+:) S is
min[- (og gold QQ
(5)
the value function determining the reward associ- ated to state 8t, and 7, À are hyperparameters. We use the policy model augmented with ramdomly initialized feedforward layer as the value function. A crucial component in on-policy learning regime is the reward function rı. While the end goal of the learning to have the solver answering correctly, the decomposer LM should receive some incremen- tal signal aligned to its generation as well for to stably converge to the optimal policy. With this
This step is somewhat similar to instruction tuning. where an LM asked to generate some text condi- tioned on a context, instead of following the usual sentence completion-style behavior of the LM. In- tuitively, the role of this stage is to align the LM to the task of the decomposer.
Decomposition via policy network. The previ- ous stage inculcates the ability to decompose a

---

goal in mind, construct the reward function as,
ously. In the first stage, we require the subproblems along with the reasoning steps for given prob- lem. We use samples from four existing datasets MATH (Hendrycks et al., 2021), AQuA (Ling et al., 2017), GSM8K (Cobbe et al., 2021), and Strate- gyQA (Geva et al., 2021). Each question in these four datasets contains question Qgold, step-by- step illustration of the reasoning process Sgold. and the final answer Agold- We sample 7, 000 examples from the training splits of these datasets and employ OpenAT's text-davinci-000 model to generate the corresponding subquestions. We provide the model with one-shot example illustrating how to decompose a question into subquestions based on the reasoning. In the second stage of training, we utilize the remaining training data from MATH and AQuA datasets to conduct the policy optimization since this step does not require any supervised ex- amples of subproblems.
Tt R1 R2 + R3 + R4 Rs (7) where R1 to Rs are defined as follows (see Ap- pendix B for detailed treatment on the reward computation method). Here cos-sim represents co- sine similarity. (xx) 1 if x true is the indicator function.
|Egrl
Entity coverage: R1 = TEol where Eq and Eq are the sets of distinct entities in the generated subproblems and the original question, respec- tively.
Consistency of answers to subproblems: R₂ Σ (1(e, = é) cos-sim(Qi, A.))
(8)
where & is the entity whose value has been asked in the subproblem Q₁ and ej is the entity an- swered. This reward penalizes the decomposer LM for generating questions whose answers are not consistent.
Order of operations: R3 L. where I is the number of operations matched order between and Sgold. and m the total number of opera- tions in Sgold-
LMs used. We use LLaMA 13 billion (Touvron et al., 2023) the decomposer LM. For the solver LM, we primarily use text-davinci-o0/ (hence- forth, we denote as GPT-3.5 for brevity). We also experiment with the LLaMA 13 bilion and LLaMA 33 billion models as solvers to test the model-agnostic generalizability of DaSLaM.
CoT proximity: To ensure that the distance of rea- soning produced by the model after prompting S to the gold reasoning Sgold is less than the distance of reasoning produced without prompt So to the gold reasoning steps Syold. we design a reward based on the cosine similarity of each step of Sgold- We break and Sy at new-line token to form reasoning steps. At each step j. we compute Cjj = cos-sim(S", SS and C2j cos-sim (. The reward is
Baselines. We compare DaSLaM with four ex- isting methods of prompting: Chain-of-thought prompting (CoT) (Wei et al., 2022), Least-to- most prompting (L2M) (Zhou et al., 2023), Progressive Hint Prompting (PHP) (Zheng et al., 2023), and, Demonstrate-Search-Predh (DSP) (Khattab et al., 2022a). The original setting of PHP requires 8-shot prompting; however, since all other methods including DaSLaM predict in the zero-shot setting. we use PHP in 1-shot for a fairer comparison. Addi- tionally, we experiment with three ablation variants: DaSLaM-NF does not take the solver feedback into account while generating the subproblems; Fine- tuned is the solver LM (LLaMM 13B in this case, could not finetune 33B variant due to compu- tational constraints) finetuned without any decom- poser: GPT-3.5 decomposer does away with the finetuned LLaMA 13B decomposer and uses pre- trained GPT-3.5 as the prompted decomposer.
R4 Ilas > essee Iless cu)(-11 cas),
(9)
Correctness of final answer: Rs R==I(A=A Agold)
Now, we can define the PPO objective as follows:
max E. (as(s)) (a.)) Ge JE, [K.]
(10)
where Teef is the reference model that is ini- tialized with supervised finetuned 6. K₁ = KL(TTee(-|ss) (-]+s)) is the KL-divergence be- tween the reference model and the policy model.
The resulting decomposer LM & optimized using the above mentioned three stages of finetuning can then be used with DaSLaM.
Test datasets. For evaluation purposes, use three datasets MATH (Hendrycks al., 2021), AQuA (Ling et al., 2017), and JEEBench (Arora et al., 2023). For the first two datasets, only the test splits are used during evaluation since their
5 Experiments
Training data curation. The training process of DaSLaM consists of two stages as mentioned previ-

---

Method
Dataset
CoT
L2M
РHP
DSP
GPT3.5 Decomposer
DaSLaM-NF
DaSLaM aM
PnC
16.4
16.0
10.2
16.2
16.0
20.0
21.4
NT
14.4
11.0
9.8
20.3
14.2
18.4
26.1
ALG
27.6
22.1
24.0
15.3
32.1
31.6
33.4
I-ALG
16.4
16.8
10.0
17.0
18.4
20.8
24.8
Calc.
14.0
14.58
14.28
18.8
12.0
15.1
18.2
P-ALG
32.3
28.0
26.5
28.0
35.5
38.0
44.0
Geom.
14.2
12.5
14.0
5.2
22.0
19.01
21.4
AQuA
41.6
14.7
11.4
41.0
45.4
53.2
54.5
Table Performance comparison MATH and AQuA datasets using GPT-3.5 as the solver LM. See Section 5 for abbreviations.
training splits are used while finetuning the decom- poser. The MATH dataset contains mathematical problems on multiple different domains. We report the results on each of them separately and use the following abbreviations ALG, I-ALG, and P- ALG for Algebra, Intermediate Algebra, and Pre- Algebra, respectively; Calc for Calculus, Geom for Geometry, PnC for Probability and Combina- torics, NT for Number theory. From the JEEBench dataset, we use the problems in Physics (Phy) and Mathematics (Math). Each of these two sub- jects has three types problems- - single-answer multiple-choice questions (MCQ), numerical prob- lems (Num), and multi-answer multiple-choice questions (Multi). For all these datasets, we use exact match criteria to evaluate the correctness of the model-inferree answers. Details of training and inference hyperparameters and compute resource usage are provided in Appendix C.
stronger across tasks containing algebraic reason- ing (AQuA, Pre- and Intermediate- Algebra, etc.) compared Probability and Combinatorics or Cal- culus, which require more implicit knowledge. The performance gain achieved via DaSLaM aM is signif- icantly better compared to methods like L2M or РHP. The latter methods often fail to improve over standard CoT (e.g., on Probability and combina- tories, Number Theory, and Algebra problems, L2M shows drop in accuracy). Even when im- proving over CoT, their improvement is meager compared to DaSLaM. This trend entails our earlier argument in support of offloading the problem de- composition task to a specialized LM;; methods that prompt the solver LM to decompose the problem lack the expertise achieved via dedicated finetuning in DaSLaM.
Finetuned decomposer is essential. Despite being orders of magnitude smaller, finetuned LLaMA 13B model delivers better performance compared to GPT-3.5
6 Experimental Results
decomposer (DaSLaM vs. GPT-3.5 generator in Table and 2). This fur- ther justifies our choice of separately finetuning the decomposer and the added flexibility that it offers. In fact, finetuning the decomposer is far effective compared to finctuning the solver (DaSLaM vs Finetuned solver in Table 2).
The tasks used to evaluate the performance of DaSLaM contain questions that can be answered either of the three types numerical, single correct answer MCQ, and multiple correct answer MCQ. DaSLaM is better than pure prompting We start with DaSLaM augmented with GPT-3.5 as the solver LM on MATH and AQuA datasets (see Table 1). The improvement achieved with DaSLaM prompting compared to standard CoT is staggering across all types of problems in the MATH dataset: +11.7 on Pre-Algebra, +8.4 Intermediate Algebra, +7.7 Number Theory, +72 on Geometry, +5.0 on Probability and Combinatorics, +5.8 on Algebra, and 442 on Calculus. The absolute improvement is even larger on the AQuA dataset, i.c., +12.9 over COT. is noticeable that the effects of DaSLaM are
Feedback from the solver is important. the preceding paragraph, we attributed the superi- ority of DaSLaM over other methods to the usage of specialized LM for problem decomposition. However, manipulating the problem decomposi- tion upon feedback from the solver is also an im- portant factor here. None of the existing methods does so, and therefore, remains blind towards what reasoning (and possible errors) is followed by the solver model. This further manifested when we

---

Datasct
variants. DaSLaM provides consistent improvement Table 2: Performance on
Method
PnC
NT
ALG
TAGG
Geom
Cal
Palg
AQuA
MATH and AQuA with
LLaMA billion
LLaMA 13 billion and
CoT
2.05
4.0
3.12
2.4
3.2
2.08
5.0
17.7
LLaMA billion solvers.
L2M
1.66
3.2
3.33
2.8
2.0
3.33
4.54
16.6
Finetuned
2.8
3.6
3.57
3.2
4.1
3.05
6.04
19.4
PHP is not reported
GPT3.5 Decomposer
2.05
5.0
4.68
2.8
2.08
4.0
6.66
20.4
one-shot PHP generated
DaSLaM-NF
2.93
4.8
4.68
3.2
4.0
3.9
6.2
21.6
randomly with both LLaMA
DaSLaM
4.0
5.6
4.70
3,4
4.3
4.1
8.33
22.0
LLaMA 33
billion
CoT
2.4
4.16
4.54
3.7
4.0
4.0
5.2
20.0
across all the tasks while
L2M
2.38
4.16
4.2
6.0
4.25
5.71
5.55
21.6
DaSLaM-NF
3.2
5.83
5.6
5.6
5.1
5.71
5.2
22.5
other baseline methods
DaSLaM
4.0
7.36
9.09
6.02
5.3
6.03
8.44
26.8
mostly fail.
Datasct Multi
Method
Phy MCQ
MCQ
Phy Multi
Phy Num
Num
Phy
Int
CoT
33.33
21.9
6.25
12.0
3.03
1.69
12.5
20.0
РHР
22.22
17.07
6.25
7.59
3,03
1.69
0*
4.0
L2M
22.22
21.9
6,25
12.5
3.03
3.38
10.0
20.0
DaSLaM-NF
20.8
31.7
7.5
10.12
3.03
3.38
12.5
16.0
DaSLaM
55.55
36.5
18.75
16.0
6.06
10.16
22.5
24.0
GPT4
55.35
34.14
27.5
21,5
13.15
11.8
25.0
20.0
Table Performance comparison on the JEE benchmark dataset with GPT-3.5 as the solver I.M. 0* signifies that the model was not able to answer any problem in the task correctly.
compare DaSLaM with itself without the feedback module, DaSLaM-NF, While DaSLaM-NF is able to improve upon basic CoT and other prompting meth- ods, it falls short a decomposer LM that has access to the initial response of the solver.
basic CoT prompting, often with sharp fall in per- formance (c.g., Physics MCQ questions). On the other hand, DaSLaM boosts the LMs performance, very often over 100% relative improvement (all three types problems in Physics and numeri- cal problems in Mathematics). Aggregated across question types, DaSLaM boosts the performance of GPT-3.5 to 22.420 in Physics and 22. 22.07 in Math- ematics. It is noteworthy that the same LM in its base setting performs near-random, i.e., 10.4 and 10.7 in Physics and Mathematics, respectively, whereas a random selection baseline gives scores of 9.6 and 10.3, respectively (Arora et al., 2023). Furthermore, GPT-3.5 with DaSLaM outperforms a better optimized candidate of the GPT series, GPT- 3.5 on both these subjects (note that Arora et al. (2023) reported 18.9 and 15.7 scores with GPT-3.5 Physics and Mathematics, respectively). Com- parison with GPT-4. The colossal compute used by GPT-4 makes the comparison with any of its predecessors like GPT-3.5 quite unfair. However, is tempting to observe that DaSLaM boosts the performance of GPT-3.5 often to the level of GPT- 4. For example, on arithmetic problems of the AQuA dataset, DaSLaM surprisingly outperforms both zero-shot and few-shot GPT-4 (40.6 and 50.4 respectively, compared to 54.5 with GPT-3.5 and DaSLaM), On MATH dataset, DaSLaM augmented GPT-3.5 scores an aggregate of 30.23, which is better than ChatGPT (26.4) and close to GPT-4 (35.7). On JEEBench Mathematics problems, GPT-
DaSLaM generalizes to smaller solvers. An im- portant aspect of a prompting method is its ability to work with LMS of different scales. Despite be- ing finetuned with GPT-3.5 responses only, DaSLaM is able improve upon the base performance of smaller scale LLaMA models as well (see Table 2). L.2M prompting generally fails with both LLaMA 13 billion and 33 billion variants. On the other hand, DaSLaM, with or without feedback, almost doubles the performance of the base CoT across multiple tasks of the MATH dataset. It shows substantial improvement AQuA well. The importance of feedback from the solver LM usually manifests strongly in proportion the scale of the solver.
DaSLaM generalizes harder problems. Since the decomposer LM é is trained using a subset of the training data of MATH and AQuA, we opt for a harder (in terms of benchmark performance of dif- ferent LMs) reasoning evaluation on the JEEBench dataset. Table 3 summarizes the performance of the baselines and DaSLaM with GPT-3.5 as the solver LM Mathematics and Physics questions of the JBEBench dataset. We notice that the superiority DaSLaM manifests even more profoundly on this task compared to the former ones. Both PHP and L2M prompting absolutely fail to improve upon

---

gragh the eggattim ax? emppleslly bolow the if
CoT
Thi graph of equation o fi umpplerly helow the poacis This meaze that the equation should have two real rouse.
the lorpeet possible tntegral value
Theccoore, the deciiiiaan of the equatson should be stentee thau or oqual to
Laaect
-to
DrSLaM
The dacrimmann of the equatioe is givnn by P dec. Subettonngg the vinsee of and i, we get 3f 45oPy( 6 591
dare 5 896 2
This impla that à y-s
Past, need sudeustand, the equation or"
What be the equation?
Since à intogre, the lorper possible Irtegral vaher of -30
Nest,
possible intogral
of
valuu aquaitie:
What equation of the line?
Now, we caa dur by sestingg Ea
The equazion the line y mr =.
Stnce the groph is completely below the x-oats, the equation of the line will be S.S. D
b- 4hs
Thaesione, the equation of the lisw is S
Since is un inlogge, we cas aas i 1 to get the lurgesi posstble integpai vadue ot t:
What se the value of o?
I We know than of ab Therefoee, a v a. The vilto of 7.
b- --0- - -55
What the value of 57
Taceedoo the largest possilue integua value of is -5.. What is the value of a?
The graph the equation ouf ix 6 completdy below the
This that
should have should be argation
Now, let conoidve the equarine ard Ar completely below the equation must hove ceal.
This inplis the diseriiinnt the
For the graph
the eqpation gomn by 8
Ieen o,
Thendser,, the diveriminant Ein eguatim hn oegation Discommtint Joc
So. y 4se shonld negattioe
Since 48, a 477.
Therefoe, 4/TH 6) 0
Sn. 4P Sic
<0 PPc -M6
This tupirs thus 6f 198
What the largest possiblle value of 57 the lorgee pooeclle integral vili of b 12.
Thrrefoee, the lragest posahle tutegral value is
The grophi of the equation = exd Ar 6 is completely below the
The graph of the equation y no* - by - is completely below the s-axts. itc" 40, what is the largest passible intogral value of 37 The graph Lha equation
s-aais, If o² - -1,, what is the bargest possible integral value of 67 Sieos o² 09, wn can mistristte this vnlto ioto the equatton
ou? i 6 completaly bakow the z-ooio. This that the equaton should lave nol roote
00x² bx--
Now, can sulve for 5 by willing 1
This impire that the distrautenn the by 5
should be uogatiro.
0= Mtt*+ be -i
discrinimat the equition Hean
b-
3 dae shonid negatton. Sluce 48, a 27.
integer,
gye the largest poseibin integral
So Sic
b- 10
This impitee
Tharefeer, the lagont poutble integral valun -56.
So. the largest possible lncegral value of is
Figure 2: An example case study on a problem from the MATH dataset. GPT-3.5 is used as the solver LM with three different methods of prompting standard CoT, Least-to-most, and DaSLaM. Only DaSLaM is able to guide the model to the correct answer.
4 comes up with an aggregate score of 23.1, which is pretty close to our 22.42. In Physics and Math MCQ questions. aM with GPT-3.5 outperforms GPT-4. These results definitely do not claim any assumed superiority of DaSL boooste GPT-3.5 over GPT-4 since there are multiple other cases that state otherwise. Instead, we seek to demonstrate how much leftover potential these LMs possess that can be unleashed via our proposed method of feedback-guidee automatic problem decomposi- tion.
swering incorrectly. With CoT, the solver wrongly assumes that the given equation must have two real roots though it should not have any real roots. Also. it mistakes the value of a² as The effect is promi- nent in the subproblems generated by DaSLaM as it asks find the value of a explicitly. Furthermore. the solver LM specifically announces that y S 0 to answer the first subproblem generated by DaSLaM. This helps to correct the reasoning about the sign of the discriminant.
With L2M, the confusion around the value of a and a persists, as the solver LM substitutes a in the given equation by 49 (which is the value of a²) twice in the answering process. Although it substi- tuted the correct value of a once while answering the second question, it is not explicitly declared like in DaSLaM. We observe multiple similar failure cases with L2M. It is quite likely that prompting the model to generate the final answer after each subproblem accumulates the erroneous reasoning steps that the model falls prey to.
7 Case Study
To this point, we have compared the numbers pro- duced by DaSLaM-boosted models across different datasets. While they provide an overall assess- ment, deeper analyses are needed to comprehend the actual reasoning steps adopted by these differ- ent methods. Figure shows the reasoning steps generated by GPT-3.5 given an example problem from the MATH dataset with three different prompt- ing methods vanilla CoT, L2M, and DaSLaM. Note that DaSLaM uses the CoT output to decompose the problem.
With DaSLaM, the reasoning steps followed by the solver remains robust throughout. It reaches the final answer much earlier (third and fourth sub- problems). In the final answer, the solver simply
Both CoT and L2M end up with the model an-

---

reiterates the steps that it earlier generated an- swer the subproblems. This a common behavior that we observed across multiple problems from multiple datasets. In Appendix D (see Figures and 4), we provide similar case studies on LLaMA 13B and 33B with different prompting methods. With reduced solver capacity, the difference be- tween Least-to-most and CoT generated reasoning steps further diminishes with both leading to in- correct answers; DaSLaM, on the other hand, still guides the solver through correct steps.
Limitations
A potential limitation of DaSLaM, with many sys- tem that uses an LLM-as-a-service API charging per token exchange, is the increased token usage because of the RL exploration. Asserting a token budget on the decomposer L.M is left as an avenue for future exploration. Ideally, the decomposer LM should seamlessly invoke solvers of many forms, such as retrievers (Khattab et al., 2022b) or mathe- matical calculators (Schick et al., 2023; Wolfram, 2023). Future work may extend DaSLaM such tools. DaSLaM is limited to purely text-based sub- problem decomposition; it not possible at present to incorporate reasoning through other modalities (e.g., visual inputs for geometric reasoning) into DaSLaM in its current form.
An interesting observation can be made by com- paring how the solver behaves with CoT vs. with DaSLaM. With DaSLaM, do not provide any new knowledge the solver. Yet, the same model can rectify its errors made CoT response. This may point to the intuition that current LLMMs are actu- ally underutilized, and one can unfold even more impressive performance with cleverly composed guidance.
Acknowledgments
The authors acknowledge the financial support of DYSL-AI.
We further provide case analyses with when DaSLaM fails to guide the model (GPT 3.5 in this case) to successful final answers, Appendix While we do not find any obvious pattern of er- rors, one can see that the decomposer generates questions that are not readily answerable within that context. DaSLaM does not use any method to trace back the елгог generate subproblems based the answers to the previous subproblems. This might raise such issues where the subproblems gen- erated are not actually helping the solver in the right order.
References
Daman Arora, Himanshu Gaurav Singh, and Mausam. 2023. Have Ilms advanced enough? a challenging problem solving benchmark for large language mod- cls.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchrr Nakano, et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:21 1et::21101.1166 14168.
Jacob Devlin, Ming-Wer Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
8 Conclusion
We challenged the design of ever-larger monolithic LLMs homogeneous network structures, where diverse aspects of problem decomposition and solu- tion are stored in a tangled and opaque manner. The formidable general-purpose problem-solving capa- bilities of LLMs are exceedingly resource-hungry. dependent on immense data engineering. Inspired by brain science, we took first step toward hetero- geneity - let two different L.LMs evolve indepen- dently and adapt to their roles of decomposing and solving complex reasoning problems. Through ex- tensive experiments on several benchmarks, showed that such heterogeneous network can match or exceed some of the largest contemporary LLMs, at a much smaller parameter count.
Payal Dhar. 2020. The carbon impact of artificial intel- ligence. Nature Machine Intelligence, 2(8):4334255.
Dheeru Dua, Shivanshu Gupta, Sameer Singh, and Matt Gardner. 2022. Successive Prompting for Decompos- ing Complex Questions. Empirical Methods Natural Language Processing (EMNI.).
Evelina Fedorenko and Rosemary A. Varley. 2016. Lan- guage and thought are not the same thing: evidence from neuroimaging and neurological patients. An- nals of the New York Academy of Sciences, 1369.

---

Guhao Feng, Bohang Zhang. Yuntian Gu, Haotian Ye. Di He, and Liwei Wang. 2023. Towards revealing the mystery behind chain of thought: a theoretical perspective.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21(1).
Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions the Association for Computational Linguistics, 9:346- 361.
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke 7ettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint 2rXiv:2302.04777
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS.
Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. 2022. Distilling multi-step reasoning ca- pabilities of large language models into smaller mod- els via semantic decompositions. arXiv preprint arXiv:2212.00193.
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An ceerrcctoo-followiinn Ilama model. https:// gi thub. com/ conn/tatsu-lab/ wwwtsaaaaasstaarrrrrrrrrrr stanfor_aalpaca.
Omar Khattab, Keshav Santhamam, Xiang Lisa Li, David Hall, Percy Liang. Christopher Potts, and Matei Zaharia. 2022a. Demonstrate-search- prediet: Composing retrieval and language mod- els for knowledge-intensivy nip. arXiv preprint arXiv:2212.14024
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. 2022b. Demonstrate-seand predict: Composing retrieval and language mod- els for knowledge-intensive NLP, arXiv preprint arXiv:2212.14024.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi. Quoe V Le, and Denny Zhou. 2022. Chain of thought prompt- ing elicits reasoning large language models. In Advances in Neural Information Processing Systems.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sab- harwal. 2023. Decomposed prompting: A modular approach for solving complex tasks. In The Eleventh International Conference on Learning Representa- tions.
Stephen Wolfram. 2023. ChatGPT gets its "Wolfram superpowers"! (blog).
Yunhu Ye. Binyuan Hui, Min Yang, Binhua Li, Fei Huang. and Yongbin Li. 2023. Large language models are versatile decomposers: Decompose evi- dence and questions for table-based reasoning. arXiv preprint arXiv:2301.13800
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blun- som. 2017. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158-167, Vancouver, Canada. Association for Computational Linguistics.
Chuanyang Zheng, Zhengying Liu, Enze Xie, Zhenguo Li, and Yu Li. 2023. Progressive-hint prompting improves reasoning large language models.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei. Nathan Scales, Xuezhi Wang. Dale Schuurmans. Claire Cui, Olivier Bousquet, Quoe V Le, and Ed H. Chi. 2023. Least-to-most prompting enables com- plex reasoning in large language models. In The Eleventh International Conference Learning Rep- resentations.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances Neural Information Processing Systems, 35:2507-2521.
Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko. 2023. Dissociating language and thought large language models: cognitive perspective.
OpenAI. 2023. Gpt-4 technical report.

---

Small Language Models Fine-tuned to Coordinate Larger Language Models improve Complex Reasoning (Appendix)
A Supervised Fine-tuning Dataset
Entity coverage Reward (R1). Here, we find distinct nouns and numbers the question and the sub-questions using the nltk library. The reward calculated using R
The training data for supervised fine-tuning stage was generated using text-davinci-003. Each data point in the dataset consisted of tuple (Quold, guld: Q' goga), where Qgold represents the reasoning question, Sgold represents the gold rea- soning steps, and Q' gold represent the sub-questions generated by text-davinci-00 in a one-shot set- ting. An example of a single data point is given below.
Exrl where Eqy and EEq
Eq are the sets of distinct nouns and numbers in the sub-questions and questions, respectively, An example of the same is shown below.
Quola:
Each good worker can paint my new house alone in hours, Each bad worker can paint my house alone in 36
12
Qgoid
hours, need my house painted in hours. If can only
find good workers, how many bad workers must also
John borrowed 3 soccer boots from Jake, and forgot them the field. if peter came across total of 15 boots on the field Jake's boots inclusive, and he took 4 boots at random, what is the probability that Jake's boots were not amongst 12 the 4 taken?. (A) 01 (B) is (C) 12 (D) 53
find order have my house painted time?
Quiii
1. How many good workers are needed to paint the house in hours?
91 (E)
15
2. How many bad workers are needed to paint the house
Sgold
in hours?
Since Jake owns 3 of the boots, the subset from which the 4 boots should he chosen are the 12 honts not owned Jake from the universe of 15eThe first boot can be one of the 12 from the 15 with prohability 12. The second boot can he one of the from the remaining with probability 11 The third boot can be one of the from the 13 remaining with probability 10
What the total number workers needed paint the house
hours?
Mgg 4 +
(11)
Eg
= 6
3
(12)
13 The fourth boot can be one of the 9 from the 12 remaining with probability 9. The total probability will 11
Eq'
5
Ri
0.55
(13)
|Eel
9
be 12
10 9
33 On cancellation, this comes to
T5
T4
T3
Consistency of answers to subproblems re- ward (R2). To ensure that the sub-question is such that the solver LLM & does not produce random answers,
Qgold
How many boots
own?
compare the entity in the solution the entity asked in the subquestion along with the cosine similarity of the embed- dings sub-question and sub-answers. To find the embeddings, we use the pre-trained model in the spacy library. Ro = (I((e = 6) + cosine-similarity(C. (Q/, A;)), where (xx) = if true, é is entity whose value has been asked in the subproblem Q/ and is the entity answered. An example for the reward calculation is given below.
2. How many boots were on the field?
3. How many boots did Peter take?
4. What is the probability choosing one of the 12 hoots
not owned by Jake from the universe
5. What is the probability of choosing the second, third and fourth boots not owned by Jake?
6. What is the total probability?
B Reward Calculation
During policy gradient optimization of the decom- poser LM é, rewards were provided in incremen- tal fashion that the generated sub-questions align well with the initial question, the answer generated by the solver LM and the sub-question solving capabilities of For this, the reward was designed as combination of five terms. The definitions of these terms are given in Section 4. Here we provide an example of the reward calculation.

---

Qí: How many good workers are needed to paint the house
Syold
in 3 hours?
We first consider the possible neighbors of the chief's mother, one must be the chief. the other is one the remaining natives. That native can. turn, have one of neighbors in addition to the chief's mother
Aı: We can calculate the amount of work done in hours by good worker by dividing 12 hours hy 3 hours. We get 4 as the answer. We know that bad worker can paint the house in 36 hours. We can calculate the amount of work done in 3 hours by bad worker by dividing 36 hours by 3 hours. We get 12 as the answer. We need to find out how many good workers We can calculate the number of good workers needed by dividing 12 by 4.
Continuing around, there are = 120 possible ar- rangements
needed to paint the house in 3 hours.
The chicf can on his mother's s left or right, for a total of 120 240 arrangements
4. Alternatively, we know that the seating arrangement is the same if is rotated, once the chief's ' spot is chosen, the rotations are eliminated
Answer: good workers can paint the house in 3 hours. cos-sim( Q A:) 0.70457435
I(e. 6.)
5. There are 2 ways to seat his wife and his mother, and
R₂ 0.70437435 1.70457435
then there are 5 distinct positions for the remaining
5 natives, which makes for arrangements
240 possible
Order of operations reward (Ra). To evaluate the correctness of the mathematical solution pro- duced by the model, we compare the order of opera- tions. R3 _, where / is the number of operations matched in order between S and Sguld, and is the total number of operations in Syold. Below is an example of R3 calculation.
So:
We can see that chicf must seated between his mother and his wife.
This means there are 3 possible positions for the chief, and possible positions for the other people. Therefore, there 3! 51 1800 different scating arrangements.
S:
The chief must between his mother wife. There are 8 people the tribe. So. the chief's mother and wife can arranged in ways.
Smode!
Original Price
Discounted Price 0.78x
The remaining people can arranged ways.
Price after coupon 0.78x 20
Therefore, total number of possible seating ar-
Price paid= 0.78x 20 1.90
rangements after removing rotation 5! 240
0.78x 20 1.90 0.5x
Cosine similarity calculation per step:
0.28x 21_90
1. Ciş 0..17633503, caj 0.14773823
78.21
Cıj 0.45063934, C2j 0..4991884
Spaa Let be the original price of item
Cij 0.5173945, C2j 0.20383501
Discounted price 0.78x
Cι 0.46866685, C2
Payment made by the customer after using the coupon
6i 0.47825924, e2j
0.788x 20
Hence, R4 0.47673503 + (-1 0.47917843)
0.78x 20 x/2 1.9
0.5173945 0.46866685 + 0.47825924 0.46187719
x = 78.20
operations Swotd: +. operations in Spast [.+
Correctness of final answer(Rs). R3 = T(Á Agold). This reward checks if the final answer matches the gold answer. negative example the same is given below.
Rs = 0.75
COT Proximity reward (R4). As defined in Sec- = (C1 > c2j)c1 + I(ezj > tion 4, R4 = =
c1y)(-1 - c2j). where T(.) is the indicator function. Here present an example to illustrate the same.

---

D Case Study on LLAMA Models
Quoidi
Three friends Alan, Roger and attempt answer question on an exam. Alan randomly guesses the answer, giving him a probability cheats by looking at the paper of student front of him, giving him & probability of answering conectly. And Peter dutifully performs giving him a probability of a correct answer. What the prohability that the question is answered correctly, but not via cheating?
Similar to Figure 2, we performed example-wise case study on the answering process of the LLaMA solvers (13 billion and 33 billion models) with dif- ferent prompting methods (see Figures 3 and 4), The example problem was selected from the MATH dataset. As can see, with relatively small scale solvers like LLaMA, Least-to-most fails to add any improvement over basic CoT prompting. DaSLaM able to guide the model to the correct answer both cases.
guessing correctly. Roger
calculations, then marks the answer,
Syaud
Prob(Alan) =
Prob(Roger) without cheating 11 Prob(Peter) = 6
Total Probability
Smodd
Alan has a & chance of getting the answer correct. Roger has chance of getting the answer correct.
Peter has a chance of getting the answer correct.
The probahility that the question is answered correctly 1 + + 13-
13
The probability that the question is answered correctly, but
not via cheating isl (
12
Agota:
Amada:
Rs 0
C Hyperparameter Selection
We performed hyperparameter tuning on a subset of 250 examples from the training dataset. Based on the improvement in accuracy and compute re- sources, we converged at the following values.
For the supervised fine-tuming stage, used LoRA 16, LoRA a 32, LoRA droput 0.05. For the RLMF stage, we finetuned the last 3 layers LoRA adapters, using batch size 8, gradient accumulation steps=8, init coef=0.01, target=4. For inference, we used the following generation parameters: temperature = 0.95, top p=0.18, 18, pad token id = 0, do sample = False, number of beams
= 1. maxi length 2048.
All the models were implemented using Hug- gingface with PyTorch, and loaded in int_8 preci- sion. For solver-L.LaMA 13B model, we instruc- tion finetuned the LLaMA13B model using the Alpaca dataset by stanford (Taori et al., 2023). For solver-LLaMA 33 billion model, we used an instruction-finetuned version of LLLAM--3 billion from Huggingface. To implement the PPO algo- rithm, we used the TRL library. We used single NVIDIA A 100 (80GB) GPU for all training and inference purposes.

---

What the greatest
and 18 The graated commonn dicisor of two numbers the largest number that divides both of them. Therefore, the greatest disisor 800
minges positive intepees, what the maximum possible value that the greatest common divisor of 80m and 18n
186
largst number
that divides both of them. The largest number that divides both of them is the largest number that divides Slli 49 and divides
Least in Most
largest that dividlz Siln 48 is 800 40 isell. The largesi number that diviles 18n I1 188 II iselff Thereforee the greates common divisar 18h + themeeles..
largest number
11 can take?
18o 11 8ls 40 and
DaSLaM
COT
What is the greatesst common divisoe 8f and 2n +
The greatest common divisor of two numbers is the largest number divides common divisor of 80n - and 180++ 11 the largest number that divides both of them. The largest number that divides both
greatest common divisar
greasest common divisor
them. Therefore. greatest
and Sn. 1.3 5- 2n I fii
The greatest common divisor of So and +
divisor
larges number that divides
The greatest common divisor of Sn 5 and 2 + is the greasest common divisor of 8i + 2n 20
the largest numher that divides 18ss 11. The lugest namber that divides Sinn 40 is 80n 49 iself. The largest number that cammon divisor of Stn 40 and 18o 11 is 8ls 40 and 18n 11 themsolves.
The grodied common divisor Sm 6 and
the greatest common divisor 86
greatest
and 2n +2 2n -1 == 1.
Henee, gressest
divisor
+ and 2n
What the greatest common dis divisur of 18x
- 57
The greatest common divisar of 18o 11 and 80 5 is the greatest common divisor of and
greatest
divisor
1in 6 So - 5 2n +
What is the greatest common divisor of 8ite 29 and 186
11?
greatest common divisor
+ greatest common
18m + and 80s + 40 186. 11 62n 38is greatidd common disisoe of 188 11 and
626--88188-- 18-111 64 greated divisor of 18a-11 and 6lt 18a-11
greatest common divisor
and
18a
11
Figure 3: A case study on I.LMM-I3B.
The product of three consecutive integers is 120. That product divided by the mean of the three integers is 24. The mean of of three integers divided by mean of the three integers is 40. The product of the three integers is 120, the product of the three integers divided by the mean of the three integers is This means that the largest of the three consecutive integers is 40.
The product three consec- usive integers 120. product divided by the mean of the three integers is 24. What is the largest of the three consecutive integers?
integers
Least Most
The sum of the three integers is 120. Therefore,
COT
DaSLaM
The product of three consecutive integers 120.
What is the product of three consecutive Integers?
That product divided by the mean of the integers is 24.
The product of three consecutive integers 120.
The of the three integers is the sum of the three integers divided by 3.
What is the mean of the three integers?
The product of three consecutive integers is 120.
The sum of the three integers is 120.
That product divided by the mean of the three integens is 24.
Therefore, the mean of the three integers is 40.
So, 120/mean 24
lurgest three consecurive integers is 40 64.
Mean 120/24
What is the largest of the three consecutive integers?
The product of three consecutive integers is 120.
That product divided by the mean of the three integers is 24.
So, 120/nean =
Mean 120/24
So, the three consecutive integers 5,
The largest of the three consecutive integers is
Figure 4: A case study on LLaMA-33B as solver model.

---

E
Failure cases with DaSLaM
Sub problem 3 What is the time taken cover the distance?
E.1
Case
Sub Answer
Question A train running at speed of 100 miles/hour, takes 10 hours to reach its destination. After covering quarter of the distance, it starts raining and the train has to be slowed to speed of 75 miles/hour. What is the total journey duration? A)10 B)11.5 C)12.5 D)13.5 E)15
Time taken to cover the first quarter of the distance (d/4)/100 hours d/400 hours
Time taken to cover the remaining distance = (3d/4/75 hours d/300 hours
Total time taken d/400 d/300 (5d 4dy/1 200 hours 9d/ 200 hours.
Putting d=1000,
Initial Answer The train covers the first quarter of the distance at speed of 100 miles/hour.
Final answer: The time taken cover the dis- tance x 1000/1200 hours.
So, the time taken to cover the first quarter of the distance (1/4) 10 hours 2.5 hours
Sub problem What is the total duration of the journey?
The train covers the remaining three quarters of the distance at speed of 75 miles/hour.
Sub Answer 4 Time taken to cover the first quarter of the distance (d/4)/100 hours d/400 hours
So, the time taken to cover the remaining three quarters the distance = (3/4) 10 hours = 7.5 hours
Time taken cover the remaining three quarters of the distance (3d/4)/75 hours 4d/300 hours
Total time taken d/400 4d/300 (5d 4d)/(400 + 300) 9d/700 hours Total duration of the journey = 9d/700 hours 9 hours and (d/700) hours = 9 hours and (100/700) hours = 9 hours and 14.29 minutes
Therefore, the total journey duration = 2.5 7.5 10 hours
Sub problem What is the total distance to be covered?
Final Answer Step 1: The train covers the first quarter of the journey at speed of 100 miles/hour. Step 2: The time taken to cover the first quarter of the journey is given by: Time Distance/Speed Time (1/4) x 100/100 Time = 1/4 hour Step 3: The train covers the remaining three quarters of the journey at a speed of 75 miles/hour. Step 4: The time taken to cover the remaining three quarters of the journey is given by: Time Distance/SSeed Time (3/4) 75/100 Time 3/4 hour Step 5: The total duration of the journey the sum of the time taken to cover the first quarter and the remaining three quarters of the journey. Total Time = 1/4 3/4 Total Time = 4/4 Total Time = hour Therefore, the total duration the journey hour.
Sub Answer Let the total distance to be covered be
Time taken to cover the first quarter of the distance
= (1/4)d/100 = 10/400 hours
Time taken to cover the remaining distance (3/4)d/75 300/400 hours
Total time taken 10/400 300/400 10/4 hours
2.5 hours Since the train takes 10 hours to reach its destina- tion,
Total distance to be covered = 10 x 100 = 1000 miles
Sub problem 2
What is the speed of the train?
Sub Answer 2
After covering quarter of the distance, the train has to be slowed to speed of 75 miles/hour.
Let the remaining distance be x. Therefore, x d (d/4) 1000 (1000/4) = 750 miles.
The time taken to cover the remaining distance 750 miles speed of 75 miles/hour (750/75) hours 10 hours.
Therefore, the speed of the train x/time taken
= 750/10 = 75 miles/hour.

---

E.2 Case II
E.3 Case III
Question Consider there an staircase elevator and you are coming down. you walk 20 steps and stop, then you reach bottom in 10 minutes. If you walk 10 steps and stop, you reach to the ground in 20 minutes. What is the speed of the elevator? A)T step/minute B)2 step/minute C)3 step/minute Dy4 step/minute E)None of the above Initial Answer
Question Let < 0 <
Suppose an and 1
13 2x sec = 0
are the roots of the equation and 02 and $2 are the roots of the equation x²² + 2x tan 0 - = 0. If on > 31 and 02 > 32, then
0n + B2 equals
(A) (scc 0 tan
(B) 2 sec 0
(C) 2 tan A
Sub Question What the total number of steps
(D) 0
in the elevator?
Initial Answer
We can solve the equations for alphai and alpha2
Sub Question What is the speed of the elevator?
to get
alpha1 sec + tan 0
Sub Question 3
alpha2 sec tan 0 Since 01 and 0,232, we have sec lan 0 Adding the two equations, we get scc > 0
What is the time taken cover 20 steps?
Sub Answer 3
sec tan
Time taken to cover 20 steps is minutes
sec + tan
Sub Question
What is the time taken to cover 10 steps?
Sub Answer 4
Therefore, 01 = 2ec 0.
Time taken to cover 10 steps is 20 minutes Final Answer Let the speed of the elevator be Time taken to reach the bottom when walking 20
The correct answer is (B) 2 sec A. What is the equation of the line x² 2x sec 0 +1 0?
steps and stopping 10 minutes
Given, - < < - We need find the equation of the line ...2 2x see s +1 We know that the equation of a line given by y = me Now, let us substitute the given equation in the equation of a line, x² 2x sec 0 + -
Time taken reach the bottom when walking 10 steps and stopping 20 minutes Therefore,
20x = 10
10x 20 x Therefore, the speed of the elevator is 2 step- s/minute.
m. +
x² 2x sec 0 ma x² + 2x see 0 -
me
=0
mæ x²2 + 2r sec 0 ==
Comparing the coefficients of æ on both sides,
m sec 0
Comparing the constant terms on both sides,
= Therefore, the equation of the line x² 2x sec O + 1 = 0 is given by 2sec Or
Hence, the answer is y 2sec fr + 1.
What is the equation of the line x2 + 2x tan 0 1 0?
Given, Ι < 12
We know that,
sec 0 rox
7 tan
sin nos
Therefore,
x² 2x sec + 2 2t
cos
x² + 2x tan 9 - = _22
2.
sin 8
-
0

---

Hence, the equation of the line x² +20 tan 0-1 is x² + 2xr sin
on 31 and 02 > 32
O.
We need to find the value of a1 + 32
cos
Final Answer: x² + 2x sin cos
1 = 0
Let us consider the equation x² 2x sec 1 = 0 a1 and B1 are the roots of the equation. Therefore,
What is the equation of the line
x² 2x scc A + = 0 minus the equation 0?
01 + 31 = - f
of the line x² 2x tan
Let us consider the equation x²2 + 2x lan 0 - = =0 02 and 32 are the roots of the equation. Therefore,
The equation of the line x²² 2x sec
is
given by x² 2x sec + 1 = 0
The equation of the line x² + 2x tan given by x² + 2x tan Subtracting the two equations, we get
is
Q2 + B2 tan 0
= 0
Now,
a1 + 32 = a1 + +2 a2 + 32 - a2 - 32
x² 2x scc f + 1 x² 2x tan A 1 0 Simplifying the equation, get
= =a 81 - 02 B2
-2 + 2 tan 0
2x(sec 0 + lan 0)
2(scc 0 tan 0)
Therefore, the equation of the line
Hence, the answer is (A) 2(soc tan 0).
x² 2x soe f = 0 minus the equation of the line x² + 2x tan Hence, the answer is 2scc + tan 8) = 0.
0 is -er((sssce 8 + tan 0) =
What is the equation of the line plus the equation
2² 2x seco
of the line x2
Given,
< <
a1 and
81 are the roots of the equation
x² 2x sec
+ = 0
02 and 32 are the roots of the equation
2 + 2x tan - -
> 31 and 02 32
We need to find the value of 82
Let consider the equation x² 2z sec 0 + 1 a1 and 11 are the roots of the equation. Therefore.
a1 + 81 = scc f
Let us consider the equation x 2x tan -
02 and B3 are the roots of the equation.
Therefore,
a2 + B2 = 2 tan 0
Now.
01 $2 01 + 32
02 + $2 02 B3
=a1+3 31 02 32
-e2 8 + 2 tan 0
2(soc 0 tan B)
Hence, the answer is (A) 2(sec tan 8).
Final Answer Given,
< 0 <
a1 and 31 the roots of the equation 22
2x sec 0 -1= 0
02 and 82 are the roots of the equation 2
2x 0 =0


Reviews:
[{"review_id": "1xa1eKD4nK", "reviewer": "Reviewer_KUis", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive criticism of our paper. Following are the responses to the concerns you raised under **Reasons to reject**:\n\n> only prompting methods used as baselines. The approach uses detailed finetuning with extra architectural elements and tuning to solvers but no baselines contain any elements similar to these. It is not clear from the experiments if finetuning a baseline could also yield the benefits shown.\n\nOur proposed approach DaSLaM **does not finetune** the solver model. It only finetunes the decomposer model (LLAMA 13B) to generate subquestions based on the output of the solver. Yet, as per the comment, we have recently experimented with full finetuning of an LLAMA 13B model on the training dataset we used for tuning the decomposer; the results are shown below:\n\n\n| Method | PnC | NT | ALG | iALG | Geom. | Cal | Palg | AQuA |\n| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| DaSLaM (LLaMA 13B)     | 4.0    | 5.6    | 4.70     | 3.4     | 4.3    | 4.1    | 8.33     | 22.0     |\n| Finetuned (LLaMA 13B)    | 2.8   | 3.6   | 3.57    | 3.2  | 4.1     | 3.05   | 6.04  | 19.4  |\n\nClearly, despite using a non-tuned solver, DaSLaM delivers better performance compared to full finetuning. Even though the decomposer is finetuned, it is solver-agnostic. So, it can be used with different solvers (as we have shown in the paper with three different solvers). Furthermore, it is prohibitive to finetune most existing LLMs. Our method provides a strong solution to this problem.\n\n> The approach requires knowing solver errors and annotating with decompositions to tune the decomposer. This seems like a limitation, since that would require significant after-the-fact human effort in most domains especially with more complex answers than multiple choice or simple numeric.\n\nThere is a misunderstanding. DaSLaM **does not require** a step-by-step error annotation of the solver output. While training the decomposer, we only need whether the final output from the solver is correct or not as supervision signal. The whole point of using RL to train the decomposer is to bypass such costly human intervention using simple, automated rewards computed from the existing chain-of-thought annotated datasets.\n\n> some aspects of the description was not clear, such as the cosine similarity which seems to use a different model\n\nWe used Spacy's pretrained pipeline to compute cosine similarity between two text segments. We mentioned this in Appendix B.\n\n> Reproducibility: The settings of parameters are underspecified or subjectively determined; the training/evaluation data are not widely available.\n \nWe provided the full codebase and datasets used in the Supplementary along with the hyperparameter details in Appendix C.\n\nWe commit to include the additional experimental results presented in the response in the revised version of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ addresses an important problem\n+ observes that simpler models could produce better results by fine tuning---not a major insight, but still important to note\n+ It looks like the results on JEEBench are good, which suggests some capacity for generalization for the decomposer model.", "reasons_to_reject": "- some aspects of the description was not clear, such as the cosine similarity which seems to use a different model\n- only prompting methods used as baselines. The approach uses detailed finetuning with extra architectural elements and tuning to solvers but no baselines contain any elements similar to these. It is not clear from the experiments if finetuning a baseline could also yield the benefits shown.\n- The approach requires knowing solver errors and annotating with decompositions to tune the decomposer. This seems like a limitation, since that would require significant after-the-fact human effort in most domains especially with more complex answers than multiple choice or simple numeric.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Language Models Separately Tuned for Decomposition and Solution Improve Complex Reasoning\n\nIn this paper the authors suggest training a separate LLM that learns to decompose a problem in order to improve reasoning in LLMs. They do this by first finetuning the LM to produce the right subproblems. Then they use reinforcement learning to sensitize the decomposer to errors made by the solver. They show that this approach works better than other baselines such as chain of thought prompting in many cases.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yMhFHLTiFP", "reviewer": "Reviewer_fnE5", "summary": "", "questions": "- E. Is this the first paper to employ RL to train an auxiliary model to guide complex reasoning, or other forms of problem solving, by very large LLMs? This is not explicitly stated in the paper, and if there are other papers, discussion of these would help readers better understand the paper's contribution.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive criticism of our paper. Following are the responses to the concerns you raised under **Reasons to reject** and **Questions For The Authors**:\n\n> (\"cerebral functions are specialized and spatially localized\") does not support the need for separate LMs for decomposition and solution [...] (1) the connection assumes that LMs and human brains work in the same way [...] (2) multiple cognitive abilities of LMs can also be localized in terms of model parameters, without the need for separate models [...] step-by-step supervision of a monolithic LLM can greatly improve complex reasoning [...] without the need for specialized models.\n\nAs you correctly point out, the neuroscience reference is not essential to develop our ideas, and will be removed.  However, from (1) above, you might have misunderstood our intent. We need not assume that LMs and human brains work in the same way (we know for sure they do not) to want to try something that works on humans on a LM.  CoT and RLHF work with both human brains and LLMs, after all! As for (2) above, we regard localization and modular editing research to be in nascent stages with much left to explain and invent.\n\n> paper does not need to include this speculative motivation. [...] The motivation to train a separate external model could come from the fact that foundational LLMs are typically not available for fine-tuning and it is computationally prohibitive to do so. In fact, the **versatility of the proposed method**, which only requires fine-tuning of a small language model and can be applicable to black-box models for the underlying solver model, is a **major strength of the proposed method**. This could be highlighted further in the introduction.\n\nThanks! This sounds *much* better than we ourselves pitched the work! We will update our manuscript accordingly.\n\n>  The case study (Section 7) only pertains to one sample, suggesting that it may have been cherry-picked. While this is effective at highlighting the strength of the proposed method, overall qualitative analysis based on a number of random samples would help readers to understand the overall strengths and weaknesses. This could also be supported with more samples provided in the paper, highlighting both success and failure cases of the method.\n\nWe greatly appreciate this suggestion. We included only one example due to space constraint in the main paper. We commit to include more examples from different datasets in the revised version, highlighting *both* the strengths and the weaknesses of our proposed method.\n\n> If results are available, it would be helpful for the readers to provide a full performance table for GPT-4 baselines\n\nResults of GPT-4 on the JEEBench dataset are available in [1]. We will include them under Table 3 in the revised version. For the MATH dataset, we could only find the aggregate results and not the topic-specific ones.\n\n> The writing is generally easy to follow, but off-putting or erroneous at times. It is evident that the writing could benefit from more meticulous attention. I would encourage the authors to revise the text for a more polished and refined presentation, given the significant contribution of their work.\n\nWe greatly appreciate the suggestion. We commit to revising the current version and correcting such mistakes wherever present.\n\n> Is this the first paper to employ RL to train an auxiliary model to guide complex reasoning, or other forms of problem solving, by very large LLMs? This is not explicitly stated in the paper, and if there are other papers, discussion of these would help readers better understand the paper's contribution.\n\nTo the best of our knowledge, ours is **the first such proposal**. Training verifier-like auxiliary models is not new, but they do not *actively* guide the solver along the complex reasoning requirements like the DaSLaM decomposer.\n\n[1] Arora, Daman, and Himanshu Gaurav Singh. \"Have LLMs Advanced Enough? A Challenging Problem Solving Benchmark For Large Language Models.\" arXiv preprint arXiv:2305.15074 (2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors propose a novel approach to apply RL to deeply tune mid-sized LLMs to guide complex reasoning (of much larger LLMs). The proposed approach differentiates itself from previous works from a technical standpoint, as these are based on prompting.\n- The paper tackles the important issue of hallucinations and errors in complex reasoning, highlighted by the abundance of recent work. Relevant literature appears to be thoroughly discussed throughout the paper, but I am not certain.\n- The method shows remarkable performance improvements over prompt-based baselines and are shown to generalize well to novel solver models and benchmarks.\n- The method is applicable to black-box solver LLMs, which is the case for recent state-of-the-art models such as GPT-4 and PaLM-2.", "reasons_to_reject": "- A. The motivation proposed by the authors that \"... decomposition and solution generation .... need distinct capabilities\" (Line 13) is not sufficiently supported. The citation from the neuroscience literature in Line 67 (\"cerebral functions are specialized and spatially localized\") does not support the need for a separate language models for decomposition and solution generation, for two reasons: (1) the connection assumes that language models and human brains work in the same way, which is risky conjecture and (2) multiple cognitive abilities of language models can also be localized in terms of model parameters, without the need for separate models–in fact, pre-trained language models already demonstrate a wide variety of abilities. On the contrary, concurrent work [1] shows that step-by-step supervision of a monolithic LLM can greatly improve complex reasoning performance, without the need for specialized models.\n  - A1. In my opinion, the paper does not need to include this speculative motivation. The well-known issue of {hallucinations/errors from chain-of-thought reasoning, and the limitations of existing prompting methods to mitigate this issue} provide sound motivation to apply more explicit forms of supervision via RL to guide the model. The motivation to train a separate external model could come from the fact that foundational LLMs are typically not available for fine-tuning and it is computationally prohibitive to do so. In fact, the versatility of the proposed method, which only requires fine-tuning of a *small* language model and can be applicable to *black-box* models for the underlying solver model, is a major strength of the proposed method. This could be highlighted further in the introduction.\n- B. The case study (Section 7) only pertains to one sample, suggesting that it may have been cherry-picked. While this is effective at highlighting the strength of the proposed method, overall qualitative analysis based on a number of random samples would help readers to understand the overall strengths and weaknesses. This could also be supported with more samples provided in the paper, highlighting both success and failure cases of the method. Comprehensive analysis of both strengths and weaknesses would greatly improve the paper.\n- C. If results are available, it would be helpful for the readers to provide a full performance table for GPT-4 baselines (near Line 515), even if the results may not be favorable for the proposed method.\n- D. The writing is generally easy to follow, but off-putting or erroneous at times. It is evident that the writing could benefit from more meticulous attention. I would encourage the authors to revise the text for a more polished and refined presentation, given the significant contribution of their work. A non-exhaustive list of potential issues are noted in \"Typos Grammar Style And Presentation Improvements\".\n  - D1. The organization and writing for page 6 are notably very good.\n\nI'd like to note that the idea and contribution of the paper are quite strong, but my initial scores will have to reflect the issues above. I am open to adjusting my score if these issues are addressed.\n\n[1] Lightman 2023, Let's Verify Step by Step", "questions_to_authors": "- E. Is this the first paper to employ RL to train an auxiliary model to guide complex reasoning, or other forms of problem solving, by very large LLMs? This is not explicitly stated in the paper, and if there are other papers, discussion of these would help readers better understand the paper's contribution.", "paper_topic_and_main_contributions": "This paper proposes a method to train a small (e.g. 13B) *decomposition* model that guides step-by-step problem-solving by a black-box *solver* model (e.g., GPT-3.5) via reinforcement learning (RL). Unlike recent methods which use prompt-based methods to explicitly guide problem solving, this work uses RL to deeply tune a decomposition model to guide and thus improve the performance of the solver model. The results are very strong, demonstrating notable improvement over baseline prompting methods, and show clear generalization to other solver models (e.g., LlaMA) and harder benchmarks (e.g., JEE) unseen during training.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "See \"Questions for The Authors\"."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- There is a small error in terminology regarding OpenAI models. `text-davinci-003` is also a variant of GPT 3.5 rather than GPT 3 (Line 108), according to OpenAI (https://platform.openai.com/docs/models/gpt-3-5).\n- In the caption of Figure 1, it is stated at \"In step 1, the solver ... fails to answer correctly\", but this is not always the case. The reader may ask, \"what happens when it answers correctly?\". The explanation could be improved by clarifying that the solver LM provides an initial answer, which is *often* found to have errors, for which the decomposition model takes into account when generating subproblems.\n- In Line 171 (\"the decomposition actuates with complete knowledge of the solver's actions\"), I was confused by the phrase *complete knowledge*. A more appropriate explanation may be, e.g., that the decomposition is informed by the actions of the solver.\n- The text explanations and equations in Section 3 could both be better polished and organized to enhance readability. I had/have many confusions including: (1) whether the decomposition model (DM) uses the entire chain-of-thought reasoning from the solver model (SM) or just the first step, as in Figure 1, (2) that the DM generates all subproblems in a single step, and the SM answers each of these independently, (3) whether the solver uses the solutions from previous subproblems to answer the current subproblem, (4) what is the difference between $\\hat{A}_i$ and $S_i$ in Eq 4, i.e., \"is the solver prompted to infer a final answer based on the local solution of each subproblem, at *each* step?\", etc.\n  - The mathematical expression `argmax [ p(...) argmax(...) ]` used throughout Section 3 is not standard syntax, therefore can be confusing."}}, {"review_id": "MywiIXS0Qa", "reviewer": "Reviewer_xvjb", "summary": "", "questions": "See weaknesses", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive criticism of our paper. Following are the responses to the concerns you raised under **Reasons to reject**:\n\n> The comparisons feel quite unfair. The approach fine-tunes their model, while the alternatives are all exclusively prompting-based. To me, the most important ablation to include would have been one where the model is asked to decompose and solve the problems without training.\n\nWe emphasize that **the solver model is not finetuned in any of the experiments**. Only the decomposer is finetuned. Since this is a LLaMA 13B model, our preliminary experiments suggest that the question generation capability of the decomposer **before** finetuning is very poor, as shown below:\n\n| Method | PnC | NT | ALG | iALG | Geom. | Cal | Palg | AQuA |\n| -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |\n| DaSLaM (text-davinci-003)     | 21.4    | 26.1    | 33.4     | 24.8     | 18.2    | 44.0    | 21.4     | 54.5 |\n| DaSLaM (text-davinci-003) prompted decomposer     | 12.4 | 9.6 | 23.2 | 7.6 | 8.33 | 17.8 | 8.1 | 39.0 |\n| DaSLaM (LLaMA 13B)     | 4.0    | 5.6    | 4.70     | 3.4     | 4.3    | 4.1    | 8.33     | 22.0 |\n| DaSLaM (LLaMA 13B) prompted decomposer     | 3.8 | 3.83 | 4.6 | 3.2 | 3.66 | 3.05 | 5.0 | 19.6 |\n| DaSLaM (LLaMA 33B)     | 4.0 |  7.36 | 9.09 | 6.02 | 5.3 | 6.03 | 8.44 | 26.8     |\n| DaSLaM (LLaMA 33B) prompted decomposer     | 1.6 | 3.2 | 5.6 | 4.7 | 3.2 | 2.4 | 6.66 | 21.6 |\n\nAs we can see, there is >50% reduction in accuracy in most tasks when we replace the finetuned decomposer with a prompted one. \n\nWe claim that the model-level separation of the decomposition and solution steps in DaSLaM is superior to single-model decomposition-solution approaches like [1], irrespective of the power of the solver. We used Least-to-most prompting as a baseline to validate this claim. \n\n> sometimes knowing what question to ask next requires an answer to a previous question - however, as far as I understand, the approach in this paper has no way of handling that\n\nWhile this is not the central question of our paper, our proposed method partially addresses this. While [1] seeks to generate a subproblem based on the original problem and the answer to the subproblem, they do not take into account how the solver acts against the original question. We showed that this feedback plays a crucial role; the decomposer in our ablation variant DaSLaM-NF generates the subproblem independent of the initial response of the solver. This ablation variant performs pretty poorly, irrespective of the decomposer finetuning.\n\n> the main contribution of the paper is not that decomposing and solving helps, but that this can be improved with fine-tuning as they propose\n\nWe would like to point out that our contribution is not just limited to finetuning. A major bottleneck with finetuning is that most existing LLMs are either prohibitively large to finetune using a constrained resource or the API providers like OpenAI do not provide such a facility. On the other hand, the superiority of such large models as solvers compared to (even finetuned) smaller models is generally accepted. Our method provides a novel way to incorporate finetuning-based knowledge transfer into such large models while treating them in a black-box manner.\n\n> Missing references\n\nWe commit to include discussions on the mentioned papers in the revised version.\n\n> Reproducibility: The settings of parameters are underspecified or subjectively determined; the training/evaluation data are not widely available.\n\nWe have provided the full codebase and datasets used in the Supplementary along with the hyperparameter details in Appendix C.\n\nWe commit to include the additional experimental results presented in the response in the revised version of our paper.\n\n[1] Dua, Dheeru, et al. \"Successive prompting for decomposing complex questions.\" arXiv preprint arXiv:2212.04092 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea is fairly natural and simple to motivate. The paper is well-presented and compelling.", "reasons_to_reject": "The comparisons feel quite unfair. The approach fine-tunes their model, while the alternatives are all exclusively prompting-based. To me, the most important ablation to include would have been one where the model is asked to decompose and solve the problems without training. In my eyes, the main question that this paper has to answer is not whether decomposition helps in answering reasoning questions - much prior work (e.g. [1-3]) has already shown this. As indicated by [2], sometimes knowing what question to ask next requires an answer to a previous question - however, as far as I understand, the approach in this paper has no way of handling that (though that is not a reason to reject it in itself - not every paper has to solve every problem - but it weakens the message).\n\nIn other words, the main contribution of the paper is not that decomposing and solving helps, but that this can be improved with fine-tuning as they propose. However, as far as I understand, this paper does not perform this analysis which is central to the work.\n\n1. [1] Successive Prompting for Decomposing Complex Questions, Dua et al. 2022\n2. [2] Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP, Khattab et al. 2022\n3. [3] Parsel: Algorithmic Reasoning with Language Models by Composing Decompositions, Zelikman et al. 2022", "questions_to_authors": "See weaknesses", "paper_topic_and_main_contributions": "This paper proposes to split up LM problem solving with two separately finetuned LMs, one for decomposition and one for solving subproblems.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] Successive Prompting for Decomposing Complex Questions, Dua et al. 2022\n[2] Parsel: Algorithmic Reasoning with Language Models by Composing Decompositions, Zelikman et al. 2022"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers have a consensus that work is sound and exciting. The finetuning comparison as well as the clarrification in the review discussion should be included in further edits of the paper.
Decision: Accept-Main