Learning Math Reasoning from Self-Sampled Correct and Partially-Correct Solutions
Authors: Ansong Ni
Jeevana Priya Inala
Chenglong Wang
Alex Polozov
Christopher Meek
Dragomir Radev
Jianfeng Gao
Venue: ICLR 2023 poster (2023)
Keywords: mathematical reasoning
multi-target learning
self-sampling
large language models

Abstract: Pretrained language models have shown superior performance on many natural language processing tasks, yet they still struggle at multi-step formal reasoning tasks like grade school math problems. One key challenge of finetuning them to solve such math reasoning problems is that many existing datasets only contain one reference solution for each problem, despite the fact that there are often alternative solutions resembling different reasoning paths to the final answer. This way, the finetuned models are biased towards the limited reference solutions, which limits their generalization to unseen examples. To mitigate this issue, we propose to let the model perform sampling during training and learn from both self-sampled fully-correct solutions, which yield the correct answer upon execution, and partially-correct solutions, whose intermediate state matches an intermediate state of a known correct solution. We show that our use of self-sampled correct and partially-correct solutions can benefit learning and help guide the sampling process, leading to more efficient exploration of the solution space. Additionally, we explore various training objectives to support learning from multiple solutions per example and find they greatly affect the performance. Experiments on two math reasoning datasets show the effectiveness of our method compared to learning from a single reference solution with MLE, where we improve PASS@100 from 35.5% to 44.5% for GSM8K, and 27.6% to 36.2% PASS@80 for MathQA. Such improvements are also consistent across different model sizes.

Published as a conference paper at ICLR 2023
LEARNING MATH REASONING FROM SELF-SAMPLED CORRECT AND PARTIALLY-CORREG SOLUTIONS
Ansong Ni Jeevana Priya Inala? Chenglong Wang
Oleksandr Polozov Christopher Meek Dragomir Radev! Jianfeng Gao
Yale University, "Microsoft Research, GGoogle, 'University of Washington
ansong ni si@yyale.... edu, {iinala, cherwanggemmccrooof...o
ABSTRACT
Pretrained language models have shown superior performance on many natural language processing tasks, yet they still struggle at multi-step formal reasoning tasks like grade school math problems. One key challenge of finetuning them to solve such math reasoning problems is that many existing datasets only contain one reference solution for each problem, despite the fact that there are often alter- native solutions resembling different reasoning paths to the final answer. This way, the finetuned models are biased towards the limited reference solutions, which limits their generalization to unseen examples. To mitigate this issue, we pro- pose to let the model perform sampling during training and learn from both self- sampled fully-correct solutions, which yield the correct answer upon execution, and partially-comred
solutions, whose intermediate state matches intermediate state of a known correct solution. We show that our use of self-sampled correct and partially-correc solutions process, leading to more efficient exploration of the solution space. Additionally, we explore various training objectives to support learning from multiple solutions per example and find they greatly affect the performance. Experiments on two math reasoning datasets show the effectiveness of our method compared to learn- ing from single reference solution with MLE, where we improve PASS @ 100 from 35.5% to 44.5% for GSM8K, and 27. 27.6% to 36.2% @A for MathQA. Such improvements are also consistent across different model sizes. Our code is available ht
benefit learning and help guide the sampling
INTRODUCTION
Recent progress on pretrained language models shows that they are able to achieve human-level performance on various natural language processing tasks with finetuning(Devlig et al.) 2019 Brown
et al. 2020 Raffel et al., 2020). However, such models still lack the ability to perform multi niut math reasoning even for problems that intended for grade-school students (Cobbe et al. 2021)
Current methods for solving math problems typically rely on generating solutions (a sequence of computation steps) and executing them to obtain the final answer (Cobbe et al.) [202]] Austin al.
2021
Chen et al. 2021a; Chowdhery et al.] 2022). as directly generating the final answer would
require computational abi aiilit that even the largest models do not possess (Brown al. 2020
Chowdhery et al.) (022).
When finetuning such models on math reasoning, existing methods often rely on the MLE objective that aims to maximize the log-likelihood of the reference solution for each natural language input. However, in addition to the reference solution, there are often multiple correct solutions for each question, resembling alternative reasoning paths to the final answer. However, those alternative solutions are unseen during training, and this results in model overfitting: the model becomes overly confident in its predictions because it sees the same solution over multiple epochs of training (Bunel et al. 2018 Austin et al. 2021 Cobbe et al. 2021). This leads to poor generalization on unseen
Majority of the work done during internship at Microsoft Research. Work started while at Microsoft Research, paper contribution limited proof-reading. Work initiated while Microsoft.

---

Published as a conference paper at ICLR 2023
inputs and is reflected by the low PASS@1 performance, where the model is unable to predict the right answer even when allowed multiple attempts per question.
To mitigate this issue, we propose learning from self- self-sampled solutions. Concretely, during training time, the model samples alternative solutions, and keeps track of all solutions that are semantically correct with respect to the gold execution result, and learns from all of these correct solutions as opposed to only from the reference. To further improve the effectiveness of learning from self- sampled solutions, we allow the model learn from partially-correc solutions, whose intermediate states are consistent with intermediate states of known correct solutions. This new technique allows the model to maximally utilize the self-samliing and more efficiently explore the solution space. We also study various common loss functions for learning from multiple targets for single natural lan- guage input, including augmented-Me E.,E, Maximize Marginal Likelihood (MML) and f-smooohed MML. (Guu et al. 2017) and find that their different gradient equations greatly affect the learning capabilities of the model.
We perform experiments two math reasoning tasks, namely MathQA-Python (Austin et al. 202 and Grade-School-Marl (GSM) (Cobbe et al., (021), and finetune GPT-Neo models (Black et al., 2021) to generate Python program as solutions from the problem description in natural language. Results show that learning from self-sampled solutions can improve the PASS@ 100 from 35.5% to 44.5% for GSM, and 27.6% to 36.2% for PASS @ 80 on filtered version of MathQA-Python
Moreover, we find that learning from partially-correct solutions generally improves performance over learning from just fully-correct solutions (e-g., +3.0% PASS 100 for GSM8K) it guides the sampling process, discovering more alternative solutions for learning. Such performance boosts from our proposed methods are also consistent for different model sizes. Ablation on different loss functions shows that MLE-Aug loss is the most effective in learning from multiple targets and yields the most improvements over MLE loss.
2 OVERVIEW
Problem formulation. We consider the task of generating solutions from math problem descrip- tions in natural language (NL). Given an NL input X' and the executor E: y + Z, the goal is to generate a solution y E y that executes to the expected answer =" E 2, i.., E(v) =²
Standard approach and its limitation. The standard approach is to assume that we have a dataset of paired NL input and reference solution y". Most datasets typically only provide one reference solution for particular NL input. Then, a parameterized model Pd is learned with the Maximum Likelihood Estimation (MLE) objective from the NL-Solution pair (æ,, y²) as:
= - log Po(y"l [zx
The builtin assumption of using Fg.. 1 for learning is that only the reference solution y" is correct. However, this assumption is clearly untrue for the math reasoning problem as typically multiple reasoning paths can achieve the correct final result. With only one reference solution as target for learning, Eq. I would encourage the model to put all probability mass on y" which could easily lead to overfitting (Bunel al. 2018 Austin et al. 2021 Cobbe et al. 2021).
Overview of our approach. While manually collecting additional reference solutions for each specification is laborious process (Austin et al. 2021;; Cobbe al. 2021: [Schuster et al. 2021), in our work, we explore an alternate approach: where the model self self-ssmples additional correct (or example: for the question x, our model was able to self-sample an alternative solution ÿ that is different from the reference solution y provided in the dataset. Looking at the intermediate states shown on the right, we can see that both these solutions execute to produce the sample desired output, i.e., á = x", as noted with solid red boxes. Taking this one step further, our approach can also identify partially- correct solutions from its samples. For example, on the bottom left, we show sampled solution D' that is incorrect only because of an error its last two steps. But we identify a prefix j'zs of it as partially-correet because the intermediate state & for this prefix matches the intermediate state ss of known correct solution y" (noted dashed red boxes) and yet syntactically different from y". Based on these observations and intuitions, we introduce our approach in the following sections.
partially-corrent solutions and learns from them during training. FFg. shoows
We choose different for evaluating PASS @ la be consistent with previous work.
2

---

Published as a conference paper at ICLR 2023
goods train speed 72kmph and 250M platform in 26 seconds. What the length of the goods train?
Algorithm 1 Training Update
Input:
[22}
Ruf Solution
250
[72, 250)
Parameterized model Po(y)a::; Executor & y Z; A training example (z, y =²); Buffer B for this imput ir
n2 26
(72, 250, 26)
0.2778
(22, 250, 250,_26, 23,011
n1 / te
1172, 258, 26, 28.0, 12.51
[72, 256, 26, 26.8, 12.5.
onswer to t2
output
278
1: if |
then
Self-Sampled Solutions
Evaluate with Exerutor
2: B 4- B {v²) initialize buffer
3: end if
Cornect Solution
nê 72
(72)
4: Y + SampleSolutions(;, Po. B)
n1 250
(72, 250
Correct
n2
[72, 258, 26)
5: for i in do
to no 0.2778
(72, 250, 26, 20.0)
6: 2 3 E(j) 1* exccute solution */
t1 n2
[72, 250, 26, 20.0, 520)
7:
if isCorrect( x*) then if not isDuplicate( B) then B + ÿ / save to buffer
output 278
2
8:
FettalyCorrd Salition
Frofte Proft
(22)
Partially-
9:
n1 250
(72, 250)
Correct
10:
end
f'ss
(72, 258, 26)
11: end if
to no / 3.6
122-250, 25, 28.0)
01.01/1 10
12.5% 72, 250, 26, 20,0, 12.5
12: end for
12 n0 11
[72, 238, 26, 28.0, 12.5, 59.5
13: 0 update
VoL(u, Po)
answer to t11
output 1547
Solutions
Intermediate
Figure I: Examples of self-sampled correct and partially- correct solutions from MathQA (more in[ppeendix DJ. The steps and intermediate states marked in red are incorrect.
3 LEARNING FROM SELF-SAMPLED SOLUTIONS
We formally present our approach. There are three main steps: 1) sampling 2) filtering and 3) learning as shown in Alg. Here we mainly introduce the self-sampling framework using only fully-correct solutions and the extensions with partially-correct solutions will be introduced in 3.3
3.1 ONLINE SAMPLING AND FILTERING
For each specification æ, we maintain a buffer B to save the different solutions that are correct, i.e., evaluate to the correct result. Note that the buffers persistent and cumulative across training epochs. To add more solutions in B, we perform online sampling and filtering as follows. Online sampling (line 4 in Alg. 1}: With the NI. question from each example (3r, y =²") as input, the model samples a set candidate solutions Y {0 (o]lzz);
Alg. 1 As not all sampled solutions in Ý are correct (thus not suitable for learning), we filter out all incorrect solutions in Y, i.e., Y {ūlı E E = 2 }; Filtering duplicate solutions (line 8 in Alg. I): Because the model can sample solutions that correct but are "trivial variants" of other already saved solutions (e.g., the solution differs from an-
Filtering incorrect solutions(line
other solution only in white spaces, comments or trivial steps like "x x * 1. 1")"), we further filter the buffer to remove them. This is essential as all saved solutions will be directly used for learning and such undesired behavior from the model will be encouraged without the filtering pro- cess Concretely, we first perform filtering based on the linearized abstract syntax trees (ASTs) to eliminate the differences in white space, etc; then we set a constraint on maximum number of lines using the number of lines in y² the reference prevent saving solutions with trivial steps.
3.2 LEARNING FROM MULTIPLE TARGETS
With self-sampling, each natural language question is paired with multiple solutions as targets for learning. Here we discuss some common loss functions for the multi-target learning problem, with
Our preliminary experiments also show that the performance greatly degenerates when such trivial variants left in the buffer for learning.
3

---

Published as a conference paper at ICLR 2023
Name
Loss Functions L(x, Pb)
Gradients B,
MLE
log P' (y (1(1z)
Vo Volog (s (x)
MLE-Aug
EBEn log Po(((a))
-E BEB log Po Po(j)z
MMI.
log EBEs Po(()ar)
Σ JEB Leee P
Patúle)
Vo log (o(
8-MML
- log E, Po(ū)=)
Σ Eyee SEB
Polújm)
V log (o(())
LaeR Ps(a)
Table 1: Comparison of loss functions and their gradients over multiple reference B. Note that they all degenerates to MLE when only the gold reference solution is used as target, i.e., {v*}.
a focus on how each target contributes to the gradient. The loss functions and their gradients are shown in Tab.
Augmented MLE (MLE-Aug): This objective augments MLE with multiple targets simply by summing the loss from multiple solutions in B, which is equivalent as minimizing the KL-divergence from Po ()) to (1 x)) [8] 1l 1s sv)) where 1 (-) is a set indicator function. It encourages the model to put equal weights on all targets by ensuring that all targets equally contribute to the gradient.
Maximum Marginal Likelihood (MML): MML attempts to approximate Pa (x)) by marginaliz-
ing over the correct solutions in B. However, for each target E B, the gradient of it is in proportion to the likelihood Po (o((ar)) ()) given by the model, which results in a positive feedhack loop during gradi-
ent updates. It encourages the model to still put a majority of the probability in B as noted in (Guu et al. (017).
of the solutions
(017). the 8-MML objective an extension of MML with a hyperparameter B E (0. 1] to adjust weights of the gradient from each target. It an interpolation between MML and MLE-Aug objectives, more specifically, it recovers MML when and its gradient is equivalent to that of MLE-Aug when / 0.
3-smoothed MML (3-MML): Proposed in Guu
Empirically, we find that these distinctions between those loss functions greatly affects the model performance Fig. especially when partially-correct solutions are included for learning.
3.3 LEARNING FROM PARTIALLY-CORRECT SOLUTIONS
Besides learning from self- sample fully-correct solutions (FCSs), we can also let the model learn from partially-correct solutions (PCSs). Our motivation is that the model often encounter solutions that are close to being correct as they only make mistakes in the last few steps (e.g.. Fig. T), and these partially-correct solutions provide additional learning opportunities. Learning from PCSs could also address the issue that the sampler may have a low chance of encountering fully-correct solutions for complex tasks due the sparse solution space.
3.3.1 IDENTIFYING PARTIALLY-CORIE
SOLUTIONS
When the model samples a solution that does not produce the desired answer, we want identify a prefix of this solution is partially correct, i.e., it performs some of the necessary computation steps needed for the correct solution, so that the model can additionally learn from these potentially unseen prefixes in the next iteration. A challenge here is figuring out when a prefix is partially correct. Ideally, we want say prefix yçe is partially correct if there exists suffix y>s such that their concatenation (ygal (y>+) is correct solution. There are two caveats here: (1) if there is no length restriction on the suffix, it is always possible find a suffix that complements any prefix (e.g., a full gold solution is one such suffix); and (2) it is computationally very expensive to search for all suffixes (even with a length restriction) to check if prefix can be completed to a correct solution.
To overcome these challenges, we leverage the gold reference solutions and any self-sampled fully- correct or even partially-corred solutions to help identify new partially-correct prefixes. The idea is to identify a prefix as partially correct if it produces set of intermediate values (upon execution) that exactly matches the set intermediate values produced by a prefix of a known correct or partially- correct solution. For such prefix, know that there exists reasonable complement suffix based on the suffix of the known solutions. Note that, this definition of partial correctness is conservative compared to the ideal definition above, but it makes the computation significantly tractable.
4

---

Published as a conference paper at ICLR 2023
Algorithm 2 SampleeSllutions(z,, Po, 1)) with partially-comeed solutions
Input: Model (yla); the NL input x and set of partially-correct solutions B
Output: Solution samples Y.
I: Select igi E B (0j E(ū) = uniformly at random /* sample PCS prefix for completion */
2: Sample set of completions Y, ~ Po Po(O>=l0ge,zz
3: + -[0sc bssee,,,,, / concatenate completions with the solution prefix "/
4: return Y
Below, we formally define this notion of partial solutions that leverages existing known fully and partially correct solutions.
Intermediate state. Given a solution y = (u,, .... U.) where u, is the i-th reasoning step, we define the intermediate state 8; the set of all variables values in the scope after executing the first steps yg. (u....., uz), which we call a prefix of this solution. It is easy to see that the prefixes y<; and intermediate states s, of a solution construct bijective function, which is also illustrated in Fig. 1 Note that the state representation is name-agnostic since variable names do not typically contributes to the semantics of the solutions.
State-based equivalence and partial correctness. Given the definition of the intermediate state, we say the prefixes of two solutions, yęı and Vlej. are semantically equivalent if and only if s; = i.e, those two solutions produces the exact same set of variable values. And then we define partial correctness as follows: solution prefix Vşi partially-corred if and only if it is semantically equivalent to the prefix of another known partially-correct solution yeg- As we keep all known partially-correet solutions in the buffer B, formally:
PartiallyCorrectys)) + By B. 3j S 1/01 8.1. 83 81
3.3.2 MODIFICATIONS TO THE MAIN ALGORITIMM
To support learning from partial solutions, we modify Alg. I as follows to enable buffering and sampling from partial solutions. The fully updated algorithm is shown in Ampppedix C
Samped-Sampling: In 3.1 we mentioned that full solutions are sampled for each question as I Po(ū(=). With PCS prefixes, compared with sampling a solution from scratch, generating solutions with these prefixes reduces the generation length thus the model can more efficiently explore the solution space. This guided sampling process is described in more detail in Alg. 2] Note that since the empty solution yd is in the buffer B since initialization, therefore model can still generate and explore the space from scratch and not always follows the existing solution prefixes.
prefixes: As mentioned in [S 3.3 if a solution ğ does not produce the expected result =x but its prefix dgv is partially-correct, the model can still learn from its prefix. However, an important task here to identify the longest partially-corred other words, locate the exact step that the solution deviates from a correct reasoning path. We can achieve this simply by backtracking the intermediate states and find the first state that is equivalent to any of the states from a saved solution. Fl
Identify partially-corres
prefix for learning, in
Filtering solution prefixes: With the inclusion of partially-correct solutions, we need to slightly change the two filtering criteria in S 3.1 For deduplication, while still use AST to rule out changes with non-semantic tokens such as white space, we also check the partially-corret solution prefix Uz ; is prefix of another known PCS in B3 For the same reason, when saving new partially- correct solution ÿ. we need to prune out any existing solution in B that is a prefix of ÿ. As for the length constraint, the same principle still applies, but now is compared against other partially- correct solution that executes to the same state.
Learning objective: As partially-correct solutions are solution prefixes yeu missing the later part y>4, with an auto-regressive generation model, the learning of Pe Po(ys(z)) is independent of y>-- Thus the learning objectives in 3.2 do not need to change with the inclusion of PCS in the buffer for learning. The only difference is that the end-of-sequent " "(eos)" token is not appended to the PCS as those solutions are not yet finished.
In practice we use state solution on prefix dictionary and the lookup takes a negligible amount of time.
5

---

Published as a conference paper at ICLR 2023
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. We evaluate on two math reasoning datasets, in which we generate straight-line Python programs as solutions to solve math problems described in natural language. We finetune the lan- guage models to output such program solutions using only the natural language problem description as the input.
The original MathQA-Python consists of 19.2K training examples of NI and Python program pairs (Austin et al. 2021). However, find the raw dataset contain many questions that share the same question templates and only differ in concrete number across the train/dev/test sets. To better understand the generalization of the trained models, we derive deduplicated version of the dataset by first merging the train and dev data and then perform template- based deduplication. Partly inspired by Preegan-Doolate et al. (2018), we re-split the train and dev set based on the question templates, resulting in 6.8K/0.7K train/dev data for the filtered version F While we mainly experiment on the filtered version, we report performance on both versions when compared with previous methods.
> MlathoA-Python-illle
& SGMS5.5KPythonn The grade-school-man mmath (GSM8K) dataset (Cobbe al., (02T) contains 7.5K training data points. Since it only provides natural language solutions with math formulas and does not have a dev set, we first reserved 20% of the training data as dev set, then automatically converted the formulas to program solutions in the same style MathQA-Python. As the result, we finetune our models with the 5.5K successfully converted training examples. Note that the natural language solutions/explaaaatios are not used input the models in our experiments.
Evaluation metrics: Following recent work in neural program synthesis (Austin et al. 2021 Chen et al. 2021a Chowdhery et al. 2022) and math reasoning (Cobbe et al., [222], J. we use PASS @ as our main evaluation metric. It allows the model to sample A solutions for each question and the task considered solved if any one of the solutions is correct, so PASS @ k can also be seen as the fraction of problems in the test/dev set being solved given k attempts. More details (e.g., temperature) can found in Appendix A
Model training: We use GPT-Neo Bllck et al. 2021) as our language model and mainly study two model sizes, 125M and 2.7B Following previous work (Austin al., 2021). we evaluate all PASS @ k on the same model checkpoint that has the best @ score, but note that it might not be the best checkpoint for other & values (more discussion in Appendix E). Detailed hyperparameter settings can also be found in Appendix A.
4.2
MAIN RESULTS
shows the performance on the two datasets by learning from self-sampled FCSs and PCSs using MLE-Aug (orange bars), com- pared with MLE on single reference solution (blue bars). We can see that our proposed method can greatly improve @ASS@k, k, especially for higher & values. By comparing different model sizes, see that learning from self-sampled solutions help with both small and large models, with a +12.3% and +9.0% PASS @ 100 improvement on S.M555K-Pythen GPT-Neo-2.7B, respectively and +3.1% and +8.6% PASS @ 80 improvement on MathQA-Python Filtered for GPT-Neo-11 125M and GPT--Ne-22.78 respectively. We note that our approach does not improve PASS which is expected as learning from multiple targets mainly helps with increasing the diversity of the sampled solutions rather than improving the most-beo solution (for which MLE is better suited).
Learning from self-sampled solutions improves PASS k. Fig. 2
for GPT-Neo- 125M and
Partially-correct solutions improve model performance. We next show the effects of including partially-corres solutions on PASS @k performance in Fig. 2 (green bars vs orange bars) and the number of saved FCSs and PCSs in Fig. 4 First, we observe from Fig. that using partial correctness not only results in PCSs being saved and directly learned from, but also boosts the number of FCSs being found with the
process. As result, most PASS @ k performances drop if we
We will release the processing scripts for replication and comparison. We choose GPT-Neo because it was the only public language model that have been pretrained code when we conduct the experiments.

---

Published as a conference paper at ICLR 2023
50
44.
29-2
32.5
40
34.4
32.9
16.5
40.4
192
22.7
30
25.1
28.2885
155
20.2207206
15. 13.1262222
20
20
Past
Pass55
Pa
Pars@20
Pas10900
100101000
Fsc101
Pass55
Pass8
Pano@20
Phe1893
Phesp 100
[BSel-Smmppigg
(a) GSM555K-Pytthon with GPT-Neo 125M
(b) GSM5.5K-Python with GPT-Neo 2.7B
36.2
25.0
18.5
21.92L.
25.1
25,2285
27.3
2544
90.8
31.5
20.0
20.320.7
15.0
12.0
30
10.0
Psseg1
ParggS
Pa
ParogP20
ParAd 50
PaxaDRO
1sngt
Pass
Pasa
PlaugF20
Paug
2ML
BSett-Samping w/
Salli Swr.---mppiiggg w// FiSHecs
EIIEE
m5 Samplig w/ FCS
Self-Sampping w// FCS+RCS
(c) SaMnOAA-Pytho--tteeeee with GPT-Neo 125M
(d) MaaihQAA-yyhho-ilteeeee with GPT-Neo 2.7B
Figure 2: Percentage of the problems solved (PASS@k) on the dev set of GSM5.5K 11.55--Pyyy 5K-Python and MathhA-Pyyton-FFiiee comparing our self-sampling approach and the common MLE objective. All our methods include partially-correct solutions and use the MLE-Aug loss for learning.
32.3
3.50
27.3
27.9
22.7
23.7
22.7
3.00
PCS
8 PCS
18.7
2.50
0.16
15
2.00
10
5
150
1.00
3.00
2.55
Self-Sampingw// FCS
Selt-Samplin w/ FCS
G/MLE DMMEE Aug
II MML CB MML
Figure 3: PASS@ 100 comparison of various loss functions ( 3.2 under dif-
F35 onl
RC5orly
ferent self-sampling strategies. Results are on the dev set of GSM5.5K-Pyth with finetuned GPT-Neo 125M model. = 0.25 for 3-MML. Full results avail- able as Tab. in Appendix
Figure Number of saved FCSs and PCSs per prob- lem for GSM5. GSM5.5K-Pytie 5K-Python (left) and MathQA-Pythnn Filtered (right), with different self-sampling strategies and model sizes. # FCSs includes the reference solution.
do include partially-corree solutions the buffer, the model learns from smaller number of FCSs and PCSs as targets. The one exception is the GPT-Neo 125M model on the MathQA-Python- Filtered dataset, where we do not observe any advantage/dissadwaaaag of using PCSs.
MLE-Aug loss function works the best. We next study the effects of different objective functions for learning from multiple targets as described in [S 3.2 We also experiment under different self- sampling strategies (i.e., FCS only or FCS + PCS), and our experiment results GSM5.5K-PM --Pthon with the GPT-Neo 125M model are shown in Tab. 5 We can see that MLE-Aug loss results in the biggest improvement compared other losses both with just FCSs and with FCSs PCSs. MML performs the worst: only marginally improves over MLE with only FCS and performs worse than MLE when also learning from PCSs. As discussed in S 3.2 and Tab. 1 the gradient of MML is in proportional to the likelihood given by the model, thus encourages the model to put all weight on one solution in the buffer. As MLE already learns from the gold reference solution, it is hard for

---

Published as a conference paper at ICLR 2023
Original Version PASS@80
Filtered Version
Models
PASS@
PASS@1 1 P
Previous work:
Codex Davinci" Chen el al.
2021a)
6.0
42.0
5.0
40.0
LaMDA 68B" (Austin
2021
79.5
LaMDA 137B
Austin qusstn et al.
2021
81.2
Ours:
GPT-Neo 125M w/ self-sampling FCS + PCS
77.6
84.7
11.7
28.2
GPT-Neo 2.7B w/ self samplingg FCS PCS
20.7
36.2
Table 2: Comparison with previous methods on the original (test set used) and filtered version (dev set used) of the MathQA-Pyther dataset. *: model not pretrained code. t; few-shot learning results.- +: no results available.
MML to make improvements with self- self-sampled solutions, and the performance may decrease when MML puts all weight on an incomplete partially-corres solution. In contrast, the gradients of MLE-Aug objective equally distributed among the targets, which leads to more diversity in its generation due to more balanced source of learning signals. B-MML loss is proposed to alleviate the aforementioned issue for MML loss, but we do not observe an advantage of using it instead of the MLE-Aug loss in our experiments.
4.3 ADDITIONAL ANALYSIS
Diversity of the solutions. By inspecting the & generated solutions for each task, we find that there more diversity in the solutions that the model generates using our method. More specifically, calculate the ratio of unique solutions from the 100 samples for the comparison in Fig. 2a and find that 30.5% of them are unique for our approach but only 20.8% for the model trained with MLE.
Dynamics between # of PCSs and FCSs saved in the buffer. As discussed above, more saved solutions typically results in better PASS @ performance. Interestingly, when comparing different model sizes, we can see that while the sum of partially and fully-correct solutions sampled and saved in the buffer are about the same (i.e. 3.36 and 3.31) for ESM5.5K-Python dataset in Fig. 4| around 60% of them are FCS for the small model while it is 76% for the larger model. The difference in percentage of PCSs left the buffer also reflects the model's ability for completing partially-corred solution prefixes. We also find that during early stages of training, the number of PCSs rapidly grows while the model is relatively weak to sample FCSs, thus the PCSs help enriching the learning signal and preventing overfitting early-on. More discussions about this can be found in Appendix E
Comparison to previous works Here we compare with previous work on both the original and the filtered versions of MathQA-Python datasets in Tab. 2 On the original dataset, self-sampling with GPT-Neo 125M is able to outperform previous methods that finetune 137B model pretrained on natural language. We also compare with Codex model used in a few-shot setting (more details in(Apeendxx A). and find that on the much harder filtered dataset, a 2.7B GPT-Neo model finetuned with our methods obtains much better PASS @1 but with lower PASS 80. By inspecting the output from Codex, we discover that its outputs are much more diverse than finetuned models, which contributes to a higher @0 even under the few-shot setting. Comparison with previous work on the GSM dataset is in Appendix B due to limited space.
5 LIMITATIONS AND FUTURE WORK
More general definition of (partial) correctness. In this work, we define partial correctness based on state-based solution equivalence. This is a conservative way for defining solution equiva- lence as it requires exact match of the sets of variable values, but a solution could be partially correct and yet, not have an exact match of variable values because some these values may not needed for future computation. In the future, find more partially correct solutions in an efficient manner. Besides, our partial correctness defini- tion requires the existence of at least one fully-correct solution and when such reference solution is
want to explore ways to relax this restriction that will help us
8

---

Published as a conference paper at ICLR 2023
not available from the dataset (i.e., in seekky--upervvied setting), would need to first sample an FCS that matches the gold execution result to begin with. In addition, we simply use the matching of execution results to define correctness, which is susceptible to spurious solutions that achieves the correct result by coincidence. For math reasoning, we find such spurious solutions to be quite rare the correct answer is typically numeric which is less likely for semantically wrong solution to obtain the correct answer by chance. But methods as Zhong et al. (2020); Chen et al. (2022) may be needed for this definition of correctness be more robust on other domains.
Towards generating general programs. While we focus on the domain of generating solutions for math reasoning in this work, here we reflect on how our method can be applied to program syn- thesis in general. However, general programs might contain complex structures such as conditions (e.g. se) or loops (e.g., while-do) as opposed to straight-line programs in the math- reasoning domain. Dealing with these complex structures poses additional challenges because most neural program synthesis models perform left-to-right auto-regressive generation, and the changes to the control flow break the alignment between program generation and program execution (Chen) et al. 2018 2021b; Nye et al., 202T). There are two potential ways to extend our technique to ad- dress the problem. First, can treat branch loop as atomic unit (i.e., block whose state is the state after executing all statements within it), then we can apply state-based equivalence in the same way. Second, because technique only requires execution after the full programs generated, we can still evaluate and compare program states based on intermediate states.
6 RELATED WORK
Weakly-supervised semantic parsing. Many previous work in learning semantic parsers from weak supervision follows the same process of sampling programs and maximizing the probability of the correct ones (Krishnamurthy et al. 2017 Guu et al. 2017; Min et al. 2019 Ni et al. 2020). tasks contain one reference solution for each task as opposed
Our work differs final answer like weakly-supervinn semantic parsing tasks. Thus, our work leverages the reference solution for sampling and defines partial correctness based known solutions. Because of the problem setup difference, we found that the conclusions in Guu et al. (207)) about loss functions do not generalize our case.
only the
Execution-guiddet code generation. Our work relates to execution-guidee code generation as we leverage intermediate states of math solutions to guide the sampling process. In code generation lit- erature, intermediate program execution states are used to prune the search space (Liang et al.) 2017] Wang et al. 2018 Li et al. 2022) or condition further generation on the execution states(Chen al, 2018 Ellis et al. 2019 Nye et al.) 22200] Chen et al., 2021b] et al., (221I). The key difference of these methods from ours that they require doing both decoding and execution at inference time, while our work only uses execution during training, which reduces decoding overhead.
Learning from partial reward for program synthesis. There are parallels between multi-target learning and the reinforcement learning setting with sparse rewards for generating programs (Liang eet al. 2017 2018 Simmons- Cadle et al., 2018 Bunel et al. 2018 Agarwal et al. 2019). Similarly, approach of identifying partial correctness of solutions is similar to partial rewards. But instead of discounting an entire trajectory with low reward as in RL, we truncate the solution to a partially- correct prefix and assign it the "full reward", which is a main contribution of this work.
7 CONCLUSION
We propose to let pretrained language models sample additional solutions for each problem and learn from the self-sampled solutions that are correct or partially-correet. We define partial cor- rectness by tracing and matching intermediate execution states. We experiment on different math reasoning tasks and show that such partially-correet solutions can help more efficient exploration of the solution space and provide useful learning signal, which improves the PASS @k performance. Overall, our proposed method can improve PASS @ from 3.1 % to 12.3% compared to learning from single solution with MLE.
'W manually inspected the self-sampled FCSs by GPT-Neo 2.7B 100 tasks of GSM5.5K and found spurious solutions only exist for 3 of them.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
The authors would like to thank Jackson Woodruff, Pengcheng Yin, and the anonymous reviewers for the useful discussion and comments.
REFERENCES
Rishabh Agarwal, Chen Liang, Dale Schuurmans, and Mohammad Norouzi. Learning to generalize from sparse and underspecified rewards. In International Conference on Machine Learning, pp- 130-140. PMLR, 2019.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoc Le, et al. Program synthesis with large language models. arXiv preprint 0007022008..773, 2021.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Au- toregressive Language Modeling with Mesh-Tensorfion March 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901,200
Rudy Bunel, Matthew Hausknecht, Jacob Devlin, Rishabh Singh, and Pushmeet Kohli. Leveraging grammar and reinforcement learning for neural program synthesis. In International Conference on Learning Representations, 2018.
Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin. Jian Jian-Guang Lou, and Weizhu Chen. Codet: Code generation with generated tests. arXiv preprint arXiv:2207. 10399 2022.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda. Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03372 2021a.
Xinyun Chen, Chang Liu, and Dawn Song. Execution- Execution-guided neural program synthesis. In Interna- tional Conference on Learning Representations, 2018.
Xinyun Chen, Dawn Song, and Yuandong Tian. Latent execution for neural program synthesis beyond domain-specific languages. Advances in Neural Information Processing Systems, 34, 2021b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.0233 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 14168, 2021.
Jacob Devlin, Ming-Wos Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers). pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/NII 1423.
Kevin Ellis, Maxwell Nye, Yewen Pu, Felix Sosa, Josh Tenenbaum, and Armando Solar-Lezama. Write, execute, assess: Program synthesis with repl. Advances in Neural Information Processing Systems, 32, 2019.
Catherine Finegan- Finegan-Dollak, Jonathan K Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasi- vam, Rui Zhang, and Dragomir R Radev. Improving text-to-sql evaluation methodology. ACL, 2018.
10

---

Published as a conference paper at ICLR 2023
Kelvin Guu, Panupong Pasupat, Evan Zheran Liu, and Percy Liang. From language to programs: Bridging reinforcement learning and maximum marginal likelihood. In ACL, 2017.
Jayant Krishnamurthy, Pradeep Dasigi, and Matt Gardner. Neural semantic parsing with type con- straints for semi-structured tables. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 1516-1526, 2017.
Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling. Felix Gimeno, Agustin Dal Lago, et. Competition- code generation with alphacode. arXiv preprint arXiv: 2203.07814, 2022.
Chen Liang, Jonathan Berant, Quoe Le, Kenneth Forbus, and Ni Lao. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. In ACL, pp. 23-33, 2017.
Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoe V Le, and Ni Lao. Memory augmented policy optimization for program synthesis and semantic parsing, Advances in Neural Information Processing Systems, 31, 2018.
Sewon Min, Danqi Chen, Hannaneh Hajishirzi, and Luke Zettlemoyer. A discrete hard em approach for weakly supervised question answering. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2851-2864, 2019.
Ansong Ni, Pengcheng Yin, and Graham Neubig. Merging weak and active supervision for semantic parsing. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 8536- 8543, 2020.
Maxwell Nye, Yewen Pu, Matthew Bowers, Jacob Andreas, Joshua B Tenenbaum, and Armando Solar-Lezama. Representing partial programs with blended abstract semantics. In International Conference on Learning Representations, 2020.
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowyez, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:21 14, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou. Wei Li, Peter J Liu, al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(11)) 2020.
Tal Schuster, Ashwin Kalyan, Alex Polozov, and Adam Tauman Kalai. Programming puzzles. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Riley Simmons-Edler, Anders Miltner, and Sebastian Seung. Program synthesis through reinforce- ment learning guided tree search. arXiv preprint arXiv:/806.022 02932,, 2018.
Chenglong Wang, Kedar Tatwawadi, Marc Brockschmidt, Po-Sen Huang, Yi Mao, Oleksandr Polo- zov, and Rishabh Singh. Robust text-to-sql generation with execution-guided decoding. arXiv preprint
2018.
Ruiqi Zhong, Tao Yu, and Dan Klein. Semantic evaluation for text-to-sql with distilled test suites. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 396 411, 2020.
11

---

Published as a conference paper at ICLR 2023
APPENDIX
A EXPERIMENT SETTING DETAILS
Name
MathQA
GSM8K.
Training Steps
50K
25K
Learning Rate (LR)
1.0e-4
Optimizer
AdamW W
Adam Betas
(0.9, 0.999)
Adam Eps
1.0e-8
Weight Decay
0.1
LR Scheduler
Linear w/ Warmup 100
# LR Warm-up Steps
Effective Batch Size
32
Precision
FP 32 125M, FP16 for 2.7B
Gradient Clipping
1.0
Table 3: The hyperparameters used for model training two different types of datasets.
Hyperparameters. All hyperparameters for training is shown in Tab. We use 0.25 in the experiments with B-MMI., as a result of enumeration search among the values of (0.1, 0.25, 0.5, 0.9). We use the default AdamW optimizer settings and slightly tuned the learn- ing rate by trying out several values between 1 1.0e-3 and 1.0c-5. The difference in floating point precision to fit the GPT-Neo 2.7B model into the memory of the GPUs. All experiments are conducted on V100-32GB GPUs.
P & evaluation. We use temperature sampling and sample n solutions with T 0.8, where 80 for MathQA and n = 100 for GSM to evaluate PASS@n, to be maximally consistent with previous work (Austin report PASS @ {5, 10, 20, 50} using the samples and the unbiased estimator proposed in Chen et al. (021a).
al. 202T] Cobbe et af.) 202T] Chowdhery et al. 2022). We also
use = 0.2 to sample solution per specification and evaluate PASS@
Codex few-shot settings. We estimate the Codex (Chen et al., 2021a) performance der the few-shot settings. More specifically, the prompt consists of a natural lan- guage task description "# Generate Python code to solve the following math word probl ems " and four examples, following previous work (Chowdhery et al. 2002). Each example consists of the NL specification one-line comment and the gold program solutions. We evaluate PASS@I k for Codex using the same sampling methods as above.
Details for self-sampling. During a training step, we sample one solution for each task (i.e, natural language problem) in the batch, i.e., | in Alg. and Alg. 2 Thus for each gradient update, we first compute the loss for each task based on the saved solutions in the buffer and loss functions described in Tab. I, then total number of samples we generate per task throughout training is also scaled up by the number of training epochs, which is 235 for MathQA-Pytthn-FFllee for GSM5.5K- S.M55.5-Python. Python. For sampling temperature, we use the same setting as inference time, with 0.8.
averaged across the 32 tasks in the batch. Note that the
83 for MathQA- MathQA-Python and 145
B ADDITIONAL EXPERIMENT RESULTS
Comparing GSM performance with previous work. Here compare method with pre- vious work on the original test sets of GSM8K. The results are shown as Tab. 4 On GSM8K, some the prior works evaluated on a different format NL inputs than ours, so they are not directly comparable, but we still include them to help better position the performance of our meth- ods. We test Codex using the same input few-shot setting, and we find that similar with the
We also experiment with higher sampling budgets but do observe significant improvements.
12

---

Published as a conference paper at ICLR 2023
Models
PASS@1
PASS 100
Previous work:
OpenAI 6B
(Cobbe el al.
2021)
21.8
70.9
Pal.M-Coder 5408
(Chowdhery al. 2022)
50.9
LaMDA 137B" -
Chowdhery ct al. 2022)
7.6
Codex Cushman'
Chen
2021
5.0
58.0
Codex Davinci
Chen et al
2021a
17.0
71.0
Ours:
GPT-Neo 2.7B w/ self-sampling FCS PCS
19.5
41.4
Table 4: Compare with previous methods on the original test set of GSM8K dataset. ": model not pretrained on code. i; few-shot learning results. + different setting from our
Sols. in B
PASS k( %)
Self Self-Sampling
Loss Func.
FCS
PCS
k=
k=
k=10
k=
k=50
k= 100
MLE
7.4
10.6
12.7
15.3
19.2
22.7
MML
1.48
6.9
11.0
13.3
16.0
20.1
23.7
FCS only
MLE-Aug
1.76
7.6
13.1
16.5
20.5
26.8
32.3
8 A-ML
1.57
7.5
11.7
14.5
17.9
23.1
27.3
MML
1.40
1.10
5.5
9.0
11.0
13.1
16.2
18.7
FCS PCS
MLE-Au Aug
2.00
1.36
7.5
13.6
17.5
22.1
29.2
35.0
8-MML
1.62
1.14
7.2
12.0
14.9
18.4
23.6
27.9
with different self-sampling strategies. Results are on the dev set of GSM5.5K K--yytnn with GPT-Neo 125M the base model. Best per- formance within the same category is in bold and ones worse than MLE is underlined. / = 0.25 for 3-MMI.
Table 5: Full comparison of various loss functions
$ 3
method achieves better PASS while being significantly worse in @ 100 compared with Codex. We hypothesize that as Codex model is used tested few-shot setting and not finetuned, does not suffer from the overfitting issue we mentioned. This leads to great diversity but poor accuracy during generation. However, due to the little information we have about Codex (e.g., model size, training data), it is hard derive any further conclusion.
result MathQA in Tab. 2
Ablation results on loss functions. Here show the full results on the ablation of loss functions in Tab. 5] We can see that trends observed from PASS@ 100 in Fig. are consistent with other PASS results, as MLE-Aug loss beats other two loss functions loss when adding PCSs for learning results in worse performance than MLE for PASS as well. Moreover, from the number of FCSs and PCSs saved in the buffer B, we can also observe that using MLF-Aug loss results in more FCSs and PCSs being saved, thus further encourages diversity in generation.
all PASS@A. k. And using MML
C FULL LEARNING ALGORITHM WITH PARTIAL CORRECTNESS
Our general learning framework shown as Alg. I and further extended in $ 3.3 Here we show a complete version of the algorithm with using partially-correct solutions in Alg Additionally, here are the detailed explanation of the data structure and functions used in it: b Mapping M: This is a data structure that maps an intermediate state to a set of solution (prefixes) that execute to that state, i.e., M S y In this mapping, we save all PCSs and their intermedi- ate states, including all prefixes of any PCS. We use this to significantly speed up the lookup process as mentioned 3.3.2.
Function CariaaCCoreccnnssSrtteri M): Since all states for all known PCSs saved in M, to know whether a prefix UEi is partially-correc, we only need to check if its state matches any
Natural language explanations of the solutions are used as input and the few-shot exemplars are not in the format
13

---

Published as a conference paper at ICLR 2023
Algorithm 3 Training Update with Partially Correctness
Initialize: (only once before training starts)
Solutions buffer B {v" ,y" with empty and the reference solution Reference solution states (si $2. (.)))) where sf 7 (Vss)
State-prefixes mapping
{s] - (ys.)kim
Input:
Parameterized model Po(y)a))
A training example Vz.,',,² :²")
Tracing function T /: 3 obtain intermediate states
1: Y <- SampleSolutions(z, Po. Alg. %
2: for in Y do
for i + 0
4: 8; +- ( 1 get intermediate state for each solution prefix Øsa */
5:
if
M) then
6:
Ys .M(s.) f get existing prefixes that executes to state s; +/ if isDuplicate (g Ys) then
8:
+ updateBuffer
9:
M - uppateMupping(Esst M)
10: end if
11: continue / only need the longest matching prefix */
12: end
13: end for
14:
end for
15: VoL(r, B, Po)
of the known states for PCS, ie.., if & M
AST and length constraint rule out "trivial variants" and identify new PCSs to save in the buffer B. Here the solutions to compare are the of solutions Ys that reaches the same intermediate state, i.e., being state-based equivalent; B): Here we not only need to add the new PCS into the buffer B, but
Function isDuplicate (ū Ys): As mentioned $ 3.3.2
> Function updateBuffer( also need to prune out the saved solutions that are prefix of Úı: & Function
(
Here we need to save the states of all prefixes of an identified
partially-corres correct solution, thus we will loop through all prefixes of tıí and obtain its execution state, then update M accordingly. As mentioned above, existing PCSs may be prefix of the new PCS, so we also need to prune out such existing PCSs from mapping M.
D QUALITATIVE ANALYSIS
In Tab. 6 show more examples of the fully-correct and partially-coree solutions that the mod- els found during self-sampling, from both the MathQA and GSM datasets. First, we can see that for some NI. problems, it is possible that FCS PCS can be found with self-sampling, MathQA-EEample / and MathhAA-Example tion quite straightforward thus leaves very little room for the existence of other correct solu- tions, as the reference solution is already very short. Moreover, we can also observe that the ways self-sam self-sampled FCS and PCS differ from the reference solution vary a lot. In MathQA Math(AA-Exampe- GSM 1MM-Exxmple-I and GSM-ExaMpaa Example-2 the sampled FCSs complete the task with very different paths compared with the reference solution, and actually result using fewer lines of code. Another way of getting FCS or PCS is to perform small and local perturbations, e.g., switch the two sides of addition or re-order the two non-dependent statements, as shown in other examples. We find that these local perturbations are more common in general in both datasets, such patterns are easier for the model to learn.
in Take Math@A-Example-2 as an example, the ques-
E TRACKING TRAINING PROGRESS
Learning from self-sampled solutions mitigates overfitting. Here shown the PASS@ per- formance curve with respect to the training process in Fig. From the curves, we can observe that for MILE, while PASS and PAss@5 generally improves during training, other PASS @k for
14

---

Published as a conference paper at ICLR 2023
NL Problem Deseriptions
Ref. Solution
Self-Sampled FCS
Self-Sampled PCS
MathhA--xxample-I)
n0=70.0
rC=70.0
The charge single room hotel is percent less than the charge for
n1-10.0
n1=10.0
t0=100. D-n0
t0=100. 0-n1
single room at hotel R and 10 pereent less than the charge for single room at hotel G. The charge for single room
t1=100. C-n1
t1=100.0-a
t2=t0/t1
t2=t0/t1
t33-22+100.0
t3-t2.100.0
hotel R what percent greater than
1:-100.0-t.3
answer=t3-100.0
the charge for single room at hotel G?
t5-t4/tt3
answer=t5<100.0
n0=9.0
If john runs in the speed of 9 km/hr from his house, what time will he reach the park which 300m long from his house?
n1=300.0
t0=n0+1000.0
tl=n1/t0
answer=t1+60.0
n0=15.0
n0=15.0
n0=15.0
A class consists of 15 biology students and 10 chemistry students. If you pick two students at the same time, what's the probability that one maths and one is chemistry?
n1=10.0
m1=10.0
n1=10.0
t0=n0+n1
t0=n0+n1
t0=n0+n1
t1=n0/t0
t1=n0/t0
t1=n0/t0
t2-n1/t0
t2-n1/t0
t2-n1/t0
t.3-t0-1.0
t33-00-1.0
t3-t0-1.0
t4-n1/t3
t4-n1/t3
t4-n0/t3
t5=n0/t3
t5=n0/t3
t5=n1/t3
t6=t1+t4
t6=t1*t4
t7=t5+t2
t7=t5*t2
answer=t6+t7
answer=t7+t6
(GSM-Exumlle-1):
n0=2
n0=10
n0=10
Ellie has found an old bicycle in field and thinks it just needs some oil to work well again. She needs 10ml of oil to fix cach wheel and will need an- other 5ml of oil to fix the rest of the bike. How much oil does she need in total fix the hike?
n1=10
m1=5
n1=5
n2=5
t0=n0+n1
n2=2
t0=n0+n1
answcr=n0+t0
answer=t0+n2
(GSM-Exxmpple
n0=20
n0=25
no-2
There very little traffic Happy
n1-100
nl=2
nl=25
Street. During the week, most cars
n2-25
n2-20
n2-20
pass on Tuesday 25. On Mon-
n3=2
m3=100
n3=100
day. 20% less than on Tuesday, and on Wednesday. more than Mon-
n4=10
m4=10
n4=10
t0=n0/n1#n2
tC=n0-n1
day. On Thursday and Friday, it about 10 cars each day. On the week- end, traffic drops to 5 cars per day. How many cars travel down Happy Street from Monday through Sunday?
t1=n2-t0
t1=n2/n3*n0
t2-t1+n3
t2=t0-t1
t3=n4+n3
t3=t2+n4
t4=t0+n3
t4=n0-t3
answer=t3+n2
answer=t4tn3
+E2+t3+14
Table 6: More examples of self-sampled fully-correct (FCS) and partially-correct solutions (PCSs). "MathQA" denotes the MathQA-Pythan Pet---yttoottteeecee Filtered dataset and "GSM" denotes the GSM5.5K-Python dataset. All solutions from the final buffer after training GPT-Neo 2.7B model, while learning from self-sampled FCS+PCS with the MLE-Aug loss.
higher A actually decreases after reaching the peak performance in early epochs, which is consis- tent with previous findings (Cobbe et al., 2021). This is due to overfitting: in the early stage of training. the model is less confident about its predictions thus the sampled & solutions verse, and while training continues, it overfits to the one reference solution provided for learning thus leads poor generalization when evaluated by PASS @ with high k values. Fig. also shows
very di-
15

---

Published as a conference paper at ICLR 2023
passga1 FOR-RR-NLELLIN
passg5
FCP-MAL
.LE
FOP-MLELNN PEPHECP-PLELAIEE FCP-MP MLLE
0.00
0.1
0.04
0.0
0.4
0.2
pass@No
passg20
-FCP-MEEAAD FOP-RCP-LLELNN NN FEP-WNL
ALE
- -FOPHLELLNN PCPHEC--PEE FCP-MML MLE
03
0.1
0.
tranerg stpp
035
tratnegbotal stop
12k
33%
104
324
pass@5o POP--EEEEEEN FCP-MNL
passgr10
-FCR-NEEAA4
FOP-NLE.NNE RCPHECP-PLE FCP-IPL
0.5
04
026
0.5
616
53
toaner stop
0.1
10k 32k 30%
21
30c
254
200
Figure 5: How PASS @ on the dev set evolve during training. Results shown CSM5.5K-Python dataset with GPT-Neo 125M model. Exponential moving average smoothing is applied for more clarity, but original curve is shown in shade.
how our proposed self- saff-aan method can mitigate the overfitting problem, as it keeps improv- ing or maintaining PASS@ (5, 10, 20) while such performances start decreasing for MLE. Though also shows improvements for PASS @ (50, 100), but the performance still decreases in later training stages. Here we can also see the importance of suitable learning objective, as MML has almost no effect in mitigating such overfitting issue.
Early stopping is needed when prioritizing high k value for PASS@k. In our experiments, we select the model checkpoint with the best PASS @1 performance evaluate all PASs@k. This setup aims to choose the best model that can solve the task with a small number of attempts (which cor- responds to smaller value). as studied in (Austin et al., (002). We can also observe that with our methods, the best PASS @1 checkpoint also yields the best or close to the best PASS @ (5, 10, 20) per- formances. However, in certain applications where large number of attempts are allowed, PASS @ with high i values should be prioritized. An example is to generate candidate solutions before reranking (Cobbe et al.) 202I). In this case, an earlier checkpoint (e.g., one with best PASS@ 100) should be used instead, which is not the best checkpoint for PASS @ l where k is small. Also note that our proposed method are not suitable for these applications, as we observe no improvement on the peak PASS @ (50, 100) performances. We think this because when such peak performance is reached, it is still in the early stage of training thus not many FCSs or PCSs have been saved in the buffer yet.
cartially-corred solutions help in early training stages. To show how self-sampling effects training. in Fig. 6a we show how the size of the buffer progresses during training. From the curves, can see that in the early training stages (L.e first 5k steps), the number of saved PCSs rapidly grows while the number of FCSs only slightly increases. In later stages of training, the growth of buffer size is mainly contributed by more FCSs being sampled and saved while the number of
16

---

Published as a conference paper at ICLR 2023
Not Executable mLengthy Noc Cornect Dupllicace
-Oplicatee New FCS PIS
25
080
11
04
02
03
(a) Growth of the number of saved FCS and PCS dur- ing training. FCS includes the gold solutions.
(b) Distribution of the characterization of self-sampled solutions during training.
Figure 6: How samm-igg evolves throughout the training process. Results shown as training the GPT-Neo 125M model the CSM5.5K-Python dataset with MLE-Aug loss.
PCSs stays steady. Also when compared to learning only with FCSs, learning with FCSs PCSs eventually accumulates more FCSs in the buffer (green dotted line vs yellow dotted line). In addition, show how the distribution of the outcomes of self-sampled solutions changes throughout training Fig. 6b We can see that in the early training stages, the ratio of not executable/incoreee solutions quickly drops to almost zero. At the same time, the ratio of new FCS PCS being saved reaches the peak. As training proceeds, the models are mostly sampling known FCS or PCS as the size of the buffer converges as well. But the number of self-samples fully-correct solutions gradually overtakes the partially-correct ones.
17


Reviews:
[{"review_id": "rE7QZCZSis6", "reviewer": "Reviewer_2ima", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Making pre-trained language models (PLMs) generalize to perform multi-step math reasoning problems is a challenging task. One of the main challenges is that during the fine-tuning stage, only one reference solution is used as supervision, preventing the model from exploring a more diverse solution space. To mitigate this issue, this paper proposed learning from self-sampled solutions. The self-sampling stage yields both fully-correct solutions and partially correct solutions, leading to more efficient exploration of the solution space. They conducted experiments on two math reasoning datasets, namely,  MathQA and GSM8K. The experimental results show that with training on FCS and PCS, the generated solutions are more diverse and improved the Pass@K over the baseline when $k$ is large. They also compared different loss functions and found that MLE-Aug loss function works the best when training with multiple solutions.", "summary_of_the_review": "In general, this paper is well-motivated and contains several interesting ideas. However, I think the experiments are not convincing enough to show the generalizability of the proposed method (see weaknesses for details). At the current stage, I don't think it is not fully qualified as a top-tier conference publication. ", "strength_and_weaknesses": "Strengths:\n- This paper successfully integrates several of the ideas in the literal, e.g., learning from execution, and semi-supervised semantic parsing, and improves the model performance on complex math reasoning.\n- I liked the idea of using partially-correct solutions to encourage the model to explore the solution space. However, it would be better if the authors could explore a more general definition of the PCS.\n\nWeaknesses:\n- One of my major concerns about this paper is that the experiments were only conducted on two similar math reasoning datasets. Theoretically, the method could be generalized to other semantic parsing tasks like text-to-SQL and more general code synthesis. It would be more convincing if they can show the proposed method also works on other datasets/domains. \n- I am also not totally convinced by the improvement presented in Sec 4.2, where the authors use Pass@k as the evaluation metric. We see that when $k$ is small, the improvement is quite marginal. It is quite possible that although the generated solutions are more diverse, the correct solution only appears a few times. I think using the majority vote of the generated $k$ solutions as the answer would be a better metric because, in the real use case, we only want to give the users a few solutions.\n \nOther comments:\n- One experiment I would like to see is *Number of saved FCSs and PCSs vs. Training iterations vs. dev performance*. When is the model able to sample diverse solutions? Do the diverse solutions decrease the convergence speed while improving the dev performance?\n- In Table 2, the comparison to Codex is not fair as you can easily achieve much better performance with chain-of-thought prompting.\n- I think the comparison of the different loss functions is not closely tied to the main idea of this paper. It would be better if the authors can use more space to explore the boundaries of the proposed method. For example, experiment with other semantic parsing tasks. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iaTRcBc-ekE", "reviewer": "Reviewer_bPyF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the application of language models (LM) to mathematics reasoning tasks, noting that many existing datasets only contain one reference solution. This is problematic since many derivations may lead to the same answer, and therefore maximum-likelihood estimation of the LM may lead to overfitting. The main idea is the observation that samples generated from the model during training (“self-sampled”) can be evaluated to see if they yield the correct answer, and if so those alternate solutions may also be rewarded for a suitable choice of loss function. To this end, three loss functions are compared: MLE augmented with the additional samples (MLE-Aug) yielding the correct answer, maximum marginal likelihood (MML) which marginalizes over the candidate set, and smoothed MML which interpolates between MLE-Aug and MML.\n\nAdditionally, the paper proposes learning from partially-correct solutions. The challenge is to identify sequences which contain a prefix which evaluates to a necessary step of the correct solution. To do so, the idea is to exploit both self-sampled (correct) solutions and the gold reference to identify semantically equivalent prefixes (produce the exact same set of variable values). The longest partially-correct prefix is used for learning, which is distinguished from other sequences only by the lack of a distinguished end-of-sequence token.\n\nExperiments on MathQA-Python and GSM5.5K-Python demonstrate that both self-sampled solutions and the inclusion of partially correct solutions improve model performance. The MLE-Aug loss function outperforms the MML losses in all cases. In addition, the resulting model produces diverse solutions, which likely contributes to the improved pass@k results.\n", "summary_of_the_review": "While the scope of the paper is fairly limited (math reasoning), the contributions are worthwhile and soundly evaluated. There is a story (discussed in the limitations) for broader applicability of the proposed ideas. While the relatively small model sizes considered may limit the absolute performance of the model, this is also a strength when it comes to reproducibility, so in the balance I would say this is a positive. ", "strength_and_weaknesses": "Overall, while the contributions are straightforward and, in the case of learning from partially-correct solutions, of somewhat limited scope (doesn’t generalise to more general coding tasks), the paper addresses important limitations of existing training methods for math reasoning tasks.\n\nRegarding the proposed multi-target learning objectives, it’s not clear why MML would be expected to work. If multiple solutions produce the correct output, shouldn’t they all be rewarded? As in contrastive learning, wouldn’t we want to reward correct solutions, but downweight the probability mass assigned to incorrect ones? Additionally, I would have liked to see other objectives considered, such as ranking-based objectives or sequence-level (e.g. minimum bayes risk).\n\nRegarding learning from partially-correct solutions, the additional filtering required (line 8 in Alg. 1) seems like an important limitation, given the footnote about the impact on performance. In general, identifying “trivial variants” seems like a hard problem, given the potential for malformed code during training which could make it hard to filter based on compiled ASTs. \n\nIn the limitations section, the issue of spurious solutions which achieve the correct solution by chance is raised. While it’s stated that this issue is “rare” in the domain under consideration, how was this quantified? Furthermore, given the persistent buffer of solutions maintained during training, is there a concern that early versions of the model may be more likely to produce spurious solutions, which would then remain throughout training in the solutions buffer?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "O13a4tU5_1", "reviewer": "Reviewer_dnGk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an approach to solve math word problems (MWPs) via fine-tuning with self-sampled solutions that can be partially correct. The approach relies on a text-to-code formulation: a generative model, which is to be fine-tuned, receives the MWP text and aims to generate a snippet of python code, which, upon execution, returns the solution to the input MWP. Under this formulation, the paper proposes to augment the training data with samples from the generative model itself. Samples whose computation returns the correct solution to the problem are marked as correct and are used as additional data for fine-tuning. Samples whose computation returns an incorrect solution but some intermediate steps are correct are marked as partially correct and are also used for fine-tuning. The paper establishes a method to find such partially correct solutions (via backtracking the implicit computation state of the generated code/program). Evaluation results suggest that fine-tuning with self-sampling and with partially correct solutions both contribute to improved performance. ", "summary_of_the_review": "The paper describes an interesting approach (text-to-code, self-sampling with partially correct solutions) to solve math word problems. I have some concerns on the problem formulation compared to the (seemingly more natural to me) text-to-text approach. I am very open to increase my score. ", "strength_and_weaknesses": "### strengths \nIn my opinion (authors please correct me if I'm wrong) the proposed approach seems like a clever data augmentation method. The technical contribution lies in finding partially correct solutions, which, thanks to the problem formulation, can be identified through the computation state of the generated solution (a code snippet/program). More broadly, I think this approach can be useful beyond MWP solving; for example, I can imagine applying similar approach to identify partially correct code/programs, which can potentially be useful for improving text-to-code models, with additional implications for computer science education for identifying and hinting a learner's partially correct code submissions.\n\n### weaknesses\nI have a question on the problem formulation. There is certainly nothing wrong with formulating the MWP solving as a text-to-code problem. My concern is: is this the best way to solve MWPs? Or put another way, is MWP the best setting to showcase the proposed method's effectiveness? In my opinion (correct me if I'm wrong), the trending approach for MWP solving specifically (I'm not talking about other math reasoning tasks) is to take advantage of models trained on natural language and generate solutions in natural language. This is because much of MWP's reasoning involves simple mathematics (often simple arithmetics, i.e., the two datasets that the authors considered) but a lot more commonsense reasoning and natural language narratives. I think there is a reason that recent methods take advantage of language model's capability in generating texts, and tackle MWP solving via chain-of-thought prompting, majority voting, etc., all of which rely on generating natural language texts that lead to the final solution. \n- Could the author comment on why they take the text-to-code approach, rather than text-to-text approach, for MWP solving, and the pros and cons of each?\n- Additionally, it would be helpful to compare these two approaches experimentally, although I understand it could be very laborious. Note that currently some of the better performing approaches for MWP solving can reach solution accuracy of > 50% [1, 2]\n\nOne more question: is it the case that the correctness criterion for a sampled solution only rely on whether the execution of the sampled solution gives the correct final answer? If this is the case, the fine-tuning data perhaps, and inevitably, will include samples whose final result is correct but intermediate computation step is wrong. How would this scenario impact the performance of the proposed approach?\n\n\n[1] https://arxiv.org/abs/2110.14168\n\n[2] https://arxiv.org/abs/2206.14858", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DsGPJjZPhxq", "reviewer": "Reviewer_bc3k", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies finetuning on model sampled solutions to coding problems. The idea is to sample python solutions to problems, keep solutions that return the correct result in a buffer and finetune on them. The authors experiment with a few different losses for finetuning on the self-sampled solutions and evaluate their approach on two math reasoning datasets. They find that while pass@1 is not always improved from this approach, pass@k (for k >1) shows decent improvement.", "summary_of_the_review": "I am going with a weak accept. See weakness section for explanation. Experiments on more code generation problems (where pass@k is more reasonable metric) would have been more convincing, but I like the overall approach of this paper.", "strength_and_weaknesses": "Strengths\n- The idea is really interesting and there are some nice improvements in pass rates for k > 1.\n- I liked the exploration of the different loss functions as they tend to weight self-generated samples differently.\n- The approach to keep partially correct solutions is also interesting as it potentially allows for bigger solution set to finetune on.\n\nWeaknesses\n- The problem seems to be very code motivated, so it is not clear to me why more code datasets were not tried instead of math datasets (which are converted to code).\n- pass@1 is not generally improved by this approach. This is a bit unsatisfying as, at test time, given a math problem one does not typically have test-cases to run the generated code against and verify the answer, so it is really pass@1 that matters more than any pass@k. Another concern is that maybe the pass@k just improves because there is more data (not necessarily correct) to finetune on that improves the diversity of model samples.\n- Experiment details are not fully explained and it could be tough to reproduce.\n\nQuestions for authors:\n- What if you were to just sample solutions and finetune on them without caring about correctness? Will this yield an improvement on pass@k as well or is it much worse? \n- The supplementary says that only one solution is sampled per problem. This is a bit confusing. Are you saying that one sample is enough to get a correct solution from the model? Please clarify: (1) how many samples are there for each problem (2) how many problems are in a gradient update (3) how many solutions of those sampled execute to correct answer (as a function of training steps).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster