Solving Quantitative Reasoning Problems with Language Models
Authors: Aitor Lewkowycz
Anders Johan Andreassen
David Dohan
Ethan Dyer
Henryk Michalewski
Vinay Venkatesh Ramasesh
Ambrose Slone
Cem Anil
Imanol Schlag
Theo Gutman-Solo
Yuhuai Wu
Behnam Neyshabur
Guy Gur-Ari
Vedant Misra
Venue: NeurIPS 2022 Accept (2022)
Keywords: language models
quantitative reasoning
transformers
math and science word problems

Abstract: Language models have achieved remarkable performance on a wide range of tasks that require natural language understanding. Nevertheless, state-of-the-art models have generally struggled with tasks that require quantitative reasoning, such as solving mathematics, science, and engineering questions at the college level. To help close this gap, we introduce Minerva, a large language model pretrained on general natural language data and further trained on technical content. The model achieves strong performance in a variety of evaluations, including state-of-the-art performance on the MATH dataset. We also evaluate our model on over two hundred undergraduate-level problems in physics, biology, chemistry, economics, and other sciences that require quantitative reasoning, and find that the model can correctly answer nearly a quarter of them.

Solving Quantitative Reasoning Problems with Language Models
Aitor Lewkowycz, Anders Andreassen, David Dohan', Ethan Dyer', Henryk Michalewskil, Vinay Ramaseshi, Ambrose Slone, Cem Anil, maaol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur", Guy Gur-Ari', Vedant Misra"
Google Research
Abstract
Language models have achieved remarkable performance wide range of tasks that require natural language understanding. Nevertheless, state-of-the-ae models have generally struggled with tasks that require quantitative reasoning, such as solving mathematics, science, and engineering problems at the college level. To help close this gap, we introduce Minerva, a large language model pretrained on general natural language data and further trained on technical content. The model achieves saate-oftthe-e art performance on technical benchmarks without the use of external tools. We also evaluate our model on over two hundred undergraduate- level problems in physies, biology, chemistry, economics, and other sciences that require quantitative reasoning, and find that the model can correctly answer nearly a third them.
Introduction
Artificial neural networks have seen remarkable success in a variety of domains including computer vision, speech recognition, audio and image generation, translation, game playing, and robotics. In particular, large language models have achieved excellent performance across a variety of natural language tasks including common-sense reasoning, question answering, and summarization [1, 2, 4, 5]. However, these models have struggled with tasks that require quantitative reasoning, such as solving mathematies, science, and engineering problems [6, 8J..
Quantitative reasoning problems they test the capability of models on several fronts. They require the solver to correctly parse a natural language input, potentially recall world knowledge that pertains to the problem, and apply an algorithm or series of computations to the information provided in order to arrive at correct solution. They also require that the solver able correctly parse and generate precise sequences of mathematical tokens, as well as apply computational procedure to tokens via symbolic and numerical manipulation. Finally, such problems are proving ground for research toward robust quantitative reasoning solvers that are useful in supporting the work of humans in scientific and technical fields.
interesting domain of application for language models because
Previous research has shown that large language models achieve impressive performance on math and programming questions after training domain specific datasets [9, 10, 11]. In this work, we apply this approach to quantitative reasoning problems, in which the model required to provide complete and self-contained solution, without relying on external tools. Such tasks include mathematics word problems, competition mathematics evaluations, and many problems in science and engineering.
"Equal leadership and advising contribution
1 Equal contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Question: A line parallel to y 4x + 6 passes through (5, 10). What is the y- coordinate the point where this line crosses the y-axis?
Question: Assume that variance of first nat- ural numbers variance first m even natural numbers is 16. Compute
Model output: Variance of first n natural numbers is n²=1 Variance of first
Target: Since the line is parallel to 4x + 6, we know that its slope 4. Thus
natural numbers is 22(m²-1) So, _²= 10 12
the equation of the line is y 4z + b where b the y-intercept. Plugging (5, 10) into this gives 10 11-5-- b = b what wanted.
12
12
and (m 16. Solving for n and m, get n and m So, m + n 18.
which is
Model output: The line is parallel to y 4x + 6, which means that it has the same slope = 4x + 6. Since the line passes through (5, 10), can the point-slope form of line find the equation of the line:
Question: Prove that for every a, b, b & a. holds 4. (atb)2
Model output: We have
y 10 (xx 5)
a² 2
+
b²
++0
y 10 4x 20
2
4x 10
a² + b²
a²
+ 2ab +
2
4
Now that have the equation of the line, we find the y-coordinate of the point where the line crosses the y-axis by substituting x = 0 into the equation:
2a + +22
>
2
+
2ab -b
a² 2ab + >0
(a 6)2 > 0,
= 4 10 10
which is true for all and b.
Figure 1: Example questions with solutions from Minerva 62B. These solutions were selected by majority vote over 256 samples, majl @ @56... All questions were prompted with the same prompt 4-shot prompt from MATH; see appendix). Left: Question from the MATH dataset (Algebra 4). The model derives the solution by writing down a line equation, simplifying it, substituting æ = O, and solving for y. The model outputs valid ВTT: along with natural language explanations. This detailed approach is different from the one used in the ground truth target. Top right: Example question from the Joint Entrance Examination Main Math 2020 exam taken cach year by almost 2M Indian high-school students intended study engineering and similar fields. Bottom right: Example question from National Math Exam Poland (May 2022). The exam is taken by approximately 270K high-school students every year.
1.1 Our Contribution
We present Minerva, language model that achieves strong performance on many quantitative reasoning tasks. Our model is able to process scientific and mathematical questions formulated in natural language. and generate step-by-step solutions using correct ІТиК notation. Figures and show example responses from Minerva to mathematics and physics questions.
Minerva is based on PaLM general language models [5] that are further trained on a high-quality dataset containing scientific and mathematical data. In particular, we start with 8B, 62B, and 540B parameter pretrained models, and continue training them on our technical content dataset. We achieve ffate-oftthe-- performance on MATH [6]. GSM8k [7], and STEM subset of the MMLU [12] dataset, all of which include math and science questions formulated in natural language. Notably, our models achieve robust performance in a few-shot setting with no explicit training on these datasets.
The main novelty of this paper is the use of large training dataset that juxtaposes natural language with correctly-formatie preprint server and from web pages that we carefully process to minimise the loss of mathematical content. In addition to achieving state-of-the-art results, our work sets a new baseline for the performance achievable on quantitative reasoning by increasing data quality and model size.
mathematical content in ETT₃X format. The data is collected from the arXiv
To further increase the coverage of quantitative reasoning benchmarks, we build a dataset of over 200 undergraduate--lvee questions in science and mathematics from MIT's OpenCourseWare (OCW).

---

particle moves so that is at (3 sin { cos 2) at time Find speed of particle, measured in unit of distance per unit of time.
Question:
Target: The parametric curve (Ir, y) = sin { cos {) describes a circle with radius 3. Furthermore, it makes full revolution at time 8x. (figure) Therefore, the speed of the particle is 4
The speed is the magnitude of the velocity vector. Therefore, we must first find the velocity vector, which is the derivative of the position vector. We have
Model output:
3sin t 3 cos
t sin
3
3
dt 4
4
3
cos
4
4
4
COS
4'
4
sin
4
Therefore, the magnitude of the velocity vector is
((
3 cos
t 1
9
9
t
M
+
-
sin
1
=
V
16
cas
2
16
sin
4
3 cas?
3
V
+
sin?
4
Therefore, the speed of the particle is
Figure 2: Example question from the MATH dataset (Precalculus 4), along with the ground truth target and the Minerva solution. The model correctly takes the derivative of a position vector compute the velocity. then computes the norm of the resulting vector, uses a trigonometric identity to obtain a final numerical answer. Minerva takes a completely different approach from the ground truth solution. The model output was not modified, except for few newlines that were removed to save space.
This provides a measure our model's quantitative reasoning abilities a chain-of-thought context beyond pure mathematical setting.
1.2 Related Works
Solving quantitative reasoning problems expressed in natural language has been active area of study [13, 14]. Prompting language models using scratchpad [15] or chain-of-thought [16] solutions can lead them generate step-by-step solutions unseen problems. The GSM8k work [7] showed that training verifiers to rerank model outputs can lead to improved performance. The original version of GSM8k included special syntax for algebraic caleulations, which were processed by a calculator. In this work we focus on self-contained models without access to external tools.
The standard method for evaluating language models on generative tasks is to greedily sample one solution per problem. Recent works [9, 17, 18, 19] have shown that it is advantageous to sample multiple solutions per problem, and then filter those down to final answer. We find that majority voting [19] significantly improves performance over greedy decoding.
The work [11] includes an evaluation of davinci-002, OpenAl's
's latest publicly available language model, on subset of 90 problems from the MATH dataset. Due to the focus on subset of questions, as well as changes made to the way questions are formatted, difficult to directly compare our results with those of [11]. J. In Section 3, we compare OpenAl davinci-002 with our models under the same experimental conditions.
Code generation. Applying code generating models to mathematical problems has been active area of exploration. PaLM [5] showed that large language model with code in its training dataset can achieve good performance on a code version of GSM8k. Furthermore, the Codex model [9] can generate code solutions to MATH problems []]]. These solutions often rely on external libraries to perform mathematical operations such as solving equations or taking limits. This is complementary approach to ours, in which we directly probe the model's s ability to arrive at an answer by relying only its reasoning capability.
3

---

Formal mathematics. Mathematics developed discipline based in natural language, but its axiomatic fundamentals make possible to simulate mathematical thinking. This can be achieved using specialized programming languages that facilitate the simulation of logical and mathematical
thinking using a computer, such as Coq [20]. Isabelle [21], HOL.4 [22], Lean [23], Metamath [24] and Mizar [25]. Work on automation of proof assistants and automated theorem provers such as
[26], leanCoP [27], and Vampire [28] has substantially benefited from integration with machine
learning methods [29, 30, 31, 32, 33].
Language models applied to formal and synthetic mathematical problems. Previous work trained language models to predict mathematical expressions [34, 31, 32, 35, 36, 37, 38, 39]. In turn, such a predictive model can be used to guide proof search, as done by [32]. Large language models excel in modelling natural language, though in the case of formal languages, models that facilitate retaining information about the graph structure of given mathematical formula, such GNNs, are still very competitive.
Modelling mathematics as discipline of natural language. New benchmark datasets [6, 40] cover more advanced mathematical topics. For specific subtasks that pertain to understanding mathematics in natural language, such as searching for related equations, is possible to use graph convolutional networks [41]. However for directly solving such tasks from natural language, language models are facing limited competition from other classes of models.
2 Training and Evaluation
2.1 Mathematical Training Dataset
Our models were trained on a dataset of 38.5B tokens from webpages filtered for mathematical content and from papers submitted to the arXiv preprint server. In addition, the dataset includes general natural language data, which is randomly sampled from the same dataset that was used for pretraining PaLM. Our mathematical webpage dataset was constructed by collecting pages that contain mathematical expressions MathJax format. The pages underwent cleaning process that removes most HTML tags but preserves natural language well as mathematical notation, including LTTX symbols and formatting. The result is that mathematical formulae like e + = or E me² are presented in full to the model during training. This procedure makes it possible for the model to perform well on tasks that require calculation and symbolic manipulation. Table 1 provides a breakdown of the training dataset. See Appendix B for more details.
Table 1: Proportion of data, and number of tokens, from each source the technical training dataset. The General Natural Language dataset is a subset of the dataset used to pretrain the model.
Data source
Proportion of data
Tokens
Present during pretraining
Math Web Pages arXiv General Natural Language Data
47.5%
17.5B
No
47.5%
21.0B
No
5%
>100B
Yes
2.2 Models and Training Procedure
Our approach is to start with the PaLM pretrained decoder-only transformer language models [5] and further train (finetune) them on our mathematical dataset using an autoregressive objective. Table contains the main model and training hyperparameters. The largest model, with 540B parameters, was finetuned on 26B tokens. While this model is highly undertrained compared to the 8B and 62B models, it still achieves superior performance. Additional details can be found in Appendix C.
2.3 Evaluation Datasets
We mainly focus few shot evaluation, though see Appendix E.3 for a discussion of finetuned evaluation. For evaluation, we truncate the inputs from the left to 1024 tokens and we use the model to generate up to 512 tokens. When sampling once per problem, we sample greedily. When sampling

---

Table 2: Model architecture and continued training hyperparameters. Model training was resumed from pretrained PaLM models, and the number of steps quoted refers only to continued training on our technical dataset.
Model
Layers
Heads
dmedel
Parameters
Steps
Tokens
Minerva 8B
32
16
4096
8.63B
624k
164B
Minerva 62B
64
32
8192
62.50B
416k
109B
Minerva 540B
118
48
18 432
540.35B
399k
26B
MATH Subtopics
Accuracy MNLU Subtopics
Minara mallgge
60
Pggl 528
Minancs 510
Mincra 928 majtas
60-
40
507A OpenAl davind
20
20
IntermedAAg
Homtheery
PreCakulia
Probabiity
Geometry
Algetia
Prodipubraa
Grand
Engineering
Chembory
Math
Physic
Computer Science
Blatngy
Overall
Figure 3: Performance on MATH and MMLU-STEM by subtopic. Minerva achieves state-of-the-art results on both datasets. ma @k denotes evaluations where k samples were generated for each problem and only the majority vote (most common answer) was selected [19]. For MATH, 256 for Minerva 8B and 62B, and 540B. AMLU-STEM, k davinci-002 the latest publicly available model from OpenAI.
multiple times per problem use nucleus sampling [42] with temperature T = 0.6, p 0.95. For generative tasks, the model produces a chain-of-thoupg answer and demarcates a final answer. We evaluate solution as correct if the final answer matches the ground truth solution, independent of the quality of the chain-of-thought preceding it. To evaluate correctness, we parse the final answers and compare them using the SymPy library [43]. This done order correctly identify answers that are mathematically equivalent such as 1/v3 and v3/3. See Appendix D.I for further details.
The existing datasets which focus
MATH: dataset of 12K middle school and high school mathematics problems [6]. Problem statements are written in ПЛрХ. We prompt the model with fixed 4-shot prompt (listed in Appendix D.2). This prompt includes four random examples from the training dataset whose ground truth targets are not too long.
GSMSk: middle school math word problems [7]. Models are evaluated using the chain-of- thought prompt from Wei et al. [16]. Previous models evaluated on GSMSk made use of an external calculator. In this work, our model does not have access to any external tools.
MMLU-STEM: subset the MMLU dataset [12] focused on science, technology, engi- neering, and mathematics (STEM). For the original version, we use the 5-shot prompt from the development set for each task. We also consider tfthuuh prompting for this task, where we prompt the model with examples that include step-by-step solutions. We use a multiple-choic version of the MATH prompt for topics that involve mathematical reasoning, and add step-by-step solutions the standard 5-shot prompts for the rest of the topics. See Appendix G for more details.
2.4 Undergraduate-Leva STEM Problems
To evaluate the scientific reasoning capabilities of Minerva, we harvested a set of STEM problems at the undergraduate level, most of which involve multi-step reasoning, which we refer to in this paper
5

---

Table 3: Model performance on several quantitative reasoning datasets. For majority voting we use
k = 256 (64 for 540B) samples for MATH, & 64 for OCWCourses, k 100 (40 for S40B) for GSM8K and & = 16 for MMLU-STEM, The PaLM GSM8K results do not use calculator and were reported in [5]. We evaluated datasets that did not have published results recent models on OpenAI davinci-002. Despite MMLU-STEM being multiple choice task, can apply majority vote by prompting the model to generate rationale prior to the final answer, sampling multiple times, and then using majority vote on the final answers. Superscripts denote results that are quoted from previous work: GPT-2 [6]. PaLM 540B ma maj104 [19]. and Chinchilla [44].
MATH OCWCourses
GSM8k
MMLU-STEM
PaLM 8B
1.5%
1.5%
4.1%
22.0%
Minerva 8B
14.1%
7.7%
16.2%
35.6%
Minerva 8B, maj1@k
25.4%
12.5%
28.4%
43.4%
PaLM 62B
4.4%
5.9%
33.0%
39.1%
Minerva 62B
27.6%
12.9%
52.4%
53.9%
Minerva 62B, ma j10k
43.4%
23.5%
68.5%
63.5%
PaLM 540B
8.8%
7.1%
56.5%
58.7%
Minerva 540B
33.6%
17.6%
58.8%
63.9%
Minerva 540B, maj10k
50.3%
30.8%
78.5%
75.0%
OpenAl davinci-002
19.1%
14.8%
Published SOTA
6.9%
74.4%
54.9%
OCWCourses. Using publicly avvilable course materials offered by MIT (OpenCourse Ware), collected problems with automatically-verffable solutions (either numeric or symbolically verifiable via SymPy) from courses including "solid-state chemistry", "information and entropy", "differential equations", and "special relativity." These problems were processed by contractors to be self- contained and have a clearly-delinatee final answer. Problems asking for proof or open-ended short answer were not included. In total we curated 272 problems, 191 of which have numeric solutions and 81 have symbolic solutions. In Appendix we detail the contributions from each course, and the process of converting these course materials into format suitable for processing by language models. We also provide the text of all problems. We plan release these part of an open-sone open-source dataset which will be detailed in an upcoming manuscript.
2.5 Inference- Timen Techniques
We find that we can considerably outperform greedy decoding by sampling k > solutions (with non-zero temperature) and selecting one using majority voting [19]. This consists of grouping predictions with respect to their final answer and selecting the most common answer. We denote this maji jajkk, following [17]. A variation of this algorithm, denoted ma jn@k, involves selecting the most common answers. Intuitively, the reason majority voting improves performance is that while there are many ways answer question incorrectly, there are typically very few ways to answer correctly. In other words, the truth is a Schelling point.
Contrast majority voting with pass@k, where a task is considered solved if any single sample solves it out of k samples. See Section 4.2 for more details on pass@k performance. In Appendix E. we report on how performance depends on & for different metrics. We find that while pass@k continues improve as is increased, majority voting performance saturates faster: 97% the large I accuracy is achieved at k = 64 for MATH and l: = 16 for GSM8k. This is likely because majority voting selects the most common answer in the modeled distribution, and the error of this estimate decreases with increasing k. This is in contrast to passēk k where the performance improvement comes from the tail of the distribution, which can keep improving as i is increased.
Log- likelihood is another metric that can used to rerank samples. We found that majority voting performs significantly better than log- likeelllooood reranking (see Appendix E.2).

---

Results
Table 3 summarizes the results for Minerva models and other models, on the evaluation datasets described in Section 2.3. Figure 3 presents a breakdown of the MATH dataset results by subtopic. For MMLU evaluations, unless otherwise noted, performance is measured by using the standard 5-shot prompt per topie and picking the answer with the highest score. When evaluating MMLU with majority voting, sample k 16 model answers using chain-of-thought prompt.
We present model output samples in Figures 1 and 2, and additional output samples listed in the Appendix. In addition, we evaluated Minerva 62B on the National Math Exam in Poland and found that it achieves score of 57%, which happened to be the national average in 2021 [45, p. 23]. The 540B model achieves 65%.
We include results on the latest publicly available language model from OpenAl, davinci-002, evaluated using the OpenAl API with temperature combination of training data, scale and inference techniques yields state of the art results on all the technical tasks that considered. For all tasks (with the exception of GSM8K), the improvement with respect to previous results is considerable.
the official recommendation (7 0.2). The
While our main focus is on few shot evaluation, we also tried to finetune Minerva on MATH; this did yield any improvement; interestingly, however, finetuning PaLM on MATH did give significant improvement. Further details can be found in Appendix E.3.
3.1 Basic arithmetic
In Appendix I, we study the performance Minerva 540B on simple arithmetic tasks. The model achieves over 80% accuracy on 1i-di addition and over 20% accuracy 18-digit addition.
4
Performance Analysis
4.1 Model Mistakes
To better understand the types of mistakes our models make, we compare the performance of Minerva 8B and Minerva 62B on 216 problems with high confidence majority decisions of both models. Specifically, we selected examples where the top answer received at least 15% of votes, and that either Minerva 8B was correct and Minerva 62B was incorrect (15 samples), or vice versa (201 samples). The categories and examples for each category are described in Appendix J.2.
As shown Table 4, the prevailing errors of the 8B model were related incorrect reasoning or calculations. Many of the calculation errors were relatively benign arithmetic mistakes. Solutions that were too short were relatively rare (in these cases, the model immediately produces an incorrect answer without any intermediate reasoning steps). Finally, in few cases, the model hallucinates an equation or mathematical fact that not real.
In the samples where the 62B model was incorrect, the dominating failure modes were again incorrect reasoning and incorrect calculations. In summary, we find that the 62B Minerva model retains most of the skills of the 8B model and improves upon both reasoning and calculation robustness.
Table 4: Failure modes of the 8B Minerva model, out of 201 samples which the 62B model solved correctly and the 8B model did not.
Type of mistakes
Occurrences
Type of mistakes
Occurrences
Incorrect reasoning
82
Incorrect calculation
70
Misunderstands question
22
Uses incorrect fact Hallucinated math objects
16
Solution too short
4
4.2 False Positives
In our approach to solving quantitative reasoning problems, we are able to automatically verify whether the final answer to a problem is correct, but do not have automatic way to verify the
7

---

model's 's chain of reasoning. This leaves open the possibility of false positives: samples which have the correct final answer, but for which the reasoning is incomplete or incorrect.
We selected 100 random questions from MATH (20 per difficulty level), along with answers sampled at zero temperature from the 62B model. We then manually inspected the answers to determine the false positive rate, which the ratio between number of false positive examples and number of examples for which the final answer is correct; for difficulty levels 1, 2, 3, 4, and 5, the false positive rates are < 5%, 10%, < 5%, 15%, and 30%, respectively. Averaging these per-level false positive rates (weighted by true positive rates), the overall false positive rate is estimated to be 8%. Although this overall rate is low, we find that it does increase with difficulty level.
Our focus on pass@1 and majority voting as the primary evaluation metries due in part to the fact that they are less susceptible to false positives than pass @k [17]. While the pass 0256 accuracy 84.5% for the 62B model, false positives account for part of it. We inspected the samples that failed in majority voting but passed false positive rate for pass@256 to be 30% among samples selected in this way. After removing false positives, we estimate that the pass 256 accuracy at greater than 68%; see Appendix J.3 for details.
pass@k due to single correct answer, and estimate the
5 Memorization
A central question in interpreting Minerva's solutions is whether performance reflects genuine analytic capability or instead rote memorization. This is especially relevant as there has been much prior work indicating that language models often memorize some fraction of their training data [46, 47, 48]. When examining model solutions, we find that memorization of intermediate facts, such as numerical values of square roots or trigonometric identities, are crucial elements of model solutions. We would like to investigate a strong form of memorization, where model performance is result of memorizing the explicit problems and solutions in our evaluation set, but also a weaker form, where the model has memorized alternate answers to the same questions.
In order to evaluate the degree which our models solve problems by recalling information memo- rized from training data, we conduct three analyses the MATH dataset. First directly search for problems and solutions in our training corpus. Next, we generate modified versions of problems and evaluate our models' robustness to these changes. Finally, we measure the degree of overlap between the ground truth solutions and solutions generated by our model and measure the effect of this similarity model performance. Overall, we find little evidence that the model's performance can be attributed to memorization.
5.1 Training and Evaluation Dataset Overlap
We selected the problems for which our 62B parameter model produced a correct answer, and filtered them to the 100 problems with the highest majority vote score, expecting that problems with a high majority vote score are more likely to have been memorized. For each of these question-answe pairs, we compute the BLEU score across chunks of 500 characters in our Math Web Pages dataset (a histogram of the BLEU scores is shown Appendix Figure 9). We then manually inspect the 250 documents with the highest BLEU scores. While many of the top matches were from homework help sites with math questions and solutions, none of the questions matched the questions in the subset of MATH under consideration. We have included these 250 segments in Appendix K.1. We note that some problems from MATH can be found on the web. Nevertheless, this analysis concludes that these problems did not make it through our data collection process.
5.2 Performance on Modified MATH Problems
To further investigate memorization, we randomly selected twenty problems which the 62B model answered correctly under majority voting. We manually modified each problem either by introducing minor changes to problem wording (framing) or by changing the numbers which appeared in the problem and modifying the solution accordingly. We then compared the accuracy over sampled solutions before and after the modification. Results are shown in Figure 4. In both cases the accuracy before and after modifications are correlated, with no clear bias in favor of the original formulation. This is suggestive of minimal memorization. The modified problems are listed in Appendix K.2.
8

---

Modified framinng
Modified
1.0
1.0
Majorily-voting accuracy samples
Accurary (modffed)
0.6
0.6
Accuracy
30
0.4
0.2
0.0
0.0
40 62 20
100
Accuracy (origine))
Accuracy (originel)
BLEU score cutoff [to ground-truid truth solution)
Figure 4: Results indicating lack of memorization MATH. Left, Center: Accuracy of original questions from MATH dataset and their modified versions. Each point represents question. æ axis is accuracy on the original question, and the axis is accuracy on the modified one. Right: Majority vote accuracy, computed only on samples with BLEU score to the ground truth solution less than or equal to the a-axis value.
5.3 BLEU Score Between Ground Truth and Generated Solutions
We seek to detect memorization of solutions by computing BLEU score between ground truth answers and model generated answers. We use the 62B model and analyze 256 samples per problem in the MATH dataset. First, we compute overlap statistics for all correct samples. We find that 160 out of 5,000 test questions have sample with BLEU score greater than or equal to 80 (see Appendix K.3). We note that they tend be short solutions. To understand the effect of answer similarity performance, we remove model samples above certain BLEU score threshold, and recompute the majority vote accuracy. We find that majority vote performance is robust even down to relatively low similarities (see Figure 4), indicating that performance cannot be attributed to model outputs that are very similar to ground truth answers.
Conclusions and Discussion
In this work, we take an approach to quantitative reasoning that relies on solving problems using mathematical reasoning expressed natural language. We show that by training large language model on high quality mathematical dataset, we are able to achieve strong performance on tasks that require logical reasoning, numerical calculation, and symbolic manipulation. Our model does not make use of external tools, and at inference time relies exclusively on autoregressive sampling to achieve this performance. Complementary approaches quantitative reasoning include code- generating models and formal methods. These are all different routes toward common goal: an agent that can reason about and solve quantitative problems.
6.1 Limitations of Our Approach
Our approach to quantitative reasoning has several limitations. First, we have no automatic way of verifying the correctness of the model's 's answers. This is in contrast to formal approaches, for which automatic verification intrinsic. Second, our model has no access to external tools such as calculator or Python interpreter. It is therefore limited in its ability to perform quantitative reasoning tasks that require complicated numerical calculations. Third, because our model was trained on a large amount of data, we have little direct control over the specific capabilities that the model acquired.
Our inference time strategy also has limitations. Majority voting requires generating many samples and selecting the most common final answer. This can be computationally costly, and importantly only works for problems with well defined final answers, and so is less well suited for domains such as theorem proving or open-ended scientific reasoning.
6.2 Societal Impact
Artificial neural networks capable of solving quantitative reasoning problems in a general setting have the potential of substantial societal impact. Minerva, while step in this direction, is still far from achieving this goal, and its potential societal impact is therefore limited. The model's performance still well below human performance, and furthermore, we do not have automatic way of verifying

---

the correctness of its outputs. If these issues could be solved, we expect the impacts of this model to be broadly positive. direct application could be an accessible and affordable math tutor which could help improve educational inequalities.
7 Acknowledgments
We thank David Andre, Jacob Austin, Maarten Bosma, Aakanksha Chowdhery, Sergey Ioffe, Colin Raffel, Charles Sutton, Christian Szegedy, Stanislas Polu, and Harrison Edwards for helpful discus- sions.
References
[1] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv preprint arXiv: 10683, 2019.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33: 13:1877-1901, 2020.
[3] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint ar2iv:2112.11446, 2021.
[4] Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron to train negatron-turing nlg 530b, a large-scale generative language model. arXiv preprint 12Ni::2201.11990 2022.
[5] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope. James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai. Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways, 2022. URL https: / arxiv. org/ / abs / 2204 02311.
[6] Dan Hendryeks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https: / /arxivv.vv/. org/ abs /2103.11.000 03874.
[7] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL /tttpp::///xtiie erg/abs/ /2110 14168.
[8] Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, Peter Clark, Chitta Baral, and Ashwin Kalyan. Numglue: A suite of fundamental yet challenging mathematical reasoning tasks, 2022. URL https: /ttp/aaivvv..o/..e..a..........a abs / 2204. 05660.
[9] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray. Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad
10

---

Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert. Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Mísra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. URL https: / //arxiv.org/////eoo.i........ als 2107 03374.
[10] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint 27X7722108.07732, 2021.
(11) Iddo Drori, Sarah Zhang. Reece Shuttleworth, Leonard Tang, Albert Lu, Elizabeth Ke, Kevin Liu, Linda Chen, Sunny Tran, Newman Cheng, Roman Wang, Nikhil Singh, Taylor L.. Patti, Jayson Lynch, Avi Shporer, Nakul Verma, Eugene Wu, and Gilbert Strang. A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level, 2021. URL https:/ /arxiv.vvvv.////ii///aaaaaaaaaaa org/ abs /212.. 15594.
[12] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. CoRR, 0bs/2009.03300, 2020. URL https: http://aaxvvvv.o/ org/ haa///bss000// /2009. 03300.
[13] Rik Koncel-Kedziorsii, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Du- mas Ang. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3, 2015.
[14] Mohammad Javad Hosseini, Hannanch Hajishirzi, Oren Etzioni, and Nate Kushman. Learning solve arithmetic word problems with verb categorization. EMNLP, 523533, 2014.
[15] Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski. Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz. Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show Your Work: Scratchpads for Intermediate Computation with Language Models. arXiv e-prints, art. arXiv:2112.001 14, November 2021.
[16] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models, 2022. URL https: / arxiv. org/ abs /2011 11200.1.11100 1903.
[17] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode, 2022. URL https: arxiv. org/ abs 2203. 0781 8
[18] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha. Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslic Baker, Yu Du YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri. Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung- Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Seeier-Hellstenn, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera- Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. Lamda: Language models for dialog applications, 2022. URL httph:tpp... : wttpiviiiiv.////ii///aaa///aaavv .rg/ abs / 220022211100 10220008800 08239.
[19] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2022. URL https: / arxiv. org/ abs//203..322.. 11171.
11

---

[20] The Coq development team. The Coq reference manual, 2022. URL http::/coo...iieeeeeeee inria. fr. Version 8.15.
[21] Makarius Wenzel, Lawrence C. Paulson, and Tobias Nipkow. The isabelle framework. In Otmane Ait Mohamed, César Muñoz, and Sofiène Tahar. editors, Theorem Proving in Higher Order Logics, pages 33-38, Berlin, Heidelberg, 2008. Springer Berlin Heidelberg. ISBN 978-3-540-7106777.
[22] John Harrison. Hol light: A tutorial introduction. In Mandayam Srivas and Albert Camilleri, editors, Formal Methods in Computer-Aided Design. pages 265-269, Berlin, Heidelberg, 1996. Springer Berlin Heidelberg. ISBN 978-3-540-49567-3,
[23] Leonardo Mendonça de Moura, Soonho Kong, Jeremy Avigad, Floris van Doorn, and Jakob von Raumer. The lean theorem prover (system description). In Amy P. Felty and Aart Mid- deldorp, editors, CADE. volume 9195 of Lecture Notes in Computer Science, pages 378-388. Springer, 2015. ISBN 978-3-319-21400-9, URL http: /tppp.blpp../////////// anplpp.uni... ceit-trierr de db/
conf/ canef//ae/aeeeeo cade201dee200 html #MouraKADR15.
[24] Norman D. Megill and David A. Wheeler. Metamath: A Computer Lan- guage for Pure Mathematics. Lulu Press, Morrisville, North Carolina, 2019.
URL http:/ http://us..etatt wrggdowwloadodssoooooooooooooooe /eeeaadss//ttt/eetee///ee /et amath.pde pdf. http: / /us. wtw.://eetaaaat..ooooooo.ooooooo headddds//mmmmaaaaaa pdf.
[25] Adam Grabowski, Artur Komilowicz, and Adam Naumowicz. Mizar in nutshell. J. Formalized Reasoning, 3(2):153-245, 2010. doi: (tt.////sm..972-577777777
URL https:: /doi.
orq/10. 6092 http/issn..e 1922
[26] Stephan Schulz. System Description: E 1.8. In Ken McMillan, Aart Middeldorp, and Andrei Voronkov, editors, Proc. of the 19th LPAR, volume 8312 of LNCS. Springer, 2013.
[27] Jens Otten. leancop 2.0 and ileancop 1.2: High performance lean theorem proving in classical and intuitionistic logic (system descriptions). In Alessandro Armando, Peter Baungartner, and Gilles Dowek, editors, Automated Reasoning, pages 283-291, Berlin, Heidelberg, 2008. Springer Berlin Heidelberg. ISBN 978-3-540-71070-7.
[28]
Laura Kovács and Andrei Voronkov. First-order theorem proving and Vampire. In 20V.3013
[29] Alexander A. Alemi, François Chollet, Niklas Een, Geoffrey Irving, Christian Szegedy, and Josef Urban. Deepmath Deep Sequence Models for Premise Selection. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pages 2243-2251, USA, 2016. Curran Associates Inc. ISBN 978-1-5108 118--3881-9 URL http:/ / /dl........ acm org/ i tatiintinn.. htt.
[30] Zarathustra Amadeus Goertzel, Karel Chvalovský, Jan Jakubuv, Miroslav Olsák, and Josef Urban. Fast and slow enigmas and parental guidance. CoRR, abs/2107.06750, 2021. URL https: / arxiv. org/ abs 22107.211. 06750..
[31] Wenda Li, Lei Yu, Yuhuai Wu, and Lawrence C. Paulson. Isarstep: a benchmark for high-level mathematical reasoning. In ICLR, 2021.
[32] Stanislas Polu and Ilya Sutskever. Generative language modeling for automated theorem proving. arXiv preprini arXiv:2009.033: 2020.
[33] Cezary Kaliszyk, Josef Urban, Henryk Michalewski, and Miroslav Olšák. Reinforcement learning of theorem proving. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett, editors, Advances in Newral Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL http:::://///oee/eeeeeee ngss neurips www..ps.cccpppaaapeeeeeee ctppsccccppprr//ppppppppppppp .df.
[34] Markus Norman Rabe, Dennis Lee, Kshitij Bansal, and Christian Szegedy. Mathematical reasoning via self-supervised skip-tree training. 2021. URL wtw.tttt///opereeeeee www.wwewwrre... forum? Derrm?id-YmAArna ddYYmgAnYYCME,,
12

---

[35] Yuhuai Wu, Markus N. Rabe, Wenda Li, Jimmy Ba, Roger B. Grosse, and Christian Szegedy. learning inductive bias for primitives of mathematical reasoning. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICMI. 2021, 18-24 July 2021, Virtual Event, volume 139 Proceedings of Machine Learning Research, pages 11251-11262 PMLR, 2021. http:// /ttp:/ed///ceeddeeeee nig. mlr press/ v139/w http///2lc...mm//. html..
LIME:
[36] Jesse Michael Han, Jason Rute, Yuhuai Wu, Edward Ayers, and Stanislas Polu. Proof artifact co-training for theorem proving with language models. International Conference on Learning Representations, 2022. URL https: http////ppeeeeieeeeeeeeeee wttp:/reeeeeeeeeeeeteeeeeee wwwttoooumniieimee http:/oormmm1ddrrxxxeee5.4....
[37] Stanislas Polu, Jesse Michael Han, Kunhao Zheng, Mantas Baksys, Igor Babuschkin, and Ilya Sutskever. Formal mathematics statement curriculum learning. CoRR, abs/2202.01344,, 2022. URL https:tt///// /aw.iviiiiv..//a.a//aaaaaaaaaava www.wiwww./w...............o.eee abs/ abs/aass/.2a//// 2202. 01444.
[38] Albert Q. Jiang, Wenda Li, Szymon Tworkowski, Konrad Czechowski, Tomasz Odrzygózdz, Piotr Milos, Yuhuai Wu, and Mateja Jamnik. Thor: Wielding hammers to integrate language models and automated theorem provers. CoRR, abs/2205. 10893, 2022. doi: 10.48550/arXiv. 2205. 10893. URL https:://///t/ct //doi. org / /10. ..o..... 4 48550 20550//axiiio 2205 10993.
[39] Yuhuai Wu, Albert Jian, Wenda Li, Markus N. Rabe, Charles Staats, Mateja Jamnik, and Christian Szegedy. Autoformalization with large language models. CoRR, abs/2205. 12615, 2022. doi: 00.48550/a2Xi.22255.2222 2205. 12615.
URL https: //oii /10/ 48550/arx1yy
[40] Sean Welleck, Jiacheng Liu, Ronan Le Bras, Hannaneh Hajishirzi, Yejin Choi, and Kyunghyun Cho. Naturalproofs: Mathematical theorem proving in natural language. CoRR, abs/2104.01 102.01 2021. URL https:// //axiiv org/ abs / 2104. 01112.
[41] Lukas Pfahler and Katharina Morik. Semantic search in millions of equations, 2020.
[42] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration, 2019. URL httts:://aaat
orgg abs////////// 1904. 09751.
[43] Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondřej Čertík, Sergey B. Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rath- nayake, Sean Vig, Brian Granger, Richard Muller, Francesco Bonazzi, Harsh Gupta. Shivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel. Štěpán Roučka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert Cimrman, and Anthony Scopatz. Sympy: symbolic computing in python. PeerJ Computer Science, 3:3,03, January 2017. ISSN 2376- 5992. doi: 10. 70.77112 (0.7771//peee 17/p7////////......// 103. URL https::pp/.............tt / doi org /00. 771 7 http//eeerjjee 103.
[44] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models, 2022. URL WtRLhtts:///:xxv.. abs/2203. 15556.
[45] CKE. Skale centylowe wyników - matura 2021, 2021. URL petttps:/q..iiee wtt.iioorr.ppp////mm files/371 0007771200001100 hat///mmmaurrr 2ataaa2021-222 2021-centyyll centyle
pdf.
[46] Trieu H Trinh and Quoc Le. A simple method for commonsense reasoning, arXiv preprint arXiv:1806.02847. 2018.
[47] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 188):9 2019.
[48] Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models, 2022. URL https: / arxiv. org. aaag/abb//e abs/ 2202.. 07646.
13

---

[49] Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowyez, Alex Salcianu, Mare van Zee, Jacob Austin, Sehastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp. Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepar Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. Scaling up models and data with x and seqio, 2022. 17189.
URL https: /awpivrxivv//..////ii//...//a/ia org/abgsss abs /2003.
[50] Abhilasha Ravichander, Aakanksha Naik, Carolyn Rose, and Eduard Hovy. Equate: A bench- mark evaluation framework for quantitative reasoning in natural language inference, 2019. URL https: / //axxvv.... orgo abs 1901. 03735.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 6
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No] The code and models are proprictary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] We only run the experiments once. This is the standard approach for work with large language models because their significant compute requirements.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] OCWCourses will be released with
license CC BY-NC-SA 4.0. [No] The mathematical webpages dataset is proprictary.
(c) Did you include any new assets either in the supplemental material as URL? [Yes] OCWCourses questions are part of supplementary material
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes]: We share the main instructions given to contractors.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals. applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [No] We hired dedicated workforce for this task rather than using crowdsourcing service. As such, our approach to hiring and compensation is proprictary.
15


Reviews:
[{"review_id": "uxZSxiVFd4r", "reviewer": "Reviewer_8LQc", "summary": "This paper introduces a large language model and new training and evaluation dataset for quantitative reasoning problems. It first overviews existing methods and resources for quantitative reasoning, then discusses the datasets collected for training and evaluation, and finally provides comprehensive results and analysis of a model that is finetuned on the collected training data, finding that in comparison to existing models, the proposed large language model performs well.", "questions": "* Clarification question: Table 1 lists tokens in the entire dataset, right? Not the amount sampled for the technical training dataset? Otherwise the proportions don't make sense. \n* Is the general NL data for the technical dataset sampled randomly from the full general NL dataset?\n* Why truncate inputs from the left? Besides fitting in the limited transformer input. How often does this happen?\n* What does it look like for reasoning to be wrong but the answer to be right? It's a bit surprising that the false positive rate is 30% in difficulty 5 examples -- without knowing what those examples look like, I find it very surprising that there are so many false positives.\n* How many times each are the 20 examples in 5.2 modified to test memorization? ", "limitations": "I'm wondering about copyright/consent issues when it comes to scraping/re-releasing data from math webpages and ArXiv. Are you planning to release the training data?\n\nSimilarly, details on the contract work used to clean the dataset -- typically crowdsourcing contributions will have (at least in the appendix) details on compensation, etc. but I couldn't find that.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n* Good overview on challenges associated with quantitative reasoning\n* Very comprehensive error analysis (and examples in appendix)\n* Great analysis of the model robustness wrt. memorization\n\n---\n\nWeaknesses:\n\n* The main contribution here is collecting (scraping and filtering) a dataset, and fine-tuning a large model on it. By itself, I am not sure this is a huge contribution. However, the analysis in the latter half of the paper is quite comprehensive.\n* I was confused about the majn@k format in Fig 1 and 2.3 until section 2.5. Would help to clarify in the Fig 1 caption and 2.3 that it's majority voting on the final answer, not the whole generated text (how does this work for something like a proof, though, that's inherently a chain of reasoning?).\n* I'd like to see more discussion on the limitations of existing quantitative benchmarks. I was expecting this in the related work section but couldn't find anything (just briefly mentioned on L86).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fWyUVKIcadp", "reviewer": "Reviewer_9sZQ", "summary": "The authors fine-tune large PLMs (pre-trained on a corpus of scientific and mathematical text) on a new quantitative reasoning dataset, evaluating on existing benchmarks and a new, custom benchmark. The main contribution is the formulation of these datasets, which allows for sota on MATH and strong performance on other benchmark. Notably, this approach relies exclusively on a language model, which has no outside access to libraries, calculators, or other tools in performing the task.", "questions": "- The claim in L175-177 seems unfounded to me. Why wouldn't you be sure you haven't found a local minimum with the fine-tuned model by training on the wrong dataset, whereas you haven't with the base model?\n- In L57-58, I cannot parse \"Prompting language models... unseen problems\". Do you mean to say \"apply\" instead of \"output\"?\n- I cannot tell whether the pre-trained model, on which Minerva is based, is a contribution of this paper or another paper?\n- I don't understand the need for the OCW benchmark introduced, in addition to MATH, GSM8k, and others. Which part of the new dataset do these more established benchmarks lack?\n", "limitations": "The authors rightly identify 3 limitations of their method. There is nothing set in stone, however, about these limitations (one could engineer a dataset to give specific capabilities to a model, or use a model that had access to outside tools, etc), and the authors do not discuss why they chose the model, data, etc. that they did __with respect to__ these limitations.", "rating": 2, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The most obvious strength of the paper is the considerable improvement over SOTA scores, although I'm quite surprised at how low such scores are, especially when GPT-3 also so exceeds them. \n- I think the majority voting idea is really cool, since sampling obviously provides a limited view of the branching probability tree characterizing the language model's processing of the original prompt, and majority rule arguably offers a more fulsome picture of the outputs. The obvious problem with it is that cost scales linearly.\n- Checking the answers using sympy is a clever way to bin different expressions of answers.\n\nWeaknesses\n- The main problem with the paper, in my view, is that the training data is the main contribution of the paper, but I cannot tell what it is like. Basically the only thing I can tell is that it contains mathematical expressions. Fig. 1 and Fig. 2 show the evaluation data, which is already outlined in other work, and I wonder why that space wasn't used to explain this paper's training corpus. At one point, the authors mention \"equations and diagrams\" juxtaposed with natural language, but how would a language model process diagrams, or how would they be converted to text? I have many more questions, and they might be answered in the appendix, which wasn't available to me, but this lack of clarity seems to me to preclude the publication of this paper at NeurIPS.\n- The benchmark introduced may be too small to be meaningful, and I don't think the paper makes it clear how representative it is. Furthermore, I worry, when it includes questions from \"‘solid-state chemistry’, ‘information and entropy’, ‘differential equations’, and ‘special relativity'\", that quantitative reasoning is not what we are testing, but specific scientific knowledge that makes use of quantitative reasoning. \n- Doesn't reference EQUATE (Ravichander et al. 2019), one such recent, decently-cited quantitative reasoning benchmark.\n- The authors say in L51 that \"existing benchmarks are limited with respect to quantitative reasoning\", and I'd like to know what they mean by that. I couldn't find any corresponding discussion.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YS4dVxZ35XK", "reviewer": "Reviewer_iAwH", "summary": "State-of-the-art language models have generally struggled with tasks that require quantitative reasoning, such as solving mathematics, science, and engineering problems at the college level. This paper introduced Minerva, a large language model pretrained on general natural language data and further trained on technical content. The authors conducted experiments in a few-shot setting across a range of tasks that require quantitative/numerical/symbolic reasoning, including one they manually collected for better evaluation (OCWCourses). The results demonstrated that it achieved strong performance on all the datasets and set state-of-the-art\n performance on the MATH dataset. Through an error analysis, they also found the two most common cases in which their model made wrong predictions are *Incorrect reasoning* and *Incorrect calculation*.\n\n\n", "questions": "n/a", "limitations": "The authors have sufficiently addressed the limitations in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Originality: The idea of further training a pretrained language model on a math-heavy corpus is quite straightforward yet effective (as can be seen from the results).\n\n- Quality: The proposed approach is technically sound and the experimental results showed that it outperformed the previous language model on MATH and OCWCourses by a large margin. However, I don't think there are many insights I can draw from this paper. My main takeaway about this paper is that we can achieve much better performance on quantitative/numerical/symbolic reasoning by further training a pretrained language model on high-quality corpus containing equations and diagrams. What if we just train a language model from scratch using the same corpus? What is the role of the general language data played here? I think answering such questions could make this paper stronger.\n\n- Clarity: This paper is well-organized and easy to follow. \n\n- Significance: This paper achieved strong results across a range of different datasets and the reasoning skill demonstrated by the model is amazing. As pointed out by the authors that one of the main challenges is to automatically verify the correctness of the generated reasoning chains/answers, I believe this work would prosper the work in such directions.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JMO2HywlVNq", "reviewer": "Reviewer_3teb", "summary": "This work proposes a large language model *Minerva* to solve scientific and mathematical questions in natural language and generate step-by-step solutions using correct  Latex notation. This model is trained on a high-quality data collection composed of scientific and mathematical information. This model got impressive results across a wide variety of quantitative reasoning tasks, including Precalculus, Geometry, and many more.\n\n", "questions": "1. Any reason behind the incorrect answers for the solutions that were short?\n2. Can it derive math expressions instead of numbers as the final answer like solving an indefinite integral?", "limitations": "There is currently no automated mechanism to check the correctness of the model's answers, and the authors also acknowledge it. This work has no major negative societal impact, but people may use it to cheat on some STEM tests.", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* This paper provides a well-documented analysis of results and the reason behind the usage of metrics.\n* The work will be a founding stone in making AI for tutoring. This can have a significant impact on students with fewer resources.\n* The authors also addressed the issues of False positives and Memorization with a detailed investigation.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4pHn3WfOaXM", "reviewer": "Reviewer_h3bW", "summary": "The paper focuses on the problem of quantitative reasoning in large language models, and introduces Minerva, a large language model that is pretrained on general language data and then further trained on a dataset containing scientific and mathematical data. \n\nThe training data is collected from arXiv and web pages that were preprocessed to conserve mathematical content. The authors then use this data to finetune pretrained decoder-only transformer LMs (8B and 62B parameter models) using an autoregressive objective. During inference time, the paper suggests sampling k > 1 solutions and selecting one using majority voting. The authors show empirically that this approach considerably outperforms greedy decoding. \n\nThe authors compare their models against previous works on the MATH, GSM8k, and MMLU-STEM datasets, and also build a dataset (OCWCourses) of 272 undergraduate questions in science and mathematics to evaluate the model's quantitative reasoning abilities in a chain-of-thought context. The authors report that the Minerva 62B significantly outperforms state-of-the-art results on the MATH dataset, and achieves similar performance on MMLU-STEM. Furthermore, the authors conduct analyses to understand the types of mistakes the models make, the effect of false positives (cases where the answer is correct but the chain of reasoning is not), and whether the model is memorizing the answers. They find that most of the errors are incorrect reasoning and incorrect calculations, that false positive rate increases with difficulty level (with an estimated average of 8% on the MATH dataset), and that there is little evidence that the model's performance can be attributed to memorization. \n\n\n\n", "questions": "1- Are the \"BaseModel\" models used publicly available? Appendix B does not contain sufficient information for reproducibility.\n2- Table 4 only reports the distribution of failure modes of the 8B model, and the text only mentions the dominating failure modes for the 62B model. What is the distribution for the 62B model, where the 8B model solved correctly and the 62B model did not? Is there a further breakdown of incorrect reasoning?", "limitations": "The authors discuss the limitations and potential societal impact of their work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\nThe topic of quantitative reasoning with large language models is of interest to the research community. By finetuning pretrained LMs on large math and science datasets and comparing against models that were not trained with such data, the paper highlights the importance of data quality for achieving quantitative reasoning abilities. Furthermore, the authors build a new dataset for evaluating quantitative reasoning, and conduct thorough evaluation of the type of mistakes the models make, highlighting the issue with false positives, and investigating the impact of memorization on performance.\n\n**Weaknesses**\n\nAlthough the authors mention that the dataset is the main novelty of their paper (line 45), there isn't sufficient discussion of it in the paper and appendix. Will this approach scale to more complex math and science problems? Are further gains expected with larger models? Furthermore, what is the breakdown of papers by area/topic? How are diagrams (as mentioned in the intro) processed? What are the examples of heuristics used to process the pages as mentioned in the appendix? etc.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept