Inversely Eliciting Numerical Reasoning in Language Models via Solving Linear Systems
Authors: Fan Zhou
Haoyu Dong
Qian Liu
Zhoujun Cheng
Shi Han
Dongmei Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: numerical reasoning
language model

Abstract: Numerical reasoning over natural language has been a long-standing goal for the research community. However, recent language models have proven difficult to reliably generalize to a broad range of numbers, although they have shown proficiency in reasoning over common and simple numbers. In this paper, we propose a novel method to elicit and exploit the numerical reasoning knowledge hidden in pre-trained language models using simple anchor numbers. Concretely, we first leverage simple numbers as anchors to probe the implicitly inferred arithmetic expressions from language models, and then explicitly apply the expressions on complex numbers to get corresponding answers. To inversely elicit arithmetic expressions, we transform and formulate the task as an analytically solvable linear system. Experimental results on several numerical reasoning benchmarks demonstrate that our approach is highly effective. More importantly, our approach works in the inference phase without extra model training, making it highly portable and achieving significant and consistent performance benefits across a variety of language models in zero-shot, few-shot, and fine-tuning scenarios.

Under review as a conference paper at ICLR 2023
INVERSELY ELICITING NUMERICAL REASONING IN LANGUAGE MODELS VIA SOLVING LINEAR SYSTEMS
Anonymous authors
Paper under double-blind review
ABSTRACT
Numerical reasoning over natural language has been long-standing goal for the research community. However, recent language models have proven difficult to reliably generalize to a broad range of numbers, although they have shown profi- ciency in reasoning over common and simple numbers. In this paper, we propose a novel method to elicit and exploit the numerical reasoning knowledge hidden in pre-trained language models using simple anchor numbers. Concretely, we first leverage simple numbers as anchors to probe the implicitly inferred arithmetic expressions from language models, and then explicitly apply the expressions on complex numbers to get corresponding answers. To inversely elicit arithmetic ex- pressions, transform and formulate the task system. Experimental results on several numerical reasoning benchmarks demon- strate that approach is highly effective. More importantly, our approach works in the inference phase without extra model training. making it highly portable and achieving significant and consistent performance benefits across variety of language models in zero-shot, few-shot, and fine-tuning scenarios.
analytically solvable linear
INTRODUCTION
Language Models (LMs) have demonstrated great success on wide range of natural language tasks (Devlin et al. 2018 Brown et al.] 20206 Chowdhery et al. 2022), and recent works even explore use LMs as general-purpose interface for diverse modalities (Hao al., 2022 Xie iet al. 2022 He et al. 2022). But when it comes to reasoning about numbers, the crucial parts of text, tables, and knowledge bases, the performance of LMs slumps. A key challenge of numerical reasoning for now is number calculation. Even rational numbers, a small subset of real numbers, readily constitute an infinite space that cannot be completely covered by pre-training corpora, hence posing significant obstacle to LMs. Recent works have shown strong context understanding ca- pabilities of LMs in numerical reasoning datasets (Dua et al., 2019; Cobbe et al., 202T), but LMs still far from being robust on implicit numerical calculation:
numbers grow bigger and more complex, LMs are more likely to fail, c.g., 8, 534.5 + 17. 17.85; and even for small number additions, e-g.. 512 + 128 and 513 + 129, LMs not stable enough produce the correct answer consistently. Similar observations are also reported by Razeghi et al. (2022), showing that end-to-end LMs easily fail to calculate numbers that rarely appear pre-training corpora.
Fortunately, by reverse thinking, we have a positive perspective: with the exact same context, LMs significantly more accurate and stable on simple numbers tyyicallly small integers that appear frequently in the pre-training corpora than complex numbers, indicating that LMs have a strong capability of applying arithmetic results simple numbers after pre-training. This motivates us to leverage simple numbers as "anchors" to probe the implicitly inferred arithmetic expressions from language models and then explicitly apply the expressions on complex numbers. Specifically, as Figure illustrates, when detecting complex numbers (10, 477 and 7, 459) that are challenging for LMs. to first replace them by anchor numbers(10 and 7, etc) and use LMs to output answers (3, etc) that are more much accurate than complex numbers, then inversely elicit the hidden arithmetic relationship (x1 - x2) implicitly inferred by LMs through anchor inputs/outputs (10.7.3, etc), and finally explicitly doing the arithmetic using the initial complex numbers (10, 477-7, 459) to produce the precise answer (3, 018). In this way, our method combines the advances of LMs on understanding complex context and memorizing simple numbers for reliable numerical reasoning.

---

Under review as a conference paper at ICLR 2023
Operand Proposal
Number Substitution
Reasninng LM
Arithmetic Relationship Inversion
Paragraph:
Paragraph:
3 ())
As the census of 2000, there 10,477 houscholds and 7,459 families in the county.
As of the census of 2000, there
Solving Algorithm
were 10 households and
7 families in the county.
(2y), (x))
y=
Question:
Question:
How many more households
How many more households
are there than families?
are there familless
(x,)) ) 7,459 (x22
Answer: 3,018
Figure I:
The illustration of our proposed framework, which elicits numerical reasoning in language models via Solving Linear Systems (SOLIS).
This paper introduces a new idea of eliciting and exploiting the numerical reasoning knowledge hidden pre-trained LMs through probing with simple anchor numbers. Importantly, our frame- work does not need any additional model training or labeled data, because it simply works during the test-time inference phase, and it is portable to all existing tnee-uuedd/feww-hoozzerr with decoders. Thus it is significantly different from existing neural symbolic methods that need continuous training (Liang et al., 2016) and program synthesis from examples that need specific and human-provided input-output pairs for each example in the inference phase (Gulwani 2011).
LMs
To inversely elicit arithmetic relationships in LMs through anchor numbers, we propose SOLIS, a novel method to transform and formulate this problem to a linear system that can be straightfor- wardly solved in an analytic way. Alternative search-basee and heuristic- based methods are further devised to promote robustness for noisy linear systems. Experimental results show significant and consistent gains over various language models and diverse zero-shot, few-shot and fine-tuning set- tings on several representative numerical reasoning datasets.
2 PRELIMINARY STUDY
In this section, we will first demonstrate the brittleness of language models' ability on arithmetically- related tasks. Unlike arithmetic benchmarks such as AddSub or MultiArith (Roy & Roth 2015) which contain natural language context for each sample, we directly generate and feed the arith- metic expressions and test the performance on language models. This done to reduce potential perturbing factors and highlight the models' calculating ability. We impose constraints the com- plexity of the expressions: we only study the four fundamental operations, and demand no more than operands, where each operand's 'ss integer range is less then 10, 000 and floating point precision is less than 4. To conduct a systematic investigation, we first produce FF which represents the set of all the expressions satisfying our constraints. We randomly sample numbers within the limits of range and precision as the operands. For one expression f E IF with a specified range and precision, we randomly generate 50 samples. We evaluate the language model on these samples and denote this synthesized task as MathExp which stands for Math Expressions.
We sample a maximum of 50 expressions for each different settings of complexity, and test these samples using large scale language model
Two Operands
Three Operands
Four Operands
GPT-3 (Brown et al., 2020a). We conduct the study on GPT-3 in a few-shot manner: to un- leash its potential, we pre-pend 10 to 20 expres- sions (having the same f. integer range, and floating point precision as the tested sample) to- gether with the answers as the prompt. We then call the OpenAl AP to get all the predictions, and evaluate the performance accordingly.
Exaet Match
80
60
40
20
0
0
2
210' 200" 300"
Floating Point Precision
Integer Range
Figure 2: Performance with different floating point precision (left) and integer range (right).
Results in Figure 2 indicate that even the latest fails to achieve a satisfactory performance: (i) the prediction
powerful Gotee3(Cul accuracy decreases largely as the number gets more complex, i.c., integer range or floating point
https://openai.com
2

---

Under review as a conference paper at ICLR 2023
precision of operands increases; (ii) the prediction aceuracy also drops dramatically as the arithmetic relationship getting more complex, i.c., number of operands increases. In Appendix [A] we also present the performance with our SOLIS framework, which is more robust to influence of floating point precision and integer range.
NUMERICAL REASONING VIA SOLVING LINEAR SYSTEMS
The preliminary study demonstrates that the current language models are vulnerable to complex numbers. For example, they have chance to guess the answer to the sum of two floating point numbers with three decimal places, However, the language model can perform reliably well when the operands are simple, i.e., relatively small integers. Such observations motivate us simplify the numbers before feeding them into language models, thus enabling reliable neural-bases numerical reasoning. In this section, we first provide an overview of our framework SoLIS, and then we elaborate on each part of our framework in detail.
3.1 METHOD OVERVIEW
As mentioned above, our method can be integrated into language models in plug-and-play manner at test time. For the sake of clarification, in the following we refer to LMs that can steadily perform numerical reasoning as reasoning LMs. They can be either LMs obtained by fine-tuning on datasets involving numerical reasoning, or LMs that perform numerical reasoning via in-context learning.
As shown in Figure I. our method generally involves three stages: (1) Operand Proposal: given paragraph, we first identify the numbers which are necessary for the reasoning LM to perform nu- merical reasoning (e.g.. 10, 10,477) 477); (2) Number Substitution: these proposed operand are generally complex for language models, and thus they need to be substituted with randomly chosen simple numbers (c.g., 10) to make the model input simpler. Using the reasoning LM, we can obtain a set of predicted answers with respect to each substituted paragraph after several substitutions. (3) Arith- metic Relationship Inversion: using these paragraphs and their answers as observed data, we can inversely derive the internal reasoning flow from the reasoning LM, i.e. the arithmetic expression between the operands (e.g.. I X2). By applying the expression on the original numbers, the answer the original paragraph can be obtained.
3.2 OPERAND PROPOSAL
There are often many numbers involved in a paragraph, and it is quite challenging to model the arith- metic relationships among all these numbers simultaneously. Consequently, is important during the operand proposal step to trim the prospective operands to a manageable size. straightforward strategy would be to select only the numbers pertinent to the answer candidate operands, which is not trivial in practice since there is no intermediate supervision on the relevance between each number and the answer.
To address the issue, we provide a novel technique that employs number perturbation and the reason- ing LM to measure the relevance systematically. It largely inspired by prior works that leverage an image classifier to quantify the relevance of pixels with image categories (Samek et al. 2017) and its application on natural language tasks (Liu et al., 202T). In their works, relevance is assessed by the degradation of the classifier score after erasing each pixel, where a substantial degradation indicates a strong relevance. Similarly, we consider a number be essential to the final answer if there is a difference between the model predictions before and after perturbing it. Regarding perturbations, we implement it by adding a small adjustment to each number in the paragraph (e-g.. 98.5 - 98.6) and evaluate whether the model prediction changes correspondingly. Despite the fact that the reasoning LM hardly perform accurate calculations over numbers, we observe that LMs have strong context understanding capabilities about numbers and are sensitive to slight changes in the numbers used to forecast answer. More details about the operand proposal mechanism can be found in Appendix B
We use the terms number and operand interchangeably.
3

---

Under review as a conference paper at ICLR 2023
3.3 NUMBER SUBSTITUTION
After the operand proposal stage, a random set of numbers is generated to substitute the proposed operands sequentially. These numbers referred anchor numbers below. Each anchor number an integer between and 20, a range that we believe reasoning LMs can easily handle. Meanwhile, to minimize the effects of number substitution, the numbers. Taking the example from Figure we make the substitution number corresponding to 10, 477 larger than the one corresponding to 7, 459 since 10, 477 larger than 7, 459.
strive maintain the order relationships among
Notably, the random number substitution must be repeated several times (e.g.. three times in Fig- ure 1 obtain group of anchor numbers. Along with the original question, each of these para- graphs is fed into the reasoning LM to predict the answer, which we call the anchor answer. Typi- cally, the number of anchor answers must exceed the number of operands for the subsequent arith- metic relationship inversion stage to be feasible.
3.4 ARITIMETIC RELATIONSHIPP INVERSION
Given collection of anchor numbers and anchor answers, the arithmetic relationship inversion stage investigates the relationship between these numbers and induces expression reflect it. Taking the example from Figure L typical expression can be y = I1 22, where Ii and z2 are both anchor numbers while y the anchor answer.
Although the example expression appears intuitive, deriving such an expression from data points is tremendously difficult because the solution space is theoretically infinite. To make it practicable, as a first step, we begin by limiting the problem-solving space to compositions of binary operators, where each operator operators in numerical reasoning (Dua et al.) 2019). Mcanwhile, there can be up to three compo- sitions, which means the expression contains a maximum of four operands. With such priors, the insoluble expression induction problem can be turned into a linear system solving problem, where the anchor numbers, the anchor answer, and their compositions constitute linear system. In this way, the problem of expression induction can be tackled by the solving algorithms for linear sys- tems, which will be elaborated in Section 4 Finally, the answer can reached in a trustworthy and interpretable manner by applying the derived expression to the original numbers.
be addition, subtraction, multiplication division, the four most prevalent
4 SOLVING ALGORITHM
In this section, we introduce three algorithms that can derive expressions merely from anchor num- bers and anchor answers, namely analytical-basee search-based based and heuristic-based algorithm.
4.1 FORMULATION
Formally, given a paragraph and a question, we denote a group of anchor numbers as æ (I1, T2. : and the arithmetic relationship swer by = f(x). The goal is to recover from different groups of anchor numbers X and corresponding anchor answers y. We propose to transform and formulate the arithmetic relationship inversion as solving system of linear equations. Given expression f(x) with four fundamental arithmetic operations, transform the equation y f(x) by multiplying denominators on both sides when operator division exists, then we get:
expression J. which should produce the an-
ag C 01 Σ1 + 02 I2 ag y a4 11 + .... + ag (x)02... ... x,
For example, y equivalent to solving a = (ag. Qi..... ax), which are coefficients all possible polynomial basis combined by æ1,, Zn and y. denoted constitute multiple groups of values of polynomial basis, denoted as P. then Equation I can be denoted as Pa 0, which is typical of linear equations.
x1/22 can be transformed to I2 I1 Σ2y 0. Then uncovering f(x) is
p, where io = 2 - 1. Multiple groups of anchors X and
4

---

Under review as a conference paper at ICLR 2023
4.2 ANALYTICAL-BASED ALGORITHM
To solve Pa- = b, we can simply generate k+1 groups of anchor numbers X and LMs' answers as y. compute P hased on X and y. and finally get a (P) when P is in full rank. But notice that y can be a linear weighted summation of 20..... In by itself, the coefficient matrix P may not be full-ranked. To address this, we generate k groups of anchor numbers and add an additional constraint by setting |a| E (4 1. So we augment P with an all-one vector to P and finally get a = (P*)) b, where b = (0. o.... 0, 1). In practice, randomly sampled groups of anchor numbers can form full-ranked p with a very high probability, and one can even add a buffer by sampling a bit more groups of anchor numbers than k to constitute different P' s for cross validation.
The analytic method is theoretically complete deduce arithmetic expressions in our pre-defined problem space. But in practice, LMs may produce incorrect results even for anchor numbers, es- pecially when given a complex expression, so to violate the analytic method which needs purely correct anchor answers. To best tolerate them, we then propose search-based and heuristic-based methods better solve noisy linear system. Gladly, the analytic method theoretical supports other methods in aspects such as guiding the number of anchors to sample to ensure unique expression.
4.3 SEARCH- BASED ALGORITHM
The search-based algorithm exhaustively explores the search space and finds out the most preferable arithmetic expression in the space. We constrain the search space of a in Equation [ by: requiring d1-n 0, for all coefficients of the non-constant terms, and for coefficient ag of constant term C, one can restrict the search range to pre-defined set, e.g., ag E [00000 -1, 1, O, 1, 100} in our experiments for efficiency, and different from the analytic method that can easily solve constants in expressions. Constraints here mean that we only let this search algorithm cover f(x) with no more than one constant for efficiency. We then transform all searched polynomial- basis-baseeddd equations backwards into expressions because they have one-to-one mappings, e.g., from I22 I1 I2²/ 0 to y = Ii / We denote the space of expressions as F, and for each ʃ; E and each group of anchor numbers Xj (using m to denote the number of groups), we get Vij by applying Si to Xj.
We define the prediction error between the target ex- pression f and f. as c(f. fi), which calculated by e(f. E, Si = =,, ber of occurrence of exact matching as C, We then find the most preferable expression with the mini- mum prediction error and the maximum number of exact matching. Specifically, when the number of exact matching exceeds pre-defined Chhreshold. we pick the expression fa with the highest e;; otherwise, we pick the expression fi with the lowest 6, The search process is sketched in Algorithm [
Algorithm SEARCH
Input: parameters X, U, F, Cthreshold Output: Most preferable expression f while j < m do
(ūg-W1)) and the num-
2:
for fi E F do
Vi + f((x))
q + G + (1y = (1))
G 5 G; + |03 -iisl
6: end for
j j- -+J+1
8: end while
This method is robust for probably incorrect predic- tions, i.e., when model does not have sufficient num- ber of exact matching, it is still capable to return the most nearest expression by selecting the one with the minimum error. However, the search-based method can be challenged by exponentially explosive search space when the number of operands surges, and it's not efficient to search constant numbers that has a wide and even infinite range, neither.
9: 1 <- arg max e, 1. + +gg mine
10: if c.. 2 Cchreshold then f + ft;
11: elsef
fiz fvg
12: end if
4.4 HEURISTIC-BASED ALGORITHM
In this section, we introduce a heuristic- algorithm, simulated annealing, which is efficient and does not need to search for the whole problem space, though it may produce sub-optimal results given a limited number of exploration steps. We follow the formulation introduced Section 4.1
and proposed optimization target Lu to measure the L1 loss of Pa. The pipeline includes: (1)
randomly initialize a with values {-1, 0, 1} and calculate initial Lui (2) randomly select from to k and perturb a; by adding or subtracting a constant number (we use 1 here); (3) calculate La.,
5

---

Under review as a conference paper at ICLR 2023
and adopt the perturbation with large probability if Cи decreases and with low probability if it increases, balanced by a pre-defined temperature T. which decreases over steps; (4) return a if the number of steps is enough Li equals to zero, otherwise repeat from step 1. Note that, restrict coefficients a to be integers for simplicity, so different from the analytical method restricting
Σi 9, = 1, we ensure only one of the coefficients of y-related polynomial basis {y, T1y. .... T1.2. Tn be non-zero (with static value 1) and at least two coefficients in a are non-zero during the whole initialization and perturbation process to avoid some infeasible local optimal.
In summary, Table shows the strong and weak points of these algorithms. In the problem space introduced in Section 3.4] within at most four operands, the search- based method does not have scalability issues, so it achieves best performance in experiments because it's robust to LMs' predictions and can retrieve optimal expression through exhaustive search except rare constants.
Table 1: Comparison of solving algorithms.
Optimum Robustness Scalability
Analytical Search Heuristic
/
X
X
/
/
EXPERIMENTS
In this section, we integrate SOLIS with various language models as backbones and evaluate the effectiveness of SOLIS on two well-known numerical reasoning benchmarks.
5.1 EXPERIMENTAL SETUP
Datasets We perform experiments DROP (Dua et al. 2019). AddSub and MultiArith, of which the latter two are widely used subsets from MAWPS (Roy Roth 2015). DROP is a reading comprehension benchmark that focuses on numerical reasoning and has variety of answer types, including span, number, and date. The experimental results of DROP are evaluated with the official evaluation metrics Exact Match (EM) and FI. As for MAWPS, it consists of math word problems which also require numerical reasoning ability. The subset AddSub features relatively easier numer- ical reasoning, whereas Auith necessitates multi-step step numerical calculations. The EM metric is used to evaluate the results of AddSub and MultiArith. More details can be found in Appendix
Backbone and Baselines The fine-tuning evaluation is conducted on DROP, where we adopt two kinds of fine-tuned LMs backbones, including (i) Vanilla LMs: BART (Lewis et al. 2020) and T5 (Raffel et al.) 2020), (ii) Reasoning LMs: GENBERT Geva et al. 2020). TAPEX (Luu et al.] 2022) and POET Pi et al. 2022). We compare the performance of our method with previous spe- cialized models designed for DROP, such as NumNet [Ran et al. 2019), NeRd (Chen et al., 2020b), MTMSN (Hu et al. 2019) and QDGAT (Chen et al., 2020a). All models are fine-tuned on the DROP train set, and the best validation set performance is reported. The few-shot and zero-shot evalua- tions are done using AddSub and MultiArith, where adopt GPT-3 code-davinci-002 (Brown) eet al. 2020a) with different prompting as our backbones: Few-shot Chain-of-Thouge Prompt- ing (Few-shot Chain) (Wei al. 2022b) and Zero-shot Chain-of-Thought Prompting (Zero-shot Chain) (Kojima et al.) 2022p. These models perform numerical reasoning by in-context learning, and the few-shot demonstrations are the samples released by Wei al. (2022b).
Design Choices on DROP Following previous work, we apply two general- purpose numerical de- signs on the DROP dataset. First, we employ the character-level rather than subword-level number representation, which proves to be more effective (Wallace et al., 2019: Pi et al.) 2022). Second, we employ the reverse decoding technique, which proves to be successful design to mimic arith- metic carry (Geva et al., 22020). Meanwhile, as mentioned above, the search-based algorithm has difficulties in covering expressions including constants. Considering the constant 100 is frequently used for percentage calculations (c.g., "How many percent of the national population does not live in Bangkok?"", we add it to be one candidate DROP.
5.2 EXPERIMENTAL RESULTS
Since our work focuses on addressing arithmetic problems, we first evaluate suggested solving algo- rithms via their performance on the DROP subset whose answers are numbers (i.e., numeric subset).

---

Under review as a conference paper at ICLR 2023
Table Experimental results of SoLIS various solving algorithms on the DROP numeric subset.
LM
Algorithm
F1(%) on Hard
FI( %)) on Total
30.4
66.4
Analytical
46.4 (+16.0)
69.3 (+2.9)
BART
Search
64.8 (+30.4)
75.2 (+8.8)
Ileuristic
52.8 (+2444)
71.7 (15.3)
66.8
78.4
Analytical
73.3 (+6.5)
80.0 (+1.6)
POET-SQL
Search
76.9 (+10.1)
81.4 (+3.0)
Heuristic
73.0 (.))
80.5 ( (12.1)
Table 3: Fine-tuning evaluation on the validation set of DROP dataset.
Models
EM(%)
F1(%)
Specialized Models
NumNet Ran ct al 2019
64.9
68.3
MTMSN FTu et al 2019
76.7
80.5
NeRd CChen ct al. 20206
78.6
81.9
QDGAT Chen al. 2020a
84.1
87.1
Vanilla I.Ms
BART (Lewis et
2020
67.4
70.6
SOLIS
72.9 (+55.5
76.1 (+5.5)
T5 Raffel al. 2020
61.0
64.6
SOLIS
69.9 (+-8.9)
73.5 (+88.)
Reasoning L.Ms
GENBERT Geva et al
2020
68.8
72.3
w. SOLIS
70.5 (+1.7)
74.4 (+2.1)
TAPEX (Liu 2022
76.3
79.3
SOLTS
78.5 (+2.2)
81.6 (+2.3)
POET-SQL (Pi al. 2022
76.9
80.0
SoLIS
78.2 1..)
82.0 (12.0)
Meanwhile, we select cases in which the answer is greater than 1000, identify them as "hard" cases, and additionally report the average performance them. As shown in Table 2 all of proposed algorithms significantly improve the performance of LMs, especially in hard cases. For example, the search-based algorithm boosts BART with absolute 30.4% improvement hard cases. The full results of the performance comparison can be found in Appendix D Notably, since the search-based algorithm is the most effective, we apply the default algorithm in SoLIS.
Table shows the experimental results of different models on DROP dataset. As shown, SOLIS can bring consistent and significant improvements over all backbone LMs, especially for the vanilla LMs. Taking the T5 model as an example, it could be boosted by maximum of 8.9% with SoLIS. Even for PoEr-SQL which are already pre-trained for numerical reasoning, our method yields 2.0% Fl improvement, pushing the best LM performance to 82.0% F1. T presents the exper- imental results on AddSub and Arith. The results indicate that our approach is surprisingly effective for giant LMs and can further boost the performance of chain-of-thought prompting.
Table 4: Few-shot evaluation on the AddSub and MultiArith dataset.
Language Model
Setting
AddSub
MultiArith
Few-shot Vanilla Chowdhery ct al.
2022
42,2
PaIMM (540B)
Few-shot Chain (Wi et al. 20226
91.9
94.7
Zero-shot Chain (Koimaa et al. 2022
66.6
63,8
SoLIS
89.4 (+-22.8)
80.0 ( 16.2)
GPT-3 (175B)
Few-shot Chain Wei ct al. 2022h
88.4
96.7
w. SOLIS
90.9 (+2.5)
98.7 (+2.0)

---

Under review as a conference paper at ICLR 2023
Table 5: Case study on derived expressions using PoEr-SQL w. SoLIS on DROP. Listed are, the intention, the example question with intention trigger words (i.e., the colorful spans) and the derived expression, and the proportion of each intention.
Question Intention
Example Question with [Derived Expression]
Proportion
Addition
How many total yards of touchdown passes were there?
8.92%
ly =x + X2 x2 + xs]
Diff Constant
How many in percent in the county from the census of 2000
36.49%
weren't English? ly 100 x]
Subtraction
How many more percentages of people were germans compared
54.25%
irish? ly x X2]
Composition
How many more Albanian citizens were there compared to Bul-
0.34%
garian and Georgia citizens combined ? ly Xo (x1 + xa]]
6
MODEL ANALYSIS
Arithmetic Relationship Inversion In addition to performance improvement, SOLIS features the ability to derive an arithmetic expression for each question, whereas no such information is available during training. To better understand if these expressions align with question intentions, we collect all derived expressions and categorize them into four types in T As demonstrated, the majority of expressions contain addition and subtraction between variables and constants, which are largely consistent with the question intention, highlighting the superior interpretability of SOLIS.
The possibility that the anchor answers provided by reasoning LMs are inaccurate presents a challenge for the solving algorithms. To measure the robustness of our solving algorithms, we roughly decrease the probability that anchor answers are correct by decreas- ing the number of few-shot demonstrations in Figure 3 As shown, even though the backbone LM performance drops to 60.0%, the improvement of SoLIS is still as high as to 5.1%, suggesting its robustness.
Solving Algorithm Robustness
Number Substitution To study the impact of different factors during the number substitution stage, we conduct experiments on MathExp in Figure 4 As demonstrated, expanding the range of anchor numbers results in minor performance drop, showing that the reasoning LM more familiar with small integers. Furthermore, increasing the size of anchor number groups gives a large improvement on the performance, especially when there are four operands.
Limitation Discussion The first limitation of our framework is that we cannot support expressions that cannot be solved with linear systems. For example, with respect to the question "How many yards was Donovan McNabb's 's longest rushing TD?", the expected expression [y max, (x+)] is not supported by SoLIS. Second, the framework is less efficient when there are many operands. On the one hand, the group of anchor numbers would be quite huge, making the algorithm's 's runtime unacceptable. For example, when expanding to 5 operands, number substitution must be performed at least 50 times. On the other hand, for the search-based algorithm, the search space will increase exponentially, making the algorithm impracticable. Last, we assume a certain level of numeracy understanding of the reasoning LM. Therefore, if the reasoning LM is unable to comprehend the numeracy relationship, our method would not work well.
RELATED WORK
Numerical Understanding and Reasoning via Specialized Models Since our work focuses on numerical reasoning, it is related to previous works on numerical understanding, which has been found to be important for deep learning models (Spithourakis & Riedel 2018 Wallace et al. 2019 Naik et al. 2019 Zhang et al. 2020 Sundararaman et al. 2020 Thawani et al. 2021). In general, previous work on numerical understanding aims to develop better numeracy embeddings that accu- synthetic tasks. For example, Wallace et al. (2019) proposes List Maximum, an evaluation task by predicting the index of the maximal number in list of five numeracy embeddings. When it comes to downstream
rately reflect their properties, so they typically evaluate numeracy embeddings
8

---

Under review as a conference paper at ICLR 2023
Few-shot Chain Il Few shot Chain w: SoLIS
Two Operands
Three Operands
Four Operands
100
90
90.9
68.4
Exnet Match (%)
100
Exact Match
80
80
60
70
60.1
40
60
20
50
55
10
20 100
15
20
25
30
# of Few-shot Examples
Anchor Number Range
Anchor Number Group Size
Figure 3: Experimental results of Few-shot Chain with or without SOL1S on AddSub as the number of few-shot examples decreases.
Figure 4: The experimental results SoLiS on Math- Exp with different choices of anchor number range (left) and anchor number groups (right).
applications, the majority of previous works (Jiang al. 2020 Duan et al. 2021 are studied on classification or regression tasks. Differently, numerical reasoning involves flexible answers that go beyond classification (e.g., requiring complex calculation), which is our focus. As for numercial reasoning, previous works generally design trainable specialized modules and equip LMs with them to tackle different kinds of numerical reasoning problems (e.g., counting). While these methods work well on specific datasets (Dua et al.) 2019; Andor et al. 2019 Hu et al.] [2019: Ding et al.) 2019), they are hardly suited across different datasets and backbone LMs (Chen et al. 2020b). Dif- ferently, since our method does not require additional model training. it is applicable to almost all models, even those that only provide an inference interface (c.g.. GPT-3). As for methods that first generate programs logic forms, it is quite laborious to define domain-specifer language and col- lect corresponding training data (Berant et al., 2013). Unlike them, our methods does not require extra annotated programs. Instead, our method allows for the program discovery from examples via solving linear systems.
Numerical Reasoning via Pre-training This line of work always focuses on the pre-training of lan- guage models with corpus which involves reasoning. The corpus can be reasoning-orientel natural language texts from Internet (Deng et al. [202]] B Lewkowyez al., 2022), human-designed templates filled by different data sources GGeaa et al. |2000) Yoran et al. (022). or programs with rich reason- ing semantics (Liu al. 2022 Pi al. 2022). Although this kind of pre-training allows language models to perform better reasoning, they still require considerable computation budgets during pre- training and may still be challenged by complex numbers. In contrast, our method efficient since it can be integrated into existing models without further training or pre-training.
Numerical Reasoning in Giant Language Models Recent works demonstrate that with proper prompting, giant language models (e.g.. GPT-3) perform much better than smaller ones on several reasoning tasks (Wei et al.] [022b]a Kojima et al.) 2022 Li et al.) 2022 Zhou al., 2022; Wang a 2022). For example, with the chain-of-thought prompting, the few-shot PaLM model (Chowd- hery et al. 2022) can beat the previous best fine-tuned model on math word problems. However, their conclusions do not generalize non-giant language models. Different from them, our method can be simultaneously applied to language models ranging from millions (e.g., BART) to billions (e.g., GPT-3). Moreover, our work is orthogonal to these giant LMs and can be complementary to each other. For example, Section 5 shows that our approach can further boost the numerical reasoning capability of GPT-3 with chain-of-thought prompting.
CONCLUSION
In this work, we present SOLIS, a framework which can elicit numerical reasoning in language models at test time. Motivated by the fact that language models usually excel at simple numbers, SOLIS uses simple numbers as anchors inversely derive the implicitly inferred arithmetic ex- pressions from language models, and subsequently apply these expressions to complex numbers to perform numerical reasoning. With modeling the expression derivation as solving linear systems, we propose three kinds of algorithms to achieve SOL.S with noisy signals. Experimental results on several numerical reasoning benchmarks demonstrate that SOL.S can be integrated to a variety of language models, and can greatly improve their performance under zero-shot, few-shot, and fine- tuning scenarios. Our work provides a new perspective towards tackling numerical reasoning, which can be potentially applied to more language models and numerical reasoning tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Daniel Andor, Luheng He, Kenton Lee, and Emily Pitler. Giving bert a calculator: Finding oper- ations and arguments with reading comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5947-5952, 2019.
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1533-1544, Seattle, Washington, USA, October 2013. Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffiey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020a.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems,
2020b.
Kunlong Chen, Weidi Xu, Xingyi Cheng, Zou Xiaochuan, Yuyu Zhang, Le Song, Taifeng Wang. Yuan Qi, and Wei Chu. Question directed graph attention network for numerical reasoning over text. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing (EMNLP), pp. 6759-6766. 2020a.
Xinyun Chen, Chen Liang, Adams Wei Yu, Denny Zhou, Dawn Song, and Quoc V Le. Neural symbolic reader: Scalable integration of distributed and symbolic representations for reading comprehension. In ICLR, 2020b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023. 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Xiang Deng, Yu Su, Alyssa Lees, You Wu, Cong Yu, and Huan Sun. ReasonBERT: Pre-trained to reason with distant supervision. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6112-6127, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10. (0.18653/v1/2211 .et.0221eemlppmmiii 494.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:/8100004805 2018.
Ming Ding, Chang Zhou, Qibin Chen, Hongxia Yang, and Jie Tang. Cognitive graph for multi-hop reading comprehension at scale. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2694 2703, Florence, Italy, July 2019. Association for Com- putational Linguistics. doi: 10.18653/v 1259.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp- 2019.
10

---

Under review as a conference paper at ICLR 2023
Hanyu Duan, Yi Yang, and Kar Yan Tam. Learning Numeracy: A Simple Yet Effective Number Embedding Approach Using Knowledge Graph. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2597-2602, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 18.133/3/v 1/2021 http:///.......nnneeeee findings-emmlp.221. URL.
htt.ps://aclaattee www.aatttoooooo......o....o lgt. f nns-ininggsemmn 12.221 221
Mor Geva, Ankit Gupta, and Jonathan Berant. Injecting numerical reasoning skills into language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pp. 946-958, 2020.
Sumit Gulwani. Automating string processing in spreadsheets using input-output examples. ACM Sigplan Notices, 46(1):317-330, 2011.
Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei. Language models are general-purpose interfaces. arXiv preprint arXiv:2206.0633 2022.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 16000-16009, 2022.
Minghao Hu, Yuxing Peng, Zhen Huang, and Dongsheng Li. A multi-type multi-span network for reading comprehension that requires discrete reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNLP), pp. 1596-1606, 2019.
Chengyue Jiang, Zhonglin Nian, Kaihao Guo, Shanbo Chu, Yinggong Zhao, Libin Shen, and Kewei Tu. Learning Numeral Embedding. In Findings of the Association for Computational Linguis- fics: EMNLP 2020, pp. 2586-2599, Online, November 2020, Association for Computational URL pettps://aclantho..o
Linguistics. doi: 10. 18.18 www.:00...nninngemmmii. org/2020. findings-enntnn
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916, 2022.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising eqqeeceeto-sequeene pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880 2020.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. arXiv preprint 2rXhv:2206.14856 2022.
Yifei Li. Zeqi Lin, Shizhuo Zhang, Qiang Fu, Bei Chen, Jian-Guang Lou, and Weizhu Chen. On the advance of making language models better reasoners. arXiv preprint arXiv:2206.02333 2022.
Chen Liang, Jonathan Berant, Quoc Le, Kenneth D) Forbus, and Ni Lao. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. arXiv preprint 1eXiiv/6/1..000 2016.
Qian Liu, Dejian Yang, Jiahui Zhang, Jiaqi Guo, Bin Zhou, and Jian- Lou. Awakening latent grounding from pretrained language models for semantic parsing. In Findings of the Associa- tion for Computational Linguistics: ACI-IICNIIP 2021, pp. 1174-1189, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/202 http:///.22.....nnneeeeeeee findings-acl.
Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. TAPEX: Table pre-training via learning a neural SQL executor. In International Conference on Learning Representations, 2022.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL /ttpttts///oeeeeeeieee... wwt.:/eeeeeeooa wtwttoortt/ttoeeeoooom wtt.::/wwwm.....d.......... CqY7

---

Under review as a conference paper at ICLR 2023
Aakanksha Naik, Abhilasha Ravichander, Carolyn Rose, and Eduard Hovy. Exploring Numeracy in Word Embeddings. In Proceedings of the 57th Anmual Meeting of the Association for Computa- tional Linguistics, pp. 3374-3380, Florence, Italy, July 2019. Association for Computational Lin- URL https: ac /aalaalllaanhtccc anthol horechhoogoy... org/P1 9--11--1222
guistics. doi: 111118661/1191//
Xinyu Pi, Qian Liu, Bei Chen, Morteza Ziyadi, Zeqi Lin, Yan Gao, Qiang Fu, Jian-Guang Lou, and Weizhu Chen. Reasoning like program executors. arXiv preprint arXiv: 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu, Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020.
Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, and Zhiyuan Liu. Numnet: Machine reading comprehen- sion with numerical reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2474-2484, 2019.
Yasaman Razeghi, Robert Logan IV, Matt Gardner, and Sameer Singh. Impact of pretraining term frequencies on few-shot reasoning. arXiv preprint arXiv:220..07206 2022.
Subhro Roy and Dan Roth. Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1743-1752, 2015.
W. Samek, A. Binder, G. Montavon, S. Lapuschkin, and K. Müller. Evaluating the visualization of what deep neural network has learned. IEEE Transactions on Neural Networks and Learning Systems, 28(11):2600-2673 2017. doi: 10.1 1.L119/TNNNLS2221655988822
Georgios Spithourakis and Sebastian Riedel. Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Melbourne, Australia, July 2018. Association for Computational Linguistics.
Dhanasekar Sundararaman, Shijing Si, Vivek Subramanian, Guoyin Wang, Devamanyu Hazarika, and Lawrence Carin. Methods for Vlumeracy-Preservingg Word Embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4742- 4753, Online, November 2020. Association for Computational Linguistics. doi: 10. 18653/vvv
2020.emnlp- maiammnlpmain38.... URL htt.h:tts://aa/ /ttt::::/aalaantoooooo.or////// /ttpaa//AAAAAtttt////aaaaaattaat
em..p-0mmmmmmmaa.......m....m. 384
Avijit Thawani, Jay Pujara, and Filip Ilievski. Numeracy enhances the Literacy of Language Models. In Proceedings of the 202/ Conference Empirical Methods in Natural Language Processing, pp. 6960-6967, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguisties. doi: 10.18653/v 1/2553//11222..omm 1222222cmmlmpmmai..55555 org/ 2021 emn p- www.n-nmaaa. In. 557
URL
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. Do nlp models know numbers? probing numeracy in embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 5307-5315. 2019.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.1117 2022.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. ArXiv, Abs/2206.07688 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 11903, 2022b.
12

---

Under review as a conference paper at ICLR 2023
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art art natural language processing. In Proceedings of the 2020 conference empirical methods in natural language processing: system demonstrations, pp. 38-45, 2020.
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga. Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida Wang, al. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. arXiv preprint arXiv:2201.05966 2022.
Ori Yoran, Alon Talmor, and Jonathan Berant. Turning tables: Generating examples from semi- structured tables for endowing language models with reasoning skills. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume Longg Papers), pp- 6016-6031, 2022.
Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar, and Dan Roth. Do Language Embeddings capture Scales? In Findings of the Association for Computational Linguistics: EMNLP 2020. pp. 4889-4896 Online, November 2020. Association for Computational Lin- egy. org/
guistics. doi: 10. 10.18653/v1/000.0 000.8663///2200...nnnggemm////// 2020. -indfiiss-smmmnn 439
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schu- urmans, Olivier Bousquet, Quoe Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205. 10625, 2022.
13

---

Under review as a conference paper at ICLR 2023
A PRELIMINARY STUDY DETAILS
Here present the model performance MathExp of GPT-3 with different solving algorithms in Figure and Figure 6 We can conclude that: (1) both algorithms are not sensitive with either the
floating point precision the integer range; (2) the search-based algorithm is most robust than the
analytical-base algorithm with respect to the number of operands.
-O- Twe Operands
-0
Three Operands
Four Operands
Two Operands
Three Operands
Four Operands
100
100
Exact Matc (8)
Exad Match (S)
80
40
20
$10' 300"000"gi0
S10' g10² Si0"
Floating Point Precision
Integer Range
Floating Point Precision
Integer Range
different floating point precision (left) and integer range (right) on MathExp of GPT-3 search-based algorithm.
Figure 5: Performance
Figure 6: Performance over different floating point precision (left) and integer range (right) on MathExp of GPT-3 w. analytical-based algorithm.
B OPERAND PROPOSAL DETAILS
we mention that the textual context on a realistic dataset may be noisy, i.c., contains irrelevant numbers, thus we need to locate the operand number first. We substitute 10 times for each number appearing in the paragraph, the output gives 2 3 different prediction numbers out of 10, decide the current tested number is involved to the answer. Moreover, substitute numbers fol- lowing a template: suppose the original number x is with precision p. then the substituted numbers can be represented
In Section 3.2
10P, where (-5, -4, -2,
3, 4,5
C EXPERIMENTS
C.1 EXPERIMENTAL SETUP
Table 6: Statistics of DROP dataset
Train
Dev
Dataset
# Questions
# Docs
# Questions
Docs
DROP
77,409 409
5, 565
9, 536
582
Table 7: Statistics of MAWPS dataset
Subset
# Questions
AddSub 395
MultiArith
600
For BART, we implement the fine-tuning methods using the Huggingface transformers library Wolf ett al,, 2020) on 4 V100 16GB GPUs. We use BART (Lwiss et al., 2020) as our backbone. We same-scale reasoning-pretraine PoET-SQL. and TAPEX models in experiments. For T5,
14

---

Under review as a conference paper at ICLR 2023
implement its fine-tuning the Huggingface transformers library A100 GPUs. We use T5LARGE (Raffel et al. 2020) as our backbone.
apply Adam (Loshchilov & Hutter 2019) optimizer. The fine-tuning epochs are set as 50. For BART models (i.e., BART and PoET- SQL), we follow previous works (Pi et al. 2022 to set the batch size as 128 and the learning rate as 3 x 10-0 For T5, we decrease the batch size to 32 due to the computational budget. The early stop technique is used to save training time. For zero-shot few-shot evaluation, we employ the GPT-3 API. We keep the temperature as default setting O, and set the maximum output tokens to 128. As for anchor number groups: the group size is 6/8/ /110 corresponding corresponding to 2/3/4 operands on DROP; the group size is 4 on AddSub, and 10 on MultiArith because MultiArith requires more compositional operations.
Hyperparameter Selection For fine-tuning evaluation,
C.2 EXPERIMENTAL DETAILS ON DROP
Fine-tuning Details For all fine-tuning methods, select the default max token length for each model. We set the max token length of generation as 96. To save training time, we set early stop mechanism: evaluate the EM and F1 score per 500 1000 steps, if the performance does not increase in the latest 20 evaluations, we stop the training and save the best checkpoint.
On DROP, pre-pend the question the given paragraph. For multi-span answer, insert "." between each span and make up the final answer. For T5LARGE, we also insert "
" token between the question and the given paragraph. Since most LMs' checkpoints on DROP is currently not off-the-shelf,, we re-in r--mmllmennt them and compare to the results reported in previous works. We present the comparison results Table g
Table 8: Performance Comparison DROP between reported results in previous works and our re-implementation. Results marked with represent our re-implementation results.
Models
EM (%)
F1 (%)
BART (Pi et al.
2022
66.2
69.2
BART
67.4
70.6
T5 Yoran al. T5
2022
64.6
61.0
64.6
POET-SQL. (i et al.
2022)
77.7
80.6
PoEt-SQL*
76.9
80.0
C.3 COMBINATION WITH MAJORITY VOTING (WANG ET AL. 2022
Wang et al. 2022) proposed self-consistency, which samples different reasoning paths, and then pick the most consistent final answer by majority voting. Such majority voting method proves to improve LLMs' performance over various reasoning benchmarks. We have also conducted experi- ments combining majority voting and SoLIS, to check if there further improvement. We present the performance comparison in Table 9
Table 9: Combination with Majority Voting (Wang et
2022
Language Model
Setting
MultiArith
Zero-shot Chain Kojima al. 2022
63.8
GPT-3 (175B)
majority voring
73.5 (+9.7)
majority voting +SOLIS
83.5 (+ 19.7)
15

---

Under review as a conference paper at ICLR 2023
D MORE RESULTS ON DROP
We present the performance breakdown of on dev set of DROP in To from fine-tuning models on DROP dataset, also use GPT-3 conduct study on few-shot learning. We pre-pend 10 random training samples in train set, and run all cases where answer type equals to "number". We also apply our search-based algorithm on GPT-3. To save API calling time, we only substitute the number for one time. Table II presents the F1 score comparison.
We also summarize common calculation error cases in our tested language models and present some of them for case study in Table 12 which again illustrates the unreliability of language models.
Table 10: Breakdown of model score by answer types on the dev set of DROP
Models
Number
Span
Spans
Date
Total
BART
66.3
80.3
66.0
56.7
70.6
SOLIS
75.2
80.5
66.7
55.7
76.1
T5"
55.5
8I.6
73.0
53.5
'64.6"
SOLIS
69.8
81.8
73.9
53.5
73.5
TAPEX
77.8
84.3
72.9
62.8
79.3
SOLIS
81.4
84.4
73.0
61.7
81.6
POET-SQL
78.4
84.6
76.6
63.4
80.0
w. SOLIS
81.4
84.9
76.9
62.6
82.0
Table 11: Few-shot evaluation of GPT-3 w. SoLIS on the DROP numeric subset.
Language Model
Algorithm
F1(%) on Hard
F1(%) on Total
42.5
64.7
GPT-3 (175B)
Scarch
59.9 (+17.4)
68.7 (+4.9)
Table 12: Common calculation error cases on DROP dataset.
Error Type
Example
Prediction
Label
Carry Error
... the size of the black-white IQ gap in the United States decreased from 16.33 to 9.94 IQ points.
6.49
6.39
Q: How many IQ points did the black-white IQ gap de- crease in the United States in a 2013 analysis of the Na- tional Assessment of Educational Progress?
Missing High Digit
The Department of Tourism recorded 26,861,093 Thai 499287 and 11,361,808 foreign visitors Bangkok in 2010.
15499287
Q: How many Thai visitors did Bangkok have in 2010 compared to other foreign visitors?
Extra Integer digit
Rayner nailed a 23-yard field goal ... Rayner got a 54- y yarder and 46-yarder to end the half
111113
123
Q: How many total yards of field goals did Dave Rayner have?
Extra Float Number Digits
have estimated the IQ means of 17-year-old black, 3.75 white, and Hispanic students range respectively from 90.45-94.15
3.7
Q: How many points difference the range in 17-
year-old black students?
Insufficient Precision
The Diocese of Karelia has 22,000 church members 12 parishes....
1833
1833.33
Q: How many church members approximately are in cach one of the 12 parishes?
16


Reviews:
[{"review_id": "jItdnZFAzVl", "reviewer": "Reviewer_Lb3T", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The author proposed a novel method combining a large language model with solving linear systems to improve the calculation over large number performance.\nContributions:\n1. A novel approach that takes advantage of neural networks' ability to solve a simple math problem, and generalizes the ability through number substitution and linear system solving.\n2. Natural language model is agnostic so the approach is generalizable to different models.\n3. Have good performance gain even with zero shot\n\n---------\n\nUpdate about my final recommendation: \nI am convinced with the rebuttal material and happy to accept this paper. \n", "summary_of_the_review": "The author proposed a novel method combining a large language model with solving linear systems to improve the calculation over large number performance. The experiment shows this model-agnostic approach can improve the performance of different natural language models by a non-trivial amount. ", "strength_and_weaknesses": "Strengths: \n1. The paper writing is very clear and the story is convincing.\n2. The study using different solving strategies demonstrates the approach can be configured toward different needs\n3. The experimental studies show it is pretty generic and can improve different neural baselines\n\nWeakness:\n1. The experiment setup is a little bit vague. There are multiple setups, including zero-shot, few shots, and fine-tuning. However, it is hard to tell which setup is used to perform one experiment just from the figure and the caption in the evaluation section. \n2. Expressiveness: The system can only handle up to the linear system, and cannot generalize to aggregation, or recursion yet. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bMl-qj3fYFc", "reviewer": "Reviewer_8HSz", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors introduced a clever and simple method that improves the arithmetic capabilities of LMs.", "summary_of_the_review": "The authors introduced a clever and simple method that improves the arithmetic capabilities of LMs. Assessing the full usefulness of this method may require more work.", "strength_and_weaknesses": "The method is easy to understand and implement and offers improvements on various reasoning datasets. It needs to be clarified, though, whether there exists a path toward applications beyond systems of linear equations. I also would like to know whether the method helps when combined with majority voting.\n\nThe Drop dataset used for evaluations in this paper is excellent, but its arithmetic is quite restricted. Are there any non-synthetic NLP datasets that have systems of linear equations? How much gain would we see on GSM8K and MATH? Can you perform a qualitative analysis for Drop, GSM8K, and MATH, highlighting the types of errors that were fixed?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WfxKEJaqEIy", "reviewer": "Reviewer_qbBT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes SOLIS, where during model inference, complex numbers are substituted with simpler numbers (anchors) from 1 to 20, the language model outputs the arithmetic between the anchors, and through these, the original answer of the arithmetic between the complex numbers are derived through solving linear systems. The operand (to be substituted by anchors) is chosen from the input text through the significance of their perturbations on the model performance. Finally, after having the operands and their anchors, the required output is generated through either one of the proposed 3 approaches to solving linear systems - analytical, search-based, and heuristics. \n\nAlthough missing some crucial baselines (discussed in the section below), the authors show that LMs can improve their performance for numeral prediction with the incorporation of their SOLIS framework.\n\n## Edit After Reviewing Author Response\n\nAfter reviewing the author response, I have updated my recommendation to a weak rejection of the paper.  This is primarily due to the clarification of the goals by the authors and incorporating (and demonstrating improvement on) the GenBERT baseline. The reason for non-acceptance despite the paper having some novel contributions and being well-written has to do with my initial comment on the lack of the proposed model to work with both textual and numeric data which is something (also acknowledged by authors in the response) that some previous works are able to do (see references in `summary of review` below).\n", "summary_of_the_review": "The lack of important baselines I can overlook as the idea is interesting, however, I base my recommendation on the first part of the paper weakness - the offered solution is antithetical to the premise of the paper. If the problem statement is that language models struggle to understand numbers that appear in tables, text, and knowledge bases, the solution cannot be to approximate a calculator as an addition to the inference chain. To elaborate, take a supposed instance of any QA dataset (SQuAD or DROP) where the context is \"X scored 4 goals while Y managed to only score 3\", the question is \"Who scored more goals?\" and the answer is textual - \"X\". The framework suggested can compute arithmetic between 4 and 3, but does not help the LM better understand what 4 and 3 are such that it can make an informed decision. There are several lines of work that work on the premise that this paper takes, but provide interventions that can be applied to both textual and numeric data:\n\n1. Spithourakis, Georgios, and Sebastian Riedel. \"Numeracy for Language Models: Evaluating and Improving their Ability to Predict Numbers.\" Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018.\n\n2. Jiang, Chengyue, et al. \"Learning Numeral Embedding.\" Findings of the Association for Computational Linguistics: EMNLP 2020. 2020.\n\n3. Zhang, Xikun, et al. \"Do Language Embeddings capture Scales?.\" Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP. 2020.\n\n4. Geva, M., Gupta, A., & Berant, J. (2020). Injecting numerical reasoning skills into language models.\n\n5. Sundararaman, Dhanasekar, et al. \"Methods for Numeracy-Preserving Word Embeddings.\" Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020. \n", "strength_and_weaknesses": "## Strengths:\n1. The ideation is interesting - authors leverage the fact that LMs are good at arithmetic for numerals 1-20 that are highly frequent in the training corpus and use this for eliciting arithmetic operations between more complex numbers.\n\n\n2. The ideas are clearly expressed in the writing/figures/tables and thus are easy to grasp.\n\n## Weakness:\n\nAmong the two weaknesses of the paper, the first is a deal-breaker. The authors establish the premise for the need of numeric comprehension in language models as 1) numeric reasoning forms the crux of understanding text, tables, and knowledge bases 2) extrapolation for numerals outside of the pre-training corpora. Here, the approach that the authors have taken simply appends a system on the top of untouched LMs that does the computations for the LM. This appendage does improve performance when the task is strictly to output a number. However, tracing this back to the original premise of the paper, unless the representations of numbers of these LMs change or adapt, this appendage makes no difference in the understanding of numerals by the LM. Case in point, a DROP dataset instance where the LM has to understand numerals in the paragraph but the output is textual - I believe this appendage provides no significance in the understanding of numerals. This is further corroborated by the paper itself - section 5.3 - where the authors state that for their experimentation, they have specifically chosen instances of the DROP dataset that have numeric outcomes.\n\nTwo essential works that also aim to improve numerical comprehension and also evaluate themselves on the DROP dataset (including instances where the output is textual), one of which is cited in the paper, is not used as baselines:\n\n1. Geva, M., Gupta, A., & Berant, J. (2020). Injecting numerical reasoning skills into language models.\n\n2. Sundararaman, Dhanasekar, et al. \"Methods for Numeracy-Preserving Word Embeddings.\" Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "B1gb6jezIRw", "reviewer": "Reviewer_Dv6a", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a method to identify relevant numbers in text, replace them with smaller numbers that are more accurate to the LM, and use the input-output pair to search for a linear system.", "summary_of_the_review": "The proposed method is novel and thoroughly evaluated, and proven to be effective. It is however not clear if the method can be extended to more complex numerical reasoning tasks.\n\n-----------After rebuttal--------------\n\nI am happy with the rebuttal and I still recommend acceptance for this paper.", "strength_and_weaknesses": "Strength:\n1. Novelty is high in my opinion. Numerical instability is a long-standing problem in neural-based reasoning. While most research attempts to use neural-symbolic methods, this paper figured out a way to identify relevant numbers and outsource the finding solution steps to classicial methods.\n2. Experiments are through with good analysis. Comprehensive ablation studies are performed to identify the contributions of variations in LMs and different linear system search methods.\n\nWeakness:\n1. Currently the method is limited to linear system, which probably is OK since most arithmetic in natural language text are not complex. But there are areas where non-linear solutions are required (imagine solving high-school math), how can the method be extended to that case?\n2. It would be great if the authors can show some failure cases. I wonder which step leads to the most failures? Is it finding the right number in the text, or the search for system solution?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-Ukb3Aek1q", "reviewer": "Reviewer_AWSU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method to infer the arithmetic expressions performed by a language model while solving a mathematical reasoning question with a chain-of-thought style output. The inferred expressions are then calculated outside of the LM. This aims to improve the performance of LLMs on mathematical reasoning tasks by increasing the accuracy on arithmetic calculations. ", "summary_of_the_review": "The paper presents an interesting idea for eliciting the mathematical expressions that the LM is implicitly computing, and offloading such computations to an external calculator. However, the proposed approach is extremely complicated compared to the problem that it's trying to solve, and does not generalize well to other situations nor scale to more complex arithmetic expressions. There are simpler and more general approaches that are likely to perform well and these alternatives are not benchmarked. Therefore I think the current paper does not convincingly demonstrate the value of the proposed approach.", "strength_and_weaknesses": "Strength:\n\n- The method itself appears to be novel. It involves extracting the relevant numbers and substituting large numbers in the chain-of-thought answer with random small numbers where the LM can reliably perform calculations on. By sampling a set of these input and output correspondences, the authors specify and solve a linear system of equations to elicit the arithmetic relationships. \n\n- The method leads to performance gains in the settings evaluated in this paper.\n\nWeakness:\n\n- The main issue with this work is that this method is a very complicated way to do something that is much simpler in comparison (simple arithmetic calculations). Despite its complexity, the method does not guarantee that the correct calculations are extracted and does not scale well to more complex expressions. It seems that similar performance gains can be achieved by prompting the LLMs to output its reasoning steps in the format of mathematical expressions, and sending the evaluation to an external calculator. This approach was already explored in [1] with some success. One can also imagine finetuning the model to output well-formatted arithmetic expressions in a task-agnostic way. A version of this was explored in [2] on the GSM8k dataset. These alternatives are much simpler and more general than the proposed approach, and the paper has not benchmarked against these simpler approaches to show that it offers a meaningful improvement. \n\n- The idea of formalizing the reasoning steps of a LM through perturbing its output and inferring the relationships may be an interesting framework, but there is no discussion of extending this particular approach to something that cannot be readily achieved by simple methods. \n\n[1] https://arxiv.org/pdf/2201.11903.pdf\n\n[2] https://arxiv.org/pdf/2110.14168.pdf", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject