ELASTIC: Numerical Reasoning with Adaptive Symbolic Compiler
Authors: Jiaxin Zhang
Yashar Moshfeghi
Venue: NeurIPS 2022 Accept (2022)
Keywords: Numeral Reasoning
Encoder
Decoder
Deep Learning
AI
Hierarchical Decoder
Mathematical Symbols

Abstract: Numerical reasoning over text is a challenging task of Artificial Intelligence (AI), requiring reading comprehension and numerical reasoning abilities. Previous approaches use numerical reasoning programs to represent the reasoning process. However, most works do not separate the generation of operators and operands, which are key components of a numerical reasoning program, thus limiting their ability to generate such programs for complicated tasks. In this paper, we introduce the numEricaL reASoning with adapTive symbolIc Compiler (ELASTIC) model, which is constituted of the RoBERTa as the Encoder and a Compiler with four modules: Reasoning Manager, Operator Generator, Operands Generator, and Memory Register. ELASTIC is robust when conducting complicated reasoning. Also, it is domain agnostic by supporting the expansion of diverse operators without caring about the number of operands it contains. Experiments show that ELASTIC achieves 68.96 and 65.21 of execution accuracy and program accuracy on the FinQA dataset and 83.00 program accuracy on the MathQA dataset, outperforming previous state-of-the-art models significantly.

ELASTIC: Numerical Reasoning with Adaptive Symbolic Compiler
Jiaxin Zhang University of Strathclyde 16 Richmond Street, Glasgow, GI IXQ jiaxin. zhangōstrath.ac ac uk
Yashar Moshfeghi University of Strathclyde 16 Richmond Street, Glasgow, GI 1XQ
yashar www.f moshfeghi@stratteeee ac. uk
Abstract
Numerical reasoning over text is challenging task of Artificial Intelligence (AI), requiring reading comprehension and numerical reasoning abilities. Previous approaches use numerical reasoning programs to represent the reasoning process. However, most works do not separate the generation of operators and operands, which are key components of a numerical reasoning program, thus limiting their ability to generate such programs for complicated tasks. In this paper, introduce the numEricaL reASoning with adapTiive symbollc Compiler (ELASTIC) model, which is constituted of the RoBERTa the Encoder and Compiler with four modules: Reasoning Manager, Operator Generator, Operands Generator, and Memory Register. ELASTIC robust when conducting complicated reasoning. Also, it is domain agnostic by supporting the expansion of diverse operators without caring about the number of operands it contains. Experiments show that ELASTIC achieves 68.96 and 65.21 of execution accuracy and program accuracy on the FinQA dataset and 83.00 program accuracy on the MathQA dataset, outperforming previous state-of- tafte-of-the-eee -hee-art art models significantly/
Introduction
5 show astonishing performance over reading comprehension tasks like SQuAD (6). However, PLMs fall short of numerical reasoning over text (7). which requires conducting numerical reasoning based on understanding the text. Hence, numerical reasoning over text is more challenging than reading comprehension
Recently, Pre-trained Language Models (PLMs) 12
S and attracts the architecture to
interest of the Al community. Previous approaches adopt the sequence-to-sequeer
generate the sequential format of numerical reasoning programs (see (b) in Table I 9 10). However, the sequential format could produce invalid expressions such as "3 ((2)" because of the wrong position of parentheses [TT. To avoid this, some methods convert the reasoning program to the binary tree, then use the tree-decoder to generate the pre/post-order traversal sequence (see (c) in Table Alternatively, FinQANet [15] represents the reasoning program in a flattened format and generates the right parentheses forcibly after generating two consecutive operands. To increase the scalability, NeRd 7 introduces the symbolic operations and generates the reasoning program as the nested compositional format (see (e) in Table [ Researchers also investigate capture valuable information between entities and numbers to improve numerical reasoning ability. Some works use PLMs [8] 7 [6), while others, like Li et al. and Ran et al. [18), adopt graph neural network to encode the text.
12
Currently, proposed approaches struggle with two significant problems. Firstly, they are vulnerable to complicated numerical reasoning problems. The complicated numerical reasoning problems usually
ELLSTTC code found at http://///gttttoo///////////ttta
Submission-
358
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

contain a long reasoning program, in which the types of operators are diverse, and the number of operands is dynamic. Since most works do not separate the generation of operators and operands. their performance is hindered by cascading errors when encountering complicated tasks. Secondly, previous works lack extensibility for the operators, which arises from either the flaw of the model architecture or the representation format of the program, making them hard to apply to different data domains.
Table 1: An Example (from MathQA [9] dataset) requires solving the problem by conducting numerical reasoning. The numerical reasoning program could be represented by four different formats: sequential format, tree- traversv format, flatten format [15], nested format. #n refers to the executable result from the nth sub-program, and const refers to the constant number
Problem: A small table has a length of 12 inches and a breadth of inches. Cubes are placed on the surface of the table so as to cover the entire surface. The maximum side of such cubes is found to be inches. Also, few such tables are arranged to form a square. The minimum length of side possible for such square is 80 inches. What is the number for b?
(a) Numerical Reasoning Program: = 80 2 122
(b) Sequential Format:
/ ((80 + 4) x (80 + 4) -(12 x 12))
(c) Pre-order Traverse Format: fi= +, 4, +,80, 4, x, 12, 12, none
(d) Flattened Format: divide(80,4))pa)))) Hllpower(12,comg)) 2olppower(HH.constttsssssssaaaee ((rrT))))
(e) Nested Format:
sqrtsubbrrcc(oowerdddiddee. 4), const_ 2), power( const_ 2)))
Hence, we present the numEricaL reASoning with adap'Tive symbolle Compiler (ELASTIC) model. ELASTIC separates the generation of operators and operands, allowing to be less influenced by the cascading error from the complicated reasoning. Moreover, ELASTIC is adaptable to the number of operands following an operator, making it domain-agnosten Specifically, ELASTIC contains an Encoder part extracting the contextual representations of the passage and question and a Compiler part generating the numerical reasoning program. The Compiler consists of four modules: Reasoning Manager, Operator Generator, Operands Generator, and Memory Register. We conduct experiments on two challenging datasets: FinQA [15], and MathQA [19]. Since FinQA and MathQA are collected from different domains: annual financial reports and GRE/GMAT, ELASTIC demonstrates its adaptability by achieving state-of-the-ar results on both datasets. Furthermore, our ablation studies investigate how the length of the numerical reasoning program influences the model's s numerical reasoning ability, which shows that ELASTIC is less liable to being influenced by the cascading error. In addition, Departing Distance (M-MDD), which measures how difficult for the mode to use the executable results from the previous sub- sub-program. We use M-MDD to demonstrate the necessity of the Memory Register in ELASTIC. The contributions of our work are: (1) we present a numerical reasoning model ELASTIC with good adaptability and elasticity, which separates the generation of operators and operands. ELASTIC achieves state-of-the-ae results on two challenging datasets: FinQA and MathQA: (2) we introduce the design of separate modules and Memory Register, making ELASTIC perform stably on complicated numerical reasoning problems; (3) the proposed ELASTIC is domain agnostic because it supports diverse operators.
support diverse operators.
introduce the maximum Memory
2 Related Work
Making models to conduct numerical reasoning has attracted the AI community since the last century 20 Previous research has investigated making the model do numerical reasoning over text by using statistical learning methods to find similar equation pattern 21
23 24 Since deep learning has recently achieved great success in many tasks, Wang al. 25 propose DNS, which the first
22
deep learning model for solving math word problems far as we know. After their work, rescarchers

---

try to find better way to represent the numerical reasoning program. For example, Wang et al. 126] and Wang et al. [27] use the expression tree to represent the reasoning program. Sun et al.
create
tree-decoder GTS, which generates prefix traverse sequence of the tree. Chiang et al. [8 and Qin et al. [13] extract the semantic information from the question and passage texts and want to connect them with the reasoning steps. In addition, Li et al. (29). and Zhang et al. [30] introduce the graph encoder to capture the structural information or syntactic information to capture the relation between numbers and entities. Shen et al. [31] propose unified model, which uses both sequential and graph as the encoder, then uses seq2seq and tree decoder to generate the reasoning program
Furthermore, several datasets are proposed to evaluate the model's numerical reasoning ability, such as Math23K [32] and HWMP [[3]. There are also more challenging datasets, like ASdiv [33], and MathQA [9]. At the same time, Dua et al. propose DROP [4]. which requires more than arithmetic operations to conduct numerical reasoning. State-of-the-a models like NeRd (7) and NumNet [88] are introduced to solve the DROP dataset. More recently, dataset called FinQA [5] has been proposed, which constructed from the annual financial report.
Despite the considerable success achieved by these approaches, Patel et al. 135 argues that some state-of-the-art models, like GTS [I] and Graph2Tree [30], only learn the statistical relation instead of numerical reasoning ability. Unlike previous works, our model ELASTIC separates the generation of operators and operands, allowing it to conduct complicated numerical reasoning. Moreover, ELASTIC is adaptable to the number of operands following an operator, making it domain agnostic.
3 Approach
Figure D ssows the architecture of our ELASTIC model. ELASTIC consists of an Encoder part encoding the question text and problem text into contextual vectors and a Compiler part producing the numerical reasoning programs. The Compiler part consists of four modules: Reasoning Manager, Operator Generator, Operands Generator, and Memory Register. The Reasoning Manager leverages other modules to produce the numerical reasoning program. Since a complete numerical reasoning program usually contains several sub-programs, the generation steps between operators and operands are interchangeable. To help the following sub-programs use executable results from the previous sub-programs, Memory Register stores the sub-programs executable results into corresponding pre-defined cache tokens embeddings
Table 2: Task Definition Notation
Notation
Description
P. Q, R
Problem Text, Question Text, Numerical Reasoning Pro- gram
NUM
The numbers in P and Q
CONS
Constants defined in DSL
OP
All mathematical operators The ith operator in R
op;
OE
All operands
oe
All operands belonging to op;
oe)
The jth operands of op;
From either OP or OF, s constitute R
The i-th sub uub-program of R
r
ri = op,
oee
Task Definition Given the problem text P and question text Q, the task to generate a numerical reasoning program R. Both problem text P and question text Q consist of words and numbers (denoted by NUM). The Numerical reasoning program R represents the numerical reasoning process,
"See Appendix F for example showing how different modules work.
3

---

which is a sequence of symbols (denoted by s) from mathematical operators (denoted by OP) and operands (denoted by Operanes OE are from either constant numbers (denoted by CONS) defined in Domain Specific Language (DSL.) or NUM. CONS are the special numbers that do not exist in either the problem text P and question text Q, such as const_pi(m)) piss__p(m......)) Finally, the pattern of the numerical reasoning program At is defined as R = opi [oe)] -
j=0 f === where E OP, it is the dsh operator in R, and op; contains several operands je In addition, regard group of one operator and operands as the sub- sub-program r. For example, op; [oe]] is the ich sub-program ri, which can be executed since is a complete arithmetic program
3.1 Encoder Part
As shown in Figure 1 (Encoder), the Encoder takes the concatenated sequence of Q and P as input. The Encoder encodes the input sequence and outputs the contextual vectors h the Next, th used for the Compiler to produce the numerical reasoning program R. In this work, use RoBERTa as the Encoder. The outputs from the final layer RoBERTa is used as h me E Rh-s, where s is the maximum input length of the RoBERTa, and is the hidden size of RoBERTa. Note that ELASTIC not dependent on the specific type of encoder. Any model providing contextual vectors of the sequence be used.
Eap (ope-1)
Operator Generator
Reasoning Manager
opt
Question <seg>> Passage
b
g"
E
RoBERT1
A Jmc
softmax
Attention
or
GRU
ops
nhc
g2"
acf
Update
Memory Register
1=0 1b0
Eap (op.)
Reonsnum (0E,-1)
softmax
Operands
Encoder
Compiler
Figure 1: The overall architecture of the ELASTIC model. The Encoder part takes the sequence of question text Q and passage text as input, then generates the contextual vectors he The Compiler part consists of four modules: Reasoning Manager, Operator Generator, Operands Generator, and Memory Register. The right part of the figure shows a complete process of the generation of sub-program Tz. Firstly, Reasoning Manager sends the guidance vectors gdP to the Operator Generator, which guides the generation suspends the Operator Generator, then the Operands Generator takes nod and op: from the Operator Generator to produce the first operand oei- When finish the generation the sub-program rt, the Memory Register stores the results and updates the embedding vectors of cache token #t by of" Again, the Compiler repeats to generate next sub-program 1*+1-
operator opt. Secondly, Reasoning Manager
3.2 Compiler Part
Decoding Vocabulary and Token Embedding We first describe the decoding vocabulary. The decoding vocabulary consists of OP and OE, where OE can be further categorized into NUM and CONS. The embedding of symbol of the decoding vocabulary is represented by the embedding Popp.coms.mmm! (s), which the embedding look-up function. Hence, the embedding for symbol deñined as:
See Table 2 for the definition of all the notations. Also, Appendix E for example.

---

Eup (s)
if OP
e,
E Ecoms (s)
i E CONS
(I)
Enum (s) = h
if E NUM
The symbols embeddings of OP and CONS are two trainable embedding matrices Eop E R and
Ecoms E Rh oNo and Teyons refers sizes of OP and CONS respectively). The embedding for the symbol of NUM is hete R.a, where denotes the index position in the sequence of and P.
Reasoning Manager As shown in Figure (Reasoning Manager), the Reasoning Manager outputs the vector g, which guides the Operator Generator and the Operands Generator to produce op and The inputs for the Reasoning Manager from the Encoder and embedding of the previously generated symbol st-1- The Reasoning Manager first calculates the context vector by the normalized vectors of hede and the attention weights o;:
contextual vectors hd (ha for generating operators)
Σ h
(2)
exp(score(e,s (e hrm²)) 0; E, exp(score c..-.. hn(())
(3)
score (core(es.-1:1i h(
W1 W2h
(4)
where Wi E Rish and W2 E I and both are trainable parameters. The e summarizes the encoded information from the Encoder according the previous generated symbol Next, the Reasoning Manager adopts the GRU [36 network to generate the guidance output
g, h, GRU(Relu Wa/c:
E Dop. cop.cons,mum (8)- h.
(5)
where ":" represents concatenation. W; Rh is trainable parameter, and Relu is the activation function. h E Ih the hidden state of GRU from the previous step, and hy is O.
Operator Generator As shown in Figure I OOpeatoor Generator). Firstly, the Operator Generator receives the guidance vector g;" from the Reasoning Manager by inputting: contextual vectors het of tokens from the question Q, and embedding (opt.-- of the previously generated operator. Next, the Operator Generator calculates the probabilities i-th operator (denoted as i-op) of the OP:
exp E (i-op) Relu W
op
Pi--op| (opp:--))))) Σ Ejoopeor exp (j-op) Relu( W oPpg?")
where Wop Bith trainable parameter. The Operator Generator selects the operator with the highest probability as the predicted op. Next, unlike other models, the Reasoning Manager suspends the generation of operators and starts to generate operands {oe") through the Operands Generator.
Operands Generator As shown in Figure 1 (Operands Generator). The inputs from Operands Generator the Reasoning Manager are different from Operator Generator's s. Because oe could be number in Q or P, the contextual vectors ohe of all tokens used. Furthermore, the Operands Generator initializes the embedding the initial operand ((or)) as Relu( (W Eop g(]) (W4 E RR),2)), leveraging information of ope to produce oe Next, the Reasoning Manager outputs g for n-th step generation operand oef, Finally, the probability of i-th operand (denoted as i-oe) of the OE:
expl ET coms-mam (i-oe) Relu W oee
PP(i-ee] Excons (ve (0e Σj Ejoeeoo exp( ET (i-oE) Relu W Waegfe)
5

---

where oo E Rhth is trainable parameter. The Operands Generator selects the operand with the highest probability as the predicted oe. After one operand has been generated. the Operands Generator continues producing operands for the sub-program r.. The decoding process for the operands terminates when the token none produced.
Memory Register When generating sub sub-program ri, its operands could be the executable results from the previous sub-program (p < i). To make the Operands Generator be able to use the results from previous sub- sub--rograms. Inspired by Chen et al. [15], we introduce cache token # to the CONS of DSL, which is used for storing the information of executable results. Unlike other constants, #n does not point to a static value. It is different according to the different sub -roogrrgamm Tn- As the results, ELASTIC needs to update the representation of # after the sub-progran I' is generated. Specifically, the Memory and Register module update the cache #n by replacing its embedding with output gn. which is the guidance vector from Reasoning Manager to guide the generation of the last operands belonging to the sub-program I'n
Training Objective Given the data ID with size of N containing P.. Qu op', oe', where P. and Q: refer to the passage and question in the the 7th training data, likewise, op' and de' are the golden operators and operands. Our training goal is to minimize the sum of the negative log- likeleiddoood over the entire data, so the training loss is Ei - (log P(OPi)P' Q') + log POOE PP Q-)}-
4 Experimental Set-up
Datasets
We conduct evaluation experiments on two datasets: FinQA 15 and MathQA 19
FinQA a dataset created from the annual financial reports. contains 8,281 data, split into train, eval, and test parts with 6,251, 883, and 1,147 examples. We adopt the evaluation metrics from the original FinQA paper: execution accuracy (Exe Acc) and program accuracy (Prog Acc). The program accuracy calculates the accuracy of the operators and operands between the predicted program and the golden program. The execution accuracy calculates the accuracy between the golden executable result and the result from the predicted program. Since the FinQA dataset only contains operators with two operands, we extend it by creating questions required to be solved by the operators with more than two operands. We use the extended FinQA dataset to evaluate our models' adaptability to the number of operands (See Appendix A).
FinQA:
MathQA: MathQA is created from GRE/GMAT examinations, containing 37,200 math word problems. The dataset is split into 80%, 12%, and 8% of train, dev, and test data. Compared with the FinQA dataset, the examples of MathQA require more advanced reasoning ability, which challenges the model to conduct advanced numerical reasoning (sce Appendix B). A significant difference with FinQA is that the number of operands following an operator is not explicit in the MathQA dataset. Each MathQA question contains one correct of several answer options, calculated by the reasoning program with the knowledge of the operation semantics. Since we do not have this kind of knowledge, we adopted the same way NeRd 77. by only using program accuracy to evaluate models' performances. Note that program accuracy is stricter than execution accuracy because the model could find the correct answer by spurious reasoning programs.
We compare our ELASTIC model with several state-of-ther of------rtherr the-art models. (1) FinQANet [15]: It adopts the Encoder-Decoder architecture with cache updating mechanism generate the program. Since FinQANet only supports generating operators with exact two operands, we manage to train and evaluate FinQANe Net on the MathQA dataset by discarding the operators containing more than two operands. (2) NeRd 77: it uses the BERT and a
Baselines
based model to generate the symbolic nested program. (3) Graph2Tree [17: It models the dependency information of the text sequence by the GGE (38) like model, and generates the program in a tree-structured way. (4) NumNet [18): NumNet models the numeracy information by GNN network. We also train the NumNet+. which replaces the Encoder of the NumNet by RoBERTa-large Note that program
'We do not select other datasets because of: (1) too small in size (around 1000), e.g., MAWPS (37 and A 33 (2) language are not English. e.g. Math23K [10] and HMWP 13 (3) lack intermediate annotated program, like DROP (3].
https: gi haa.b..oommmmaaa hatra//laaazzinn//// http:///iinnnuuuttupuuuu..

---

accuracy does not apply to NumNet, since NumNet does not generate compositional reasoning programs. (5) Human Performance: We also report the human performance of both experts and non-experts in the FinQA dataset. The results are taken from the original FinQA paper [5
Implementation Details The model is implemented by Pytorch [39] and Transformer 40 then trained on server with an NVIDIA Tesla A100 GPU of 40G memory. Training epochs are set to 50 and 100 for FinQA and MathQA, respectively. The batch size for all datasets is set to 10. We use Adam as optimizer [4] to update the parameters of the models. The initial learning rate is set to le-5 equally, and it would be halved in every 25 epochs and 50 epochs for FinQA and MathQA. During training, the dropout rate and the weight decay are set 0.1 and 1e-5 prevent over-fitting The parameters of the RoBERTa are fine-tuned during training. For the GRU cell in the decoder, the hidden size is the same as the RoBERTa, and the GRU layers number is 4. During inference, we use greedy decoding to generate the reasoning program.
5 Results
Overall Results Table 3 shows the performances of our ELASTIC model and baselines on FinQA and MathQA. Overall, ELASTIC (RoBERTa-large achieves the highest scores on both datasets. In the FinQA dataset, see significant lead in our ELASTIC (RoBERTa- large) model compared to the best bascline FinQANet (RoBERTa-large), with 3.91 points higher exccution accuracy and 1.69 points higher program accuracy. When we change the Encoder part of ELASTIC from RoBERTa-large to GOBERTa-base it still achieves better results than FinQANet using the same size of RoBERTa, Since both ELASTIC and FinQANet use the RoBERTa as the encoder, the results demonstrate the improvements brought by separating the generation procedures for operators and operands. Both ELASTIC models outperform the NeRd with large margin. It is worth mentioning that NeRd defines external rules for different operators in their model 77. which is not the case with the ELASTIC. ELASTIC also outperforms the NumNet and NumNet+ by considerable margin. This could be due to the internal structure of these models limiting their scalability in generating reasoning programs, thus struggling to produce reasoning steps systematic manner [42). Finally, Graph2Tree achieves only 0.37 accuracy on the FinQA test dataset, which is much lower compared to its 69.96 program accuracy on the MathQA dataset. We suspect that this is because of the data leak problem existing in FAnQA train and eval data (see detailed explanation in Appendix D). Although ELASTIC surpasses the non-expert performance, we can still find a large gap between our ELASTIC model and Human Expert.
For the MathQA dataset, ELASTIC (RoBERTa-large) the best performing model, with 3.3 higher program accuracy than NeRd and 13.04 points higher than Graph2Tree. We further investigate the performance of ELASTIC using RoBERTa-base which still achieves higher accuracy of 82.27 than 79.7 of NeRd. The slight performance difference between ELASTIC (RoBERTa-large) and ELASTIC (RoBERTa-base) suggests that the extracted contextual semantic information of passage and question is sufficient. Finally, FinQANet achieves promising results on MathQA, 79. 99.20% program accuracy by FinQANet (RoBERTa-large large) and 74. 12% program accuracy by FinQANet (RoBERTa- base). Note that we discarded the data of MathQA containing more than two operands for the FinQANet, so that performance of FinQANet on MathQA is the overestimation of its numerical reasoning ability. Even with such consideration, ELASTIC outperforms FinQANet significantly, demonstrating that ELASTIC more adaptable by supporting diverse operators than FinQANet.
Performance Breakdown To demonstrate the strength of ELASTIC, we investigate the importance of the Memory Register. Also, we show ELASTIC performance when generating different lengths of program steps.
Necessity of Memory Register As discussed in the Section Memory Register ELASTIC stores the executable results of each sub-program into a special cache token #n, and updates its embedding after n-th sub-progras program is generated. The longer the reasoning program is, the higher the probability of the generating process using the previous sub- sub-program result. This section investigates the effect of the Memory Register on improving numerical reasoning performance.
First, we present an ablation study of using and not using the Memory Register for ELASTIC. From Table we find that ELASTIC with Memory Register performs slightly better than it without. Similar
7

---

Table 3: Overall Results for the baselines and ELASTIC on the testing data from three datasets. means that the scores are taken from the original papers. I means that the scores are taken from the FinQA paper [15]. * The program accuracy does not apply to the NumNet on FinQA and MathQA datasets because NumNet does not generate the intermediate reasoning program. In addition, NumNet could only solve reasoning program involving add and subtract operations. However, the proportions of examples only use add and subtract operations in MahiQA are 0.055% and 0.056%, respectively. As a result, we choose not to train NumNet on MathQA.
FinQA (test)
MathQA (test)
Datasets & Metrics
Exe Acc
Prog Acc
Prog Acc
Graph2Tree
0.37
0.0
69.96
NumNet NumNet+
2.32
n/a*
n/a*
10.29
n/a*
n/a*
NeRd
52.48t
49.90%
79.70
FinQANet (RoBERTa- base)
60.10
58,38
74.12
FinQANet (RoBERTa-large)
65.05
63.52
79.20
ELASTIC (RoBERTa-base)
62.66
59.28
82.27
ELASTIC (RoBERTa-large)
68.96
65.21
83.00
Human Expert
91.16t
87.49F
n/a
Human Non-Expert
50.68
48.
n/a
Table 4: The performances of ELASTIC with or without memory register (MR). ELASTIC with MR performs better than without MR FinQA and MathQA datasets. Both ELASTIC with without MR performs better than FinQANet. All models use the RoBERTa-large as the encoder.
FinQA (test)
MathQA (test)
Datasets & Metrics
Exe Acc
Prog Acc
Prog Acc
ELASTIC MR
68.96
65.21
83.00
ELASTIC w/o MR
68.79
64.78
82.68
FinQANet
65.06
63.52
79.20
observations can found for the MathQA dataset. This observation demonstrates the value of the Memory Register. Next, since ELASTIC and FinQANet store the executable results from the previous sub-program in different way, we conduct a comparison between the two models. The results from Table 4 show that the ELASTIC with Memory Register achieves significantly higher scores than FinQANet on both datasets.
Next, given two sub-program belonging to the same R: Ty (i < f). where the executable result of r; is used as the operand for ry. Then, we introduce the Memory Departing Distance (MDD) for and ry as j - i, and the maximum Memory Departing Distance (M-MDD) the longest MDD between all {r] of R The bigger M-MDD is, the more challenging to select the correct previous sub-program result, since the model tends to forget the information passing from long steps before. As the result, we investigate how models perform when dealing with different M-MDD.
From Figure and Figure 2b, the ELASTIC with Memory Register performs better than ELASTIC without it at each M-MDD on FinQA and MathQA datasets. Particularly in the MathQA dataset, when M-MDD is larger than 5, ELASTIC with Memory Register can achieve better results than the ELASTIC without it. This demonstrates the importance of the Memory Register when using
Forr example, in flatten program (2dd(20, 3), subtract(6, 1), add #1, 10), subtract(#O0, #2)", the MDD for ro. F1, and r2 are 3, 1, and 1. Obviously, the maximum M-MDD is 3.
8

---

100
LEASTI PROH HAL
Accmaay
66
70
LLAGEE 1il Halic Iegilim Malga
00
LLAST SocREh Pripa Notiin
20
P1 p-tam Conear
Maximun MDD
Program Steps
(a)
(b)
(c)
Figure 2: (a) Program Accuracy on FinQA according to the M-MDD. (b) Program Accuracy on MathQA according the M-MDD. (c) Program Accuracy and Operator Accuracy ELASTIC (RoBERTa-large) on different program steps in MathQA dataset, compared with Program Accuracy of FinQANet (RoBERTa-large).
executable results from long steps before. Worth mentioning that ELASTIC performs better than FinQANet on both datasets, even without the Memory Register.
Performance on Different Program Steps When producing the long numerical reasoning program, ELASTIC is less influenced by the cascading error. To demonstrate this superiority of ELASTIC, we investigate how different lengths of programs influence models' performances.
Table 5: ELASTIC and FinQANet performances FinQA dataset in terms of different program steps. The "# Train & Dev" the number of training and development data. All models use the Rarg as the encoder. means the results of that model are taken from the original FinQA paper. $ means that the FinQANet (RoBERTa-largee is re-trained by ourselves.
15
ELASTIC
FinQANeti
FinQANett
Program Steps
Train & Dev
Exc Acc
Prog Ace
Exe Acc
Prog Ace
Exe Acc
Prog Acc
76.30
75.66
70.27
68.77
73.70
71.25
4240
-2
66.01
66.01
63.69
61.79
62.34
59.65
2300
23
31.78
31.10
31.65
31.65
28.57
23.80
594
Table 5 displays the models' performances when generating programs with different steps. ELASTIC (RoBERTa-large) performs better than FinQANet (RoBERTa-large) when the program step either I or 2, indicating ELASTIC also performs well on the shorter program steps. Surprisingly, with the program step increasing from 3, the accuracy for both ELASTIC and FinQANet tumbles by half compared with the performances on program steps equal 2. We suspect that the FinQA dataset lacks sufficient training examples for the data with more than 3 program steps. Table 5 shows that the number of training data with more than program steps is 594 compared to the numbers of data available for program steps equal to (4240) or 2 (2300). For fair comparison, we retrained FinQANet GoBBRRTa-lagge on the FinQA dataset, but ELASTIC still outperforms it in execution accuracy and program accuracy.
From Figure 2c ELASTIC (RoBERTa-large) surpasses the FinQANet (RoBERTa-large) on MathQA dataset almost on every program step. Meanwhile, although MathQA challenging, ELASTIC (RoBERTa-lane large) still achieves program accuracy over 80.0 when program steps are less than or equal The model's performance drops when the program steps are equal to 9 and 10, but starts to soar when the program steps are bigger than 10. This demonstrates that ELASTIC performs well when generating longer program steps. As shown Figure 2c we plot the accuracy for the operator generation, which ignores the correctness of operands generation. We could find that the operation accuracy is always higher than the program accuracy regarding different program steps (except for program steps equal to 12). This finding demonstrates the advantage of separating the generation

---

procedure for operators and operands. This finding also reveals that the wrong predictions are because ELASTIC selects the wrong operands. We suspect this is due too much noise from the context.
Finally, our ELASTIC (RoBERTa-large) model (with approximately 500 million trainable parame- ters) outperforms Austin et al.'s smallest model [ (with 8 billion trainable parameters) but only marginally underperforms their largest model (with 137 billion trainable parameters). The parameter size of these models are 1600 and 274000 times bigger than our model respectively, resulting the training resources required significantly considerable.
6 Conclusion and Future Work
This paper presents the numEricaL reASoning with adap'Tive symbollc Compiler (ELASTIC) model aiming to solve the numerical reasoning over text problem. ELASTIC separates the generation of operators and operands, allowing the model to generate the long and complicated reasoning program. Also, ELASTIC is domain agnostic and supports diverse operators, increasing adaptabilityy In addition, we introduce the Memory Register and improve the performance of the model by using executable results from the preceding sub-pre cul-programs. We evaluated the performance of the ELASTIC model on FinQA and MathQA datasets and conducted an extensive comparison with state-of-the- models. The results show ELASTIC gained significant improvement over the state-of-the-ard baselines. Furthermore, We investigated the model's performance in terms of different M-MDD, demonstrating the necessity of the Memory Register. Finally, we compared models' performances regarding different lengths of numerical reasoning programs, showing ELASTIC is adept at producing long numerical reasoning programs. In the future, We plan to improve the accuracy of matching numbers and entities of the text. In addition, ELASTIC requires annotated reasoning programs, which labor intensive. It is worth investigating how generate reasoning programs from the trained model.
References
[1] Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages Association for Computational Linguistics, 2019.
[2] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907. 1692, 2019.
[3] Zhilin Yang. Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14 14, 2019, Vancouver, BC, Canada, pages 5754 5764, 2019.
[4] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Hersert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Lin, editors, Advances in Neural Information Processing Systems 33; Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
[5] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine I Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1114000 2020.
10

---

[6] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang, Squad: 100, 000+ questions for machine comprehension of text. In Jian Su, Xavier Carreras, and Kevin Duh, editors, Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 2383-2392. The Association for Computational Linguistics, 2016.
[7] Xinyun Chen, Chen Liang, Adams Wei Yu, Denny Zhou, Dawn Song, and Quoe V. Le. Neural symbolic reader: Scalable integration of distributed and symbolic representations for reading comprehension. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
[8] Mor Geva, Ankit Gupta, and Jonathan Berant. Injecting numerical reasoning skills into language models. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 9589558. Association for Computational Linguisties, 2020.
[9] Yan Wang, Xiaojiang Liu, and Shuming Shi. Deep neural solver for math word problems. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 845- Association for Computational Linguisties, 2017.
[10]
Yan Wang, Xiaojiang Liu, and Shuming Shi. Deep neural solver for math word problems. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen. Denmark, September 9-11, 2017, pages 845-854, Association for Computational Linguisties, 2017.
[11) Zhipeng Xie and Shichao Sun. A goal-driven tree-structured neural model for math word prob- lems. In Sarit Kraus, editor, Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages ijeai.org, 2019.
[12] Qianying Liu, Wenyv Guan, Sujian Li, and Daisuke Kawahara. Tree-structured decoding for solving math word problems. Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNI 2019, Hong Kong. China, November 3-7, 2019, pages 2370 2379. Association for Computational Linguistics, 2019.
[13] Jinghui Qin, Lihui Lin, Xiaodan Liang, Rumin Zhang, and Liang Lin. Semantically-aligne universal tree-structured solver for math word problems. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 3780-3789. Association for Computational Linguistics, 2020.
[14] Jipeng Zhang, Roy Ka-Wei Lee, Ee-Peng Lim, Wei Qin, Lei Wang, Jie Shao, and Qianru Sun. Teacher-student networks with multiple decoders for solving math word problem. In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, pages 4011-4017. ijcai.org, 2020.
[15] Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Har Hao Huang, Bryan R. Routledge, and William Yang Wang. Finqa: A dataset of numerical reasoning over financial data. In Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Viriual Event Dominican Republic, 7-11 November, 2021, pages 3697-3711. Association for Computational Linguistics, 2021.
Moens, Xuanjing
/ Punta Cana,
[16) Zhenwen Liang, Jipeng Zhang, Jie Shao, and Xiangliang Zhang. MWP-BERT: strong baseline for math word problems. CoRR, abs/2107. 2021.
11

---

[17] Shucheng Li, Lingfei Wu, Shiwei Feng, Fangli Xu, Fengyuan Xu, and Sheng Zhong. Graph- to-tree neural networks for learning structured input-output translation with applications to semantic parsing and math word problem. In Trevor Cohn, Yulan He, and Yang Liu, editors, Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACI., pages 2841-2855 Association for Computational Linguistics, 2020.
[18]
Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, and Zhiyuan Liu. Numnet: Machine reading com- prehension with numerical reasoning. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference Natural Language Processing, EMNLP-IJCNLPP 2019, Hong Kong, China, November 3-7, 2019, pages for Computational Linguisties, 2019.
Association
[19] Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik
Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation- based formalisms. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Human Language Technologies, NAACL-IILT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pages 2357-2367. Association for Computational Linguistics, 2019.
Linguistics:
[20] Edward A Feigenbaum, Julian Feldman, et al. Computers and thought. New York McGraw-Hill, 1963.
[21] Subhro Roy, Tim Vieira, and Dan Roth. Reasoning about quantities in natural language. Transactions of the Association for Computational Linguistics, 3:1-13, 2015.
[22] Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman, Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 523-533, 2014.
[23] Sowmya Sundaram and Deepak Khemani. Natural language processing for solving simple word problems. In Proceedings of the 12th International Conference Natural Language Processing, pages 3944402, 2015.
[24] Chao-Chun Liang, Kuang-Yi Yi Hsu, Chien-Tsung Huang, Chung-Min Li, Shen- Yu Miao, and Keh-Yih Su. A tag-based statistical english math word problem solver with understanding, reasoning and explanation. In IJCAI, pages 4254 4255, 2016.
[25] Yan Wang, Xiaojiang Liu, and Shuming Shi. Deep neural solver for math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 845-854, 2017.
[26] Lei Wang, Yan Wang, Deng Cai, Dongxiang Zhang, and Xiaojiang Liu. Translating a math word problem an expression tree. CoRR, abs/181 1.05632, 2018.
[27] Lei Wang, Dongxiang Zhang, Jipeng Zhang, Xing Xu, Lianli Gao, Bing Tian Dai, and Heng Tao Shen. Template-based math word problem solvers with recursive neural networks. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7144-7151, 2019.
[28] Ting-Rui Chiang and Yun-Nung Chen. Semantically-aligne equation generation for solving and reasoning math word problems. In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 2656-2668. Association for Computational Linguistics, 2019.
[29] Shucheng Li, Lingfei Wu, Shiwei Feng. Fangli Xu, Fengyuan Xu, and Sheng Zhong. Graph- to-tree neural networks for learning structured input-outpur translation with applications to semantic parsing and math word problem. In Trevor Cohn, Yulan He, and Yang Liu, editors, Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20
12

---

November 2020, volume EMNLP 2020 of Findings of ACL, pages 2841-28521 Association for Computational Linguisties, 2020.
[30] Jipeng Zhang, Lei Wang. Roy Ka-We Lee, Yi Bin, Yan Wang, Jie Shao, and Ee-Peng Lim. Graph-to-tree learning for solving math word problems. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, ACI. 2020, Online, July 5-10, 2020, pages 3928-3933 Association for Computational Linguistics, 2020.
[31] Yibin Shen and Cheqing Jin. Solving math word problems with multi-encoderss and multi- decoders. In Donia Scott, Núria Bel, and Chengqing Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020, pages 2924 2934. International Committee on Computational Linguistics, 2020.
[32] Yan Wang, Xiaojiang Liu, and Shuming Shi. Deep neural solver for math word problems. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 845-854, Association for Computational Linguisties, 2017.
[33] Shen-Yor Miao, Chao-Chun Liang, and Keh-Yih Yih Su. A diverse corpus for evaluating and developing english math word problem solvers. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 975-984. Association for Computational Linguistics, 2020.
[34] Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. pages 1368-2378 2019.
[35] Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve simple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tüss Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 2080-2094, 2094. Association for Computational Linguistics, 2021.
[36] Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, abs/1412.33555 2014.
[37] Rik Koncel-Keede Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. MAWPS: A math word problem repository. In Kevin Knight, Ani Nenkova, and Owen Rambow, editors, NAACL HIT 2016, The 2016 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 1152-1157. The Association for Computational Linguistics, 2016.
[38] William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 1024-10344 2017.
[39] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances Neural Information Processing Systems 32, pages 8024-8035 Curran Associates, Inc., 2019.
13

---

[40] Thomas Wolf. Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics.
[41] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors. 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[42] Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compo- sitional skills of sequence-to-sequenee Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmüssunn Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 2879-2888, PMLR, 2018.
recurrent networks. In Jennifer G. Dy and Andreas
[43] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoe V. Le, and Charles Sutton. Program synthesis with large language models. CoRR, abs/2108.07732, 2021.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] . You strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license the code and datasets? [Yes] See Section ??.
Did you include the license to the code and datasets? [No] The code and the data are proprictary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] We provide the URL of our code in of the paper.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section [mplementation Details
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] We think the error bars are not related to the core result of experiments.
14

---

(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] See Section Implementation Details
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Section Datasets and Section [Baselines
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you' 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "Xe9UFGW1s2q", "reviewer": "Reviewer_vH6J", "summary": "The authors present a novel model (ELASTIC) for numerical reasoning problems. The proposed model consists of an encoder and a compiler which separates the generation of operators and operands. The authors claim that the separated generators enhance the adaptability and elasticity of the model. The proposed model also incorporates a memory register to improve the model’s performance on the numerical problems with large M-MDD. The proposed model achieves state-of-the-art performance on two datasets: FinQA and MathQA.\n", "questions": "Results in Figure 2 are a bit confusing. The accuracy with M-MDD=2 is much lower than that with M-MDD=3 in figure 2 (a), why? What’s the possible reason of the large fluctuations in figure 2 (b) and 2(c)? Why the performance of ELASTIC starts to soar when the program steps are bigger than 10 in figure 2 (c)?  Does the similar phenomenon exist in the other dataset?\n", "limitations": "The experiments might be designed more carefully. When analyzing the necessity of memory register, the effect of separated generators is not disentangled with the effect of the memory register. For more effective analysis, there should be a setting ELASTIC w/o separated generator, or FinQANet w memory register if possible.\n\nTo make section 3 easier to follow, it might be better to use more textual examples instead of notations when possible.\n\nTypos: line 107-108, 144, 262\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: Numerical reasoning is an important area in machine learning and natural language processing. This work proposes novel methods to enhance the numerical reasoning ability of the model. On one hand, the generation of operators and operands are separated. On the other hand, a memory register is introduced to make it easier for the model to use the previous sub-program results. The proposed model outperforms FinQANet on both FinQA and MathQA. \n\nWeaknesses: \n1. The motivation and effectiveness of separating the generation of operands and operators is not clear. The authors claim that previous work that does not use separated generators suffers from cascading errors when encountering complicated tasks (line 36-37), but the results shown in figure 2 (c) and Table 4 may not be sufficient to support this claim. The proposed model does not consistently stabilize the performance on long reasoning programs.  \n\n2. The improvement achieved by memory register seems to be marginal (Table 4). \n\n3. The writing should be improved. The approach section is not easy to follow. The authors use notations throughout this section and some terms and notations are not well explained at their first appearance. \n\nFor example, DSL (line 108), NGS (line 136), W_{op} in eq 6, and the index i in eq 6 and 7 have different meanings. \n\nFigure 1 shows the reasoning manager generates g^{op} and g^{oe} in two different flows and the inputs are different, but the difference is not explained in line 134-143. Although the following introduction of two generators explains the difference, it might be better to move this content to line 134-143. Figure 1 and line 160-172 shows how the results from the RM updates the memory register but does not show how the vectors stored in the memory register is used during the generation.  I guess there should be another arrow from the memory register to the generator in Figure 1.  Please clarify. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "52p1szSO7gz", "reviewer": "Reviewer_3k7j", "summary": "The authors propose ELASTIC, a multi-modular network for numerical reasoning on MathQA and FinQA datasets. It consists of an encoder and a compiler with four different modules. Each module is specifically designed to solve a part of the numerical reasoning task. For example, the operator generator decides the next operation, and then the operand generator picks the operands to be used. The memory register stores some intermediate computations for future use. Overall, they achieve strong improvements over prior baselines. Additionally, this method generalizes better to higher program steps.", "questions": "Writing suggestions:\n* Please rewrite Section 3 introduction using less technically repetitive terms and more general ones/synonyms. Currently, it's a bit hard to read through.\n* Please add a figure demonstrating the terms in Table 2 clearly.\n* Similarly, I feel it would help to have a motivating figure to explain the need and functionality of the memory register, for completeness.", "limitations": "Please try to add a section specific to the limitations of the method. Other than the generalization issues, I feel these modular approaches might need more training data compared to the end-to-end frameworks. You may consider discussing some points along these lines.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is very nicely motivated, with good supporting experiments and ablations\n* The design of each component is generally well described and easy to follow through (except some parts mentioned below)\n* This direction of using specific modules for a fixed set of tasks (similar to NMN's), is indeed very interesting for this area\n\nWeakness:\n* Why is NumNet+ the only baseline from the DROP dataset leaderboard used in the paper? There are several other meaningful baselines that can be tried by just keeping the arithmetic heads at classification time. Please try adding more of these baselines, as applicable, to make Table 3 stronger.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1zIHA2raYr", "reviewer": "Reviewer_2nDD", "summary": "This paper proposes a new model called numEricaL reASoning with adapTive symbolIc Compiler (ELASTIC) to solve numerical reasoning QA problems. ELASTIC separates the generation of operators and operands, and is adaptable to the number of operands, making it more accurate and flexible. They show promising results on two datasets including FinQA and MathQA.", "questions": "1. Comparison with large language models. The authors claim that pretrained language models fall short of numerical reasoning from the text. However, recent advances in program synthesis show that large language models perform well in this area: they achieve 83.8% on MathQA which outperforms the 83.00% in this paper. I agree that this paper is valuable but for the completeness could you make more comparisons with these large language models in terms of number of parameters, runtime costs, and generalization ability, etc.? \n\n2. Some details introduced in Section 3 are unclear. For example, what does “NGS” mean? What’s the size of the memory register (size = max program steps or size = 1)? Is it possible to train another network to predict which memory slot should be overwritten when the memory is full [2]?\n\n3. Although the authors say ELASTIC is domain agnostic, it seems that the performance of these program-based models can be significantly influenced by the programming language. Can ELASTIC solve MathQA-python[3], which replaces the original MathQA DSL with python?\n\n[1] Program Synthesis with Large Language Models\n\n[2] Automatic Program Synthesis of Long Programs with a Learned Garbage Collector\n\n[3] https://github.com/google/trax/blob/master/trax/examples/MathQA_Python_generation_notebook.ipynb", "limitations": "Please see questions listed above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Numerical reasoning is an important and challenging task of AI. Although the idea of generating operators and operands separately is already been adopted in other areas like program synthesis, it is still novel and valuable in numerical reasoning to alleviate the cascading error problem. Experimental results are solid and promising, which well supports the authors’ claims.\n\nThe writing needs to be improved, especially the method part, which is somewhat confusing. It seems that the method proposed cannot be applied to QA problems without labeled programs. Also, the influence of different programming language has not been discussed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept