A Generation-based Deductive Method for Math Word Problems
Authors: Yuxuan Hu
Jing Zhang
Haoyang Li
Cuiping Li
Hong Chen
Venue: EMNLP 2023 Main (2023)
Keywords: math word problem
natural language processing

Abstract: Math word problems (MWP) involving advanced operators such as linear equation solver cannot be easily tackled by earlier MWP methods, because the existing generation methods suffer from repeated sub-expression generation and deductive methods are restricted to dealing with binary operations. This paper propose a new multivariate directed acyclic graph (mDAG) as an alternative to the generation methods' binary expression tree or the deductive methods' binary directed acyclic graph. Then to produce the topological ordering of mDAG, we propose a generation-based deductive (GeDe) model, which equips a generation model with a re-encoder to keep the deductive property but avoid the expensive enumeration of the deductive methods. GeDe performs well on math problems with many operators on the widely used benchmarks as well as solving multivariate operators on our own CMWPA benchmark. Our code is available at https://github.com/hyx1999/GeDe

A Generation-based Deductive Method for Math Word Problems
Yuxuan Hul Jing Zhang' Haoyang Li Cuiping Li Hong Chen
'School of Information, Renmin University of China, Beijing, China Key Laboratory of Data Engineering and Knowledge Engineering, MOE, China Engineering Research Center of Database and Business Intelligence, MOE, China (huyuxuan1999 thang-jing, aesiing,iaaaeee lihaoyang cs, licuiping, chong) | @ruc. edu. on
Abstract
generate the binary expression tree, where each node denotes an operator or quantity. These generation-bast based methods, however, suffer from a fatal flaw in that they require repeated generation of the same sub-expression (or sub-tree), which makes them inefficient. For example, in Figure/ (a), the sub-expression (91 - 35 x 2) + (1 2) is generated four times. Humans, the other hand, can represent repeated sub-expressions with an in- termediate quantity that can be naturally reused in the following computation process.
Math word problems (MWP) involving ad- vanced operators such linear equation solver cannot be easily tackled by earlier MWP meth- ods, because the existing generation methods suffer from repeated sub-expression generation and deductive methods are restricted to deal- ing with binary operations. This paper pro- pose new multivariate directed acyclic graph (mDAG) as an alternative to the generation methods' binary expression tree or the deduc- tive methods' binary directed acyclic graph. Then to produce the topological ordering of mDAG, we propose generation-baset deduc- tive (GeDe) model, which equips a generation model with re-encoder to keep the deduc- tive property but avoid the expensive enumer- ation of the deductive methods. GeDe per- forms well math problems with many oper- ators on the widely used benchmarks as well as solving multivariate operators on our own CMWPA benchmark. Our code is available at https://githul: ccm
Deductive approaches (Cao al., 2021: Jie al., 2022) are suggested to address the aforementioned reuse issue. Specifically, deductive methods con- vert the math expression into a binary Directed Acyclic Graph (bDAG), where each node repre- sents an operation that consists of binary operator and two input quantities. The calculation result of operation is represented by quantity. Then, these methods need to generate a topological ordering, i.e., an operation sequence. of the bDAG. By doing this, subsequent operations can casily reuse the previously generated intermedi- ate quantities. As shown in Figure I (b), quantity Q3 represents the sub-expression (94-2x35)=44-2)) which is then reused by two subsequent operations denoted by quantity 44 and 4s. When the operation sequence is inferred, these operations computed consecutively produce the final answer. Beyond the ability to reuse the intermediate quantity, de- ductive methods are more interpretable because the step-by-step generation of operations helps people understand how the reasoning works. To generate the operation at each reasoning step, existing deduc- tive methods follow an "emumerate-thens then-classify" procedure. To be more precise, they create collec- tion of candidate operations by listing every pos- sible combination of the quantities and operators, and then they use a classifier choose the opera- tion that has the highest probability. which can be viewed as a greedy search strategy.
intermediate
com/hyx/ /hyx/ 9999999999 e
1 Introduction
Solving Math Word Problems (MWPs) the task of answering natural language problems that require mathematical reasoning ability (Bobrow, 1964). To achieve such skill, researchers have proposed variety of MWP solvers, each of which seeks produce specific logic form that can be used calculate the answer the problem.
Deductive methods and generation- based meth- ods are typically the two main approaches used to solve MWPs. Inspired by advances in ma- chine translation, some generation- based methods directly adopt a sqquence-to-sequemee (seq2seq) model to generate the sequence of the math ex- pression according to the problem (Wang et al., 2017). To further capture the structure of the math expression, some sequence-to-tree (seq2tree) meth- ods (Xie and Sun, 2019) adopt a tree decoder to
*Corresponding author.

---

Question:
There are several chickens and rabbits in a eage, Inside, we observe 94 feet and 35 heads. A chicken has I head and 2 feet. A rabbit has heod and 4 foct, The
of rubbits and chickens are dewoted x y. respectively, Tell me the of x. xy:
Answer:
Muthematical Expression:
673
((94-2X35)= (4-2)) ((-4)) +(3- 1X (94-2X35) i (4-2))) + +1)X 1X (194-2X35) (4-2))) +11
Binary Expression Tree
Binary Directed Acyelic Graph
Multivariate Directed Acyelic Graph (our method)
(Amwer))mnng
(Answer)=Q;"0)
-9 +
9,%%
X4
1
"q. Xij
25 %
1
@9
C
X
X
9: 95=2x33
@
P2
Pa
Tax 9:
linear equation solver
2
L
One topological ordering of the abuve bDAG:
Two internatiate
advanced
operndds
quanities
operster
Pre-urder mnversal of the above binury expression
%
R4
One topological ordening the above mDAG:
x
(b.4h
(a) The generation method
(b) The deductive method
(c) The generation based deductive method
Figure I: Illustration of MWP example with natural language input problem and corresponding mathematical expression output that can be used to calculate the answer. The repeated sub-expression is underlined red. In order get the answer, three methods are presented: (a) a seq2seq or seq2tree generation method to generate an expression sequence binary expression tree; (b) deductive method reason out topological ordering of the bDAG; and (c) the proposed generation-bast deductive method to generate a topological ordering of the mDAG.
One obvious limitation of the aforementioned ap- proaches is that they only take into account the ba- sic binary operators such as +. - x Although binary operators are the most fundamental in math- ematics, there are some templated problems, such as solving linear equations, finding the extreme val- ues of quadratic equations, and even integrating a function, that can be solved by existing advanced operators. Thus, we can abstract advanced op- erator to tackle each templated problem. With these advanced operators, we can inject prior math- ematical knowledge to reduce the difficulty of solv- ing MWPs. However, problems requiring advanced operators are difficult to tackle using carlier MWP methods: generation-based methods inherently suf- fer from the reuse issue; deductive methods are limited by the assumption of binary operations.
Compared to generation- based techniques, GeDe has the deductive property that enables the reuse of intermediate quantities. Compared to deduc- tive methods, GeDe employs generation model generate each multivariate operation, which avoids the need to enumerate a large number of possible multivariate operations.
In order achieve this generative-based deduc- tive capacity, we equip a generation model with a re-encoding strategy that jointly encodes the prob- lem and intermediate quantities at each step of reasoning, yiclding embeddings of the intermedi- ate quantities that could be reused in the subse- quent steps. In addition, we switch from the tradi- tional greedy beam search to hierarchical beam search strategy, which is well suited to the equation generation requirement.
To address this issue, first define multi- variate Directed Acyelic Graph (mDAG) with each node involving multivariate operation that con- sists of a basic or advanced operator and multiple input quantities. Compared to basic binary opera- tors, advanced operators can receive multiple quan- tities and return multiple output quantities. For example, in Figure (c), linear equation solver requires 6 quantities (1, 1, 4, 35, 94) and returns 2 intermediate quantities (go. 91). Then, similar to the bDAG, we use the topological ordering of the mDAG to obtain a sequence of multivariate opera- tions. To generate such sequence, we propose GeDe, a Generation-bain Deductive approach.
Contributions. (1) By extending bDAG to mDAG, we can directly address complex mathematical problems using pre-defined advanced operators. (2) We propose GeDe, a generation- based deduc- tive model that keeps the deductive property while avoiding the high cost of enumeration. GeDe equips generation model with the re-encoding and hierarchical beam search strategies to achieve the objective. (3) We automatically create dataset named CMWPA for solving complicated MWPs that require both the basic binary operators and the advanced operators. has been shown that GeDe not only effectively adapts advanced opera- tors but also performs better on three existing MWP

---

datasets when more operations are involved.
et al.. 2022). This tuning-free methods have achieved promising performance, and their success mainly relies on the reasoning power of large-scale PLMs. However, the reasoning power is extremely expensive due to the large number of parameters, massive pre-training data, carefully designed pre- training objectives, and huge overhead of compu- tational resources. In contrast, we investigate fine- tuning the small models.
2 Related Work
2.1 Math Word Problem
Early efforts to solve MWPs use rule-based ap- proaches, which are only able to address limited number of MWP scenarios (Kushman al., 2014; Liguda and Pfeiffer, 2012; Roy and Roth, 2018). Deep learning models, on the other hand, are better capable of addressing a wider range MWPs. The first seq2seq model for MWPs is proposed by Wang et al. (2017). This model employs RNN to encode the problem and produce mathematical expressions. To enhance the seq2seq model, additional tech- niques have been developed, including reinforce- ment learning (Huang et al., 2018), template-ber based methods (Wang al., 2019), and group attention mechanisms (Li et al., 2019). Seq2tree, a tree structure decoder, is developed by Xie and Sun (2019). It replaces the original sequence decoder and greatly outperforms seq2seq models in terms of performance. KA-S2T (Wu et al., 2020) and MWP- BERT (Liang et al., 2022) inject commonsense knowledge and quantities' properties to improve model performance. In order encode the relation- ships between quantities in MWPs, Graph2tree (Li et al., 2020; Zhang et al., 2020) encodes the input problem using graph neural networks.
3 Problem Definition
The goal of MWP is to generate a specific logic form that can be executed to answer the problem P = {p1, Pr. Pr. which consists of n tokens and m quantity tokens Q = {q1, 22.2............... ....... Im commonsense constants, such as and c, may not explicitly appear in the problem; thus, we addition- ally add them to the quantity set Q.
Some
In this paper, we define the multivariate' directed acyclic graph (mDAG) as our target logic form, which describes the process of solving MWPs. The nodes of mDAG denote operations that consist of an operator and multiple quantities, and the edges represent the dependency between nodes. Our goal is to generate operation sequence O = (ot oe,....,oo)) which can be obtained from the topological ordering of mDAG. |O] is the number of operations. The t-th operation tokens of = (a², ab.... ..., d( with each token rep- resenting an operator or a quantity. Each operator is selected from the operator set V. which is pre- defined by the provided dataset. Each quantity is choosen from Q, which is initialized with the quantity tokens in P and can gradually grow the steps of reasoning progress. |o'] is the number of tokens of the t-th operation.
sequence of
In addition to the generation models with seq2seq, seq2tree, or graph2tree structures, other efforts use deductive methods to solve MWPs step by step rather than directly generating the entire ex- pression. Cao et al. (2021) represent the calculation process by bDAG and extract the bDAG structure by aggregating quantities and sub-expressions iter- atively. Jie et al. (2022) view the task as a complex relation extraction problem and predict the rela- tion of two quantities gradually. Compared with generation methods, deductive methods can easily employ the intermediate values to avoid repetitive generation. We expand the deductive methods to handle more complex advanced operators.
Approach
4.1 Overview
In general, the proposed GeDe method consists of two main components: the re-encoder and de- coder. The former aims to jointly encode the prob- lem and quantities, which can support the reuse of intermediate quantities. The latter is designed to generate an operation according to the output of the re-encoder. Since our target is an operation sequence, we need to perform multiple reasoning steps, with each step generating operation. We illustrate the reasoning process in Figure 2. At each
2.2 Large-scale Pre-trained Language Model
In-context few-shot learning or even zero-shot learning based on large-scale pre-trained language models, such as GPT-3 (Brown et al., 2020), PaLM (Chowdhery et al., 2022), and OPT (Zhang et al., 2022), has been thoroughly studied for mul- tiple tasks, including math word problem solv- ing (Cobbe et al., 2021; Wang et al., 2022; Wei
'The term "multivariate" means that the operator can re- ocive multiple quantities and output multiple quantities.

---

Prohlem P.
Reasoning step
Reasoning
There are several chickens and rabbire in cage. Inside, can observve qo fect and heads. A chicken has Qa head and q3 fent. A vabbit has 94 head and 95 feet...
initialize inputs pd
update inputs
pt Concan(P9,oo)
update inputs
(PO)
Mc(Pi)
R
@: 1ear 9od
Q: (q.
Note: specific quantities in the original problem are replaced by 90- As-
Mn Q*)
Mo(R R² Q³)
qo 9: -linear equation solvert S4
921
0q.
02-
9: 9o
Illustration of iteratively generating the operation sequence by the proposed GeDe, At cach reasoning step, GeDe re-encodes the input by adding new intermediate quantities and then generates new operation.
Figure 2:
reasoning step, we update the input sequence by adding new intermediate quantities generated in the previous step. The updated input sequence is fed into the re-encoder and the decoder to generate an operation. The generation process is equipped with a hierarchical beam search strategy to enable both token-level beam search within an operation and operation-leve level beam search in the whole operation sequence.
quantities [OTTc] and [QTT7] and concatenate the sequence
(QrTa//Crri///USSSTTTTTTTT--- (QTT,) (2)
with the original input P, obtain Pf We instantiate the re-encoder ME by a PLM (e.g., BERT or GPT) to represent the input se- quence and obtain the reasoning state, i.e.
R = =E(())
(3)
4.2 Re-Encoder
where R E R represents the reasoning state at step t. N denotes the length of the input quence and H denotes the hidden size.
This section delves into the re-encoder by explain- ing the input and the encoder respectively.
only interested in the semantics of the quantities rather than their precise values, first substitute each quantity in the original prob- lem P with a general special token, [QTT;]. This leaves P, devoid of any specific quantities. order to obtain the encoder's input sequence. P we con- catenate P. with all intermediate quantities, where each quantity signifies its corresponding operation.
Since
For the subsequent generation module, we ex- tract the representation of each quantity from R according to their positions in :
Q' = (R'(i ( I,).
(4)
where Q' E RMXH M denotes the number of quantities, Ia saves the indexes of all the quantities in P and R']:] denotes the i-th row of R In summary, the original input is re-encoded with the previously generated intermediate quantities at each reasoning step to update the reasoning state and record all intermediate quantities, which may be reused in the subsequent generation process.
We take the example in Figure explain the input. The given math problem contains six quan- tities, which are replaced by [QTTo] to [QTT5]. At reasoning step f, we have already generated the following operation:
4.3 Decoder
QTTc QTTT
QTTT QTT:
@ (1)
LES]
We adopt a Gated Recurrent Unit (GRU) network (Chung et al., 2014) combined with the attention mechanism (Vaswani et al., 2017) 7) as the decoder Mp- Following the majority of the earlier works (Liang al., 2022: Tan et al., 2021; Xic and Sun, 2019), we choose GRU instead of transformer for fair comparison. Although some works choose pre- trained transformer (Shen et al., 2021), their per- formance might not be improved due to the larger parameters but limited labeled data.
[ES]]QTT]]] (TTT. CeTTTTT[TTTTTTT [|TTTT (TTTIJQTT[|[QTTT])
where LES stands for multivariant operator of lin- car equation solver given the operands of matrix made up of [QTT2]. [QTTs]. [QTTa], [QTTs] and vector made up of [QTTp] ] and [QTT,]. In practice, the operation is represented by a sequence that expands the matrix and vector by row. Then we denote the outputs of this operation by two new

---

Operation Generation. The decoder aims pro- vide an operation of (aj ab. .... ) at each reasoning step t. To enable the auto-regressive generation, we insert a special beginning token ([BOS]) before the first token of and add a special ending token ( ((OSS) or [E0O]) after the last to- ken d to re-create o' = (ah, ai, a², .... / + While [ECS] only signifies the termination of the current operation, [EOO] stands for the final to- ken of the complete operation sequence. The hid- den state hf of each token af can be obtained by h = GRU(h a²) where h E R rep- resents the hidden state of the previous step, h is initialized from the hidden state of the [CLS] token produced by the encoder, and at E R1x is the representation of the token az. Next, us- ing h as the query to attend to current reasoning state Rt, obtain the attention-enhancee state A MH, R'), where MHA denotes multi- head attention (Vaswani et al., 2017). Finally, we determine the likelihood of the output token by determining how well A resembles the represen- tation of quantities and operators, i.e.,
refined version of greedy search (Tillmann and Ney, 2003). However, using beam search in the deductive methods is difficult because the search space of the operation sequence is nested. In other words, we need to generate each operation based on tokens and generate the entire operation sequence based on operations. Therefore, previous deduc- tive methods (Cao et al., 2021; Jie et al., 2022) only adopt the greedy search and leave the imple- mentation of the beam search as further work. To address this challenge, we propose a hierarchical beam search strategy. Compared with the tradi- tional beam search, the hierarchical beam search can control the generation process at two levels. Specifically, the hierarchical beam search con- sists an inner beam search and an outer beam search. The former is standard beam search which seeks series of tokens form a candidate oper- ation. The latter is designed search a complete operation sequence. The beam score of the inner beam search purely relies on the probabilities of tokens predicted by the decoder. Suppose the t-th step generates / tokens, the inner beam score ibs' is calculated as:
p(affost d'e,.P = softmax(AA((V)) Q())²), (5)
- Σ log p(a)),
ibs log II p(a)))
(7)
where oe represents 0², 0² o²,...,o'-I 0'_ before reason- ing step t, d'z, represents a/. ai. -م_ before the i-th token of step L. is the matrix concatena- tion operator, V E RIV|*H and Q' E RM de- note the representations of operators and 1-th step's quantities respectively. When obtaining new op- eration o², we can determine the number of new quantities by the operator in o' and record these new intermediate quantities for the subsequent rea- soning steps. When [EOS] has the highest proba- bility, the decoding process of the current operation ends but a new operation generation starts instead. When [EOO] has the highest probability, the entire decoding process is complete.
where p(a{) is computed by Eq (5). We use the inner beam scores of generated operations to ap- proximate the distribution of operations to support the outer beam search. The probability of the t-th operation of can be calculated as the softmax score of its inner beam score, i.e.,
p(o') softmax(cxp(ihe)))
(8)
Suppose the entire operation sequence contains T operations, the outer beam score is computed as:
obs log pl t=1
T M logp(ot)
(9)
Training Objective. Given problem P and its ground truth operation sequence O, we maximize the probability of generating O by P. iee.
Algorithm presents the hierarchical beam search algorithm. Each outer beam is denoted by the symbol beam, which keeps track of both the current operation sequence and the beam score. The empty operation sequence and score of zero are used to construct the initial outer beam ini- tially (line 1). Then, we iteratively expand outer beams until they are all finished, i.e., all the outer beams are terminated with [EOO] (line 4-).). For
10| 1o'I+1
p P)
II
II
p(aflo 1 P).
(6)
1=1
1=1
4.4 Hierarchical Beam Search
To enhance the generation quality during inference, beam search is used in many generation tasks as

---

Algorithm 1 Hierarchical Beam Search
widely-adopted MWP datasets show the effec- tiveness of our model on binary operations.
Input: Math World Problem P, Beam size
Output: beams with Top- operation sequences
1: beams é- IruitialBeant
2: while not all beams are over do
5.1 Experimental Setup
3: booms, f- -]:
4: for beam in beams do
S: if beam is over then
Datasets. We consider four MWP datasets in- cluding our created CMWPA and three widely- used existing MWP datasets: MAWPS (Koncel- Kedziorski et al., 2016), Math23k (Wang al., 2017), MathQA (Amini et al., 2019), and SVAMP (Patel al., 2021). We use CMWPA to verify the validity of multivariate operations. Following (Tan et al., 2021), I), we perform pre-processing to filter out unsolvable problems. In all the datasets, we take into account the basic binary operators addition (+), subtraction (-), multiplication (x), division (+), and exponentiation (*). For advanced opera- tors used in the CMWPA dataset, we consider the linear equation solver, the quadratic function ex- tremum solver, and the quadratic function integral solver. Appendix A.2 presents the statisties for each dataset.
6:
beams, (pamms..ppeed
7:
else
8:
ups < InuerBeanSearch((P beam, K);
9:
for op in ops do beamtes <- (xtend(beem,, op):
10:
11:
12:
end for end if
13:
14:
end for
15:
beams +- Get K);
16:
end while
each extensible outer beam, we search candidate operations ops using the inner beam search (line 8). The inner and the outer beam search share the same beam size K. Next, extend outer beams with these candidate operations (line 9-12). At the end each step, we only maintain the top-K outer beams according to their scores computed by Eq. (9) (line 15). Finally, beams save the top-K operation sequences. We discuss the complexity of GeDe in Appendix A.1
Evaluation Metric. Following previous work (Jie et al., 2022), we compare the predicted and the gold answer to calculate the accuracy as the evaluation metric. We parse out the operator and operands from the model predicted expression sequence and then use the corresponding operator executor to calculate the answers. We explain the details of the parsing and execution in Appendix A.3.
4.5 Decoding Constraint
Logic forms need to obey clear grammatical rules. In order to guarantee the validity of the output, we provide two constraint strategies, one during and one after the decoding process. Inspired by PICARD (Scholak et al., 2021), an incremental grammar checker proposed for Text-to-SQL task, the constraint strategy during the decoding process is to filter out illegal beams at each decoding step in the inner beam search to prevent potential syn- tax errors in the generated operation. For example, when detect that the current token generation step needs generate an operator, we will reject all non-operators. Following (Jic et al., 2022), the after decoding constraint strategy eliminates candi- date operations that are improbable to exist in real- world mathematical problems, such as [[QTT,] [OTT;]" and " "[OTT,] [orT,)"
Implementation Details. We adopt RoBERTa- base (Liu et al., 2019) as our re-encoder for En- glish datasets, and Chiines-RoBBRRTa-baasee (Cui et al., 2020) for Chinese datasets. The purpose of using the Roberta model is to make more fair comparison with previous work. We can also use unidirectional attention models (e.g., GPT). We use AdmaW to optimize the loss function with a learning rate of 2e-5, weight decay of 1e-2, and a batch size 8. During inference, the beam size K set to by default. For CMWPA, Math23K, MathQA, and SVAMP we report accuracy on their test set. For MAWPS and Math23k, we follow pre- vious works and also report 5-fold cross-validation performance. We conduct all experiments with RTX 3090 (24G) GPU.
5 Experiments
this section, we establish a dataset for multivari- ant advanced operators and show that the proposed GeDe capable of doing these types of operations successfully. We also conduct experiments on four
hhtps//hhugginggicecccooooottttt hhtp::/wwngiiiiaccccoo////eeeeer

---

Model
MAWPS S-fold
Math23k Test Set
Math22k 5-fold
MathQA Test Set
SVAMP Test Set
GroupAtto (L al. 2019)
76.1
69.5
66.9
21.3
MBERT+LSTM (Tan et al,, 2021)
75.1
77.1
$28
RoBERIaaienn (Lan et al., 2022)
88.4
76.9
76.6
30.3
Generate&Rank (Shen al., 2021)
84.0
85.4
84.3
GTS (Xie Sun, 2019)
82.6
75.6
74.3
41.0
Graph2Tree (Zhang et al., 2020)
77.4
75.5
69.5
HMS in 2021)
76.1
S2T/G2T
MultiE&D (Shen and 2020)
78.4
76.9
BERT-CL al., 2022)
82.4
73.8
MWP-RoBEKIa (Liang et al., 2022)
84.5
82.0
76,6
RoBERTa-DR (Jie al., 2022)
92.0
85.1
83.0
78.6
47.3
GeDe
92.3
85.4
84.2
81.5
45.7
Table I: Accuracy on three existing MWP datasets (%).
5.2 Experiment on CMWPA
Logic Form
Accuracy
Inference Time
The existing MWP datasets only use basic binary operators as target logic form. Rewriting these logic forms to support advanced operators ex- pensive, Therefore, based on handcraft templates, we create a synthetic dataset named CMWPA (Complex Math Word Problems with Advanced operators).
BET mDAG
32.0
600 ms/per sample
95.0
400 ms/per sample
Table 2: Accuracy (%) and time cost CMWPA of GeDe with different annotation (BET: Binary Ex- pression Tree, mDAG: Multivariate Directed Acyclic Graph).
To create the CMWPA dataset, we first define needed operators which include five binary oper- ators (addition (+), subtraction (-), multiplica- tion (x) ), division (,), and exponentiation (")) as well as three advanced operators, which can be used to solve linear equations (the [linear equa- tion solver] operator), find the maximum value of quadratic functions (the [quadratic function tremum solver] operator), and find the definite inte- grals of quadratic functions (the [quadratic function integral solver] operator). For each operator, we write one or more templates to generate a text de- scription and its operation. We only consider the quadratic function because the operations related to the quadratic function can be transformed to series of binary operations for training the baseline model. The templates of CMWPA is described in Appendix A.4. In this dataset, for each problem, we provide two types of logic forms: multivariate operation sequence and binary operation sequence. An example is given in Appendix Table
annotation, GeDe achieves 95.0% accuracy, which indicates that our proposed method can effectively support advanced operators to generate the multi- variate operation sequence. However, when using the binary expression tree as the generation target, GeDe only achieves 32.0% accuracy. Because the average number of advanced operators in multi- variate operation sequences is 2.98, which is sig- nificantly less than the average number of binary operators (i.e., 35.03) in binary expressions tree, using advanced operators solve MWPs can es- sentially reduce the learning difficulty and lead improved both the accuracy and efficiency.
5.3 Experiment on Existing MWP Datasets
Baselines. The baselines can be broadly catego- rized into four groups, seguence-to-scquenneeo requeece-to-tree(S22)
graph-to-trreeGG2T)) and dediective-reasoninn((RR where the first three of these are all generation-basee methods but are in- stantiated with different encoders or decoders. We select basclines having reported the performances on at least one of the three datasets.
We conduct experiments CMWPA to demon- strate that using advanced operators to solve com- plex MWPs more effective than only using basic binary operators. Concretely, our proposed GeDe is applied to generate multivariate operation se- quences. Then for fair comparison, we adopt GeDe to generate binary operation sequence.
Experiment Results. We start by running tests MAWPS and Math23k. As shown in Table 1, our model achieves promising performance on both the datasets compared to previous state-of-the-art (SOTA) methods. Given that MAWPS only has an average of 1.41 binary operations, the proposed
Experiment Results. Table 2 shows the accuracy and inference time on CMWPA, using mDAG as

---

GeDe only slightly improves 0.3% accuracy on MAWPS compared earlier baselines. This is not enough to demonstrate the benefits of the pro- posed model. On Math23k, GeDe performs equally well as the earlier SOTA method Generate Rank. However, Generate&Rank fine-tunes a mBART- large (Liu et al., 2020) model with 610M parame- ters. In contrast, GeDe only involves 126M param- eters and thus reveals better parameter-efficiennyy We further evaluate our method on MathQA, the most challenging MWP dataset with an average of 4.25 binary operations, and show results Table 1. Our model greatly beats all baselines (+2.9%), which demonstrates the model's efficacy in hand- ing complex MWPs. In summary, on three existing MWP datasets, the performances of GeDe are on par better than those of the closest competitors. SVAMP is also a challenging dataset that manu-
Operations
RoBERTa-DR
GeDe
77.4
78.0
83.5
81.8
3
83.4
85.1
81.7
84.0
71.4
77.5
Overall
78.6
81.5
Table 3: Fine-grained accuracy on MathQA (%).
Model variant
Accuracy
GeDe
81.5
w/o dynamic quantity embeddings
80.3
wwo re-encoder
75.8
w/o hierarchical beam scarch
81.0
Table Ablation study on MathQA (%).
ally created to evaluate a model's robustness. On this dataset, GeDe achieves an accuracy of 15.7%. which can outperform the vast majority baselines except the DR model.
in Section 4.2 can update both new quantities and reasoning state each reasoning step. We investi- gate the two functions respectively.
Instead of using dynamic quantity embeddings, develop variant model with static quantity em- beddings. In other words, instead of having distinct embeddings updated based on various contexts in various math problems, [QTT; in various math problems is assigned a unified embedding that is updated globally. Note we keep re-encoding the original problem with the newly produced quan- tities at each step f, but only the updated reason- ing state R is leveraged. The comparison results in Table 4 show that without the dynamic quan- tity embeddings, the performance drops 1.2% MathQA's test set. Since different MWPs' quanti- ties reflect different semantics, it preferable for them to be dynamically updated with their contexts.
In addition, we conduct experiments based on Roberta-latee large the Math23k dataset. The model achieves an accuracy of 86.7% on the Math23K test set. Using Roberta-large improves the accuracy by 1.3% over using Roberta-base. This shows that using a larger PLM improves the performance of our method and outperforms the baseline Generate & Rank model on the Math23K test set.
To further highlight the advantages of the pro- posed GeDe. following (Jie al., 2022), pro- vide fine- mne-grained analysis on MathQA based on more specific,
various numbers of operations. we compare our model with the most powerful baseline RoBERTa-DR (Jie et al., 2022) and dis- play the analysis results in Table 3. We observe that GeDe performs better samples with 3, and 4 operations, particularly samples with at least 5 operations. This comparison indicates model is more robust to problems requiring more reasoning steps, because the designed re-encoder can capture adequate interactions between the newly produced quantities and the original problem.
Then completely remove the re-encoder and only allow the encoder encode the original prob- lem. Instead, we directly use the hidden state in the decoder's GRU network to represent the reasoning state. Table shows that without the re-encoder, the performance drops 5.7%. In this variant model, although the quantities are dynamically updated according to various problems, the interactions be- tween the quantities and the input problem not fully exploited as the re-encoder does.
5.4 Ablation Study
In this section, we take a thorough ablation study on MathQA dataset to verify the effectiveness of the re-encode and the hierarchical beam search strategies in the proposed GeDe.
Effect of Hierarchical Beam Search. Previous de- ductive methods (Cao al., 2021; Jie al., 2022) generate the operation sequence based on hierarchi- cal greedy search, and regard the implementation of beam search as a future challenge. We imple-
Effect of Re-encoder. The proposed re-encoder

---

ment hierarchical beam search in our GeDe to im- prove greedy search. We compare them, where the beam size set to to create a greedy search. As shown in Table 4, when the hierarchical beam search is disabled (beam size = 4) and replaced with the hierarchical greedy scarch (beam size 1), the performance drops 0.5%. By observing the inner and outer beam scores in the generation process, for most of the samples, find that the score of the first beam significantly greater than that of the remaining beams, resulting in a rela- tively small gap between greedy and beam search. This problem, also referred to as neural networks' "over-confidence" has been studied by some works (Miao et al., 2021; Wang et al., 2021). Such im- provement is left in the further.
leading different influences. To provide an eth- ical analysis of this work and others on the same line, we will address three aspects: the possible positive or negative effects of our work, the impact of harmful information datasets, and the equality and differences between different languages.
First, the point of studying MWP is explore the mathematical reasoning capabilities of artifi- cial intelligence (Wei et al., 2022). However, the developed models may still be applied to harmful aspects, such as cheating in math exams.
On the other hand, the presence of harmful in- formation in the training data may lead the model to learn some implicit biases (Liang et al., 2021; Steed al., 2022). In our experiments, for the three existing datasets, we exactly follow the ex- perimental setup of previous works to pre-procest, and remove the potential harmful information. For our manually created dataset CMWPA, our tem- plates also do not contain any harmful information. However, in the inference phase, model cannot reject answers when the user provides malicious in- put. Therefore, we need employ extra efforts avoid this issue when the model is deployed online.
6 Conclusion and Future Work
This paper proposes multivariant direct acyclic graph (mDAG) to describe math expression in order handle the advanced multivariant opera- tors. Then to generate the topological ordering of mDAG, we propose generation model equipped with a re-encode strategy to keep the deductive property but avoid the expensive enumeration of existing deductive methods. A hierarchical beam search algorithm is implemented to enable the in- ner token and outer operation searches. Extensive experiments three standard datasets and one automatically created dataset demonstrate the pro- posed model's advantage in solving math problems with binary operators and advanced operators.
Finally, we use both English and Chinese datasets in our experiments to respect linguistic equality and better take into account language dif- ferences. The experimental results validate the robustness of our model across languages. Nev- ertheless, English and Chinese are the two most popular languages, and we should make greater efforts to concentrate on and preserve the devel- opment of minor languages in the field of natural language processing (Zhang et al., 2021).
7 Limitations
From the time complexity analysis in Ap- pendix A.I, we can see that our model will face the efficiency issue when needs to generate a long operation sequence. At the same time, the re- encode module needs to concatenate the problem description with generated operations, which may reach the input length limit of PLM. Therefore, our future work will study how compress the input sequence during the generation process address above issues.
Acknowledgments
This work is supported by National Natural Sci- Foundation of China (62322214 62072460, 621 62172424,,2276277) Beijing Natural Science Foundation (4212022); the Public Computing Cloud at Renmin University of China.
References
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Yejin Choi, and Hannanch Ha- jishirzi. 2019. MathQA: Towards interpretable math word problem solving with operation- based for- malisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages
8 Ethics Statement
For many years, public opinion has debated the pros and cons associated with artificial intelligence technology. One consensus is that advances in technology may be used in a variety of scenarios,

---

2357-2367, Minneapolis, Minnesota. Association for Computational Linguistics.
5944-5955, Dublin, Ireland. Association for Compu- tational Linguistics.
Daniel G. Bobrow. 1964. Natural language input for a computer problem solving system. Technical report, USA.
Rik Koncel- Kenzee-kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannanch Hajishirzi. 2016. MAWPS: A math word problem repository. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 1152-1157, San Diego, California. Association for Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Cerbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. 2014. Learning to automatically solve algebra word problems. In Proceedings of the 52nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 271-281, Baltimore, Maryland. Association for Com- putational Linguistics.
Yihuai Lan, Lei Wang. Qiyuan Zhang, Yunshi Lan, Bing Tian Dai, Yan Wang, Dongxiang Zhang, and Ee-Peng Lim. 2022. Mwptoolkit: An open-soure framework for deep saaming-baseed math word prob- lem solvers. Proceedings of the AAAI Conference on Artificial Intelligence, 36(1 (6(11):133 11:11113188113333 18-1831900
Yixuan Cao, Feng Hong, Hongwei Li, and Ping Luo. 2021. A bottom-up dag structure extraction model for math word problems. In AAAI Conference on Artificial Intelligence.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. urXiv preprint arXiv:2204.02311.
Jierui Li, Lei Wang, Jipeng Zhang, Yan Wang, Bing Tian Dai, and Dongxiang Zhang. 2019. Modeling intra- relation in math word problems with different func- tional multi-head attentions. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics. pages 6162 6167, Florence, Italy. Association for Computational Linguistics.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho,
and Yoshua Bengio. 2014, Empirical evaluation of gated recurrent neural networks on sequence model- ing, arxiv preprint arXiv:1412.3555
Shucheng Li, Lingfei Wu, Shiwei Feng, Fangli Xu, Fengyuan Xu, and Sheng Zhong, 2020. Graph-to- tree neural networks for learning structured input- output translation with applications to semantic pars- ing and math word problem. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2020. pages 2841 2851-2.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word prob- lems.
Zhongli Li, Wenxuan Zhang, Chao Yan, Qingyu Zhou, Chao Li, Hongzhi Liu, and Yunbo Cao. 2022. Seek- ing patterns, not just memorizing procedures: Con- trastive learning for solving math word problems. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2486-2496, Dublin, Ireland. Association for Computational Linguistics.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. Revisiting pre-trained models for Chinese natural language processing. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing: Findings. pages 657-666 668, Online. Association for Computa- tional Linguisties.
Paul Pu Liang. Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2021. Towards understanding and mitigating social biases in language models. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 202 I, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 6565-6576. PMLR.
Danqing Huang, Jing Liu, Chin-Yer Yew Lin, and Jian Yin. 2018. Neural math word problem solver with re- inforcement learning. In Proceedings of the 27th International Conference on Computational Linguis- tics, pages 213 223, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Zhenwen Liang, Jipeng Zhang, Lei Wang, Wei Qin, Yunshi Lan, Jie Shao, and Xiangliang Zhang 2022. MWP-BERT
Zhanming Jie. Jierui Li, and Wei Lu. 2022. Learning to reason deductively: Math word problem solving complex relation extraction. In Proceedings of the 60th Annual Meeting of the Association for Compu- lational Linguistics (Volume I: Long Papers), pages
pre-training for math word problem solving. In Findings of the Asso- ciation for Computational Linguistics: NAACL 2022, pages 997-1009, Seattle, United States. Association for Computational Linguistics.

---

Christian Liguda and Thies Pfeiffer. 2012. Modeling math word problems with augmented semantic net- works. In Natural Language Processing and Infor- mation Systems, pages 247-252, Berlin, Heidelberg. Springer Berlin Heidelberg.
Yibin Shen and Cheqing Jin. 2020. Solving math word problems with colti-enccoder and multi-decoders. In Proceedings of the 28th International Conference on Computaaionnll Linguistics. pages 2924-2934, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Xin Lin, Zhenya Huang, Hongke Zhao, Enhong Chen, Qi Liu, Hao Wang, and Shijin Wang. 2021. Hms: un-
Ryan Steed, Swetasudha Panda, Ari Kobren, and Michael L. Wick. 2022. Upstream mitigation is not all you need: Testing the bias transfer hypothesis in pre- pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Compu- national Linguistics (Volume 1: Long Papers), ACI 2022, Dublin, Ireland, May 22-27, 2022. pages 3524- 3542. Association for Computational Linguistics.
hierarchical solver with dependency-enhance derstanding for math word problem. Proceedings of the AAAI Conference Artificial Intelligence, 35(5):423.-4240.
Yinhan Liu, Jiatao Gu. Naman Goyal, Xian Li. Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke 7ettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Trans. Assoc. Comput. Linguistics, 8:726-742.
Minghuan Tan, Lei Wang. Lingxiao Jiang, and Jing Jiang. 2021. Investigating math word problems us- ing pretrained multilingual language models. arXiv preprint arXiv:2 11Xv:2205.08922
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. Cite arxiv: 1901..
Christoph Tillmann and Hermann Ney. 2003. Word reordering and dynamic programming beam search algorithm for statistical machine translation. Compu- tational Linguistics, 29(1)997-133
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need, Advances in neural information processing systems, 30.
Mengqi Miao, Fandong Meng, Yijin Liu, Xiao-Hua Zhou, and Jie Zhou. 2021. Prevent the language model from being overconfident in neural machine translation. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 1th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 3456-3468, Online, Association for Computa- tional Linguistics.
Deng-Bay Bao Wang. Lei Feng, and Min-Ling Zhang. 2021. Rethinking calibration of deep neural networks: Do not be afraid of overconfidence. In Advances in Neu- ral Information Processing Systems 34: Annual Con- ference on Neural Information Processing Systems 1021, NeurlPS 2021, December 6-14, 2021, virtual, pages 1809-1 1820.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 2080-2094.
Lei Wang. Dongxiang Zhang, Jipeng Zhang, Xing Xu. Lianli Gao, Bing Tian Dai, and Heng Tao Shen. 2019. Template-bast math word problem solvers with re- cursive neural networks. In Proceedings of the Thirty- Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial In- telligence Conference and Nimth AAAI Symposium Educational Advances in Artificial Intelligence, AAAI' 19/IAAI 19/EAAI 19, AAAI Press
Subhro Roy and Dan Roth. 2018 Mapping to declara- tive knowledge for word problem solving. Transac- tions of the Association for Computational Linguis- tics, 6:159-177 172.
Torsten Scholak, Nathan Schucher, and Dzmitry Bah- danau. 2021. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. Proceedings of the 2027 Conference on Empirical Methods in Natural Language Processing, pages 9895-9901, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022. Self- cofsistesit im- proves chain of thought reasoning in language mod- cls. arXiv preprint arxiv:2203.1117.
Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. In Pro- ceedings of the 2017 Conference on Empirical Meth- ods in Natural Language Processing, pages 845 854, Copenhagen, Denmark. Association for Computa- tional Linguistics.
Jianhao Shen, Yichun Yin. Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu. 2021. Generate & rank: A multi- task framework for math word prob- lems. In Findings of the Association for Computa- tional Linguistics: EMNLP 2021, pages 2269-2279, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models.

---

Qinzhuo Wu, Qi Zhang, Jinlan Fu, and Xuanjing Huang. 2020. A knowledge-awan sequence-to-tee toetre network for math word problem solving. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Processing (EMNLP), pages 7137- 7146, Online. Association for Computational Lin- guistics.
A.2 Datasets Statistics
The statistics of datasets are presented in Table 6. CMWPA is synthetic English dataset with 1000 training samples and 100 test and validation sam- ples. MAWPS and MathQA are public English MWP datasets that contain 1.9K math problems and 20K math problems, respectively. Math23K is a public Chinese MWP dataset that contains 23K math problems. We use the average number of oper- ations to assess the difficulty of MWP dataset. As we can see, MAWPS is the simplest dataset because almost all problems require only ations. MathQA is the most challenging dataset, requiring more operations and, hence, more steps in the reasoning process to obtain the answer. SVAMP is also a challenging dataset that is manually cre- ated to evaluate a model's robustness. They apply variations to the instances sampled from MAWPS, Such variations could include adding extra quanti- ties, swapping the positions between noun phrases, etc.
Zhipeng Xie and Shichao Sun. 2019. A goal-driven tree-structured neural model for math word problems. In IJCAI, pages 5299-5305.
Jipeng Zhang. Lei Wang, Roy Ka-Wei Lee, Yi Bin, Yan Wang. Jie Shao, and Be-Peng Lim. 2020. Graph-to- tree learning for solving math word problems. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 3928- 3937, Online. Association for Computational Lin- guistics.
two oper-
Shiyue Zhang, Benjamin Frey, and Mohit Bansal. 2021. Chrentranslate: Cherokee- englishh machine transla- tion demo with quality estimation and corrective feed- back. In Proceedings of the Joint Conference of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL 2021- System Demonstrations, Online, August 1-6, 2021, pages 272-279. Association for Computational Linguistics.
A.3 Parsing and Execution
Due to the existence of higher-order operators, the way we calculate the answer different from pre- vious works. We implement the corresponding solving function using Python for each pre-defined operator, which is also included in our published code. During inference, for the generated operation sequence, we sequentially calculate the returned quantities for cach operation. Naturally, the turned quantities of the last operation denote the answer to the problem. For generated operation, we first parse out its operator and several operands. Then, we call the solving function corresponding to the operator to obtain the returned quantities.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprini arXiv:2005.01068
A Appendix
A.1 Complexity Analysis
Consider a math problem that has n words in the problem description and |O] operations in the solv- ing process. A total of k words are needed to describe the |O] operations, i.e, Æ 1! Jo'). GeDe needs to perform |O| operation re-encode steps and k token generation steps. For the 7-th re-encode step, its computational complexity is O((n + ΣT (o'1)2). For generating the tokens in oT, its computational complexity is O(1o") (n EE= lo')). Therefore, the overall time complexity is 1 EE= |07|
A.4 CMWPA Templates
We show the templates corresponding to the ad- vanced operators as follows.
Two templates for the [linear equation solver] operator:
Text description: [4o] * [oo] [qı] * [or] = [qu]: [qa] [ou] + Iqal * [o,]=[qal.
(n + E= < 0(10) n² + * (n + x)). If we use the unidircctional attention model as re-encoder, the complexity can be lowered to 0(n2+ko (n)) which is the same as what the current seq2seq generation methods achieve. The additional time complexity is acceptable because O is typically not very large.
Operation: [linarr equation solver] Iqal [qu] [qx] [qs]
[q2] Iqal
Text description: Determine [ou]. [oy] as the result of inverse of matrix [I [qu)_[91) J. [qu)] [qa] ]] times
vector [ [qu]. [qs]].

---

Operation: [linear equation solver] [qa] [qu] [qa] [qa] [q4] [9s]
One template for the [quadratic function integral solver] operator:
Text description: Determine [oo] the definite inte- gral of quadratic function [qo] x"2 [q:] x [qa]
between the intervals [qs] and [q.].
Operation: Iquadratic function integral solver] [qs] [q4] [qol 192l
One template for the [quadratic function ex- tremum solver] operator:
Text description: Determine [oo] as the the extremum
value of quadratic function [qo] x'2 [qi x [qa]-
Operation: Iquadratic function extremum solver] [qo] [g2l
Based on the templates, we generate a sample follows. First, we randomly initialize candi- date set of quantities. Then, we randomly seleet a template and fill in slots by randomly selecting quantities from the quantity candidate set. We re- input the returned quantities of the operation into the candidate set and repeat the above process sev- eral times. In this way, problem description and its operation sequence are generated. We also con- vert the operation sequence into pre-order binary expression as another type of annotation for train- ing the seq2seq baseline.
A.5
CMWPA Example
We provide a sample of CMWPA in Table 5. This sample is initialized with 6 quantities and involves four types of operators: the subtraction operator, the [linear equation solver] operator, the lquadratic function integral solver] operator, and the Iquadratic function extremum solver] opera- tor. Two types of annotations are provided: the multivariant operation sequence and the pre-order binary expression (pre-order binary expression can be transformed into a binary operation sequence (bDAG) or a binary expression tree). For each op- cration in the multivariant operation sequence, provide the operation, the input quantities, and the returned output quantities.

---

Problem:
Given I 0 = Iqal=
=qs1= = 0.266
Determine [qu] the
[q4] minus [qa] Determine [4=] [qs] as the result of inverse of matrix [q4] [qx] ) [ga]. [qp] times
vector [qu] [qo] Determine [qa] the definite integral of quadrutic function [qa] =x" [qr] [qs]
between the intervals [qı] and [qs] Determine [qio] as the the extremum value of quadratic function [qs] x² [qo] [qa] Output the value [q:0]-
Multivariant Operation Sequence:
1. operation; [QTTa). [QTT:]]
returned quantities operation:: [[QTTall
operationg: [llinear equation solver). [QTTs). (0TT
returedd quantities of operationy: [[QTT+], [QTT,]]
operationg: I[quadratic function integral solver] (
returned quantities of operationg: [[QTTa]]
operation.: [[quadratic function extremum solver]. (QTT)))(QTTa).0QTTT))
returned quantities of operations: [[QTTwo]]
Pre-order binary expression:
+ 1.../.....00. [[QTT]) - 1
[QTTal,",* //,,,./.,1__--
(OTT).....IQITT. (
(.TTTT
(0TT))
[ [OTTs)][c]]
(0TT (1TTT) ".[QT,
2. [QTTa).-. (.TTTTTTT (-. (QTT)) [QTT2)][QTTT]] (c1). [QTT.) [el],
[QTTal.+, +,1.,,,, 1
( (62)) ..,, IQT=L [QTTal.
1QT::1.
Table sample of CMWPA. (QTT, represents the i-th quantity, [cl], [c2], and [c3] represent three constants 1, 2, and 3 respectively.
Dataset
#Train
#Dev
#Test
Avg. #Operations
Avg. PDL
Operation Types
Language
CMWPA
1.000
329.55
Basic Advanced
English
MAWPS
1,589
199
299.31
Basic
English
Math23k
21.162
1,000
1,000
2.27
156.28
Basic
Chinese
MathQA
16,191
2,411
1,605
374.89
Basic
English
SVAMP
3.138
1,000
159.6
Basic
English
Table 6: Detailed statistics of all datasets. PDL. means problem description length.


Reviews:
[{"review_id": "YbihCJ1vzL", "reviewer": "Reviewer_vqVh", "summary": "", "questions": "Question A. You've cited Vaswani's work about multi-head attention in line 300, but the equation 5 seems like a single-head attention. Which thing did you use? If you used single-head attention, then why?\n\nQuestion B. How did you construct CWMPA dataset? What is the base data to synthesize the dataset? I know that the authors already stated it on the appendix, but I think this should be appear at the main paper.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Question A:\nIn fact, we used multi-head attention but with **num_attention_heads**=1  in our experiments. This hyperparameter was chosen through validation set analysis, revealing that the model's optimal performance is achieved when **num_attention_heads**=1.  we will revise our paper to ensure consistency.\n\nQuestion B:\nWe appreciate your suggestions, and we will transfer the primary description of the CWMPA dataset to the main paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Succssful extension of previous deduction approaches to N-ary operators.\n- Devise a hierarchical beam search procedure for the decoding procedure.\n- Achieved state-of-the-art performance using less parameters.", "reasons_to_reject": "- Paper organization: The explanation about CMWPA dataset is not sufficient. (I think the authors should bring some details about the dataset from the appendix).", "questions_to_authors": "Question A. You've cited Vaswani's work about multi-head attention in line 300, but the equation 5 seems like a single-head attention. Which thing did you use? If you used single-head attention, then why?\n\nQuestion B. How did you construct CWMPA dataset? What is the base data to synthesize the dataset? I know that the authors already stated it on the appendix, but I think this should be appear at the main paper.", "paper_topic_and_main_contributions": "This paper develops a new model that does deduction on math word problems by generating operations. They proposed a model called GeDe, which generates an operation for each deduction step using mDAG structure. To see the effect of their proposal, they tested their model on the synthesized dataset CWMPA and other well-known public benchmarks. Also, they conducted an ablation study to verify whether the components proposed are working as expected. Through these experiments, they showed that their model can handle the N-ary operators properly (as expected) and achieves the state-of-the-art performance using less parameters.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The introduction about CWMPA should appear, when you use the term CWMPA (line 434). The introduction about CWMPA is stated in lines 477-485, which is very far from line 434.\n- Could you add the number of parameters for each model, in Table 1?"}}, {"review_id": "XcedvzMPYA", "reviewer": "Reviewer_SjPM", "summary": "", "questions": "Did you conduct experiments with alternative methods on your proposed CMWPA dataset to ensure a fair comparison of various solving techniques, particularly S2S approaches, given that binary or advanced operations should not affect S2S methods? Additionally, did you test your models with Roberta-large, does it improve the performance? It will give us more details about the extent to which your proposed method can achieve, as well as another fair comparison with previous sota models, e.g. generate&rank.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "The dataset CMWPA is primarily used to demonstrate that using mDAG as the output logic form is superior to using bDAG as the logic form, rather than to compare the performance of different models. We complement the performance of Generate&Rank on CMWPA. The accuracy of the model is 97% in the case of using mDAG annotations, while it is 32% in the case of using bDAG annotations. This further illustrates the advantages of mDAG.\n\nWe replaced Roberta-large with Roberta-base in our model and evaluated it on the Math23K dataset. The model achieved an accuracy of 86.7% on the Math23K test set. Using Roberta-large resulted in a 1.3% improvement in accuracy compared to using Roberta-base. This demonstrates that using Roberta-large in our model  enhances performance and outperforms the baseline Generate & Rank model on this Math23K test set.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The writing is clear and they performed an ablation analysis. In addition, they collected a dataset to show the effectiveness of their proposed method.", "reasons_to_reject": "Although the study includes the use of a re-encoder, hierarchical beam search, and quantity embeddings, the improvements in performance is limited. Furthermore, the authors did not report the setting of random seeds or the average performance during multiple experiments.", "questions_to_authors": "Did you conduct experiments with alternative methods on your proposed CMWPA dataset to ensure a fair comparison of various solving techniques, particularly S2S approaches, given that binary or advanced operations should not affect S2S methods? Additionally, did you test your models with Roberta-large, does it improve the performance? It will give us more details about the extent to which your proposed method can achieve, as well as another fair comparison with previous sota models, e.g. generate&rank.", "paper_topic_and_main_contributions": "The work proposed a multivariate acyclic graph expression and a generation-based model with a re-encoder and decoder to generate the operation sequence  in an iterative manner.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "T0xo2G1iXX", "reviewer": "Reviewer_asVd", "summary": "", "questions": "Q1: Lines #272-#274: How to obtain the representation of the intermediate quantities (such as [QQT_6] and [QQT_7] if they are new quantities)?\n\nQ2: Table 1: Missing related dataset and experimental results descriptions of SVAMP.\n\nQ3: Line #473: CMWPA is a collection of number-word questions synthesized by artificially handcrafted templates that do not contain natural language descriptions in the real world. There are publicly challenging MWP datasets (such as Math or ASDIV). Did the authors conduct experiments on these datasets?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: We have detailed the representation of new quantities through equations 2, 3, and 4. In equation 2, new quantities are represented as the output of an operation. Subsequently, as illustrated in equation 3, this expression from equation 2 is concatenated with the original input and then re-encoded to yield $R^t$. Lastly, as depicted in equation 4, we extract the vectors associated with these new quantities within $R^t$ to serve as their representations.\n\nQ2: Thank you for pointing out the issue, and we will follow up by revising our paper.\n\nSVAMP is a more challenging dataset that is manually created to evaluate a model’s robustness. They apply variations to the instances sampled from MAWPS. Such variations could include adding extra quantities, swapping the positions between noun phrases, etc. On this dataset, GeDe can outperform the vast majority of baselines except the DR model.\n\nQ3:  Evaluating our model on these two datasets necessitates the implementation of a multitude of intricate parsers and solvers to process and execute the mathematical operations it generates. While this task falls outside the scope of our research forcus and the workload involved is substantial, we will defer it to future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed GeDe approach (includes dynamic quantity embeddings, re-encoder, and multivariate operation mechanisms) is promising, and the experimental results illustrate the effectiveness of solving challenging MWPs. Insights of related experiments could be beneficial for future research on MWP solving. ", "reasons_to_reject": "To demonstrate the effectiveness of the proposed method, the authors additionally construct a synthetic data set instead of using public datasets with challenging MWPs. Consequently, related issues (such as Q3) are to be discussed and explored. Furthermore, the evaluation results of SVAMP in Table 1 are unclear and required to be explained. ", "questions_to_authors": "Q1: Lines #272-#274: How to obtain the representation of the intermediate quantities (such as [QQT_6] and [QQT_7] if they are new quantities)?\n\nQ2: Table 1: Missing related dataset and experimental results descriptions of SVAMP.\n\nQ3: Line #473: CMWPA is a collection of number-word questions synthesized by artificially handcrafted templates that do not contain natural language descriptions in the real world. There are publicly challenging MWP datasets (such as Math or ASDIV). Did the authors conduct experiments on these datasets?\n", "paper_topic_and_main_contributions": "This paper presents a generation-based deductive (GeDe) approach to construct a multivariate directed acyclic graph (mDAG) for MWP solving to address the repeated subtree generation issue and the binary tree expression limitation. The GeDe constructs a mDAG by generating an operation sequence iteratively. GeDe re-encodes the input for each reasoning step by adding new intermediate quantities (concatenating the original MWP) and decodes to generate a new multivariate operation. A hierarchical beam search (operation candidates seeking and operation sequence selecting) is also proposed to improve the model performance. Experimental results demonstrate the effectiveness of three public datasets and one newly released CMWPA dataset.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Measuring Mathematical Problem Solving With the MATH Dataset, Hendrycks et al., NeurIPS 2021\n\nA Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers, Miao et al. ACL-2020\n", "Typos_Grammar_Style_and_Presentation_Improvements": "(1)Table2, Row1: “bDAG” -> BET?\n\n(2)Lines #717-#721. Missing the reference source.\n\n(3)Table 6: SVAMO -> SVAMP\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers highlight the novelty and exciting contributions of this paper. The paper is of good quality, both in terms of writing and analyses. There are some reasons to reject the paper, but they are mainly concerned with adding some clarifications or restructuring the paper.

Paper Topic And Main Contributions:

This paper introduces a novel deductive approach called GeDe for solving math word problems (MWP) by generating operations iteratively using a multivariate directed acyclic graph (mDAG) structure. GeDe addresses issues related to repeated subtree generation and binary tree expression limitations. It achieves this by re-encoding the input at each reasoning step, introducing new intermediate quantities, and decoding to generate multivariate operations. The paper also presents a hierarchical beam search to enhance model performance. Experimental results on multiple datasets, including the newly released CMWPA dataset, demonstrate GeDe's effectiveness in achieving state-of-the-art performance with fewer parameters while properly handling N-ary operators.

Reasons To Accept:

* Novelty of the Proposed Approach
    * Successful extension of previous deduction approaches to N-ary operators, addressing a critical limitation in math word problem-solving.
    * Devising a hierarchical beam search procedure for the decoding process, enhancing the model's efficiency and performance.
    * Achievement of state-of-the-art performance while using fewer parameters, demonstrating the paper's contribution to optimization in the field.
* New Data Set
    * Collection and utilization of a dataset to showcase the effectiveness of the proposed GeDe approach, including dynamic quantity embeddings, re-encoder, and multivariate operation mechanisms.
* Quality of the Writing
    * Clear and well-structured writing
* Quality of the Analyses
    * Insightful ablation analysis.
    * Promising results and insights from related experiments can potentially inform and benefit future research in the field of math word problem-solving.

Reasons To Reject:

* The core of the paper lacks some details that should be included in there and not in the appendix.
* The authors did not report the setting of random seeds or the average performance during multiple experiments.
* The authors need to add more explanations of the results.
Decision: Accept-Main