Non-Autoregressive Math Word Problem Solver with Unified Tree Structure
Authors: Yi Bin
Mengqun Han
WENHAO SHI
Lei Wang
Yang Yang
See-Kiong Ng
Heng Tao Shen
Venue: EMNLP 2023 Main (2023)
Keywords: MWP solving
non-autoregressive solver
unified tree structure
TL;DR: This paper devises a non-autoregressive goal-driven method to generate the expression with unified MTree structure. Besides, an MTree based metric is introduced to make the expression evaluation more accurate.

Abstract: Existing MWP solvers employ sequence or binary tree to present the solution expression and decode it from given problem description. However, such structures fail to handle the  variants that can be derived via mathematical manipulation, e.g., $(a_1+a_2)*a_3$ and $a_1 * a_3+a_2 * a_3$ can both be possible valid solutions for a same problem but formulated as different expression sequences or trees. The multiple solution variants depicting different possible solving procedures for the same input problem would raise two issues: 1) making it hard for the model to learn the mapping function between the input and output spaces effectively, and 2) wrongly indicating \textit{wrong} when evaluating a valid expression variant. To address these issues, we introduce a unified tree structure to present a solution expression, where the elements are permutable and identical for all the expression variants. We  propose a novel non-autoregressive solver, named \textit{MWP-NAS}, to parse the problem and deduce the solution expression based on the unified tree. For evaluating the possible expression variants, we  design a path-based metric to evaluate the partial accuracy of expressions of a unified tree. The results from extensive experiments conducted on Math23K and MAWPS demonstrate the effectiveness of our proposed MWP-NAS. The codes and checkpoints are available at: \url{https://github.com/mengqunhan/MWP-NAS}.

Non-Autoregresive Math Word Problem Solver with Unified Tree Structure
Yi Bin', Mengqun Han', Wenhao Shii, Lei Wang', Yang Yang²
See-Kiong Ng', Heng Tao Shen²
National University of Singapore, University of Electronic Science and Technology of China Singapore Management University
fyi.bin, shenhengtao] @ hhommail.com, hanmengqun @ foxmail.com
[shiwenhao16, dlyyang) @ gmail.com, lei, wang. wwang.2019.. 2019@ 20de@s wenddssssmu.coo. smu.edu.sg, seekiong @ nus.edu.sg
Abstract
Problem: Dana earn $13 per hour, She worked 10 hours on Saturday and 3 hours on Sunday, and spent $40 on Saturday. How much money did Dana have?
Existing MWP solvers employ sequence bi- nary tree to present the solution expression and decode it from given problem descrip- tion. However, such structures fail to handle the variants that can be derived via mathemat- ical manipulation, e.g., (a1 a2) a3 and 41 ag 02 *0g can both be possible valid solu- tions for a same problem but formulated as dif- ferent expression sequences trees. The mul- tiple solution variants depicting different possi- ble solving procedures for the same input prob- lem would raise two issues: 1) making it hard for the model to learn the mapping function between the input and output spaces effectively, and 2) wrongly indicating wrong when eval- uating a valid expression variant. To address these issues, we introduce unified tree struc- ture to present a solution expression, where the elements are permutable and identical for all the expression variants. We propose novel aon-auutoregesssvve solver, named MWP-NAS, parse the problem and deduce the solution expression based on the unified tree. For eval- uating the possible expression variants, we de- sign a path-based metric evaluate the partial accuracy of expressions of unified tree. The results from extensive experiments conducted on Math23K and MAWPS demonstrate the ef- fectiveness of our proposed MWP-NAS. The codes and checkpoints are available at: https: //gi thub. com/ com//mengguuhaan//MMWWW
13 33x10+13x 40 Sat income + Sun income all expense
Reasonable Solutions
1 110 40 income expense
(13 133x00-440 1100--40 40x Saturday Sunday
Binary Trees
40
13
30
10
Unified MTrre
10
Answer
129
Figure I: An MWP example with multiple reasonable solutions and corresponding binary trees, but only one unique output based on the unified MTTre structure.
mulate the corresponding solution expression by employing suitable mathematical operation sym- bols (e.g.. -, / to link the relevant numbers in the problem. Due the complex reasoning in text and expression, automatically solving math word problems has always been considered as one of the key testbeds for evaluating the intelligence level of an Al agent (Charniak, 1969; Clark, 2015).
1 Introduction
The approaches for automatically solving MWP proposed by various researchers over the years can be categorized into three stages: manual features and rule-based at early stage, followed by facilitat- ing quantitative reasoning with semantic parsing, and the use of deep learning in recent years (Zhang et al., 2019). Methods from the first two stages did not work well as they were limited in their capacity for feature representation and language understanding. With the success of deep learning in natural language understanding, a series of neu-
Automatically solving math word problems (MWP) is an important and fundamental task in Artifi- cial Intelligence that has attracted great deal of research attention throughout the years (Bobrow, 1964; Wang al., 2017; Zhang et al., 2020; Huang al., 2018). Figure illustrates an MWP example and its solutions. Given an input word problem de- scription, the MWP solver needs to understand the problem in natural language, figure out the math- ematical logic underlying the problem, then for-

---

ral solvers for MWP have been proposed recently and achieved superior performance. Inspired by the sequence to sequence modeling in machine transla- tion, several works (Wang et al., 2017; Huang et al., 2018; Wang et al., 2019) considered MWP as a se- quence to sequence mapping problem, from natural language to mathematical expression, and sequen- tially generate the numbers and operators of the ex- pression. However, such sequential representation ignored the arithmetic properties of the expression. As such, the binary tree was introduced to present the expression and achieved tremendous progress with Seq2Tree models (Xie and Sun, 2019; Shen and Jin. 2020; Lin et al., 2021) and Graph2Treee models (Zhang et al., 2020; Li et al., 2020). More recently. the application of pre-trained language models to MWP solving (Liang et al., 2022; Shen et al., 2021) have led to new state-of-the-ar due to the superior capacity and performance in language modelling.
lem and expression. However, such codes break the mathematical relations between numbers (Jic et al.. 2022) and may lead to inferior performance. Furthermore, the code dictionary has to be prede- fined which limits the flexibility and may incur the issue of out-of-vocabulary (OOV) during in- ference. Motivated by the intuition of goal-driven strategy for tree-structure decoding, we propose non-autoregressive MWP solver, MWP-NAS for short, to integrate with goal-driven decoding based on MTree. Specifically, our MWP-NAS first ob- tains the semantic representation of problem text via a neural language model, and uses as the initial goal for MTTree decoding. As there exist mul- tiple unordered child nodes for each parent, we em- ploy a non-autoregressive Transformer to explore the comprehensive interactions among the nodes and predict them in parallel. For each sub-goal, we recursively implement such -uooreeeeereessivve decoding until all the child nodes are numbers. A cross-goo -goal attention strategy is devised to enable the interactions between goals and make the de- composed sub-goals more accurate. In MTree, as each number could be applied in one of four forms: {n, -n, 7 - }.), we design simple classification module to learn the form, which is jointly trained with the non-autoregressive decoder. We also ob- serve that under the current evaluation approaches, the variants of the same expression would be re- ported as wrong by expression evaluation, a.k.a false negative, which may underestimate the ability of the methods. Furthermore, we want to be able to evaluate the ability of a solver not only from the final expression, but also considering the partial correctness of the sub-expressions, Towards this end, we propose two MTree based evaluation met- rics: 1) MTree Ace measuring the exact matching accuracy unified MTree, and 2) MTree IoU cal- culating the intersection over union of MTTree paths between predicted and ground- truth expressions. In summary, the main contributions of this paper as follows:
Most existing approaches learn to match exactly the target expression and ignore the alternative ones that can be derived by the arithmetic laws, e.g., commutative law, associative law, and distributive law. As the example shown in Figure 1, we can calculate the answer following either the rules of income expense Saturday + Sunday re- sulting in different expressions and binary trees. However, translating the same problem description to different expression variants inconsistent with the function mapping, which introduces output in- determinacy and makes the model difficult to opti- mize. Furthermore, the issue of variant expressions may lead to problems in expression evaluation due to possible false negatives and thus underestimation of the performance of the methods.
To address the above issues, we propose to learn the MWP solver based on unified form of expres- sion. Specifically, we climinate the diversity of expression variants and unify them to an identical format with MTree structure (Wang et al., 2022). Unlike having only two branches in binary tree, MTrre may contain multiple branches for each node, as shown in Figure 1. Since the expression diversity is mainly introduced by the computational priority, MTree removes the / and operators, and introduces and +/ to make all the operators are with the same superiority and the child nodes of the same parent are permutable. Based on such MTree, Wang et al. (2022) design MTree codes to represent and learn the mapping between prob-
We propose a novel non-autoregressive solver for MWP, dubbed MWP-NAS, which imple- ments a goal-driven non-autoregressive man- ner to decompose the goal to sub-goals based on the unified MTTree structure. To enable in- formation passing between goals for sub-goal decomposition, we devise novel cross-goal attention to make the model leverage informa- tion across goals.

---

We design two metrics for expression eval- uation based on MTree, to better assess the effectiveness of MWP solvers.
bottom-up manner. Bin al. (2023a) introduced the reexamination process of human and devised model-agnostic pseudo-dual learning scheme to fur- ther improve the performance. With the booming of pre-trained language models (PLM), researchers have been trying to strengthen the models with PLM. Liang et al. (2022) built number-aware MWP-BERT with PLM effectively learn the con- textual number representation. Using the PLM encoder, e.g., BERT or RoBERTa, to extract prob- lem text representation also significantly boosts the MWP solving accuracy (Jie et al., 2022; Li et al., 2022; Wang et al., 2022).
Extensive experiments are conducted on Math23K and MAWPS, and the results demonstrate that our MWP-NAS outperforms all the SoTAs. We also compare several mile- stone baselines evaluated by our MTree Acc and loU, and analyze the effectiveness of our MTrre-basee basee metrics.
2
Related Works
2.1 MWP Solving
2.2 Non-Autoregressive Transformer
MWP solving has been attracting wide research attention for a long time (Bobrow, 1964; Bakman, 2007; Kushman et al., 2014; Shi et al., 2015; Mitra and Baral, 2016). Inspired by the superiority of deep neural networks, deep learning based meth- ods have dominated this area and achieved impres- sive progress. Wang et al. (2017) first regarded MWP as sequence sequence mapping problem, and designed an RNN-based seq2seq model, GRU encoder and LSTM decoder in specific, sequen- tially generate the numbers and operators of the ex- pression. Based on the seq2seq model, Wang et al. (2018b) and Huang et al. (2018) introduced deep re- inforcement learning to MWP solving and achieved promising performance. Robaidek et al. (2018) 8) also made an attempt to employ convolutional neu- ral networks encoder and decoder. Wang al. (2019) proposed predict the expression template and then fill the operator into the template by de- signing a recursive neural network.
Non-Autoregressive Transformer (NAT) (Gu et al., 2018) is proposed to accelerate the decoding pro- cess in machine translation by generating all the words in parallel. As pointed in (Gu al., 2018), however, NAT suffers from the multimodality prob- lem and exhibits complete conditional indepen- dence, resulting in severe repetition issuc in gen- erated texts and inferior generation performance. To address these issues, many works have been proposed to enhance the latent representations by integrating extra refinement process (Li et al., 2019b: Guo al., 2020), design a semi- autoregressive fashion (Mallinson et al., 2022: Wang et al., 2018a). Besides, Huang et al. (2022) introduced a directed acyclic transformer to capture multiple translations and facilitate fast predictions in NAT, demonstrating superior performance. In ad- dition, Bin et al. (2023b) employed NAT for order- ing problem which could benefit from the bi-lateral dependencies modelling and avoid the repetition is- sue by designing an exclusive loss, as well as outfit generation (Ding et al., 2023)
Subsequently, Xie and Sun (2019) proposed a goal-driven expression tree generation strategy, which recursively decomposes the current goal into sub-goals via top-down manner. The tree-based expression template significantly improved the so- lution accuracy. Zhang et al. (2020) introduced Quantity Cell Graph and Quantity Comparison Graph to enrich the problem representation, and de- signed a Graph2Tree framework to learn the expres- sion tree. Cao et al. (2021) applied Direct Acyclic Graph (DAG) to represent the expression and de- vised Seq2DAG model, aggregating the numbers and sub-expressions, to obtain the DAG of expres- sion. Jie et al. (2022) started from deductive rea- soning and regarded MWP solving as a complex relation extraction problem, then proposed to learn the relation between two quantities iteratively in a
Following (Wang et al., 2022), our work tries to unify the expression using MTree and learn the mapping function between problem text and MTTre. Wang et al. (2022) predefined a code dictionary over the training set and may result in OOV and poor generalization. To address these issues, we propose non- an--autrregrssivee decoder to learn the MTTree of expression with a goal-driven strategy.
Methodology
To handle multiple branches in MTree, we propose a non-autoregressive solver, termed as MWP-NAS, for the child nodes generation. As depicted on the left of Figure 2, our MWP-NAS mainly consists

---

of Problem Encoder and Goal-Driven MTree Generator to understand the problem and reason the MTTree structure of solution expression, which used to compute the final answer. To handle multi- branch decoding in MTTree generator, we devise a novel non -utooorrreressiv goal decomposer, shown on the right side in Figure 2. All the details will be introduced in the subsequent parts.
to compact problem representation, and decode the problem representation to MTree, as shown on the left of Figure 2. Two kinds of language models are commonly used in previous works: the recurrent neural networks RRNN-based), e.g., LSTM or GRU, and the pre-trained language models (PLM-based)) e.g., BERT or RoBERTa. Inspired by the supe- rior representation ability of PLM, recent works tend to employ PLMs problem encoder. We follow (Jie et al., 2022: Wang al., 2022) im- plementing RoBERTa-bas (Liu et al., 2019) or BERT (Devlin et al., 2019) to extract the prob- lem representation. Specifically, given the problem S { [w1, U22 .... ( containing numerical val- ues V = (v1, 12, .... 1m), we concatenate a [CLS] and [SEP] at the beginning and end of the problem text, respectively. Then the output of [CLS] is em- ployed as the entire problem representation. The problem encoding process could be formulated as:
3.1 Preliminary of MTree
MTree is first introduced to MWP by (Wang et al.. 2022), unify the expression tree structure. Given an expression, we first transform it to a plain ex- pression by removing the brackets with SymPy'. an arithmetical Python package. The plain expres- sion only contains numbers and four arithmetical operators {+, -+,-,x//}> x,,
where the operands of {+, } can be swapped while the ones for {-./} cannot. The unswappable operators also cannot be applied in multi-branch tree because different order of operands would lead different results. To tackle this issue, two new operators {x-,+/} are introduced replace (-./-- x- indicates to get the opposite value of the product of mul- tiple numbers, e.g., x- with operands {2 is equal to (2 x 3). +/ means calculating the re- ciprocal for the sum of the operands, e.g., 2+3 for operands 3). Finally, the operators for MTTree are {+, X, x-, +/). where all of them able handle multiple operands and the operands swappable. The summarized meanings of operators are as follows:
(1) where E, is the output of [CLS] for the entire prob- lem representation, and Ey denotes the contextual representations of numbers. We also finetune the PLMs during training make the learned repre- sentations better the MWP task.
Eg, Fly ::
1
3.3 Goal-Driven MTTre Generator
Expression tree decoding has been well explored in MWP solving, e.g., sequential decoding with pos- torder traversal and goal-driven decomposing. Mo- tivated by the intuition and success of goal-driven decomposing (Xie and Sun, 2019), we implement the MTree generator following the top-down de- composition with goal-driven mechanism. Specif- ically, we utilize the problem representation (E, in Equation as the root goal of the MTree, then recursively generate the sequence of sub-goals with the top-down fashion, until the sub-goal not an operand. The sub-goals are categorized as an ex- plicit token, e.g., operand or operator, by measuring the similarity between sub-goals and candidates. The candidate representations are defined as:
+ (x) means the sum (product) of operands, e.g., 1.g.1+2-4 3 and 1 2 x for the operands (1,2,3).
x- -+/) means the opposite (reciprocal) of the productive (sum) of the operands, e.g., -(1 2 x 3) and I1213 for the operands (1,2,3).
Besides, to handle the different forms of numbers, e.g., negative numbers and fractions, MTree also introduces four types of variants [n, 4, nn, 1 to denote them, as the example shown in Figure 2.
Ev.
if it is number
3.2 Problem Encoder
Eops
if it is an operator
E.
(2)
Given math word problem, we first employ a neu- ral language model to convert the discrete words
Econ:
if it is constant
Ey,
if it is the special token N₂
httpp::/ http://wwwwpppwwtptttppp
where Fly is the encoder output in Equation 1, and other E, are learned embedding. Nú, is a special to- ken indicating the end and number of branches of a
'oor the operations beyond these four arithmetical oper- ators, such as a", we follow the processing in (Wang et al., 2022) to convert it to multiple times product.

---

Problem Encoder
Non-Autoregressivee Goal Decomposer
Sub-goal Outputs: X, X,
Problem: Dana earns $ N,(13) hour, She worked N_(10) hours Saturday and Ny(3) Sunday, spent N,(40) Saturday. How much money Dana have?
N1
22
N
N
X-
+
N
Type Classifier
Goal-Driven MTrre Generator
Pointer Network
initial goal
K, V
Multi-head Inter-attention
FFN
X
-40
x
sub-gools
Mead Self attention
Legend
Goal decomposer
V Goal feature
13
10
13
Goal
P
P4
Token feature
Figure 2: The overall pipeline of our MWP-NAS, which consists of a problem encoder and goal-driven MTTree generator. To implement top-down decomposing, we devise a non-autoregressie goal decomposer (shown on the right side) to implement the multi-branch decomposition.
goal, which will be detailed in next section. Unlike the binary tree where the left and right branches are ordered, the multiple branches (may more than two branches) of our MTrre are unordered. Different from (Xie and Sun, 2019) generating left and right sub-goals with different modules, need to treat all the sub-goals the same and generate them with identical module.
where Wo, WK. and Wy are learned mapping matrices. To make the communications between sub-goals available, we propose cross-goal atten- tion to implement the self-attention of positions. As the example shown in Figure 3(b), when we de- compose the left "x" node, also peek at the in- formation of nodes "-40" and another "x", while the vanilla attention (shown in Figure 3(a)) only capture the information the left "x" node. For the leaf node 10, we attach dummy child nodes to pass the information to other child nodes. Note that the dummy nodes are only used for cross-goal interaction, and would not be used for loss com- putation. Through such cross-goal attention, the sub-goals could interact with each other and avoid duplicate decomposition.
3.4 Non-Autoregressive Goal Decomposer
Towards this end, design novel autoregressive goal decomposer (NAGD) han- dle the unordered multi-branch decomposing as se- quence generation, based on the non-autoregressive Transformer (Gu et al., 2018). The pipeline of our proposed NAGD is illustrated on the right in Fig- ure 2. Given the goal representation Eg. the NAGD first integrates it with every positional embedding via element-wise sum, and the fused representation is denoted Ep. Following (Vaswani et al., 2017), we embed positions with sine and cosine functions:
We then connect the positions and token candi- dates by a multi-head attention block, which employs Ép as Q, and learns K and V from candi- dates, as:
F (e (E,Wv), (6) Vdz
Pt2j sin 2in(2/10000
(3)
E, soft
P1,2++1 cos(i/ cos(i/100002)/d4)
(4)
where E, is the representation of target candidates defined in Equation 2. Finally, the contextual rep- resentation Ep could be used select the most relevant candidates, egg., operands or operators, via a pointer network (Vinyals et al., 2015). The prob- ability of position to choose the j-th candidate is
where denotes the position and the j-th di- mension in Pi- To explore relative information and dependencies between positions, we implement a multi-head self-attention as:
Es MHAu( E, F,WK, F,W,W,,,WWWWW
(5)

---

4 Evaluation
40
Most cxisting works employ expression accuracy and value accuracy to evaluate and compare the performance of MWP methods (Xie and Sun, 2019; Wang et al., 2017; Jie et al., 2022). Value accuracy measures the accuracy of the final answer, but fails to evaluate the validity of the solving procedure. Expression accuracy trics to address this issue and measure the exact matching accuracy between pre- dicted and ground-truth expression, and results in much lower performance due to the diverse variants of the same solution expression. Obviously, the ex- pression accuracy should be the same value accu- racy in theory, slightly lower for some exception cases, The ideal metrie evaluate the expression (7) should be capable of handling the reasonable vari- ants of solution expressions. In other words, all the variants derived from the gold expression with arithmetic rules would lead to the gold answer, and (8)
E
E
P
P
P1
P
P.
P₂
P
Pa
MHAtt
MHAtt
E,
E
(a)
(b)
Illustration of vanilla multi head self-attention (a), and our proposed cross-goal attention (b). MHAtt means multi-head self-attention.
Figure 3:
formulated as:
Wij = u² tan + Whef),
Ptr, = softmaz(ws),
should be considered as true.
where W. are learned parameters, and u is a col- umn weight vector. ee" and ef are the representa- tions of i-th position and candidate Pir is the probabilistic distribution across all the candidates for i-th position.
However, it is hard to include all the variants and conduct an exhaustive evaluation. Based on the uniqueness of MTrre in expression variants, we propose MTree Accuracy and MTree IoU to evaluate the expression accuracy more accurately. Specifically, following existing expression accu- racy verify the exact matching accuracy the whole tree, MTree accuracy measures the match- ing accuracy between predicted and gold unified MTrees. While such evaluation on the whole ex- pression tree fails to measure the partial correct- ness of expressions, which is also an important way to evaluate the ability of solvers, as well as humans. For example, given the example in Fig- ure 1, 13 x (10 3) + 40 and (13 10 + 40) are two false expressions, and the former one only uses the wrong operation "4" for the expense, but all the items are correct. While the latter applies incorrect calculation of income on Sunday, which should obtain a lower score. To this end, we pro- pose MTree IoU to calculate the accuracy of paths connecting root and leaves to measure the partial correctness of expressions. MTree IoU first trans- forms the predicted and ground-truth expressions to MTrees, then constructs root-leaf path sets P, and Pg for them, respectively. Inspired by the eval- uation object detection (Ren et al., 2015), we calculate the Intersection over Union (IoU) as:
To calculate the loss between predictions and ground-truthe we need align the positions and the ground-truth uob-goals. We therefore define a pseudo order for ground-truth tokens of each goal. The operators are ahead, followed by the constants and operands in the problem. For the example in Figure 2, the pseudo order of ground- truth sub- goals for node + is [x, x, 400. Besides, vanilla non-autoregressive Transformer (Gu et al., 2018) implements fertility prediction to decide the decod- ing length. In our NAGD, we a max length as 8 for all the samples, because more than 99% sam- ples in the datasets are with MTree branches less than 8. Meanwhile, we also append a special token N₁ at the end of sub-goals (similar with the <END> token in machine translation) to indicate the num- ber of branches for the current goal decomposing, and the loss and prediction after the N₂ would be ignored during training and inference, respectively.
After predicting the token of each sub-goal, we need to indicate the type (fn, 1 -n, } men- tioned in Section 3.1) for predicted operands. We design simple MLP for type classification, and employ focal loss to mitigate the issue of imbal- anced classes. The loss is finally integrated with pointer loss to jointly train the whole MWP-NAS.
P P
MTreeloU
(9)

---

where |.1 denotes the size of the set. The over- all MTree IoU of test set is averaged over all the test samples. Note that there may exist duplicate paths in an MTree due to the duplicate numbers and multiple usages, such as the number "13" in Figure is used multiple times in the MTree. We treat such duplicate paths different paths in the IoU caleulation, because if the solver only predicts of them, it should not be totally correct. In our codes, we actually use 'List' as the type of Po and Pg in Equation 9, rather than 'Set' that may ignore the duplicated paths.
Table 1: Performance comparison with basclines. ?
means the results referred from (Wang et al., 2022) means the results based on SUMC split. + indicates the results of the data split (Jie et al., 2022), which implements 5-fold CV train-test setting for Math23K (we call it Reasoner Split).
Model
Math23K
MAWPS
Seq2Seq (Wang et al., 2017)
58.1
59.5
T-RNN (Wang 2019)
66.9
66.8
GROUP-ATT (l.i al., 2019a)
69.5
76.
GTS (Xie and Sun, 2019)
75.6
82.6
Graph2Tree (Zhang et al., 2020)
77.4
83.7
NeuralSymbolic (Qin et al., 2021)
75.7
HMS (Lin et al., 2021)
76.1
80.3
NUMS2T (Wu al., 2021)
78.1
5 Experiments
BERT-Tree (Li al., 2022)
82.4
SAU-Solver (Qin et al., 2020)
76,2
75.5
UniLM (Dong al., 2019)
77.5
78.0
5.1 Datasets
DeductReasoner (Jie et al., 2022)
84.3
$6.0
To evaluate the effectiveness of the proposed MWP- NAS and metrics, conduct extensive experi- ments on Math23K and MAWPS, and compare with SoTAs. Math23K (Wang et al., 2017) con- tains 23162 arithmetical problems. For fair com- parison, we use the splits following (Wang et al., 2022; Zhang et al., 2020), resulting in 21162, 1000, and 1000 for training, validation, and test, respec- tively. MAWPS (Koncel-Kedziorsii et al., 2016) is a much smaller MWP dataset, which contains 2373 samples. We follow (Wang al., 2022) to prepro- cess for MTree and obtain 2163 samples. Due its small size, following previous works (Zhang et al., 2020; Jie et al., 2022), we conduct 5-fold cross- validation on MAWPS, resulting in 433 samples for four folds and 431 samples for another fold. We report the averaged results across five folds.
DeductReasoner (Jic et al., 2022)
85,1 4
91.2
SUMC-Solver (Wang ct al., 2022)
82.5
82,0
MWP-NAS (SUMC Split)
84.8
86.7
MWP-NAS (Reasoner Split)
86.1
91.4*
size of our non-autoregressive goal decomposer is 768 for every layer. We run 1000 and 2000 epochs on Math23K and MAWPS for training, and choose the checkpoints with the best performance on vali- dation set for test.
5.3 Comparisons with Baselines
We first evaluate and analyze the effectiveness of our proposed non-auloregressive solver for MTrre by comparing its performance with the state-of-the-- arts. Here we only compare the accuracy of the fi- nal answer and leave the expression accuracy evalu- ation in the next section, because of two points: (I) The expression generated by our MWP-NAS based on MTTree could be written as multiple reasonable expression variants, which is hard to evaluate the expression accuracy, and (2) only few numbers of previous works reported the expression accuracy, would be many N/A in the comparison. As the results shown in Table 1, our proposed MWP-NAS outperforms the baselines and sets a new state- of-the-art on both datasets, which demonstrates the effectiveness and superiority of our method. As the only two methods based on MTree, the SUMC- Solver employs codes prediction of leaf node to re- construct the expression tree, which performs much lower than our MWP-NAS. The reason might be that the code prediction of leaf nodes makes the nodes independent and break the arithmetical re- lation between the nodes. While our MWP-NAS implementing attention mechanism is able to ex-
5.2 Experimental Settings
We implement the proposed MWP-NAS with Py- Torch and conduct experiments with an NVIDIA RTX A6000 GPU. The maximum size of MTree branch set as 8, because more than 99% sam- ples in the datasets have fewer than 8 child nodes. We fine-tune the pre-trained language models, i.e., RoBERTa/BERT-bas during training and initial- ize the global learning rate as 25-5 and 5 for Math23K and MAWPS, respectively. The hidden
PIn the main part of (Jie et al., 2022), the authors imple- ment a different training setting, combining training and validation sets for training. and run five times with different random seeds for averaging on Math23K. The 5-fold split of MAWPS also has two different settings: the split in (Wang et al., 2022) and others. For fair and comprehensive compar- isons, we implement our MWP-NAS on these two splittings, and term them as Reasoner Split and SUMC Split in Table 1. The main experiments (including the ablations) are based on SUMC split.

---

6000
244
3 P: 315x
G&P: 6000 + 6000x3.24%X2
G&P: 360x 5+
G: 315x
Figure Several cases our MWP-NAS with MTree structure. G and indicate the Ground truth and Prediction. respectively. We merge the ground truth and prediction for correct predictions, i.e. (a) and (b), and illustrate failure case in (c).
Table 2:
The results of ablation study of cross-goal attention on Math23K and MAWPS.
SUMC-Salver
Model
Math23K
MAWPS
MWP-NAS
w/o Cross-Goal Attention with Cross-Goal Attention
84.1
86.1
Val (%)
70
84.8
86.7
plore and capture the relations between numbers, and yield better results. Besides, also note De- ductReasoner (Jic et al., 2022), implementing plex relation modeling and deductive reasoning. performs very close to our work for both data split settings, which may imply that introducing the de- ductive reasoning to MTree structure would bring some new insights.
40
25
Branch Number MTrre
Figure Performance comparison on different branch numbers of MTrre.
5.4 Effectiveness of Cross-Goal Attention
imum number of branches in it. For example, the branch number the MTrre in Figure three, because the "4" node is the one with the most branches in it and has three branches. The experi- mental results are illustrated in Figure 5. From the results, we can observe that our MWP-NAS outper- forms the SUMC-Solver all the branch numbers, and exhibits remarkable boosting larger branch numbers, which means that our MWP-NAS works better than SUMC-Solver for more complicated problems. Besides, with the branch number creasing, both methods show decreasing trend for value accuracy, which exhibits more inferior perfor- mance for more difficult problems and is consistent with common sense.
We also conduct ablation studies investigate the effectiveness of the proposed cross-gon goal atten- tion, and show the results in Table 2. The results demonstrate that equipped with our cross-gos -goa at- tention, our model gains significant improvement, e.g., from 84.1 to 84.8 on Math23K. This suggests that with such a cross-goal attention mechanism, the information belonging to different goals could be passed and aggregated. Such cross-goal infor- mation integration apparently improves the accu- racy of single-goal decomposition and benefits the expressiveness of the model overall.
5.5 Analysis on Different Branch Numbers
5.6 Illustration with MTTre Structure
In MWP solving, complex problems always con- tain more operands and result in more branches in MTrre. To investigate the performance on different difficult problems, we test and compare the value accuracy with different branch numbers. We first define the branch number of an MTree as the max-
To comprehensively probe the proposed MWP- NAS, we also visualize several solving cases with MTree structure, as illustrated Figure We ob- serve that our MWP-NAS capable of generating accurate MTree structure, even for the complex

---

Table Evaluation comparison with our proposed MTree Acc and MTree IoU. We reproduce and evaluate all the results based on the released codes.
Model
Exp Acc
Val Acc
MTree Ace
MTree IoU
Seq2Seq (Wang et al., 2017)
51.7
58.3
58.2
62.99
GTS (Xie and Sun, 2019)
64.4
75.7
75.3
82.77
Graph2Tree (Zhang et al., 2020)
66.0
77.4
76.9
83.53
DeductReasoner (Jie et al., 2022)
76.4
84.3
83.6
88.86
SUMC-Solver (Wang et al., 2022)
82.9
82.4
87.26
MWP-NAS (Ours)
84.8
83.8
89.73
Problem:
The fruit shop sells bananas, oranges and pineapples total of 150 Sunday, which bananas are 27.5 kg, number of oranges sold times that of bananas. How many kilograms pineapples are sold in the fruit shop on Sunday?
Ground-Truth Expression: x 3x333x2. 27.5 DeductReasoner: 150 27.5x3 Exp Acc: X (False) MTrre Acc: v/ (True)
Problem: There are 36 chickens, and the number of ducks is twice that of chickens. How many chickens and ducks are there?
Ground-Truth Expression: (1+2)x33
DeductReasoner: 36x2+36
Exp Acc: X (False) MTrre Acc: V (True)
Figure Examples evaluated by Exp Acc and our proposed MTree Acc, predicted with DeductReasoner, Traditional Exp Ace fails to capture the same mathematical semanties with different expressions, while MTrre Acc successfully handles this.
arithmetic expressions. For example, as shown in Figure 4(b), could easily capture the composi- tion between and 4. From the exemplar shown in Figure 4(c), we note our MWP-NAS may fail to predict right type for operands, which may because the optimization objectives of number type is an extra loss. This also motivates us keep exploration of this direction for further improvements.
wrong ground-truth expressions, and two of them give the wrong expression but result in the gold an- swers. For the MTree IoU, it further evaluates the expressions with local paths, and is able to assess the partial correctness of an expression.
Conclusion
We presented novel non-autoregressive MWP solver (MWP-NAS) based the unified MTree structure. Our proposed MWP-NAS implements goal-driven manner for multi-branch decompo- sition to generate the MTree of expressions. We devised cross-goal attention strategy to pass in- formation between goals during goal decomposing. We have also designed two metrics based on MTrre for better expression evaluation. Experiments con- ducted Math23K and MAWPS demonstrated the effectiveness of our approach and metrics.
5.7 Evaluation on MTree Metrics
To study the effectiveness of our proposed M'Tree- based metrics, implement 5 representative MWP methods with the open source codes and compare them with our MWP-NAS with more met- rics on Math23K. The comparison results are illus- trated in Table 3. As aforementioned, the expres- sion accuracy should be consistent with the value accuracy, while it is much lower than value accu- racy in practice. The MTree accuracy measures the performance by transforming the ground-truth ex- pression and prediction to the unified MTree struc- ture, demonstrating slightly lower than the value accuracy. We visualize several examples in Fig- ure 6 to check whether the predicted expression is right and indicated as wrong by expression accu- racy. We observe that MTTree accuracy is able to handle different variants derived from ground-truit egg., commutative law for the first case and distribu- tive law for the second one in Figure 6. We have also checked the samples (only six samples in test set) failed by MTree accuracy but the final value is right, and found that three of them are annotated
7 Acknowledgement
This research is supported by A A*STAR, CISCO Systems (USA) Pte. Ltd and National University of Singapore under its Cisco-NUS Accelerated Digital Economy Corporate Laboratory (Award 12 1001E000).. This research is also partially sup- ported by the National Natural Science Foundation of China under grant 62102070, 62220106008, and U20B2063. This research is also partially sup- ported by Sichuan Science and Technology Pro- gram under grant 2023NSFSC1392.

---

8 Limitations
Yujuan Ding, PY Mok. Yunshan Ma, and Yi Bin. 2023. Personalized fashion outfit generation with user coor- dination preference learning. IPM, 60(5):10343)
To obtain the MTree, we need first to sim- plify and expand the original expression using sympy. simpl.s and sympy. expand of SymPy package. Some cases (about 0.2 percent, i.e. cases in test set) are failed to automatically trans- form to MTrre, due to the expression unification. We discard them during training and use the origi- nal form as ground-truth for test evaluation. This makes our reported performance decrease by 0.2 percent. Therefore, automatically unifying the ex- pressions could be small issue and limitation of work. Besides, with the blooming of large language models (LLMs), we only conducted ex- periments with some relatively small models, i.e., BERT and RoBERTa, for fair comparison, and ig- nored the integration of MTree and LLMs at the moment, which also could be a limitation of our work and will be explored in the future.
Li Dong, Nan Yang, Wenhui Wang. Furu Wei, Xi- aodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Hon. 2019. Unified language model pre-training for natural language understanding and generation. In NIPS, pages 13042- 13054.
Gu. J Bradbury, Xiong, VOK Li, and Socher. 2018. Non-autoregressive neural machine translation. In ICLR.
Junliang Guo, Xu Tan, Linli Xu, Tao Qin, Enhong Chen, and Tie-Ya Liu. 2020. Fine-tuning by curriculum learning for non-autoregressive neural machine trans- lation. In AAAI, pages 7839-7846,
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. In NeurlPS.
Danqing Huang, Jing Liu, Chin- Yew Lin, and Jian Yin. 2018. Neural math word problem solver with rein- forcement learning. In COLING, pages 213-222
References
Fei Huang, Hao Zhou, Yang Liu, Hang Li, and Minlic Huang. 2022. Directed acyclic transformer for non- autoregressive machine translation. In ICML, pages 9410-9428. PMLR.
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Yejin Choi, and Hannaneh Ha- jishirzi. 2019. Mathqa: Towards interpretable math word problem solving with operation-bused for- malisms. NAACL-HLT, pages 2357-2367
Zhanming Jie, Jierui Li, and Wei Lu. 2022. Learning to reason deductively: Math word problem solving as complex relation extraction. In ACL, pages 5944- 5955.
Yefim Bakman. 2007. Robust understanding of word problems with extraneous information. arXiv preprint math/0701393.
Rik Kedzioske,iii,,,i, Subhro Roy, Aida Amini. Nate Kushman, and Hannaneh Hajishirzi. 2016. Mawps A math word problem repository. In NAACL- HLT. pages 1152- 1157.
Yi Bin. Wenhao Shi, Yujuan Ding, Yang Yang. and See- Kiong Ng. 2023a. Solving math word problems with reexamination. arXiv preprint 1xXiv:2310.09590.
Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. 2014. Learning to automatically solve algebra word problems. In ACL, pages 271- 281.
Yi Bin, Wenhao Shi, Bin Ji, Jipeng Zhang, Yujuan Ding, and Yang Yang. 2023b. Non-autoregressive sentence ordering. In Findings of EMNLP.
Daniel G Bobrow. 1964. Natural language input for a computer problem solving system. pages 146-28 146-226.
Jierui Li, Lei Wang, Jipeng Zhang. Yan Wang, Bing Tian Dai, and Dongxiang Zhang. 2019a. Modeling intra- relation in math word problems with different func- tional multi-head attentions. In ACL, pages 6162- 6167.
Yixuan Cao, Feng Hong, Hongwei Li, and Ping Luo. 2021. A bottom-up dag structure extraction model for math word problems. In AAAI, pages 39-46 46.
Shucheng Li, Lingfei Wu, Shiwei Feng, Fangli Xu, Fengyuan Xu, and Sheng Zhong, 2020. Graph-to- tree neural networks for learning structured input- output translation with applications to semantic pars- ing and math word problem. In Findings of EMNLP. pages 2841-2852.
Eugene Charniak. 1969. Computer solution of calculus word problems. In IJCAI, pages 303-316.
Peter Clark. 2015. Elementary school science and math tests as driver for ai: Take the aristo challenge! In AAAI, volume 29, pages 4019-4021.
Zhongli Li. Wenxuan Zhang, Chao Yan, Qingyu Zhou, Chao Li, Hongzhi Liu, and Yunbo Cao. 2022. Seek- ing patterns, not just memorizing procedures: Con- trastive learning for solving math word problems. Findings of ACI, pages 2486-2496.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers language under- standing. In NAACL-HLLT, pages 4171 11866..

---

Zhuohan Li, Zi Lin, Di He, Fei Tian, Tao Qin, Liwei Wang, and Tie-Yan Yan Liu. 2019b. Hint-based train- ing for non-autoregressive machine translation. In EMNLP-IJCNLP, pages 5707-5712.
Yibin Shen and Cheqing Jin. 2020. Solving math word problems with multi aucti-en and multi-decoders. In COLING, pages 2924-2934.
Shuming Shi, Yuehui Wang, Cein-Yew Lin, Xiaojiang
Zhenwen Liang, Jipeng Zhang, Lei Wang. Wei Qin. Yunshi Lan, Shao, and Xiangliang Zhang. 2022. Mwp-hert: Humeracy-augmentt math word problem solving. In Findings of NAACL, pages 997-1009.
Liu, and Yong Rui. 2015. Automatically solving number word problems by semantic parsing and rea- soning. In EMNLP, pages 1132-1145
pre-training for
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. NeurIPS, 30.
Xin Lin, Zhenya Huang, Hongke Zhao, Enhong Chen, Qi Liu, Hao Wang, and Shijin Wang. 2021. HMS: un-
A hierarchical solver with dependency-enhaune derstanding for math word problem. In AAAI, pages 4232-4246
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. NeurlPS, 28.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint 1907.
Bin Wang. Jiangzhou Yang Fan, Xinyu Dai, Shujian Huang, and Jiajun Chen. 2022. Structure-unified m-tree coding solver for math word problem. In EMNLP, pages 3122-8132
Chunqi Wang. Ji Zhang. and Haiqing Chen. 2018a. Seme-auttoeerressvve neural machine translation. EMNLP, pages 479-488.
Jonathan Mallinson, Jakub Adamek, Eric Malmi, and
Aliaksei Severyn. 2022. Edit5: semi--utoreerrsssvve text editing with t5 warm-start. In Findings of EMNLP, pages 2126-2138.
Lei Wang, Dongxiang Zhang, Lianli Gao, Jingkuan Song, Long Guo, and Heng Tao Shen. 2018b. Math- dgn: Solving arithmetic word problems via deep inforcement learning. In AAAI, volume 32.
Shen-Yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020. A diverse corpus for evaluating and developing english math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975-984.
Lei Wang, Dongxiang Zhang, Jipeng Zhang, Xing Xu, Lianli Gao, Bing Tian Dai, and Heng Tao Shen. 2019. Template- based math word problem solvers with re- cursive neural networks. AAAI, pages 7144-7151.
Arindam Mitra and Chitta Baral. 2016. Learning to use formulas to solve simple arithmetic problems. In ACI, pages 21 2144-2153.
Yan Wang, Xiaojiang Liu, and Shuminng Shi. 2017. Deep neural solver for math word problems. EMNLP, pages 845-854.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are alp models really able solve simple math word problems? NAACL-HLT, pages 2080- 2094.
Qinzhuo Wu, Qi Zhang, Zhongyu Wei, and Xuanjing Huang. 2021. Math word problem solving with ex- numerrcal values. ACL-IICNN, pages 5859- 5869.
Jinghui Qin, Xiaodan Liang, Yining Hong, Jianheng Tang, and Liang Lin. 2021. Neural-symbolic solver for math word problems with auxiliary tasks. In ACL-IJCNLP, pages 5870-5881.
Zhipeng Xie and Shichao Sun. 2019. A goal-driven tree-structured neural model for math word problems. In IJCAL, pages 5299-5305 5305.
Jinghui Qin, Lihui Lin, Xiaodan Liang. Rumin Zhang. and Liang Lin. 2020. Semantically-aligne universal tree-structured solver for math word problems. In EMNLP, pages 3780-3789.
Dongxiang Zhang, Lei Wang, Luming Zhang, Bing Tian Dai, and Heng Tao Shen. 2019. The gap of semantic parsing: A survey on automatic math word problem solvers. IEEE TPAMI, 22877228772303
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. NewrIPS, 28.
Jipeng Zhang, Lei Wang, Roy Ka-Wei Lee, Bin, Yan Wang, Jie Shao, and Re-Peng Lim, 2020. Graph-to- tree learning for solving math word problems. In ACL, pages 3928-3937
Benjamin Robaidek, Rik Koncel-Kedziorski, and Han- nanch Hajishirzi. 2018. Data-driven methods for solving algebra word problems. arXiv preprint arXiv:1804.100 10718.
A Appendix
A.1
Refine the MTree Structure
In this part, we attempt to provide an under-mature refinement for MTree structure and hope to encour- age more research attention and insights on unify- ing the expression representation. As mentioned in
Jianhao Shen, Yichun Yin, Lin Li, I Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu. 2021. Generate & rank: A task framework for math word prob- lems. In Findings of EMNLP, pages 2269-2279.

---

Table 4: Results and comparisons with preliminary re-
2019). MATH (Hendrycks et al., 2021), AS- Div (Miao et al., 2020), and SVAMP (Patel et al., 2021). To comprehensively evaluate the effective- ness of our proposed MWP-NAS, we also con- duct complementary experiments on more datasets, MathQA and SVAMP in specific". The results are reported in Table 5, from which we observe that the comparison results across different meth- ods show inconsistency on two datasets, even hibiting sigmificant contradictions for some compar- isons. For example, GROUP-ATI (Li al., 201 9a) outperforms Graph2Tree (Zhang et al., 2020) on MathQA by 0.9, but shows remarkable perfor- mance decrease, i.e., 15.0, on SVAMP. We also observe that our MWP-NAS outperforms almost all the baselines on these datasets, except for the Graph2Tree (Zhang et al., 2020) on SVAMP. This may come from the graph representations, e.g., Quantity Cell Graph and Quantity Comparison Graph, which are good at handling the problem variants in SVAMP dataset. When we go beyond this aspect, comparing with the strong baseline De- duetReasoner (Jie et al., 2022), our MWP-NAS substantially outperforms it and achieves the state- of thethh-art performance, which further verifies the effectiveness of out proposed MWP-NAS.
fined MTree (RefMTreee.
Model
Val Acc
MTree Acc
MTree IoU
SUMC-Solver 82.9
82.4
87.26
-ReiMTree
79.8
79.5
86.99
MWP-NAS 84.8
83.8
89.73
-RefMTree 84.3
83.4
89.71
Section 3.1, in the design of original MTTree (Wang et al., 2022), there needs to introduce additional indicator denote the form of numerical values, {n, ;/ n, - } in specific. We observe that it is possible omit the form indicator with the new operators +/). Rethinking the operation for x-, it calculates the opposite value of the prod- uct of the operands. Suppose we have only one operand n for x the result could be -- n. Similar transformations also can be implemented to obtain with +/. and with the sequential combina- tion of +/ and x- With such modification, the MTTree in Figure only needs to revise "-40" to "40" and add an internal node between 40 and the root. We also conduet several preliminary experi- ments with the refined MTree (we call RefMTree) on Math23K. The experimental settings are the same as the ones in Section 5.2, except for remov- ing the type classifier of MWP-NAS and the form code of SUMC-Solver. From the results shown in Table 4, we observe that all the performances are decreased, which suggests that such simple and naive modification for MTree refinement is far from good enough. We also note that our MWP-NAS decreases slightly, while the SUMC-Solver gets sig- nificant performance drop. We assume it may be that MTree refinement changed the format of path codes of SUMC, and led to inferior results. Though the additional internal nodes deepen the MTree and make our MWP-NAS to predict the MTTree more difficult, MWP-NAS still benefits from removing the type classifier and yields slightly lower results. summary, the simple refinement of MTree does not work well for both MTree- based methods, and encourages us to keep the exploration in future.
Table 5: Performance comparison with baselines two extra datasets, MathQA and SVAMP. A means the results referred from (Li et al., 2022), and & indicates the results referred from (Jie et al., 2022), otherwise from the original paper.
Model
MathQA
SVAMP
Seq2Seq (Wang
2017)
T-RNN (Wang al., 2019)
GROUP-ATT (Li et al., 2019a)
70.4
21.5*
GTS (Xie and Sun, 2019)
30.8
Graph2Tree (Zhang al., 2020)
69.5
36,5
NeuralSymbolic (Qin
2021)
HMS (Liin al., 2021)
NUMS2T (Wu al., 2021)
BERT-Tree (Li et al., 2022)
73.8
32.4
SAU-Solver (Qin et al., 2020)
UniLM (Dong et al., 2019)
DeductRcasoner (Jic et al., 2022)
80.6
35,3
SUMC-Solver ( (Wang et al., 2022)
AWP-NAS
81.2
35,5
A.2 Complementary Experimental Results
FFor MAWPS and SVAMP, we use BERT base as the prob- encoder.
MWP solving fundamental task for evaluating the reasoning ability of language models, there- fore there exist many benchmark datasets, such as Math23K (Wang et al., 2017). MAWPS (Koncel- Kedziorski et al., 2016), MathQA (Amini et al.,
'We do not include the results on these two datasets in the main part because only a few baselines conducted experiments on them. Similarly, we did not conduct experiments on the other datasets due to the less baselines on them. Therefore we only regard these results as complementary to the main results reported in Table 1.


Reviews:
[{"review_id": "80dtVi4HYG", "reviewer": "Reviewer_fZF3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the endeavors and efforts put into reviewing our paper. We're grateful to all the valuable reviews. In this rebuttal, we detailly answer the reviewer’s questions one-by-one, and hope the responses clear up all the concerns.\n\n**Q1:** The performance of the proposed approach on other datasets is missing from the paper.\n>**Response:** Thanks for the suggestive review. As most representative baselines conduct experiments on Math23k and MAWPS, such as seq2seq, Graph2Tree, GTS, DeductReasoner, and SUMC-Solver, we choose these two benchmarks to verify the effectiveness of our approach for fair comparison. As the reviewer mentioned, we also note some methods conduct experiments on other datasets, e.g., MathQA and SVAMP, and will conduct experiments on these datasets for comprehensive comparison. \n\n\n\n**Q2:** Comparison of performance with LLMs is missing from the paper.\n>**Response:** Thanks for the suggestion. LLMs demonstrate superb generation and emergent ability in many NLP tasks, as well as MWP solving. We compare with LLMs referring the results of famous Chain-of Thought paper on MAWPS dataset. The results are as below:\n\\begin{matrix}\n\\hline\n\\text{UL2 20B} & \\text{LaMDA 137B} & \\text{GPT-3 175B} & \\text{Codex} & \\text{PaLM 540B} & \\text{ChatGPT} & \\text{MWP-NAS (121M)} \\\\\\\\ \\hline\n19.1 & 57.9 & 87.1 & 92.6 & 93.3 & 95.7 & 86.7 \\\\\\\\ \\hline\n\\end{matrix}\n\n>We note that with huge number of parameters (various from tens billion to hundreds billion), the LLMs demonstrate different performance. GPT-3, Codex, PaLM, and ChatGPT with strong emergent ability could solve the MWPs with high accuracy, while smaller models such as UL2 and LaMDA cannot perform well in the Chain-of-Thought scenario (with few-shot learning). Our MWP-NAS only has 121M parameters (including 110M of BERT-base) could achieve considerable performance. We must admit that the LLMs don’t need additional training for different datasets, while our small MWP-NAS needs such training. To make our MTree structure benefit from the LLMs, we will try to integrate the MTree structure with LLMs prompt learning to explore more potentials in future work. Thanks again for your constructive and valuable reviews.\n\n**Q3:** Qualitative analysis of the unified tree structure is important but it is missing from the paper.\n>**Response:** Thanks for the suggestion. We present some qualitative analyses for better evaluation based on MTree in Figure 5, in Page 8, which verifies the superiority of evaluation based on unified tree structure (here we have not directly exhibited the case with tree structure). We will visualize some cases with tree structure of predicted MTree in the revision to qualitatively analyze the unified tree structure, and explore more potential points with failure case study.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Ambiguous but correct expression tree for MWP is a very real problem and affect the performance of MWP solvers, so a the idea of using a unified tree structure for expressions is an interesting idea.\n2. The proposed technique achieves best performance on Math23K dataset.\n3. Overall the paper is well written and clear.", "reasons_to_reject": "- The performance of the proposed approach on other datasets is missing from the paper.\n- Comparison of performance with LLMs is missing from the paper.\n- Qualitative analysis of the unified tree structure is important but it is missing from the paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a unified tree structure for the expression representation where the elements are permutable and identical for all expression variants while decoding solution for a MWP. They propose a technique to parse the MWP and generate an expression for the MWP in unified tree structure.\n\nThe main contributions of the paper are as follows:\n1. Authors aim to solve and important problem of representing an ambiguous correct expression for a MWP.\n2. Authors propose a non-autoregressive tree decoding strategy which does goal driven decoding using the unified tree structure.\n3. Authors also propose two new metrics for evaluation the expression tree MtreeAcc and MtreeIoU", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2mDA5sH6SW", "reviewer": "Reviewer_jd6M", "summary": "", "questions": "Q1: Lines #322-#323: What is the effect of the special token Nb? Is Nb a predefined constant (Lines #399-#401)? Do all MWPs share the same Nb? How to decide the branch number in the inference? Related details are unclear.\n\nQ3: Line #353: Does cross-goal attention only consider other subgoals under the same parent goal? Or will it also consider the parent subgoal, grandparent subgoal, even the main goal?\n\nQ3: Lines #453-#460: Although the MTree IoU assesses the partial correctness of the expression, are there other contributions to MWP solving? For example, can it be used to improve the model performance?\n\nQ4: Line #465-#467: By Eq(9), the proposed IoU formula can't directly distinguish two different paths with duplicated numbers. Thus, the assessment may be overestimated. How to handle such cases?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the endeavors and efforts put into reviewing our paper. We're grateful to all the valuable reviews. In this rebuttal, we detailly answer the reviewer’s questions one-by-one, and hope the responses clear up all the concerns.\n\n**Q1:** Lines #322-#323: What is the effect of the special token Nb? Is Nb a predefined constant (Lines #399-#401)? Do all MWPs share the same Nb? How to decide the branch number in the inference? Related details are unclear.\n\n>**Response:** We apologize for the ambiguous description. The special token $N_b$ is somehow like the **“<END>”** token in traditional text generation task, which is shared by all the MWPs. **$N_b$ is not a predefined constant**, it is a token that is appended at the end of branches for the same goal/subgoal. It does not directly present the number of branches of a goal, but it can indicate the paragraph length indirectly, because all the branches must be setting before it. As mentioned in L396-402, during the inference, we set a max number, 8 in this work, for parallel decoding of branches, and ignore the predictions after the generated $N_b$ token. We will make description more detail and clear in the revision. Thanks for your suggestive reviews.\n\n\n**Q2:** Line #353: Does cross-goal attention only consider other subgoals under the same parent goal? Or will it also consider the parent subgoal, grandparent subgoal, even the main goal?\n\n>**Response:** Thanks for your comments. As shown in Figure 3(b), the cross-goal attention considers **all the subgoals in the same depth** of the MTree, which could belong to different parent goals. The cross-goal attention makes the interactions between goals available and avoids wrong decomposition. For the example in Figure 3(b), there exist two '$\\times$' goals in the second layer, which could share very similar latent representations because they belong to the same '$\\times$' token class. If we decompose them independently, it may result in the same child nodes and induce repetition. When we employ cross-goal attention, the decomposition process of left '$\\times$' would peek at the information of the right '$\\times$' and interact with it. Then it could avoid to generate the same child nodes with the right '$\\times$', and vice versa for the decomposition of right '$\\times$'.  In this paper, we did not consider the parent subgoal, grandparent subgoal, and the main goal here, because the motivation of cross-goal attention is to make the interactions across goals available. **The interactions across parent subgoals could be explored during the grandparent subgoal decomposition**, and so on. Hope our response is clear and helpful.\n\n**Q3:** Lines #453-#460: Although the MTree IoU assesses the partial correctness of the expression, are there other contributions to MWP solving? For example, can it be used to improve the model performance?\n>**Response:** Thanks for your suggestive review. In this work, we only employ MTree IoU as a metric to assess the partial correctness of the expression. We believe it could be used to improve the model performance, such as **adding an optimization goal based on MTree IoU** to improve the MTree IoU during training, it could make the discrete/binary score of traditional accuracy of single expression to a continuous score that is easy to optimize.\n\n**Q4:** Line #465-#467: By Eq(9), the proposed IoU formula can't directly distinguish two different paths with duplicated numbers. Thus, the assessment may be overestimated. How to handle such cases?\n>**Response:** Thanks for your review. Yes, the vanilla IoU formula in Eq(9) can't directly distinguish two different paths with duplicated numbers. To address this issue, **as described in L465-467,** we treat the duplicated paths as different paths. **In our code implementation, we actually use ‘List’ as the type of $P_p$ and $P_g$ in Eq(9), rather than ‘Set’ that may ignore the duplicated paths.** We apologize for the unclear description, and will add the implementation details in the revision.\n\n**Clarifications of Reasons to Reject:**\n\n**Q5:** One of the motivations for introducing MTree is to address the challenge of learning efficiency issues for handling multiple solution variants. However, the authors didn’t provide related experimental results to support this claim.\n>**Response:** Thanks for the review. We assume the reviewer means the claim about **‘effectively’ in L15 in Abstract.** First we apologize for the unclear description. Actually we want to claim that it is hard to learn the ‘mapping function’ between input and output domain, because a problem might has multiple variants for the solution expression. This will push the model to learn multiple different outputs for the same input, which violates the **exactly one output rule** in the definition of function [1]. So it will make the model hard to optimize and might lead to inferior performance, which we claimed ```“making it hard for the model to learn the mapping function between the input and output spaces effectively”``` in Abstract. To verify the effectiveness of mapping function learning, we think the performance could be the straightforward way, and will try other perspecitves to evaluate the effectiveness. Hope this makes our motivation clearer.\n>>[1] The definition of function from Wikipedia: a function from a set X to a set Y assigns to each element of X **exactly one element** of Y. https://en.wikipedia.org/wiki/Function_(mathematics)\n\n**Other issues:**\n\n**Q6:** Lines #031: What is “xxx”? It is suggested to revise the statement if “xxx” represents “to be released upon acceptance.”\n>**Response:** Thanks for the suggestion. We use ‘xxx’ as placeholder for code link that will be available after double-blind review. We apologize for the informal presentation.\n\n**Q7:** Line #299: Is E_p or E_s(in Eq(1)) ?\n>**Response:** We apologize for the carelessness. It should be $E_s$ in L299.\n\n**Q8:** Lack of assessments for publicly challenging MWP datasets (such as Math, SVAMP, or ASDIV)\n>**Response:** Thanks for your constructive review. As many baselines, such as Seq2Seq, GTS, Graph2Tree, and DeductReasoner, conduct experiments on Math23k and MAWPS, we also choose them for fair comparison. For the benchmarks you mentioned, we will try to conduct experiments on them and compare with the baselines in the revision.\n\nFor other minor issues that are not listed above, e.g., missing references and notation ambiguous, we will address them in the revision, and will conduct thorough proof-reading to make the paper clearer. Thanks again for the valuable reviews. Hope the above responses are clear and helpful, and could address your concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Unlike previous work on goal-driven tree decoders, the proposed method is like the human's problem-solving strategy, that is, to generate subgoals by the expansion from top (main-goal) to bottom (subgoals) (rather than from left to right). Experimental results support the claim of the proposed cross-goal attention mechanism to enhance the system performance. This work and experimental results will benefit future research on MWP solving.", "reasons_to_reject": "(1) One of the motivations for introducing MTree is to address the challenge of learning efficiency issues for handling multiple solution variants. However, the authors didn’t provide related experimental results to support this claim.\n\n(2) Lack of assessments for publicly challenging MWP datasets (such as Math, SVAMP, or ASDIV).", "questions_to_authors": "Q1: Lines #322-#323: What is the effect of the special token Nb? Is Nb a predefined constant (Lines #399-#401)? Do all MWPs share the same Nb? How to decide the branch number in the inference? Related details are unclear.\n\nQ3: Line #353: Does cross-goal attention only consider other subgoals under the same parent goal? Or will it also consider the parent subgoal, grandparent subgoal, even the main goal?\n\nQ3: Lines #453-#460: Although the MTree IoU assesses the partial correctness of the expression, are there other contributions to MWP solving? For example, can it be used to improve the model performance?\n\nQ4: Line #465-#467: By Eq(9), the proposed IoU formula can't directly distinguish two different paths with duplicated numbers. Thus, the assessment may be overestimated. How to handle such cases?", "paper_topic_and_main_contributions": "This paper presents a non-autoregressive approach to deducing the multiple solution variants (i.e., valid solutions for the same MWP but formulated as different expressions) based on a unified tree (in which the unified tree presents a solution expression where nodes in a tree are permutable are identical for all expression variants). The authors also propose a cross-goal attention mechanism to leverage the information across goals to enhance the model's robustness. Furthermore, the authors introduce a partial-tree evaluation metric to assess the capability of an MWP solver. Experimental results demonstrate the effectiveness of the proposed approaches.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Measuring Mathematical Problem Solving With the MATH Dataset, Hendrycks et al., NeurIPS 2021\n\nA Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers, Miao et al. ACL-2020\n\nAre NLP Models really able to Solve Simple Math Word Problems? Patel et al., ACL-2021", "Typos_Grammar_Style_and_Presentation_Improvements": "(1)\tLines #031: What is “xxx”? It is suggested to revise the statement if “xxx” represents “to be released upon acceptance.”\n\n(2): Line #299: Is E_p or E_s(in Eq(1)) ?\n\n(3) Table 1: The same notations but are different data splitting methods for ReduceReasoner models. It is suggested to use different notations to distinguish them."}}, {"review_id": "amZZSpGV3V", "reviewer": "Reviewer_P6mx", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the endeavors and efforts put into reviewing our paper. We're grateful to all the valuable reviews. In this rebuttal, we detailly answer the reviewer’s questions one-by-one, and hope the responses clear up all the concerns.\n\n\n**Q1:** It is not clear how to handle the order of the prediction in each sub-goal, which seems to be crucial for the result. More specifically, in the left subfigure of Figure 2, why is p1 connected to \"x\" and p3 connected to \"N4\", can we change their order?\n>**Response:** Thanks for the review. As described in L386-393, we define a **pseudo-order for the subgoals** to align positions and subgoals during training. Specifically, in this work we make the operators are ahead and followed by the constants and operands. Following such pseudo-order rule, $p_1$ connects to ‘$\\times$’ and $p_3$ connects to ‘$N_4$’. With certain rule, the order cannot be changed, but we can also define other rules for the pseudo-order.\n\n**Q2:** The writing of Section 3.3 and Section 3.4 could be improved. You may first introduce what p1 to p4 means and how they are connected to outputs before introducing the details. In lines around 339 where you introduce the positional embeddings, the readers don't know what are there in each position. The sentence between lines 400 to 402 is hard to understand. E_s in Equation (1) is introduced as E_p in line 299.\n>**Response:** We apologize for the unclear description. We will improve these parts as the reviewer suggested, and conduct thorough proof-reading over the whole paper. For $E_s$ and $E_p$ in Eq(1) and L299, it should be $E_s$ in L299, we will revise this in the revision. Thanks for the suggestive and valuable comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The method proposed by this paper is interesting and sound. It combines goal-driving, MTree (a unified, unique representation of math equations), and cross-attention methods in harmony.\n2. The proposed evaluation method is a more fine-grained metric than the expression and result accuracy.", "reasons_to_reject": "A. It is not clear how to handle the order of the prediction in each sub-goal, which seems to be crucial for the result. More specifically, in the left subfigure of Figure 2, why is p1 connected to \"x\" and p3 connected to \"N4\", can we change their order?\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a method for math word problem-solving. It is a goal-driving method that starts from the root of the expression tree and decodes the tree layer by layer to the leaf nodes. It extends this approach by 1. predicting all child nodes in the next layer in parallel and 2. using cross-attention to make the parallel prediction aware of each other.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The writing of Section 3.3 and Section 3.4 could be improved. You may first introduce what p1 to p4 means and how they are connected to outputs before introducing the details. In lines around 339 where you introduce the positional embeddings, the readers don't know what are there in each position.\nThe sentence between lines 400 to 402 is hard to understand.\n$E_s$ in Equation (1) is introduced as $E_p$ in line 299."}}]

Author rebuttal:

Meta-review: Paper Topic And Main Contributions:
* This paper proposes a method for math word problem-solving. It is a goal-driving method that starts from the root of the expression tree and decodes the tree layer by layer to the leaf nodes. 
* It extends this approach by 1. predicting all child nodes in the next layer in parallel and 2. using cross-attention to make the parallel prediction aware of each other. 
* The authors introduce a partial-tree evaluation metric to assess the capability of an MWP solver. 
* Experimental results demonstrate the effectiveness of the proposed approaches.

Reasons to accept:
* The proposed method is interesting and sound. It combines goal-driving, MTree (a unified, unique representation of math equations), and cross-attention methods.
* The proposed evaluation methods are valuable; they are more fine-grained than expression and result accuracy.

Reasons to reject:
* The performance of the proposed approach on other datasets (such as Math, SVAMP, or ASDIV) is missing. The authords explain that the datasets they do use (Math23k and MAWPS) are used by many of the baselines. Experiments on MathQA and SVAMP will be added to the final version.
* Comparison of performance with LLMs is missing. The authors address this point in the rebuttal.
* Qualitative analysis of the unified tree structure is missing. The authors point to existing qualitiative analysis and will add further visualization.
Decision: Accept-Main