Venue: ACL 2020

# Graph-to-Tree Learning for Solving Math Word Problems

Jipeng Zhang\({}^{1,2,*}\), Lei Wang\({}^{2,*}\), Roy Ka-Wei Lee\({}^{3}\), Yi Bin\({}^{1}\), Yan Wang\({}^{4}\), Jie Shao\({}^{1,5,\#}\), Ee-Peng Lim\({}^{2}\)

\({}^{1}\)Center for Future Media, University of Electronic Science and Technology of China

\({}^{2}\)School of Information Systems, Singapore Management University

\({}^{3}\)Department of Computer Science, University of Saskatchewan

\({}^{4}\)Tencent AI Lab

\({}^{5}\)Sichuan Artificial Intelligence Research Institute zhangjipeng20@std.uestc.edu.cn, lei.wang.2019@phdcs.smu.edu.sg, roylee@cs.usask.ca, yi.bin@hotmail.com, bradenwang@tencent.com, shaojie@uestc.edu.cn, eplim@smu.edu.sg &

###### Abstract

While the recent tree-based neural models have demonstrated promising results in generating solution expression for the math word problem (MWP), most of these models do not capture the relationships and order information among the quantities well. This results in poor quantity representations and incorrect solution expressions. In this paper, we propose _Graph2Tree_, a novel deep learning architecture that combines the merits of the graph-based encoder and tree-based decoder to generate better solution expressions. Intcluded in our _Graph2Tree_ framework are two graphs, namely the _Quantity Cell Graph_ and _Quantity Comparison Graph_, which are designed to address limitations of existing methods by effectively representing the relationships and order information among the quantities in MWPs. We conduct extensive experiments on two available datasets. Our experiment results show that _Graph2Tree_ outperforms the state-of-the-art baselines on two benchmark datasets significantly. We also discuss case studies and empirically examine _Graph2Tree_'s effectiveness in translating the MWP text into solution expressions1.

Footnote 1: Code could be found at [https://github.com/2003pro/Graph2Tree](https://github.com/2003pro/Graph2Tree)

+
Footnote †: FOOTNOTE:6]Footnote 6: footnotetext: [ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote †: ENDFOOTNOTE]

+
Footnote†: footnotetext: [ENDFOOTNOTE]


\(470-348\), before associating the price attribute of $17. The existing deep learning models are not effective in capturing such relationships and order information among the quantities in MWPs, thus resulting in an inaccurate representation of the final solution expressions.

To enrich the representation of a quantity, the relationships between the descriptive words associated with a quantity need to be modeled. However, such relationships cannot be effectively modeled using recurrent models, which are commonly used in the existing MWP deep learning methods. Inspired by the concept of Quantity Schema Roy and Roth (2015) and Qset Koncel-Kedziorski et al. (2015), we design the _Quantity Cell Graph_ to associate informatively descriptive words to quantity. We first extract associated nouns, verbs, adjectives, units, and rates that describe a quantity in the MWP text. Next, we construct a graph where the extracted descriptive words are represented as neighbor nodes directly linked to a quantity. Finally, a neural network model is used to learn enriched latent representations of the quantities based on the constructed _Quantity Cell Graph_.

The loss of quantities' numerical qualities in existing MWP methods can also result in poor quantity representations. Most of the existing MWP methods often replace quantities with special symbols (e.g., "\(n_{1}\)", "\(h_{2}\)", etc.) Wang et al. (2017, 2018); Liu et al. (2019). The loss of quantities' numerical qualities could be problematic when generating solution expressions. Take the example in Table 1, without modeling the numerical qualities of quantities, an MWP method may learn a solution expression "\(384-470\)" which results in a negative number that is unlikely to occur in MWPs. To address this limitation, we introduce the _Quantity Comparison Graph_, which was inspired by a numerical machine reading comprehension model proposed by Ran et al. (2019). The intuition of _Quantity Comparison Graph_ is to retain the numerical qualities of the quantity and leverage certain heuristics to represent the relationships among quantities in MWPs such that solution expressions reflect a more realistic arithmetic order.

Besides improving the quantity representation, we also aim to improve the solution expression generative process. For longer solution expressions in MWPs, as some quantities are repeatedly used in different arithmetic sub-solution expressions, the existing methods which utilized recurrent neural networks may not be able to learn the underlying reasoning process and arithmetic order. For example, in Table 1, the quantity 348 is being used in "\(348*23\)" and "\((470-348)*17\)". To address this limitation, we propose to use a graph encoder to guide the learning of representations of quantities and a tree decoder to explicitly model the multi-stage reasoning process.

**Contribution.** In this paper, we combine the above-proposed solutions and introduce the **Graph2Tree** solver to address the existing MWPs methods' limitations. The contributions of this paper are as follows:

* We construct the **Quantity Cell Graph** and **Quantity Comparison Graph** to enrich the quantity representations by capturing relationships between quantities and their attributes and retaining the quantities' numerical qualities.
* We propose the **Graph2Tree** to improve the learning of solution expressions' generation. The _Graph2Tree_ model uses a graph transformer to learn the latent quantity representations from our proposed graphs, and a tree structure decoder to generate a solution expression tree. To the best of our knowledge, this is the first graph-to-tree model for MWPs.
* We conduct extensive experiments on two available large-scale MWPs datasets, and our results show that our proposed _Graph2Tree_ model outperforms state-of-the-art baselines on MWP task.

## 2 Problem Formulation

We denote the text of the math word problem as \(P\), where \(P\) is a sequence of word tokens and numeric values. We let \(V_{p}=\{v_{1},\cdots,v_{m}\}\) denote the word tokens in \(P\) and \(n_{P}=\{n_{1},\cdots,n_{l}\}\) denote the set of quantities in \(P\). Our goal is to map \(P\) to a valid and correct mathematical expression \(E_{p}\).

Solving MWPs requires an understanding of quantities in problem and their complex mathematical relationships. MWPs are often expressed in a linear textual sequence form, which is not ideal for learning the quantities' complex interactions. Thus, we propose to formulate the problem into graph form so that the relationships between quantities can be expressed more explicitly. The problem text \(P\) is transformed into graph \(\mathcal{G}\) by augmenting the text sequences with other structural information like dependency parsing and POS tagging 

The final mathematical expression \(E_{p}\) that we aim to construct can always be represented as a solution expression tree \(T\). \(T\) may include constant quantities, operators and quantities in \(n_{P}\). The set of constant quantities \(V_{con}\) contains some special values not appeared in text like \(\pi,1\). The set of math operators \(V_{op}\) contains \(\left\{+,-,*,/\right\}\). Overall, the target vocabulary of \(P\) can be denoted as \(V_{dec}=V_{op}\cup V_{con}\cup n_{P}\) (\(V_{dec}\) varies in different problems as \(n_{P}\) varies) . The goal of our _Graph2Tree_ model here is to estimate the conditional probability \(P(E_{p}|P)\), which can be transformed as \(P(T|\mathcal{G},V_{dec})\).

## 3 Methodology

Figure 1 shows our proposed _Graph2Tree_ framework. _Graph2Tree_ first encodes the MWP text input using BiLSTM and simultaneously constructs _Quantity Cell Graph_ and _Quantity Comparison Graph_. The output of BiLSTM, word-level representations, are used as node representations. Together with the two constructed graphs, the node representations are input into a graph transformer to learn a graph representation of the MWP. The multiGCN component of the graph transformer is modified to learn the graph representation based on the _Quantity Cell Graph_ and _Quantity Comparison Graph_. This enriches the final graph representation with quantities' relationship information and numerical qualities. Pooling is used to aggregate all nodes into a pool-based graph embedding vector as the graph transformer's output. Finally, the output graph representation and the updated node representations are used as input to a tree-structure decoder to infer the final solution expression tree.

### Graph-Based Encoder

There have been some graph-based models Sahu et al. (2019) intending to grab the complicated relations in text. The graph-based encoder in our _Graph2Tree_ framework is inspired by the graph transformer model Koncel-Kedziorski et al. (2016); Cai and Lam (2019). We first discuss the initialization of node representations based on MWPs' input problem text. Next, we introduce the construction of the _Quantity Cell Graph_ and _Quantity Comparison Graph_. Finally, we discuss the learning of graph representation using the graph transformer module.

#### 3.1.1 Node Representation Initialization

To initialize the node representations, we first learn the word-level hidden state representations of the input MWP text using a BiLSTM neural network, \(H=\left\{h_{1},\cdots,h_{N}\right\}\in R^{N\times d},N=m+l\). Here \(d\) denotes the dimension of hidden vectors, \(m\) represents the number of words, and \(l\) represents the number of quantities. The learned hidden state representations will be used as the input node representations for the graph encoder.

#### 3.1.2 Quantity Cell

We refer all quantities \(n_{P}\) and words \(V_{p}\) from the problem as nodes in the graph. Next, we define a _quantity cell_ as a subset of nodes in the graph that are associated with a quantity. Formally, each

Figure 1: Overview of the proposed model. In order to initialize representation of text \(P\), a BiLSTM is used to compute node representation \(H\). Later, after extracting Quantity Cells from text \(P\), we construct Quantity Comparison Graph and Quantity Cell Graph. With two graphs and \(H\), we use the proposed graph transformer to get the internal representation. Finally, a tree-based decoder is implemented to generate the target euqation \(E_{q}\).

 

MWP \(P\) is transformed into multiple _quantity cells_\(QC=\{Q_{1},Q_{2},\cdots,Q_{m}\}\), where \(m\) is the number of quantities in \(P\). Each quantity cell \(Q_{i}\in QC\) contains a quantity token \(\{n_{i}\}\) and the corresponding attributes \(\{v_{1i},\cdots,v_{qi}\}\). These _quantity cells_ are sub-graph representations of quantity-related information in the MWPs. Dependency parsing, constituency parsing and POS tagging implemented with _stanford corenlp toolkit_[4] are used to extract and construct the _quantity cells_.

A _quantity cell_ in an MWP \(P\) consists of the following properties:

* _Quantity._ The quantity numeric value.
* _Associated Nouns._ We consider the nouns related to the _Quantity_ in the dependency parse tree. _Associated Nouns_ are the nouns related by the _num_, _number_ and _prep_of_ relations.
* _Associated Adjectives. Associated Adjectives are the adjectives related to _Quantity_ or _Associated Nouns_ with the _amod_ relation, which is detected by the dependency parser.
* _Associated Verbs._ For each _Quantity_, we detect the related verbs, _Associated Verbs_, according to _nsubj_ and _dobj_ relations.
* _Units and Rates._ We detect the nouns related to _Associated Nouns_ by _prep_of_ as the _Unit_. The nouns related _Associated Nouns_ which own the key words such as "each", "every" and "per" are regarded as _Rates_.

If the _quantity cell_ detection process does not grab any attributes, we will use a window centered on _Quantity_ to select neighboring words as the attributes of the _Quantity_. An example of the _quantity cell_ is illustrated in the left part of Figure 1.

#### 3.1.3 Quantity Graph Construction

From the quantity cells, we construct two graphs: _Quantity Cell Graph_ and _Quantity Comparison Graph_. The goal of the _Quantity Cell Graph_ is to associate informative descriptive words to quantity so as to enrich the quantity's representation. Similarly, the goal of the _Quantity Comparison Graph_ is to retain the numerical qualities of the quantity and leverage heuristics to improve representations of the relationships among quantities. Formally, we define the construction of two graphs as follow:

* **Quantity Cell Graph**\(G_{qcell}\). For each _Quantity Cell_\(Q_{i}=\{n_{i}\}\cup\{v_{1i},\cdots,v_{ni}\}\), the undirected edge \(e_{ij}\) between \(n_{i}\) and each \(v_{j}\in\{v_{1i},\cdots,v_{qi}\}\) will be added to the graph \(G_{qcell}\).
* **Quantity Comparison Graph**\(G_{qcomp}\). For two quantity nodes \(n_{i},n_{j}\in n_{P}\), a directed edge \(e_{ij}=(n_{i},n_{j})\) pointing from \(n_{i}\) to \(n_{j}\) will be added to the graph \(G_{qcomp}\) if \(n_{i}>n_{j}\). This heuristic constraint can prevent the subtracting a larger number from a smaller number, which results in a negative number.

We represent the two graphs using adjacency matrices. For graph \(G\), an adjacency matrix \(A\in R^{N\times N}\) is first initialized. If there exists an edge between the \(i\)-th and \(j\)-th nodes, we need to assign value \(1\) to corresponding position of the adjacency matrix \((i,j,A_{i,j})\) for this edge. Otherwise, \(0\) would be assigned. Thus, we compute the adjacency matrix \(A_{qcomp}\) for graph \(G_{qcomp}\) and \(A_{qcell}\) for \(G_{qcell}\).

#### 3.1.4 Graph Transformer

The inputs to the graph transfer module are adjacency matrices of multiple graphs \(\{A_{k}\}_{k=1}^{K},A_{k}\in\{A_{qcomp},A_{qcell}\}\) and initial node embeddings \(H\), where \(K\) is the number of graphs and each \(A_{k}\in R^{N\times N}\) is the adjacency matrix for \(k\)-th graph. \(K\) graphs are used as we adopt a multi-head structure in our model and they are split evenly between _Quantity Cell Graphs_ and _Quantity Comparison Graph_.

The graph transformer first utilizes graph convolution networks (GCNs) [56] to learn the graph node features. For multiple graphs, we use a \(K\)-head graph convolution setup. This is similar to the transformer model proposed in Vaswani et al. (2017), where \(K\) separate graph convolution networks are used and concatenated before a residual connection is applied.

Specifically, a single GCN has its parameter \(W_{gk}\in R^{d\times d_{k}}\), where \(d_{k}=d/K\). Given an adjacency matrix \(A_{k}\) representing graph structure and a feature matrix \(X\) (in the beginning, \(X\) is set as \(H\)) meaning the input feature for all nodes, we define learning of GCN as follow:

\[GCN(A_{k},X)=GConv_{2}(A_{k},GConv_{1}(A_{k},X)) \tag{1}\]

Here, the GCN contains 2 different graph convolution operations:

\[GConv(A_{k},X)=relu(A_{k}X^{T}W_{gk}) \tag{2}\]

For each graphs \(\{A_{k}\}_{k=1}^{K}\), we perform learning of GCN in parallel, yielding \(d_{k}\)-dimensional output values. The output values are concatenated and projected, resulting in the final values:

\[Z=\mathop{\parallel}\limits_{k=1}^{K}GCN(A_{k},H)\] 

Here, \(\|\) denotes the concatenation of the \(K\) GCN heads.

Graph transformer then augments this \(K\)-head graph convolution network with a feed-forward network, layer-norm layer, and residual connection:

\[\hat{Z}=Z+LayerNorm(Z) \tag{4}\] \[\overline{Z}=\hat{Z}+LayerNorm(FFN(\hat{Z})) \tag{5}\]

here, \(FFN(x)\) is a two-layer feed-forward network with a \(relu\) function between layers:

\[FFN(x)=max(0,xW_{f1}+b_{f1})W_{f2}+b_{f2} \tag{6}\]

The resulting node representations \(\overline{Z}\) represent quantities, entities and relations. In order to learn the global context graph representation, we apply the element-wise min-pooling operation on all learned node representations. Finally, the global feature is fed into a fully connected neural network (FC) to generate the graph representation \(z_{g}\):

\[z_{g}=FC(MinPool(\overline{Z})) \tag{7}\]

### Tree-Based Decoder

Inspired by the the Goal-driven Tree Structure (GTS) [38], we build a tree-based decoder to construct the solution expressions. We set the quantity nodes to be the leaf nodes and each operator node must have two child nodes. As such, the specialized tree decoder generates an equation following the pre-order traversal ordering. As part of the tree construction process, the centermost operator is first produced, followed by the left child node. This process is repeated until the leaf node is produced. Subsequently, we generate the right child nodes recursively.

#### 3.2.1 Tree Initialization

To start the above mentioned tree generation process, our model initializes the root node vector \(q_{root}\) according to the global context graph representation \(z_{g}\). For each token \(y\) in the target vocabulary \(V_{dec}\) of \(P\), the representation for a certain token \(e(y|P)\) is defined as:

\[\left\{\begin{array}{ll}e_{(y,op)}&if~{}y\in V_{op}\\ e_{(y,con)}&if~{}y\in V_{con}\\ \overline{z}^{p}_{loc(y,P)}&if~{}y\in n_{P}\end{array}\right. \tag{8}\]

The expression trees in our decoder contain three kinds of nodes: operators, constant quantities, and quantities that appeared in \(P\). Constant quantities and quantities in \(n_{P}\) are always set to be in leaf nodes position. Operators will always take up the positions of the non-leaf nodes. The quantities' representations in \(n_{P}\) are dependent on certain MWPs, i.e., \(y\) will take the corresponding \(\overline{z}^{p}_{loc(y,P)}\) from \(\overline{Z}\). The representations of operators and constant quantities are independent, i.e., their representations are obtained by 2 independent embedding matrices \(M_{op}\) and \(M_{con}\).

#### 3.2.2 Pre-Order Tree Generation

We adopt the pre-order traversal manner to construct the expression tree:

* **Step 1.** The generation starts with a derivation tree with only a root node \(q_{root}\). We use attention module of GTS to encode the node embedding \(\overline{Z}\) into global graph vector \(G_{c}\): \[G_{c}=\mathrm{GTS}-\mathrm{Attention}(q_{root},\overline{Z})\] (9)
* **Step 2.** This tree decoder applies left sub-node generation module to the derivation in a top-down manner, generating new left child node \(q_{l}\) conditioned on the parent node \(q_{p}\) and global graph \(G_{c}\). Note that the token \(\hat{y}\) is predicted when generating the new node: \[\begin{split} q_{l}=\mathrm{GTS}-\mathrm{Left}(q_{p},G_{c})\\ \hat{y}=\mathrm{GTS}-\mathrm{Predict}(q_{l},G_{c})\end{split}\] (10) If the generated \(\hat{y}\) is an operator, two empty child node positions are created and we will keep executing **Step 2**. This step works like decomposing the whole goal into multi-stage reasoning. If the generated \(\hat{y}\) is a quantity (constant or from \(n_{P}\)), we will get into **Step 3**.
* **Step 3.** The tree decoder switches to use the right sub-node generation module and populate the empty right node position. At every decoding step, we use the left child node \(q_{l}\), global graph vector \(G_{c}\) and a sub-tree embedding \(t_{l}\) as the input to the right generation module and generate the right child node \(q_{r}\) and the corresponding token \(\hat{y}_{r}\): \[\begin{split} q_{r}=\mathrm{GTS}-\mathrm{Right}(q_{l},G_{c},t_{ l})\\ \hat{y}_{r}=\mathrm{GTS}-\mathrm{Predict}(q_{r},G_{c})\end{split}\] (11) The addition of the sub-tree embedding works similarly to incorporating a sub-tree copying mechanism. The additional sub-tree embedding \(t_{l}\) is computed by using sub-tree embedding component of GTS: \[t_{l}=\mathrm{GTS}-\mathrm{SubTree}(\hat{y}_{l},q_{l})\] (12) 

If \(\hat{y}_{r}\) is an operator, the next step should go back to **Step 2**. If \(\hat{y}_{r}\) is a quantity, we will get into **Step 4**.
* **Step 4.** The model switches to backtracking to find the new empty right node position. If the model cannot find the new empty right node position, the generation is completed. If the empty right node position still exists, go back to **Step 2**.

### Model Learning

For each problem-tree expression example, \((p,T)\), the loss function \(L(T,P)\) is defined as the a sum of the negative log-likelihoods of probabilities for predicting \(t\)-node token \(y_{t}\). Formally, our training goal is to minimize the following loss function:

\[L(T,P)=\sum^{E}_{t=1}-\mathrm{log}prob(y_{t}|q_{t},G_{c},P) \tag{13}\]

where \(q_{t}\) is the goal vector, \(G_{c}\) is the global graph context, \(E\) is the number of tokens in \(T\), and \(prob\) is computed by distribution computation function in GTS.

## 4 Experiment

In this section, we compare our proposed _Graph2Tree_ model with state-of-the-art baselines. We also conduct ablation study and analysis to investigate the effectiveness of various components of our model.

**Datasets.** Two commonly-used MWP datasets are used in our experiments: MAWPS Koncelk et al. (2016) with 2,373 problems and Math23K Wang et al. (2017) with 23,162 problems.

**Baselines.** We compare _Graph2Tree_ to an extensive set of baselines and state-of-the-art models: **DNS**Wang et al. (2017) uses a vanilla seq2seq model to generate expressions. **Math-EN**Wang et al. (2018) benefits from an equation normalization to reduce target space. **T-RNN**Wang et al. (2019) applies recursive neural networks over predicted tree-structure templates. **S-Aligned**Chiang and Chen (2019) designs the decoder with a stack to track the semantic meanings of operands. **GROUPATT**Li et al. (2019) borrows the idea of multihead attentions from Transformer Vaswani et al. (2017). **AST-Dec**Liu et al. (2019) creates an expression tree with a tree LSTM decoder. **STS**Xie and Sun (2019) develops a tree structured neural networks in a goal-driven manner to generate expression trees. **IRE**Sahu et al. (2019) is another baseline that was first proposed in relation extraction and has something in common with our method.

**Implementation Details and Evaluation Metric.** In the _Graph2Tree_ model, we use a word embedding (not pre-trained) with \(128\) units, a one layer graph transformer with \(4\) GCNs, each of which has the dimension of the hidden state set to \(128\). The dimensions of the hidden state for all the other layers are set to \(512\). Our model is trained for \(80\) epochs. Mini-batch size and dropout rate are set to \(64\) and \(0.5\), respectively. For optimizer, we use Adam with learning rate set to \(0.001\), \(\beta_{1}=0.94\) and \(\beta_{2}=0.99\), and the learning rate will be halved every \(20\) epochs. Also, we use a beam size of \(5\) in beam search.

For the Math23K dataset, some methods are evaluated using 5-fold cross-validation, expressed in "Math23K*", and others are evaluated using the available test set (expressed as "Math23K"). We evaluate _Graph2Tree_ on both settings. For the MAVPS dataset, the models are evaluated with 5-fold cross-validation. Following previous works, we use solution accuracy as the evaluation metric.

### Overall Results

Table 2 shows the solution accuracy of _Graph2Tree_ and various baselines. We observe that _Graph2Tree_ outperforms all baselines in the two MWP datasets. As the code for GTS is made available2, we implemented GTS and tested it on all dataset settings. We also statistically test the improvement of _Graph2Tree_ over the strongest baseline (i.e., GTS) and found that the improvement to be significant at 0.01 level using paired t-test. The superior performance of _Graph2Tree_ demonstrates

\begin{table}
\begin{tabular}{l|c|c|c}  & **MAWPS** & **Math23K** & **Math23K*** \\ \hline DNS & \(59.5\) & - & \(58.1\) \\ Math-EN & \(69.2\) & \(66.7\) & - \\ T-RNN & \(66.8\) & \(66.9\) & - \\ S-Aligned & - & - & \(65.8\) \\ GROUP-ATT & \(76.1\) & \(69.5\) & \(66.9\) \\ AST-Dec & - & \(69.0\) & - \\ GTS & \(82.6\) & \(75.6\) & \(74.3\) \\ IRE & - & \(76.7\) & - \\ Graph2Tree & **83.7** & **77.4** & **75.5** \\ \end{tabular}
\end{table}
Table 2: Solution accuracy of _Graph2Tree_ and various baselines. Note that Math23K denotes results on public test set and Math23K* denotes 5-fold cross-validation.

 

the importance of enriching quantity's representations in handling the MWP task.

### Ablation Study and Parameter Analysis

To understand the effects of the various components and hyperparameters in our _Graph2Tree_ model, we conduct ablation studies and parameter analysis on the Math23K dataset.

#### 4.2.1 Effect of Quantity Graph

We investigate the effects of _Quantity Cell Graph_ and _Quantity Comparison Graph_ in our model. The results of our ablation study are shown in Table 3. We find that the _Graph2Tree_ with both _Quantity Cell Graph_ and _Quantity Comparison Graph_ performs the best. We also observe that having either _Quantity Cell Graph_ and _Quantity Comparison Graph_ still outperforms the implementation without either graph (i.e., full-connected graph). More interestingly, we also noted that enriching the quantity representation with either graph would also outperform the baseline GTS model in this task, suggesting the importance of quantity representation in MWP task. From this study, we also infer that improving quantity representation, modeling the relationships among quantities, and retaining their numerical qualities help to achieve better results for the MWP task. Also, if two types of graphs are merged into an integrated graph, the performance drops. We postulate that a possible reason for the inferior performance may be due to the noise introduced by the integration of multiple graphs.

#### 4.2.2 Effect of Graph Number

The number of GCNs is a tuneable hyperparameter in our _Graph2Tree_ model. Thus, we investigate the effect of the number of GCNs on our model's performance. We varied the number of GCNs from 2, 4, 8. Note that even numbers are used as the GCNs are split evenly to model the _Quantity Cell Graph_ and _Quantity Comparison Graph_. Table 4 shows the study's results. We observe that the 4-GCN version achieves the best performance. A potential reason could be due to the optimal capacity of information aggregation is achieved using 4 GCNs over the two quantity graphs.

#### 4.2.3 Impact of Length of Expression

To investigate how well our _Graph2Tree_ model performs with the increasing expression complexity as compared to state-of-the-art models using explicit tree decoders, we analyze the increasing number of operators in the test set. From the results shown in Table 5, we note that:

(1) Our proposed _Graph2tree_ outperforms the other two models in most cases except that the number of operators equals to 5. In other cases with less than 5 operators, our model shown statistically significant improvements over other two models.

(2) All the models' performances follow an accuracy descending pattern when the length of expression becomes longer. This is intuitive as longer expressions often associate with more complex questions that are more difficult to solve and have fewer data for training.

#### 4.2.4 Impact of Numerical Comparison

One of the primary goals of our _Graph2Tree_ model is to address the situation where the wrong arithmetic order leads to incorrect solution expression generation. We evaluate this aspect of our model by investigating how _Graph2Tree_ has improved the arithmetic order errors. We first retrieve the MWPs with incorrectly predicted expressions. As we are interested in arithmetic order errors, we check that the incorrectly predicted expressions' length is equal to their corresponding ground truth

\begin{table}
\begin{tabular}{c|c}  & **Math23K** \\ \hline w/ 2 GCN & \(76.7\) \\ w/ 4 GCN & \(77.4\) \\ w/ 8 GCN & \(76.9\) \\ \end{tabular}
\end{table}
Table 4: Solution accuracy with varying number of GCNs.

\begin{table}
\begin{tabular}{l|c}  & **Math23K** \\ \hline Graph2Tree & \(77.4\) \\ \hline only Quantity Cell Graph & \(76.8\) \\ only Quantity Comparison Graph & \(76.9\) \\ \hline only Full-Connected Graph & \(75.3\) \\ \hline merge two graphs as single one & \(76.4\) \\ \end{tabular}
\end{table}
Table 3: Solution accuracy with various graph configurations in _Graph2Tree_.

\begin{table}
\begin{tabular}{c|c|c|c|c}  & Pro (\%) & AST-Dec (\%) & GTS (\%) & Our (\%) \\ \hline \(1\) & \(17.3\) & \(82.7\) & \(84.9\) & **85.5** \\ \(2\) & \(52.2\) & \(74.5\) & \(80.6\) & **83.7** \\ \(3\) & \(19.1\) & \(59.9\) & \(70.7\) & **71.7** \\ \(4\) & \(6.6\) & \(42.4\) & \(50.0\) & **51.5** \\ \(5\) & \(3.4\) & **44.1** & \(38.2\) & \(38.2\) \\ \(6\) & \(0.9\) & **55.6** & \(44.4\) & **55.6** \\ \end{tabular}
\end{table}
Table 5: Accuracy for increasing length of templates. #_Op_ is the number of operators in expressions. _Pro_ denotes the proportion of MWPs for different expression lengths.

 

expressions' length. In total, we retrieved 103 incorrect predicted expressions for _Graph2Tree_ and 119 for GTS. Next, we manually count the number of incorrectly predicted expression attributed to arithmetic order error among the initially retrieve set. We found that \(Graph2Tree\) has generated 7 expressions with arithmetic order error, while GTS has generated 27 arithmetic order error expressions. This suggests that \(Graph2Tree\) is able to significantly improve the arithmetic order in MWP task.

### Case Study

Finally, we perform a case study on the solution expressions generated by GTS and _Graph2Tree_. Selected case studies are shown in Table 6. In Case 1, there are essential words, i.e., "each," "group," and "students" around the quantity "15", and "students" around the quantity "76". However, GTS predicts operator "\(+\)" between these two quantities with obviously different units as GTS is unable to model quantity representation effectively using BiLSTM. For the second case, we observe that GTS gives a wrong prediction "\(3-5\)" as GTS does not model quantities' numerical qualities. For the last case, this MWP requires models to have the ability to handle situation where quantities are repeatedly and frequently used. _Graph2Tree_ is able to handle this situation better than the GTS model as our model encodes the MWP in richer graph representation. The three case studies demonstrate _Graph2Tree_ model strengths in generating more accurate and realistic solution expressions for MWPs.

Besides, further analysis is performed on error cases. We found that our model, like other baselines, performed poorly in solving MWPs with long solution expressions. Answering these MWPs requires complex reasoning which opens the possibility for future works.

## 5 Related Work

### Math Word Problems Solving

The earlier works on math word problems (MWPs) are mainly tested on small-scale datasets. These works can be broadly divided into statistical machine learning based [48; 16; 17; 18] and semantic parsing based [26; 15; 16; 17; 18].

Recently, deep learning based models have become a new trend in solving math word problems. wang2017deep applied a vanilla seq2seq model to map the language text to an expression. li2019learning applied multi-head attention to model different types of MWP features. Both wang2018deep and chiang2019learning proposed to generate expressions with the implicit tree structure. huang2018deep designed a new intermediate form to generate. Other models [26; 17; 18] have generated an expression tree explicitly to derive the final answer.

### Graph Transformer

Transformer is a self-attention based neural network which has shown potential in tasks like neural machine translation [20] and language modeling [19]. However, there are only a fewer works which focus on extension of transformer to graph-structure data. In community of natural language processing, the first

\begin{table}
\begin{tabular}{|p{227.6pt}|} \hline
**Case 1**: The class organized students to climb the mountain. The female students were divided into 4 groups, and each group had 15 students. There were 76 male students in total. How many students joined climbing last week? \\ \hline
**GTS**: \((15+76)*4\); **(error)****Graph2Tree**: \(15*4+76\); \\ \hline \hline
**Case 2**: Lingling and Yaya are 200 meters apart. Lingling is in the front and runs 3 meters per second. Yaya is in the rear and runs 5 meters per second. They set off at the same time, running in the same direction. How long will it be before Yaya could catch up with Lingling? \\ \hline
**GTS**: \(200/(3-5)\); **(error)****Graph2Tree**: \(200/(5-3)\); \\ \hline \end{tabular} 
\begin{tabular}{|l 

graph transformer was introduce in a knowledge-graph-to-text task (Koncel-Kedziorski et al., 2019), where a graph attention Network Velickovic et al. (2018) is used with a transformer style architecture. Another graph transformer Cai and Lam (2019) extends vanilla multi-head attention mechanism into relation-enhanced global attention mechanism. Our work aims to explore the adaptation of transformer in modeling multiple heterogeneous graph in parallel for the MWP task.

## 6 Conclusion

In this paper, we proposed a novel MWP solver, _Graph2Tree_, which improves the task performance by enriching the quantity representations in the problem. We conducted extensive experiments to evaluate our model against state-of-the-art baselines. Our experiments shown that _Graph2Tree_ is able to outperform the baselines on the MWP task. For future work, we aim to consider more complex relationships among the quantities and other attributes to enrich quantity representations further. We will also explore adding heuristic in the tree-based decoder to guide and improve the generation of solution expression.

## Acknowledgments

This work is supported by the National Natural Science Foundation of China (No. 61832001 and No. 61672133), Sichuan Science and Technology Program (No. 2019YFG0535 and No. 2018GZDZX0032) and the National Research Foundation, Prime Minister's Office, Singapore under its International Research Centres in Singapore Funding Initiative.

## References

* B. Bobrow (1964)Natural language input for a computer problem solving system. pp. 146-226. Cited by: SS1.
* D. Cai and W. Lam (2019)Graph transformer for graph-to-sequence learning. CoRRabs/1911.07470. External Links: Link, 1911.07470 Cited by: SS1.
* T. Chiang and Y. Chen (2019)Semantically-aligned equation generation for solving and reasoning math word problems. In NAACL-HLT, pp. 2656-2668. Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pp. 4171-4186. Cited by: SS1.
* M. Javad Hosseini, H. Hajishirzi, O. Etzioni, and N. Kushman (2014)Learning to solve arithmetic word problems with verb categorization. In EMNLP, pp. 523-533. Cited by: SS1.
* D. Huang, S. Shi, C. Lin, and J. Yin (2017)Learning fine-grained expressions to solve math word problems. In EMNLP, pp. 805-814. Cited by: SS1.
* D. Huang, J. Yao, C. Lin, Q. Zhou, and J. Yin (2018)Using intermediate representations to solve math word problems. In ACL, pp. 419-428. Cited by: SS1.
* T. N. Kipf and M. Welling (2017)Semisupervised classification with graph convolutional networks. In ICLR, Cited by: SS1.
* R. Koncel-Kedziorski, D. Bekal, Y. Luan, M. Lapata, and H. Hajishirzi (2019)Text generation from knowledge graphs with graph transformers. In NAACL-HLT, pp. 2284-2293. Cited by: SS1.
* R. Koncel-Kedziorski, A. Hajishirzi, A. Sabharwal, O. Etzioni, and S. Dumas Ang (2015)Parsing algebraic word problems into equations. TACL3, pp. 585-597. Cited by: SS1.
* R. Koncel-Kedziorski, S. Roy, A. Amini, N. Kushman, and H. Hajishirzi (2016)MAWPS: a math word problem repository. In NAACL, pp. 1152-1157. Cited by: SS1.
* N. Kushman, L. Zettlemoyer, R. Barzilay, and V. Artzi (2014)Learning to automatically solve algebra word problems. In ACL, pp. 271-281. Cited by: SS1.
* J. Li, L. Wang, J. Zhang, Y. Wang, B. Tian Dai, and D. Zhang (2019)Modeling intra-relation in math word problems with different functional multi-head attentions. In ACL, pp. 6162-6167. Cited by: SS1.
* Q. Liu, W. Guan, S. Li, and D. Kawahara (2019)Tree-structured decoding for solving math word problems. In EMNLP-IJCNLP, pp. 2370-2379. Cited by: SS1.
* C. D. Manning, M. Surdeanu, J. Bauer, J. Finkel, S. J. Bethard, and D. McClosky (2014)The Stanford CoreNLP natural language processing toolkit. In ACL, pp. 55-60. Cited by: SS1.
* A. Mitra and C. Baral (2016)Learning to use formulas to solve simple arithmetic problems. In ACL, Cited by: SS1.
* Q. Ran, Y. Lin, P. Li, J. Zhou, and Z. Liu (2019)NumNet: machine reading comprehension with numerical reasoning. In EMNLP-IJCNLP, pp. 2474-2484. Cited by: SS1.
* S. Roy and D. Roth (2015)Solving general arithmetic word problems. In EMNLP, pp. 1743-1752.

 

* Roy and Roth (2018) Subhro Roy and Dan Roth. 2018. Mapping to declarative knowledge for word problem solving. _TACL_, 6:159-172.
* Sahu et al. (2019) Sunil Kumar Sahu, Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2019. Inter-sentence relation extraction with document-level graph convolutional neural network. In _ACL_, pages 4309-4316.
* Shi et al. (2015) Shuming Shi, Yuehui Wang, Chin-Yew Lin, Xiaojiang Liu, and Yong Rui. 2015. Automatically solving number word problems by semantic parsing and reasoning. In _EMNLP_, pages 1132-1142.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _NeurIPS_, pages 5998-6008.
* Velickovic et al. (2018) Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph Attention Networks. _ICLR_.
* Wang et al. (2018) Lei Wang, Yan Wang, Deng Cai, Dongxiang Zhang, and Xiaojiang Liu. 2018. Translating a math word problem to a expression tree. In _EMNLP_, pages 1064-1069.
* Wang et al. (2019) Lei Wang, Dongxiang Zhang, Jipeng Zhang, Xing Xu, Lianli Gao, Bing Tian Dai, and Heng Tao Shen. 2019. Template-based math word problem solvers with recursive neural networks. In _AAAI_.
* Wang et al. (2017) Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. In _EMNLP_, pages 845-854.
* Xie and Sun (2019) Zhipeng Xie and Shichao Sun. 2019. A goal-driven tree-structured neural model for math word problems. In _IJCAI_, pages 5299-5305.
* Zou and Lu (2019a) Yanyan Zou and Wei Lu. 2019a. Quantity tagger: A latent-variable sequence labeling approach to solving addition-subtraction word problems. In _ACL_, pages 5246-5251.
* Zou and Lu (2019b) Yanyan Zou and Wei Lu. 2019b. Text2Math: End-to-end parsing text into math expressions. In _EMNLP-IJCNLP_, pages 5326-5336.