SlowGuess/ACL_2020_OCR huggingface dataset:
Jipeng Zhang $^{1,2,*}$ , Lei Wang $^{2,*}$ , Roy Ka-Wei Lee $^{3}$ , Yi Bin $^{1}$ , Yan Wang $^{4}$ , Jie Shao $^{1,5,\#}$ , Ee-Peng Lim $^{2}$
<sup>1</sup>Center for Future Media, University of Electronic Science and Technology of China
$^{2}$ School of Information Systems, Singapore Management University
$^{3}$ Department of Computer Science, University of Saskatchewan
Sichuan Artificial Intelligence Research Institute
zhangjipeng20@std.uestc.edu.cn, lei.wang.2019@phdcs.smu.edu.sg, roylee@cs.usask.ca,
yi.bin@hotmail.com, bradenwang@tencent.com, shaojie@uestc.edu.cn, eplim@smu.edu.sg
While the recent tree-based neural models have demonstrated promising results in generating solution expression for the math word problem (MWP), most of these models do not capture the relationships and order information among the quantities well. This results in poor quantity representations and incorrect solution expressions. In this paper, we propose Graph2Tree, a novel deep learning architecture that combines the merits of the graph-based encoder and tree-based decoder to generate better solution expressions. Included in our Graph2Tree framework are two graphs, namely the Quantity Cell Graph and Quantity Comparison Graph, which are designed to address limitations of existing methods by effectively representing the relationships and order information among the quantities in MwPs. We conduct extensive experiments on two available datasets. Our experiment results show that Graph2Tree outperforms the state-of-the-art baselines on two benchmark datasets significantly. We also discuss case studies and empirically examine Graph2Tree's effectiveness in translating the MWP text into solution expressions<sup>1</sup>.
Math Word Problem (MWP), which involves automatically answering a mathematical question according to a textual description, is an important natural language understanding task that has been studied by researchers since the 1960s (Bobrow, 1964). A typical MWP is a short narrative that describes a problem and poses a question about an unknown quantity. Table 1 provides an example of a typical MWP where the reader is required to infer the revenue of a store after selling all the teddy
Problem: 348 teddy bears are sold for  $23 each. There are total 470 teddy bears in a store and the remaining teddy bears are sold for$ 17 each. How much did the store earn after selling all the teddy bears?
Expression:  $x = 348 \times 23 + (470 - 348) \times 17$
Solution: 10078
Table 1: A math word problem.
bears. Earlier studies have attempted to perform the MWP task via statistical machine learning methods (Kushman et al., 2014; Hosseini et al., 2014; Mitra and Baral, 2016; Roy and Roth, 2018) and semantic parsing approaches (Shi et al., 2015; Koncel-Kedziorski et al., 2015; Roy and Roth, 2015; Huang et al., 2017). However, these methods are nonscalable as tremendous efforts are required to design suitable features and expression templates.
In recent years, deep learning-based models have been developed to solve MwPs. These deep learning methods are able to automate the learning of features and generalize well by returning new solution expressions that are unseen in the training datasets. Wang et al. (2017) proposed a large-scale MWP dataset and applied a vanilla sequence to sequence (seq2seq) model to translate the language text to a solution expression. Since then, many research efforts mainly focused on improving the generation of solution expressions. Some researchers have proposed seq2seq models to improve solution expression generation using implicit (Wang et al., 2018; Chiang and Chen, 2019) and explicit (Wang et al., 2019; Liu et al., 2019; Xie and Sun, 2019) tree structures. Improving the representation of quantity is a potential approach to achieve better solution expressions. For example, to get the correct solution expression for the problem described in Table 1, an ideal MWP model should be able to associate quantity, i.e., 348 teddy bears, with its price attribute of $23, and understand the arithmetic order by deriving 122 remaining teddy bears, i.e.,
$^{1}$ Code could be found at https://github.com/2003pro/Graph2Tree
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3928-3937
470 - 348, before associating the price attribute of $17. The existing deep learning models are not effective in capturing such relationships and order information among the quantities in MwPs, thus resulting in an inaccurate representation of the final solution expressions.
To enrich the representation of a quantity, the relationships between the descriptive words associated with a quantity need to be modeled. However, such relationships cannot be effectively modeled using recurrent models, which are commonly used in the existing MWP deep learning methods. Inspired by the concept of Quantity Schema (Roy and Roth, 2015) and Qset (Koncel-Kedziorski et al., 2015), we design the Quantity Cell Graph to associate informatively descriptive words to quantity. We first extract associated nouns, verbs, adjectives, units, and rates that describe a quantity in the MWP text. Next, we construct a graph where the extracted descriptive words are represented as neighbor nodes directly linked to a quantity. Finally, a neural network model is used to learn enriched latent representations of the quantities based on the constructed Quantity Cell Graph.
The loss of quantities' numerical qualities in existing MWP methods can also result in poor quantity representations. Most of the existing MWP methods often replace quantities with special symbols (e.g., "n1", "n2", etc.) (Wang et al., 2017, 2018; Liu et al., 2019). The loss of quantities' numerical qualities could be problematic when generating solution expressions. Take the example in Table 1, without modeling the numerical qualities of quantities, an MWP method may learn a solution expression "384 - 470" which results in a negative number that is unlikely to occur in MWPs. To address this limitation, we introduce the Quantity Comparison Graph, which was inspired by a numerical machine reading comprehension model proposed by Ran et al. (2019). The intuition of Quantity Comparison Graph is to retain the numerical qualities of the quantity and leverage certain heuristics to represent the relationships among quantities in MWPs such that solution expressions reflect a more realistic arithmetic order.
Besides improving the quantity representation, we also aim to improve the solution expression generative process. For longer solution expressions in MWPs, as some quantities are repeatedly used in different arithmetic sub-solution expressions, the existing methods which utilized recurrent neural
networks may not be able to learn the underlying reasoning process and arithmetic order. For example, in Table 1, the quantity 348 is being used in "348 * 23" and "(470 - 348) * 17". To address this limitation, we propose to use a graph encoder to guide the learning of representations of quantities and a tree decoder to explicitly model the multistage reasoning process.
Contribution. In this paper, we combine the above-proposed solutions and introduce the Graph2Tree solver to address the existing MWPs methods' limitations. The contributions of this paper are as follows:
- We construct the Quantity Cell Graph and Quantity Comparison Graph to enrich the quantity representations by capturing relationships between quantities and their attributes and retaining the quantities' numerical qualities.
- We propose the Graph2Tree to improve the learning of solution expressions' generation. The Graph2Tree model uses a graph transformer to learn the latent quantity representations from our proposed graphs, and a tree structure decoder to generate a solution expression tree. To the best of our knowledge, this is the first graph-to-tree model for MWPs.
- We conduct extensive experiments on two available large-scale MwPs datasets, and our results show that our proposed Graph2Tree model outperforms state-of-the-art baselines on MwP task.
We denote the text of the math word problem as  $P$ , where  $P$  is a sequence of word tokens and numeric values. We let  $V_{p} = \{v_{1},\dots ,v_{m}\}$  denote the word tokens in  $P$  and  $n_P = \{n_1,\dots ,n_l\}$  denote the set of quantities in  $P$ . Our goal is to map  $P$  to a valid and correct mathematical expression  $E_{p}$ .
Solving MwPs requires an understanding of quantities in problem and their complex mathematical relationships. MwPs are often expressed in a linear textual sequence form, which is not ideal for learning the quantities' complex interactions. Thus, we propose to formulate the problem into graph form so that the relationships between quantities can be expressed more explicitly. The problem text  $P$  is transformed into graph  $\mathcal{G}$  by augmenting the text sequences with other structural information like dependency parsing and POS tagging.
Figure 1: Overview of the proposed model. In order to initialize representation of text  $P$ , a BiLSTM is used to compute node representation  $H$ . Later, after extracting Quantity Cells from text  $P$ , we construct Quantity Comparison Graph and Quantity Cell Graph. With two graphs and  $H$ , we use the proposed graph transformer to get the internal representation. Finally, a tree-based decoder is implemented to generate the target euqation  $E_{q}$ .
The final mathematical expression  $E_{p}$  that we aim to construct can always be represented as a solution expression tree  $T$ .  $T$  may include constant quantities, operators and quantities in  $n_P$ . The set of constant quantities  $V_{con}$  contains some special values not appeared in text like  $\pi, 1$ . The set of math operators  $V_{op}$  contains  $\{+, -, *, / \}$ . Overall, the target vocabulary of  $P$  can be denoted as  $V_{dec} = V_{op} \cup V_{con} \cup n_P$  ( $V_{dec}$  varies in different problems as  $n_P$  varies). The goal of our Graph2Tree model here is to estimate the conditional probability  $P(E_{p}|P)$ , which can be transformed as  $P(T|\mathcal{G}, V_{dec})$ .
Figure 1 shows our proposed Graph2Tree framework. Graph2Tree first encodes the MWP text input using BiLSTM and simultaneously constructs Quantity Cell Graph and Quantity Comparison Graph. The output of BiLSTM, word-level representations, are used as node representations. Together with the two constructed graphs, the node representations are input into a graph transformer to learn a graph representation of the MWP. The multiGCN component of the graph transformer is modified to learn the graph representation based on the Quantity Cell Graph and Quantity Comparison Graph. This enriches the final graph representation with quantities' relationship information and numerical qualities. Pooling is used to aggregate all nodes into a pool-based graph embedding vector as the graph transformer's output. Finally, the output graph representation and the updated node
representations are used as input to a tree-structure decoder to infer the final solution expression tree.
3.1 Graph-Based Encoder
There have been some graph-based models (Sahu et al., 2019) intending to grab the complicated relations in text. The graph-based encoder in our Graph2Tree framework is inspired by the graph transformer model (Koncel-Kedziorski et al., 2016; Cai and Lam, 2019). We first discuss the initialization of node representations based on MwPs' input problem text. Next, we introduce the construction of the Quantity Cell Graph and Quantity Comparison Graph. Finally, we discuss the learning of graph representation using the graph transformer module.
3.1.1 Node Representation Initialization
To initialize the node representations, we first learn the word-level hidden state representations of the input MwP text using a BiLSTM neural network,  $H = \{h_1,\dots ,h_N\} \in R^{N\times d},N = m + l.$  Here  $d$  denotes the dimension of hidden vectors,  $m$  represents the number of words, and  $l$  represents the number of quantities. The learned hidden state representations will be used as the input node representations for the graph encoder.
3.1.2 Quantity Cell
We refer all quantities  $n_P$  and words  $V_p$  from the problem as nodes in the graph. Next, we define a quantity cell as a subset of nodes in the graph that are associated with a quantity. Formally, each
MWP  $P$  is transformed into multiple quantity cells  $QC = \{Q_{1}, Q_{2}, \dots, Q_{m}\}$ , where  $m$  is the number of quantities in  $P$ . Each quantity cell  $Q_{i} \in QC$  contains a quantity token  $\{n_{i}\}$  and the corresponding attributes  $\{v_{1i}, \dots, v_{qi}\}$ . These quantity cells are sub-graph representations of quantity-related information in the MWPs. Dependency parsing, constituency parsing and POS tagging implemented with stanford corenlp toolkit (Manning et al., 2014) are used to extract and construct the quantity cells.
A quantity cell in an MWP  $P$  consists of the following properties:
- Quantity. The quantity numeric value.
- Associated Nouns. We consider the nouns related to the Quantity in the dependency parse tree. Associated Nouns are the nouns related by the num, number and prep_of relations.
- Associated Adjectives. Associated Adjectives are the adjectives related to Quantity or Associated Nouns with the amod relation, which is detected by the dependency parser.
- Associated Verbs. For each Quantity, we detect the related verbs, Associated Verbs, according to nsubj and dobj relations.
- Units and Rates. We detect the nouns related to Associated Nouns by prep_of as the Unit. The nouns related Associated Nouns which own the key words such as "each", "every" and "per" are regarded as Rates.
If the quantity cell detection process does not grab any attributes, we will use a window centered on Quantity to select neighboring words as the attributes of the Quantity. An example of the quantity cell is illustrated in the left part of Figure 1.
3.1.3 Quantity Graph Construction
From the quantity cells, we construct two graphs: Quantity Cell Graph and Quantity Comparison Graph. The goal of the Quantity Cell Graph is to associate informative descriptive words to quantity so as to enrich the quantity's representation. Similarly, the goal of the Quantity Comparison Graph is to retain the numerical qualities of the quantity and leverage heuristics to improve representations of the relationships among quantities. Formally, we define the construction of two graphs as follow:
- Quantity Cell Graph  $G_{qcell}$ . For each Quantity Cell  $Q_i = \{n_i\} \cup \{v_{1i}, \dots, v_{ni}\}$ , the undirected edge  $e_{ij}$  between  $n_i$  and each  $v_j \in \{v_{1i}, \dots, v_{qi}\}$  will be added to the graph  $G_{qcell}$ .
- Quantity Comparison Graph  $G_{qcomp}$ . For
two quantity nodes  $n_i, n_j \in n_P$ , a directed edge  $e_{ij} = (n_i, n_j)$  pointing from  $n_i$  to  $n_j$  will be added to the graph  $G_{qcomp}$  if  $n_i > n_j$ . This heuristic constraint can prevent the subtracting a larger number from a smaller number, which results in a negative number.
We represent the two graphs using adjacency matrices. For graph  $G$ , an adjacency matrix  $A \in R^{N \times N}$  is first initialized. If there exists an edge between the  $i$ -th and  $j$ -th nodes, we need to assign value 1 to corresponding position of the adjacency matrix  $(i,j,A_{i,j})$  for this edge. Otherwise, 0 would be assigned. Thus, we compute the adjacency matrix  $A_{qcomp}$  for graph  $G_{qcomp}$  and  $A_{qcell}$  for  $G_{qcell}$ .
3.1.4 Graph Transformer
The inputs to the graph transfer module are adjacency matrices of multiple graphs  $\{A_k\}_{k=1}^K$ ,  $A_k \in \{A_{qcomp}, A_{qcell}\}$  and initial node embeddings  $H$ , where  $K$  is the number of graphs and each  $A_k \in R^{N \times N}$  is the adjacency matrix for  $k$ -th graph.  $K$  graphs are used as we adopt a multi-head structure in our model and they are split evenly between Quantity Cell Graphs and Quantity Comparison Graph.
The graph transformer first utilizes graph convolution networks (GCNs) (Kipf and Welling, 2017) to learn the graph node features. For multiple graphs, we use a  $K$ -head graph convolution setup. This is similar to the transformer model proposed in Vaswani et al. (2017), where  $K$  separate graph convolution networks are used and concatenated before a residual connection is applied.
Specifically, a single GCN has its parameter  $W_{gk} \in R^{d \times d_k}$ , where  $d_k = d / K$ . Given an adjacency matrix  $A_k$  representing graph structure and a feature matrix  $X$  (in the beginning,  $X$  is set as  $H$ ) meaning the input feature for all nodes, we define learning of GCN as follow:
$$
G C N \left(A _ {k}, X\right) = G C o n v _ {2} \left(A _ {k}, G C o n v _ {1} \left(A _ {k}, X\right)\right) \tag {1}
$$
Here, the GCN contains 2 different graph convolution operations:
$$
G C o n v \left(A _ {k}, X\right) = r e l u \left(A _ {k} X ^ {T} W _ {g k}\right) \tag {2}
$$
For each graph  $\{A_k\}_{k=1}^K$ , we perform learning of GCN in parallel, yielding  $d_k$ -dimensional output values. The output values are concatenated and projected, resulting in the final values:
$$
Z = \prod_ {k = 1} ^ {K} G C N \left(A _ {k}, H\right) \tag {3}
$$
Here,  $\parallel$  denotes the concatenation of the  $K$  GCN heads.
Graph transformer then augments this  $K$ -head graph convolution network with a feed-forward network, layer-norm layer, and residual connection:
$$
\hat {Z} = Z + \operatorname {L a y e r N o r m} (Z) \tag {4}
$$
$$
\overline {{Z}} = \hat {Z} + L a y e r N o r m (F F N (\hat {Z})) \quad (5)
$$
here,  $FFN(x)$  is a two-layer feed-forward network with a relu function between layers:
$$
F F N (x) = \max  \left(0, x W _ {f 1} + b _ {f 1}\right) W _ {f 2} + b _ {f 2} \tag {6}
$$
The resulting node representations  $\overline{Z}$  represent quantities, entities and relations. In order to learn the global context graph representation, we apply the element-wise min-pooling operation on all learned node representations. Finally, the global feature is fed into a fully connected neural network (FC) to generate the graph representation  $z_{g}$ :
$$
z _ {g} = F C (M i n P o o l (\bar {Z})) \tag {7}
$$
3.2 Tree-Based Decoder
Inspired by the the Goal-driven Tree Structure (GTS) (Xie and Sun, 2019), we build a tree-based decoder to construct the solution expressions. We set the quantity nodes to be the leaf nodes and each operator node must have two child nodes. As such, the specialized tree decoder generates an equation following the pre-order traversal ordering. As part of the tree construction process, the centermost operator is first produced, followed by the left child node. This process is repeated until the leaf node is produced. Subsequently, we generate the right child nodes recursively.
3.2.1 Tree Initialization
To start the above mentioned tree generation process, our model initializes the root node vector  $q_{\text{root}}$  according to the global context graph representation  $z_g$ . For each token  $y$  in the target vocabulary  $V_{dec}$  of  $P$ , the representation for a certain token  $e(y|P)$  is defined as:
$$
\left\{ \begin{array}{l l} e _ {(y, o p)} & i f y \in V _ {o p} \\ e _ {(y, c o n)} & i f y \in V _ {c o n} \\ \overline {{z}} _ {l o c (y, P)} ^ {p} & i f y \in n _ {P} \end{array} \right. \tag {8}
$$
The expression trees in our decoder contain three kinds of nodes: operators, constant quantities, and quantities that appeared in  $P$ . Constant quantities
and quantities in  $n_P$  are always set to be in leaf nodes position. Operators will always take up the positions of the non-leaf nodes. The quantities' representations in  $n_P$  are dependent on certain MwPs, i.e.,  $y$  will take the corresponding  $\overline{z}_{loc(y,P)}^p$  from  $\overline{Z}$ . The representations of operators and constant quantities are independent, i.e., their representations are obtained by 2 independent embedding matrices  $M_{op}$  and  $M_{con}$ .
3.2.2 Pre-Order Tree Generation
We adopt the pre-order traversal manner to construct the expression tree:
- Step 1. The generation starts with a derivation tree with only a root node  $q_{root}$ . We use attention module of GTS to encode the node embedding  $\overline{Z}$  into global graph vector  $G_{c}$ :
$$
G _ {c} = \mathrm {G T S} - \operatorname {A t t e n t i o n} \left(q _ {\text {r o o t}}, \bar {Z}\right) \tag {9}
$$
- Step 2. This tree decoder applies left subnode generation module to the derivation in a top-down manner, generating new left child node  $q_{l}$  conditioned on the parent node  $q_{p}$  and global graph  $G_{c}$ . Note that the token  $\hat{y}$  is predicted when generating the new node:
$$
q _ {l} = \operatorname {G T S} - \operatorname {L e f t} \left(q _ {p}, G _ {c}\right) \tag {10}
$$
$$
\hat {y} = \operatorname {G T S} - \operatorname {P r e d i c t} \left(q _ {l}, G _ {c}\right)
$$
If the generated  $\hat{y}$  is an operator, two empty child node positions are created and we will keep executing Step 2. This step works like decomposing the whole goal into multi-stage reasoning. If the generated  $\hat{y}$  is a quantity (constant or from  $n_P$ ), we will get into Step 3.
- Step 3. The tree decoder switches to use the right sub-node generation module and populate the empty right node position. At every decoding step, we use the left child node  $q_{l}$ , global graph vector  $G_{c}$  and a sub-tree embedding  $t_{l}$  as the input to the right generation module and generate the right child node  $q_{r}$  and the corresponding token  $\hat{y}_{r}$ :
$$
q _ {r} = \operatorname {G T S} - \operatorname {R i g h t} \left(q _ {l}, G _ {c}, t _ {l}\right) \tag {11}
$$
$$
\hat {y} _ {r} = \operatorname {G T S} - \operatorname {P r e d i c t} \left(q _ {r}, G _ {c}\right)
$$
The addition of the sub-tree embedding works similarly to incorporating a sub-tree copying mechanism. The additional sub-tree embedding  $t_l$  is computed by using sub-tree embedding component of GTS:
$$
t _ {l} = \operatorname {G T S} - \operatorname {S u b T r e e} \left(\hat {y} _ {l}, q _ {l}\right) \tag {12}
$$
If  $\hat{y}_r$  is an operator, the next step should go back to Step 2. If  $\hat{y}_r$  is a quantity, we will get into Step 4.
- Step 4. The model switches to backtracking to find the new empty right node position. If the model cannot find the new empty right node position, the generation is completed. If the empty right node position still exists, go back to Step 2.
3.3 Model Learning
For each problem-tree expression example,  $(p,T)$ , the loss function  $L(T,P)$  is defined as the a sum of the negative log-likelihoods of probabilities for predicting  $t$ -node token  $y_{t}$ . Formally, our training goal is to minimize the following loss function:
$$
L (T, P) = \sum_ {t = 1} ^ {E} - \log p r o b \left(y _ {t} \mid q _ {t}, G _ {c}, P\right) \tag {13}
$$
where  $q_{t}$  is the goal vector,  $G_{c}$  is the global graph context,  $E$  is the number of tokens in  $T$ , and  $prob$  is computed by distribution computation function in GTS.
In this section, we compare our proposed Graph2Tree model with state-of-the-art baselines. We also conduct ablation study and analysis to investigate the effectiveness of various components of our model.
Datasets. Two commonly-used MWP datasets are used in our experiments: MAWPS (Koncel-Kedziorski et al., 2016) with 2,373 problems and Math23K (Wang et al., 2017) with 23,162 problems.
Baselines. We compare Graph2Tree to an extensive set of baselines and state-of-the-art models: DNS (Wang et al., 2017) uses a vanilla seq2seq model to generate expressions. Math-EN (Wang et al., 2018) benefits from an equation normalization to reduce target space. T-RNN (Wang et al., 2019) applies recursive neural networks over predicted tree-structure templates. S-Aligned (Chiang and Chen, 2019) designs the decoder with a stack to track the semantic meanings of operands. GROUP-ATT (Li et al., 2019) borrows the idea of multihead attentions from Transformer (Vaswani et al., 2017). AST-Dec (Liu et al., 2019) creates an expression tree with a tree LSTM decoder. GTS (Xie and Sun, 2019) develops a tree structured neural networks in a goal-driven manner to generate