SlowGuess/ACL_2020_OCR huggingface dataset:
expression trees. IRE (Sahu et al., 2019) is another baseline that was first proposed in relation extraction and has something in common with our method.
Implementation Details and Evaluation Metric. In the Graph2Tree model, we use a word embedding (not pre-trained) with 128 units, a one layer graph transformer with 4 GCNs, each of which has the dimension of the hidden state set to 128. The dimensions of the hidden state for all the other layers are set to 512. Our model is trained for 80 epochs. Mini-batch size and dropout rate are set to 64 and 0.5, respectively. For optimizer, we use Adam with learning rate set to 0.001,  $\beta_{1} = 0.94$  and  $\beta_{2} = 0.99$ , and the learning rate will be halved every 20 epochs. Also, we use a beam size of 5 in beam search.
For the Math23K dataset, some methods are evaluated using 5-fold cross-validation, expressed in "Math23K*", and others are evaluated using the available test set (expressed as "Math23K"). We evaluate Graph2Tree on both settings. For the MAwPS dataset, the models are evaluated with 5-fold cross-validation. Following previous works, we use solution accuracy as the evaluation metric.
MAWPS	Math23K	Math23K*
DNS	59.5	-	58.1
Math-EN	69.2	66.7	-
T-RNN	66.8	66.9	-
S-Aligned	-	-	65.8
GROUP-ATT	76.1	69.5	66.9
AST-Dec	-	69.0	-
GTS	82.6	75.6	74.3
IRE	-	76.7	-
Graph2Tree	83.7	77.4	75.5
Table 2: Solution accuracy of Graph2Tree and various baselines. Note that Math23K denotes results on public test set and Math23K* denotes 5-fold cross-validation.
Table 2 shows the solution accuracy of Graph2Tree and various baselines. We observe that Graph2Tree outperforms all baselines in the two MWP datasets. As the code for GTS is made available $^{2}$ , we implemented GTS and tested it on all dataset settings. We also statistically test the improvement of Graph2Tree over the strongest baseline (i.e., GTS) and found that the improvement to be significant at 0.01 level using paired t-test. The superior performance of Graph2Tree demonstrates
$^{2}$ https://github.com/ShichaoSun/math_seq2tree
the importance of enriching quantity's representations in handling the MWP task.
4.2 Ablation Study and Parameter Analysis
To understand the effects of the various components and hyperparameters in our Graph2Tree model, we conduct ablation studies and parameter analysis on the Math23K dataset.
4.2.1 Effect of Quantity Graph
We investigate the effects of Quantity Cell Graph and Quantity Comparison Graph in our model. The results of our ablation study are shown in Table 3. We find that the Graph2Tree with both Quantity Cell Graph and Quantity Comparison Graph performs the best. We also observe that having either Quantity Cell Graph and Quantity Comparison Graph still outperforms the implementation without either graph (i.e., full-connected graph). More interestingly, we also noted that enriching the quantity representation with either graph would also outperform the baseline GTS model in this task, suggesting the importance of quantity representation in MWP task. From this study, we also infer that improving quantity representation, modeling the relationships among quantities, and retaining their numerical qualities help to achieve better results for the MWP task. Also, if two types of graphs are merged into an integrated graph, the performance drops. We postulate that a possible reason for the inferior performance may be due to the noise introduced by the integration of multiple graphs.
Math23K
Graph2Tree	77.4
only Quantity Cell Graph	76.8
only Quantity Comparison Graph	76.9
only Full-Connected Graph	75.3
merge two graphs as single one	76.4
Table 3: Solution accuracy with various graph configurations in Graph2Tree.
4.2.2 Effect of Graph Number
The number of GCNs is a tuneable hyperparameter in our Graph2Tree model. Thus, we investigate the effect of the number of GCNs on our model's performance. We varied the number of GCNs from 2, 4, 8. Note that even numbers are used as the GCNs are split evenly to model the Quantity Cell Graph and Quantity Comparison Graph. Table 4 shows the study's results. We observe that the 4-GCN
version achieves the best performance. A potential reason could be due to the optimal capacity of information aggregation is achieved using 4 GCNs over the two quantity graphs.
Math23K
w/ 2 GCN	76.7
w/ 4 GCN	77.4
w/ 8 GCN	76.9
Table 4: Solution accuracy with varying number of GCNs.
4.2.3 Impact of Length of Expression
To investigate how well our Graph2Tree model performs with the increasing expression complexity as compared to state-of-the-art models using explicit tree decoders, we analyze the increasing number of operators in the test set. From the results shown in Table 5, we note that:
(1) Our proposed Graph2tree outperforms the other two models in most cases except that the number of operators equals to 5. In other cases with less than 5 operators, our model shown statistically significant improvements over other two models.
(2) All the models' performances follow an accuracy descending pattern when the length of expression becomes longer. This is intuitive as longer expressions often associate with more complex questions that are more difficult to solve and have fewer data for training.
#Op	Pro (%)	AST-Dec (%)	GTS (%)	Our (%)
1	17.3	82.7	84.9	85.5
2	52.2	74.5	80.6	83.7
3	19.1	59.9	70.7	71.7
4	6.6	42.4	50.0	51.5
5	3.4	44.1	38.2	38.2
6	0.9	55.6	44.4	55.6
Table 5: Accuracy for increasing length of templates. #Op is the number of operators in expressions. Pro denotes the proportion of MwPs for different expression lengths.
4.2.4 Impact of Numerical Comparison
One of the primary goals of our Graph2Tree model is to address the situation where the wrong arithmetic order leads to incorrect solution expression generation. We evaluate this aspect of our model by investigating how Graph2Tree has improved the arithmetic order errors. We first retrieve the MWPs with incorrectly predicted expressions. As we are interested in arithmetic order errors, we check that the incorrectly predicted expressions' length is equal to their corresponding ground truth
Case 1: The class organized students to climb the mountain. The female students were divided into 4 groups, and each group had 15 students. There were 76 male students in total. How many students joined climbing last week?
GTS: (15 + 76) * 4; (error) Graph2Tree: 15 * 4 + 76;
Case 2: Lingling and Yaya are 200 meters apart. Lingling is in the front and runs 3 meters per second. Yaya is in the rear and runs 5 meters per second. They set off at the same time, running in the same direction. How long will it be before Yaya could catch up with Lingling?
GTS: 200/(3 - 5); (error) Graph2Tree: 200/(5 - 3);
Case 3: A bus and a truck departed from the two cities of A and B, which are 900 kilometers apart. They went in opposite directions. It takes 10 hours for the bus to travel from A to B, and 15 hours for the truck to travel from B to A. How many hours would it be before the bus, and the truck meet?
GTS: 900/(900/10 + 1/15); (error) Graph2Tree: 900/(900/10 + 900/15);
Table 6: Three examples of solving MwPs with our Graph2Tree model.
expressions' length. In total, we retrieved 103 incorrect predicted expressions for Graph2Tree and 119 for GTS. Next, we manually count the number of incorrectly predicted expression attributed to arithmetic order error among the initially retrieve set. We found that Graph2Tree has generated 7 expressions with arithmetic order error, while GTS has generated 27 arithmetic order error expressions. This suggests that Graph2Tree is able to significantly improve the arithmetic order in MWP task.
Finally, we perform a case study on the solution expressions generated by GTS and Graph2Tree. Selected case studies are shown in Table 6. In Case 1, there are essential words, i.e., "each," "group," and "students" around the quantity "15", and "students" around the quantity "76". However, GTS predicts operator "+" between these two quantities with obviously different units as GTS is unable to model quantity representation effectively using BiLSTM. For the second case, we observe that GTS gives a wrong prediction "3 - 5" as GTS does not model quantities' numerical qualities. For the last case, this MWP requires models to have the ability to handle situation where quantities are repeatedly and frequently used. Graph2Tree is able to handle this situation better than the GTS model as our model encodes the MWP in richer graph representation. The three case studies demonstrate Graph2Tree model strengths in generating more accurate and realistic solution expressions for MWPs.
Besides, further analysis is performed on error cases. We found that our model, like other baselines, performed poorly in solving MwPs with long solution expressions. Answering these MwPs re
requires complex reasoning which opens the possibility for future works.
5.1 Math Word Problems Solving
The earlier works on math word problems (MwPs) are mainly tested on small-scale datasets. These works can be broadly divided into statistical machine learning based (Kushman et al., 2014; Hosseini et al., 2014; Mitra and Baral, 2016; Roy and Roth, 2018; Zou and Lu, 2019a) and semantic parsing based (Shi et al., 2015; Koncel-Kedziorski et al., 2015; Roy and Roth, 2015; Huang et al., 2017; Zou and Lu, 2019b).
Recently, deep learning based models have become a new trend in solving math word problems. Wang et al. (2017) applied a vanilla seq2seq model to map the language text to an expression. Li et al. (2019) applied multi-head attention to model different types of MwP features. Both Wang et al. (2018) and Chiang and Chen (2019) proposed to generate expressions with the implicit tree structure. Huang et al. (2018) designed a new intermediate form to generate. Other models (Wang et al., 2019; Liu et al., 2019; Xie and Sun, 2019) have generated an expression tree explicitly to derive the final answer.
5.2 Graph Transformer
Transformer is a self-attention based neural network which has shown potential in tasks like neural machine translation (Vaswani et al., 2017) and language modeling (Devlin et al., 2019). However, there are only a fewer works which focus on extension of transformer to graph-structure data. In community of natural language processing, the first
graph transformer was introduce in a knowledge-graph-to-text task (Koncel-Kedziorski et al., 2019), where a graph attention Network (Velicković et al., 2018) is used with a transformer style architecture. Another graph transformer (Cai and Lam, 2019) extends vanilla multi-head attention mechanism into relation-enhanced global attention mechanism. Our work aims to explore the adaptation of transformer in modeling multiple heterogeneous graph in parallel for the MWP task.
In this paper, we proposed a novel MWP solver, Graph2Tree, which improves the task performance by enriching the quantity representations in the problem. We conducted extensive experiments to evaluate our model against state-of-the-art baselines. Our experiments shown that Graph2Tree is able to outperform the baselines on the MWP task. For future work, we aim to consider more complex relationships among the quantities and other attributes to enrich quantity representations further. We will also explore adding heuristic in the tree-based decoder to guide and improve the generation of solution expression.
This work is supported by the National Natural Science Foundation of China (No. 61832001 and No. 61672133), Sichuan Science and Technology Program (No. 2019YFG0535 and No. 2018GZDZX0032) and the National Research Foundation, Prime Ministers Office, Singapore under its International Research Centres in Singapore Funding Initiative.
D. Bobrow. 1964. Natural language input for a computer problem solving system. pages 146-226.
Deng Cai and Wai Lam. 2019. Graph transformer for graph-to-sequence learning. CoRR, abs/1911.07470.
Ting-Rui Chiang and Yun-Nung Chen. 2019. Semantically-aligned equation generation for solving and reasoning math word problems. In NAACL-HLT, pages 2656-2668.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT*, pages 4171–4186.
Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. In EMNLP, pages 523-533.
Danqing Huang, Shuming Shi, Chin-Yew Lin, and Jian Yin. 2017. Learning fine-grained expressions to solve math word problems. In EMNLP, pages 805-814.
Danqing Huang, Jin-Ge Yao, Chin-Yew Lin, Qingyu Zhou, and Jian Yin. 2018. Using intermediate representations to solve math word problems. In ACL, pages 419-428.
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In ICLR.
Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh Hajishirzi. 2019. Text generation from knowledge graphs with graph transformers. In NAACL-HLT, pages 2284-2293.
Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equations. TACL, 3:585-597.
Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. MAwPS: A math word problem repository. In NAACL, pages 1152-1157.
NateKushman,LukeZettlemoyer,ReginaBarzilay, andYoavArtzi.2014.Learningtoautomatically solvealgebrawordproblems.InACL,pages271- 281.
Jierui Li, Lei Wang, Jipeng Zhang, Yan Wang, Bing Tian Dai, and Dongxiang Zhang. 2019. Modeling intra-relation in math word problems with different functional multi-head attentions. In ACL, pages 6162-6167.
Qianying Liu, Wenyv Guan, Sujian Li, and Daisuke Kawahara. 2019. Tree-structured decoding for solving math word problems. In EMNLP-IJCNLP, pages 2370-2379.
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In ACL, pages 55-60.
Arindam Mitra and Chitta Baral. 2016. Learning to use formulas to solve simple arithmetic problems. In ACL.
Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, and Zhiyuan Liu. 2019. NumNet: Machine reading comprehension with numerical reasoning. In EMNLP-IJCNLP, pages 2474-2484.
Subhro Roy and Dan Roth. 2015. Solving general arithmetic word problems. In EMNLP, pages 1743-1752.
Subhro Roy and Dan Roth. 2018. Mapping to declarative knowledge for word problem solving. TACL, 6:159-172.
Sunil Kumar Sahu, Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2019. Inter-sentence relation extraction with document-level graph convolutional neural network. In ACL, pages 4309-4316.
Shuming Shi, Yuehui Wang, Chin-Yew Lin, Xiaojiang Liu, and Yong Rui. 2015. Automatically solving number word problems by semantic parsing and reasoning. In EMNLP, pages 1132-1142.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS, pages 5998-6008.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph Attention Networks. ICLR.
Lei Wang, Yan Wang, Deng Cai, Dongxiang Zhang, and Xiaojiang Liu. 2018. Translating a math word problem to a expression tree. In EMNLP, pages 1064-1069.
Lei Wang, Dongxiang Zhang, Jipeng Zhang, Xing Xu, Lianli Gao, Bing Tian Dai, and Heng Tao Shen. 2019. Template-based math word problem solvers with recursive neural networks. In AAAI.
Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. In EMNLP, pages 845-854.
Zhipeng Xie and Shichao Sun. 2019. A goal-driven tree-structured neural model for math word problems. In *IJCAI*, pages 5299–5305.
Yanyan Zou and Wei Lu. 2019a. Quantity tagger: A latent-variable sequence labeling approach to solving addition-subtraction word problems. In ACL, pages 5246-5251.
Yanyan Zou and Wei Lu. 2019b. Text2Math: End-to-end parsing text into math expressions. In EMNLP-IJCNLP, pages 5326-5336.