Venue: NAACL 2018

# A Meaning-based Statistical English Math Word Problem Solver

Chao-Chun Liang, Yu-Shiang Wong, Yi-Chung Lin and Keh-Yih Su

Institute of Information Science, Academia Sinica, Taiwan

{ccliang, yushiangwtw, lyc, kysu}@iis.sinica.edu.tw

###### Abstract

We introduce _MeSSy_, a _meaning-based_ approach, for solving English math word problems (MWPs) via _understanding and reasoning_ in this paper. It first analyzes the text, transforms both body and question parts into their corresponding logic forms, and then performs inference on them. The associated context of each quantity is represented with proposed _role-tags_ (e.g., _nsubj_, _verb_, etc.), which provides the flexibility for annotating an extracted _math quantity_ with its associated context information (i.e., the _physical meaning_ of this quantity). Statistical models are proposed to select the operator and operands. A noisy dataset is designed to assess if a solver solves MWPs mainly via understanding or mechanical pattern matching. Experimental results show that our approach outperforms existing systems on both benchmark datasets and the noisy dataset, which demonstrates that the proposed approach understands the meaning of each quantity in the text more.

## 1 Introduction

The _math word problem_ (MWP) (see Figure 1) is frequently chosen to study natural language understanding and simulate human problem solving Bakman (2007); Hosseini et al. (2014); Liang et al. (2016) for the following reasons: (1) the answer to the MWP cannot be simply extracted by performing keyword/pattern matching. It thus shows the merit of understanding and inference.

(2) An MWP usually possesses less complicated syntax and requires less amount of domain knowledge, so the researchers can focus on the task of understanding and reasoning. (3) The body part of MWP that provides the given information for solving the problem consists of only a few sentences. The understanding and reasoning procedures thus could be more efficiently checked. (4) The MWP solver has its own applications such as _Computer Math Tutor_ (for students in primary school) and _Helper for Math in Daily Life_ (for adults who are not good in solving mathematics related real problems).

According to the approaches used to identify entities, quantities, and to select operations and operands, previous MWP solvers can be classified into: (1) Rule-based approaches Mukherjee and Garain (2008); Hosseini et al. (2014), which make all related decisions based on a set of rules; (2) Purely statistics-based approaches Kushman et al. (2014); Roy et al. (2015); Zhou et al. (2015); Upadhyay et al. (2016), in which all related decisions are done via a statistical classifier; (3) DNN-based approaches Ling et al. (2017); Wang et al. (2017), which map the given text into the corresponding math operation/equation via a DNN; and (4) Mixed approaches, which identify entities and quantities with rules, yet, decide operands and operations via statistical/DNN classifiers. This category can be further divided into two subtypes: (a) Without understanding Roy and Roth (2015); Koncel-Kedziorski et al. (2015); Huang et al. (2017); Shrivastava et al. (2017), which does not check the entity-attribute consistency between each quantity and the target of the given question; and (b) With understanding Lin et al. (2015); Mitra and Baral (2016); Roy and Roth (2017), which also checks the entity-attribute consistency while solving the problem.

Figure 1: An example of math word problem.



However, a widely covered rule-set is difficult to construct for the rule-based approach. Also, it is awkward in resolving ambiguity problem. In contrast, the performance of purely statistics-based approaches deteriorates significantly when the MWP includes either irrelevant information or information gaps (Hosseini et al., 2014), as it is solved without first understanding the meaning.

For the category (4a), since the physical meaning is only implicitly utilized and the result is not generated via inference, it would be difficult to explain how the answer is obtained in a human comprehensible way. Therefore, the categories (2), (3) and (4a) belong to the less favored direct translation approach2 (Pape, 2004).

Footnote 2: According to (Pape, 2004), the meaning-based approach of solving MWPs achieves the best performance among various behaviors adopted by middle school children.

In contrast, the approaches of (4b) can avoid the problems mentioned above. However, among them, Mitra and Baral (2016) merely handled _Addition_ and _Subtraction_. Only the _meaning-based framework_ proposed by Lin et al. (2015) can handle general MWPs via understanding and reasoning. Therefore, it is possible to explain how the answer is obtained in a human comprehensible way (Huang et al., 2015). However, although their design looks promising, only a few Chinese MWPs had been tested and performance was not evaluated. Accordingly, it is hard to make a fair comparison between their approach and other state-of-the-art methods. In addition, in their prototype system, the desired operands of arithmetic operations are identified with predefined lexico-syntactic patterns and ad-hoc rules. Reusing the patterns/rules designed for Chinese in another language is thus difficult even if it is possible.

In this paper, we adopt the framework proposed by Lin et al. (2015) to solve English MWPs (for its potential in solving difficult/complex MWPs and providing more human comprehensible explanations). Additionally, we make the following improvements: (1) A new statistical model is proposed to select operands for arithmetic operations, and its model parameters can be automatically learnt via weakly supervised learning (Artzi and Zettlemoyer, 2013). (2) A new informative and robust feature-set is proposed to select the desired arithmetic operation. (3) We show the proposed approach significantly outperforms other existing systems on the common benchmark datasets reported in the literature. (4) A noisy dataset with more irrelevant quantities in MWPs is created and released. It could be used to check if an approach really understands what a given MWP looks for. (5) An experiment is conducted to compare various approaches on this new dataset. The superior performance of our system demonstrates that the proposed meaning-based approach has good potential in handling difficult/complex MWPs.

## 2 System Description

The adopted meaning-based framework (Lin et al., 2015) is a pipeline with following four stages (see Figure 2): (1) _Language Analysis_, (2) _Solution Type Identification_, (3) _Logic Form Transformation_ and (4) _Logic Inference_. We use the Stanford CoreNLP suite (Manning et al., 2014) as the language analysis module. The other three modules are briefly described below. Last, we adopt the _weakly supervised learning_(Artzi and Zettlemoyer, 2013; Kushman et al., 2014) to automatically learn the model parameters without manually annotating each MWP with the adopted solution type and selected operands benchmark.

### Solution Type Identification (STI)

After language analysis, each MWP is assigned with a specific solution type (such as _Addition_, _Multiplication_, etc.) which indicates the stereotype math operation pattern that should be adopted to solve this problem. We classify the English MWPs released by Hosseini et al. (2014) and Roy Roth (2015) into 6 different types: _Addition_, _Subtraction_, _Multiplication_, _Division_, _Sum_ and _TVQ-F3_. An SVM (Chang and Lin, 2011) is used to identify the solution type with 26 features. Most of them are derived from some important properties associated with each quantity.

Figure 2: The diagram of _MeSys_ framework



In addition to the properties _Entity4_ and _Verb_(Hosseini et al., 2014) associated with the _quantity_, we also introduce a new property _Time_ which encodes the tense and aspect of a verb into an integer to specify a point in the timeline. We assign

2, 4, and 6 to the tenses _Past_, _Present_ and _Future_, respectively, and then adjust it with the aspect-values -1, 0 and 1 for _Perfect_, _Simple_, and _Progressive_, respectively.

Footnote 4: In our works, the term “_Entity_” also includes the unit of the quantity (e.g., “_cup of coffee_”).

Another property _Anchor_ is associated with the _unknown quantity_ asked in the question sentence. If the subject of the question sentence is a noun phrase (e.g., "_how many apples does__Iohn have?_"), _Anchor_ is the _subject_ (i.e., _John_). If the subject is an expletive nominal (e.g. "_how many apples are there in the box?_"), then _Anchor_ is the associated nominal modifier _nmod_ (i.e., "_box_"). Otherwise, _Anchor_ is set to "_Unknown_".

Inspired by (Hosseini et al., 2014), we transform _Verb_ to _Verb-Class_ (\(VC\)) which is _positive_, _negative_ or _stative_. A verb is _positive/negative_ if it increases/decreases the associated quantity of the subject. For example, in the sentence "_Tom borrowed 3 dollars from Mike_", the verb is _positive_ because the money of subject "_Tom_" increases.

However, a _positive_ verb does not always imply the _Addition_ operation. If the question is "_How much money does Mike have now?_" for the above body sentence, _the operation_ should be _Subtraction_. Two new properties _Anchor-Role_ (\(AR\)) and _Action_ (\(A\)) are thus proposed: \(AR_{i}\) indicates the role that _Anchor_ associated with \(q_{i}\), and is set to _nsubjloyhymodl\(\phi\)_. \(A_{i}\) is determined by following rules: (1) \(A_{i}\)=_positive if (\(VC_{i}\)_, \(AR_{i}\))_ is either (_positive_, _nsubj_) or (_negative_, _objhmod_). (2) \(A_{i}\)=_negative if (\(VC_{i}\)_, \(AR_{i}\))_ is either (_negative_, _nsubj_) or (_positive_, _objhmod_). (3) Otherwise, _\(A_{i}\)=_VC\({}_{i}\)._

To rule out the noisy quantities introduced by irrelevant information, we further associate each known quantity with the property _Relevance_ (\(R\)) according to the unknown quantity asked in the question sentence. Let \(q_{i}\) denote the \(i\)-th known quantity, \(E_{i}\) denote the entity of \(q_{i}\), \(X_{i}\) denote the property \(X\) of \(q_{i}\), \(q_{U}\) denote the unknown quantity asked, and \(X_{U}\) denote the property \(X\) of \(q_{U}\). \(R_{i}\) is specified with following rules: (1) \(R_{i}\)=2 (Directly-Related) if either {_Anchor is Unknown & E\({}_{i}\) entails \(E_{U}\)_} or {_Anchor is not Unknown & AR\(\neq\)\(\emptyset\) & \(E_{i}\)_ entails \(E_{U}\)_} (2) \(R_{i}\)=1 (Indirectly-Related) if there is a \(q_{i}\) which maps5 to \(q_{i}\) and \(R_{i}\)=2 (i.e., \(q_{i}\) is Directly-Related). (3) \(R_{i}\)=0 (Unrelated) otherwise.

Footnote 5: That is, \(q_{i}\) is linked to a directly-related quantity \(q_{j}\) under an expression such as “2 _pencils weigh 30 grams_”.

The solution type is identified by an SVM based on 26 binary features. Let the symbols **p**, **n**, **s**, \(A\), \(E\), \(R\), \(T\), \(V\), \(S_{B}\), \(S_{Q}\) and \(w_{Q}\) stand for _positive_, _negativc_, _static_, _Action_, _Entity_, _Relevance_, _Time_, _Verb_, "_a body sentence_", "_the question sentence_" and "_a word in question sentence_" respectively. Also, let \(I(x)\) be the indicator function to check if \(x\) is true. The 26 features are briefly described as follows:

(1) \(VC_{C}\)=p; (2) \(\exists R\)=2 s. t. \(A_{i}\)=p; (3) \(\exists R\)=2 s.t. \(A_{i}\)=n;

(4) \(\exists R\)=2 s.t. \(A_{i}\)=s; (5) \(\sum_{i}I\) (\(R_{i}\)=2) > 2;

(6) \(\sum_{i}I\) (\(R\)=2 & \(A_{i}\in\{\textbf{p},\textbf{ n}\}\) ) = 2;

(7) \(\exists R\)=2 s.t. \(A_{i}\)=p & \(T_{U}\)<\(T_{i}\);

(8) \(\exists R\)=2 s.t. \(A_{i}\)=s at \(T_{U}\)<\(T_{i}\);

(9) \(\exists R\)=2 s.t. \(A_{i}\)=s & \(T_{i}\)==max \(T_{i}\);

(10) \(\exists R\)=2 s.t. \(A_{i}\)=s & \(T_{i}\)<\(v_{i}\);

(11) \(T_{U}\geq\) max \(T_{i}\); (12) \(T_{U}\leq\) min \(T_{i}\);

(13) \(\forall R\)=2, \(V_{i}\) are the same; (14) \(\forall R\)=2 s.t. \(T_{i}\)=\(T_{U}\);

(15) \(\forall R\)=2, \(T_{i}\) are the same;

(16) \(\exists R\)=2, \(\exists R\)=1 s.t. \(q_{i}\) maps to \(q_{i}\) & \(q_{i}\) ? \(q_{i}\) 

* A sandwich is priced at $0.75. A pudding is priced at $0.25. Tim bought 2 sandwiches and 4 puddings. Mary bought 2 puddings. How much money should Tim pay?
* ...price(sandwich,0.75)&price(pupdding,0.25)... quan(q1,#,sandwich)=2&verb(q1,buy)&nsubj(q1,Tim)... qual(q2,#,pudding)=4&verb(q2,buy)&nsubj(q2,Tm)... a&SkuSum(qua/[39,tobar],#)ver(Q,pay)&nsubj(q,Timm)
* quan(q, dollar,#)=quan(?q,%u,70)&vrb($q,pay) & nsubj($q,?a)

[MISSING_PAGE_POST]

5&web(q,par)&nsubj(q,Tm)... quan(q;@, dollar,#)=1.5&web(q,par)&nsubj(q,Tm)... quan(q;@, dollar,#)=1.5&web(q,par)&nsubj(q,Tm)... quan(q;@, dollar,#)=1.5&web(q,par)&nsubj(q,Tm)... quan(q;@, dollar,#)=0.5&web(q,pay)&nsubj(q,May)
* ... qun(q;@, dollar,#)=1.5&web(q,par)&nsubj(q,Tm)... quan(q;@, dollar,#)=1.5&web(q,par)&nsubj(q,Tm)... quan(q;@, dollar,#)=1.

 

the last line in Figure 4(b), three newly derived quantity facts _q4_, _q5_ and _q6_ (in 4(d)) can be unified with the first argument _quan(?q,dollar,#)_ in 4(c), but only _q4_ and _q5_ satisfy the second argument _verb(?q,pay)&nsubj(?q,Tim)_. As a result, the answer 2.5 is returned by taking sum on the values of the quantity facts _quan(q4,dollar,#)_ and _quan(q5,dollar,#)_.

### Probabilistic Operand Selection

The most error-prone part in the LFT module is instantiating the utility function of math operation especially if many irrelevant quantity facts appear in the given MWP. Figure 5 shows the LFT module needs to select two quantity facts (among 4) for _Addition_. Please note that the _question quantity_\({\mathfrak{q}}_{0}\), transformed from "_how many flowers_", is not a candidate for operand selection.

Lin et al. (2015) used predefined lexico-syntactic patterns and ad-hoc rules to instantiate utility functions. However, their rule-based approach fails when the MWP involves more quantities. Therefore, we propose a statistical model to select operands for the utility functions _Addition_, _Subtraction_, _Multiplication_ and _Division_. The operand selection procedure can be regarded as finding the most likely configuration \((o^{n}_{1},r)\), where \(o^{n}_{1}=o_{1},\cdots,o_{n}\) is a sequence of random indicators which denote if the corresponding quantity will be selected as an operand, and \(r\) is a tri-state variable to represent the relation between the values of two operands (i.e., \(r=-1,0\text{ or }1:\) which denote that the first operand is less than, equal to, or greater than the second operand, respectively). Given a solution type \(s\), the MWP logic expressions \(\mathbb{L}\) and the \(n\) quantities \(q^{n}_{1}=q_{1},\cdots,q_{n}\) in \(\mathbb{L}\). The procedure is formulated as:

\[P(r,o^{n}_{1}|q^{n}_{1},\mathbb{L},s)\approx P(r|s)\times P(o^{n}_{1}|q^{n}_{ 1},\mathbb{L},s), \tag{1}\]

\(P(r|s)\) simply refers to _Relative Frequency_ (as it has only a few parameters and we have enough training samples). \(P(o^{n}_{1}|q^{n}_{1},\mathbb{L},s)\) is further derived as:

\[\begin{split}& P(o^{n}_{1}|q^{n}_{1},\mathbb{L},s)\\ &\approx\prod_{l=1}^{n}P(o_{l}|q_{l},\mathbb{L},s)\approx\prod_{ l=1}^{n}P\big{(}o_{l}|\Phi(q_{l},\mathbb{L},s)\big{)},\end{split} \tag{2}\]

where \(\Phi(\cdot)\) is a feature extraction function to map \(q_{i}\) and its context into a feature vector. Here, the probabilistic factor \(P\big{(}o_{i}\big{|}\Phi(q_{i},\mathbb{L},s)\big{)}\) is obtained via an SVM classifier Chang Lin (2011).

\(\Phi(\cdot)\) extracts total 25 features (specified as follows, and 24 of them are binary) for \(q_{i}\). The following 11 of them are independent on the question in the MWP:

1. Four features to indicate if \(s\) is _Addition_, _Subtraction_, _Multiplication_ or _Division_.
2. A feature to indicate if \(q_{i}\) is within a \(qmap(...)\).
3. A feature to indicate if \(q_{i}=1\).
4. Five features to indicate if \(n<2\), \(n=2\), \(n=3\), \(n=4\text{ or }n>4\); where \(n\) is the number of quantities in Eq (1).

\(\Phi(\cdot)\) also extracts features by matching the logic expressions of \(q_{i}\) with those of _question quantity_\({\mathfrak{q}}_{0}\) to check the role-tag consistencies between \(q_{i}\) and \({\mathfrak{q}}_{0}\). Another fourteen features are extracted with three indicator functions \(I_{m}(\cdot)\), \(I_{e}(\cdot)\), \(I_{3}(\cdot)\) and one tri-state function \(T_{m}(\cdot)\) as follows:

\[\begin{array}{ll}[&I_{m}(q_{i},{\mathfrak{q}}_{0},\mathsf{entity}),I_{e}( q_{i},{\mathfrak{q}}_{0},\mathsf{entity}),\\ &I_{m}(q_{i},{\mathfrak{q}}_{0},\mathsf{web}),I_{e}(q_{i},{\mathfrak{q}}_{0}, \mathsf{verb}),\\ &I_{3}({\mathfrak{q}}_{0},\mathit{nsubj}),T_{m}(q_{i},{\mathfrak{q}}_{0}, \mathit{nsubj}),\\ &I_{3}({\mathfrak{q}}_{0},\mathit{modifier}),I_{m}(q_{i},{\mathfrak{q}}_{0}, \mathit{modifier}),\\ &I_{3}({\mathfrak{q}}_{0},\mathit{place}),I_{m}(q_{i},{\mathfrak{q}}_{0}, \mathit{place}),\\ &I_{3}({\mathfrak{q}}_{0},\mathit{temporal}),I_{m}(q_{i},{\mathfrak{q}}_{0}, \mathit{temporal}),\\ &I_{3}({\mathfrak{q}}_{0},\mathit{xcomp}),I_{m}(q_{i},{\mathfrak{q}}_{0}, \mathit{xcomp})~{}]\end{array}\]

where the indicator functions \(I_{m}(x,y,z)\) checks if the z of \(x\) matches the z of y, \(I_{e}(x,y,z)\) checks if the z of x entails the z of y and \(I_{3}(y,z)\) checks if the z of y exists. \(T_{m}(q_{i},{\mathfrak{q}}_{0},\mathsf{nsubj})\) returns "exactmatch" (if _nsubj_ of \(q_{i}\) matches _nsubj_ of \({\mathfrak{q}}_{0}\) ), "quasi-match" (if _nsubj_ of \({\mathfrak{q}}_{0}\) does not exist or is a plural pronoun), and "unmatch".

\(I_{e}(\cdot)\) uses the WordNet hypernym and hyponym relationship to judge whether one entity/verb entails another one or not via checking if they are in an inherited hypernym-path in WordNet. The _entity_, _verb_ and _nsubj_ of a quantity are determined according to the logic expressions. The modifier, place, temporal and _xcomp_ of a quantity are extracted from the dependency tree with some lexico-syntactic patterns. For example, the _modifier_ and _place_ of the quantity in the sentence "_There are 30 red flowers in the garden_." are "_red_" and "_garden_" respectively. The _temporal

Figure 5: An example for operand selection



and _xcomp_ of a quantity are extracted according to the dependency relations "_mod_" (i.e., _temporal modifier_) and "_xcomp_" (i.e., _open clausal complement_), respectively.

## 3 Datasets for Performance Evaluation

The AI2 dataset provided by Hosseini et al. (2014) and the IL dataset released by Roy and Roth (2015) are adopted to compare our approach with other state-of-the-art methods. The AI2 dataset has 395 MWPs on addition and subtraction, with 121 MWPs containing irrelevant information (Hosseini et al., 2014). It is the most popular one for comparing different approaches. On the other hand, the IL dataset consists of 562 elementary MWPs which can be solved by one of the four arithmetic operations (i.e., \(+\), \(-\), \(\times\), and \(+\)) without any irrelevant quantity. It is the first publicly available dataset for comparing performances that covers all four arithmetic operations.

However, the difficulty of solving an MWP depends not only on the number of arithmetic operations required, but also on how many irrelevant quantities inside, and even on how the quantities are described. One way to test if a proposed approach solves the MWPs with understanding is to check whether it is robust to those irrelevant quantities. Therefore, it is desirable to have a big enough dataset that contains irrelevant quantities which are created under different situations (e.g., confusing with an irrelevant agent, entity, or modifier, etc.) and allow us to probe the system weakness from different angles. We thus create a new dataset with more irrelevant quantities8. But before we do that, we need to know how difficult the task of solving the given MWPs is. Therefore, we first propose a way to measure how easy that a system solves the problem by simply guessing.

Footnote 8: The IL dataset does not include any irrelevant information; on the other hand, the AI2 dataset only contains 121 MWPS with irrelevant information (but not systematically created).

### Perplexity-flavor Measure

We propose to adopt the _Perplexity_ to measure the task difficulty, which evaluates how likely a solver will get the correct answer by guessing. Every MWP in the datasets can be associated with a solution expression template, such as "\(\boxed{}+\boxed{}\)" or "\(\boxed{}-\boxed{}\)", where the symbol \(\boxed{}\) represents a slot to hold a quantity. The solution can be obtained by placing correct quantities at appropriate slots. A random baseline is to solve an MWP by guessing. It first selects a solution expression template according to the prior distribution of the templates and then places quantities into the selected template according to the uniform distribution.

The expected accuracy of the random baseline on solving an MWP is a trivial combination and permutation exercise9. For example, the expected accuracy of solving an MWP associated with "\(\boxed{}\) + \(\boxed{}\)" template is \(p_{\boxed{}+\boxed{}}\times~{}nC_{2}^{~{}-1}\), where the factor \(p_{\boxed{}+\boxed{}}\) denotes the prior probability of the template "\(\boxed{}+\boxed{}\)" and \(n\) is the total number of quantities (including irrelevant ones) in the MWP. On the other hand, expected accuracy of solving an MWP associated with "\(\boxed{}-\boxed{}\)"10 template is \(p_{\boxed{}-\boxed{}\Box}\times~{}nP_{2}^{~{}-1}\). Let \(A_{l}\) denote the expected accuracy of solving the \(i\)-th MWP in a dataset. The accuracy of the random baseline on the dataset of size \(N\) is then computed as \(A=(1/N)\times\sum_{i=1}^{N}A_{i}\).

Footnote 9: Let \(nC_{k}\) denote \(k\)-combinations of \(n\) and \(nP_{k}\) denote \(k\)-permutations of \(n\).

The word "_Accuracy_" comprises the opposite sense of the word "_Perplexivity_"11 (i.e., in the sense of how hard a prediction problem is). The lower the _Accuracy_ is, the higher the _Perplexity_ is. Therefore, we transform the _Accuracy_ measure in-to a _Perplexity-Flavor_ measure (PP) via the formula:

Footnote 10: We assume the operands have different values and, therefore, they are not permutable for the _subtraction_ operator.

Footnote 11: The _Perplexity_ of a uniform distribution over \(k\) discrete events (such as casting a fair \(k\)-sided dice) is \(k\).

\[\text{pp}=2^{-\log_{2}A}\]

For instance, the _Perplexity-Flavor_ measures of AI2 and IL datasets are 4.46 and 8.32 respectively.

### Noisy Dataset

Human Math/Science tests have been considered more suitable for judging AI progress than Turing test (Clark and Etzioni, 2016). In our task, solving MWPs is mainly regarded as a test for intelligence (not just for creating a _Math Solver_ package). By injecting various irrelevant quantities into original MWPs, a noisy dataset is thus created to assess if a solver solves the MWPs mainly via _understanding_ or via _mechanical/statistical pattern matching_. If a system solves an MWP mainly via pattern matching, it would have difficulty in solving a similar MWP augmented from the original one with some irrelevant quantities. Therefore, we first create a noisy dataset by selecting some 

MWPs that can be correctly solved, and then augmenting each of them with an additional noisy sentence which involves an irrelevant quantity. This dataset is created to examine if the solver knows that this newly added quantity is irrelevant.

Figure 6 shows how we inject noise into an MWP (a). (a.1) is created by associating an irrelevant quantity to a new subject (i.e., _Mary_). Here the ellipse symbol "..." denotes unchanged text. (a.2) is obtained by associating an irrelevant quantity to a new entity (i.e., _books_). In addition, we also change modifiers (such as _yellow_, _red_, ...) to add new noisy sentence (not shown here). Since the noisy dataset is not designed to assess the lexicon coverage rate of a solver, we reuse the words in the original dataset as much as possible while adding new subjects, entities and modifiers.

136 MWPs that both Illinois Math Solver12[Roy and Roth, 2016] and our system can correctly solve are selected from the AI2 and IL datasets. This subset is denoted as OSS (Original Sub-Set). Afterwards, based on the 136 MWPs of OSS, we create a noisy dataset of 396 MWPs by adding irrelevant quantities. This noisy dataset is named as NDS13. Table 1 lists the size of MWPs, _Perplexities_ (PP), and the average numbers of quantities in each MWP of these two datasets.

Footnote 12: We submit MWPs to Illinois Math Solver ([https://cogcomp.cs.illinois.edu/page/demo](https://cogcomp.cs.illinois.edu/page/demo)\({}_{view}\)/Math) in May and June, 2017.

Footnote 13: The noisy dataset can be downloaded from [https://github.com/chaochun/nlu-mwp-noise-dataset](https://github.com/chaochun/nlu-mwp-noise-dataset). It includes 102 Addition, 147 Subtraction, 101 Multiplication and 46 Division MWPs.

## 4 Experimental Results and Discussion

We compare our approach with [58] and [58] because they achieved the state-of-the-art performance on the IL dataset. In the approach of [58], each quantity in the MWP was associated with a _quantity schema_ whose attributes are extracted from the context of the quantity. Based on the attributes, several statistical classifiers were used to select operands and determine the operator. They also reported the performances on the AI2 dataset to compare their approach with those of others (e.g., kushman2014learning, which is a purely statistical approach that aligns the text with various pre-extracted equation templates). Roy and Roth [2017] further introduced the concept of _Unit Dependency Graphs_ to reinforce the consistency of physical units among selected operands associated with the same operator.

To compare the performance of the statistical method with the DNN approach, we only implement a Bi-directional RNN-based _Solution Type Identifier_ (as our original statistical _Operand Selector_ is relatively much better). It consists of a word embedding layer (for both body and question parts), and a bidirectional GRU layer as an encoder. We apply the attention mechanism to scan all hidden state sequence of body by the last hidden state of question to pay more attention to those more important (i.e., more similar between the body and the question) words. Lastly, it outputs the solution type by a softmax function. We train it for 100 epochs, with mini-batch-size = 128 and learning-rate = 0.001; the number of nodes in the hidden layer is 200, and the drop-out rate is 0.714.

Footnote 14: Since the dataset is not large enough for splitting a development set, we choose those hyper parameters based on the test set in coarse grain. Therefore, the DNN performance we show here might be a bit optimistic.

We follow the same n-fold cross-validation evaluation setting adopted in [58] exactly. Therefore, various performances could be directly compared. Table 2 lists the accuracies of different systems in solving the MWPs

\begin{table}
\begin{tabular}{|p{113.8pt}|p{56.9pt}|} \hline  & OSS & NDS \\ \hline \# MWPs & 136 & 396 \\ \hline _Perplexity_ (PP) & 7.42 & 18.83 \\ \hline \#Quantities & 2.64 & 3.62 \\ \hline \end{tabular}
\end{table}
Table 1: _Perplexity_ measures of OSS and NDS

\begin{table}
\begin{tabular}{|l|c|c|} \hline  & AI2 & IL \\ \hline Surr (Statistical) & **81.5** & **81.0** \\ \hline Our system (DNN) & 69.8 & 70.6 \\ \hline Roy and Roth, 2017) & 76.2 & 74.4 \\ \hline Roy and Roth, 2015) & 78.0 & 73.9 \\ \hline Kushman et al., 2014) & 64.0 & 73.7 \\ \hline \end{tabular}
\end{table}
Table 2: Performances of various approaches

Figure 6: Examples of noisy sentences



of various datasets. The performance of [14] system is directly delivered by their code15. The last two rows are extracted from [13]. The results show that our performances of the statistical approach significantly outperform that of our DNN approach and other systems on every dataset.

Footnote 15: [https://github.com/CogComp/arithmetic](https://github.com/CogComp/arithmetic).

The performances of STI and LFT modules are listed in Table 3. As described in section 2, the benchmark for both solution type and the operand selection benchmark are automatically determined by weakly supervised learning. The first and second rows of Table 3 show the solution type accuracies of our statistical and DNN approaches, respectively. The third row shows the operand selection accuracy obtained by given the solution type benchmark. Basically, LFT accuracies are from 92% to 95%, and the system accuracies are dominated by STI. We analyzed errors resulted from our statistical STI on AI2 and IL datasets, respectively. For AI2, major errors come from: (1) failure of ruling out some irrelevant quantities (40%), and (2) making confusion between _TVQ-F_ and _Sum_ these two solution types (20%) for those cases that only involve addition operation (however, both types would return the same answer). For IL, major errors come from: (1) requiring additional information (35%), and (2) not knowing PartWhole relation (17%). Table 4 shows a few examples for different STI error types.

The left-half of Table 5 shows the performances on the OSS and NDS datasets. Recall that OSS is created by selecting some MWPs which both Illinois Math Solver [16] and our system16 can correctly solve. Therefore, both systems have 100% accuracy in solving the OSS dataset. However, these two systems behave very differently while solving the noisy dataset. The much higher accuracy of our system on the noisy dataset shows that our meaning-based approach understands the meaning of each quantity more. Therefore, it is less confused17 with the irrelevant quantities.

Footnote 16: In evaluating the performances on OSS and NDS datasets, our system is trained on the folds 2-5 of the IL dataset.

Footnote 17: Since the gap between two different types of approaches is quite big, those 396 examples on OSS and 196 examples on NDS are sufficient to confirm the conclusion.

One MWP in the noisy dataset that confuses _Illinois Math Solver_ (IMS) is "_Tom has 9 yellow balloons. Sara has 8 yellow balloons. Bob has 5 yellow flowers. How many yellow balloons do they have in total?_", where the underlined sentence is the added noisy sentence. The solver sums all quantities and gives the wrong answer 22, which reveals that IMS cannot understand that the quantity "_5 yellow flowers_" is irrelevant to the question "_How many yellow balloons?_". On the contrary, our system avoids this mistake.

Although the meaning of each quantity is explicitly checked in our LFT module, our system still cannot correctly solve all MWPs in NDS. The error analysis reveals that the top-4 error sources are STI, LFT, CoreNLP and incorrect problem construction (for 27%, 27%, 18%, 18%), which indicates that our STI and LFT still cannot completely prevent the damage caused from the noisy sentences (which implies that more consistency check for quantity meaning should be done). The remaining errors are due to incorrect quantity extraction, lacking common-sense or not knowing entailment relationship between two entities.

A similar experiment is performed to check if the DNN approach will be affected by the noisy information more. We first select 124 MWPs (denoted as OSS') from OSS that can be correctly solved by both our statistical and DNN approaches and then filter out 350 derived MWPs (denotes as NDS') from NDS. The right-half of Table 5 shows that the performance of the DNN approach drops more than the statistical approach does in the noisy dataset, which indicates that our statistical approach is less sensitive to the irrelevant quantities and more close to human's approach.

\begin{table}
\begin{tabular}{|p{113.8pt}|p{284.5pt}|} \hline Error Type & Example \\ \hline Confusing TVQ-F and Sum solution type & Sally found 9 seashells, Tom found 7 seashells, and Jessica found 5 seahells on the beach. How many seabells did they find together? \\ \hline Requiring additional information information & A garden has 52 rows and 15 columns of bean plans. How many plants are there in all? \\ \hline Not knowledge Parting Purch Whole reclationship & Eric wants to split a collection of peanuts into groups of 8. Eric has 64 peanuts. How many groups will be created? \\ \hline \end{tabular}
\end{table}
Table 4: Examples for different STI error types

\begin{table}
\begin{tabular}{|c|c|c|c|c|c|} \hline  & Statis- & R\&R, & & Statis- & DNN \\ \hline OSS & 100 & 100 & OSS’ & 100 & 100 \\ \hline NDS & 82.1 & 28.5 & NDS’ & 81.4 & 75.4 \\ \hline \end{tabular}
\end{table}
Table 5: Performances on the OSS and NDS 

Related Work

To the best of our knowledge, MWP solvers proposed before 2014 all adopted the rule-based approach. Mukherjee and Garain (2008) had given a good survey for all related approaches before 2008. Afterwards, Ma et al. (2010) proposed a MSWPAS system to simulate human arithmetic multi-step addition and subtraction behavior without evaluation. Besides, Liguda and Pfeiffer (2012) proposed a model based on augmented semantic networks, and claimed that it could solve multi-step MWPs and complex equation systems and was more robust to irrelevant information (also no evaluation).

Recently, Hosseini et al. (2014) proposed a _Container-Entity_ based approach, which solved the MWP with a sequence of state transition. And Kushman et al. (2014) proposed the first statistical approach, which heuristically extracts some algebraic templates from labeled equations, and then aligns them with the given sentence. Since no semantic analysis is conducted, the performance is quite limited.

In more recent researches Roy and Roth (2015); Koncel-Kedziorski et al. (2015); Roy and Roth (2017), quantities in an MWP were associated with attributes extracted from their contexts. Based on the attributes, several statistical classifiers were used to select operands and determine operators to solve multi-step MWPs. Since the physical meaning of each quantity is not explicitly considered in getting the answer, the reasoning process cannot be explained in a human comprehensible way. Besides, Shi et al. (2015) attacked the _number word problem_, which only deal with numbers, with a semantic parser. Mitra and Baral (2016) mapped MWPs into three types of problems, including Part-Whole, Change and Comparison. Each problem was associated with a generic formula. They used a log-linear model to determine how to instantiate the formula with quantities and solve the only one _Unknown_ variable. They achieved the best performance on the AI2 dataset. However, their approach cannot handle _Multiplication_ or _Division_ related MWPs. Recently, DNN-based approaches Ling et al. (2017); Wang et al. (2017) have emerged. However, they only attacked algebraic word problems, and required a very large training-set.

Our proposed approach mainly differs from those previous approaches in _combining the statistical framework with logic inference_, and also in _adopting the meaning-based statistical approach for selecting the desired operands_.

## 6 Conclusion

A _meaning-based_ logic form represented with _role-tags_ (e.g., _nsubj_, _verb_, etc.) is first proposed to associate the extracted math quantity with its physical meaning, which then can be used to identify the desired operands and filter out irrelevant quantities. Afterwards, a statistical framework is proposed to perform understanding and reasoning based on those logic expressions. We further compare the performance with a typical DNN approach, the results show the proposed approach is still better. We will try to integrate domain concepts into the DNN approach to improve the learning efficiency in the future.

The main contributions of our work are: (1) Adopting a meaning-based approach to solve English math word problems and showing its superiority over other state-of-the-art systems on common datasets. (2) Proposing a statistical model to select operands by explicitly checking the meanings of quantities against the meaning of the question sentence. (3) Designing a noisy dataset to test if a system solves the problems by understanding. (4) Proposing a perplexity-flavor measure to assess the complexity of a dataset.

## References

* Artzi and Zettlemoyer (2013) Yoav Artzi and Luke Zettlemoyer. 2013. Weakly supervised learning of semantic parsers for mapping instructions to actions. _Transactions of the Association for Computational Linguistics_, 1(1):49-62.
* Bakman (2007) Yefim Bakman. 2007. Robust understanding of word problems with extraneous information. [http://lanl.arxiv.org/abs/math.GM/0701393](http://lanl.arxiv.org/abs/math.GM/0701393).
* Chang and Lin (2011) Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: a library for support vector machines. _ACM Transactions on Intelligent Systems and Technology_, 2(3):27:1-27:27. Software available at [http://www.csie.ntu.edu.tw/cjlin/libsvm](http://www.csie.ntu.edu.tw/cjlin/libsvm).
* but how Intelligent is it? Standardized Tests as a Measure of AI. _AI Magazine_, pages 5-12.
* Hosseini et al. (2014) Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_, EMNLP 2014, October 25-29, 2014, Do 

ha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 523-533.
* Huang et al. (2015) Chien-Tsung Huang, Yi-Chung Lin, and Keh-Yih Su. 2015. Explanation generation for a math word problem solver. _International Journal of Computational Linguistics and Chinese Language processing_ (IJCLCLP), 20(2):7-44.
* Huang et al. (2017) Danqing Huang, Shuming Shi, Chin-Yew Lin and Jian Yin. 2017. Learning fine-grained expressions to solve math word problems. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 816-825.
* Koncel-Kedziorski et al. (2015) Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equations. _Transactions of the Association for Computational Linguistics_, 3:585-597.
* Kushman et al. (2014) Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. 2014. Learning to automatically solve algebra word problems. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics_, pages 271-281.
* Liang et al. (2016) Chao-Chun Liang, Shih-Hong Tsai, Ting-Yin Chang, Yi-Chung Lin and Keh-Yih Su. 2016. A meaning-based English math word problem solver with understanding, Reasoning and Explanation. In _Proceedings of COLING 2016_.
* Liguda and Pfeiffer (2012) Christian Liguda and Thies Pfeiffer. 2012. Modeling math word problems with augmented semantic networks. In Gosse Bouma, Ashwin Ittoo, Elisabeth Metais, and Hans Wortmann, editors, _Natural Language Processing and Information Systems/17th International Conference on Applications of Natural Language to Information Systems_, volume 7337. Springer.
* Lin et al. (2015) Yi-Chung Lin, Chao-Chun Liang, Kuang-Yi Hsu, Chien- Tsung Huang, Shen-Yan Miao, Wei-Yau Ma, Lun-Wei Ku, Chunr-Jung Liau, and Keh-Yih Su. 2015. Designing a tag-based statistical math word problem solver with reasoning and explanation. _International Journal of Computational Linguistics and Chinese Language processing_ (IJCLCLLP), 20(2):1-26.
* Ling et al. (2017) Wang Ling, Dani Yogatama, Chris Dyer and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_, pages 158-167.
* Ma et al. (2010) Yuhui Ma, Ying Zhou, Guangxuo Cui, Yun Ren, and Ronghuai Huang. 2010. Frame-based calculus of soling arithmetic multi-step addition and subtraction word problems. _Education Technology and Computer Science, International Workshop_ on, 2:476-479.
* Manning et al. (2014) Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In _ACL Demonstrations_.
* Mitra and Baral (2016) Arindam Mitra and Chitta Baral. 2016. Learning to use formulas to solve simple arithmetic problems. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics_, pages 2144-2153.
* Mukherjee and Garain (2008) Anirban Mukherjee and Utpal Garain. 2008. A review of methods for automatic understanding of natural language mathematical problems. _Artif. Intell. Rev._, 29(2):93-122.
* Pape (2004) Stephen J. Pape. 2004. Middle school children's problem-solving behavior: A cognitive analysis from a reading comprehension perspective. _Journal for Research in Mathematics Education_, 35(3):187-219.
* Roy and Roth (2015) Subbro Roy and Dan Roth. 2015. Solving general arithmetic word problems. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1743-1752.
* Roy and Roth (2016) Subbro Roy and Dan Roth. 2016. Illinois Math Solver: Math reasoning on the web. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations_, pages 52-66.
* Roy and Roth (2017) Subbro Roy and Dan Roth. 2017. Unit dependency graph and its application to arithmetic word problem solving. In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI-17)_.
* Roy et al. (2015) Subbro Roy, Tim Vieira, and Dan Roth. 2015. Reasoning about quantities in natural language. Transactions of the _Association for Computational Linguistics_, 3:1-13.
* Russell and Norvig (2009) Stuart Russell and Peter Norvig. 2009. _Artificial Intelligence_: A modern approach. Prentice Hall Press, Upper Saddle River, NJ, USA, 3rd edition.
* Shi et al. (2015) Shuming Shi, Yuehui Wang, Chin-Yew Lin, Xiaojiang Liu, and Yong Rui. 2015. Automatically solving number word problems by semantic parsing and reasoning. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1132-1142.
* Shrivastava and Sharma (2017) Manish Shrivastava and Dipti Misra Sharma. 2017. Deep neural network based system for solving arithmetic word problems. In _Proceedings of the 2017 International Joint Conference on Natural Language Processing_, pages 65-68.
* Upadhyay et al. (2016) Shyam Upadhyay, Ming-Wei Chang, Kai-Wei Chang, and Wen-tau Yih. 2016. Learning from explicit and implicit supervision jointly for algebra word prob 

lems. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 297-306.
* Wang et al. (2017) Yan Wang, Xiaojiang Liu and Shuming Shi. 2017. Deep neural solver for math word problems. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 856-865.
* Zhou et al. (2015) Lipu Zhou, Shuaixiang Dai, and Liwei Chen. 2015. Learn to solve algebra word problems using quadratic programming. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 817-822.