Venue: NAACL 2019

# Semantically-Aligned Equation Generation

for Solving and Reasoning Math Word Problems

 Ting-Rui Chiang Yun-Nung Chen

National Taiwan University, Taipei, Taiwan

r07922052@csie.ntu.edu.tw y.v.chen@ieee.org

###### Abstract

Solving math word problems is a challenging task that requires accurate natural language understanding to bridge natural language texts and math expressions. Motivated by the intuition about how human generates the equations given the problem texts, this paper presents a neural approach to automatically solve math word problems by operating symbols according to their semantic meanings in texts. This paper views the process of generating equations as a bridge between the semantic world and the symbolic world, where the proposed neural math solver is based on an encoder-decoder framework. In the proposed model, the encoder is designed to understand the semantics of problems, and the decoder focuses on tracking semantic meanings of the generated symbols and then deciding which symbol to generate next. The preliminary experiments are conducted in a benchmark dataset Math23K, and our model significantly outperforms both the state-of-the-art single model and the best non-retrieval-based model over about 10% accuracy, demonstrating the effectiveness of bridging the symbolic and semantic worlds from math word problems.1

Footnote 1: The source code is available at [https://github.com/MiuLab/E2EMathSolver](https://github.com/MiuLab/E2EMathSolver).

## 1 Introduction

Automatically solving math word problems has been an interesting research topic and also been viewed as a way of evaluating machines' ability Mandal and Naskar (2019). For human, writing down an equation that solves a math word problem requires the ability of reading comprehension, reasoning, and sometimes real world understanding. Specifically, to solve a math word problem, we first need to know the goal of the given problem, then understand the semantic meaning of each numerical number in the problem, perform reasoning based on the comprehension in the previous step, and finally decide what to write in the equation.

Most prior work about solving math word problems relied on hand-crafted features, which required more human knowledge. Because those features are often in the lexical level, it is not clear whether machines really understand the math problems. Also, most prior work evaluated their approaches on relatively small datasets, and the capability of generalization is concerned.

This paper considers the reasoning procedure when writing down the associated equation given a problem. Figure 1 illustrates the problem solving process. The illustration shows that human actually assigns the semantic meaning to each number when manipulating symbols, including operands (numbers) and operators (\(+-\times\div\)). Also, we believe that the semantic meaning of operands can help us decide which operator to use. For example, the summation of "_price of one pen_" and "_number of pens Tom bought_" is meaningless; therefore the addition would not be chosen.

Following the observation above, this paper proposes a novel encoder decoder model, where the encoder extracts semantic meanings of numbers in the problem, and the decoder is equipped with a stack that facilitates tracking the semantic meanings of operands. The contributions of this paper are 4-fold:

* This paper is the first work that models semantic meanings of operands and operators for math word problems.
* This paper proposes an end-to-end neural math solver with a novel decoding process that utilizes the stack to generate associated equations.



* This paper achieves the state-of-the-art performance on the large benchmark dataset Math23K.
* This paper is capable of providing interpretation and reasoning for the math word problem solving procedure.

## 2 Related Work

There is a lot of prior work that utilized handcrafted features, such as POS tags, paths in the dependency trees, keywords, etc., to allow the model to focus on the quantities in the problems Kushman et al. (2014); Hosseini et al. (2014); Roy et al. (2015); Roy and Roth (2015); Koncel-Kedziorski et al. (2015); Roy et al. (2016); Upadhyay et al. (2016); Upadhyay and Chang (2017); Roy and Roth (2018); Wang et al. (2018). Recently, Mehta et al. (2019) attempted at learning models without predefined features. Following the recent trend, the proposed end-to-end model in this paper does not need any hand-crafted features.

Kushman et al. first extracted templates about math expressions from the training answers, and then trained models to select templates and map quantities in the problem to the slots in the template. Such two-stage approach has been tried and achieved good results Upadhyay and Chang (2017). The prior work highly relied on human knowledge, where they parsed problems into equations by choosing the expression tree with the highest score calculated by an operator classifier, working on a hand-crafted "trigger list" containing quantities and noun phrases in the problem, or utilizing features extracted from text spans Roy et al. (2015, 2016); Koncel-Kedziorski et al. (2015). Shi et al. defined a Dolphin language to connect math word problems and logical forms, and generated rules to parse math word problems. Upadhyay et al. parsed math word problems without explicit equation annotations. Roy and Roth classified math word problems into 4 types and used rules to decide the operators accordingly. Wang et al. trained the parser using reinforcement learning with hand-crafted features. Hosseini et al. modeled the problem text as transition of world states, and the equation is generated as the world states changing. Our work uses a similar intuition, but hand-crafted features are not required and our model can be trained in an end-to-end manner. Some end-to-end approaches have been proposed, such as generating equations directly via a seq2seq model Wang et al. (2017). Ling et al. tried to generate solutions along with its rationals with a seq2seq-like model for better interpretability.

This paper belongs to the end-to-end category, but different from the previous work; we are the first approach that generates equations with stack actions, which facilitate us to simulate the way how human solves problems. Furthermore, the proposed approach is the first model that is more interpretable and provides reasoning steps without the need of rational annotations.

## 3 End-to-End Neural Math Solver

Our approach composes of two parts, an encoder and a decoder, where the process of solving math word problems is viewed as transforming multiple text spans from the problems into the target information the problems ask for. In the example shown in Figure 1, all numbers in the problem are attached with the associated semantics. Motivated by the observation, we design an encoder to extract the semantic representation of each number in the problem text. Considering that human usually manipulates those numbers and operators (such as addition, subtraction, etc.) based on their semantics for problem solving, a decoder is designed to construct the equation, where the semantics is aligned with the representations extracted by the encoder. The idea of the proposed model

Figure 1: The solving process of the math word problem “_Each notebook takes $0.5 and each pen takes $1. Tom has $10. How many notebooks can he buy after buying 5 pens?_” and the associated equation is \(x=(10-1\times 5)+0.5\). The associated equation is \(x=(10-1\times 5)\div 0.5\).

 

is to imitate the human reasoning process for solving math word problems. The model architecture is illustrated in Figure 2.

### Encoder

The encoder aims to extract the semantic representation of each constant needed for solving problems. However, the needed constants may come from either the given problem texts or domain knowledge, so we detail these two procedures as follows.

#### 3.1.1 Constant Representation Extraction

For each math word problem, we are given a passage consisting of words \(\{w_{t}^{P}\}_{t=1}^{m}\), whose word embeddings are \(\{e_{t}^{P}\}_{t=1}^{m}\). The problem text includes some numbers, which we refer as constants. The positions of constants in the problem text are denoted as \(\{p_{i}\}_{i=1}^{n}\). In order to capture the semantic representation of each constant by considering its contexts, a bidirectional long short-term memory (BLSTM) is adopted as the encoder [6]:

\[h_{t}^{E},c_{t}^{E}=\mathrm{BLSTM}(h_{t-1}^{E},c_{t-1}^{E},e_{t}^{P}), \tag{1}\]

and then for the \(i\)-th constant in the problem, its semantic representation \(e_{i}^{c}\) is modeled by the corresponding BLSTM output vector:

\[e_{i}^{c}=h_{p_{i}}^{E}. \tag{2}\]

#### 3.1.2 External Constant Leveraging

External constants, including \(1\) and \(\pi\), are leveraged, because they are required to solve a math word problem, but not mentioned in the problem text. Due to their absence from the problem text, we cannot extract their semantic meanings by BLSTM in (2). Instead, we model their semantic representation \(e^{\pi},e^{1}\) as parts of the model parameters. They are randomly initialized and are learned during model training.

### Decoder

The decoder aims at constructing the equation that can solve the given problem. We generate the equation by applying stack actions on a stack to mimic the way how human understands an equation. Human knows the semantic meaning of each term in the equation, even compositing of operands and operators like the term "\((10-1\times 5)\)" in Figure 1. Then what operator to apply on a pair operands can be chosen based on their semantic meanings accordingly. Hence we design our model to generate the equation in a postfix manner: a operator is chosen base on the semantic representations of two operands the operator is going to apply to. Note that the operands a operator can apply to can be any results generated previously. That is the reason why we use "stack" as our data structure in order to keep track of the operands a operator is going to apply to. The stack contains both symbolic and semantic representations of operands, denoted as

\[S=[(v_{l_{t}}^{S},e_{l_{t}}^{S}),(v_{l_{t}-1}^{S},e_{l_{t}-1}^{S}),\cdots,(v_{ 1}^{S},e_{1}^{S})], \tag{3}\]

where \(v^{S}\) of each pair is the symbolic part, such as \(x+1\), while \(e^{S}\) is the semantic representation, which is a vector. The components in the decoder are shown in the right part of Figure 2, each of which is detailed below.

### Decoding State Features

At each decoding step, decisions are made based on features of the current state. At each step, fea

Figure 2: The encoder-decoder model architecture of the proposed neural solver machine.

 

tures \(r^{sa}\) and \(r^{opd}\) are extracted to select a stack action (section 3.3.2) and an operand to push (section 3.3.3). Specifically, the features are the gated concatenation of following vectors:

* \(h_{t}^{D}\) is the output of an LSTM, which encodes the history of applied actions: \[h_{t}^{D},c_{t}^{D}=\mathrm{LSTM}(h_{t-1}^{D},c_{t-1}^{D},\mathrm{res}_{t-1}),\] (4) where \(\mathrm{res}_{t-1}\) is the result from the previous stack action similar to the seq2seq model Sutskever et al. (2014). For example, if the previous stack action \(o_{t-1}\) is "push", then \(\mathrm{res}_{t-1}\) is the semantic representation pushed into the stack. If the previous stack action \(o_{t-1}\) is to apply an operator \(\diamond\), then \(\mathrm{res}_{t-1}\) is the semantic representation generated by \(f_{\diamond}\).
* \(s_{t}\) is the stack status. It is crucial because some operators are only applicable to certain combinations of operand semantics, which is similar to the type system in programming languages. For example, operating multiplication is applicable to the combination of "_quantity of an item_" and "_price of an item_", while operating addition is not. Considering that all math operators supported here (\(+,-,\times,\div\)) are binary operators, the semantic representations of the stack's top 2 elements at the time \(t-1\) are considered: \[s_{t}=[e_{l_{t}}^{S};e_{l_{t}}^{S}].\] (5)
* \(q_{t}\) incorporates problem information in the decision. It is believed that the attention mechanism Luong et al. (2015) can effectively capture dependency for longer distance. Thus, the attention mechanism over the encoding problem \(h_{1}^{E},h_{2}^{E},\cdots\) is adopted: \[q_{t}=\mathrm{Attention}(h_{t}^{D},\{h_{i}^{E}\}_{i=1}^{m}),\] (6) where the attention function in this paper is defined as a function with learnable parameters \(w,W,b\): \[\mathrm{Attention}(u,\{v_{i}\}_{i=1}^{m})=\sum_{i=1}^{m}\alpha_{ i}h_{i},\] (7) \[\alpha_{i}=\frac{\exp(s_{i})}{\sum_{l=1}^{m}\exp(s_{i})},\] (8) \[s_{i}=w^{T}\tanh(W^{T}[u;v_{i}]+b).\] (9)

In order to model the dynamic features for different decoding steps, features in \(r_{t}^{sa}\) is gated as follows:

\[r_{t}^{sa} =[g_{t,1}^{sa}\cdot h_{t}^{D};g_{t,2}^{sa}\cdot s_{t};g_{t,3}^{ sa}\cdot q_{t}], \tag{10}\] \[g_{t}^{sa} =\sigma(Ws^{aa}\cdot[h_{t}^{D};s_{t};q_{t}]), \tag{11}\]

where \(\sigma\) is a sigmoid function and \(W^{sa}\) is a learned gating parameter. \(r_{t}^{opol}\) is defined similarly, but with a different learned gating parameter \(W^{opd}\).

#### 3.3.1 Stack Action Selector

The stack action selector is to select an stack action at each decoding step (section 3.3.2) until the unknowns are solved. The probability of choosing action \(a\) at the decoding step \(t\) is calculated with a network \(\mathrm{NN}\) constituted of one hidden layer and ReLU as the activation function:

\[P(Y_{t} |\{y_{i}\}_{i=1}^{t-1},\{w_{i}\}_{i=1}^{m}) \tag{12}\] \[=\mathrm{StackActionSelector}(r_{t}^{sa})\] \[=\mathrm{softmax}(\mathrm{NN}(r_{t}^{sa})),\]

where \(r_{t}^{sa}\) is decoding state features as defined in section 3.3.

Figure 3: Illustration of the inference process. The purple round blocks denote the transformed semantics, while the green ones are generated by the variable generator.

 



#### 3.3.2 Stack Actions

The available stack actions are listed below:

* **Variable generation**: The semantic representation of an unknown variable is generated dynamically as the first action in the decoding process. Note that this procedure provides the flexibility of solving problems with more than one unknown variables. The decoder module can decide how many unknown variables are required to solve the problem, and the semantic representation of the unknown variable is generated with an attention mechanism: \[e^{x}=\mathrm{Attention}(h_{t}^{D},\{h_{i}^{E}\}_{i=1}^{m}).\] (13)
* **Push**: This stack action pushes the operand chosen by the operand selector (section 3.3.3). Both the symbolic representation \(v_{*}\) and semantic representation \(e_{*}\) of the chosen operand would be pushed to the stack \(S\) in (3). Then the stack state becomes \[S=[(v_{*}^{S},e_{*}^{S}),(v_{l_{t}}^{S},e_{l_{t}}^{S}),\cdots,( v_{1}^{S},e_{1}^{S})].\] (14)
* **Operator \(\diamond\) application** (\(\diamond\in\{+,-,\times,\div\}\)): One stack action pops two elements from the top of the stack, which contains two pairs, \((v_{i},e_{i})\) and \((v_{j},e_{j})\), and then the associated symbolic operator, \(v_{k}=v_{i}\diamond v_{j}\), is recorded. Also, a semantic transformation function \(f_{\diamond}\) for that operator is invoked, which generates the semantic representation of \(v_{k}\) by transforming semantic representations of \(v_{i}\) and \(v_{j}\) to \(e_{k}=f_{\circ}(e_{i},e_{j})\). Therefore, after an operator is applied to the stack specified in (3), the stack state becomes \[S= [(v_{l_{t}}^{S}\diamond v_{l_{t}-1}^{S},f_{\diamond}(e_{l_{t}}^{ S},e_{l_{t}-1}^{S})),\] (15) \[(v_{l_{t}-2}^{S},e_{l_{t}-2}^{S}),\cdots,(v_{1}^{S},e_{1}^{S})].\]
* **Equal application**: When the equal application is chosen, it implies that an equation is completed. This stack action pops 2 tuples from the stack, \((v_{i},e_{i}),(v_{j},e_{j})\), and then \(v_{i}=v_{j}\) is recorded. If one of them is an unknown variable, the problem is solved. Therefore, after an OP is applied to the stack specified in (3), the stack state becomes \[S=[(v_{l_{t}-2}^{S},e_{l_{t}-2}^{S}),\cdots,(v_{1}^{S},e_{1}^{S})].\] (16)

#### 3.3.3 Operand Selector

When the stack action selector has decided to push an operand, the operand selector aims at choosing which operand to push. The operand candidates \(e\) include constants provided in the problem text whose semantic representations are \(e_{1}^{c},e_{2}^{c},\cdots,e_{n}^{c}\), unknown variable whose semantic representation is \(e^{x}\), and two external constants \(1\) and \(\pi\) whose semantic representations are \(e^{1},e^{\pi}\):

\[e=[e_{1}^{c},e_{2}^{c},\cdots,e_{n}^{c},e^{1},e^{\pi},e^{x}]. \tag{17}\]

An operand has both symbolic and semantic representations, but the selection focuses on its semantic meaning; this procedure is the same as what human does when solving math word problems.

Inspired by addressing mechanisms of neural Turing machine (NTM) Graves et al. (2014), the probability of choosing the \(i\)-th operand candidate is the attention weights of \(r_{t}\) over the semantic representations of the operand candidates as in (8):

\[P(Z_{t}\mid\{y_{i}\}_{i=1}^{t-1},\{w_{i}\}_{i=1}^{m}) \tag{18}\] \[=\mathrm{OperandSelector}(r_{t}^{opd})\] \[=\mathrm{AttentionWeight}(r_{t}^{opd},\{e_{i}\}_{i=1}^{m}\cup\{e^{1 },e^{\pi},e^{x}\}),\]

and \(r_{t}^{opd}\) is defined in section 3.3.

#### 3.3.4 Semantic Transformer

A semantic transformer is proposed to generate the semantic representation of a new symbol resulted from applying an operator, which provides the capability of interpretation and reasoning for the target task. The semantic transformer for an operator \(\diamond\in\{+,-,\times,\div\}\) transforms semantic representations of two operands \(e_{1},e_{2}\) into

\[f_{\circ}(e_{1},e_{2})=\tanh(U_{\diamond}\mathrm{ReLU}(W_{\diamond}[e_{1};e_{ 2}]+b_{\diamond})+c_{\diamond}), \tag{19}\]

where \(W_{\diamond},U_{\diamond},b_{\diamond},c_{\diamond}\) are model parameters. Semantic transformers for different operators have different parameters in order to model different transformations.

### Training

Both stack action selection and operand selection can be trained in a fully supervised way by giving problems and associated ground truth equations. Because our model generates the equation with stack actions, the equation is first transformed into its postfix representation. Let the postfix representation of the target equation be

where \(y_{t}\) can be either an operator (\(+,-,\times,\div,=\)) or a target operand. Then for each time step \(t\), the loss can be computed as

\[L(y_{t})=\begin{cases}L_{1}(\mathrm{push\_op})+L_{2}(y_{t})&$y_{t}$ is an operand\\ L_{1}(y_{t})&$otherwise$\end{cases},\]

where \(L_{1}\) is the stack action selection loss and \(L_{2}\) is the operand selection loss defined as

\[L_{1}(y_{t}) =-\log P(Y_{t}=y_{t}~{}|~{}\{o_{i}\}_{i=1}^{t-1},\{w_{i}\}_{i=1}^ {m}),\] \[L_{2}(y_{t}) =-\log P(Z_{t}=y_{t}~{}|~{}r_{t}).\]

The objective of our training process is to minimize the total loss for the whole equation, \(\sum_{t=1}^{T}L(y_{t})\).

### Inference

When performing inference, at each time step \(t\), the stack action with the highest probability \(P(Y_{t}|\{\tilde{y}_{i}\}_{i=1}^{t-1},\{w_{i}\}_{i=1}^{m})\) is chosen. If the chosen stack action is "_pusl_", the operand with the highest probability \(P(Z_{t}|\{\tilde{Y}_{i}\}_{i=1}^{t-1},\{w_{i}\}_{i=1}^{m})\) is chosen. When the stack has less than 2 elements, the probability of applying operator \(+,-,\times,\div,=\) would be masked out to prevent illegal stack actions, so all generated equations must be legal math expressions. The decoder decodes until the unknown variable can be solved. After the equations are generated, a Python package SymPy (Meurer et al., 2017) is used to solve the unknown variable. The inference procedure example is illustrated in Figure 3. The detailed algorithm can be found in Algorithm 1.

```
functionSolveProblem(problem_text) \(v\leftarrow\text{ExtractConstants}(problem\_text)\triangleright v\) is a list of constants in the problem. \(h^{E},h_{c}^{D},o_{c}^{D},E\leftarrow\text{Encoder}(problem\_text{text})\) \(S\leftarrow\text{Stack}\)  ret, \(\operatorname*{loss},\operatorname*{t},\operatorname*{equations}\gets \operatorname*{padding},0,1,\{\}\) while not solvableequations\(\mathbf{do}\) \(h_{t}^{D}\leftarrow\operatorname*{LSTM}(h_{t-1}^{D},e_{t-1},\operatorname*{ret})\) \(s_{t}\gets S.\operatorname*{get.pop}2^{D}\) \(h^{E}\leftarrow\operatorname*{Attention}(h_{t-1}^{D},h^{E})\) \(r_{t}\leftarrow[h_{t}^{D},s_{t},h^{E}]\) \(p_{sa}\leftarrow\text{StackActionSelector}(r_{t})\) \(p_{polq}\leftarrow\text{OperandSelector}(r_{t})\) iftrainingthen \(\triangleright\) Target equation \(y\) is available when training. \(Y_{t}\gets y_{t}\) if\(y_{t}\)is operand then \(\operatorname*{loss}\leftarrow\operatorname*{loss}+L_{1}(\mathrm{push})+L_{2}(y_{ t})\) else \(\operatorname*{loss}\leftarrow\operatorname*{loss}+L_{1}(y_{t})\) endif else \(Y_{t}\leftarrow\text{StackActionSelector}(r_{t}^{sa})\) if\(Y_{t}=\text{push}\)then \(Z_{t}\leftarrow\text{OperandSelector}(r_{t}^{apd})\) endif endif if\(Y_{t}=\text{gen.var}\)then \(e^{x}\leftarrow\text{Attention}(h_{t}^{D},h^{E})\)  ret \(e^{x}\leftarrow e^{x}\) elseif\(Y_{t}=\text{push}\)then \(S.\text{push}(y_{x_{t}},z_{x_{t}})\)  ret \(\gets z_{t}\) elseif\(Y_{t}\in\{+,-,\times,+\}\)then \((v_{a},e_{a}),(v_{b},e_{b})=S.\text{pop}(),S.\text{pop}()\) \(S.\text{push}(x_{t}v_{b}),f_{x}(e_{a},e_{b}))\)  ret \(\leftarrow f_{y}(e_{a},e_{b})\) elseif\(Y_{t}=\text{equal}\)then \((v_{a},e_{a}),(v_{b},e_{b})=S.\text{pop}()\),S.\text{pop}()\)  equations = equations \(\cup\)''\(v_{a}=v_{b}\)'' \(\text{ret}\gets S.\text{top}()\) endif endwhile returnsolve(equations) endfunction
```

**Algorithm 1** Training and Inference

## 4 Experiments

To evaluate the performance of the proposed model, we conduct the experiments on the benchmark dataset and analyze the learned semantics.

### Settings

The experiments are benchmarked on the dataset Math23k Wang et al. (2017), which contains 23,162 math problems with annotated equations. Each problem can be solved by a single-unknown-variable equation and only uses operators \(+,-,\times,\div\). Also, except \(\pi\) and \(1\), quantities in the equation can be found in the problem text. There are also other large scale datasets like Dolphin18K Shi et al. (2015) and AQuA Ling et al. (2017), containing 18,460 and 100,000 math word problems respectively. The reasons about not evaluating on these two datasets are 1) Dolphin18k contains some unlabeled math word problems and some incorrect labels, and 2) AQuA contains rational for solving the problems, but the equations in the rational are not formal (e.g. mixed with texts, using \(x\) to represent \(\times\), etc.) and inconsistent. Therefore, the following experiments are performed and analyzed using Math23K, the only large scaled, good-quality dataset.

### Results

The results are shown in Table 1. The retrieval-based methods compare problems in test data with problems in training data, and choose the most 

similar one's template to solve the problem (Kushman et al., 2014; Upadhyay and Chang, 2017). The classification-based models choose equation templates by a classifier trained on the training data. Their performance are reported in Robaidek et al.. The seq2seq and hybrid models are from Wang et al., where the former directly maps natural language into symbols in equations, and the latter one ensembles prediction from a seq2seq model and a retrieval-based model. The ensemble is the previous state-of-the-art results of Math23K.

Our proposed end-to-end model belongs to the generation category, and the single model performance achieved by our proposed model is new state-of-the-art (\(>65\%\)) and even better than the hybrid model result (64.7%). In addition, we are the first to report character-based performance on this dataset, and the character-based results are slightly better than the word-based ones. Among the single model performance, our models obtain about more than 7% accuracy improvement compared to the previous best one (Wang et al., 2017). The performance of our character-based model also shows that our model is capable of learning the relatively accurate semantic representations without word boundaries and achieves better performance.

### Ablation Test

To better understand the performance contributed by each proposed component, we perform a series of ablation tests by removing components one by one and then checking the performance by 5-fold cross validation. Table 2 shows the ablation results.

Char-Based v.s. Word-BasedAs reported above, using word-based model instead of character-based model only causes 0.5% performance drop. To fairly compare with prior word-based models, the following ablation tests are performed on the word-based approach.

Word-Based - GateIt uses \(r_{t}\) instead of \(r^{sa}_{t}\) and \(r^{opr}_{t}\) as the input of both \(\mathrm{StackActionSelector}\) and \(\mathrm{OperandSelector}\).

Word-Based - Gate - AttentionConsidering that the prior generation-based model (seq2seq) did not use any attention mechanism, we compare the models with and without the attention mechanism. Removing attention means excluding \(q_{t-1}\) in (11), so the input of both operator and operand selector becomes \(r_{t}=[h_{t}^{D};s_{t}]\). The result implies that our model is not better than previous models solely because of the attention.

Word-Based - Gate - Attention - StackTo check the effectiveness of the stack status (\(s_{t}\) in (11)), the experiments of removing the stack status from the input of both operator and operand selectors (\(r_{t}=h_{t}^{D}\)) are conducted. The results well justify our idea of choosing operators based on semantic meanings of operands.

Word-Based - Semantic TransformerTo validate the effectiveness of the idea that views an operator as a semantic transformer, we modify the semantic transformer function of the operator \(\diamond\) into \(f_{\diamond}(e_{1},e_{2})=e_{\diamond}\), where \(e_{\diamond}\) is a learnable parameter and is different for different operators. Therefore, \(e_{\diamond}\) acts like the embedding of the operator \(\diamond\), and the decoding process is more similar to a general seq2seq model. The results show that the semantic transformer in the original model encodes not only the last operator applied on the operands but other information that helps the selectors.

Word-Based - Semantic RepresentationTo explicitly evaluate the effectiveness of operands' semantic representations, we rewrite semantic representation of the \(i\)-th operand in the problem texts

\begin{table}
\begin{tabular}{l l c} \hline
**Model** & \multicolumn{1}{c}{**Accuracy**} \\ \hline \hline Retrieval & Jaccard & 47.2\% \\  & Cosine & 23.8\% \\ \hline Classification & BLSTM & 57.9\% \\  & Self-Attention & 56.8\% \\ \hline Generation & Seq2Seq w/ SNI & 58.1\% \\  & Proposed Word-Based & **65.3\%** \\  & Proposed Char-Based & **65.8\%** \\ \hline Hybrid & Retrieval + Seq2Seq & 64.7\% \\ \hline \end{tabular}
\end{table}
Table 1: 5-fold cross validation results on Math23K.

\begin{table}
\begin{tabular}{p{113.8pt} p{113.8pt}} \hline
* Gate & 64.1\% \\ Word-Based
- Stack & 62.5\% \\ Word-Based
- Semantic Transformer & 60.1\% \\ Word-Based
- Semantic Representation & 61.7\% \\ \hline \end{tabular}
\end{table}
Table 2: 5-fold cross validation results of ablation tests.
 

from (2) to \(e_{i}^{c}=b_{i}^{c}\), where \(b_{i}^{c}\) is a parameter. Thus for every problem, the representation of the \(i\)-th operand is identical, even though their meanings in different problems may be different. This modification assumes that no semantic information is captured by \(b_{i}^{c}\), which can merely represent a symbolic placeholder in an equation. Because the semantic transformer is to transform the semantic representations, applying this component is meaningless. Here the semantic transformer is also replaced with \(f_{\diamond}(e_{1},e_{2})=e_{\diamond}\) as the setting of the previous ablation test. The results show that the model without using semantic representations of operands causes a significant accuracy drop of 3.5%. The main contribution of this paper about modeling semantic meanings of symbols is validated and well demonstrated here.

## 5 Qualitative Analysis

To further analyze whether the proposed model can provide interpretation and reasoning, we visualize the learned semantic representations of constants to check where the important cues are,

### Constant Embedding Analysis

To better understand the information encoded in the semantic representations of constants in the problem, a self-attention is performed when their semantic representations are extracted by the encoder. Namely, we rewrite (2) as

\[e_{i}^{c}=\mathrm{Attention}(h_{p_{i}}^{E},\{h_{t}^{E}\}_{t=1}^{m}. \tag{20}\]

Then we check the trained self-attention map (\(\alpha\) in the attention function) on the validation dataset.

For some problems, the self-attention that generates semantic representations of constants in the problem concentrates on the number's quantifier or unit, and sometimes it also focuses on informative verbs, such as "_gain_", "_get_", "_fill_", etc., in the sentence. For example, Figure 4 shows the attention weights for an example math word problem, where lighter colors indicate higher weights. The numbers "_58_" and "6" focus more on the quantifier-related words (e.g. "_every_" and "_how many_"), while "9" pays higher attention to the verb "_fill_". The results are consistent with those hand-craft features for solving math word problems proposed by the prior research Hosseini et al. (2014); Roy and Roth (2015); Roy et al. (2015). Hence, we demonstrate that the automatically learned semantic representations indeed capture critical information that facilitates solving math word problems without providing human-crafted knowledge.

### Decoding Process Visualization

We visualize the attention map (\(q_{t}\) in (6)) to see how the attention helps the decoding process. An example is shown in the top of Figure 5, where most attention focuses on the end of the sentence. Unlike the machine translation task, the attention shows the word-level alignment between source and target languages, solving math word problems requires high-level understanding due to the task complexity.

To further analyze the effectiveness of the proposed gating mechanisms for stack action and operand selection, the activation of gates \(g^{sa},g^{opd}\) at each step of the decoding process is shown in the bottom of Figure 5. It shows that most of time, the gate activation is high, demonstrating that the proposed gating mechanisms play an important role during decoding. We also observe a common phenomenon that the activation \(g_{2}^{sa}\), which controls how much attention the stack action selector puts on the stack state when deciding an stack action, is usually low until the last "operator application" stack action. For example, in the example of Figure 5, \(g_{2}^{sa}\) is less than \(0.20\) till the last argument selection stack action, and activates when deciding the _division operator application_ (\(\div\)) and the _equal application_ (\(=\)). It may result from the higher-level semantics of the operand \((6.75-2.75)\) on the stack when selecting the stack action _division operator application_ (\(\div\)). In terms

Figure 4: The self-attention map visualization of operands’ semantic expressions for the problem “_There are 58 bananas. Each basket can contain 6 bananas. How many bananas are needed to be token off such that exactly 9 baskets are filled?_”.

 

of the activation of \(g^{opd}\), we find that three features are important in most cases, demonstrating the effectiveness of the proposed mechanisms.

### Error Analysis

We randomly sample some results predicted incorrectly by our model shown in Table 3. In the first example, the error is due to the language ambiguity, and such ambiguity cannot be resolved without considering the exact value of the number. From the second example, although our model identifies the problem as a comparison problem successfully, it handles the order of the operands incorrectly. For the third problem, it cannot be solved by using only the surface meaning but requires some common sense. Therefore, above phenomena show the difficulty of solving math word problems and the large room for improvement.

## 6 Conclusion

We propose an end-to-end neural math solver using an encoder-decoder framework that incorporates semantic representations of numbers in order to generate mathematical symbols for solving math word problems. The experiments show that the proposed model achieves the state-of-the-art performance on the benchmark dataset, and empirically demonstrate the effectiveness of each component in the model. In sum, the proposed neural math solver is designed based on how human performs reasoning when writing equations, providing better interpretation without the need of labeled rationals.

 



## References

* Graves et al. (2014) Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural turing machines. _arXiv preprint arXiv:1410.5401_.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Computation_, 9(8):1735-1780.
* Hosseini et al. (2014) Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing_, pages 523-533.
* Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Herve Jegou, and Tomas Mikolov. 2016. Fasttext.zip: Compressing text classification models. _arXiv preprint arXiv:1612.03651_.
* Kingma and Ba (2014) Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _CoRR_, abs/1412.6980.
* Koncel-Kedziorski et al. (2015) Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equations. _TACL_, 3:585-597.
* Kushman et al. (2014) Nate Kushman, Luke Zettlemoyer, Regina Barzilay, and Yoav Artzi. 2014. Learning to automatically solve algebra word problems. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014_, pages 271-281.
* Ling et al. (2017) Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017_, pages 158-167.
* Luong et al. (2015) Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1412-1421.
* Mandal and Nakkar (2019) Sourav Mandal and Sudip Kumar Taskar. 2019. Solving arithmetic mathematical word problems: A review and recent advancements. In _Information Technology and Applied Mathematics_, pages 95-114. Springer.
* Mehta et al. (2017) Purvanshi Mehta, Pruthwik Mishra, Vinayak Athavale, Manish Shrivastava, and Dipti Misra Sharma. 2017. Deep neural network based system for solving arithmetic word problems. In _Proceedings of the IJCNLP 2017_, pages 65-68.
* Meurer et al. (2017) Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondrej Certik, Sergey B. Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thinlina Rathnayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, Stepan Roucka, Ashutosh Sahoo, Isuru Fernando, Sumith Kulat, Robert Cimrman, and Anthony Scopatz. 2017. Sympy: symbolic computing in python. _PeerJ Computer Science_, 3:e103.
* Robajdek et al. (2018) Benjamin Robajdek, Rick Koncel-Kedziorski, and Hannaneh Hajishirzi. 2018. Data-driven methods for solving algebra word problems. _CoRR_, abs/1804.10718.
* Roy and Roth (2015) Subbro Roy and Dan Roth. 2015. Solving general arithmetic word problems. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015_, pages 1743-1752.
* Roy and Roth (2018) Subbro Roy and Dan Roth. 2018. Mapping to declarative knowledge for word problem solving. _TACL_, 6:159-172.
* Roy et al. (2016) Subbro Roy, Shyam Upadhyay, and Dan Roth. 2016. Equation parsing : Mapping sentences to grounded equations. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_, pages 1088-1097.
* Roy et al. (2015) Subbro Roy, Tim Vieira, and Dan Roth. 2015. Reasoning about quantities in natural language. _TACL_, 3:1-13.
* Shi et al. (2015) Shuming Shi, Yuehui Wang, Chin-Yew Lin, Xiaojiang Liu, and Yong Rui. 2015. Automatically solving number word problems by semantic parsing and reasoning. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015_, pages 1132-1142.
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In _Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014_, pages 3104-3112.
* Upadhyay and Chang (2017) Shyam Upadhyay and Ming-Wei Chang. 2017. Annotating derivations: A new evaluation strategy and dataset for algebra word problems. In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics_, pages 494-504.
* Upadhyay et al. (2016) Shyam Upadhyay, Ming-Wei Chang, Kai-Wei Chang, and Wen-tau Yih. 2016. Learning from explicit and implicit supervision jointly for algebra word problems. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 297-306.
* Wang et al. (2018) Lei Wang, Dongxiang Zhang, Lialni Gao, Jingkuan Song, Long Guo, and Heng Tao Shen. 2018. MathDQN: Solving arithmetic word problems via deep

reinforcement learning. In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence_.
* Wang et al. (2017) Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 845-854.

## Appendix A Algorithm Detail

The training and inference procedures are shown in Algortihm 1.

## Appendix B Hyperparameter Setup

The model is trained with the optimizer adamKingma and Ba (2014), and the learning rate is set to \(0.001\). Pretrained embeddings using FastText Joulin et al. (2016) are adopted. The hidden state size of LSTM used in the encoder and decoder is 256. The dimension of hidden layers in attention, semantic transformer and operand/stack action selector is 256. The dropout rate is set as \(0.1\) before inputting the decoder LSTM, before the stack action selector and after the hidden layer of the stack action selector and attention. The reported accuracy is the result of 5-fold cross-validation, same as Wang et al. for fair comparison.

## Appendix C Error Analysis between Seq2Seq

We implement the seq2seq model as proposed by Wang et al. and compare the performance difference between our proposed model and the baseline seq2seq model. Table 4 shows the generated results seq2seq predicts correctly but our model predicts incorrectly. Table 5 show the results our model can predict correctly but seq2seq cannot 



**Problem & Results**

\begin{tabular}{l l} \hline
* [noitemsep] -1em \(\downarrow\) & 5 \(\downarrow\) , \(\varotimes\) , \(\varotimes\) , \(\varotimes\) \\ \end{tabular}  ? (There are 5 people in front of Little Red and 7 people behind. How many persons are there in total?)

_Proposed Model_: \(5+7\)

_Seq2Seq Model_: \(5+7+1\)

* [noitemsep] -2em \(\uparrow\) & 3 , \(\varotimes\) , \(\v



\begin{table}
\begin{tabular}{l l} \hline
**Problem \& Results** \\ \hline \hline  & \(\bigcirc\) \\ \end{tabular} (In hospital, it is often necessary to give glucose injection to patient. This glucose water is prepared by mixing glucose and water at 1:19. Based on this information, what do you know?)

_Proposed Model_: \(1\div(1+19.0)\)

_Seq2Seq Model_: \(1\times(1+19.0)\)

* [noitemsep]