Venue: NAACL 2021

# Are NLP Models really able to Solve Simple Math Word Problems?

Arkil Patel Satwik Bhattacharya Navin Goyal

Microsoft Research India

arkil.patel@gmail.com, {t-satbh,navingo}@microsoft.com

###### Abstract

The problem of designing NLP solvers for math word problems (MWP) has seen sustained research activity and steady gains in the test accuracy. Since existing solvers achieve high performance on the benchmark datasets for elementary level MWPs containing one-unknown arithmetic word problems, such problems are often considered "solved" with the bulk of research attention moving to more complex MWPs. In this paper, we restrict our attention to English MWPs taught in grades four and lower. We provide strong evidence that the existing MWP solvers rely on shallow heuristics to achieve high performance on the benchmark datasets. To this end, we show that MWP solvers that do not have access to the question asked in the MWP can still solve a large fraction of MWPs. Similarly, models that treat MWPs as bag-of-words can also achieve surprisingly high accuracy. Further, we introduce a challenge dataset, SVAMP, created by applying carefully chosen variations over examples sampled from existing datasets. The best accuracy achieved by state-of-the-art models is substantially lower on SVAMP, thus showing that much remains to be done even for the simplest of the MWPs.

+
Footnote †: _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2080–2094

June 6-11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

A Math Word Problem (MWP) consists of a short natural language narrative describing a state of the world and poses a question about some unknown quantities (see Table 1 for some examples). MWPs are taught in primary and higher schools. The MWP task is a type of semantic parsing task where given an MWP the goal is to generate an expression (more generally, equations), which can then be evaluated to get the answer. The task is challenging because a machine needs to extract relevant information from natural language text as well as perform mathematical reasoning to solve it. The complexity of MWPs can be measured along multiple axes, e.g., reasoning and linguistic complexity and world and domain knowledge. A combined complexity measure is the _grade level_ of an MWP, which is the grade in which similar MWPs are taught. Over the past few decades many approaches have been developed to solve MWPs with significant activity in the last decade Zhang et al. (2020).

MWPs come in many varieties. Among the simplest are the one-unknown arithmetic word problems where the output is a mathematical expression involving numbers and one or more arithmetic operators \((+,-,*,/)\). Problems in Tables 1 and 6 are of this type. More complex MWPs may have systems of equations as output or involve other operators or may involve more advanced topics and specialized knowledge. Recently, researchers have started focusing on solving such MWPs, e.g. multiple-unknown linear word problems Huang et al. (2016a), geometry Sachan and Xing (2017) and probability Amini et al. (2019), believing that existing work can handle one-unknown arithmetic MWPs well Qin et al. (2020). In this paper, we question the capabilities of the state-of-the-art

\begin{table}
\begin{tabular}{l l} \hline \hline Problem: & \\ Text: Jack had 8 pens and Mary had 5 pens. Jack gave 3 \\ pens to Mary. How many pens does Jack have now? \\ Equation: 8 - 3 = 5 \\ \hline Question Sensitivity Variation: & \\ Text: Jack had 8 pens and Mary had 5 pens. Jack gave 3 \\ pens to Mary. How many pens does Mary have now? \\ Equation: 5 + 3 = 8 \\ \hline Reasoning Ability Variation: & \\ Text: Jack had 8 pens and Mary had 5 pens. Mary gave 3 \\ pens to Jack: How many pens does Jack have now? \\ Equation: 8 + 3 = 11 \\ \hline Structural Invariance Variation: & \\ Text: Jack gave 3 pens to Mary. If Jack had 8 pens and Mary had 5 pens initially, how many pens does Jack have now? \\ Equation: 8 - 3 = 5 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Example of a Math Word Problem along with the types of variations that we make to create SVAMP.



(SOTA) methods to robustly solve even the simplest of MWPs suggesting that the above belief is not well-founded.

In this paper, we provide concrete evidence to show that existing methods use shallow heuristics to solve a majority of word problems in the benchmark datasets. We find that existing models are able to achieve reasonably high accuracy on MWPs from which the question text has been removed leaving only the narrative describing the state of the world. This indicates that the models can rely on superficial patterns present in the narrative of the MWP and achieve high accuracy without even looking at the question. In addition, we show that a model without word-order information (i.e., the model treats the MWP as a bag-of-words) can also solve the majority of MWPs in benchmark datasets.

The presence of these issues in existing benchmarks makes them unreliable for measuring the performance of models. Hence, we create a challenge set called SVAMP (**S**imple **V**ariations on **A**rithmetic **M**ath word **P**roblems; pronounced _swamp_) of one-unknown arithmetic word problems with grade level up to 4 by applying simple variations over word problems in an existing dataset (see Table 1 for some examples). SVAMP further highlights the brittle nature of existing models when trained on these benchmark datasets. On evaluating SOTA models on SVAMP, we find that they are not even able to solve half the problems in the dataset. This failure of SOTA models on SVAMP points to the extent to which they rely on simple heuristics in training data to make their prediction.

Below, we summarize the two broad contributions of our paper.

* We show that the majority of problems in benchmark datasets can be solved by shallow heuristics lacking word-order information or lacking question text.
* We create a challenge set called SVAMP 1 for more robust evaluation of methods developed to solve elementary level math word problems. Footnote 1: The dataset and code are available at: [https://github.com/arkilpatel/SVAMP](https://github.com/arkilpatel/SVAMP)

## 2 Related Work

**Math Word Problems.** A wide variety of methods and datasets have been proposed to solve MWPs; e.g. statistical machine learning Roy and Roth (2018), semantic parsing Huang et al. (2017) and most recently deep learning Wang et al. (2017); Xie and Sun (2019); Zhang et al. (2020); see Zhang et al. (2020) for an extensive survey. Many papers have pointed out various deficiencies with previous datasets and proposed new ones to address them. Koncel-Kedziorski et al. (2016) curated the MAWPS dataset from previous datasets which along with Math23k Wang et al. (2017) has been used as benchmark in recent works. Recently, ASDiv Miao et al. (2020) has been proposed to provide more diverse problems with annotations for equation, problem type and grade level. HMWP Qin et al. (2020) is another newly proposed dataset of Chinese MWPs that includes examples with muliple-unknown variables and requiring non-linear equations to solve them.

**Identifying artifacts in datasets** has been done for the Natural Language Inference (NLI) task by McCoy et al. (2019), Poliak et al. (2018), and Gururangan et al. (2018). Rosenman et al. (2020) identified shallow heuristics in a Relation Extraction dataset. Cai et al. (2017) showed that biases prevalent in the ROC stories cloze task allowed models to yield state-of-the-art results when trained only on the endings. To the best of our knowledge, this kind of analysis has not been done on any Math Word Problem dataset.

**Challenge Sets** for NLP tasks have been proposed most notably for NLI and machine translation Belinkov and Glass (2019); Nie et al. (2020); Ribeiro et al. (2020). Gardner et al. (2020) suggested creating _contrast sets_ by manually perturbing test instances in small yet meaningful ways that change the gold label. We believe that we are the first to introduce a challenge set targeted specifically for robust evaluation of Math Word Problems.

## 3 Background

### Problem Formulation

We denote a Math Word Problem \(P\) by a sequence of \(n\) tokens \(P=({\mathbf{w}}_{1},\ldots,{\mathbf{w}}_{n})\) where each token \({\mathbf{w}}_{i}\) can be either a word from a natural language or a numerical value. The word problem \(P\) can be broken down into _body_\(B=({\mathbf{w}}_{1},\ldots,{\mathbf{w}}_{k})\) and _question_\(Q=({\mathbf{w}}_{k+1},\ldots,{\mathbf{w}}_{n})\). The goal is to map \(P\) to a valid mathematical expression \(E_{P}\) composed of numbers from \(P\) and mathematical operators from the set \(\{+,-,/,*\}\) (e.g. \(3+5-4\)). The metric used to evaluate models on the MWP task is Execution Accuracy, which is obtained from com 

paring the predicted answer (calculated by evaluating \(E_{P}\)) with the annotated answer. In this work, we focus only on one-unknown arithmetic word problems.

### Datasets and Methods

Many of the existing datasets are not suitable for our analysis as either they are in Chinese, e.g. Math23k Wang et al. (2017) and HMWP Qin et al. (2020), or have harder problem types, e.g. Dolphin18K Huang et al. (2016b). We consider the widely used benchmark MAWPS Koncel-Kedziorski et al. (2016) composed of 2373 MWPs and the arithmetic subset of ASDiv Miao et al. (2020) called ASDiv-A which has 1218 MWPs mostly up to grade level 4 (MAMPS does not have grade level information). Both MAWPS and ASDiv-A are evaluated on 5-fold cross-validation based on pre-assigned splits.

We consider three models in our experiments:

**(a) Seq2Seq** consists of a Bidirectional LSTM Encoder to encode the input sequence and an LSTM decoder with attention Luong et al. (2015) to generate the equation.

**(c) GTS Xie and Sun (2019) uses an LSTM Encoder to encode the input sequence and a tree-based Decoder to generate the equation.

**(d) Graph2Tree Zhang et al. (2020) combines a Graph-based Encoder with a Tree-based Decoder.

The performance of these models on both datasets is shown in Table 2. We either provide RoBERTa Liu et al. (2019) pre-trained embeddings to the models or train them from scratch. Graph2Tree Zhang et al. (2020) with RoBERTa embeddings achieves the state-of-the-art for both datasets. Note that our implementations achieve a higher score than the previously reported highest score of 78% on ASDiv-A Miao et al. (2020) and 83.7% on MAWPS Zhang et al. (2020). The implementation details are provided in Section A in the Appendix.

## 4 Deficiencies in existing datasets

Here we describe the experiments that show that there are important deficiencies in MAWPS and ASDiv-A.

### Evaluation on Question-removed MWPs

As mentioned in Section 3.1, each MWP consists of a body \(B\), which provides a short narrative on a state of the world and a question \(Q\), which inquires about an unknown quantity about the state of the world. For each fold in the provided 5-fold split in MAWPS and ASDiv-A, we keep the train set unchanged while we remove the questions \(Q\) from the problems in the test set. Hence, each problem in the test set consists of only the body \(B\) without any question \(Q\). We evaluate all three models with RoBERTa embeddings on these datasets. The results are provided in Table 3.

The best performing model is able to achieve a 5-fold cross-validation accuracy of 64.4% on ASDiv-A and 77.7% on MAWPS. Loosely translated, this means that nearly 64% of the problems in ASDiv-A and 78% of the problems in MAWPS can be correctly answered without even looking at the question. This suggests the presence of patterns in the bodies of MWPs in these datasets that have a direct correlation with the output equation.

Some recent works have also demonstrated similar evidence of bias in NLI datasets Gururangan et al. (2018); Poliak et al. (2018). They observed that NLI models were able to predict the correct label for a large fraction of the standard NLI datasets based on only the hypothesis of the input and without the premise. Our results on question-removed examples of math word problems resembles their observations on NLI datasets and similarly indicates the presence of artifacts that help statistical

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Model** & **MAWPS** & **ASDiv-A** \\ \hline Seq2Seq (S) & 79.7 & 55.5 \\ Seq2Seq (R) & 86.7 & 76.9 \\ \hline GTS S (Xie and Sun, 2019) & 82.6 & 71.4 \\ GTS R & 88.5 & 81.2 \\ \hline Graph2Tree (S) Zhang et al. (2020) & 83.7 & 77.4 \\ Graph2Tree (R) & **88.7** & **82.2** \\ \hline Majority Template Baseline2  & 17.7 & 21.2 \\ \hline \hline \end{tabular}
\end{table}
Table 2: 5-fold cross-validation accuracies (\(\uparrow\)) of baseline models on datasets. (R) means that the model is provided with RoBERTa pretrained embeddings while (S) means that the model is trained from scratch.

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Model** & **MAWPS** & **ASDiv-A** \\ \hline Seq2Seq & 77.4 & 58.7 \\ GTS & 76.2 & 60.7 \\ Graph2Tree & **77.7** & **64.4** \\ \hline \hline \end{tabular}
\end{table}
Table 3: 5-fold cross-validation accuracies (\(\uparrow\)) of baseline models on Question-removed datasets.

 

models predict the correct answer without complete information. Note that even though the two methods appear similar, there is an important distinction. In Gururangan et al. (2018), the model is _trained_ and tested on hypothesis only examples and hence, the model is forced to find artifacts in the hypothesis during _training_. On the other hand, our setting is more natural since the model is trained in the standard way on examples with both the body and the question. Thus, the model is not explicitly forced to learn based on the body during training and our results not only show the presence of artifacts in the datasets but also suggest that the SOTA models exploit them.

Following Gururangan et al. (2018), we attempt to understand the extent to which SOTA models rely on the presence of simple heuristics in the body to predict correctly. We partition the test set into two subsets for each model: problems that the model predicted correctly without the question are labeled _Easy_ and the problems that the model could not answer correctly without the question are labeled _Hard_. Table 4 shows the performance of the models on their respective _Hard_ and _Easy_ sets. Note that their performance on the full set is already provided in Table 2. It can be seen clearly that although the models correctly answer many _Hard_ problems, the bulk of their success is due to the _Easy_ problems. This shows that the ability of SOTA methods to robustly solve word problems is overestimated and that they rely on simple heuristics in the body of the problems to make predictions.

### Performance of a constrained model

We construct a simple model based on the Seq2Seq architecture by removing the LSTM Encoder and replacing it with a Feed-Forward Network that maps the input embeddings to their hidden representations. The LSTM Decoder is provided with the **average** of these hidden representations as its initial hidden state. During decoding, an attention mechanism Luong et al. (2015) assigns weights to individual hidden representations of the input tokens. We use either RoBERTa embeddings (non-contextual; taken directly from Embedding Matrix) or train the model from scratch. Clearly, this model does not have access to word-order information.

Table 5 shows the performance of this model on MAWPS and ASDiv-A. The constrained model with non-contextual RoBERTa embeddings is able to achieve a cross-validation accuracy of \(51.2\) on ASDiv-A and an astounding 77.9 on MAWPS. It is surprising to see that a model having no word-order information can solve a majority of word problems in these datasets. These results indicate that it is possible to get a good score on these datasets by simply associating the occurence of specific words in the problems to their corresponding equations. We illustrate this more clearly in the next section.

### Analyzing the attention weights

To get a better understanding of how the constrained model is able to perform so well, we analyze the attention weights that it assigns to the hidden representations of the input tokens. As shown by Wiegreffe and Pinter (2019), analyzing the attention weights of our constrained model is a reliable way to explain its prediction since each hidden representation consists of information about only that token as opposed to the case of an RNN where each hidden representation may have information about the context i.e. its neighboring tokens.

We train the contained model (with RoBERTa embeddings) on the full ASDiv-A dataset and observe the attention weights it assigns to the words of the input problems. We found that the model usually attends to a single word to make its prediction, irrespective of the context. Table 6 shows some representative examples. In the first example, the model assigns an attention weight of 1 to the representation of the word _'every'_ and predicts the correct equation. However, when we make a subtle change to this problem such that the corresponding equation changes, the model keeps on attending over the word _'every'_ and predicts the same equa

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{2}{c}{**MAWPS**} & \multicolumn{2}{c}{**ASDiv-A**} \\ \cline{2-5}
**Model** & _Easy_ & _Hard_ & _Easy_ & _Hard_ \\ \hline Seq2Seq & 86.8 & 86.7 & 91.3 & 56.1 \\ GTS & 92.6 & 71.7 & 91.6 & 65.3 \\ Graph2Tree & 93.4 & 71.0 & 92.8 & 63.3 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Results of baseline models on the _Easy_ and _Hard_ test sets.

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Model** & **MAWPS** & **ASDiv-A** \\ \hline FFN + LSTM Decoder (S) & 75.1 & 46.3 \\ FFN + LSTM Decoder (R) & **77.9** & **51.2** \\ \hline \hline \end{tabular}
\end{table}
Table 5: 5-fold cross-validation accuracies (\(\uparrow\)) of the constrained model on the datasets. (R) denotes that the model is provided with non-contextual RoBERTa pretrained embeddings while (S) denotes that the model is trained from scratch.

 

tion, which is now incorrect. Similar observations can be made for the other two examples. Table 22 in the Appendix has more such examples. These examples represent only a few types of spurious correlations that we could find but there could be other types of correlations that might have been missed.

Note that, we do not claim that every model trained on these datasets relies on the occurrence of specific words in the input problem for prediction the way our constrained model does. We are only asserting that it is possible to achieve a good score on these datasets even with such a brittle model, which clearly makes these datasets unreliable to robustly measure model performance.

## 5 Svamp

The efficacy of existing models on benchmark datasets has led to a shift in the focus of researchers towards more difficult MWPs. We claim that this efficacy on benchmarks is misleading and SOTA MWP solvers are unable to solve even elementary level one-unknown MWPs. To this end, we create a challenge set named SVAMP containing simple one-unknown arithmetic word problems of grade level up to 4. The examples in SVAMP test a model across different aspects of solving word problems. For instance, a model needs to be sensitive to questions and possess certain reasoning abilities to correctly solve the examples in our challenge set. SVAMP is similar to existing datasets of the same level in terms of scope and difficulty for humans, but is less susceptible to being solved by models relying on superficial patterns.

Our work differs from adversarial data collection methods such as Adversarial NLI Nie et al. (2020) in that these methods create examples depending on the failure of a particular model while we create examples without referring to any specific model. Inspired by the notion of _Normative evaluation_Linzen (2020), our goal is to create a dataset of simple problems that any system designed to solve MWPs should be expected to solve. We create new problems by applying certain variations to existing problems, similar to the work of Ribeiro et al. (2020). However, unlike their work, our variations do not check for linguistic capabilities. Rather, the choice of our variations is motivated by the experiments in Section 4 as well as certain simple capabilities that any MWP solver must possess.

### Creating Svamp

We create SVAMP by applying certain types of variations to a set of seed examples sampled from the ASDiv-A dataset. We select the seed examples from the recently proposed ASDiv-A dataset since it appears to be of higher quality and harder than the MAWPS dataset: We perform a simple experiment to test the coverage of each dataset by training a model on one dataset and testing it on the other one. For instance, when we train a Graph2Tree model on ASDiv-A, it achieves 82% accuracy on MAWPS. However, when trained on MAVPS and tested on ASDiv-A, the model achieved only 73% accuracy. Also recall Table 2 where most models performed better on MAWPS. Moreover, ASDiv has problems annotated according to types and grade levels which are useful for us.

To select a subset of seed examples that sufficiently represent different types of problems in the ASDiv-A dataset, we first divide the examples into groups according to their annotated types. We dis

\begin{table}
\begin{tabular}{l|c c c} \hline \hline
**Input Problem** & & **Predicted Equation** & **Answer** \\ \hline John delivered 3 letters at & easy & house. If he deitvered for 8 houses, how many letters did John defiever? & 3*8 & 24 ✓ \\ John delivered 3 letters at & easy & house. He delivered 24 letters in all. How many houses did John visit to deliver letters? & 3*24 & 72 ✗ \\ \hline Sam made 8 dollars moving lawns over the Summer. He charged 2 bucks for & hard & lawn. How many laws did he now? & 8/2 & 4✓ \\ Sam moved 4 lawns over the Summer. If the charged 2 bucks for & hard & lawn, how much did he earn? & 4/2 & 2 ✗ \\ \hline
10 apples were in the box. 6 \(\mathit{arcd}\) red and the & hard & are green. how many green apples are in the box? & 10 - 6 & 4✓ \\
10 apples were in the box. & hard & apple is either red or green. 6 apples are red. how many green apples are in the box? & 10/6 & 1.67 ✗ \\ \hline \hline \end{tabular}
\end{table}
Table 6: Attention paid to specific words by the constrained model.

 

card types such as _TVQ-Change_, _TVQ-Initial_, _CeilDivision_ and _Floor-Division_ that have less than 20 examples each. We also do not consider the _Difference_ type since it requires the use of an additional modulus operator. For ease of creation, we discard the few examples that are more than 40 words long. To control the complexity of resulting variations, we only consider those problems as seed examples that can be solved by an expression with a single operator. Then, within each group, we cluster examples using K-Means over RoBERTa sentence embeddings of each example. From each cluster, the example closest to the cluster centroid is selected as a seed example. We selected a total of 100 seed examples in this manner. The distribution of seed examples according to different types of problems can be seen in Table 7.

#### 5.1.1 Variations

The variations that we make to each seed example can be broadly classified into three categories based on desirable properties of an ideal model: _Question Sensitivity_, _Reasoning Ability_ and _Structural Invariance_. Examples of each type of variation are provided in Table 8.

**1. Question Sensitivity.** Variations in this category check if the model's answer depends on the question. In these variations, we change the question in the seed example while keeping the body same. The possible variations are as follows:

_(a) Same Object, Different Structure_: The principal object (i.e. object whose quantity is unknown) in the question is kept the same while the structure of the question is changed.

_(b) Different Object, Same Structure:_ The principal object in the question is changed while the structure of question remains fixed.

_(c) Different Object, Different Structure:_ Both, the principal object in the question and the structure of the question, are changed.

**2. Reasoning Ability.** Variations here check whether a model has the ability to correctly de

\begin{table}
\begin{tabular}{p{69.3pt} p{341.4pt} p{341.4pt}} \hline \hline Category & Variation & Examples \\ \hline  & \multirow{4}{*}{
\begin{tabular}{l} Question \\ Sensitivity \\ \end{tabular} } & \multicolumn{1}{p{341.4pt}}{Different Object, Same Structure} & Original: Allan brought two balloons and Jake brought four balloons to the park. How many balloons did Allan and Jake havre in the park? \\  & & \multicolumn{1}{p{341.4pt}}{Partition: Allan brought two balloons and Jake brought four balloons to the park. How many more balloons did Jake have than Allan in the park?} \\  & & \multicolumn{1}{p{341.4pt}}{Optional: He about how balloons and Jake brought four balloons to the park. How many more balloons did Jake have than Allan in the park?} \\  & & \multicolumn{1}{p{341.4pt}}{Optional: He about how balloons and Jake brought four balloons to the park. How many more balloons did Jake have than Allan in the park?} \\  & & \multicolumn{1}{p{341.4pt}}{Optional: He about how balloons and Jake brought four balloons to the park. How many more balloons did Jake have than Allan in the park?} \\  & & \multicolumn{1}{p{341.4pt}}{Optional: He about how balloons and Jake brought four balloons to the park. How many more balloons did Jake have than Allan in the park?} \\  & & \multicolumn{1}{p{341.4pt}}{Optional: He about how balloons and Jake brought four balloons to the park. How many more balloons did Jake have than Allan in the park?} \\  & & \multicolumn{1}{p{341.4pt}}{Optional: He about how balloons and Jake brought four balloons to the park. How many more balloons did Jake have than Allan in the park?} \\  & & \multicolumn{1}{p{341.4pt}}{Optional: He about how balloons and Jake brought four balloons to the park. How many more balloons did Jake have been among all the balloons and Jake have 

termine a change in reasoning arising from subtle changes in the problem text. The different possible variations are as follows:

_(a) Add relevant information:_ Extra relevant information is added to the example that affects the output equation.

_(b) Change information:_ The information provided in the example is changed.

_(c) Invert operation:_ The previously unknown quantity is now provided as information and the question instead asks about a previously known quantity which is now unknown.

## 3 Structural Invariance

Variations in this category check whether a model remains invariant to superficial structural changes that do not alter the answer or the reasoning required to solve the example. The different possible variations are as follows:

_(a) Add irrelevant information:_ Extra irrelevant information is added to the problem text that is not required to solve the example.

_(b) Change order of objects:_ The order of objects appearing in the example is changed.

_(c) Change order of phrases:_ The order of number-containing phrases appearing in the example is changed.

#### 5.1.2 Protocol for creating variations

Since creating variations requires a high level of familiarity with the task, the construction of SVAMP is done in-house by the authors and colleagues, hereafter called the _workers_. The 100 seed examples (as shown in Table 7) are distributed among the _workers_.

For each seed example, the _worker_ needs to create new variations by applying the variation types discussed in Section 5.1.1. Importantly, a combination of different variations over the seed example can also be done. For each new example created, the worker needs to annotate it with the equation as well as the type of variation(s) used to create it. More details about the creation protocol can be found in Appendix B.

We created a total of 1098 examples. However, since ASDiv-A does not have examples with equations of more than two operators, we discarded 98 examples from our set which had equations consisting of more than two operators. This is to ensure that our challenge set does not have any unfairly difficult examples. The final set of 1000 examples was provided to an external volunteer unfamiliar with the task to check the grammatical and logical correctness of each example.

### Dataset Properties

Our challenge set SVAMP consists of one-unknown arithmetic word problems which can be solved by expressions requiring no more than two operators. Table 9 shows some statistics of our dataset and of ASDiv-A and MAWPS. The Equation Template for each example is obtained by converting the corresponding equation into prefix form and masking out all numbers with a meta symbol. Observe that the number of distinct Equation Templates and the Average Number of Operators are similar for SVAMP and ASDiv-A and are considerably smaller than for MAWPS. This indicates that SVAMP does not contain unfairly difficult MWPs in terms of the arithmetic expression expected to be produced by a model.

Previous works, including those introducing MAWPS and ASDiv, have tried to capture the notion of _diversity_ in MWP datasets. Miao et al. (2020) introduced a metric called Corpus Lexicon Diversity (CLD) to measure lexical diversity. Their contention was that higher lexical diversity is correlated with the quality of a dataset. As can be seen from Table 9, SVAMP has a much lesser CLD than ASDiv-A. SVAMP is also less diverse in terms of problem types compared to ASDiv-a. Despite this we will show in the next section that SVAMP is in fact more challenging than ASDiv-A for current models. Thus, we believe that lexical diversity is not a reliable way to measure the quality of MWP datasets. Rather it could depend on other factors such as the diversity in MWP structure which preclude models exploiting shallow heuristics.

### Experiments on SVAMP

We train the three considered models on a combination of MAWPS and ASDiv-A and test them on SVAMP. The scores of all three models with and without RoBERTa embeddings for various subsets of SVAMP can be seen in Table 10.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Dataset** & **\# Problems** & **\# Equation** & **\# Avg Ops** & **CLD** \\  & & **Templates** & & \\ \hline MAWPS & 2373 & 39 & 1.78 & 0.26 \\ ASDiv-A & 1218 & 19 & 1.23 & 0.50 \\ SVAMP & 1000 & 26 & 1.24 & 0.22 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Statistics of our dataset compared with MAWPS and ASDiv-A.

 

The best performing Graph2Tree model is only able to achieve an accuracy of 43.8% on SVAMP. This indicates that the problems in SVAMP are indeed more challenging for the models than the problems in ASDiv-A and MAWPS despite being of the same scope and type and less diverse. Table 23 in the Appendix lists some simple examples from SVAMP on which the best performing model fails. These results lend further support to our claim that existing models cannot robustly solve elementary level word problems.

Next, we remove the questions from the examples in SVAMP and evaluate them using the three models with RoBERTa embeddings trained on combined MAWPS and ASDiv-A. The scores can be seen in Table 11. The accuracy drops by half when compared to ASDiv-A and more than half compared to MAWPS suggesting that the problems in SVAMP are more sensitive to the information present in the question. We also evaluate the performance of the constrained model on SVAMP when trained on MAWPS and ASDiv-A. The best model achieves only 18.3% accuracy (see Table 12) which is marginally better than the majority template baseline. This shows that the problems in SVAMP are less vulnerable to being solved by models using simple patterns and that a model needs contextual information in order to solve them.

We also explored using SVAMP for training by combining it with ASDiv-A and MAWPS. We performed 5-fold cross-validation over SVAMP where the model was trained on a combination of the three datasets and tested on unseen examples from SVAMP. To create the folds, we first divide the seed examples into five sets, with each _type_ of example distributed nearly equally among the sets. A fold is obtained by combining all the examples in SVAMP that were created using the seed examples in a set. In this way, we get five different folds from the five sets. We found that the best model achieved about 65% accuracy. This indicates that even with additional training data existing models are still not close to the performance that was estimated based on prior benchmark datasets.

To check the influence of different categories of variations in SVAMP, for each category, we measure the difference between the accuracy of the best model on the full dataset and its accuracy on a subset containing no example created from that category of variations. The results are shown in Table 13. Both the _Question Sensitivity_ and _Struc

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & **Seq2Seq** & \multicolumn{2}{c}{**GTS**} & \multicolumn{2}{c}{**Graph2Tree**} \\ \cline{2-7}  & \(S\) & \(R\) & \(S\) & \(R\) & \(S\) & \(R\) \\ \hline Full Set & 24.2 & 40.3 & 30.8 & 41.0 & 36.5 & **43.8** \\ \hline One-Op & 25.4 & 42.6 & 31.7 & 44.6 & 42.9 & **51.9** \\ Two-Op & 20.3 & **33.1** & 27.9 & 29.7 & 16.1 & 17.8 \\ \hline ADD & 28.5 & **41.9** & 35.8 & 36.3 & 24.9 & 36.8 \\ SUB & 22.3 & 35.1 & 26.7 & 36.9 & 41.3 & **41.3** \\ MUL & 17.9 & **38.7** & 29.2 & **38.7** & 27.4 & 35.8 \\ DIV & 29.3 & 56.3 & 39.5 & 61.1 & 40.7 & **65.3** \\ \hline \hline \end{tabular}
\end{table}
Table 10: Results of models on the SVAMP challenge set. \(S\) indicates that the model is trained from scratch. \(R\) indicates that the model was trained with RoBERTa embeddings. The first row shows the results for the full dataset. The next two rows show the results for subsets of SVAMP composed of examples that have equations with one operator and two operators respectively. The last four rows show the results for subsets of SVAMP composed of examples of type Addition, Subtraction, Multiplication and Division respectively.

\begin{table}
\begin{tabular}{l l} \hline \hline
**Model** & **SVAMP** \\ \hline FFN + LSTM Decoder (S) & 17.5 \\ FFN + LSTM Decoder (R) & **18.3** \\ \hline Majority Template Baseline & 11.7 \\ \hline \hline \end{tabular}
\end{table}
Table 12: Accuracies (\(\uparrow\)) of the constrained model on SVAMP. (R) denotes that the model is provided with non-contextual RoBERTa pretrained embeddings while (S) denotes that the model is trained from scratch.

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Model** & **SVAMP w/o ques** & **ASDiv-A w/o ques** \\ \hline Seq2Seq & 29.2 & 58.7 \\ GTS & 28.6 & 60.7 \\ Graph2Tree & **30.8** & **64.4** \\ \hline \hline \end{tabular}
\end{table}
Table 11: Accuracies (\(\uparrow\)) of models on SVAMP without questions. The 5-fold CV accuracy scores for ASDiv-A without questions are restated for easier comparison.

\begin{table}
\begin{tabular}{l c} \hline \hline
**Model** & **SVAMP** \\ \hline FFN + LSTM Decoder (S) & 17.5 \\ FFN + LSTM Decoder (R) & **18.3** \\ \hline Majority Template Baseline & 11.7 \\ \hline \hline \end{tabular}
\end{table}
Table 13: Change in accuracies when categories are removed. The Change in Accuracy \(\Delta=Acc(Full-Cat)-Acc(Full)\), where \(Acc(Full)\) is the accuracy on the full set and \(Acc(Full-Cat)\) is the accuracy on the set of examples left after removing all examples which were created using Category \(Cat\) either by itself, or in use with other categories.

 

tural Invariance_ categories of variations show an increase in accuracy when their examples are removed, thereby indicating that they make SVAMP more challenging. The decrease in accuracy for the _Reasoning Ability_ category can be attributed in large part to the _Invert Operation_ variation. This is not surprising because most of the examples created from _Invert Operation_ are almost indistinguishable from examples in ASDiv-A, which the model has seen during training. The scores for each individual variation are provided in Table 14.

We also check the break-up of performance of the best performing Graph2Tree model according to the number of numbers present in the text of the input problem. We trained the model on both ASDiv-A and MAWPS and tested on SVAMP and compare those results against the 5-fold cross-validation setting of ASDiv-A. The scores are provided in Table 15. While the model can solve many problems consisting of only two numbers in the input text (even in our challenge set), it performs very badly on problems having more than two numbers. This shows that current methods are incapable of properly associating numbers to their context. Also, the gap between the performance on ASDiv-A and SVAMP is high, indicating that the examples in SVAMP are more difficult for these models to solve than the examples in ASDiv-A even when considering the structurally same type of word problems.

## 6 Final Remarks

Going back to the original question, are existing NLP models able to solve elementary math word problems? This paper gives a negative answer. We have empirically shown that the benchmark English MWP datasets suffer from artifacts making them unreliable to gauge the performance of MWP solvers: we demonstrated that the majority of problems in the existing datasets can be solved by simple heuristics even without word-order information or the question text.

The performance of the existing models in our proposed challenge dataset also highlights their limitations in solving simple elementary level word problems. We hope that our challenge set SVAMP, containing elementary level MWPs, will enable more robust evaluation of methods. We believe that methods proposed in the future that make genuine advances in solving the task rather than relying on simple heuristics will perform well on SVAMP despite being trained on other datasets such as ASDiv-A and MAWPS.

In recent years, the focus of the community has shifted towards solving more difficult MWPs such as non-linear equations and word problems with multiple unknown variables. We demonstrated that the capability of existing models to solve simple one-unknown arithmetic word problems is overestimated. We believe that developing more robust methods for solving elementary MWPs remains a significant open problem.

## Acknowledgements

We thank the anonymous reviewers for their constructive comments. We would also like to thank our colleagues at Microsoft Research for providing valuable feedback. We are grateful to Monojit Choudhury for discussions about creating the dataset. We thank Kabir Ahuja for carrying out preliminary experiments that led to this work. We also thank Vageesh Chandramouli and Nalin Patel for their help in dataset construction.

\begin{table}
\begin{tabular}{l c c} \hline \hline
**Dataset** & **2 nums** & **3 nums** & **4 nums** \\ \hline ASDiv-A & 93.3 & 59.0 & 47.5 \\ VSAMP & 78.3 & 25.4 & 25.4 \\ \hline \hline \end{tabular}
\end{table}
Table 15: Accuracy break-up according to the number of numbers in the input problem. **2 nums** refers to the subset of problems which have only 2 numbers in the problem text. Similarly, **3 nums** and **4 nums** are subsets that contain 3 and 4 different numbers in the problem text respectively.

 



## References

* A. Amini, S. Gabriel, S. Lin, R. Koncel-Kedziorski, Y. Choi, and H. Hajishirzi (2019)MathQA: towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 2357-2367. External Links: Link Cited by: SS4.
* Y. Belinkov and J. Glass (2019)Analysis methods in neural language processing: a survey. Transactions of the Association for Computational Linguistics7, pp. 49-72. External Links: Document, ISSN 1874-405X Cited by: SS1.
* Z. Cai, L. Tu, and K. Gimpel (2017)Pay attention to the ending:strong neural baselines for the ROC story cloze task. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Vancouver, Canada, pp. 616-622. External Links: ISBN 978-0-321-4655-3 Cited by: SS1.
* M. Gardner, Y. Artzi, V. Basmova, J. Berant, B. Bogin, S. Chen, P. Dasigi, D. Dua, Y. Elazar, A. Gottumukkala, N. Gupta, H. Hajishirzi, G. Ilharco, D. Khashabi, K. Lin, J. Liu, N. F. Liu, P. Mulcaire, Q. Ning, S. Singh, N. A. Smith, S. Subramanian, R. Tsarfaty, E. Wallace, C. Zhang, and B. Zhou (2020)Evaluating models' local decision boundaries via contrast sets. External Links: Link Cited by: SS1.
* S. Gururangan, S. Swayamdipta, O. Levy, R. Schwartz, S. Bowman, and N. A. Smith (2018)Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), New Orleans, Louisiana, pp. 107-112. External Links: Link Cited by: SS1.
* D. Huang, Shuming Shi, Chin-Yew Lin, Jian Yin, and Wei-Yang Ma (2016)How well do computers solve math word problems? large-scale dataset construction and evaluation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, pp. 887-896. External Links: Link Cited by: SS1.
* D. Huang, S. Shi, C. Lin, J. Yin, and W. Ma (2016)How well do computers solve math word problems? large-scale dataset construction and evaluation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, pp. 887-896. External Links: Link Cited by: SS1.
* D. Huang, S. Shi, C. Lin, and J. Yin (2017)Learning fine-grained expressions to solve math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, Copenhagen, Denmark, pp. 805-814. External Links: Link Cited by: SS1.
* D. Huang, S. Shi, C. Lin, J. Yin, and W. Ma (2016)How well do computers solve math word problems? large-scale dataset construction and evaluation.. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, pp. 887-896. External Links: Link Cited by: SS1.
* D. Huang, S. Shi, C. Lin, J. Yin, and W. Ma (2016)How well do computers solve math word problems? large-scale dataset construction and evaluation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Online, pp. 889-896. External Links: Link Cited by: SS1.
* T. Lizen (2020)How can we accelerate progress towards human-like linguistic generalization?. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 5210-5217. External Links: Link Cited by: SS1.
* Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019)Roberta: a robustly optimized bert pretraining approach. External Links: 1905.02187 Cited by: SS1.
* T. Luong, H. Pham, and C. D. Manning (2015)Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisriben, Portugal, pp. 1412-1421. External Links: Link Cited by: SS1.
* T. McCoy, E. Pavlick, and T. Linzen (2019)Right for the wrong reasons: diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 3428-3448. External Links: Link Cited by: SS1.
* S. Miao, C. Liang, and K. Su (2020)A diverse corpus for evaluating and developing english math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 975-984. External Links: Link Cited by: SS1.
* Y. Nie, A. Williams, E. Dinan, M. Bansal, J. Weston, and D. Kiela (2020)Adversarial nli: a new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 4885-4901. External Links: Link Cited by: SS1.
* A. Poliak, J. Naradowsky, A. Haldar, R. Rudinger, and B. Van Durme (2018)

Hypothesis only baselines in natural language inference. In _Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics_, pages 180-191, New Orleans, Louisiana. Association for Computational Linguistics.
* Qin et al. (2020) Jinghui Qin, Lihui Lin, Xiaodan Liang, Rumin Zhang, and Liang Lin. 2020. Semantically-aligned universal tree-structured solver for math word problems.
* Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4902-4912, Online. Association for Computational Linguistics.
* Rosenman et al. (2020) Shachar Rosenman, Alon Jacovi, and Yoav Goldberg. 2020. Exposing Shallow Heuristics of Relation Extraction Models with Challenge Data. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 3702-3710, Online. Association for Computational Linguistics.
* Roy and Roth (2018) Subhro Roy and Dan Roth. 2018. Mapping to declarative knowledge for word problem solving. _Transactions of the Association for Computational Linguistics_, 6:159-172.
* Sachan and Xing (2017) Mrinmaya Sachan and Eric Xing. 2017. Learning to solve geometry problems from natural language demonstrations in textbooks. In _Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017)_, pages 251-261, Vancouver, Canada. Association for Computational Linguistics.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, volume 30. Curran Associates, Inc.
* Wang et al. (2017) Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 845-854, Copenhagen, Denmark. Association for Computational Linguistics.
* Wiegreffe and Pinter (2019) Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not explanation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 11-20, Hong Kong, China. Association for Computational Linguistics.
* Xie and Sun (2019) Zhipeng Xie and Shichao Sun. 2019. A goal-driven tree-structured neural model for math word problems. In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_, pages 5299-5305. International Joint Conferences on Artificial Intelligence Organization.
* Zhang et al. (2020) D. Zhang, L. Wang, L. Zhang, B. T. Dai, and H. T. Shen. 2020. The gap of semantic parsing: A survey on automatic math word problem solvers. _IEEE Transactions on Pattern Analysis and Machine Intelligence_, 42(9):2287-2305.
* Zhang et al. (2020) Jipeng Zhang, Lei Wang, Roy Ka-Wei Lee, Yi Bin, Yan Wang, Jie Shao, and Ee-Peng Lim. 2020. Graph-to-tree learning for solving math word problems. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 3928-3937, Online. Association for Computational Linguistics.

 

Implementation Details

We use 8 NVIDIA Tesla P100 GPUs each with 16 GB memory to run our experiments. The hyperparameters used for each model are shown in Table 16. The best hyperparameters are highlighted in bold. Following the setting of Zhang et al. (2020), the arithmetic word problems from MAWPS are divided into five folds, each of equal test size. For ASDiv-A, we consider the 5-fold split [238; 238; 238; 238; 266] provided by the authors Miao et al. (2020).

## Appendix B Creation Protocol

We create variations in template form. Generating more data by scaling up from these templates or by performing automatic operations on these templates is left for future work. The template form of an example is created by replacing certain words with their respective tags. Table 17 lists the various tags used in the templates.

The \(\left[NUM\right]\) tag is used to replace all the numbers and the \(\left[NAME\right]\) tag is used to replace all the Names of Persons in the example. The \(\left[OBJs\right]\) and \(\left[OBJP\right]\) tags are used for replacing the objects in the example. The \(\left[OBJs\right]\) and \(\left[OBJp\right]\) tags with the same index represent the same object in singular and plural form respectively. The intention when using the \(\left[OBJs\right]\) or the \(\left[OBJP\right]\) tag is that it can be used as a placeholder for other similar words, which when entered in that place, make sense as per the context. These tags must not be used for collectives; rather they should be used for the things that the collective represents. Some example uses of \(\left[OBJs\right]\) and \(\left[OBJP\right]\) tags are provided in Table 18. Lastly, the \(\left[MOD\right]\) tag must be used to replace any modifier preceding the \(\left[OBJs\right]\)/\(\left[OBJP\right]\) tag.

A preprocessing script is executed over the _Seed Examples_ to automatically generate template suggestions for the _workers_. The script uses Named Entity Recognition and Regular Expression matching to automatically mask the names of persons and the numbers found in the _Seed Examples_. The outputs from the script are called the _Script Examples_. An illustration is provided in Table 19.

Each _worker_ is provided with the _Seed Examples_ along with their respective _Script Examples_ that have been alloted to them. The _worker's_ task is to edit the _Script Example_ by correcting any mistake made by the preprocessing script and adding any new tags such as the \(\left[OBJs\right]\) and the \(\left[OBJP\right]\) tags in order to create the _Base Example_. If a _worker_ introduces a new tag, they need to mark it against its example-specific value. If the tag is used to mask objects, the _worker_ needs to mark both the singular and plural form of the object in a comma-seperated manner. Additionally, for each unique index of \(\left[OBJs\right]\)/\(\left[OBJP\right]\) tag in the example, the _worker_ must enter atleast one alternate value that can be used in that place. Similarly, the _worker_ must enter atleast two modifier words that can be used to precede the principal \(\left[OBJs\right]\)/\(\left[OBJP\right]\) tags in the example. These alternate values are used to gather a lexicon which can be utilised to scale-up the data at a later stage. An illustration of this process is provided in Table 20.

In order to create the variations, the _worker_ needs to check the different types of variations in Table 8 to see if they can be applied to the _Base Example_. If applicable, the _worker_ needs to create the _Variation Example_ while also making a note of the type of variation. If a particular example is the result of performing multiple types of variations, all types of variations should be listed according to their order of application from latest to earliest in a comma-seoperated manner. For any variation, if a _worker_ introduces a new tag, they need to mark it against its example-specific value as mentioned before. The index of any new tag introduced needs to be one more than the highest index already in use for that tag in the _Base Example_ or its previously created variations.

To make the annotation more efficient and streamlined, we provide the following steps to be followed in order:

1. Apply the _Question Sensitivity_ variations on the _Base Example_.
2. Apply the _Invert Operation_ variation on the _Base Example_ and on all the variations obtained so far.
3. Apply the _Add relevant information_ variation on the _Base Example_. Then considering these variations as _Base Examples_, apply the _Question Sensitivity_ variations.
4. Apply the _Add irrelevant information_ variation on the _Base Example_ and on all the variations obtained so far.
5. Apply the _Change information_ variation on the _Base Example_ and on all the variations obtained so far 

6. Apply the _Change order of Objects_ and _Change order of Events or Phrases_ variations on the _Base Example_ and on all the variations obtained so far.

Table 21 provides some variations for the example in Table 20. Note that two seperate examples were created through the _'Add irrelevant information'_ variation. The first by applying the variation on the _Original Example_ and the second by applying it on a previously created example (as directed in Step-4).

To make sure that different _workers_ following our protocol make similar types of variations, we hold a trial where each worker created variations from the same 5 seed examples. We observed that barring minor linguistic differences, most of the created examples were the same, thereby indicating the effectiveness of our protocol.

## Appendix C Analyzing Attention Weights

In Table 22, we provide more examples to illustrate the specific word to equation correlation that the constrained model learns.

## Appendix D Examples of Simple Problems

In Table 23, we provide a few simple examples from SVAMP that the best performing Graph2Tree model could not solve.

## Appendix E Ethical Considerations

In this paper, we consider the task of automatically solving Math Word Problems (MWPs). Our work encourages the development of better systems that can robustly solve MWPs. Such systems can be deployed for use in the education domain. E.g., an application can be developed that takes MWPs as input and provides detailed explanations to solve them. Such applications can aide elementary school students in learning and practicing math.

We present a challenge set called SVAMP of one-unknown English Math Word Problems. SVAMP is created in-house by the authors themselves by applying some simple variations to examples from ASDiv-A Miao et al. (2020), which is a publicly available dataset. We provide a detailed creation protocol in Section B. We are not aware of any risks associated with our proposed dataset.

To provide an estimate of the energy requirements of our experiments, we provide the details such as computing platform and running time in Section A. Also, in order to reduce carbon costs from our experiments, we first perform a broad hyperparameter search over only a single fold for the datasets and then run the cross validation experiment over a select few hyperparameters.

\begin{table}
\begin{tabular}{l c c c c c c|c c} \hline \hline  & \multicolumn{2}{c}{**Seq2Seq**} & \multicolumn{2}{c}{**GTS**} & \multicolumn{2}{c}{**Graph2Tree**} & \multicolumn{2}{c}{**Constrained**} \\ \cline{2-9}
**Hyperparameters** & Scratch & RoBERTa & Scratch & RoBERTa & Scratch & RoBERTa & Scratch & RoBERTa \\ \hline Embedding Size & [128, **256**] & [768] & [**128**, 256] & [768] & [**128**, 256] & [768] & [**128**, 256] & [768] \\ Hidden Size & [**256**, 384] & [**256**, 384] & [384, **512**] & [**384**, **512**] & [**256**, **384] & [**256**, **384] & [**256**, **384] & [**256**, 384] \\ Number of Layers & [1, 2] & [1, 2] & [1, 2] & [1, 2] & [1, 2] & [1, 2] & [1, 2] & [1, 2] & [1, 2] \\ Learning Rate & [54, 8+k, 1e-4, **1e-4, 2e-4**, & [8+k, 1e-3, **5**d-8, & **8+k-4**, & **8+k-4**, & **5+k-4**, & **1e-3**, 2e-3**] & [1e-3, 2e-3] \\  & **1e-3**] & [5e-4] & **2e-3**] & **1e-3**] & [2e-3] & [1e-3] & [1e-3] & [1e-3] & [1e-3] \\ Embedding LR & [5e-4, **8+k-4**, & [5e-6, **8e-6**, & [8+k-4, 1e-3, & **5**+k-4**, & **8**+k-4**, & **1e-3**, & **5**+k-4**, & **1e-3**, 2e-3**] & [1e-3] \\  & 1e-3] & [5e-5] & **2e-3]** & [1e-5] & 2e-3] & **1e-5]** & & & \\ Batch Size & [8, 16] & [4, 8] & [8, 16] & [4, 8] & [8, 16] & [4, 8] & [8, 16] & [4, 8] \\ Dropout & [0,1] & [0.1] & [0.5] & [0.5] & [0.5] & [0.1] & [0.1] & [0.1] \\ \hline \# Parameters & 8.5M & 130M & 15M & 140M & 16M & 143M & 5M & 130M \\ Epochs & 60 & 50 & 60 & 50 & 60 & 50 & 60 & 50 \\ Avg Time/Epoch & 10 & 40 & 60 & 120 & 60 & 120 & 10 & 15 \\ \hline \hline \end{tabular}
\end{table}
Table 16: Different hyperparameters and the values considered for each of them in the models. The best hyperparameters for each model for 5-fold cross-validation on ASDiv-A are highlighted in bold. Average Time/Epoch is measured in seconds.

\begin{table}
\begin{tabular}{c c} \hline \hline
**Tag** & **Description** \\ \hline \multirow{2}{*}{\begin{tabular}{c} \(\left[\text{NUMx}\right]\) \\ \(\left[\text{NAMEx}\right]\) \\ \(\left[\text{OBJsr}\right]\) \\ \(\left[\text{OBYprx}\right]\) \\ \(\left[\text{MODr}\right]\) \\ \end{tabular} } & \multirow{2}{*}{
\begin{tabular}{c} Number \\ Names of Persons \\ Singular Object \\ Plural Object \\ Modifier \\ \end{tabular} } \\ \hline \hline \end{tabular}
\end{table}
Table 17: List of tags used in annotated templates. \(x\) denotes the index of the tag.

 



\begin{table}
\begin{tabular}{l l} \hline
**Excerpt of Example** & Beth has 4 packs of red crayons and 2 packs of green crayons, Each pack has 10 crayson in it. \\
**Template Form** & \([NAME1]\) has \([NUM1]\) packs for \([MOD1]\)\([OBJp1]\) and \([NUM2]\) packs for \([MOD2]\)\([OBJp1]\). \\ \hline
**Excerpt of Example** & In a game, Frank defeated 6 enemies. Each enemy earned him 9 points. \\
**Template Form** & In a game \([NAME1]\) defeated \([NUM1]\)\([OBJp1]\). Each \([OBJs1]\) earned him \([NUM2]\) points. \\ \hline \end{tabular}
\end{table}
Table 18: Example uses of tags. Note that in the first example, the word _‘packs’_ was not replaced since it is a collective. In the second example, the word _‘points’_ was not replaced because it is too instance-specific and no other word can be used in that place.

\begin{table}
\begin{tabular}{l l} \hline
**Script Example Body** & Both has 4 packs of crayons. Each pack has 10 crayons in it. She also has 6 extra crayons. \\
**Scred Example Question** & How many crayons does Beth have altogether? \\
**Seed Example Equation** & 4*10+6 \\
**Script Example Body** & \([NAME1]\) has \([NUM1]\) packs of crayons . Each pack has \([NUM2]\) crayons in it . She also has \([NUM3]\) extra crayons. \\
**Script Example Question** & How many crayons does \([NAME1]\) have altogether ? \\
**Script Example Equation** & \([NUM1]*[NUM2]+[NUM3]\) \\ \hline  & & \\ \end{tabular}
\end{table}
Table 19: An example of suggested templates. Note that the preprocessing script could not succesfully tag _crayons_ as \(\left[OBJP1\right]\).

\begin{table}
\begin{tabular}{l l} \hline
**Excerpt of Example** & Beth has 4 packs of red crayons and 2 packs of green crayons, Each pack has 10 crayson in it. \\
**Template Form** & \([NAME1]\) has \([NUM1]\) packs for \([MOD1]\)\([OBJp1]\) and \([NUM2]\) packs for \([MOD2]\)\([OBJp1]\). \\ \hline
**Excerpt of Example** & In a game, Frank defeated 6 enemies. Each enemy earned him 9 points. \\
**Template Form** & In a game \([NAME1]\) defeated \([NUM1]\)\([OBJp1]\). Each \([OBJS1]\) earned him \([NUM2]\) points. \\ \hline \end{tabular}
\end{table}
Table 18: Example uses of tags. Note that in the first example, the word _‘packs’_ was not replaced since it is a collective. In the second example, the word _‘points’_ was not replaced because it is too instance-specific and no other word can be used in that place.

\begin{table}
\begin{tabular}{l l} \hline
**Script Example Body** & \([NAME1]\) has \([NUM1]\) packs of crayons . Each pack has \([NUM2]\) crayons in it . She also has \([NUM3]\) extra crayons. \\
**Script Example Question** & How many crayons lose \([NAME1]\) have altogether ? \\ \hline
**Base Example Body** & \([NAME1]\) has \([NUM1]\) packs of \([OBJp1]\) . Each pack has \([NUM2]\)\([OBJp1]\) in it . She also has \([NUM3]\) extra crayon \([OBJp1]\). \\
**Base Example Question** & How many \([OBjp1]\) does \([NAME1]\) have altogether ? \\
**Alternate for \([OD]\)** & \([MOD]\) & small, large \\ \hline \end{tabular}
\end{table}
Table 20: An example of editing the Suggested Templates. The edits are indicated in green.

\begin{table}
\begin{tabular}{l l} \hline
**Excerpt of Example** & Beth has 4 packs of red crayons and 2 packs of green crayons, Each pack has 10 crayson in it. \\
**Template Form** & \([NAME1]\) has \([NUM1]\) packs for \([MOD1]\)\([OBJp1]\) and \([NUM2]\) packs for \([MOD2]\)\([OBJp1]\). \\ \hline
**Excerpt of Example** & In a game, Frank defeated 6 enemies. Each enemy earned him 9 points. \\
**Template Form** & In a game \([NAME1]\) defeated \([NUM1]\)\([OBJp1]\). Each \([OBJs1]\) earned him \([NUM2]\) points. \\ \hline \end{tabular}
\end{table}
Table 19: An example of suggested templates. Note that the preprocessing script could not succesfully tag _crayons_ as \(\left[OBJP1\right]\).





\begin{table}
\begin{tabular}{l c c}
* 2 & 2