SlowGuess/NAACL_2021_OCR huggingface dataset:
Arkil Patel Satwik Bhattamishra Navin Goyal  
Microsoft Research India
arkil.patel@gmail.com, {t-satbh,navingo}@microsoft.com
The problem of designing NLP solvers for math word problems (MWP) has seen sustained research activity and steady gains in the test accuracy. Since existing solvers achieve high performance on the benchmark datasets for elementary level MwPs containing one-unknown arithmetic word problems, such problems are often considered "solved" with the bulk of research attention moving to more complex MwPs. In this paper, we restrict our attention to English MwPs taught in grades four and lower. We provide strong evidence that the existing MWP solvers rely on shallow heuristics to achieve high performance on the benchmark datasets. To this end, we show that MWP solvers that do not have access to the question asked in the MWP can still solve a large fraction of MwPs. Similarly, models that treat MwPs as bag-of-words can also achieve surprisingly high accuracy. Further, we introduce a challenge dataset, SVAMP, created by applying carefully chosen variations over examples sampled from existing datasets. The best accuracy achieved by state-of-the-art models is substantially lower on SVAMP, thus showing that much remains to be done even for the simplest of the MwPs.
A Math Word Problem (MWP) consists of a short natural language narrative describing a state of the world and poses a question about some unknown quantities (see Table 1 for some examples). MWPs are taught in primary and higher schools. The MWP task is a type of semantic parsing task where given an MWP the goal is to generate an expression (more generally, equations), which can then be evaluated to get the answer. The task is challenging because a machine needs to extract relevant information from natural language text as well as perform mathematical reasoning to solve it. The complexity of MWPs can be measured along multiple axes, e.g., reasoning and linguistic
PROBLEM:
Text: Jack had 8 pens and Mary had 5 pens. Jack gave 3 pens to Mary. How many pens does Jack have now? Equation:  $8 - 3 = 5$
QUESTION SENSITIVITY VARIATION:
Text: Jack had 8 pens and Mary had 5 pens. Jack gave 3 pens to Mary. How many pens does Mary have now? Equation:  $5 + 3 = 8$
REASONING ABILITY VARIATION:
Text: Jack had 8 pens and Mary had 5 pens. Mary gave 3 pens to Jack. How many pens does Jack have now? Equation:  $8 + 3 = 11$
STRUCTURAL INVARIANCE VARIATION:
Text: Jack gave 3 pens to Mary. If Jack had 8 pens and Mary had 5 pens initially, how many pens does Jack have now? Equation:  $8 - 3 = 5$
Table 1: Example of a Math Word Problem along with the types of variations that we make to create SVAMP.
complexity and world and domain knowledge. A combined complexity measure is the grade level of an MWP, which is the grade in which similar MWPs are taught. Over the past few decades many approaches have been developed to solve MWPs with significant activity in the last decade (Zhang et al., 2020).
MwPs come in many varieties. Among the simplest are the one-unknown arithmetic word problems where the output is a mathematical expression involving numbers and one or more arithmetic operators  $(+, -, *, /)$ . Problems in Tables 1 and 6 are of this type. More complex MwPs may have systems of equations as output or involve other operators or may involve more advanced topics and specialized knowledge. Recently, researchers have started focusing on solving such MwPs, e.g. multiple-unknown linear word problems (Huang et al., 2016a), geometry (Sachan and Xing, 2017) and probability (Amini et al., 2019), believing that existing work can handle one-unknown arithmetic MwPs well (Qin et al., 2020). In this paper, we question the capabilities of the state-of-the-art
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2080-2094 June 6-11, 2021. ©2021 Association for Computational Linguistics
(SOTA) methods to robustly solve even the simplest of MWPs suggesting that the above belief is not well-founded.
In this paper, we provide concrete evidence to show that existing methods use shallow heuristics to solve a majority of word problems in the benchmark datasets. We find that existing models are able to achieve reasonably high accuracy on MwPs from which the question text has been removed leaving only the narrative describing the state of the world. This indicates that the models can rely on superficial patterns present in the narrative of the MwP and achieve high accuracy without even looking at the question. In addition, we show that a model without word-order information (i.e., the model treats the MwP as a bag-of-words) can also solve the majority of MwPs in benchmark datasets.
The presence of these issues in existing benchmarks makes them unreliable for measuring the performance of models. Hence, we create a challenge set called SVAMP (Simple Variations on Arithmetic Math word Problems; pronounced swap) of one-unknown arithmetic word problems with grade level up to 4 by applying simple variations over word problems in an existing dataset (see Table 1 for some examples). SVAMP further highlights the brittle nature of existing models when trained on these benchmark datasets. On evaluating SOTA models on SVAMP, we find that they are not even able to solve half the problems in the dataset. This failure of SOTA models on SVAMP points to the extent to which they rely on simple heuristics in training data to make their prediction.
Below, we summarize the two broad contributions of our paper.
- We show that the majority of problems in benchmark datasets can be solved by shallow heuristics lacking word-order information or lacking question text.
- We create a challenge set called SVAMP<sup>1</sup> for more robust evaluation of methods developed to solve elementary level math word problems.
Math Word Problems. A wide variety of methods and datasets have been proposed to solve MwPs; e.g. statistical machine learning (Roy and Roth,
2018), semantic parsing (Huang et al., 2017) and most recently deep learning (Wang et al., 2017; Xie and Sun, 2019; Zhang et al., 2020); see (Zhang et al., 2020) for an extensive survey. Many papers have pointed out various deficiencies with previous datasets and proposed new ones to address them. Koncel-Kedziorski et al. (2016) curated the MAwPS dataset from previous datasets which along with Math23k (Wang et al., 2017) has been used as benchmark in recent works. Recently, ASDiv (Miao et al., 2020) has been proposed to provide more diverse problems with annotations for equation, problem type and grade level. HMWP (Qin et al., 2020) is another newly proposed dataset of Chinese MwPs that includes examples with multiple-unknown variables and requiring non-linear equations to solve them.
Identifying artifacts in datasets has been done for the Natural Language Inference (NLI) task by McCoy et al. (2019), Poliak et al. (2018), and Gururangan et al. (2018). Rosenman et al. (2020) identified shallow heuristics in a Relation Extraction dataset. Cai et al. (2017) showed that biases prevalent in the ROC stories cloze task allowed models to yield state-of-the-art results when trained only on the endings. To the best of our knowledge, this kind of analysis has not been done on any Math Word Problem dataset.
Challenge Sets for NLP tasks have been proposed most notably for NLI and machine translation (Belinkov and Glass, 2019; Nie et al., 2020; Ribeiro et al., 2020). Gardner et al. (2020) suggested creating contrast sets by manually perturbing test instances in small yet meaningful ways that change the gold label. We believe that we are the first to introduce a challenge set targeted specifically for robust evaluation of Math Word Problems.
We denote a Math Word Problem  $P$  by a sequence of  $n$  tokens  $P = (\pmb{w}_1, \dots, \pmb{w}_n)$  where each token  $\pmb{w}_i$  can be either a word from a natural language or a numerical value. The word problem  $P$  can be broken down into body  $B = (\pmb{w}_1, \dots, \pmb{w}_k)$  and question  $Q = (\pmb{w}_{k+1}, \dots, \pmb{w}_n)$ . The goal is to map  $P$  to a valid mathematical expression  $E_P$  composed of numbers from  $P$  and mathematical operators from the set  $\{+, -, /, *\}$  (e.g.  $3 + 5 - 4$ ). The metric used to evaluate models on the MWP task is Execution Accuracy, which is obtained from com
1The dataset and code are available at: https://github.com/arkilpatel/SVAMP
Model	MAWPS	ASDiv-A
Seq2Seq (S)	79.7	55.5
Seq2Seq (R)	86.7	76.9
GTS (S) (Xie and Sun, 2019)	82.6	71.4
GTS (R)	88.5	81.2
Graph2Tree (S) (Zhang et al., 2020)	83.7	77.4
Graph2Tree (R)	88.7	82.2
Majority Template Baselinen2	17.7	21.2
paring the predicted answer (calculated by evaluating  $E_P$ ) with the annotated answer. In this work, we focus only on one-unknown arithmetic word problems.
3.2 Datasets and Methods
Many of the existing datasets are not suitable for our analysis as either they are in Chinese, e.g. Math23k (Wang et al., 2017) and HMWP (Qin et al., 2020), or have harder problem types, e.g. Dolphin18K (Huang et al., 2016b). We consider the widely used benchmark MAwPS (Koncel-Kedziorski et al., 2016) composed of 2373 MwPs and the arithmetic subset of ASDiv (Miao et al., 2020) called ASDiv-A which has 1218 MwPs mostly up to grade level 4 (MAwPS does not have grade level information). Both MAwPS and ASDiv-A are evaluated on 5-fold cross-validation based on pre-assigned splits.
We consider three models in our experiments:
(a) Seq2Seq consists of a Bidirectional LSTM Encoder to encode the input sequence and an LSTM decoder with attention (Luong et al., 2015) to generate the equation.
(c) GTS (Xie and Sun, 2019) uses an LSTM Encoder to encode the input sequence and a tree-based Decoder to generate the equation.
(d) Graph2Tree (Zhang et al., 2020) combines a Graph-based Encoder with a Tree-based Decoder.
The performance of these models on both datasets is shown in Table 2. We either provide RoBERTa (Liu et al., 2019) pre-trained embeddings to the models or train them from scratch. Graph2Tree (Zhang et al., 2020) with RoBERTa embeddings achieves the state-of-the-art for both
Table 2: 5-fold cross-validation accuracies  $(\uparrow)$  of baseline models on datasets. (R) means that the model is provided with RoBERTa pretrained embeddings while (S) means that the model is trained from scratch.
Model	MAWPS	ASDiv-A
Seq2Seq	77.4	58.7
GTS	76.2	60.7
Graph2Tree	77.7	64.4
Table 3: 5-fold cross-validation accuracies  $(\uparrow)$  of baseline models on Question-removed datasets.
datasets. Note that our implementations achieve a higher score than the previously reported highest score of  $78\%$  on ASDiv-A (Miao et al., 2020) and  $83.7\%$  on MAwPS (Zhang et al., 2020). The implementation details are provided in Section A in the Appendix.
4 Deficiencies in existing datasets
Here we describe the experiments that show that there are important deficiencies in MAWPS and ASDiv-A.
4.1 Evaluation on Question-removed MWPs
As mentioned in Section 3.1, each MWP consists of a body  $B$ , which provides a short narrative on a state of the world and a question  $Q$ , which inquires about an unknown quantity about the state of the world. For each fold in the provided 5-fold split in MAWPS and ASDiv-A, we keep the train set unchanged while we remove the questions  $Q$  from the problems in the test set. Hence, each problem in the test set consists of only the body  $B$  without any question  $Q$ . We evaluate all three models with RoBERTa embeddings on these datasets. The results are provided in Table 3.
The best performing model is able to achieve a 5-fold cross-validation accuracy of  $64.4\%$  on ASDiv-A and  $77.7\%$  on MAWPS. Loosely translated, this means that nearly  $64\%$  of the problems in ASDiv-A and  $78\%$  of the problems in MAWPS can be correctly answered without even looking at the question. This suggests the presence of patterns in the bodies of MWPs in these datasets that have a direct correlation with the output equation.
Some recent works have also demonstrated similar evidence of bias in NLI datasets (Gururangan et al., 2018; Poliak et al., 2018). They observed that NLI models were able to predict the correct label for a large fraction of the standard NLI datasets based on only the hypothesis of the input and without the premise. Our results on question-removed examples of math word problems resemble their observations on NLI datasets and similarly indicate the presence of artifacts that help statistical
$^{2}$ Majority Template Baseline is the accuracy when the model always predicts the most frequent Equation Template. Equation Templates are explained in Section 5.2
Model	MAWPS	ASDiv-A
Easy	Hard	Easy	Hard
Seq2Seq	86.8	86.7	91.3	56.1
GTS	92.6	71.7	91.6	65.3
Graph2Tree	93.4	71.0	92.8	63.3
models predict the correct answer without complete information. Note that even though the two methods appear similar, there is an important distinction. In Gururangan et al. (2018), the model is trained and tested on hypothesis only examples and hence, the model is forced to find artifacts in the hypothesis during training. On the other hand, our setting is more natural since the model is trained in the standard way on examples with both the body and the question. Thus, the model is not explicitly forced to learn based on the body during training and our results not only show the presence of artifacts in the datasets but also suggest that the SOTA models exploit them.
Following Gururangan et al. (2018), we attempt to understand the extent to which SOTA models rely on the presence of simple heuristics in the body to predict correctly. We partition the test set into two subsets for each model: problems that the model predicted correctly without the question are labeled Easy and the problems that the model could not answer correctly without the question are labeled Hard. Table 4 shows the performance of the models on their respective Hard and Easy sets. Note that their performance on the full set is already provided in Table 2. It can be seen clearly that although the models correctly answer many Hard problems, the bulk of their success is due to the Easy problems. This shows that the ability of SOTA methods to robustly solve word problems is overestimated and that they rely on simple heuristics in the body of the problems to make predictions.
4.2 Performance of a constrained model
We construct a simple model based on the Seq2Seq architecture by removing the LSTM Encoder and replacing it with a Feed-Forward Network that maps the input embeddings to their hidden representations. The LSTM Decoder is provided with the average of these hidden representations as its initial hidden state. During decoding, an attention mechanism (Luong et al., 2015) assigns weights to individual hidden representations of the input
Table 4: Results of baseline models on the Easy and Hard test sets.
Model	MAWPS	ASDiv-A
FFN + LSTM Decoder (S)	75.1	46.3
FFN + LSTM Decoder (R)	77.9	51.2
Table 5: 5-fold cross-validation accuracies  $(\uparrow)$  of the constrained model on the datasets. (R) denotes that the model is provided with non-contextual RoBERTa pretrained embeddings while (S) denotes that the model is trained from scratch.
tokens. We use either RoBERTa embeddings (non-contextual; taken directly from Embedding Matrix) or train the model from scratch. Clearly, this model does not have access to word-order information.
Table 5 shows the performance of this model on MAWPS and ASDiv-A. The constrained model with non-contextual RoBERTa embeddings is able to achieve a cross-validation accuracy of 51.2 on ASDiv-A and an astounding 77.9 on MAWPS. It is surprising to see that a model having no word-order information can solve a majority of word problems in these datasets. These results indicate that it is possible to get a good score on these datasets by simply associating the occurrence of specific words in the problems to their corresponding equations. We illustrate this more clearly in the next section.
4.3 Analyzing the attention weights
To get a better understanding of how the constrained model is able to perform so well, we analyze the attention weights that it assigns to the hidden representations of the input tokens. As shown by Wegreffe and Pinter (2019), analyzing the attention weights of our constrained model is a reliable way to explain its prediction since each hidden representation consists of information about only that token as opposed to the case of an RNN where each hidden representation may have information about the context i.e. its neighboring tokens.
We train the contrained model (with RoBERTa embeddings) on the full ASDiv-A dataset and observe the attention weights it assigns to the words of the input problems. We found that the model usually attends to a single word to make its prediction, irrespective of the context. Table 6 shows some representative examples. In the first example, the model assigns an attention weight of 1 to the representation of the word 'every' and predicts the correct equation. However, when we make a subtle change to this problem such that the corresponding equation changes, the model keeps on attending over the word 'every' and predicts the same equa
Input Problem	Predicted Equation	Answer
John delivered 3 letters at every house. If he delivered for 8 houses, how many letters did John deliver?	3*8	24✓
John delivered 3 letters at every house. He delivered 24 letters in all. How many houses did John visit to deliver letters?	3*24	72✗
Sam made 8 dollars mowing lawns over the Summer. He charged 2 bucks for each lawn. How many lawns did he mow?	8/2	4✓
Sam mowed 4 lawns over the Summer. If he charged 2 bucks for each lawn, how much did he earn?	4/2	2✗
10 apples were in the box. 6 are red and the rest are green. how many green apples are in the box?	10-6	4✓
10 apples were in the box. Each apple is either red or green. 6 apples are red. how many green apples are in the box?	10/6	1.67✗
tion, which is now incorrect. Similar observations can be made for the other two examples. Table 22 in the Appendix has more such examples. These examples represent only a few types of spurious correlations that we could find but there could be other types of correlations that might have been missed.
Note that, we do not claim that every model trained on these datasets relies on the occurrence of specific words in the input problem for prediction the way our constrained model does. We are only asserting that it is possible to achieve a good score on these datasets even with such a brittle model, which clearly makes these datasets unreliable to robustly measure model performance.
5 SVAMP
The efficacy of existing models on benchmark datasets has led to a shift in the focus of researchers towards more difficult MwPs. We claim that this efficacy on benchmarks is misleading and SOTA MWP solvers are unable to solve even elementary level one-unknown MwPs. To this end, we create a challenge set named SVAMP containing simple one-unknown arithmetic word problems of grade level up to 4. The examples in SVAMP test a model across different aspects of solving word problems. For instance, a model needs to be sensitive to questions and possess certain reasoning abilities to correctly solve the examples in our challenge set. SVAMP is similar to existing datasets of the same level in terms of scope and difficulty for humans, but is less susceptible to being solved by models relying on superficial patterns.
Our work differs from adversarial data collection methods such as Adversarial NLI (Nie et al., 2020) in that these methods create examples depending on the failure of a particular model while we create examples without referring to any specific model. Inspired by the notion of Normative evaluation (Linzen, 2020), our goal is to create a dataset of
simple problems that any system designed to solve MwPs should be expected to solve. We create new problems by applying certain variations to existing problems, similar to the work of Ribeiro et al. (2020). However, unlike their work, our variations do not check for linguistic capabilities. Rather, the choice of our variations is motivated by the experiments in Section 4 as well as certain simple capabilities that any MWP solver must possess.
5.1 Creating SVAMP
We create SVAMP by applying certain types of variations to a set of seed examples sampled from the ASDiv-A dataset. We select the seed examples from the recently proposed ASDiv-A dataset since it appears to be of higher quality and harder than the MAWPS dataset: We perform a simple experiment to test the coverage of each dataset by training a model on one dataset and testing it on the other one. For instance, when we train a Graph2Tree model on ASDiv-A, it achieves  $82\%$  accuracy on MAWPS. However, when trained on MAWPS and tested on ASDiv-A, the model achieved only  $73\%$  accuracy. Also recall Table 2 where most models performed better on MAWPS. Moreover, ASDiv has problems annotated according to types and grade levels which are useful for us.
To select a subset of seed examples that sufficiently represent different types of problems in the ASDiv-A dataset, we first divide the examples into groups according to their annotated types. We dis
Table 6: Attention paid to specific words by the constrained model.
Group	Examples in ASDiv-A	Selected Seed Examples
Addition	278	28
Subtraction	362	33
Multiplication	188	19
Division	176	20
Total	1004	100
Table 7: Distribution of selected seed examples across types.
CATEGORY	VARIATION	EXAMPLES
Question Sensitivity	Same Object, Different Structure	Original: Allan brought two balloons and Jake brought four balloons to the park. How many balloons did Allan and Jake have in the park?
Variation: Allan brought two balloons and Jake brought four balloons to the park. How many more balloons did Jake have than Allan in the park?
Different Object, Same Structure	Original: In a school, there are 542 girls and 387 boys. 290 more boys joined the school. How many pupils are in the school?
Variation: In a school, there are 542 girls and 387 boys. 290 more boys joined the school. How many boys are in the school?
Different Object, Different Structure	Original: He then went to see the oranges being harvested. He found out that they harvest 83 sacks per day and that each sack contains 12 oranges. How many sacks of oranges will they have after 6 days of harvest?
Variation: He then went to see the oranges being harvested. He found out that they harvest 83 sacks per day and that each sack contains 12 oranges. How many oranges do they harvest per day?
Reasoning Ability	Add relevant information	Original: Every day, Ryan spends 4 hours on learning English and 3 hours on learning Chinese. How many hours does he spend on learning English and Chinese in all?
Variation: Every day, Ryan spends 4 hours on learning English and 3 hours on learning Chinese. If he learns for 3 days, how many hours does he spend on learning English and Chinese in all?
Change Information	Original: Jack had 142 pencils. Jack gave 31 pencils to Dorothy. How many pencils does Jack have now?
Variation: Dorothy had 142 pencils. Jack gave 31 pencils to Dorothy. How many pencils does Dorothy have now?
Invert Operation	Original: He also made some juice from fresh oranges. If he used 2 oranges per glass of juice and he made 6 glasses of juice, how many oranges did he use?
Variation: He also made some juice from fresh oranges. If he used 2 oranges per glass of juice and he used up 12 oranges, how many glasses of juice did he make?
Structural Invariance	Change order of objects	Original: John has 8 marbles and 3 stones. How many more marbles than stones does he have?
Variation: John has 3 stones and 8 marbles. How many more marbles than stones does he have?
Change order of phrases	Original: Matthew had 27 crackers. If Matthew gave equal numbers of crackers to his 9 friends, how many crackers did each person eat?
Variation: Matthew gave equal numbers of crackers to his 9 friends. If Matthew had a total of 27 crackers initially, how many crackers did each person eat?
Add irrelevant information	Original: Jack had 142 pencils. Jack gave 31 pencils to Dorothy. How many pencils does Jack have now?
Variation: Jack had 142 pencils. Dorothy had 50 pencils. Jack gave 31 pencils to Dorothy. How many pencils does Jack have now?