SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 8: Types of Variations with examples. 'Original:' denotes the base example from which the variation is created, 'Variation:' denotes a manually created variation.
card types such as TVQ-Change, TVQ-Initial, Ceil-Division and Floor-Division that have less than 20 examples each. We also do not consider the Difference type since it requires the use of an additional modulus operator. For ease of creation, we discard the few examples that are more than 40 words long. To control the complexity of resulting variations, we only consider those problems as seed examples that can be solved by an expression with a single operator. Then, within each group, we cluster examples using K-Means over RoBERTa sentence embeddings of each example. From each cluster, the example closest to the cluster centroid is selected as a seed example. We selected a total of 100 seed examples in this manner. The distribution of seed examples according to different types of problems can be seen in Table 7.
5.1.1 Variations
The variations that we make to each seed example can be broadly classified into three categories based on desirable properties of an ideal model:
Question Sensitivity, Reasoning Ability and Structural Invariance. Examples of each type of variation are provided in Table 8.
1. Question Sensitivity. Variations in this category check if the model's answer depends on the question. In these variations, we change the question in the seed example while keeping the body same. The possible variations are as follows:
(a) Same Object, Different Structure: The principal object (i.e. object whose quantity is unknown) in the question is kept the same while the structure of the question is changed.
(b) Different Object, Same Structure: The principal object in the question is changed while the structure of question remains fixed.
(c) Different Object, Different Structure: Both, the principal object in the question and the structure of the question, are changed.
2. Reasoning Ability. Variations here check whether a model has the ability to correctly de
Dataset	# Problems	# Equation Templates	# Avg Ops	CLD
MAWPS	2373	39	1.78	0.26
ASDiv-A	1218	19	1.23	0.50
SVAMP	1000	26	1.24	0.22
Table 9: Statistics of our dataset compared with MAWPS and ASDiv-A.
termine a change in reasoning arising from subtle changes in the problem text. The different possible variations are as follows:
(a) Add relevant information: Extra relevant information is added to the example that affects the output equation.
(b) Change information: The information provided in the example is changed.
(c) Invert operation: The previously unknown quantity is now provided as information and the question instead asks about a previously known quantity which is now unknown.
3. Structural Invariance. Variations in this category check whether a model remains invariant to superficial structural changes that do not alter the answer or the reasoning required to solve the example. The different possible variations are as follows:
(a) Add irrelevant information: Extra irrelevant information is added to the problem text that is not required to solve the example.
(b) Change order of objects: The order of objects appearing in the example is changed.
(c) Change order of phrases: The order of number-containing phrases appearing in the example is changed.
5.1.2 Protocol for creating variations
Since creating variations requires a high level of familiarity with the task, the construction of SVAMP is done in-house by the authors and colleagues, hereafter called the workers. The 100 seed examples (as shown in Table 7) are distributed among the workers.
For each seed example, the worker needs to create new variations by applying the variation types discussed in Section 5.1.1. Importantly, a combination of different variations over the seed example can also be done. For each new example created, the worker needs to annotate it with the equation as well as the type of variation(s) used to create it. More details about the creation protocol can be found in Appendix B.
We created a total of 1098 examples. However, since ASDiv-A does not have examples with equations of more than two operators, we discarded 98 examples from our set which had equations consisting of more than two operators. This is to ensure that our challenge set does not have any unfairly difficult examples. The final set of 1000 examples was provided to an external volunteer unfamiliar with the task to check the grammatical and logical correctness of each example.
5.2 Dataset Properties
Our challenge set SVAMP consists of one-unknown arithmetic word problems which can be solved by expressions requiring no more than two operators. Table 9 shows some statistics of our dataset and of ASDiv-A and MAWPS. The Equation Template for each example is obtained by converting the corresponding equation into prefix form and masking out all numbers with a meta symbol. Observe that the number of distinct Equation Templates and the Average Number of Operators are similar for SVAMP and ASDiv-A and are considerably smaller than for MAWPS. This indicates that SVAMP does not contain unfairly difficult MwPs in terms of the arithmetic expression expected to be produced by a model.
Previous works, including those introducing MAWPS and ASDiv, have tried to capture the notion of diversity in MWP datasets. Miao et al. (2020) introduced a metric called Corpus Lexicon Diversity (CLD) to measure lexical diversity. Their contention was that higher lexical diversity is correlated with the quality of a dataset. As can be seen from Table 9, SVAMP has a much lesser CLD than ASDiv-A. SVAMP is also less diverse in terms of problem types compared to ASDiv-a. Despite this we will show in the next section that SVAMP is in fact more challenging than ASDiv-A for current models. Thus, we believe that lexical diversity is not a reliable way to measure the quality of MWP datasets. Rather it could depend on other factors such as the diversity in MWP structure which preclude models exploiting shallow heuristics.
5.3 Experiments on SVAMP
We train the three considered models on a combination of MAWPS and ASDiv-A and test them on SVAMP. The scores of all three models with and without RoBERTa embeddings for various subsets of SVAMP can be seen in Table 10.
Seq2Seq	GTS	Graph2Tree
S	R	S	R	S	R
Full Set	24.2	40.3	30.8	41.0	36.5	43.8
One-Op	25.4	42.6	31.7	44.6	42.9	51.9
Two-Op	20.3	33.1	27.9	29.7	16.1	17.8
ADD	28.5	41.9	35.8	36.3	24.9	36.8
SUB	22.3	35.1	26.7	36.9	41.3	41.3
MUL	17.9	38.7	29.2	38.7	27.4	35.8
DIV	29.3	56.3	39.5	61.1	40.7	65.3
Table 10: Results of models on the SVAMP challenge set.  $S$  indicates that the model is trained from scratch.  $R$  indicates that the model was trained with RoBERTa embeddings. The first row shows the results for the full dataset. The next two rows show the results for subsets of SVAMP composed of examples that have equations with one operator and two operators respectively. The last four rows show the results for subsets of SVAMP composed of examples of type Addition, Subtraction, Multiplication and Division respectively.
The best performing Graph2Tree model is only able to achieve an accuracy of  $43.8\%$  on SVAMP. This indicates that the problems in SVAMP are indeed more challenging for the models than the problems in ASDiv-A and MAWPS despite being of the same scope and type and less diverse. Table 23 in the Appendix lists some simple examples from SVAMP on which the best performing model fails. These results lend further support to our claim that existing models cannot robustly solve elementary level word problems.
Next, we remove the questions from the examples in SVAMP and evaluate them using the three models with RoBERTa embeddings trained on combined MAWPS and ASDiv-A. The scores can be seen in Table 11. The accuracy drops by half when compared to ASDiv-A and more than half compared to MAWPS suggesting that the problems in SVAMP are more sensitive to the information present in the question. We also evaluate the performance of the constrained model on SVAMP when trained on MAWPS and ASDiv-A. The best model achieves only  $18.3\%$  accuracy (see Table 12) which
Model	SVAMP w/o ques	ASDiv-A w/o ques
Seq2Seq	29.2	58.7
GTS	28.6	60.7
Graph2Tree	30.8	64.4
Table 11: Accuracies  $(\uparrow)$  of models on SVAMP without questions. The 5-fold CV accuracy scores for ASDiv-A without questions are restated for easier comparison.
Model	SVAMP
FFN + LSTM Decoder (S)	17.5
FFN + LSTM Decoder (R)	18.3
Majority Template Baseline	11.7
Table 12: Accuracies  $(\uparrow)$  of the constrained model on SVAMP. (R) denotes that the model is provided with non-contextual RoBERTa pretrained embeddings while (S) denotes that the model is trained from scratch.
is marginally better than the majority template baseline. This shows that the problems in SVAMP are less vulnerable to being solved by models using simple patterns and that a model needs contextual information in order to solve them.
We also explored using SVAMP for training by combining it with ASDiv-A and MAWPS. We performed 5-fold cross-validation over SVAMP where the model was trained on a combination of the three datasets and tested on unseen examples from SVAMP. To create the folds, we first divide the seed examples into five sets, with each type of example distributed nearly equally among the sets. A fold is obtained by combining all the examples in SVAMP that were created using the seed examples in a set. In this way, we get five different folds from the five sets. We found that the best model achieved about  $65\%$  accuracy. This indicates that even with additional training data existing models are still not close to the performance that was estimated based on prior benchmark datasets.
To check the influence of different categories of variations in SVAMP, for each category, we measure the difference between the accuracy of the best model on the full dataset and its accuracy on a subset containing no example created from that category of variations. The results are shown in Table 13. Both the Question Sensitivity and Struc
Removed Category	# Removed Examples	Change in Accuracy (Δ)
Question Sensitivity	462	+13.7
Reasoning Ability	649	-3.3
Structural Invariance	467	+4.5
Table 13: Change in accuracies when categories are removed. The Change in Accuracy  $\Delta = Acc(Full - Cat) - Acc(Full)$ , where  $Acc(Full)$  is the accuracy on the full set and  $Acc(Full - Cat)$  is the accuracy on the set of examples left after removing all examples which were created using Category  $Cat$  either by itself, or in use with other categories.
Removed Variation	# Removed Examples	Change in Accuracy (Δ)
Same Obj, Diff Struct	325	+7.3
Diff Obj, Same Struct	69	+1.5
Diff Obj, Diff Struct	74	+1.3
Add Rel Info	264	+5.5
Change Info	149	+3.2
Invert Operation	255	-10.2
Change order of Obj	107	+2.3
Change order of Phrases	152	-3.3
Add Irrel Info	281	+6.9
Table 14: Change in accuracies when variations are removed. The Change in Accuracy  $\Delta = Acc(Full - Var) - Acc(Full)$ , where  $Acc(Full)$  is the accuracy on the full set and  $Acc(Full - Var)$  is the accuracy on the set of examples left after removing all examples which were created using Variation Var either by itself, or in use with other variations.
tural Invariance categories of variations show an increase in accuracy when their examples are removed, thereby indicating that they make SVAMP more challenging. The decrease in accuracy for the Reasoning Ability category can be attributed in large part to the Invert Operation variation. This is not surprising because most of the examples created from Invert Operation are almost indistinguishable from examples in ASDiv-A, which the model has seen during training. The scores for each individual variation are provided in Table 14.
We also check the break-up of performance of the best performing Graph2Tree model according to the number of numbers present in the text of the input problem. We trained the model on both ASDiv-A and MAWPS and tested on SVAMP and compare those results against the 5-fold cross-validation setting of ASDiv-A. The scores are provided in Table 15. While the model can solve many problems consisting of only two numbers in the input text (even in our challenge set), it performs very badly on problems having more than two numbers. This shows that current methods are incapable of properly associating numbers to their context. Also, the gap between the performance on ASDiv-A and SVAMP is high, indicating that the examples in SVAMP are more difficult for these models to solve than the examples in ASDiv-A even when considering the structurally same type of word problems.
Going back to the original question, are existing NLP models able to solve elementary math word
Dataset	2 nums	3 nums	4 nums
ASDiv-A	93.3	59.0	47.5
SVAMP	78.3	25.4	25.4
Table 15: Accuracy break-up according to the number of numbers in the input problem. 2 nums refers to the subset of problems which have only 2 numbers in the problem text. Similarly, 3 nums and 4 nums are subsets that contain 3 and 4 different numbers in the problem text respectively.
problems? This paper gives a negative answer. We have empirically shown that the benchmark English MWP datasets suffer from artifacts making them unreliable to gauge the performance of MWP solvers: we demonstrated that the majority of problems in the existing datasets can be solved by simple heuristics even without word-order information or the question text.
The performance of the existing models in our proposed challenge dataset also highlights their limitations in solving simple elementary level word problems. We hope that our challenge set SVAMP, containing elementary level MwPs, will enable more robust evaluation of methods. We believe that methods proposed in the future that make genuine advances in solving the task rather than relying on simple heuristics will perform well on SVAMP despite being trained on other datasets such as ASDiv-A and MAWPS.
In recent years, the focus of the community has shifted towards solving more difficult MwPs such as non-linear equations and word problems with multiple unknown variables. We demonstrated that the capability of existing models to solve simple one-unknown arithmetic word problems is overestimated. We believe that developing more robust methods for solving elementary MwPs remains a significant open problem.
We thank the anonymous reviewers for their constructive comments. We would also like to thank our colleagues at Microsoft Research for providing valuable feedback. We are grateful to Monojit Choudhury for discussions about creating the dataset. We thank Kabir Ahuja for carrying out preliminary experiments that led to this work. We also thank Vageesh Chandramouli and Nalin Patel for their help in dataset construction.
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2357-2367, Minneapolis, Minnesota. Association for Computational Linguistics.
Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49-72.
Zheng Cai, Lifu Tu, and Kevin Gimpel. 2017. Pay attention to the ending: strong neural baselines for the ROC story cloze task. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 616-622, Vancouver, Canada. Association for Computational Linguistics.
Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020. Evaluating models' local decision boundaries via contrast sets.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107-112, New Orleans, Louisiana. Association for Computational Linguistics.
Danqing Huang, Shuming Shi, Chin-Yew Lin, and Jian Yin. 2017. Learning fine-grained expressions to solve math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 805-814, Copenhagen, Denmark. Association for Computational Linguistics.
Danqing Huang, Shuming Shi, Chin-Yew Lin, Jian Yin, and Wei-Ying Ma. 2016a. How well do computers solve math word problems? large-scale dataset construction and evaluation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 887-896, Berlin, Germany. Association for Computational Linguistics.
Danqing Huang, Shuming Shi, Chin-Yew Lin, Jian Yin, and Wei-Ying Ma. 2016b. How well do computers solve math word problems? large-scale dataset construction and evaluation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 887-896, Berlin, Germany. Association for Computational Linguistics.
Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. MAwPS: A math word problem repository. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1152-1157, San Diego, California. Association for Computational Linguistics.
Tal Linzen. 2020. How can we accelerate progress towards human-like linguistic generalization? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5210-5217, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412-1421, Lisbon, Portugal. Association for Computational Linguistics.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448, Florence, Italy. Association for Computational Linguistics.
Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020. A diverse corpus for evaluating and developing English math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975-984, Online. Association for Computational Linguistics.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885-4901, Online. Association for Computational Linguistics.
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018.
Hypothesis only baselines in natural language inference. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 180-191, New Orleans, Louisiana. Association for Computational Linguistics.
Jinghui Qin, Lihui Lin, Xiaodan Liang, Rumin Zhang, and Liang Lin. 2020. Semantically-aligned universal tree-structured solver for math word problems.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902-4912, Online. Association for Computational Linguistics.
Shachar Rosenman, Alon Jacovi, and Yoav Goldberg. 2020. Exposing Shallow Heuristics of Relation Extraction Models with Challenge Data. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3702-3710, Online. Association for Computational Linguistics.
Subhro Roy and Dan Roth. 2018. Mapping to declarative knowledge for word problem solving. Transactions of the Association for Computational Linguistics, 6:159-172.
Mrinmaya Sachan and Eric Xing. 2017. Learning to solve geometry problems from natural language demonstrations in textbooks. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), pages 251-261, Vancouver, Canada. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 845-854, Copenhagen, Denmark. Association for Computational Linguistics.
Sarah Wegreffe and Yuval Pinter. 2019. Attention is not not explanation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 11-20, Hong Kong, China. Association for Computational Linguistics.
Zhipeng Xie and Shichao Sun. 2019. A goal-driven tree-structured neural model for math word problems. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 5299-5305. International Joint Conferences on Artificial Intelligence Organization.
D. Zhang, L. Wang, L. Zhang, B. T. Dai, and H. T. Shen. 2020. The gap of semantic parsing: A survey on automatic math word problem solvers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(9):2287-2305.
Jipeng Zhang, Lei Wang, Roy Ka-Wei Lee, Yi Bin, Yan Wang, Jie Shao, and Ee-Peng Lim. 2020. Graph-to-tree learning for solving math word problems. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3928-3937, Online. Association for Computational Linguistics.
We use 8 NVIDIA Tesla P100 GPUs each with 16 GB memory to run our experiments. The hyperparameters used for each model are shown in Table 16. The best hyperparameters are highlighted in bold. Following the setting of Zhang et al. (2020), the arithmetic word problems from MAwPS are divided into five folds, each of equal test size. For ASDiv-A, we consider the 5-fold split [238, 238, 238, 238, 266] provided by the authors (Miao et al., 2020).
B Creation Protocol
We create variations in template form. Generating more data by scaling up from these templates or by performing automatic operations on these templates is left for future work. The template form of an example is created by replacing certain words with their respective tags. Table 17 lists the various tags used in the templates.