SlowGuess/NAACL_2022_OCR huggingface dataset:
select three state-of-the-art MWP solvers: (1) Seq2Seq (Wang et al.) having an LSTM encoder and an attention based decoder. (2) GTS (Xie and Sun, 2019) having an LSTM encoder and a tree based decoder and (3) Graph2tree (Zhang et al.) consists of a both tree based encoder and decoder. Seq2Seq serves as our base model for experimentation. Many existing datasets are not suitable for our analysis as either they are in Chinese (Wang et al.) or they have problems of higher complexities (Huang et al., 2016). We conduct experiments across the two largest available English language datasets satisfying our requirements: (1) MaWPS (Koncel-Kedziorski et al., 2016) containing 2,373 problems (2) ASDiv-A (Miao et al., 2020) containing 1,213 problems. Both datasets have MwPs with linear equation in one variable.
Experiment Setup: We train and evaluate the three solvers on both MaWPS and ASDiv-A using five fold cross validation. Evaluation is conducted on both original and augmented datasets. We use the same hyperparameter values as recommended in the original implementation of these solvers. Further, each solver has been trained from scratch and by using BERT embeddings (Devlin et al., 2019b). We also evaluate the models on SVAMP (Patel et al., 2021) challenge set. This test set has been designed specifically to examine the robustness and adaptability of the solvers. Ablation studies have been conducted to assess the effectiveness of candidate selection algorithm and augmentation techniques.
Dataset	Evaluation Type	Seq2Seq	GTS	G2T
MaWPS	True	84.6	87.5	88.7
Paraphrasing	88.3	90.4	92.6
Substitution	89.2	89.7	91.7
Combined	91.3	92.6	93.5
ASDiv-A	True	70.6	80.3	82.7
Paraphrasing	75.6	84.2	83.6
Substitution	73.2	83.3	84.1
Combined	78.2	85.9	86.3
Table 5: Result of augmentation methods. True is original dataset, Combined is combination of paraphrasing and substitution. G2T represents Graph2Tree solver.
Problem 1: Ricardo was making baggies of cookies with 5 cookies in each bag. If he had 7 chocolate chip cookies and 3 oatmeal cookies, how many baggies could he make? Solution Equation: X = (7+3)/5
Pre Augmentation Equation: X = (7/3)/3
Post Augmentation Equation: X = (7+3)/5
Problem 2: For hallowen Destiny bought 9 pieces of candy She ate 3 pieces the first night and then her sister gave her 2 more pieces. How many pieces of candy does Destiny have now? Solution Equation: X = 9-3+2
Pre Augmentation Equation: X = ((9+3-3)
Post Augmentation Equation: X = (9+3-2)
Problem 3: Audrey needs 6 cartons of berries to make a berry cobbler. She already has 2 cartons of strawberries and 3 cartons of blueberries. How many more cartons of berries should Audrey buy? Solution Equation: X = 6-2-3
Pre Augmentation Equation: X = (6-(2)+3)
Post Augmentation Equation: X = 6-(2+3)
Table 6: Examples illustrating equation results before and after training on the full augmented dataset.
Table 5 shows the result of proposed methods. These results have been reported on BERT embeddings. Table 11 shows a comparison between training from scratch and using BERT embeddings. By training these state-of-the-art models on the augmented dataset we achieve better results for both MaWPS and ASDiv-A. On average, we were able to increase the accuracy significantly by more than five percentage points. Both paraphrasing and substitution methods have performed well independently and in combination. Further, we conduct ablation studies to analyze the performance of each augmentation method. In Table 6 we illustrate some examples on which existing models generate incorrect equations. However, after being trained with augmented dataset they generate correct equations. Additionally, in Problem 2 the base
Problem:Gavin has 6 shirts.3 are blue the rest are green.How many green shirts does Gavin have?Mean attention values:0.270.140.08
Problem:Gavin has 6 shirts.3 are blue the rest are green.How many green shirts does Gavin have?Augmented mean attention values:0.230.180.11
Problem:There are 3 pencils in the drawer.Sara placed 7 more pencils in the drawer.How many pencils are there in all?Mean attention values:0.450.110.05
Problem:There are 3 pencils in the drawer.Sara placed 7 more pencils in the drawer.How many pencils are there in all?Augmented mean attention values:0.310.160.09
Table 7: Examples illustrating distribution of top three attention weights before and after training on the full augmented dataset.
model generates syntactically incorrect solution, but post augmentation it generates syntactically correct equation. These examples show the increased robustness and solving abilities of solvers.
Attention Visualizations: Through this investigation, we aim to ascertain our hypothesis that to generate equations MWP solvers focus only on certain keywords and patterns in a region. They ignore essential information like semantics, sequence of events and content of the question text present in the problem statement. In Table 7, we show some sample problem statements with their attention weights. These weights are generated during the decoding process using Luong attention mechanism (Luong et al., 2015). Moreover, to illustrate the effectiveness of our augmentation techniques, we show the distribution of attention weights for models trained on the augmented dataset. We can infer from the examples showcased in Table 7 that before augmentation the focus of the solver is limited to a fixed region around numerical quantities and it does not pay heed to the question text. However, after training on the augmented dataset the solver has a better distribution of attention weights, the weights are not localised and and the model is also able to pay attention on the question text.
Ablation Studies: To assert the effectiveness of our methods, we conduct the following ablations:
Method	Eval Type	Seq2Seq	GTS	Graph2Tree
RSA	True	84.6	87.5	88.7
Paraphrasing	85.3	88.1	89.2
Substitution	86.8	87.3	87.9
Combined	87.0	89.2	89.5
CSA	True	84.6	87.5	88.7
Paraphrasing	88.3	90.4	92.6
Substitution	89.2	89.7	91.7
Combined	91.3	92.6	93.5
Candidate Selection Algorithm: For testing the usefulness of candidate selection algorithm, we compare it with a random selection algorithm. In this, we randomly select one of the possible candidates as augmented problem statement. We evaluate the accuracy of models trained on the augmented datasets, generated using both the algorithms. Result in Table 8, shows that candidate selection algorithm performs better than random selection algorithm and this demonstrates the effectiveness of our algorithm.
Augmentation Methods: To examine the effectiveness of proposed augmentation techniques, we evaluate the models on each of the proposed techniques independently and report the results in Table 9. Although, all the methods contribute towards increase in accuracy but Round trip translations and synonym replacement perform marginally better than others. This behaviour can be linked to the structural diversity and keyword sensitivity that round trip translations and synonym replacement bring respectively (Feng et al., 2021).
Table 8: Ablation Study for Random Selection Algorithm (RSA) and Candidate Selection Algorithm (CSA).
Augmentation	Seq2Seq	GTS	Graph2Tree
True	84.6	87.5	88.7
RRT	86.5	89.1	91.6
PR	85.9	88.4	90.7
FM	84.8	87.2	89.1
SR	85.2	90.1	91.2
NER	86.1	88.3	89.7
SVAMP Challenge Set: SVAMP (Patel et al., 2021) is a manually curated challenge test set
Table 9: Result of Ablation study for each augmentation method. True represents unaugmented MaWPS dataset, RRT, PR, FM, SR, NER represents round trip translations, problem reordering, fill masking,synonym replacement and named entity replacement respectively.
Augmentation	Seq2Seq	GTS	Graph2Tree
True	37.5	39.6	41.2
MaWPS(P+S)	39.2	40.1	42.3
ASDiv-A(P+S)	37.8	40.4	42.1
Combined	40.2	41.3	43.8
consisting of 1,000 math word problems. These problems have been cherry picked from MaWPS and ASDiv-A, then altered manually to modify the semantics of question text and generate additional equation templates. This challenge set is suitable for evaluating a solver's performance as it modifies problem statements such that solver's generalization can be checked. The results are shown in Table 10. Although, our proposed augmented dataset has very limited equation templates, still it performs comparatively better than state-of-the-art models on SVAMP challenge set. This result signifies the need for a larger and diverse dataset with enhanced variety of problems. Further, it demonstrates the effectiveness of our method which is able to perform better on SVAMP test set and increase model's accuracy despite the challenges.
Table 10: Result of augmentations on SVAMP Challenge Set. P and S represent paraphrasing and substitution methods. Combined represents augmented MaWPS and ASDiv-A. True is combined MaWPS and ASDiv-A.
Augmentation Method	MaWPS	ASDiv-A
Scratch	BERT	Scratch	BERT
True	77.2	84.6	53.2	70.6
Paraphrasing	79.8	88.3	58.1	75.6
Substitution	81.3	89.2	57.3	73.2
Combined	82.7	91.3	60.4	78.2
Table 11: Performance comparison of baseline model trained from scratch and trained using BERT embeddings. True represents unaugmented dataset.
BERT Embeddings: We train the solvers in two different settings, using pre-trained BERT embeddings and training from scratch. We chose BERT specifically as we require contextual embeddings which could be easily adapted for the task of MWP. Moreover, existing models have also shown results using BERT and it would be fair to compare their performances when trained using similar embeddings. Results obtained are shown in Table 11. We observe that for solver's trained using BERT, accuracy is higher than models trained from scratch.
Human Evaluation: To verify the quality of augmented examples, we conduct human evaluation. The focus of this evaluation is: (1) To check if the augmentations will result in the same linear equation as present in the original problem statement, (2) To evaluate if the numerical values for each augmentation example is preserved, (3) Evaluate each sample in the range 0 to 1 for its semantic similarity with the original problem statement, (4) On a scale of 1 to 5 rate each augmented example for its grammatical correctness. We conduct the human evaluations on randomly shuffled subsets consisting of around  $40\%$  of the total augmented examples for both the datasets. This process is repeated three times with different subsets, five human evaluators evaluate each example in all subsets, and the mean results are computed as shown in Table 12.
Evaluation Criteria	MaWPS	ASDiv-A
Para	Sub	Para	Sub
Preserves Equation	92.3%	89.5%	93.6%	90.1%
Preserves Numbers	88.4%	91.2%	87.3%	90.3%
Semantic Similarity	0.96	0.89	0.91	0.87
Syntactic Similarity	4.67	4.36	4.59	4.33
Table 12: Human Evaluation scores on augmented dataset. Para and Sub represents paraphrasing and substitution methods respectively.
5 Future Work and Conclusion
We showcase that the existing MWP solvers are not robust and do not generalize well on even simple variations of the problem statement. In this work, we have introduced data augmentation techniques for generation of diverse math word problems. We were able to enhance the size of existing dataset by 5 folds and significantly increase the performance of state-of-the-art solvers by over 5 percentage points. Future works could focus on developing techniques to generate data artificially and making robust MWP solvers.
We thank IHub-Data, IIIT Hyderabad<sup>3</sup> for financial support.
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang.
2018. Generating natural language adversarial examples.
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. Mathqa: Towards interpretable math word problem solving with operation-based formalisms.
Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation.
Ting-Rui Chiang and Yun-Nung Chen. 2018. Semantically-aligned equation generation for solving and reasoning math word problems. arXiv preprint arXiv:1811.00720.
J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019a. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019b. Bert: Pre-training of deep bidirectional transformers for language understanding.
Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021. A survey of data augmentation approaches for nlp.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. *Neural Computation*, 9(8):1735–1780.
Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. pages 523-533, Doha, Qatar. Association for Computational Linguistics.
Danqing Huang, Shuming Shi, Chin-Yew Lin, Jian Yin, and Wei-Ying Ma. 2016. How well do computers solve math word problems? large-scale dataset construction and evaluation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 887-896, Berlin, Germany. Association for Computational Linguistics.
Yunsu Kim, Petre Petrov, Pavel Petrushkov, Shahram Khadivi, and Hermann Ney. 2019. Pivot-based transfer learning for neural machine translation between non-English languages. pages 866-876, Hong Kong, China. Association for Computational Linguistics.
Sosuke Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations.
Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. MAwPS: A math word problem repository. In Proceedings of the 2016 Conference of the North American Chapter
3 https://ihub-data.ijit.ac.in/
of the Association for Computational Linguistics: Human Language Technologies, pages 1152-1157, San Diego, California. Association for Computational Linguistics.
Ashutosh Kumar, Satwik Bhattachamishra, Manik Bhandari, and Partha Talukdar. 2019. Submodular optimization-based diverse paraphrasing and its effectiveness in data augmentation. pages 3609-3619, Minneapolis, Minnesota. Association for Computational Linguistics.
Vivek Kumar, Rishabh Maheshwary, and Vikram Pudi. 2021. Adversarial examples for evaluating math word problem solvers. In *Findings of the Association for Computational Linguistics: EMNLP* 2021, pages 2705-2712, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. 2014. Learning to automatically solve algebra word problems. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 271-281, Baltimore, Maryland. Association for Computational Linguistics.
Shayne Longpre, Yu Wang, and Christopher DuBois. 2020. How effective is task-agnostic data augmentation for pretrained transformers?
Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation.
Rishabh Maheshwary, Saket Maheshwary, and Vikram Pudi. 2021a. Generating natural language attacks in a hard label black box setting. In AAAI.
Rishabh Maheshwary, Saket Maheshwary, and Vikram Pudi. 2021b. A strong baseline for query efficient attacks in a black box setting.
Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020. A diverse corpus for evaluating and developing English math word problem solvers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 975-984, Online. Association for Computational Linguistics.
Arkil Patel, Satwik Bhattachamishra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems?
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks.
Georgios Rizos, K. Hemker, and Björn Schuller. 2019. Augment to prevent: Short-text data augmentation in deep learning for hate-speech classification. Proceedings of the 28th ACM International Conference on Information and Knowledge Management.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data.
Hao Tan, Licheng Yu, and Mohit Bansal. 2019. Learning to navigate unseen environments: Back translation with environmental dropout.
Luke Taylor and Geoff Nitschke. 2018. Improving deep learning with generic data augmentation. In 2018 IEEE Symposium Series on Computational Intelligence (SSCI), pages 1542-1547.
Elena Voita, Rico Sennrich, and Ivan Titov. 2019. Context-aware monolingual repair for neural machine translation.
Lei Wang, D. Zhang, Jipeng Zhang, Xing Xu, L. Gao, B. Dai, and H. Shen. 2019. Template-based math word problem solvers with recursive neural networks. In AAAI.
Yan Wang, Xiaojiang Liu, and Shuming Shi. Deep neural solver for math word problems.
Jason Wei and Kai Zou. EDA: Easy data augmentation techniques for boosting performance on text classification tasks.
Sam Witteveen and Martin Andrews. Paraphrasing with large language models.
Yonghui Wu, Mike Schuster, Z. Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason R. Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. ArXiv, abs/1609.08144.
Zhipeng Xie and Shichao Sun. 2019. A goal-driven tree-structured neural model for math word problems. In *IJCAI*, pages 5299–5305.
Jipeng Zhang, Lei Wang, Roy Ka-Wei Lee, Yi Bin, Yan Wang, Jie Shao, and Ee-Peng Lim. Graph-to-tree learning for solving math word problems.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2016. Character-level convolutional networks for text classification.
7.1 Implementation Details
For conducting our experiments we have used two Boston SYS-7048GR-TR nodes equipped with NVIDIA GeForce GTX 1080 Ti computational GPU's having 11GB of GDDR5X RAM. All implementations of training and testing is coded in Python with Pytorch framework. The number of parameters range from 20M to 130M for different models. We use negative log likelihood as the loss criterion. Hyper-parameter values were not modified, and we follow the recommendations of the respective models. To reduce carbon footprint from our experiments, we run the models only on a single fold for searching hyperparameter values. We chose the number of base candidates after primary stage  $n$  as 7. Generating augmentation examples using Paraphrasing Methods took around 12 minutes on average for MaWPS and 8 minutes for ASDiv-A datasets. Substitution methods took around 5 minutes on average for both MaWPS and ASDiv-A dataset. The experiments conducted by us are not computation heavy. Each of the state-of-the-art models get trained within 5 hrs of time, with Graph2Tree taking the maximum time.
7.2 Additional Augmented Examples
In this section, we present some additional valid as well as invalid augmented examples. Additionally, we also show some more examples with their attention weights. Table 13 shows some additional examples with their attention weight distribution. These weights have been shown for the base model trained before augmentation and after augmentation on MaWPS dataset. Table 14 illustrates some additional problem statements for all the techniques in paraphrasing methods and substitution methods. In Table 15, we present some invalid augmented examples which do not satisfy our human evaluation criteria. These examples are such that they alter the semantics of the original problem statement.
Problem: A magician was selling magic card decks for 2 dollars each. If he started with 25 decks and by the end of the day he had 4 left, how much money did he earn? Mean attention values: 0.34 0.11 0.09
Problem: A magician was selling magic card decks for 2 dollars each. If he started with 25 decks and by the end of the day he had 4 left, how much money did he earn? Augmented mean attention values : 0.19 0.18 0.15
Problem: There are 18 pencils in the drawer and 6 pencils on the desk. Dan placed 4 pencils on the desk. How many pencils are there in total? Mean attention values: 0.21 0.16 0.06
Problem: There are 18 pencils in the drawer and 6 pencils on the desk.  
Dan placed 4 pencils on the desk. How many pencils are there in total?  
Augmented mean attention values : 0.29 0.19 0.12
Problem: Dan has 12 violet marbles, he gave Mary 4 of the marbles. How many violet marbles does he now have? Mean attention values: 0.23 0.21 0.17
Problem: Dan has 12 violet marbles, he gave Mary 4 of the marbles. How many violet marbles does he now have? Augmented mean attention values : 0.23 0.18 0.11
Problem: Angela has 7 tickets. Annie gives Angela  
5 more. How many tickets does Angela have in all? Mean attention values: 0.30 0.19 0.15  
Problem: Angela has 7 tickets. Annie gives Angela  
5 more. How many tickets does Angela have in all? Augmented mean attention values: 0.29 0.21 0.14
Problem: Maria had 5 bottles of water in her fridge. If she drank 1 of them and then bought 2 more, how many bottles would she have? Mean attention values: 0.48 0.14 0.04 Problem Maria had 5 bottles of water in her fridge. If she drank 1 of them and then bought 2 more , how many bottles would she have? Augmented mean attention values : 0.23 0.17 0.11
Table 13: Examples illustrating distribution of top three attention weights before and after training on the full augmented dataset.
Category	Augmentation Method	Example
Paraphrasing Methods	Round trip Translation	Original: Alyssa's dog had puppies. She gave 2 to her friends. She now has 3 puppies. How many puppies did she have to start with?
Augmented: Alyssa's dog had puppies. She gave her friends 2 . She now has 3 puppies. How many puppies did she start ?
Problem Reordering	Original: Rachel was organizing her book case making sure each of the shelves had exactly 3 books on it. If she had 4 shelves of mystery books and 2 shelves of picture books, how many books did she have total?
Augmented: How many books did she have given thatRachel was or-
organizing her book case making sure each of the shelves had exactly 3
books on it and she had 4 shelves of mystery books and 2 shelves of
picture books .
Substitution Methods	Fill Masking	Original: A cell phone company has a total of 1000 customers across
the world. If 740 of its customers live in the United States, how many
of its customers live in other countries?
Augmented: A mobile phone firm has a network of 1000 customers
across the world. If 740 of its customers live in the violetUS, How many
customers live in other locations?
Named-Entity Replacement	Original: Daniel had some noodles. He gave 20 noodles to William.
Now Daniel only has 11 noodles. How many noodles did Daniel have to
begin with?
Augmented: Matt had some noodles. He gave 20 noodles to Zeal. Now
Matt only has 11 noodles. How many noodles did Matt have initially
? Edd found 7 seashells, Alan found 12 seashells, and Royal found 5
seashells on the beach. How many seashells were found together?
Synonym Replacement	Original: There are 5 rulers in the drawer. Tim took 3 rulers from the
drawer. How many rulers are now in the drawer?
Augmented: There are 5 consonants in the drawer. Tim went 3 conso-
nants from the drawer. How many other consonants are in the drawer
now ?
Table 14: Valid Augmentation examples from all proposed methods. Coloured text represents the changes in problem statement.