Venue: NAACL 2022

# Practice Makes a Solver Perfect: Data Augmentation for Math Word Problem Solvers

Vivek Kumar, Rishabh Maheshwary and Vikram Pudi

Data Sciences and Analytics Center, Kohli Center on Intelligent Systems

International Institute of Information Technology, Hyderabad, India {vivek.k, rishabh.maeshwary}@research.iiit.ac.in, vikram@iitit.ac.in

###### Abstract

Existing Math Word Problem (MWP) solvers have achieved high accuracy on benchmark datasets. However, prior works have shown that such solvers do not generalize well and rely on superficial cues to achieve high performance. In this paper, we first conduct experiments to showcase that this behaviour is mainly associated with the limited size and diversity present in existing MWP datasets. Next, we propose several data augmentation techniques broadly categorized into _Substitution_ and _Paraphrasing_ based methods. By deploying these methods we increase the size of existing datasets by five folds. Extensive experiments on two benchmark datasets across three state-of-the-art MWP solvers shows that proposed methods increase the generalization and robustness of existing solvers. On average, proposed methods significantly increase the state-of-the-art results by over five percentage points on benchmark datasets. Further, the solvers trained on the augmented dataset performs comparatively better on the challenge test set. We also show the effectiveness of proposed techniques through ablation studies and verify the quality of augmented samples through human evaluation.

+
Footnote †: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4194 - 4206

+
Footnote †: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4194 - 4206

+
Footnote †: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4194 - 4206

## 1 Introduction

A Math Word Problem (MWP) consists of natural language text which describes a world state involving some known and unknown quantities, followed by a question text to determine the unknown values. The task is to parse the problem statement and generate equations that can help find the value of unknown quantities. An example of a simple MWP is shown in Table 1. In recent years, the challenge of solving MWP has gained much attention in the NLP community as it needs the development of commonsense multi step reasoning with numerical quantities. With the rise of deep learning, performance of math solvers has also increased significantly over the years Wang et al.; Zhang et al.). However, recent analysis conducted in Kumar et al. (2021) and Patel et al. (2021) show that these deep learning based solvers rely on shallow heuristics to solve vast majority of problems. They curated adversarial examples and SVAMP challenge set respectively to infer that MWP solvers \((1)\) do not understand the relationship between numbers and their associated entities, \((2)\) do not focus on the question text and \((3)\) ignore word order information. In this paper, we first conduct experiments to establish that the above drawbacks are due to the limited size and diversity of problems present in the existing MWP datasets. Next, we propose various augmentation methods to create diverse and large number of training examples to mitigate these shortcomings. Our methods are focused on: \((1)\) Increasing the number of problems in the existing datasets and \((2)\) enhancing the diversity of the problem set.

Training deep neural models effectively requires large number of data points Longpre et al. (2020). Constructing large datasets which are annotated, labeled and have MWPs of similar difficulty level is a very expensive and tedious task. To address these key challenges, we resort to data augmenta

\begin{table}
\begin{tabular}{l} \hline \hline
**Original Problem** \\ Problem: Nancy grew 8 potatoes. Sandy grew 5 potatoes. \\ How many potatoes did they grow in total ? \\ True Equation: X = 8+5 \\ \hline
**Paraphrasing Method** \\ Problem: How many potatoes did they grow in all given that \\ nancy grew 8 potatoes and sandy grew 5 potatoes. \\ Equation Label: X = 8+5 \\ \hline
**Substitution Method** \\ Problem: Dwight grew 8 potatoes. Juliette grew 5 potatoes. \\ How many potatoes did they grow together ? \\ Equation Label: X = 8+5 \\ \hline \hline \end{tabular}
\end{table}
Table 1: A MWP and its augmentation examples generated by our methods with preserved equation labels. Blue and Violet colours denote the changes made after the primary stage and secondary stage respectively.



tion techniques. Our motivation behind generating augmentations is that humans require sufficient practice to understand MWPs. Humans learn to solve MWPs by going through a variety of similar examples and slowly become capable enough to tackle variations of similar difficulty levels. We aim to generate augmentations such that sufficient linguistic variations of a similar problem are present in the dataset. These variations will make the solver more robust in tackling MWP, increase their reasoning ability and numerical understanding.

Data augmentation for MWPs is a challenging task as we need to preserve the equation labels while generating new samples Kumar et al. (2021). The generated samples should be semantically similar to their original counterpart, \((2)\) must have the same numerical values and preserve relationship with their respective entities and \((3)\) should maintain the same sequence of events in the problem text. Existing augmentation methods Wei and Zou cannot be directly applied due to the above mentioned reasons. Our methods can be broadly classified as follows:

* **Paaphrasing Methods:** It generates variations of the question text by re-statement such that the semantic and syntactic meaning along with the equation labels is preserved.
* **Substitution Methods:** These methods generate variations of the problem statement by identifying and substituting some of the keywords such that the augmentations are semantically and syntactically correct.

To ensure high quality augmentations 1, we propose a selection algorithm which selects samples that have high similarity with original problem and incur high loss values when tested on existing solvers. This algorithm helps selecting only those samples that can make existing solvers more robust. Further, we also verify the validity and the quality of generated augmentations through human evaluation.

Footnote 1: Codebase and augmented datasets are available at: [https://github.com/kevivk/MWP-Augmentation](https://github.com/kevivk/MWP-Augmentation)

Most of the existing MWP datasets are either in languages other than English or contain problems of varying difficulty levels Koncel-Kedziorski et al. (2016); Wang et al. (2019); Huang et al. (2016); Amini et al. (2019); Miao et al. (2020). We focus on strengthening existing English language datasets which can facilitate the development of better MWP solvers. We consider datasets containing MWP that can be solved using linear equations in one variable. These datasets include MaWPS Koncel-Kedziorski et al. (2016) and ASDiv-A Miao et al. (2020) both having \(2,373\) and \(1,213\) problems respectively. Following are the key contributions made in this paper:

* To the best of our knowledge, this is the first work that extensively evaluates data augmentation techniques for MWP solving. This is the first attempt to generate MWP problems automatically without manual intervention.
* Accuracy of the state of the art solvers increases after training on the proposed augmented dataset. This demonstrates the effectiveness of our methods. To verify the validity of generated augmentations we conduct human evaluation studies.
* We increase the diversity of the training dataset through augmentations and obtain comparatively better results than state-of-the-art solvers on the SVAMP challenge set.

## 2 Related Work

**Math Word Solvers:** Many research efforts have been undertaken in the recent past to solve the challenging MWP task. Broadly, these solvers can be categorized into statistical learning based and deep learning based models. Traditional approaches focused more on statistical machine learning Kushman et al. (2014); Hosseini et al. (2014) with the aim of categorizing equations into templates and extracting key patterns in the problem text. Recently, due to the advent of deep learning in NLP, solvers have witnessed a considerable increase in their performances. Wang et al. (2019) modelled MWP task as a sequence to sequence task and used LSTM's Hochreiter and Schmidhuber (1997) for learning problem representations. Chiang and Chen (2018) focused on learning representations for operators and operandsWang et al. (2019); Xie and Sun (2019) used tree structures for decoding process. Zhang et al. (2015) modelled question as a graph to map quantities and their attributes. Existing datasets which have been used as benchmark for english language includes MaWPS Koncel-Kedziorski et al. (2016) and Chinese language dataset Math23K Wang et al. (2019). These datasets although constrained by 

their size deal with algebraic problems of similar difficulty levels. Recently, ASDiv Miao et al. (2020) has been proposed, which has diverse problems which includes annotations for equations, problem type and grade level. Other large datasets in English language include MathQA Amini et al. (2019) and Dolphin18k Huang et al. (2016). Although, these datasets have larger problem set but they are noisy and contain problems of varied difficulty levels.

Text Data Augmentation:To effectively train deep learning models, large datasets are required. Data augmentation is a machine learning technique that artificially enlarges the amount of training data by means of label preserving transformations Taylor and Nitschke (2018). Longpre et al. (2020) hypothesize that textual data augmentation would only be helpful if the generated data contains new linguistic patterns that are relevant to the task and have not been seen in pre-training. In NLP, many techniques have been used for generating augmentations, Wei and Zou (2019) introduced noise injection, deletion, insertion and swapping of words in text. Rizos et al. (2019) used recurrent neural networks and generative adversarial networks for short-text augmentation Maheshwary et al. (2021). Recently, hard label adversarial attack models have also been used Maheshwary et al. (2021). Other frequently used methods include inducing spelling mistakes Belinkov and Bisk (2018), synonym replacement Zhang et al. (2016), identifying close embeddings from a defined search space Alzantot et al. (2018), round trip translations Sennrich et al. (2016), paraphrasing techniques Kumar et al. (2019) and words predicted by language model Kobayashi (2018) among many others. These methods are specific to the task at hand and needs to be adapted such that the generated augmentations bring diversity in the concerned dataset.

## 3 Proposed Augmentation Approach

Data augmentation generates new data by modifying existing data points through transformations based on prior knowledge about the problem domain. We introduce carefully selected transformations on well known text augmentation techniques to develop examples suited for the task of MWP. These transformations help in increasing the diversity and size of problem set in existing datasets.

### Problem Definition

A MWP is defined as an input of \(n\) tokens, \({\cal P}=\{w_{1},w_{2}..w_{n}\}\) where each token \(w_{i}\) is either a numeric value or a word from a natural language. The goal is to generate a valid mathematical equation \({\cal E}_{\cal P}\) from \({\cal P}\) such that the equation consists of numbers from \({\cal P}\), desired numerical constants and mathematical operators from the set \(\{/,*,+,-,=,(,)\}\). Let \({\cal F}:{\cal P}\rightarrow{\cal E}_{\cal P}\) be an MWP solver where \({\cal E}_{\cal P}\) is the equation to problem \({\cal P}\). Our task is to generate augmented problem statement \({\cal P}^{*}\) from the original input \({\cal P}\) such that \({\cal P}^{*}\) is: \((1)\) semantically similar to the initial input \({\cal P}\), \((2)\) preserves the sequence of events in the problem statement, \((3)\) keeps the numerical values intact and \((4)\) the solution equation is same as \({\cal E}_{\cal P}\).

### Deficiencies in Existing Models

As showcased by Patel et al. (2021), existing MWP solvers trained on benchmark datasets like MaWPS and ASDiv-A focus their attention only on certain keywords in the problem statement and do not pay much heed to the question text. We further show that even after performing significant transformations on the test set such as \((1)\) dropping the question text, \((2)\) randomly shuffling the sequence of sentences, \((3)\) random word deletion, and \((4)\) random word reordering, the solvers are still able to produce correct equations. Upon introducing these transformations we should expect a very high drop in accuracy values as the transformed problems are now distorted. Surprisingly, the decrease in accuracy scores is relatively very less than expected as shown in Table 2. We only observe a relatively moderate drop for word reordering. From this analysis, we can say that instead of focusing on the sequence of events, question text and semantic representation of the problem, solvers pick word patterns and keywords from the problem statement. We hypothesize that the drop in accuracy for word reordering experiment indicates that the solvers try to identify a contiguous window of words having some keywords and numbers in them, and generates equation based on these keywords. We further probe on this hypothesis by visualizing the attention weights in the experiment section.

### Augmentation Methods

A MWP can also be expressed as \({\cal P}=(S_{1},S_{2}..S_{k},Q)\) where \(Q\) is the question and \((S_{1},S_{2}..S_{k})\) are the sentences constituting the prob 

lem description. To mitigate the deficiencies in MWP solvers, we propose a two stage augmentation paradigm consisting of primary and secondary stage. In primary stage, we generate base augmentation candidates which then proceed to the secondary stage and get modified accordingly to become potential candidates. After identifying the potential candidates, we filter out the best candidates using proposed candidate selection algorithm. Table 1 shows changes in MWP after primary and secondary stage. Following are the details:

* **Primary Stage:** In the primary stage, our focus is on inducing variations in the question text \(Q\) of a given problem statement \(\mathcal{P}\). For this, we first generate \(n\) base candidates \(\{b_{1},b_{2},...,b_{n}\}\) from \(Q\) using \(T5\) paraphrasing model Raffel et al. (2020). The key intuition behind this step is to ensure that each augmentation of a given problem has a different question text. This will empower the solver to learn variations from the question text as well.
* **Secondary Stage:** After the generation of base candidates, we implement augmentation methods to generate potential candidates. These methods although well known, require careful tuning to adapt for MWP generation. Table 3 showcases MWP examples and their generated augmentations. Detailed description of these techniques follow.

#### 3.3.1 Paraphrasing Methods

Paraphrasing has proved to be an effective way of generating text augmentations Witteveen and Andrews). It generates samples having diverse sentence structures and word choices while preserving the semantic meaning of the text. These additional samples guide the model to pay attention to not only the keywords but its surroundings as well. This is particularly beneficial for the task of MWP solving, where most of the problem statements follow a general structure.

**Problem Reordering:** Given original problem statement \(\mathcal{P}=(S_{1},S_{2},...S_{k},Q)\), we alter the order of problem statement such that \(\mathcal{P^{*}}=(Q,S_{1},S_{2},...,S_{k})\). To preserve the semantic and syntactic meaning of problem statement we use filler phrases like 'Given that' and 'If-then'. To make these paraphrases more fluent, we use named entity recognition and co-reference resolution to replace the occurrences of pronouns with their corresponding references. Please note that this method is better than random shuffling of sentences as it preserves the sequence of events in the problem statement.

**Round Trip Translations:** Round trip translations, more commonly referred as back-translation is an interesting method to generate paraphrases. This idea has evolved as a result of the success of machine translation models Wu et al. (2016). In this technique, sentences are translated from their original language to foreign languages and then translated back to the original language. This round trip can be between multiple languages as well. The motivation behind using this technique is to utilize the different structural constructs and linguistic variations present in other languages.

Back-translation is known to diverge uncontrollably Tan et al. (2019) for multiple round trips. This may lead to change in the semantics of the problem statement. Numerical quantities are fragile to translations and their order and representation may change. To overcome these challenges, we worked with languages that have structural constructs similar with English. For instance, languages like Finnish which are gender neutral, can become problematic as they can lead to semantic variance in augmented examples. To preserve numerical quantities, we replace them with special symbols and keep a map to restore numerical quan

\begin{table}
\begin{tabular}{l|c|c|c|c} \hline \hline
**Dataset** & **Eval Type** & **Seq2Seq** & **GTS** & **Graph2Tree** \\ \hline \multirow{4}{*}{**MaWPS**} & True & 84.6 & 87.5 & 88.7 \\ \cline{2-5}  & WD & **80.2** & **81.5** & **77.3** \\ \cline{2-5}  & QR & **77.4** & **82.0** & **80.2** \\ \cline{2-5}  & SS & **77.0** & **60.4** & **66.4** \\ \cline{2-5}  & WR & **54.9** & **34.8** & **39.3** \\ \hline \multirow{4}{*}{**ASDiv-A**} & True & \(70.6\) & 80.3 & 82.7 \\ \cline{2-5}  & WD & **60.2** & **61.3** & **56.7** \\ \cline{1-1} \cline{2-5}  & QR & **58.7** & **52.4** & **54.1** \\ \cline{1-1} \cline{2-5}  & SS & **56.2** & **59.3** & **60.7** \\ \cline{1-1} \cline{2-5}  & WR & **47.1** & **32.3** & **34.6** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance of solvers on modified test sets. True represents unaugmented test set. WD,QR,SS,WR represent word deletion, question reordering, sentence shuffling and word reordering respectively.

 

tities in the generated paraphrases. We have used the following round trips:

_English - Russian - English:_ Although Russian is linguistically different from English, we still chose it as word order does not affect the syntactic structure of a sentence in Russian language Voita et al. (2019). For single round trip, we preferred Russian as it has the potential to generate different paraphrase structures.

_English - German - French - English:_German and french are structurally similar to English language Kim et al. (2019), we chose them for multiple round trips to both maintain semantic in-variance and induce minor alterations in the paraphrases.

#### 3.3.2 Substitution Methods

In this class of methods, the focus is on generating variations of the problem statement by identifying and substituting some of the keywords such that the augmentations are semantically and syntactically correct, with the equation labels preserved. Substitution is effective for MWP solving as it guides the solvers focus away from certain keywords, allowing it to distribute its attention and generalize better. We propose the following methods:

**Fill-Masking:** In this technique, we model the challenge of generating candidates as a masked language modelling problem. Instead of randomly choosing words for masking, we use part of speech tags to focus on nouns and adjectives, preferably in the vicinity of numerical quantities. We replace these identified keywords with mask tokens. These masked candidate sentences are then passed through a masked language model Devlin et al. (2019) and suitable words are filled in masked positions to generate our candidate sentences.

**Synonym Replacement:** In this method, after stop-word removal, we select keywords randomly for substitution. Unlike fill-mask technique, where masked language models were deployed, here we use Glove embeddings Pennington et al. (2014) to find the top \(k\) candidates that are close synonyms of the keywords. To ensure syntactic correctness in candidates, we maintain the part of speech tags for the substitute candidates. These synonyms are then used to substitute the keywords in the problem statement and generate augmented candidates.

\begin{table}
\begin{tabular}{p{56.9pt} p{85.4pt} p{71.1pt}} \hline \hline
**Category** & **Augmentation Method** & **Example** \\ \hline
**Paraphrasing Methods** & **Round trip Translation** & **Original:** The schools debate team had 4 boys and 6 girls on it. If they were split into groups of 2, how many groups could they make ? \\  & & **Augmented:** The school discussion group consisted of 4 boys and 6 girls. If they are divided into groups of 2 . How many groups could they have created ? \\  & **Problem Reordering** & **Original:** Lucy has an aquarium with 5 fish . She wants to buy 1 more fish . How many fish would Lucy have then ? \\  & & **Augmented:** If lucy has an aquarium with 5 fish and she wants to buy 1 more fish then how many fish would lucy have ? \\ \hline
**Substitution Methods** & **Fill Masking** & **Original:** There are 8 walnut trees currently in the park . Park workers will plant 3 more walnut trees today . How many walnut trees will the park have when the workers are finished ? \\  & & **Augmented:** There are 8 walnut trees currently in the park . Park workers will plant 3 more walnut trees soon . How many walnut trees will the park have after the workers are finished ? \\  & **Named-Entity Replacement** & **Original:** Sally found 7 seashells , Tom found 12 seashells , and Jessica found 5 seashells on the beach . How many seashells did they find together ? \\  & & **Augmented:** Edd found 7 seashells , Alan found 12 seashells , and Royal found 5 seashells on the beach . How many seashells were found together ? \\  & **Synonym Replacement** & **Original:** Katie ’s team won their dodgeball game and scored 25 points total . If Katie scored 13 of the points and everyone else scored 4 points each , how may players were on her team ? \\  & & **Augmented:** Katie’s group won their rumble game and scored 25 points total . If Katie scored 13 of the points and all else scored 4 points each, How many players was on her group ? \\ \hline \hline \end{tabular}
\end{table}
Table 3: Augmentation examples from all proposed methods. Coloured text represents the changes in problem statement.

 



**Named-Entity Replacement:** A common occurrence in MWP is the usage of named entities. These entities play a crucial role in stating the problem statement, but the solution equations do not change on altering these entities. Following this insight, we first identify the named entities2 such as person, place and organizations present in the problem statement. Then we replace these named entities with their corresponding substitutes, like a person's name is replaced by another person's name to generate the potential candidates. Table 4 reports the statistics of augmented datasets on both MaWPS and ASDiv-A. All the techniques described in paraphrasing and substitution methods are used for generating the potential candidates for a problem statement. After generation of the potential candidates for augmenting a problem statement, the best possible candidate is selected by using Algorithm 1. Key motivation behind developing this algorithm is to select candidates on which the solver does not perform well and which are similar to the original problem statement.

Footnote 2: [https://www.nltk.org/](https://www.nltk.org/)

We use negative log likelihood as the loss function \({\cal L}\) and Sentence-BERT Reimers and Gurevych (2019) fine tuned on MWP equation generation task as sentence embedding generator \({\cal S}\). We calculate the similarity of each candidate embedding with the original problem representation using cosine similarity as shown in Line \(3\) of the algorithm. Further, for each candidate sentence, we evaluate their loss values and select the candidate with the maximum mean normalized loss and similarity score.

```
0:\({\cal M}\) is augmentation method, \({\cal S}\) is similarity model, \({\cal F}\) is solver model, \({\cal L}\) is Loss function. Input: Problem text \({\cal P}\) Output: Augmented Text \({\cal P^{*}}\)
1:\({\cal E}_{p}\leftarrow{\cal F}({\cal P})\)
2:\(Candidates\leftarrow{\cal M}({\cal P})\)
3:for\(C_{j}\) in \(Candidates:\)do
4:\(S_{j}\leftarrow{\cal S}(C_{j},{\cal P})\)
5:\(L_{j}\leftarrow({\cal L}(C_{j})-{\cal L}(P))/{\cal L}(P)\)
6:\(CandidateScore.add(S_{j}*L_{j})\)
7:\({\cal P^{*}}=\mathop{\arg\max}\limits_{C_{j}}~{}CandidateScore(C_{j})\)
8:end
```

**Algorithm 1**MWP Candidate Selection Algorithm

## 4 Experiments

**Datasets and Models:** To showcase the effectiveness of proposed augmentation methods, we select three state-of-the-art MWP solvers: \((1)\)_Seq2Seq_Wang et al. (2019) having an LSTM encoder and an attention based decoder. \((2)\)_GTS_Xie and Sun (2019) having an LSTM encoder and a tree based decoder and \((3)\)_Graph2tree_Zhang et al. (2019) consists of a both tree based encoder and decoder. _Seq2Seq_ serves as our base model for experimentation. Many existing datasets are not suitable for our analysis as either they are in Chinese Wang et al. (2019) or they have problems of higher complexities Huang et al. (2016) . We conduct experiments across the two largest available English language datasets satisfying our requirements: \((1)\)_MaWPS_Koncel-Kedziorski et al. (2016) containing \(2,373\) problems \((2)\)_ASDiv-A_Miao et al. (2020) containing \(1,213\) problems. Both datasets have MWPs with linear equation in one variable.

**Experiment Setup:** We train and evaluate the three solvers on both MaWPS and ASDiv-A using five fold cross validation. Evaluation is conducted on both original and augmented datasets. We use the same hyperparameter values as recommended in the original implementation of these solvers. Further, each solver has been trained from scratch and by using BERT embeddings Devlin et al. (2019). We also evaluate the models on _SVAMP_Patel et al. (2021) challenge set. This test set has been designed specifically to examine the robustness and adaptability of the solvers. Ablation studies have been conducted to assess the effectiveness of candidate selection algorithm and augmentation techniques.

\begin{table}
\begin{tabular}{|c|c|c|} \hline
**Dataset** & **Problem Size** & **Vocabulary Size** \\ \hline MaWPS & 2,373 & 2,632 \\ ASDiv-A & 1,213 & 2,893 \\ Paraphrase & 5,909 & 3,832 \\ Substitution & 6,647 & 3,923 \\ Combined-MaWPS & **10,634** & **5,626** \\ Combined-ASDiv & **5,312** & **6,109** \\ \hline \end{tabular}
\end{table}
Table 4: Statistics of augmented dataset compared with MaWPS and ASDiv-A. Combined-Dataset represents combination of Paraphrase and Substitution methods.





### Results and Analysis

Table 5 shows the result of proposed methods. These results have been reported on BERT embeddings. Table 11 shows a comparison between training from scratch and using BERT embeddings. By training these state-of-the-art models on the augmented dataset we achieve better results for both MaWPS and ASDiv-A. On average, we were able to increase the accuracy significantly by more than five percentage points. Both paraphrasing and substitution methods have performed well independently and in combination. Further, we conduct ablation studies to analyze the performance of each augmentation method. In Table 6 we illustrate some examples on which existing models generate incorrect equations. However, after being trained with augmented dataset they generate correct equations. Additionally, in Problem 2 the base model generates syntactically incorrect solution, but post augmentation it generates syntactically correct equation. These examples show the increased robustness and solving abilities of solvers.

**Attention Visualizations:** Through this investigation, we aim to ascertain our hypothesis that to generate equations MWP solvers focus only on certain keywords and patterns in a region. They ignore essential information like semantics, sequence of events and content of the question text present in the problem statement. In Table 7, we show some sample problem statements with their attention weights. These weights are generated during the decoding process using Luong attention mechanism [3]. Moreover, to illustrate the effectiveness of our augmentation techniques, we show the distribution of attention weights for models trained on the augmented dataset. We can infer from the examples showcased in Table 7 that before augmentation the focus of the solver is limited to a fixed region around numerical quantities and it does not pay heed to the question text. However, after training on the augmented dataset the solver has a better distribution of attention weights, the weights are not localised and and the model is also able to pay attention on the question text.

**Ablation Studies:** To assert the effectiveness of our methods, we conduct the following ablations:

\begin{table}
\begin{tabular}{l|c



#### Candidate Selection Algorithm

For testing the usefulness of candidate selection algorithm, we compare it with a random selection algorithm. In this, we randomly select one of the possible candidates as augmented problem statement. We evaluate the accuracy of models trained on the augmented datasets, generated using both the algorithms. Result in Table 8, shows that candidate selection algorithm performs better than random selection algorithm and this demonstrates the effectiveness of our algorithm.

#### Augmentation Methods

To examine the effectiveness of proposed augmentation techniques, we evaluate the models on each of the proposed techniques independently and report the results in Table 9. Although, all the methods contribute towards increase in accuracy but Round trip translations and synonym replacement perform marginally better than others. This behaviour can be linked to the structural diversity and keyword sensitivity that round trip translations and synonym replacement bring respectively Feng et al. (2021).

### SEMP Challenge Set:

SVAMP Patel et al. (2021) is a manually curated challenge test set consisting of \(1,000\) math word problems. These problems have been cherry picked from MaWPS and ASDiv-A, then altered manually to modify the semantics of question text and generate additional equation templates. This challenge set is suitable for evaluating a solver's performance as it modifies problem statements such that solver's generalization can be checked. The results are shown in Table 10. Although, our proposed augmented dataset has very limited equation templates, still it performs comparatively better than state-of-the-art models on SVAMP challenge set. This result signifies the need for a larger and diverse dataset with enhanced variety of problems. Further, it demonstrates the effectiveness of our method which is able to perform better on SVAMP test set and increase model's accuracy despite the challenges.

**BERT Embeddings:** We train the solvers in two different settings, using pre-trained BERT embeddings and training from scratch. We chose BERT specifically as we require contextual embeddings which could be easily adapted for the task of MWP. Moreover, existing models have also shown results using BERT and it would be fair to compare their performances when trained using similar embeddings. Results obtained are shown in Table 11. We observe that for solver's trained using BERT, accuracy is higher than models trained from scratch.

\begin{table}
\begin{tabular}{|c|c|c|c|} \hline
**Augmentation** & **Seq2Seq** & **GTS** & **Graph2Tree** \\ \hline True & 37.5 & 39.6 & 41.2 \\ MaWPS(P+S) & 39.2 & 40.1 & 42.3 \\ ASDiv-A(P+S) & 37.8 & 40.4 & 42.1 \\ Combined & **40.2** & **41.3** & **43.8** \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 10: Result of augmentations on SVAMP Challenge Set. P and S represent paraphrasing and substitution methods. Combined represents augmented MaWPS and ASDiv-A. True is combined MaWPS and ASDiv-A.

\begin{table}
\begin{tabular}{l l c|c|c} \hline
**Augmentation** & **Seq2Seq** & **GTS** & **Graph2Tree** \\ \hline True & 84.6 & 87.5 & 88.7 \\ RRT & **86.5** & 89.1 & **91.6** \\ PR & 85.9 & 88.4 & 90.7 \\ FM & 84.8 & 87.2 & 89.1 \\ SR & 85.2 & **90.1** & 91.2 \\ NER & 86.1 & 88.3 & 89.7 \\ \hline  & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} & \multicolumn{1}{c}{} \\ \end{tabular}
\end{table}
Table 9: Result of Ablation study for each augmentation method. True represents unaugmented MaWPS dataset, RRT, PR, FM, SR, NER represents round trip translations, problem reordering, fill masking,synonym replacement and named entity replacement respectively.

\begin{table}
\begin{tabular}{|c|c|c|c|c|} \hline
**Method** & **Eval Type** & **Seq2Seq** & **GTS** & **Graph2Tree** \\ \hline \multirow{4}{*}{**RSA**} & True & 84.6 & 87.5 & 88.7 \\  & Paraphrasing & **85.3** & **88.1** & **89.2** \\ \cline{2-5}  & Substitution & **86.8** & **87.3** & **87.9** \\ \cline{2-5}  & Combined & **87.0** & **89.2** & **89.5** \\ \hline  & True & 84.6 & 87.5 & 88.7 \\ \cline{2-5}
**CSA** & Paraphrasing & **88.3** & **90.4** & **92.6** \\ \cline{2-5}  & Substitution & **89.2** & **89.7** & **91.7** \\ \cline{2-5}  & Combined & **91.3** & **92.6** & **93.5** \\ \hline \end{tabular}
\end{table}
Table 8: Ablation Study for Random Selection Algorithm (RSA) and Candidate Selection Algorithm (CSA).



Human Evaluation:To verify the quality of augmented examples, we conduct human evaluation. The focus of this evaluation is: \((1)\) To check if the augmentations will result in the same linear equation as present in the original problem statement, \((2)\) To evaluate if the numerical values for each augmentation example is preserved, \((3)\) Evaluate each sample in the range \(0\) to \(1\) for its semantic similarity with the original problem statement, \((4)\) On a scale of \(1\) to \(5\) rate each augmented example for its grammatical correctness. We conduct the human evaluations on randomly shuffled subsets consisting of around \(40\%\) of the total augmented examples for both the datasets. This process is repeated three times with different subsets, five human evaluators evaluate each example in all subsets, and the mean results are computed as shown in Table 12.

## 5 Future Work and Conclusion

We showcase that the existing MWP solvers are not robust and do not generalize well on even simple variations of the problem statement. In this work, we have introduced data augmentation techniques for generation of diverse math word problems. We were able to enhance the size of existing dataset by 5 folds and significantly increase the performance of state-of-the-art solvers by over 5 percentage points. Future works could focus on developing techniques to generate data artificially and making robust MWP solvers.

## 6 Acknowledgment

We thank IHub-Data, IIIT Hyderabad3 for financial support.

Footnote 3: [https://hub-data.iiit.ac.in/](https://hub-data.iiit.ac.in/)

## References

* Alzantot et al. [2018] Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples.
* Amini et al. [2019] Aida Amini, Saadia Gabriel, Peter Lin, Riko Koncelkoziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. Mathqa: Towards interpretable math word problem solving with operation-based formalisms.
* Belinkov and Bisk [2018] Yongatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation.
* Chiang and Chen [2018] Ting-Rui Chiang and Yun-Nung Chen. 2018. Semantically-aligned equation generation for solving and reasoning math word problems. _arXiv preprint arXiv:1811.00720_.
* Devlin et al. [2019a] J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019a. Bert: Pre-training of deep bidirectional transformers for language understanding. In _NAACL-HLT_.
* Devlin et al. [2019b] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019b. Bert: Pre-training of deep bidirectional transformers for language understanding.
* Feng et al. [2021] Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021. A survey of data augmentation approaches for nlp.
* Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Computation_, 9(8):1735-1780.
* Hosseini et al. [2014] Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. pages 523-533, Doha, Qatar. Association for Computational Linguistics.
* Huang et al. [2016] Danqing Huang, Shuming Shi, Chin-Yew Lin, Jian Yin, and Wei-Yang Ma. 2016. How well do computers solve math word problems? large-scale dataset construction and evaluation. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 887-896, Berlin, Germany. Association for Computational Linguistics.
* Kim et al. [2019] Yunsu Kim, Petre Petrov, Pavel Petrushkov, Shahram Khadivi, and Hermann Ney. 2019. Pivot-based transfer learning for neural machine translation between non-English languages. pages 866-876, Hong Kong, China. Association for Computational Linguistics.
* Kobayashi [2018] Souske Kobayashi. 2018. Contextual augmentation: Data augmentation by words with paradigmatic relations.
* Koncel-Kedziorski et al. [2016] Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. MAWPS: A math word problem repository. In _Proceedings of the 2016 Conference of the North American Chapter

\begin{table}
\begin{tabular}{|c|c|c|c|c|} \hline
**Evaluation** & \multicolumn{2}{c|}{**MaWPS**} & \multicolumn{2}{c|}{**ASDiv-A**} \\ \cline{2-5}
**Criteria** & Para & Sib & Para & Sub \\ \hline Preserves Equation & 92.3\% & 89.5\% & 93.6\% & 90.1\% \\ Preserves Numbers & 88.4\% & 91.2\% & 87.3\% & 90.3\% \\ Semantic Similarity & 0.96 & 0.89 & 0.91 & 0.87 \\ Syntactic Similarity & 4.67 & 4.36 & 4.59 & 4.33 \\ \hline \end{tabular}
\end{table}
Table 12: Human Evaluation scores on augmented dataset. Para and Sub represents paraphrasing and substitution methods respectively.

 

of the Association for Computational Linguistics: Human Language Technologies_, pages 1152-1157, San Diego, California. Association for Computational Linguistics.
* Kumar et al. (2019) Ashutosh Kumar, Satwik Bhattacharya, Manik Bhandari, and Partha Talukdar. 2019. Submodular optimization-based diverse paraphrasing and its effectiveness in data augmentation. pages 3609-3619, Minneapolis, Minnesota. Association for Computational Linguistics.
* Kumar et al. (2021) Vivek Kumar, Rishabh Maheshwary, and Vikram Pudi. 2021. Adversarial examples for evaluating math word problem solvers. In _Findings of the Association for Computational Linguistics: EMNLP 2021_, pages 2705-2712, Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Kushman et al. (2014) Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. 2014. Learning to automatically solve algebra word problems. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 271-281, Baltimore, Maryland. Association for Computational Linguistics.
* Longpre et al. (2020) Shayne Longpre, Yu Wang, and Christopher DuBois. 2020. How effective is task-agnostic data augmentation for pretrained transformers?
* Luong et al. (2015) Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation.
* Maheshwary et al. (2021a) Rishabbha Maheshwary, Saket Maheshwary, and Vikram Pudi. 2021a. Generating natural language attacks in a hard label black box setting. In _AAAI_.
* Maheshwary et al. (2021b) Rishabbha Maheshwary, Saket Maheshwary, and Vikram Pudi. 2021b. A strong baseline for query efficient attacks in a black box setting.
* Miao et al. (2020) Shen-yun Miao, Chao-Chun Liang, and Keh-Yih Su. 2020. A diverse corpus for evaluating and developing English math word problem solvers. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 975-984, Online. Association for Computational Linguistics.
* Patel et al. (2021) Arkil Patel, Satwik Bhattacharshra, and Navin Goyal. 2021. Are nlp models really able to solve simple math word problems?
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _Empirical Methods in Natural Language Processing (EMNLP)_, pages 1532-1543.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks.
* Rizos et al. (2019) Georgios Rizos, K. Hemker, and Bjorn Schuller. 2019. Augment to prevent: Short-text data augmentation in deep learning for hate-speech classification. _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_.
* Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data.
* Tan et al. (2019) Hao Tan, Licheng Yu, and Mohit Bansal. 2019. Learning to navigate unseen environments: Back translation with environmental dropout.
* Taylor and Nitschke (2018) Luke Taylor and Geoff Nitschke. 2018. Improving deep learning with generic data augmentation. In _2018 IEEE Symposium Series on Computational Intelligence (SSCI)_, pages 1542-1547.
* Voita et al. (2019) Elena Voita, Rico Sennrich, and Ivan Titov. 2019. Context-aware monolingual repair for neural machine translation.
* Wang et al. (2019) Lei Wang, D. Zhang, Jipeng Zhang, Xing Xu, L. Gao, B. Dai, and H. Shen. 2019. Template-based math word problem solvers with recursive neural networks. In _AAAI_.
* Wang et al. (2019) Yan Wang, Xiaojiang Liu, and Shuming Shi. Deep neural solver for math word problems.
* Wei and Zou (2021) Jason Wei and Kai Zou. EDA: Easy data augmentation techniques for boosting performance on text classification tasks.
* Witteveen and Andrews (2020) Sam Wittteveen and Martin Andrews. Paraphrasing with large language models.
* Wu et al. (2016) Yonghui Wu, Mike Schuster, Z. Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikum, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Kilinger, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason R. Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Gregory S. Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. _ArXiv_, abs/1609.08144.
* Xie and Sun (2019) Zhipeng Xie and Shichao Sun. 2019. A goal-driven tree-structured neural model for math word problems. In _IJCAI_, pages 5299-5305.
* Zhang et al. (2016) Jipeng Zhang, Lei Wang, Roy Ka-Wei Lee, Yi Bin, Yan Wang, Jie Shao, and EE-Peng Lim. Graph-to-tree learning for solving math word problems.
* Zhang et al. (2016) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2016. Character-level convolutional networks for text classification 

Appendix

### Implementation Details

For conducting our experiments we have used two Boston SYS-7048GR-TR nodes equipped with NVIDIA GeForce GTX 1080 Ti computational GPU's having 11GB of GDDR5X RAM. All implementations of training and testing is coded in Python with Pytorch framework. The number of parameters range from 20M to 130M for different models. We use negative log likelihood as the loss criterion. Hyper-parameter values were not modified, and we follow the recommendations of the respective models. To reduce carbon footprint from our experiments, we run the models only on a single fold for searching hyperparameter values. We chose the number of base candidates after primary stage \(n\) as 7. Generating augmentation examples using Paraphrasing Methods took around 12 minutes on average for MaWPS and 8 minutes for ASDiv-A datasets. Substitution methods took around 5 minutes on average for both MaWPS and ASDiv-A dataset. The experiments conducted by us are not computation heavy. Each of the state-of-the-art models get trained within 5 hrs of time, with Graph2Tree taking the maximum time.

### Additional Augmented Examples

In this section, we present some additional valid as well as invalid augmented examples. Additionally, we also show some more examples with their attention weights. Table 13 shows some additional examples with their attention weight distribution. These weights have been shown for the base model trained before augmentation and after augmentation on MaWPS dataset. Table 14 illustrates some additional problem statements for all the techniques in paraphrasing methods and substitution methods. In Table 15, we present some invalid augmented examples which do not satisfy our human evaluation criteria. These examples are such that they alter the semantics of the original problem statement.

[TABLE:

\begin{table}
\begin{tabular}{l|c|c|c} Problem: A magician wasi & selling magic card & decks & for 2 \\ dollars each. If he started with 25 decks and by the end of the day he had 4 left, how much money did he earn ? \\  & & & \\ \hline Problem: There are 18 pencils in the drawer and 6 pencils & on the desk. How many \\  & mean attention values: 0.21 & 0.16 & 0.06 \\  & & & \\ \end{tabular}
\end{table}
Table 13: Examples illustrating distribution of top three attention weights before and after training on the full augmented dataset.

 



\begin{table}
\begin{tabular}{l l p{142.3pt}}
**Category** & **Augmentation Method** & **Example** \\ \hline
**Paraphrasing Methods** & **Round trip Translation** & **Original:** Alyssa’s dog had puppies. She gave 2 to her friends. She now has 3 puppies. How many puppies did she have to start with ’? \\  & & **Augmented:** Alyssa’s dog had puppies . She gave her friends 2 . She now has 3 puppies . How many puppies did she start ? \\  & **Problem Reordering** & **Original:** Rachel was organizing her book case making sure each of the shelves had exactly 3 books on it. If she had 4 shelves of mystery books and 2 shelves of picture books , how many books did she have total ’? \\  & & **Augmented:**How many books did she have given that rachel was organizing her book case making sure each of the shelves had exactly 3 books on it and she had 4 shelves of mystery books and 2 shelves of picture books . \\ \hline
**Substitution Methods** & **Fill Masking** & **Original:** A cell phone company has a total of 1000 customers across the world . If 740 of its customers live in the United States , how many of its customers live in other countries ’? \\  & **Augmented:** A mobile phone firm has a network of 1000 customers across the world . If 740 of its customers live in the violetUS, How many customers live in other locations? \\  & **Named-Entity Replacement** & **Original:** Daniel had some noodles. He gave 20 noodles to William. Now Daniel only has 11 noodles. How many noodles did Daniel have to begin with ’ \\  & **Augmented:** Matt had some noodles. He gave 20 noodles to Zeal. Now Matt only has 11 noodles. How many noodles did Matt have initially ? Edd found 7 seashells , Alan found 12 seashells , and Royal found 5 seashells on the beach . How many seashells were found together ’ \\ \hline
**Synonym Replacement** & **Original:** There are 5 rulers in the drawer. Tim took 3 nulers from the drawer. How many rulers are now in the drawer ? \\  & **Augmented:** There are 5 consonants in the drawer. Tim went 3 consonants from the drawer. How many other consonants are in the drawer now ’ \\ \end{tabular}
\end{table}
Table 14: Valid Augmentation examples from all proposed methods. Coloured text represents the changes in problem statement.





\begin{table}
\begin{tabular}{p{85.4pt} p{170.7pt} p{269.3pt}}
**Category** & **Augmentation Method** & **Example** \\ \hline
**Paraphrasing Methods** & **Round trip Translation** & **Original:** Kimberly went to the store 6 times last month . She buys 9 peanuts each time she goes to the store . How many peanuts did Kimberly buy last month ? \\  & & **Augmented:** Kimberly travelled to club six times last month. She buys 9 peanuts every time she goes to the club . How many peanuts did Kimberly buy last year ? \\ \cline{2-3}  & **Problem Reordering** & **Original:** Fred has 10 blue marbles . Fred has number1 times more blue marbles than Tim . How many blue marbles does Tim have ? \\  & & **Augmented:** If red has 10 blue marbles and fred has number1 more blue marbles than Tim then how many blue marbles does tim have ? \\ \hline
**Substitution Methods** & **Fill Masking** & **Original:** Sarah had 7 homework problems . She finished 2 of them but still had 3 pages of problems to do . If each page has the same number of problems on it , how many problems are on each page ? \\  & & **Augmented:** Sarah had 7 of them . She had 2 of them but still had 3 more of them to do . If each more has the same number of them on it, How many them are on each more ? \\ \cline{2-3}  & **Named-Entity Replacement** & **Original:** Beverly had 10 dimes in his bank . His sister Maria borrowed 2 of his dims . How many dimes does Beverly have now ? \\  & & **Augmented:** Silva had 10 dimes in his bank . His sister Jeanie borrowed a pair of his dimes . How many dimes does Jeanic have now ? \\ \cline{2-3}  & **Synonym Replacement** & **Original:** Shawn’s team won their dodgeball game and scored 25 points total . If Shawn scored 13 of the points and everyone else scored 4 points each , how many players were on his team ? \\  & & **Augmented:**Shawn’s group won their rumble game and scored 25 points total . If Shawn scored 13 of the points and everyone else scored quarter points each, how many people were there ? \\ \end{tabular}
\end{table}
Table 15: Invalid Augmentation examples from all proposed methods. Coloured text represents the changes in problem statement.