SlowGuess/NAACL_2022_OCR huggingface dataset:
Vivek Kumar, Rishabh Maheshwary and Vikram Pudi
Data Sciences and Analytics Center, Kohli Center on Intelligent Systems
International Institute of Information Technology, Hyderabad, India
{vivek.k, rishabh.maheshwary}  $@$  research.iit.ac.in, vikram@iit.ac.in
Existing Math Word Problem (MWP) solvers have achieved high accuracy on benchmark datasets. However, prior works have shown that such solvers do not generalize well and rely on superficial cues to achieve high performance. In this paper, we first conduct experiments to showcase that this behaviour is mainly associated with the limited size and diversity present in existing MWP datasets. Next, we propose several data augmentation techniques broadly categorized into Substitution and Paraphrasing based methods. By deploying these methods we increase the size of existing datasets by five folds. Extensive experiments on two benchmark datasets across three state-of-the-art MWP solvers shows that proposed methods increase the generalization and robustness of existing solvers. On average, proposed methods significantly increase the state-of-the-art results by over five percentage points on benchmark datasets. Further, the solvers trained on the augmented dataset performs comparatively better on the challenge test set. We also show the effectiveness of proposed techniques through ablation studies and verify the quality of augmented samples through human evaluation.
A Math Word Problem (MWP) consists of natural language text which describes a world state involving some known and unknown quantities, followed by a question text to determine the unknown values. The task is to parse the problem statement and generate equations that can help find the value of unknown quantities. An example of a simple MWP is shown in Table 1. In recent years, the challenge of solving MWP has gained much attention in the NLP community as it needs the development of commonsense multi step reasoning with numerical quantities. With the rise of deep learning, performance of math solvers has also increased significantly over the years (Wang
Original Problem
Problem: Nancy grew 8 potatoes. Sandy grew 5 potatoes. How many potatoes did they grow in total?
True Equation:  $\mathrm{X} = 8 + 5$
Paraphrasing Method
Problem: How many potatoes did they grow in all given that nancy grew 8 potatoes and sandy grew 5 potatoes.
Equation Label:  $\mathrm{X} = 8 + 5$
Substitution Method
Problem: Dwight grew 8 potatoes. Juliette grew 5 potatoes. How many potatoes did they grow together?
Table 1: A MWP and its augmentation examples generated by our methods with preserved equation labels. Blue and Violet colours denote the changes made after the primary stage and secondary stage respectively.
et al.; Zhang et al.). However, recent analysis conducted in (Kumar et al., 2021) and (Patel et al., 2021) show that these deep learning based solvers rely on shallow heuristics to solve vast majority of problems. They curated adversarial examples and SVAMP challenge set respectively to infer that MWP solvers (1) do not understand the relationship between numbers and their associated entities, (2) do not focus on the question text and (3) ignore word order information. In this paper, we first conduct experiments to establish that the above drawbacks are due to the limited size and diversity of problems present in the existing MWP datasets. Next, we propose various augmentation methods to create diverse and large number of training examples to mitigate these shortcomings. Our methods are focused on: (1) Increasing the number of problems in the existing datasets and (2) enhancing the diversity of the problem set.
Training deep neural models effectively requires large number of data points (Longpre et al., 2020). Constructing large datasets which are annotated, labeled and have MwPs of similar difficulty level is a very expensive and tedious task. To address these key challenges, we resort to data augmenta
Human Language Technologies, pages 4194 - 4206
tion techniques. Our motivation behind generating augmentations is that humans require sufficient practice to understand MwPs. Humans learn to solve MwPs by going through a variety of similar examples and slowly become capable enough to tackle variations of similar difficulty levels. We aim to generate augmentations such that sufficient linguistic variations of a similar problem are present in the dataset. These variations will make the solver more robust in tackling MWP, increase their reasoning ability and numerical understanding.
Data augmentation for MwPs is a challenging task as we need to preserve the equation labels while generating new samples (Kumar et al., 2021). The generated samples should be (1) semantically similar to their original counterpart, (2) must have the same numerical values and preserve relationship with their respective entities and (3) should maintain the same sequence of events in the problem text. Existing augmentation methods (Wei and Zou) cannot be directly applied due to the above mentioned reasons. Our methods can be broadly classified as follows:
- Paraphrasing Methods: It generates variations of the question text by re-statement such that the semantic and syntactic meaning along with the equation labels is preserved.
- Substitution Methods: These methods generate variations of the problem statement by identifying and substituting some of the keywords such that the augmentations are semantically and syntactically correct.
To ensure high quality augmentations<sup>1</sup>, we propose a selection algorithm which selects samples that have high similarity with original problem and incur high loss values when tested on existing solvers. This algorithm helps selecting only those samples that can make existing solvers more robust. Further, we also verify the validity and the quality of generated augmentations through human evaluation.
Most of the existing MWP datasets are either in languages other than English or contain problems of varying difficulty levels (Koncel-Kedziorski et al., 2016; Wang et al.; Huang et al., 2016; Amini et al., 2019; Miao et al., 2020). We focus on strengthening existing English language datasets which can facilitate the development of
better MWP solvers. We consider datasets containing MWP that can be solved using linear equations in one variable. These datasets include MaWPS (Koncel-Kedziorski et al., 2016) and ASDiv-A (Miao et al., 2020) both having 2,373 and 1,213 problems respectively. Following are the key contributions made in this paper:
- To the best of our knowledge, this is the first work that extensively evaluates data augmentation techniques for MWP solving. This is the first attempt to generate MWP problems automatically without manual intervention.
- Accuracy of the state of the art solvers increases after training on the proposed augmented dataset. This demonstrates the effectiveness of our methods. To verify the validity of generated augmentations we conduct human evaluation studies.
- We increase the diversity of the training dataset through augmentations and obtain comparatively better results than state-of-the-art solvers on the SVAMP challenge set.
Math Word Solvers: Many research efforts have been undertaken in the recent past to solve the challenging MWP task. Broadly, these solvers can be categorized into statistical learning based and deep learning based models. Traditional approaches focused more on statistical machine learning (Kushman et al., 2014; Hosseini et al., 2014) with the aim of categorizing equations into templates and extracting key patterns in the problem text. Recently, due to the advent of deep learning in NLP, solvers have witnessed a considerable increase in their performances. (Wang et al.) modelled MWP task as a sequence to sequence task and used LSTM's (Hochreiter and Schmidhuber, 1997) for learning problem representations. (Chiang and Chen, 2018) focused on learning representations for operators and operands. (Wang et al., 2019; Xie and Sun, 2019) used tree structures for decoding process. (Zhang et al.) modelled question as a graph to map quantities and their attributes. Existing datasets which have been used as benchmark for english language includes MaWPS (Koncel-Kedziorski et al., 2016) and Chinese language dataset Math23K (Wang et al.). These datasets although constrained by
$^{1}$ Codebase and augmented datasets are available at: https://github.com/kevivk/MWP-Augmentation
their size deal with algebraic problems of similar difficulty levels. Recently, ASDiv (Miao et al., 2020) has been proposed, which has diverse problems which includes annotations for equations, problem type and grade level. Other large datasets in English language include MathQA (Amini et al., 2019) and Dolphin18k (Huang et al., 2016). Although, these datasets have larger problem set but they are noisy and contain problems of varied difficulty levels.
Text Data Augmentation: To effectively train deep learning models, large datasets are required. Data augmentation is a machine learning technique that artificially enlarges the amount of training data by means of label preserving transformations (Taylor and Nitschke, 2018). (Longpre et al., 2020) hypothesize that textual data augmentation would only be helpful if the generated data contains new linguistic patterns that are relevant to the task and have not been seen in pre-training. In NLP, many techniques have been used for generating augmentations, (Wei and Zou) introduced noise injection, deletion, insertion and swapping of words in text. (Rizos et al., 2019) used recurrent neural networks and generative adversarial networks for short-text augmentation (Maheshwary et al., 2021b). Recently, hard label adversarial attack models have also been used (Maheshwary et al., 2021a). Other frequently used methods include inducing spelling mistakes (Belinkov and Bisk, 2018), synonym replacement (Zhang et al., 2016), identifying close embeddings from a defined search space (Alzantot et al., 2018), round trip translations (Sennrich et al., 2016), paraphrasing techniques (Kumar et al., 2019) and words predicted by language model (Kobayashi, 2018) among many others. These methods are specific to the task at hand and needs to be adapted such that the generated augmentations bring diversity in the concerned dataset.
3 Proposed Augmentation Approach
Data augmentation generates new data by modifying existing data points through transformations based on prior knowledge about the problem domain. We introduce carefully selected transformations on well known text augmentation techniques to develop examples suited for the task of MWP. These transformations help in increasing the diversity and size of problem set in existing datasets.
A MWP is defined as an input of  $n$  tokens,  $\mathcal{P} = \{w_1, w_2..w_n\}$  where each token  $w_i$  is either a numeric value or a word from a natural language. The goal is to generate a valid mathematical equation  $\mathcal{E}_{\mathcal{P}}$  from  $\mathcal{P}$  such that the equation consists of numbers from  $\mathcal{P}$ , desired numerical constants and mathematical operators from the set  $\{/, *, +, -, =, (), \}$ . Let  $\mathcal{F}: \mathcal{P} \to \mathcal{E}_{\mathcal{P}}$  be an MWP solver where  $\mathcal{E}_{\mathcal{P}}$  is the equation to problem  $\mathcal{P}$ . Our task is to generate augmented problem statement  $\mathcal{P}^*$  from the original input  $\mathcal{P}$  such that  $\mathcal{P}^*$  is: (1) semantically similar to the initial input  $\mathcal{P}$ , (2) preserves the sequence of events in the problem statement, (3) keeps the numerical values intact and (4) the solution equation is same as  $\mathcal{E}_{\mathcal{P}}$ .
3.2 Deficiencies in Existing Models
As showcased by (Patel et al., 2021), existing MWP solvers trained on benchmark datasets like MaWPS and ASDiv-A focus their attention only on certain keywords in the problem statement and do not pay much heed to the question text. We further show that even after performing significant transformations on the test set such as (1) dropping the question text, (2) randomly shuffling the sequence of sentences, (3) random word deletion, and (4) random word reordering, the solvers are still able to produce correct equations. Upon introducing these transformations we should expect a very high drop in accuracy values as the transformed problems are now distorted. Surprisingly, the decrease in accuracy scores is relatively very less than expected as shown in Table 2. We only observe a relatively moderate drop for word reordering. From this analysis, we can say that instead of focusing on the sequence of events, question text and semantic representation of the problem, solvers pick word patterns and keywords from the problem statement. We hypothesize that the drop in accuracy for word reordering experiment indicates that the solvers try to identify a contiguous window of words having some keywords and numbers in them, and generates equation based on these keywords. We further probe on this hypothesis by visualizing the attention weights in the experiment section.
3.3 Augmentation Methods
A MWP can also be expressed as  $\mathcal{P} = (S_1, S_2..S_k, Q)$  where  $Q$  is the question and  $(S_1, S_2..S_k)$  are the sentences constituting the prob-
Dataset	Eval Type	Seq2Seq	GTS	Graph2Tree
MaWPS	True	84.6	87.5	88.7
WD	80.2	81.5	77.3
QR	77.4	82.0	80.2
SS	77.0	60.4	66.4
WR	54.9	34.8	39.3
ASDiv-A	True	70.6	80.3	82.7
WD	60.2	61.3	56.7
QR	58.7	52.4	54.1
SS	56.2	59.3	60.7
WR	47.1	32.3	34.6
Table 2: Performance of solvers on modified test sets. True represents unaugmented test set. WD,QR,SS,WR represent word deletion, question reordering, sentence shuffling and word reordering respectively.
lem description. To mitigate the deficiencies in MWP solvers, we propose a two stage augmentation paradigm consisting of primary and secondary stage. In primary stage, we generate base augmentation candidates which then proceed to the secondary stage and get modified accordingly to become potential candidates. After identifying the potential candidates, we filter out the best candidates using proposed candidate selection algorithm. Table 1 shows changes in MWP after primary and secondary stage. Following are the details:
- Primary Stage: In the primary stage, our focus is on inducing variations in the question text  $Q$  of a given problem statement  $\mathcal{P}$ . For this, we first generate  $n$  base candidates  $\{b_1, b_2, \dots, b_n\}$  from  $Q$  using T5 paraphrasing model (Raffel et al., 2020). The key intuition behind this step is to ensure that each augmentation of a given problem has a different question text. This will empower the solver to learn variations from the question text as well.
- Secondary Stage: After the generation of base candidates, we implement augmentation methods to generate potential candidates. These methods although well known, require careful tuning to adapt for MWP generation. Table 3 showcases MWP examples and their generated augmentations. Detailed description of these techniques follow.
3.3.1 Paraphrasing Methods
Paraphrasing has proved to be an effective way of generating text augmentations (Witteveen and Andrews). It generates samples having diverse sentence structures and word choices while preserving the semantic meaning of the text. These additional samples guide the model to pay attention to not only the keywords but its surroundings as well. This is particularly beneficial for the task of MWP solving, where most of the problem statements follow a general structure.
Problem Reordering: Given original problem statement  $\mathcal{P} = (S_1, S_2, \ldots, S_k, Q)$ , we alter the order of problem statement such that  $\mathcal{P}^* = (Q, S_1, S_2, \ldots, S_k)$ . To preserve the semantic and syntactic meaning of problem statement we use filler phrases like 'Given that' and 'If-then'. To make these paraphrases more fluent, we use named entity recognition and co-reference resolution to replace the occurrences of pronouns with their corresponding references. Please note that this method is better than random shuffling of sentences as it preserves the sequence of events in the problem statement.
Round Trip Translations: Round trip translations, more commonly referred as back-translation is an interesting method to generate paraphrases. This idea has evolved as a result of the success of machine translation models (Wu et al., 2016). In this technique, sentences are translated from their original language to foreign languages and then translated back to the original language. This round trip can be between multiple languages as well. The motivation behind using this technique is to utilize the different structural constructs and linguistic variations present in other languages.
Back-translation is known to diverge uncontrollably (Tan et al., 2019) for multiple round trips. This may lead to change in the semantics of the problem statement. Numerical quantities are fragile to translations and their order and representation may change. To overcome these challenges, we worked with languages that have structural constructs similar with English. For instance, languages like Finnish which are gender neutral, can become problematic as they can lead to semantic variance in augmented examples. To preserve numerical quantities, we replace them with special symbols and keep a map to restore numerical quan
Category	Augmentation Method	Example
Paraphrasing Methods	Round trip Translation	Original: The schools debate team had 4 boys and 6 girls on it. If they were split into groups of 2, how many groups could they make?
Augmented: The school discussion group consisted of 4 boys and 6 girls. If they are divided into groups of 2. How many groups could they have created?
Problem Reordering	Original: Lucy has an aquarium with 5 fish. She wants to buy 1 more fish. How many fish would Lucy have then?
Augmented: If lucy has an aquarium with 5 fish and she wants to buy 1 more fish then how many fish would lucy have?
Substitution Methods	Fill Masking	Original: There are 8 walnut trees currently in the park. Park workers will plant 3 more walnut trees today. How many walnut trees will the park have when the workers are finished?
Augmented: There are 8 walnut trees currently in the park. Park workers will plant 3 more walnut trees soon. How many walnut trees will the park have after the workers are finished?
Named-Entity Replacement	Original: Sally found 7 seashells, Tom found 12 seashells, and Jessica found 5 seashells on the beach. How many seashells did they find together?
Augmented: Edd found 7 seashells, Alan found 12 seashells, and Royal found 5 seashells on the beach. How many seashells were found together?
Synonym Replacement	Original: Katie's team won their dodgeball game and scored 25 points total. If Katie scored 13 of the points and everyone else scored 4 points each, how many players were on her team?
Augmented: Katie's group won their rumble game and scored 25 points total. If Katie scored 13 of the points and all else scored 4 points each, How many players was on her group?
Table 3: Augmentation examples from all proposed methods. Coloured text represents the changes in problem statement.
tities in the generated paraphrases. We have used the following round trips:
English - Russian - English: Although Russian is linguistically different from English, we still chose it as word order does not affect the syntactic structure of a sentence in Russian language (Voita et al., 2019). For single round trip, we preferred Russian as it has the potential to generate different paraphrase structures.
English - German - French - English: German and french are structurally similar to English language (Kim et al., 2019), we chose them for multiple round trips to both maintain semantic in-variance and induce minor alterations in the paraphrases.
3.3.2 Substitution Methods
In this class of methods, the focus is on generating variations of the problem statement by identifying and substituting some of the keywords such that the augmentations are semantically and syntactically correct, with the equation labels preserved. Substitution is effective for MWP solving as it guides the solvers focus away from certain keywords, allowing it to distribute its attention and generalize better. We propose the
following methods:
Fill-Masking: In this technique, we model the challenge of generating candidates as a masked language modelling problem. Instead of randomly choosing words for masking, we use part of speech tags to focus on nouns and adjectives, preferably in the vicinity of numerical quantities. We replace these identified keywords with mask tokens. These masked candidate sentences are then passed through a masked language model (Devlin et al., 2019a) and suitable words are filled in masked positions to generate our candidate sentences.
Synonym Replacement: In this method, after stop-word removal, we select keywords randomly for substitution. Unlike fill-mask technique, where masked language models were deployed, here we use Glove embeddings (Pennington et al., 2014) to find the top  $k$  candidates that are close synonyms of the keywords. To ensure syntactic correctness in candidates, we maintain the part of speech tags for the substitute candidates. These synonyms are then used to substitute the keywords in the problem statement and generate augmented candidates.
Named-Entity Replacement: A common occurrence in MWP is the usage of named entities. These entities play a crucial role in stating the problem statement, but the solution equations do not change on altering these entities. Following this insight, we first identify the named entities<sup>2</sup> such as person, place and organizations present in the problem statement. Then we replace these named entities with their corresponding substitutes, like a person's name is replaced by another person's name to generate the potential candidates. Table 4 reports the statistics of augmented datasets on both MaWPS and ASDiv-A. All the techniques described in paraphrasing and substitution methods are used for generating the potential candidates for a problem statement. After generation of the potential candidates for augmenting a problem statement, the best possible candidate is selected by using Algorithm 1. Key motivation behind developing this algorithm is to select candidates on which the solver does not perform well and which are similar to the original problem statement.
We use negative log likelihood as the loss function  $\mathcal{L}$  and Sentence-BERT (Reimers and Gurevych, 2019) fine tuned on MWP equation generation task as sentence embedding generator  $S$ . We calculate the similarity of each candidate embedding with the original problem representation using cosine similarity as shown in Line 3 of the algorithm. Further, for each candidate sentence, we evaluate their loss values and select the candidate with the maximum mean normalized loss and similarity score.
Dataset	Problem Size	Vocabulary Size
MaWPS	2,373	2,632
ASDiv-A	1,213	2,893
Paraphrase	5,909	3,832
Substitution	6,647	3,923
Combined-MaWPS	10,634	5,626
Combined-ASDiv	5,312	6,109
Table 4: Statistics of augmented dataset compared with MaWPS and ASDiv-A. Combined-Dataset represents combination of Paraphrase and Substitution methods.
Datasets and Models: To showcase the effectiveness of proposed augmentation methods, we
Algorithm 1 MWP Candidate Selection Algorithm
Requires:  $\mathcal{M}$  is augmentation method,  $S$  is similarity model,  $\mathcal{F}$  is solver model,  $\mathcal{L}$  is Loss function.
Input: Problem text  $\mathcal{P}$
Output: Augmented Text  $\mathcal{P}^*$
1:  $\mathcal{E}_P\gets \mathcal{F}(\mathcal{P})$
2: Candidates  $\leftarrow$ $\mathcal{M}(\mathcal{P})$
3: for  $C_j$  in Candidates : do
4:  $S_{j}\gets \mathcal{S}(C_{j},\mathcal{P})$
5:  $L_{j}\gets (\mathcal{L}(C_{j}) - \mathcal{L}(P)) / \mathcal{L}(P)$
6: CandidateScore.add  $(S_{j}*L_{j})$
7:  $\mathcal{P}^* = \underset {C_j}{\arg \max}$  CandidateScore(Cj)
8: end