SlowGuess/NAACL_2021_OCR huggingface dataset:
The  $[NUM]$  tag is used to replace all the numbers and the  $[NAME]$  tag is used to replace all the Names of Persons in the example. The  $[OBJs]$  and  $[OBJp]$  tags are used for replacing the objects in the example. The  $[OBJs]$  and  $[OBJp]$  tags with the same index represent the same object in singular and plural form respectively. The intention when using the  $[OBJs]$  or the  $[OBJp]$  tag is that it can be used as a placeholder for other similar words, which when entered in that place, make sense as per the context. These tags must not be used for collectives; rather they should be used for the things that the collective represents. Some example uses of  $[OBJs]$  and  $[OBJp]$  tags are provided in Table 18. Lastly, the  $[MOD]$  tag must be used to replace any modifier preceding the  $[OBJs] / [OBJp]$  tag.
A preprocessing script is executed over the Seed Examples to automatically generate template suggestions for the workers. The script uses Named Entity Recognition and Regular Expression matching to automatically mask the names of persons and the numbers found in the Seed Examples. The outputs from the script are called the Script Examples. An illustration is provided in Table 19.
Each worker is provided with the Seed Examples along with their respective Script Examples that have been allotted to them. The worker's task is to edit the Script Example by correcting any mistake made by the preprocessing script and adding any new tags such as the  $[OBJs]$  and the  $[OBJp]$
tags in order to create the Base Example. If a worker introduces a new tag, they need to mark it against its example-specific value. If the tag is used to mask objects, the worker needs to mark both the singular and plural form of the object in a comma-separated manner. Additionally, for each unique index of  $[OBJs] / [OBJp]$  tag in the example, the worker must enter at least one alternate value that can be used in that place. Similarly, the worker must enter at least two modifier words that can be used to precede the principal  $[OBJs] / [OBJp]$  tags in the example. These alternate values are used to gather a lexicon which can be utilised to scale-up the data at a later stage. An illustration of this process is provided in Table 20.
In order to create the variations, the worker needs to check the different types of variations in Table 8 to see if they can be applied to the Base Example. If applicable, the worker needs to create the Variation Example while also making a note of the type of variation. If a particular example is the result of performing multiple types of variations, all types of variations should be listed according to their order of application from latest to earliest in a comma-seperated manner. For any variation, if a worker introduces a new tag, they need to mark it against its example-specific value as mentioned before. The index of any new tag introduced needs to be one more than the highest index already in use for that tag in the Base Example or its previously created variations.
To make the annotation more efficient and streamlined, we provide the following steps to be followed in order:
1. Apply the Question Sensitivity variations on the Base Example.
2. Apply the Invert Operation variation on the Base Example and on all the variations obtained so far.
3. Apply the Add relevant information variation on the Base Example. Then considering these variations as Base Examples, apply the Question Sensitivity variations.
4. Apply the Add irrelevant information variation on the Base Example and on all the variations obtained so far.
5. Apply the Change information variation on the Base Example and on all the variations obtained so far.
Hyperparameters	Seq2Seq	GTS	Graph2Tree	Constrained
Scratch	RoBERTa	Scratch	RoBERTa	Scratch	RoBERTa	Scratch	RoBERTa
Embedding Size	[128, 256]	[768]	[128, 256]	[768]	[128, 256]	[768]	[128, 256]	[768]
Hidden Size	[256, 384]	[256, 384]	[384, 512]	[384, 512]	[256, 384]	[256, 384]	[256, 384]	[256, 384]
Number of Layers	[1, 2]	[1, 2]	[1, 2]	[1, 2]	[1, 2]	[1, 2]	[1, 2]	[1, 2]
Learning Rate	[5e-4, 8e-4, 1e-3]	[1e-4, 2e-4, 5e-4]	[8e-4, 1e-3, 2e-3]	[5e-4, 8e-4, 1e-3]	[8e-4, 1e-3, 2e-3]	[5e-4, 8e-4, 1e-3]	[1e-3, 2e-3]	[1e-3, 2e-3]
Embedding LR	[5e-4, 8e-4, 1e-3]	[5e-6, 8e-6, 1e-5]	[8e-4, 1e-3, 2e-3]	[5e-6, 8e-6, 1e-5]	[8e-4, 1e-3, 2e-3]	[5e-6, 8e-6, 1e-5]	[1e-3, 2e-3]	[1e-3, 2e-3]
Batch Size	[8, 16]	[4, 8]	[8, 16]	[4, 8]	[8, 16]	[4, 8]	[8, 16]	[4, 8]
Dropout	[0.1]	[0.1]	[0.5]	[0.5]	[0.5]	[0.5]	[0.1]	[0.1]
# Parameters	8.5M	130M	15M	140M	16M	143M	5M	130M
Epochs	60	50	60	50	60	50	60	50
Avg Time/Epoch	10	40	60	120	60	120	10	15
Table 16: Different hyperparameters and the values considered for each of them in the models. The best hyperparameters for each model for 5-fold cross-validation on ASDiv-A are highlighted in bold. Average Time/Epoch is measured in seconds.
Tag	Description
[NUMx]	Number
[NAMEx]	Names of Persons
[OBJsx]	Singular Object
[OBJpx]	Plural Object
[MODx]	Modifier
Table 17: List of tags used in annotated templates.  $x$  denotes the index of the tag.
6. Apply the Change order of Objects and Change order of Events or Phrases variations on the Base Example and on all the variations obtained so far.
Table 21 provides some variations for the example in Table 20. Note that two separate examples were created through the 'Add irrelevant information' variation. The first by applying the variation on the Original Example and the second by applying it on a previously created example (as directed in Step-4).
To make sure that different workers following our protocol make similar types of variations, we hold a trial where each worker created variations from the same 5 seed examples. We observed that barring minor linguistic differences, most of the created examples were the same, thereby indicating the effectiveness of our protocol.
C Analyzing Attention Weights
In Table 22, we provide more examples to illustrate the specific word to equation correlation that the constrained model learns.
D Examples of Simple Problems
In Table 23, we provide a few simple examples from SVAMP that the best performing Graph2Tree model could not solve.
In this paper, we consider the task of automatically solving Math Word Problems (MwPs). Our work encourages the development of better systems that can robustly solve MwPs. Such systems can be deployed for use in the education domain. E.g., an application can be developed that takes MwPs as input and provides detailed explanations to solve them. Such applications can aide elementary school students in learning and practicing math.
We present a challenge set called SVAMP of one-unknown English Math Word Problems. SVAMP is created in-house by the authors themselves by applying some simple variations to examples from ASDiv-A (Miao et al., 2020), which is a publicly available dataset. We provide a detailed creation protocol in Section B. We are not aware of any risks associated with our proposed dataset.
To provide an estimate of the energy requirements of our experiments, we provide the details such as computing platform and running time in Section A. Also, in order to reduce carbon costs from our experiments, we first perform a broad hyperparameter search over only a single fold for the datasets and then run the cross validation experiment over a select few hyperparameters.
Excerpt of Example Template Form	Beth has 4 packs of red crayons and 2 packs of green crayons. Each pack has 10 crayons in it. [NAME1] has [NUM1] packs of [MOD1] [OBJp1] and [NUM2] packs of [MOD2] [OBJp1].
Excerpt of Example Template Form	In a game, Frank defeated 6 enemies. Each enemy earned him 9 points. In a game [NAME1] defeated [NUM1] [OBJp1]. Each [OBJs1] earned him [NUM2] points.
Table 18: Example uses of tags. Note that in the first example, the word 'packs' was not replaced since it is a collective. In the second example, the word 'points' was not replaced because it is too instance-specific and no other word can be used in that place.
Seed Example Body	Beth has 4 packs of crayons. Each pack has 10 crayons in it. She also has 6 extra crayons.
Seed Example Question	How many crayons does Beth have altogether?
Seed Example Equation	4*10+6
Script Example Body	[NAME1] has [NUM1] packs of crayons. Each pack has [NUM2] crayons in it. She also has [NUM3] extra crayons.
Script Example Question	How many crayons does [NAME1] have altogether?
Script Example Equation	[NUM1] * [NUM2] + [NUM3]
Table 19: An example of suggested templates. Note that the preprocessing script could not successfully tag crayons as  $[OBJp1]$ .
Script Example Body	[NAME1] has [NUM1] packs of crayons. Each pack has [NUM2] crayons in it. She also has [NUM3] extra crayons.
Script Example Question	How many crayons does [NAME1] have altogether?
Base Example Body	[NAME1] has [NUM1] packs of [OBJp1]. Each pack has [NUM2] [OBJp1] in it. She also has [NUM3] extra [OBJp1].
Base Example Question	How many [OBJp1] does [NAME1] have altogether?
[OBJ1]	crayon, crayons
Alternate for [OBJ1]	pencil, pencils
Alternate for [MOD]	small, large
Table 20: An example of editing the Suggested Templates. The edits are indicated in green.
Base Example Body	[NAME1] has [NUM1] packs of [OBJp1]. Each pack has [NUM2] [OBJp1] in it. She also has [NUM3] extra [OBJp1]
Base Example Question	How many [OBJp1] does [NAME1] have altogether?
Base Example Equation	[NUM1] * [NUM2] + [NUM3]
Category	Question Sensitivity
Variation	Same Object, Different Structure
Variation Body	[NAME1] has [NUM1] packs of [OBJp1]. Each pack has [NUM2] [OBJp1] in it. She also has [NUM3] extra [OBJp1].
Variation Question	How many [OBJp1] does [NAME1] have in packs?
Variation Equation	[NUM1] * [NUM2]
Category	Structural Invariance
Variation	Add irrelevant information
Variation Body	[NAME1] has [NUM1] packs of [OBJp1] and [NUM4] packs of [OBJp2]. Each pack has [NUM2] [OBJp1] in it. She also has [NUM3] extra [OBJp1].
Variation Question	How many [OBJp1] does [NAME1] have altogether?
Variation Equation	[NUM1] * [NUM2] + [NUM3]
Variation Body	[NAME1] has [NUM1] packs of [OBJp1] and [NUM4] packs of [OBJp2]. Each pack has [NUM2] [OBJp1] in it. She also has [NUM3] extra [OBJp1].
Variation Question	How many [OBJp1] does [NAME1] have in packs?
Variation Equation	[NUM1] * [NUM2]
Table 21: Example Variations
Input Problem	Predicted Equation	Answer
Mike had 8 games. After he gave some to his friend he had 5 left. How many games did he give to his friend?	8 - 5	3 √
After Mike gave some games to his friend he had 5 left. If he had 8 games initially, how many games did he give to his friend?	5 - 8	-3 X
Jack bought 5 radios but only 2 of them worked. How many radios did not work?	5 - 2	3 √
Jack bought 5 radios but only 2 of them worked. How many more radios did not work than those that did?	5 - 2	3 X
Ross had 6 marbles. He sold 2 marbles to Joey. How many marbles does Ross have now?	6 - 2	4 √
Ross had 6 marbles. Joey sold 2 marbles to Ross. How many marbles does Ross have now?	6 - 2	4 X
Bob collected 7 cans. He lost 3 of them. How many cans does Bob have now?	7 - 3	4 √
Bob had 7 cans. He collected 3 more. How many cans does Bob have now?	7 - 3	4 X
Joey had 9 pens. he used 4 of them. How many pens does he have now?	9 - 4	5 √
Joey used 4 pens. If he had 9 pens initially, how many pens does he have now?	4 - 9	-5 X
Jill read 30 pages in 10 days. How many pages did she read per day?	30 / 10	3 √
Jill can read 3 pages per day. How many pages can she read in 10 days?	3 / 10	0.33 X
Mary's hair was 15 inches long. After she did a haircut, it was 10 inches long . how much did she cut off ?	15 - 10	5 √
Mary cut off 5 inches of her hair. If her hair is now 10 inches long, how long was it earlier?	5 - 10	-5 X
Table 22: Attention paid to specific words by the constrained model.
Input Problem	Correct Equation	Predicted Equation
Every day ryan spends 6 hours on learning english and 2 hours on learning
chinese. How many more hours does he spend on learning english than he
does on learning chinese?	6 - 2	2 - 6
In a school there are 34 girls and 841 boys. How many more boys than girls
does the school have?	841 - 34	34 - 841
David did 44 push-ups in gym class today. David did 9 more push-ups than
zachary. How many push-ups did zachary do?	44 - 9	44 + 9
Dan has $ 3 left with him after he bought a candy bar for $ 2. How much
money did he have initially?	3 + 2	3 - 2
Jake has 11 fewer peaches than steven. If jake has 17 peaches. How many
peaches does steven have?	11 + 17	17 - 11
Kelly gives away 91 nintendo games. How many did she have initially if she
still has 92 games left?	91 + 92	92 - 91
Emily is making bead necklaces for her friends. She was able to make 18
necklaces and she had 6 beads. How many beads did each necklace need?	18 / 6	6 / 18
Frank was reading through some books. Each book had 249 pages and it
took frank 3 days to finish each book. How many pages did he read per day?	249 / 3	(249 * 3) / 3
A mailman has to give 5 pieces of junk mail to each block. If he gives 25
mails to each house in a block, how many houses are there in a block?	25 / 5	5 / 25
Faye was placing her pencils and crayons into 19 rows with 4 pencils and 27
crayons in each row. How many pencils does she have?	19 * 4	19 * 27
White t - shirts can be purchased in packages of 53. If mom buys 57 packages
of white t - shirts and 34 trousers, How many white t - shirts will she have?	53 * 57	(53 * 57) + 34
An industrial machine can make 6 shirts a minute. It worked for 5 minutes
yesterday and for 12 minutes today. How many shirts did machine make
today?	6 * 12	5 + 12
Table 23: Some simple examples from SVAMP on which the best performing Graph2Tree model fails.