SlowGuess/ACL_2020_OCR huggingface dataset:
Wang Ling, Dani Yogatama, Chris Dyer and Phil Blunson. 2017. Program Induction for Rationale Generation: Learning to Solve and Explain Algebraic Word Problems. Association for Computational Linguistics (ACL), 2017.
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP Natural Language Processing Toolkit. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 55-60.
Anirban Mukherjee and Utpal Garain. 2008. A review of methods for automatic understanding of natural language mathematical problems. Artificial Intelligence Review, 29(2):93-122.
Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL), 2002, pages 311-318.
Subhro Roy and Dan Roth. 2015. Solving general arithmetic word problems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2015, pages 1743-1752.
Subhro Roy and Dan Roth. 2017. Unit Dependency Graph and its Application to Arithmetic Word Problem Solving. AAAI-2017.
Shuming Shi, Yuehui Wang, Chin-Yew Lin, Xiaojiang Liu, and Yong Rui. 2015. Automatically Solving Number Word Problems by Semantic Parsing and Reasoning. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), Lisbon, Portugal, 2015, pages 1132-1142.
Shyam Upadhyay and Ming-Wei Chang. 2015. DRAW: A challenging and diverse algebra word problem set. Number MSR-TR-2015-78
Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.
Zhipeng Xie, and Shichao Sun. 2019. A goal-driven tree-structured neural model for math word problems. In Proceedings of the 28th International Joint Conference on Artificial Intelligence. AAAI Press.
Appendix A: Examples of a few Selected Problem Types
Table 5 shows examples of selected types in "Basic arithmetic operations", "Aggregative operations", and "Additional domain knowledge required" categories.
Problem type	Examples
Basic arithmetic operations
Number-Operation	I have 3 hundreds, 8 tens, and 3 ones. What number am I?
TVQ-Initial	Tim's cat had kittens. He gave 3 to Mary and 6 to Sara. He now has 9 kittens. How many kittens did he have to start with?
TVQ-Change	For the school bake sale, Wendy made pastries. She baked 41 cupcakes and 31 cookies. After the sale, she had 32 to take back home. How many pastries did she sell?
TVQ-Final	Melanie had 7 dimes in her bank. Her dad gave her 8 dimes and her mother gave her 4 dimes. How many dimes does Melanie have now?
Multi-Step	They served a total of 179 adults and 141 children, if 156 of all the people they served are male, how many are female? (combination of multiple operations)
(Equation: x = (179+141)-156)
Aggregative operations
Algrbra-1	Maddie, Luisa, and Amy counted their books. Maddie had 15 books. Luisa had 18 books. Together, Amy and Luisa had 9 more books than Maddie. How many books did Amy have?
(Equation: (x+18)-9 = 15, where x: "money of Amy")
Algrbra-2	The cost of a private pilot course is $1,275. The flight portion costs $625 more than the ground school portion. What is the cost of each?
(Equation: x+y = 1275; x = y+625, where x: The cost of flight portion; y: The cost of ground school portion)
Comparison	A bookstore was selling 2 books for $15.86. You could buy 7 books for $55.93 online. Which place has a lower unit price?
Set-Operation	Sarah's team played 8 games of basketball. During the 8 games her team's scores were 69, 68, 70, 61, 74, 62, 65 and 74. What is the mean of the scores?
Ratio	There are 43 empty seats and 7 occupied seats on an airplane. What is the ratio of the number of occupied seats to the total number of seats?
Number-Pattern	The Wholesome Bakery baked 5 loaves of bread on Wednesday, 7 loaves of bread on Thursday, 10 loaves of bread on Friday, 14 loaves of bread on Saturday, and 19 loaves of bread on Sunday. If this pattern continues, how many loaves of bread will they bake on Monday?
Additional domain knowledge required
G.C.D.	A teacher is to arrange 60 boys and 72 girls in rows. He wishes to arrange them in such a way that only boys or girls will be there in a row. Find the greatest number of students that could be arranged in a row.
L.C.M.	On a track for remote-controlled racing cars, racing car A completes the track in 28 seconds, while racing car B completes it in 24 seconds. If they both start at the same time, after how many seconds will they be side by side again?
UnitTrans	Mrs. Hilt will buy a new pair of shoes in 11 days. How many minutes must she wait before she can buy her new pair of shoes?
Table 5: Examples of selected problem types
Appendix B: Problematic MwPs in MathQA
Table 6 shows examples of inconsistent MwPs in MathQA, and Figure 3 shows examples using the same sentence pattern.
Error types	Examples	Annotated Formula	Labeled Answer
Incorrect formula (67%)	Problem-1
“What is the 25 th digit to the right of the decimal point in the decimal form of 6 / 11 ?”	Annotation Formula:
divide(n1,n2)
Desired Formula:
Not available6	6
Problem-2
“In two triangles, the ratio of the area is 4 : 3 and that of their heights is 3 : 4. find the ratio of their bases.”	Annotation Formula:
multiply(n0,n0)|multi-ply(n1,n1)|divide(#0,#1)
Desired Formula:
Not available7	16:9
Problematic description (23%)	Problem-3
“The lcm of two numbers is 495 and their lcm is 5. if the sum of the numbers is 10, then their difference is”
Desired text: “...sum of the numbers is 100”	Annotation Formula:
multiply(n0,n1)|divide(#0,n2)	10
Valueless answer (10%)	Problem-4
“9886 + x = 13200, then x is ?”
Options: a) 3327, b)3237, c)3337, d) 2337, e)none of these	Annotation Formula:
subtract(n1,n0)	Annotation Operation (e): none of these
Desired Answer: 3414
A train running at the speed of  $100\mathrm{km / hr}$  crosses a pole in 18 seconds. What is the length of the train? A train running at the speed of  $100\mathrm{km / hr}$  crosses a pole in 9 seconds. What is the length of the train? A train running at the speed of  $108\mathrm{km / hr}$  crosses a pole in 7 seconds. Find the length of the train? A train running at the speed of  $110\mathrm{km / hr}$  crosses a pole in 9 seconds. What is the length of the train? A train running at the speed of  $120\mathrm{km / hr}$  crosses a pole in 12 seconds. What is the length of the train? A train running at the speed of  $120\mathrm{km / hr}$  crosses a pole in 18 seconds. What is the length of the train? A train running at the speed of  $120\mathrm{km / hr}$  crosses a pole in 9 seconds. Find the length of the train? A train running at the speed of  $126\mathrm{km / hr}$  crosses a pole in 9 seconds. Find the length of the train? A train running at the speed of  $142\mathrm{km / hr}$  crosses a pole in 12 seconds. Find the length of the train? A train running at the speed of  $162\mathrm{km / hr}$  crosses a pole in 9 seconds. Find the length of the train? A train running at the speed of  $180\mathrm{km / hr}$  crosses a pole in 18 seconds. What is the length of the train? A train running at the speed of  $180\mathrm{km / hr}$  crosses a pole in 36 seconds. What is the length of the train? A train running at the speed of  $180\mathrm{km / hr}$  crosses a pole in 7 seconds. What is the length of the train? A train running at the speed of  $180\mathrm{km / hr}$  crosses a pole in 8 seconds. Find the length of the train? A train running at the speed of  $180\mathrm{km / hr}$  crosses a pole in 9 seconds. Find the length of the train?
Table 6: Some examples of inconsistent answers on MathQA.
Figure 3: Examples in MathQA with the same sentence pattern (after sentence normalization). These MwPs all share the same sentence pattern, "a train running at the speed of [NUM1] km/hr crosses a pole in [NUM2] sec. [what is | find] the length of the train?"
There is no proper operation on the MathQA operation set. There is no proper operation on the MathQA operation set.
Appendix C: Additional Experiments for Corpus Diversity Metrics
We also provide a syntactic pattern diversity to measure the syntactic diversity of an MWP. Let  $P = \{P_{1}, P_{2}, \dots, P_{M}\}$  be a specific set of MwPs in a given corpus with the same problem type, where  $P_{i}$  is the  $i$ -th MWP in  $P$ , and  $T_{i}$  is the corresponding POS sequence of  $P_{i}$ . For example, "NNP VBZ CD NNS" is the POS tagging sequence for "Mary has 5 books." For a given  $P_{i}$ , we define its syntactic pattern diversity  $(SD_{i})$  as
$$
S D _ {i} = 1 - \max _ {j, j \neq i} \frac {B L E U (T _ {i} , T _ {j}) + B L E U (T , T _ {i})}{2},
$$
where  $BLEU(T_i, T_j)$  is measured between  $T_i$  and  $T_j$  ( $j \neq i$ ,  $j = \{1, 2, \dots, M\}$ ). Figure 4 shows that there are  $87\%$ ,  $54\%$ ,  $46\%$  and  $33\%$  identical syntactic patterns (these numbers are the percentages of MwPs with  $SD_i = 0$  w.r.t. each dataset) in the MathQA, IL, AI2, and ALGES corpora, respectively, while ASDiv only has  $4\%$ . This shows that our corpus is also more diverse in terms of syntactic patterns.
We also measure the diversity between the test-set and the training-set, as the similarity between them is a critical factor for causing exaggerated performance. The  $LD$  and  $SD$  metrics between the test-set and the training-set can be obtained by modifying the previous formulas to
$$
\begin{array}{l} \begin{array}{r} L D _ {i} = 1 - \max _ {j} \frac {B L E U (P _ {i} , P _ {j}) + B L E U (P _ {j} , P _ {i})}{2}, \\ P _ {i} \in D S _ {t e s t} a n d P _ {j} \in D S _ {t r a i n} \end{array} \\ \begin{array}{c} S D _ {i} = 1 - \max  _ {j} \frac {B L E U (T _ {i} , T _ {j}) + B L E U (T _ {j} , T _ {i})}{2}, \\ T _ {i} \in D S _ {t e s t} a n d T _ {j} \in D S _ {t r a i n} \end{array} \\ \end{array}
$$
where  $DS_{test}$  and  $DS_{train}$  are all the MwPs in the test set and the training set, respectively. For a given problem  $P_i$  from the test set,  $LD_i$  and  $SD_i$  denote the lexicon-pattern and syntactic-pattern diversity between  $P_i$  and all the problems  $P_j$  (in the training set), respectively. If  $P_i$  has a low diversity index, it can be easily solved via a training set MwP with similar patterns.
Figure 4: Syntactic pattern diversity of various corpora
Figure 5: Lexicon usage diversity of various corpora: test-set versus training-set
Figure 6: Syntactic pattern diversity of various corpora: test-set versus training-set
Figure 7: Lexicon usage diversity measured only within the test-set for various corpora. Here, the suffix "-test" denotes that it is measured only within the test-set.
However, we also need another critical factor - the CLD value within the test-set - to assess the true value of a given corpus. Suppose we take an MWP with a high LD value from the existing training-set, and then duplicate it 300 times to create a new test-set. This would still result in a high CLD value between the training-set and this new test-set. However,
this setting would be meaningless in terms of assessing the true performance of MWP solvers. Likewise, we could also enlarge the training-set by duplicating MwPs without affecting the CLD value against the test-set. However, it would be also meaningless as no new information would be provided.
For AI2, IL, KAZB, ALGES, AllArith, DRAW, and MathQA, we follow their own either n-fold cross-validation setting or the train/dev/test splits originally specified in each dataset. For MathQA-C and ASDiv-A, we follow the same n-fold cross-validation setting specified in Section 4. Figure 5 shows the LD between the test-set and the training-set for various corpora.
Also, though not directly shown in this figure, the lexicon diversity indices of  $48\%$  of the MwPs in our ASDiv-A are larger than or equal to 0.5. In contrast, they are  $35\%, 34\%, 29\%, 23\%, 23\%, 18\%$  and  $7\%$  in AI2, IL, ALGES, KAZB, AllArith, DRAW, and MathQA-C respectively. These statistics suggest that our corpus is more diverse (and thus more difficult) than other well-known MWP corpora. Figure 6 shows the SD between the test-set and the training-set for different corpora. Again, this shows that our corpus also contains more diverse POS sequences.
Last, in Figure 5, MathQA actually possesses the highest CLD between its official test-set and training-set: 0.85, surprisingly higher than that of all other corpora (e.g., they are 0.52, 0.44, 0.42 and 0.42 for ASDiv-A, IL, AllArith, and AI2, respectively). In comparison with its CLD across the whole corpus (0.05 in Figure 1), 0.85 is much higher. It is thus suspected that the MwPs within its test/training-set might be very similar to each other. As explained in Footnote #7, a test-set with very similar MwPs hinders the assessment of the true performance of an MWP solver. We thus further compare the CLDs within the test set for a few corpora. Figure 7 illustrates that the CLD=0.27 of MathQA measured within the test set is actually low in comparison with the corresponding CLDs of other corpora (e.g., the means are 0.57 for ASDiv-A and ASDiv corpora).