SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 4: Size of UnbiasedMWP data split.
Example 1
Context: There were 892 tourists in the morning, 255 left at noon, and 304 came in the afternoon.
Question: How many tourists were there at this time?
Solution Expression: (892 - 255) + 304
Variation:
(1) How many times as many tourists arrive in the afternoon as leave at noon? — 304 / 255
(2) How many more tourists came in the afternoon than left at noon? — 304 - 255
(3) How many times as many tourists come in the morning as in the afternoon? — 892 /= 304
(4) How many more tourists came in the morning than in the afternoon? — 892 - 304
(5) How many tourists came to the science park on this day? — 892 + 304
(6) How many tourists were left at noon? — 892 - 255
Example 2
Context: The school has 26 basketballs. There are 4 fewer
volleyballs than 12 times as many basketballs.
Question: How many volleysballs are there?
Solution Expression: (26*12) - 4
Variation:
(1) How many volleysballs and basketballs are there? — 26 +
((26*12) - 4)
(2) How many more volleysballs are there than basketballs? —
((26*12) - 4) - 26
Example 3
Context: 6 groups from class A of a primary school donated $624
to the earthquake-stricken area, while 5 groups from Class B
donated A total of $705 yuan.
Question: What is the average donation per group in Class A?
Solution Expression: 624 / 6
Variation:
(1) How much did the two classes contribute altogether? —
624 + 705
(2) How many times did Class A donate as much as class B? —
— 624 / 705
(3) How much more did Class B donate than Class A? — 705
— 624
(4) How much does the average group in Class B donate? —
705 / 5
(5) How much more per group did Class B donate than class
A? — (705 / 5) - (624 / 6)
(6) How many times did the average group in Class B donate as
much as the average group in Class A? — (705 / 5) / (624
/ 6)
Figure 3: Some examples of our data variation.
A.3 Data statistic
We analyze the proportions of data of different prefix expression lengths in UnbiasedMWP dataset and the result is shown in Table 5. We analyze our UnbiasedMWP to count the size of different variation data, the statistical result is shown in Table 6. Note that the count of All data is not equal to the sum of the above rows in the table, because there will be some overlap between the variation data obtained in the three data variation methods mentioned in Section 2.2.
Expression Length	3	5	7	9	11	>11
Count	6357	2560	1011	215	90	31
Table 5: Statistics analyse on prefix expression length.
Count
Source	2907
Variable assortment	5083
Sub-expression	2843
Whole expression	2205
All data	10264
We also analyze the accuracy of data of different prefix expression lengths for Bert2Tree model shown in Table 7. Experimental results show that the longer the expression, the lower the accuracy.
Table 6: Statistics analyse on variation data.
Expression Length	3	5	7	>=9
Count	85.3	74.3	60.30	26.5
Table 7: Performance of Bert2Tree on different prefix expression length of UnbiasedMWP-All.
Pytorch $^3$  is used to implement our our MWP solver on Linux with NVIDIA RTX1080Ti GPU card. Our Bert2Tree model is constructed by replacing the encoder in GTS model with the Chinese Bert(Cui et al., 2020). The learning rate is set as  $5e^{-5}$  and  $1e^{-3}$  for Bert encoder and tree-decoder respectively. Adam is set as the optimizer of Bert2Tree while  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$ , and  $\epsilon = 1e^{-8}$ . The batch size is 32. Dropout weight is set as 0.5 with weight decay  $1e^{-5}$ . For the other four models, Math-EN, Group-Attn, GTS and Graph2Tree, we follow their original parameter settings in (Hong et al., 2021). Since the data pre
processing code in Graph2Tree is not open, we do not evaluate this model on our own data.
In the experiments, we train Bert2Tree for 100 epochs on Math23K while 50 epochs on our UnbiasedMWP-Source and UnbiasedMWP-All data, because Math23K is a larger benchmark dataset which contains 23K samples. For the Del_q experiments, We intercept the last sentence (question) by detecting punctuation marks in Math23K which may cause some very small errors but does not affect the overall results of the experiment. For our UnbiasedMWP dataset, we directly use the context to do the Del_q experiment.
Math Word Problem Solving
In recent years, deep learning models especially Seq2Seq models (Wang et al., 2017; Li et al., 2019; Wang et al., 2018; Xie and Sun, 2019; Zhang et al., 2020; Qin et al., 2021; Shen et al., 2021; Wu et al., 2021), have made great progress in MwPs by learning to translate problem text in natural language into mathematical solution expression. (Wang et al., 2017) is the first to apply deep learning in MwPs and propose a widely used dataset called Math23K. (Li et al., 2019) propose a group attention mechanism to extract multi-dimensional features. (Xie and Sun, 2019) propose a tree decoder to decode expression as prefix order. Based on (Xie and Sun, 2019), (Zhang et al., 2020) improve the encoder embedding by fusing a graph encoder's output. (Qin et al., 2021) propose a framework by applying multiple auxiliary tasks to improve the problem embedding and the ability to predict commonsense constants. (Shen et al., 2021) devise a new ranking task for MWP and propose the Generate & Rank, a multi-task framework based on a generative pretrained language model. (Wu et al., 2021) propose a novel Edge-Enhanced Hierarchical Graph-to-Tree model (EEH-G2T), in which the math word problems are represented as edge-labeled graphs.
Challenging Datasets and Adversarial Examples of MWP More challenging datasets in MWP are proposed in recent years, Ape210K (Zhao et al., 2020) provides a large-scale benchmark for evaluating MWP solvers, HMWP (Qin et al., 2020) is a Chinese MWP benchmark including examples with multiple-unknown variables requiring non-linear equations to solve.
Although solvers have achieved impressive performance on these datasets, the robustness of the
<sup>3</sup>http://pytorch.org
solvers is questioned in (Kumar et al., 2021). Besides, (Patel et al., 2021) also points out that MWP solvers rely on shallow heuristics to achieve high performance and propose SVAMWP dataset. SVAMWP is more reliable and robust for measuring the performance of MWP solvers, because it raises the requirement for the model's sensitivity to question text through applying variations over word problems. Unlike SVAMWP, our variations are applied to expressions to get different expression-question pairs.
A.6 Analysis on Effects of PLM
We conduct experiments on our UnbiasedMWP dataset as shown in Table 1. In the experiments, for models without PLMs such as Math-EN (Wang et al., 2017), GROUPT (Li et al., 2019), GTS (Xie and Sun, 2019), Graph2Tree (Zhang et al., 2020), they perform worse on UnbiasedMWP-All dataset than on UnbiasedMWP-Source dataset, whereas for Bert2Tree model with Bert (Cui et al., 2020), it performs significantly better on UnbiasedMWP-All dataset. This shows that the UnbiasedMWP-ALL dataset with diverse questions is more likely to confuse the model, because the context of the sample remains unchanged and only changes the question. However, Bert2tree can better distinguish the difference between diverse questions through the pre-trained language model.