SlowGuess/NAACL_2022_OCR huggingface dataset:
Unsolvable problem 1: Answer:	The price of a ball is 6 yuan, and the price of a basketball is less than 13 times of the ball's price. How much might the price of the basketball be? ?
Unsolvable problem 2: Answer:	x is a single digit and the quotient of x72/47 is also a single digit, what is x at most? 3
Unsolvable problem 3: Answer:	In the yard there were 25 chick- ens and rabbits. Together they had 80 legs. How many rabbits were in the yard? (80-25)*2/(4-2) = 15
Table 1: This table shows three kinds of discarded MwPs in Ape210k. The first one does not have a certain answer, and the solution of the second one cannot be represented by equations. Solving the third problem requires external constants. Thus we filter those problems out in our Ape-clean dataset.
the problems without answers can not be used for fully-supervised setting. Besides, the problems without annotated equations but only answer values can be used in the weakly-supervised learning setting. Therefore, we follow the rules below to select the usable problems from Ape210k to construct an Ape-clean dataset, which can be used for the fully-supervised learning setting. (i). We remove all MwPs that have no answer values nor equations. (ii). We remove all MwPs that only have answer values without equations. (iii). We remove all MwPs with a problem length  $m > 100$  or an answer equation length  $n > 20$ , as they will bring obstacles for training. (iv). We remove all MwPs requiring external constants except 1 and  $\pi$ . (v). We remove all duplicated problems with the MwPs in Math23k, because almost all problems in Math23k can be found in Ape-210k. After data filtering, the Ape-clean dataset contains 81,225 MwPs, including 79,388 training problems and 1,837 testing problems. The remaining 129,263 problems in Ape210k are regarded as Ape-unsolvable, which can be used in the pre-training tasks in the settings of self-supervised and weakly-supervised learning.
Model Comparison. We first compare our approach with the most recent representative baselines on the benchmark Math23k dataset. The first baseline is DNS which is the pioneering work us
$^{1}$ https://github.com/ymcui/Chinese-BERT-wwm
$^{2}$ https://huggingface.co/bert-base-uncased and https://huggingface.co/roberta-base
<sup>3</sup>https://github.com/LZhenwen/MWP-BERT
Math23k	Math23k*	MathQA
DNS	-	58.1	-
Math-EN	66.7	-	-
GTS	75.6	74.3	71.3
NS-Solver	75.7	-	-
Graph2Tree	77.4	75.5	72.0
TSN-MD	77.4	75.1	-
KA-S2T	76.3	-	-
NumS2T	78.1	-	-
EEH-G2T	78.5	-	-
RPKHS	83.9	82.2	-
Encoder pre-train
RoBERTa	83.5	81.7	75.3
BERT	83.8	82.0	75.1
MWP-RoBERTa	84.5	82.0	76.6
MWP-BERT	84.7	82.4	76.2
Seq2Seq pre-train
REAL	82.3	80.0	-
BERT-CL	83.2	-	76.3
Gen&Rank	85.4	84.3	-
Table 2: Comparison of answer accuracy  $(\%)$  among our proposed models and different baselines. Math23k column shows the results on the public test set and Math23k* is 5-fold cross validation on Math23k dataset. MathQA is adapated from Li et al. (2021); Tan et al. (2021). "RoBERTa" and "BERT" represent results without pre-training. "MWP-RoBERTa" and "MWP-BERT" represent first pre-training with proposed tasks and then fine-tuning.
ing the Seq2Seq model to solve MwPs. MathEN (Wang et al., 2018) proposes an equation-normalization method and uses vanilla Seq2Seq model to get solutions. GTS (Xie and Sun, 2019) proposes a goal-driven tree-based decoder and achieves great results. Graph2Tree (Zhang et al., 2020b) constructs two graphs during data preprocessing to extract extra relationships from text descriptions. KA-S2T (Wu et al., 2020) proposes a novel knowledge-aware model that can incorporate background knowledge. NS-Solver (Qin et al., 2021) designs several auxiliary tasks to help training. NumS2T (Wu et al., 2021b) uses explicit numerical values instead of symbol placeholder to encode quantities. RPKHS (Yu et al., 2021) builds hierarchical reasoning encoder in parallel with PLM encoder. REAL (Huang et al., 2021) proposes a human-like analogical auxiliary learning
strategy. EEH-G2T (Wu et al., 2021a) injects edge label information and the long-range word relationship into graph network. Gen&Rank (Shen et al., 2021) designs a multi-task learning framework for adapting BART in MWP solving. BERT-CL (Li et al., 2021) incorporates contrastive learning strategy with PLM. To avoid the implementation error that may cause unreproducible results of baseline models, we reported the results of these baselines from the papers where they were published, as many previous papers (Zhang et al., 2020b; Shen and Jin, 2020) did.
As shown in Table 2, our MWP-BERT achieves competitive results. It is worth noting that we perform strict pre-training paradigm in MWP solving, i.e., our results come from pre-training on the different annotated MWP examples that will be applied in further fine-tuning. Here, our pre-training only uses Ape-clean and Ape-unsolvable and fine-tuning only uses Math23k/MathQA.
RPKHS, REAL and BERT-CL all incorporate BERT in their model architecture, which are orthogonal to our work. Our MWP-BERT can be utilized as an MWP-specific checkpoint for their encoder part to improve their performance. Besides, REAL, BERT-CL and Gen & Rank are all trying to make Seq2Seq pre-training (Lewis et al., 2020), which adapt both pre-training encoder as well as pretraining decoder for MWP solving. Compared with them, our model focuses on encoder pre-training and aims at obtaining better MWP representation that can be widely applied across various MWP related tasks (like quantity tagging, MWP question generation).
Another interesting observation is that BERT-based models perform better on Chinese MWP datasets while RoBERTa-based models are good at English MWP datasets. Because the Chinese RoBERTa used in this paper is actually a BERT model that uses BERT tokenization but is trained like RoBERTa (drops the Next Sentence Prediction task). Similar behaviors can be observed in Cui et al. (2020). For English setting, RoBERTa performs better than BERT, which is consistent with conclusions raised in Liu et al. (2019b).
Evaluation on Ape-clean and Math23k when being trained by a Joint MWP Set. Moreover, we combine the training set of Math23k and Ape-clean to train MWP-BERT, and then measure the accuracy on the testing set of Math23k and Ape-clean separately. Results shown in Table 2 con
Model	Math23k	Ape-clean
Equ	Ans	Equ	Ans
DNS	50.2	50.3	66.2	66.2
GTS	70.1	81.4	60.4	73.2
RoBERTa	75.8	88.8	66.7	80.2
BERT	76.7	89.4	67.0	80.4
MWP-RoBERTa	77.1	90.2	67.1	80.8
MWP-BERT	77.5	91.2	67.5	81.3
vey interesting evaluation observations. Surprisingly, the accuracy of our models on Math23k reaches above  $90\%$ , which is marvelously high (previous state-of-the-art methods can hardly reach  $80\%$  (Shen and Jin, 2020)). Compared to the results in Table 2, even GTS has a higher accuracy when trained with the big joint MWP set of Ape-clean and Math23k.
By comparing the performance of corresponding groups between Table 2 and Table 3, we can learn that our proposed MWP-BERT pre-training paradigm can achieve more significant boosting with more training examples, which proves the effectiveness of our proposed representation learning techniques.
4.3 Other MWP Understanding Tasks
Standard MWP solving is an equation generation task. To make a sufficient validation of the effectiveness of our model on number representations learning, MWP-specific understanding tasks are further considered. Following Hewitt and Manning (2019); Wallace et al. (2019), we design several number probing tasks and incorporate quantity tagging (Zou and Lu, 2019b) to enlarge the MWP understanding evaluation task.
Following the motivation mentioned in section 3.2, we re-run all the pre-training tasks as probing tasks to evaluate our modeling's understanding ability and test MWP-BERT in a zero-shot scenario, i.e. without fine-tuning the parameters of MWP-BERT and MWP-RoBERTa for the sake of fair comparison. We perform the probing evaluation on both Ape-clean and Ape-unsolvable, except that "OPred" and "TPred" are only evaluated on Ape-clean because they require equation solutions
as the ground truth.
Table 3: Comparison of answer accuracy  $(\%)$  between our proposed models and baselines when they are all trained by the combination of the training set from Apeclean and Math23k dataset.
Model	Accuracy
QT(S)	87.3
QT(R)	88.7
QT.fix)	87.7
QT	90.8
BERT	84.5
RoBERTa	84.6
MWP-BERT	91.0
MWP-RoBERTa	91.5
Table 5: Comparison of tagging accuracy  $(\%)$  between our proposed models and baselines.
Table 4 shows the performances of 4 different PLMs on the above mentioned MWP-specific understanding tasks. Significant improvements can be observed in all the tasks, and demonstrate the effectiveness of our proposed pre-training techniques in improving number representation of PLMs.
Besides, we borrow an MWP-specific sequence labeling task, quantity tagging (Zou and Lu, 2019b) ("QT"), to further compose MWP understanding evaluation settings. Quantity tagging (Zou and Lu, 2019a) is firstly proposed to solve MWP examples with only addition and subtraction operators in their solutions. Briefly speaking, this task requires the model to assign "+" or "None" for every quantity in the problem description and can serve as an MWP understanding evaluation tool to examine the model's understanding of each variable's logic role in the reasoning flow. More exactly, this is also a classification task with 3 possible targets. We extract the corresponding vectors of all quantities according to their positions in encoded problem  $Z$  from Equation 1. Next, a 2-layer feed-forward block is connected to output the final prediction.
Following the setting in baseline method  $QT$  (Zou and Lu, 2019a), we perform 3-fold cross-validation and the results are given in Table 5, which shows that PLMs benefits from the proposed mathematical pre-training and outperforms the baselines.
We run ablation study over the proposed training objectives to investigate the necessity for each of them. As Table 6 shows, all the proposed objectives can achieve improvements individually. Moreover, only using MLM results in weaker MWP solvers on Math23k (1.4% less) and Ape-clean (1.2% less),
NumCount	NTGround	ATPred	CATComp	NumMComp	OPred	TPred	QT
Metric	MSE ↓	Acc ↑	Acc ↑	Acc ↑	Acc ↑	Acc ↑	MSE ↓	Acc ↑
BERT	3.08	0.87	0.75	0.77	0.77	0.50	0.97	84.5
RoBERTa	3.20	0.86	0.76	0.78	0.77	0.51	0.99	84.6
MWP-RoBERTa	0.69	0.92	0.86	0.87	0.86	0.86	0.44	91.0
MWP-BERT	0.67	0.92	0.85	0.87	0.86	0.87	0.45	91.5
Table 4: The evaluation results on MWP-specific understanding tasks. All tasks correspond to the tasks mentioned in section 4. Note that the metric for 2 tasks is mean-squared-error, while others use classification accuracy. "QT" stands for quantity tagging.
Math23k	Ape-clean
Only MLM	89.8	80.1
Only self-supervised	90.4	80.9
w/o MLM	90.1	80.6
w/o NumCount	89.9	80.5
w/o NTGround	90.1	80.4
Only weakly-supervised	90.1	80.8
w/o ATPred	89.7	80.2
w/o CATComp	89.7	80.4
w/o NumMComp	89.6	80.5
Only fully-supervised	91.0	80.5
w/o OPred	90.5	80.3
w/o TPred	90.6	80.5
MWP-BERT	91.2	81.3
Table 6: The experimental results show the effectiveness of every pre-trained task. "Only self-supervised" means we only apply 3 tasks of self-supervised pretraining on the BERT encoder. We also investigate the influence of each task. For example, "w/o MLM" means only performing self-supervised pre-training and discarding the MLM pre-training task.
which again proves the effectiveness of our proposed pre-training tasks. Since the difficulty level of MwPs is usually in proportion to their solution length, we can easily identify that a set of MwPs exhibit a long-tail distribution over solution length, as well as the difficulty level, as shown in. Fig 3 of the Appendix. Thus, the  $87\%$  accuracy of human-level performance in Math23k (Wang et al., 2019) indicates that  $13\%$  of the MwPs are difficult to solve. Any solvers that can improve the accuracy above  $87\%$  are making significant contribution on solving the extremely difficult cases, such as MwPs whose solutions contain  $\geq 4$  variables or single variable being used multiple times. As neural models are known to be limited at dealing with these combinational and symbolic reasoning
cases (Lee et al., 2020), we exam our model on these specially difficult cases. Due to the space limit, we attach several examples of these difficult cases, statistics about solution length distribution and performance for increasing length of solution equations in the Appendix. Besides, it is worth noting that even without MLM objective, our model is able to promote the PLM competitor. Besides, we can observe that linking equation structure and number during pre-training is certainly beneficial for solving MwPs.
We propose MWP-BERT, an MWP-specific PLM model with 8 pre-training objectives to solve the number representation issue in MWP. Also, a new dataset Ape-clean is curated by filtering out unsolvable problems from Ape210k, and the filtered MwPs are useful for self- and weakly-supervised pre-training. Experimental results show the superiority of our proposed MWP-BERT across various downstream tasks on generation and understanding. In terms of the most representative task MWP solving, our approach achieves the highest accuracy, and firstly beats human performance. Better numerical understanding ability is also demonstrated in the probing evaluation. We believe that our study can serve as a useful pre-trained pipeline and a strong baseline in the MWP community.
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. In NAACL, pages 2357-2367.
Daniel Andor, Luheng He, Kenton Lee, and Emily Pitler. 2019. Giving BERT a calculator: Finding operations and arguments with reading comprehension. In EMNLP, pages 5946-5951.
Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: on meaning, form, and understanding in the age of data. In ACL, pages 5185-5198.
Ting-Rui Chiang and Yun-Nung Chen. 2018. Semantically-aligned equation generation for solving and reasoning math word problems. In NAACL.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. Revisiting pretrained models for Chinese natural language processing. In EMNLP: Findings, pages 657-668.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL*, pages 4171–4186.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In *NAACL*, pages 2368-2378.
Mor Geva, Ankit Gupta, and Jonathan Berant. 2020. Injecting numerical reasoning skills into language models. In ACL, pages 946-958.
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In *NAACL*, pages 4129-4138.
Yining Hong, Qing Li, Daniel Ciao, Siyuan Huang, and Song-Chun Zhu. 2021a. Learning by fixing: Solving math word problems with weak supervision. In AAAI.
Yining Hong, Qing Li, Ran Gong, Daniel Ciao, Siyuan Huang, and Song-Chun Zhu. 2021b. Smart: A situation model for algebra story problems via attributed grammar. In AAAI.
Danqing Huang, Shuming Shi, Chin-Yew Lin, Jian Yin, and Wei-Ying Ma. 2016. How well do computers solve math word problems? large-scale dataset construction and evaluation. In ACL, pages 887-896.
Shifeng Huang, Jiawei Wang, Jiao Xu, Da Cao, and Ming Yang. 2021. Recall and learn: A memory-augmented solver for math word problems. In Findings of EMNLP, pages 786-796.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3:585-597.
Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. Mawps: A math word problem repository. In NAACL, pages 1152-1157.
NateKushman，YoavArtzi，LukeZettlemoyer,and ReginaBarzilay.2014.Learningtoautomatically solvealgebrawordproblems.InACL, pages 271- 281.
Dennis Lee, Christian Szegedy, Markus N. Rabe, Sarah M. Loos, and Kshitij Bansal. 2020. Mathematical reasoning in latent space. In ICLR. OpenReview.net.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In ACL, pages 7871-7880.
Jierui Li, Lei Wang, Jipeng Zhang, Yan Wang, Bing Tian Dai, and Dongxiang Zhang. 2019. Modeling intra-relation in math word problems with different functional multi-head attentions. In ACL, pages 6162-6167.
Zhongli Li, Wenxuan Zhang, Chao Yan, Qingyu Zhou, Chao Li, Hongzhi Liu, and Yunbo Cao. 2021. Seeking patterns, not just memorizing procedures: Contrastive learning for solving math word problems. arXiv preprint arXiv:2110.08464.
Zhenwen Liang and Xiangliang Zhang. 2021. Solving math word problems with teacher supervision. In *IJCAI*.
Qianying Liu, Wenyv Guan, Sujian Li, and Daisuke Kawahara. 2019a. Tree-structured decoding for solving math word problems. In EMNLP, pages 2370-2379.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Jinghui Qin, Xiaodan Liang, Yining Hong, Jianheng Tang, and Liang Lin. 2021. Neural-symbolic solver for math word problems with auxiliary tasks. In ACL, pages 5870-5881.
Jinghui Qin, Lihui Lin, Xiaodan Liang, Rumin Zhang, and Liang Lin. 2020. Semantically-aligned universal tree-structured solver for math word problems. In EMNLP, pages 3780-3789.
Jianhao Shen, Yichun Yin, Lin Li, Lifeng Shang, Xin Jiang, Ming Zhang, and Qun Liu. 2021. Generate & rank: A multi-task framework for math word problems. In EMNLP, pages 2269-2279.
Yibin Shen and Cheqing Jin. 2020. Solving math word problems with multi-encoders and multi-decoders. In COLING, pages 2924-2934.
Shuming Shi, Yuehui Wang, Chin-Yew Lin, Xiaojiang Liu, and Yong Rui. 2015. Automatically solving number word problems by semantic parsing and reasoning. In EMNLP, pages 1132-1142.
Minghuan Tan, Lei Wang, Lingxiao Jiang, and Jing Jiang. 2021. Investigating math word problems using pretrained multilingual language models. CoRR, abs/2105.08928.
Avijit Thawani, Jay Pujara, Filip Ilievski, and Pedro A. Szekely. 2021. Representing numbers in NLP: a survey and a vision. In *NAACL-HLT*, pages 644-656.
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. Do NLP models know numbers? probing numeracy in embeddings. In EMNLP, pages 5306-5314.
Lei Wang, Yan Wang, Deng Cai, Dongxiang Zhang, and Xiaojiang Liu. 2018. Translating a math word problem to a expression tree. In EMNLP, pages 1064-1069.
Lei Wang, Dongxiang Zhang, Jipeng Zhang, Xing Xu, Lianli Gao, Bing Tian Dai, and Heng Tao Shen. 2019. Template-based math word problem solvers with recursive neural networks. In AAAI, volume 33, pages 7144-7151.
Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. In EMNLP, pages 845-854.
Qinzhuo Wu, Qi Zhang, Jinlan Fu, and Xuan-Jing Huang. 2020. A knowledge-aware sequence-to-tree network for math word problem solving. In EMNLP, pages 7137-7146.
Qinzhuo Wu, Qi Zhang, and Zhongyu Wei. 2021a. An edge-enhanced hierarchical graph-to-tree network for math word problem solving. In *Findings of EMNLP*, pages 1473–1482.
Qinzhuo Wu, Qi Zhang, Zhongyu Wei, and Xuanjing Huang. 2021b. Math word problem solving with explicit numerical values. In ACL, pages 5859-5869.
Zhipeng Xie and Shichao Sun. 2019. A goal-driven tree-structured neural model for math word problems. In *IJCAI*, pages 5299–5305.
Weijiang Yu, Yingpeng Wen, Fudan Zheng, and Nong Xiao. 2021. Improving math word problems with pre-trained knowledge and hierarchical reasoning. In EMNLP, pages 3384-3394.
Ma Yuhui, Zhou Ying, Cui Guangzuo, Ren Yun, and Huang Ronghuai. 2010. Frame-based calculus of solving arithmetic multi-step addition and subtraction word problems. In 2010 Second International Workshop on Education Technology and Computer Science, volume 2, pages 476-479. IEEE.
Jipeng Zhang, Roy Ka-Wei Lee, Ee-Peng Lim, Wei Qin, Lei Wang, Jie Shao, and Qianru Sun. 2020a. Teacher-student networks with multiple decoders for solving math word problem. In *IJCAI*, pages 4011–4017.
Jipeng Zhang, Lei Wang, Roy Ka-Wei Lee, Yi Bin, Yan Wang, Jie Shao, and Ee-Peng Lim. 2020b. Graph-to-tree learning for solving math word problems. In ACL, pages 3928-3937.
Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar, and Dan Roth. 2020c. Do language embeddings capture scales? In *Findings of EMNLP*, volume EMNLP 2020, pages 4889-4896. Association for Computational Linguistics.
Wei Zhao, Mingyue Shang, Yang Liu, Liang Wang, and Jingming Liu. 2020. Ape210k: A large-scale and template-rich dataset of math word problems. arXiv preprint arXiv:2009.11506.
Yanyan Zou and Wei Lu. 2019a. Quantity tagger: A latent-variable sequence labeling approach to solving addition-subtraction word problems. In ACL, pages 5246-5251.
Yanyan Zou and Wei Lu. 2019b. Text2math: End-to-end parsing text into math expressions. In EMNLP, pages 5330-5340.
Problem 1: Answer:	There are 20 questions in an exam. Solving a question cor- rectly gets 5 points, and 1 point is deducted if the answer is wrong. Jack gets 70 points. How many questions did he get right? 20-(20*5-70)/(5+1)
Problem 2: Answer:	Peter is reading a book. He reads 30% of the whole book on the first day, and 15 pages on the second day. The ratio of the number of pages that has been read to the number of pages not read is 2:3. How many pages does this book have? 15/(1-((3)/(2+3))-30%)
Problem 3: Answer:	There are 72% of 50 students can swim, and (3/5) of 25 girls can swim, how many percent of the boys can swim? (50*72%-25*(3/5))/(50-25)
A.1 Accuracy w.r.t. Solution Length
To better understand the improvement of the MWP solving performance of our model, we evaluate the problems with different lengths of solutions separately. The solution distribution details can be found in Figure 3. It is expected that getting longer solutions requires more comprehensive understanding and complex reasoning, like the three difficult examples shown in Table 7. The results in Table 8 demonstrate that our proposed MWP-BERT overcomes more difficult problems than the vanilla BERT model. Although the statistical improvement from BERT to MWP-BERT is marginal, our method really enhances the mathematical understanding and reasoning ability of PLMs.
A.2 Case Study
We perform case study as shown in Table 9. Firstly, we choose a difficult problem from Math23k dataset and use 3 different solvers to solve it. Both GTS (Xie and Sun, 2019) and Graph2Tree (Zhang et al., 2020b) fail to generate the right solution for it, while our proposed MWP-BERT solves it cor
Figure 3: The solution length distributions on Math23k and Ape-clean.
Table 7: This table shows three difficult problems in Math23k.
#op	#P	BERT	MWP-BERT
0	16	100	100
1	331	95.1	96.3
2	485	90.7	90.7
3	124	79.8	82.3
4	31	58.0	67.7
5	7	85.7	85.7
>5	6	33.3	50
Table 8: The answer accuracy of BERT and MWP-BERT on problems with different lengths in Math23k. #op denotes the number of operators in the solution. #P is the number of problems of that kind of MWPs in the public test set of Math23k.
rectly. This example shows that our encoder has a stronger capability to understand complex MWPs to guide the tree-based decoder generate correct solutions. Secondly, when solving a pair of 2 similar problems (i.e., problem 1 and problem 2 in Table 9), GTS, Graph2Tree and our MWP-BERT successfully solve the former problem. However, the baseline methods GTS and Graph2Tree both fail to solve the latter one. Our MWP-BERT generates the correct answer. This example proves that our probing tasks help the encoder to capture minor variations inside the problem description, leading to more accurate solutions.