SlowGuess/NAACL_2022_OCR huggingface dataset:
Zhenwen Liang $^{1}$ , Jipeng Zhang $^{2}$ , Lei Wang $^{3}$ , Wei Qin $^{4}$ , Yunshi Lan $^{5}$ , Jie Shao $^{6}$ , and Xiangliang Zhang $^{\text{☑}1}$
<sup>1</sup>University of Notre Dame, {zliang6, xzhang33}@nd.edu
$^{2}$ Hong Kong University of Science and Technology, jzhanggr@connect.ust.hk
$^{3}$ Singapore Management University, lei.wang.2019@phdcs.smu.edu.sg
<sup>4</sup>Hefei University of Technology, qinwei.hfut@gmail.com
$^{5}$ East China Normal University, yslan@dase.ecnu.edu.cn
<sup>6</sup>University of Electronic Science and Technology of China, shaojie@uestc.edu.cn
Math word problem (MWP) solving faces a dilemma in number representation learning. In order to avoid the number representation issue and reduce the search space of feasible solutions, existing works striving for MWP solving usually replace real numbers with symbolic placeholders to focus on logic reasoning. However, different from common symbolic reasoning tasks like program synthesis and knowledge graph reasoning, MWP solving has extra requirements in numerical reasoning. In other words, instead of the number value itself, it is the reusable numerical property that matters more in numerical reasoning. Therefore, we argue that injecting numerical properties into symbolic placeholders with contextualized representation learning schema can provide a way out of the dilemma in the number representation issue here. In this work, we introduce this idea to the popular pre-training language model (PLM) techniques and build MWP-BERT, an effective contextual number representation PLM. We demonstrate the effectiveness of our MWP-BERT on MWP solving and several MWP-specific understanding tasks on both English and Chinese benchmarks.
Recent works in math word problem (MWP) solving (Wang et al., 2018, 2019; Liu et al., 2019a; Li et al., 2019; Xie and Sun, 2019; Zhang et al., 2020b; Wu et al., 2020; Qin et al., 2021; Huang et al., 2021; Wu et al., 2021a; Yu et al., 2021; Shen et al., 2021) arrange the pipeline into a sequence-to-sequence framework. In brief, they use deep representation and gradient optimization as well as symbolic constraints to discover discrete symbolic combinations of operators and variants. Fundamentally, MWP solving system aims to perform symbolic reasoning by searching through a combinatorial solution space given the text description
Text: Some workers are producing 660 clothes. It has been 5 days and 75 clothes are produced per day. But they have to finish all clothes in 3 more days. How many clothes should be processed per day from now?
Equation:  $(660 - 75\times 5)\div 3$
Reasoning Logic:
Text: Some workers are producing 660 clothes. It has been 5 days and  $\frac{10\%}{3}$  of the total clothes are produced per day. But they have to finish all clothes in 3 more days. How many clothes should be processed per day from now?
Equation:  $660 \times (1 - 10\% \times 5) \div 3$
Figure 1: The second question is obtained from the first one by minor modifications. However, their solution equation and corresponding equation tree structure are different from each other. This demonstrates the importance of considering numerical value information and reasoning logic (equation tree) in contextual modeling.
evidences. Thus, these neurosymbolic methods mainly focus on getting more effective semantic representations (Li et al., 2019; Zhang et al., 2020b; Wu et al., 2020, 2021a; Yu et al., 2021), injecting symbolic constraints (Wang et al., 2018, 2019; Liu et al., 2019a; Xie and Sun, 2019) and how to align semantic space (text descriptions) and huge combinatorial symbolic space (symbolic solutions) (Qin et al., 2021; Shen et al., 2021; Huang et al., 2021). This line of methods has achieved great success and is still holding the lead in various MWP solving benchmarks (Wang et al., 2017; Zhao et al., 2020; Koncel-Kedziorski et al., 2016).
Despite the great performance achieved by the previous methods, there still exists fundamental challenges in number representation for MWP solving. More exactly, number values are required to be considered as vital evidence in solution exploration but existing works are known to be inefficient in
Findings of the Association for Computational Linguistics: NAACL 2022, pages 997 - 1009 July 10-15, 2022 ©2022 Association for Computational Linguistics
capturing numeracy information (Wallace et al., 2019). Intuitively, we could simply treat explicit numbers in the same way with words, i.e., assign position for all numbers in the vocabulary. However, there would be an infinite number of candidates during prediction and it would be impossible to learn their deep representations. In other words, the solution space will be extremely large and the complexity is unacceptable. Therefore, almost all existing works follow the number mapping technique Wang et al. (2017) to replace all numbers with symbolic placeholders (e.g., "x1", "x2"). The core idea here is to get a reasonable solution space by restricting neural networks to leave out numerical characteristics and focus on logic reasoning. However, most of the current MWP solvers do not consider the background knowledge in the context and are usually inefficient in capturing numeracy properties. An example is shown in Fig. 1. Small perturbations in the problem description actually bring large variations in reasoning logic and equation. If the model simply regards "75" and "10%" as the same placeholder "x3", and does not notice the small variation in the context, a wrong solution will be generated.
To this end, we incorporate several numeracy grounded pre-training objectives to inject inductive bias about numerical constraints into dynamic representations. Compared with word candidate sets, useful points in number candidate space are scattered sparsely. However, we identify that during prediction, what matters is the reusable numerical properties of number values. What's more, these properties do not suffer from the sparsity issues of specific values in MwPs. Therefore, compared with assigning a prototype vector for each single number value like (Wu et al., 2021b), it is more reasonable to inject the reusable numerical properties in deep representations, e.g., magnitude and number type. In this work, we propose to design numeracy grounded pre-training objectives to implement soft constraints between symbolic placeholders and numbers in deep representation.
Contributions. We present a suite of numeracy-augmented pre-training tasks with consideration of reasoning logic and numerical properties. More exactly, we introduce several novel pre-training tasks with access to different levels of supervision signals to make use of more available MwP data.
- One basic group of pre-training tasks is designed for the self-supervised setting. Ex
cept for masked language modeling (MLM), we give extra consideration to number-related context information by designing related objectives.
- Another set of pre-training objectives is for the weakly-supervised setting that has only answer annotations but no equation solutions. With access to the answer value, we introduce several tasks to determine the type and the value of the answer.
- The final set is for the fully-supervised setting, where both solution equation and answer are available for the MwPs.
Besides, a group of numeracy grounded pretraining objectives is designed to leverage the corpus of MWP and encourage the contextual representation to capture numerical information. Experiments conducted on both Chinese and English benchmarks show the significant improvement of our proposed approach over all competitors. To our knowledge, this is the first approach that surpasses human performance (Wang et al., 2019) in terms of MWP solving.
Math Word Problems Solving. There exist two major types of MWP, equation set MWP (Wang et al., 2017; Zhao et al., 2020) and arithmetic MWP (Qin et al., 2020; Huang et al., 2016). This work focuses on arithmetic MWP, which is usually paired with one unknown variable. Along the path of the MWP solver's development, the pioneer studies use traditional rule-based methods, machine learning methods and statistical methods (Yuhui et al., 2010; Kushman et al., 2014; Shi et al., 2015; Koncel-Kedziorski et al., 2015). Afterwards, inspired by the development of sequence-to-sequence (Seq2Seq) models, MWP solving has been formulated as a neurosymbolic reasoning pipeline of translating language descriptions to mathematical equations with encoder-decoder framework (Wang et al., 2018, 2019; Li et al., 2019; Zhang et al., 2020b; Yu et al., 2021; Wu et al., 2021a). By fusing hard constraints into decoder (Chiang and Chen, 2018; Liu et al., 2019a; Xie and Sun, 2019; Shen and Jin, 2020; Zhang et al., 2020a), MWP solvers achieve much better performance then. Several works propose to utilize multi-stage frameworks (Wang et al., 2019; Huang et al., 2021; Shen
et al., 2021; Liang and Zhang, 2021) to make more robust solvers. Also, several new works made attempts to improve MWP solver beyond supervised settings (Hong et al., 2021a,b).
Among all these previous studies, the most relevant ones to our work can be categorized into two groups. First, it has been noted that number values and mathematical constraints play a significant role in supporting numerical reasoning. Wu et al. (2021b) proposed several number value features to enhance encoder and Qin et al. (2021) designed new auxiliary tasks to enhance neural MWP solvers. Compared with their work, we first introduce pretraining language model (PLM) and concentrate on representation learning to resolve numerical understanding challenges. Second, regarding the usage of pre-training techniques for MWP solving, Shen et al. (2021) introduced BART-based (Lewis et al., 2020) MWP solver and incorporated specialized multi-task training for obtaining more effective pre-training Seq2Seq models for MWP. Compared with them, our work focuses on the number representation learning issue of MWP and achieves a more flexible pre-training representation module for MWP solving, which can be applied in various MWP related tasks other than solution generation.
Numeracy-aware Pre-training Models. Number representation has been recognized as one of the main issues in word representation learning. Existing methods make use of value, exponent, subword and character methods (Thawani et al., 2021) to obtain number representations for explicit number values. These methods are known to be less effective in extrapolation cases like testing with numbers not appearing in the training corpus.
Previous related works (Andor et al., 2019; Wallace et al., 2019; Geva et al., 2020) mainly focus on shallow numerical reasoning tasks shown in DROP dataset (Dua et al., 2019), which usually serves as a benchmark for evaluating numerical machine reading comprehension (Num-MRC) performance. Compared with MWP solving, Num-MRC's main focus is laid on extracting answer spans from a paragraph, which are more fault-tolerant with no needs to predict number tokens. Besides, their solution generation tasks only contain simple computations like addition/subtraction and there are only integers in DROP. More exactly, several research efforts have been made to deal with this kind of math-related reading comprehension task by synthesizing new training examples (Geva et al., 2020),
Figure 2: The overall architecture of our BERT-based MWP solver. Our method enables the solver to learn from unlabeled, incompletely labeled and fully labeled MwPs by different pre-training tasks.
incorporating special modules considering the numerical operation (Andor et al., 2019) and designing specific tokenization strategies (Zhang et al., 2020c). Since MWP solving requires further consideration of the complex composition of reasoning logics in MWP text, the symbolic placeholder is more effective in MWP solving. Thus, instead of dealing with explicit number values, our work focuses on improving representation for symbolic placeholders by injecting numerical properties in a probabilistic way.
The input to an MWP solver is a textual description, we denote it as  $W$  with length  $m$ , thus  $W = \{w_{1}, w_{2}, \ldots, w_{m}\}$ . We also define a subset  $W_{q}$  of  $W$  which contains all the quantities appeared in  $W$ . The output is an equation showing how to get the final answer. We denote it as  $A$  with length  $n$ , where  $A = \{a_{1}, a_{2}, \ldots, a_{n}\}$ . The vocabulary of  $A$  contains three parts, namely  $V_{op}$ ,  $V_{num}$  and  $V_{cons}$ .  $V_{op}$  is the vocabulary for all operators, i.e.,  $+, -, \times, \div$  and  $^{\wedge}$ . The vocabulary of quantities  $V_{num}$  is constructed by number mapping (Wang et al., 2017), which transforms quantities in different MwPs into a unified representation. More specifically,  $V_{num}$  does not contain the actual value of quantities appeared in  $W$ , and those quantities are denoted as  $\{n_{1}, n_{2}, \ldots, n_{k}\}$ , where  $n_{i}$  means the  $i$ -th number from  $W$  and  $k$  is the maximum number of quantities in  $V_{num}$  in order to fix the size of it.  $V_{cons}$  contains necessary constant values
e.g.,  $\pi$
3.2 PLM Encoder
Our PLM encoder maps the problem description  $W$  into a representation matrix  $Z \in \mathbb{R}^{m*h}$  where  $h$  is the dimension of the hidden feature.
$$
Z = \operatorname {e n c o d e r} (W). \tag {1}
$$
The representation vector corresponding to each word in  $Z$  will be used in the decoding process for generating the solution.
An overview of pre-training objectives and our model architecture is shown in Figure 2. In general, pre-training objectives are designed to inject contextual priori and numerical properties as soft constraints for representation learning. They are categorized into three types given provided training signals, i.e., self-supervised, weakly-supervised, and fully-supervised.
3.3 Self-supervised Objectives
In this part, we only consider input text descriptions for each example. Also, these objectives can alleviate the costs of collecting MWP corpus by constructing supervision signals without solution answers and equations.
Masked Language Modeling. We follow Devlin et al. (2019) and introduce masked language modeling (MLM) for basic contextual representation modeling. Specially, we apply masks on  $10\%$  of tokens, randomly replace  $10\%$  of tokens with other tokens and keep  $80\%$  of tokens unchanged. Later, the manipulated sentence is utilized to reconstruct the original sentence.
Number Counting. Another pre-training objective is to predict the amount of numbers that appeared in MWP description. The amount of a number corresponds with the cardinality of variable sets. This also reflects the basic understanding about the difficulty of an MWP and can act as a key contextual MWP number understanding feature. Here, we introduce a regression objective with the following formulated loss function:
$$
L _ {N u m C o u n t} = M S E (F F N (\bar {Z}), | W _ {q} |), \tag {2}
$$
where  $MSE$  stands for mean-squared-error and FFN stands for the feed-forward network which is made up of two fully-connected (FC) layers and one ReLU activation. We build a two-FC-layer block for each pre-training task (except MLM) and
discard them during fine-tuning.  $\bar{Z}\in \mathbb{R}^h$  is the mean vector of  $Z$  and represents the encoder's overall understanding of a single MwP text description.  $|W_{q}|$  is the number of quantities shown in the problem description  $W$ .
Number Type Grounding. This objective aims at linking contextual number representations with corresponding number types to tell the difference between discrete and continuous concepts/entities. For numerical reasoning in MWP solving, we only need to handle whole numbers as well as non-integer numbers (decimal, fraction and percentage). Ideas here are that whole numbers usually associate with discrete entities (for example, desks, chairs and seats) while non-integer numbers often connect with continuous concepts (for example, proportions, rate, velocity). Besides, comparisons among whole numbers got different issues compared with rational numbers. Therefore, we propose a classification objective to predict if a number is a whole number or non-integer number:
$$
L _ {N T G r o u n d} = \sum_ {i: W _ {i} \in W _ {q}} C E (F F N (Z _ {i}), y _ {i}), \tag {3}
$$
where  $W_{q}$  contains all the numbers that appeared in  $W$ , and  $CE$  is the cross-entropy loss for binary classification. Here,  $i$  is the index when  $W_{i}$  is a quantity,  $Z_{i}$  is the corresponding representation vector, and  $y_{i}$  is a binary label indicating if  $W_{i}$  is a whole number or non-integer number.
3.4 Weakly-supervised Objectives
Given both text descriptions of MwPs and corresponding answers, we can model dependencies among answer number and numbers in text descriptions so that contextual representation perceive the existence of the target variable number that does not appear in the text descriptions. In detail, we design 3 novel pre-training objectives specializing in value-annotated MwPs to improve number representation in our MWP-BERT.
Answer Type Prediction. Determining the type of answer number can provide us discrete/continuous nature of target entity/concept. Thus, we want to predict type (whole/non-integer) of the answer value given global representations of an MWP (embedded in  $Z$ ):
$$
L _ {A T P r e d} = C E \left(F F N (\bar {Z}), y _ {s}\right), \tag {4}
$$
where  $y_{s}$  is the ground truth label indicating the type of answer number.
Context-Answer Type Comparison. Besides the global context feature, an MWP-BERT also needs to associate context numbers and answer number (the target number does not explicitly appear in the text). Thus, another objective is proposed to predict if the quantities appeared in the MWP text fall into the same category as the answer (i.e. they are all whole or non-integer):
$$
L _ {C A T C o m p} = \sum_ {i: W _ {i} \in W _ {q}} C E \left(F F N \left(Z _ {i}\right), y _ {i} \oplus y _ {s}\right), \tag {5}
$$
where  $\oplus$  stands for the exclusive-or operator between two binary labels to check if they are the same, the label of a quantity  $y_{i}$  and the label of the solution value  $y_{s}$ .
Number Magnitude Comparison. Beyond type, the magnitude of a number serves as the foundation of numerical reasoning. By associating magnitudes evaluation with contextual representation, the model can get a better perception about variance over key reasoning cues like time, size, intensity and speed. Let  $\dot{y}_i$  indicate if the current quantity  $W_i$  is greater than the solution value or not. Moreover, the loss function is formulated as:
$$
L _ {\text {N u m M C o m p}} = \sum_ {i: W _ {i} \in W _ {q}} C E \left(\operatorname {F F N} \left(Z _ {i}\right), \dot {y} _ {i}\right). \tag {6}
$$
3.5 Fully-supervised Objectives
Given both equations and answers for MwPs, we can design fully-supervised training tasks to associate number representation with reasoning flows (solution equation). Mathematical equations are known to be binary tree structures with operators on root nodes and numbers on leaf nodes. The motivation is to encourage models to learn structure-aware number representations that encode the information on how to make combinations over atomic operators and numbers. We incorporate two pretraining objectives based on the solution equation tree.
Operation Prediction. The first one is a quantity-pair relation prediction task that focuses on the local feature of the equation tree. The goal is to predict the operator between two quantity nodes in the solution tree. This is in fact a classification task with 5 potential targets, i.e.,  $+, -, \times, \div$  and  $\wedge$ . The loss function of this task is:
$$
L _ {O P r e d} = \sum_ {i, j} C E (F F N ([ Z _ {i}; Z _ {j} ]), o p (W _ {i}, W _ {j})), \tag {7}
$$
where  $i,j$  are two indexes that satisfy  $W_{i},W_{j}\in W_{q}$  and  $[Z_i;Z_j]\in \mathbb{R}^{2h}$  is the concatenation of  $Z_{i}$  and  $Z_{j}$  for the quantity  $W_{i}$  and  $W_{j}$ .  $op(W_{i},W_{j})$  returns the operator between  $W_{i}$  and  $W_{j}$ .
Tree Distance Prediction. Another pre-training objective is to incorporate the global structure of the equation tree in a quantitative way. Inspired by Hewitt and Manning (2019), we consider the depth of each number and operator on the corresponding binary equation tree to be the key structure priori. Thus, we design another fully-supervised objective to utilize this information. More exactly, given the representation of two number nodes in an equation tree, this is a regression problem that predicts the distance (difference of their depth) between them. The loss is formulated as:
$$
L _ {T P r e d} = \sum_ {i, j} M S E (F F N ([ Z _ {i}; Z _ {j} ]), d (W _ {i}, W _ {j})), \tag {8}
$$
where  $d(W_i, W_j)$  is the distance between quantity  $W_i$  and  $W_j$  in the solution tree.
The final pre-training objective is the summation of Equation 2~8 and the masked language model.
3.6 Fine-Tuning
To investigate the mathematical understanding ability of our pre-training MWP-BERT, we evaluate our model of MWP solving, quantity tagging and 7 probing tasks. Moreover, we not only use BERT but also RoBERTa (Liu et al., 2019b) as the backbone of our encoder to show the adaptiveness of proposed method.
We present several empirical results with octopus evaluation settings (Bender and Koller, 2020) to prove the superiority of MWP-BERT and MWP-RoBERTa solver. In section 4.1, we illustrate the application of both solvers in the generation scenario, MWP solving, by fine-tuning them with a specific decoder (Xie and Sun, 2019). Next, we present MWP probing tasks in section 4.3 to evaluate the capability of MWP-BERT and MWP-RoBERTa on "understanding" or "capturing the meanings" of MWPs. Finally, results and analysis about ablation study are illustrated in section 4.4.
Implementation Details. We pre-train our model on 4 NVIDIA Tesla V100 graphic cards and fine-tune on 1 card. The model was pre-trained for 50 epochs (2 days) and fine-tuned for 80
epochs (1 day) with a batch size of 32. Adam optimizer (Kingma and Ba, 2014) is applied with an initial learning rate of 5e-5, which would be halved every 30 epochs. Dropout rate of 0.5 is set during training to prevent over-fitting. During testing, we use 5-beam search to get reasonable solutions. The hyper-parameters setting of our BERT and RoBERTa is 12 layers of depth, 12 heads of attention and 768 dimensions of hidden features. For the Chinese pre-training model, we use an upgrade patch of Chinese BERT and RoBERTa which are pre-trained with the whole word masking  $(\mathsf{WWM})^{1}$  (Cui et al., 2020). For the English pre-training models, we use the official source on this website<sup>2</sup>. Our code and data have been open-sourced on Github<sup>3</sup>.
4.1 MWP Solving
Experiment Settings and Datasets. Given a textual description of a mathematical problem, which contains several known variables, MWP solving targets at getting the correct answer for the corresponding question. A solver is expected to be able to predict an equation that can exactly reach the answer value. We conduct experiment based on these benchmarks, Math23k (Wang et al., 2017), MathQA (Amini et al., 2019) and Ape-210k (Zhao et al., 2020). Since there exist many noisy examples in Ape-210k, e.g., examples without equation annotations or answer values, we re-organize Ape-210k to Ape-clean and Ape-unsolvable, where the training set of Ape-clean and the whole Ape-unsolvable are used for pre-training. For the English MWP, we use the training set of MathQA (Amini et al., 2019) to perform pre-training. For the implementation of our solver, MWP-BERT is adapted as an encoder to generate intermediate MWP representation for the tree-based decoder in Xie and Sun (2019).
4.2 The Ape-clean Dataset
Ape210k is a recently released large MwPs dataset, including 210,488 problems. The problems in Ape210k are more diverse and difficult than those in Math23k as shown in 1. Not only the stronger requirement of common-sense knowledge for getting solutions, but also the missing of ground-truth solution equations or answers, will take extra obstacles for MwP solving. Among all these cases,