SlowGuess/NAACL_2022_OCR huggingface dataset:
Jeonghwan Kim Junmo Kang* Kyung-min Kim*
Giwon Hong* Sung-Hyon Myaeng
Daejeon, Republic of Korea
{jeonghwankim123, jumon.kang, kimdarwin,
giwon.hong, myaeng}@kaist.ac.kr
Numerical reasoning over text is a challenging subtask in question answering (QA) that requires both the understanding of texts and numbers. However, existing language models in these numerical reasoning QA models tend to overly rely on the pre-existing parametric knowledge at inference time, which commonly causes hallucination in interpreting numbers. Our work proposes a novel attention masked reasoning model, the NC-BERT, that learns to leverage the number-related contextual knowledge to alleviate the over-reliance on parametric knowledge and enhance the numerical reasoning capabilities of the QA model. The empirical results suggest that understanding of numbers in their context by reducing the parametric knowledge influence, and refining numerical information in the number embeddings lead to improved numerical reasoning accuracy and performance in DROP, a numerical QA dataset.
Understanding numbers in text is critical when dealing with numerical reasoning problems over text. Most previous works (Ran et al., 2019; Andor et al., 2019; Chen et al., 2020b; Gupta et al., 2019; Chen et al., 2020a; Geva et al., 2020; Saha et al., 2021) on such numerical reasoning over text have shown substantial amounts of performance gain in numerical question answering (QA) tasks such as DROP (Dua et al., 2019). While these models display stellar performance, previous studies that evaluate model's numerical reasoning robustness (Talmor et al., 2020; Kim et al., 2021; Al-Negheimish et al., 2021) suggest that these QA models that typically depend on large language models (LM) such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) suffer from a limitation: the disregard of context information at inference time for numerical reasoning.
Figure 1: Cases that testify the over-reliance on parametric knowledge by language models like BERT during numerical reasoning. These attest to the language models' lack of ability to properly understand numbers in their given context.
This issue is exemplified in Figure 1 showing a numerical reasoning question (Talmor et al., 2020), where LMs fail to decouple number values from their types; a number is understood as in the LM, not as the value to be interpreted in the local context. For example, in the first case of Figure 1, the model misinterprets the numbers "80s" and "90s" as YEAR-type from the LM, instead of AGE-type obtainable from the given local context, leading to an incorrect answer for the "older" relation. Had the model leveraged the context of the passage instead of pre-existing knowledge reflected in the LM's parameters, the model should have correctly predicted the answer. Such knowledge is commonly referred to as parametric knowledge, which is learned from the training instances that the model has encountered prior to the inference process as in the above example. Previous works (Krishna et al., 2021; Bender et al., 2021; Al-Negheimish et al., 2021) testify that such propensity to rely on previously learned knowledge instead of looking at the present context information frequently haunts large LMs. As such, we hypothesize that the capability to override the parametric knowledge captured in the number embeddings with the relevant context
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1811 - 1821
tual knowledge is crucial to properly understand the numbers at inference time, and thus perform accurate and robust numerical reasoning.
To override the detrimental influence of parametric knowledge on model's numerical reasoning capability, our work proposes a novel attention masking scheme. This Numerical-Contextual attention mask (NC-Mask) explicitly induces the number-related flow of contextual information into the number embeddings to enable the model to properly interpret the numbers according to the context given, and thereby improve the model's numerical reasoning capability. This attention mask is predicated upon the following two intuitions: (i) numbers are always related to entities in the same sentence and (ii) a number type (e.g., YEAR, AGE, QUANTITY) is defined by its surrounding words. While such entity-number and type-number interactions rely on the self-attention mechanism of the Transformer architecture (Vaswani et al., 2017), the local relations to be captured from the context are not always implicitly captured by the LM's self-attention. Our NC-Mask is intended to consolidate such relations.
On the other hand, this "overwriting" of parametric knowledge in the number embeddings can cause the characteristic numeracy information (e.g., magnitude) in the embeddings to be diluted. Previous works (Wallace et al., 2019; Sundararaman et al., 2020; Pal and Baral, 2021; Kim et al., 2021) show that LMs possess, to a certain degree, the notion of numeracy in their parameters. For example, the information that number 2 is less than or equal to  $9$  ( $2 \leq 9$ ) is contained within the embeddings of those numbers. In order to avoid losing such valuable numeracy information, we adopt the DICE regularization (Sundararaman et al., 2020), a magnitude-inducing regularizer to instill relative magnitude hierarchy into the number representations and thus replenish the diluted numeracy in the embeddings for numerical reasoning.
The empirical results of this work suggest that our attention masking strategy improves the numerical reasoning capability of the QA model. In Section 5, we also provide a detailed analysis on the optimal application of our masking strategy on the different layers and heads of the QA model's encoder. Our results also imply that our attention masking allows extra layers that leverage the knowledge instilled by the masked channels to be added, leading to the scaling up of the model without pre
training the extra layers.
2.1 Parametric Knowledge vs. Contextual Knowledge
Downstream NLP tasks often require the use of two disparate sources of knowledge: parametric and contextual knowledge. Previous works (Longpre et al., 2021) reveal that conflicts between the two types of knowledge occur from over-reliance on parametric knowledge, which is exacerbated by the significant overlap between the passage-question pairs in the training and validation sets (Krishna et al., 2021; Al-Negheimish et al., 2021). For the task of numerical QA, a related work (Talmor et al., 2020) reveals that models fail to understand numbers in the given passage because they rely on the parametric knowledge (i.e., memorization) within the pre-trained number embeddings. Such lack of contextual understanding of numbers prevents these models from properly interpreting numbers and thus inhibit effective numerical reasoning over text.
2.2 Numeracy in Language Models
Recent studies on the numeracy of large LMs reveal that number embeddings constructed by either the non-contextual embedding methods (Sundararaman et al., 2020) or large language models (Wallace et al., 2019; Talmor et al., 2020; Sundararaman et al., 2020; Kim et al., 2021) possess, to a certain degree, a prior notion of numeracy such as magnitude. However, the numeracy in the LMs are neither deterministic nor accurate (Wallace et al., 2019) like the scalar numbers. These representations, furthermore, require additional refinement measures to induce additional numeracy (Sundararaman et al., 2020). Such lack of numeracy can lead to a few of the following problems: (i) confusing numbers of similar magnitude (Talmor et al., 2020), and (ii) calculating wrong numerical answers (Geva et al., 2020). Since our masking scheme can bring about such issues due to diluted numeracy in the number embeddings, we adopt a numeracy-inducing regularization term from Sundararaman et al. (2020) to alleviate this problem and improve the overall accuracy in numerical calculations.
In this work, we propose an attention masked question answering (QA) model, namely the Numerical-Contextual BERT (NC-BERT), that learns to rely on the contextual knowledge and exhibits improved numerical reasoning capability in QA. First, we construct a tripartite attention mask (i.e., NC-Mask) over a self-attention layer on top of the encoder to direct the flow of necessary contextual information to the number embeddings, so that number-related contextual knowledge such as entity-number and type-number relations are effectively leveraged during model training. We then adopt a complementary numeracy-inducing regularizer to counteract the numeracy dilution issue caused by our masking strategy and to further enhance the accuracy of numerical calculation.
We first provide a qualitative analysis on an LM's number token embeddings to identify and reveal the type of pre-existing parametric knowledge in them, which induces model dependence on parametric instead of contextual knowledge. To this end, we first sample all the number embeddings from a pre-trained BERT-base model (Devlin et al., 2019). Then, we use an off-the-shelf similarity search tool, FAISS (Johnson et al., 2017), to calculate the cosine similarity between each number embedding and every other non-number token embeddings in BERT's vocabulary. The next step is to sample tokens with top- $k$  ( $k = 5$ ) similarity scores for each number token. Note that we exclude other numeral tokens (e.g., "two", "71"), non-alphabetical tokens, and special tokens such as [PAD] and [SEP] as seen in Table 1. However, we retain number-related tokens such as "15th" and "1990s" as indicators of DATE-related parametric knowledge within the embeddings.
In Table 1, we can see that the number embeddings are deemed to contain semantics about date, month or other quantity information. Furthermore, numbers like "2018" already contain DATE-related information, which does not seem out-of-context considering that "2018" is more often than not used in such contexts. However, this knowledge influences model decisions negatively when questions use "2018" for a non-date scenario, resulting in an error as in Figure 1. Numbers like "50" and "114" that are seldom used in DATE context also hold such DATE information, further suggesting
What is in Number Embeddings
2018	currently, october, 1990s, july, 19th
50	1950s, various, significant, many, substantial
114	12th, 11th, 14th, 15th, 13th
2	ii, several, various, 4th, iii
11	11th, 12th, 10th, 8th, 13th
Table 1: The leftmost column contains the number tokens from BERT (bert-base-uncased)'s vocabulary and the rightmost column contains the top-5 similarity-scored tokens that correspond to each number token.
that relying entirely on such parametric knowledge induces models to make a wrong prediction. It is evident, therefore, we need to pay attention to the distinction and interplay between the parametric knowledge identified in this analysis and context-specific semantics in QA models during numerical reasoning.
3.2 Base Model Architecture
A QA model for numerical reasoning needs to conduct both the span extraction from text and numerical answer generation. This steered our work to leverage GenBERT (Geva et al., 2020) as our baseline. Given a question  $(\mathbf{q})$  with  $m$  tokens and a passage  $(\mathbf{p})$  with  $n$  tokens, we construct an input sequence that consists of the BERT special tokens as follows:  $<[\mathrm{CLS}] q_1, q_2, \ldots, q_m, [\mathrm{SEP}], p_1, p_2, \ldots, p_n>$ . Then, we produce the contextualized representations,  $\mathbf{M}$ , for each token after passing the input sequence through a pre-trained LM encoder of choice (in this case, BERT).
$$
\mathbf {M} = E n c o d e r (\mathbf {q}, \mathbf {p}) \tag {1}
$$
The representation  $\mathbf{M}$  is then passed on to two different kinds of prediction modules for answer prediction; namely, span extraction and decoder modules. The span extraction module,  $\mathbf{H}_{\text{span}}$ , calculates the probabilities of start and end spans, whereas the decoder module,  $\mathbf{H}_{\text{dec}}$ , generates answers that are not found within the passage but can only be calculated by numerical reasoning such as addition and counting.
$$
\mathbf {H} _ {s p a n}: (\hat {y} _ {s t a r t}, \hat {y} _ {e n d}) = \underset {s <   = e} {\arg \max } P (\mathbf {M} _ {s}) P (\mathbf {M} _ {e}) \quad (2)
$$
$$
\mathbf {H} _ {d e c}: \hat {y} _ {i} = \underset {i} {\arg \max } P \left(y _ {i} \mid y _ {1}, y _ {2}, \dots , y _ {i - 1}, \mathbf {M}\right) \tag {3}
$$
Figure 2: A visual representation of Numerical-Contextual BERT (NC-BERT). The green layer represents the extra layer with NC-Mask on top of the encoder. The blue-highlighted path represents the digits that appear in the passage, along with the entity-number channel that is highlighted in blue in the NC-Mask layer. The decoder-number channel is presented in the upper half of the figure. Here, we omit the type-number channel for ease of understanding.
In the above equation,  $\mathbf{M}_s$  and  $\mathbf{M}_e$  denote the start and end token representations, respectively. We then train the model by minimizing the loss,  $L_{ans}$ , which is a marginal probability over the output of two modules as follows:
$$
L _ {a n s} = - \log \left(\mathbf {p} _ {\mathbf {d e c}} \cdot \prod_ {s = 1} ^ {S} p \left(y _ {s}\right) + \sum_ {\mathbf {h} \in \mathbf {q}, \mathbf {c}} \mathbf {p} _ {\mathbf {h}} \cdot \sum_ {(i, j) \in T} p _ {h} (i, j)\right) \tag {4}
$$
where  $\mathbf{p}_{\mathrm{dec}}$  and  $\mathbf{p_h}$  are the module type probabilities from a single layer feed-forward network (i.e., the type module).  $S$  is the length of the answer sequence generated by the decoder, and  $T$  is the set of all possible answer spans from the passage. We omit the conditionals for brevity.
3.3 NC-BERT for Numerical Reasoning
We describe NC-BERT, an encoder-decoder model designed for numerical reasoning, with the intuition and mechanism behind the three parts of the NC-Mask scheme. Also explained in this section is the rationale for the numeracy-inducing regularization term to deal with the numeracy dilution issue caused by the attention masking scheme. We fine-tune the model with our mask to enable the model to use the mask as a medium to effectively
aggregate numerical-contextual knowledge from texts.
3.3.1 NC-Mask
We construct an attention mask that allows the number-related contextual information to be channeled to the number embeddings. Two main intuitions are: (i) numbers are bound to the entities in the same sentence, and (ii) the words surrounding a number define its type. In order to reflect the intuitions, we construct two types of attention masks referred to as entity-number and type-number channels, which attempt to leverage the number-related input context and adjust the influence of parametric knowledge in the number embeddings. In addition, we devise the third channel, the decoder-number channel for two reasons: (i) the decoder needs only to "calculate" number sequence answers and (ii) the non-essential passage tokens act as noise during numerical calculation of the decoder.
Entity-Number Channel To construct the entity-number channel, we first extract every entity<sup>1</sup> and digit in each sentence. For each sentence, we construct an entity index set,  $E$ , and a dictionary with the digit indices as keys and  $E$  as values.  $E'$  (where  $E' \subseteq E$ ) is assigned as a value to a digit
<sup>1</sup>Using Stanford Stanza toolkit for NLP (Qi et al., 2020)
index if and only if the corresponding digit and  $E^{\prime}$  belong in the same sentence. The dictionary is then used to construct the attention mask,  $A_{E}$ , as follows:
$$
\alpha = \operatorname {s o f t m a x} \left(A _ {E} \odot \frac {Q K ^ {T}}{\sqrt {d _ {k}}}\right) V \tag {5}
$$
where  $\alpha$  is the normalized attention score and  $A_{E}$  is the attention mask that drowns out the tokens that are irrelevant to interpreting numbers in the text, namely, the non-entity tokens. To elaborate, when the query,  $q_{i}$ , of the  $i^{th}$  number token attends to all the  $m + n$  other sequence of tokens in the text during self-attention,  $A_{E}$  leaves the attention scores from the  $i^{th}$  number token to the entities in the same sentence unchanged, while zeroing out all the other attention scores to preclude their corresponding values  $(v\in V)$  from being added to the subsequent-layer representation of the  $i^{th}$  number token.
Type-Number Channel Type-number channel is constructed in a similar fashion as in Equation 5 by creating another attention mask,  $A_W$ . We first define a window of size  $k$  for every number. Within each window, we select the  $m$  (where  $m/2 <= k$ ) immediately neighboring words and construct another dictionary with the digit indices as keys and their neighboring word indices as values. Similar to  $A_E$ ,  $A_W$  is constructed to block out all the non-essential, noisy interactions from the number embeddings to irrelevant tokens in the text, while retaining the attention to the  $m$  immediately neighboring words that define the number type.
Decoder-Number Channel In a regular encoder-decoder Transformer architecture, the decoder uses its query,  $Q_{dec}$ , to attend over the value embeddings,  $V_{enc}$ , of the encoder's last hidden states (Vaswani et al., 2017). In this work, the decoder employs a new type of source attention mask,  $A_{src}$  along with the above two masks, that confines the decoder's query to attend only over the numbers and the question tokens of the encoder's last hidden states (Equation 6).
$$
\gamma = \operatorname {s o f t m a x} \left(A _ {\text {s r c}} \odot \frac {Q _ {\text {d e c}} K _ {\text {e n c}} ^ {T}}{\sqrt {d _ {k}}}\right) V _ {\text {e n c}} \tag {6}
$$
where  $\gamma$  is the source attention score. Since the number embedding is constructed to contain the number-related contextual knowledge, the decoder can learn to attend to the numbers by utilizing such knowledge within the number embeddings and perform calculations.
Figure 2 shows the overall framework, NC-BERT, with the NC-Mask applied on top of the encoder and decoder-number channel selectively attending to the last hidden state representations of the number tokens. The details of where the NC-Mask is applied in the encoder and the reason thereof is elaborated in Section 5.4
3.3.2 Numeracy-Inducing Regularization
Figure 3: A visual depiction of how the DICE-loss induces the relative magnitude hierarchy among the digit embeddings. The leftmost arrow depicts the embedding of 0 and the rightmost the embedding of 9.
Overwriting the parametric knowledge with NCMask, however, can cause pre-existing numeracy characteristic like the magnitude of a number, to be erased from the number representations. To alleviate such dilution of numeracy and to revive the magnitude characteristic of numbers in the representations, we adopt the numeracy-inducing regularization (DICE-loss in short) term (Sundararaman et al., 2020). It samples two random digits,  $a$  and  $b$ , from the given input text and their corresponding hidden states,  $\mathbf{v_a}$  and  $\mathbf{v_b}$ , from the last layer of the encoder. Then, it calculates the difference between the scalar distance of  $a$  and  $b$  and the cosine distance,  $d_{cos}$ , of their corresponding last layer representations as follows.
$$
L _ {n u m} = \left\| 2 \frac {| a - b |}{| a | + | b |} - d _ {c o s} \left(\mathbf {v} _ {\mathbf {a}}, \mathbf {v} _ {\mathbf {b}}\right) \right\| _ {2} \tag {7}
$$
DICE-loss (Figure 3) acts as an effective regularizer to induce the relative magnitude relation among the digits, thereby adjusting model parameters to generate contextualized number embeddings reflecting such hierarchy of magnitude. With the DICE-loss, the final expression for the objective ends up as follows:  $L = L_{ans} + L_{num}$ .
To validate the proposed approach, we establish the two following research questions:
Q1. "Does number-related contextual knowledge help improve numerical reasoning?"
Q2. "Does compensating for the diluted numeracy give more accurate numerical calculation?"
Q1 is assessed by the channel-wise application of the NC-Mask on our model encoder. By evaluating each channel and its effect on the model's reasoning capability, we reveal how the number-related contextual knowledge influence the baseline's numerical reasoning. We also determine the optimal placement of the NC-Mask by exploring where in the encoder should the mask be applied for better numerical reasoning. Q2 is evaluated by applying DICE-loss and thereby replenishing the diluted numeracy of the number embeddings. In addition, we experiment with the least significant digit first (LSDF) generation scheme on our encoder-decoder architecture. The goal is to explore the idea of taking the "carry" into account to bring about an additional benefit to the overall numerical reasoning capability of our model (details in the Appendix).
DROP (Dua et al., 2019) is a numerical reasoning over text dataset for QA models. This evaluation dataset consists of a total of 9,536 question-answer pairs with respect to 582 passages. The answer types are largely: Number, Date and Others, where Others refer to span type answers.
For the baseline, we use GenBERT (Geva et al., 2020), a Transformer encoder-decoder model initialized with BERT-base parameters pre-trained with simple arithmetic and textual number reasoning tasks. On top of the baseline, we empirically evaluate the effectiveness of (i) NC-Mask (ii) the numeracy-inducing regularization (DICE-loss), along with the LSDF generation technique for additional reasoning enhancement. We do not incorporate other state-of-the-art models like QDGAT (Chen et al., 2020a) and NumNet (Ran et al., 2019), since they disregard the numeracy understanding part (Wallace et al., 2019) by simply employing specialized heads that learn to assign  $\{-1,0, + 1\}$  on numbers for summation, and they do not actually perform implicit calculation to derive numerical answers by delegating the calculation part to a symbolic calculator.
4.2.1 Implementation Details
The model is based on GenBERT and is trained using RTX3090 NVIDIA GPU. With the training batch size of 16, the hidden size of 768, the learning rate of 3e-5 and an Adam optimizer with a linear warm up of 0.1. The rest of the hyperparameters are in the Appendix.
In this section, we explain the results of our NC-Mask scheme and complementary numeracy regularizer by comparing the per answer-type Exact Match (EM) and F1 scores. The results also include head- and layer-wise probing done to determine the optimal masking position in the model.
5.1 Leveraging Contextual Knowledge
As in Table 2, the addition of NC-Mask leads to noticeable performance improvements over the baseline. The Entity-Number channel proves to be the largest benefactor to the model's enhanced numerical reasoning capability. Such result can be interpreted from the fact that numbers now share high semantic similarity with the entities in question, which in turn improves the model's numerical reasoning by incorporating those numbers during the calculation. The Type-Number channel, in contrast, turns out to contribute most to the Date-type questions; the result is likely caused by the increased interaction between the Date-related tokens and numbers. For the Decoder-Number channel, through the ablation study in Table 2, we prove that the channel is a necessary component of the NC-Mask scheme, given the performance degradation in both the Number and Date type questions in its absence.
5.2 Counteracting Numeracy Dilution
With NC-Mask amplifying the influence of contextual knowledge in number embeddings, we now deal with the numeracy dilution issue. The results in Table 3 show substantial improvement in numerical reasoning performance. We also test DICE-loss after removing NC-Mask to empirically prove that the dilution of numeracy by our masking strategy indeed adversely affects model performance and requires the regularization. When adding the regularizer to the baseline alone (DICE (w/o NC-Mask)), we see a drop in performance, meaning that employing the regularizer is ineffective considering the pre-existing numeracy. On the contrary, we evidence a major increase in performance by applying