SlowGuess/NAACL_2022_OCR huggingface dataset:
Model	Number	Date	Others	All
EM	F1	EM	F1	EM	F1	EM	F1
GenBERT	73.05	75.21	52.44	56.37	70.17	74.53	68.75	72.30
+ Entity-Number	73.37	76.24	52.41	56.33	71.02	75.98	69.10	72.61
+ Type-Number	73.09	75.30	53.60	56.59	70.15	74.34	68.82	72.31
+ Decoder-Number	73.10	75.37	52.40	55.98	70.33	74.82	68.78	72.34
NC-Mask	74.16	76.89	53.27	56.32	71.24	75.10	69.17	72.65
- Decoder-Number	73.74	76.33	53.25	56.31	71.24	75.10	69.09	72.40
Table 2: Evaluation on the DROP evaluation dataset. The first half shows the results per answer type for independently applying the three channels of NC-Mask, in order to analyze the individual contextual knowledge influence on the model's numerical reasoning performance. NC-Mask provides the results of all the channels combined, with an extra ablation result on the decoder-number channel.
Model	Number	Date	Others	All
EM	F1	EM	F1	EM	F1	EM	F1
NC-Mask	74.16	76.89	53.27	56.32	71.24	75.10	69.17	72.65
+ DICE (w/ NC-Mask)	75.03	77.70	53.42	56.45	72.10	75.63	69.93	73.55
+ DICE (w/o NC-Mask)	73.36	76.12	52.42	55.98	70.17	74.40	68.87	72.38
NC-BERT	75.09	77.72	53.41	56.31	72.12	75.68	69.96	73.59
NC-Mask and DICE-loss together. The implications of this outcome are: (i) numeracy dilution occurs when overriding parametric knowledge in the embeddings and can be addressed by adopting the regularization term, and (ii) the regularization and our masking strategy are complementary.
5.3 Head Replacement with NC-Mask
Table 3: Results of NC-Mask in coordination with DICE-loss to alleviate the numeracy dilution issue. NC-BERT incorporates the LsDF generation scheme.
Heads and Layers	EM	F1
Original	68.75	72.30
All Layers & Heads	37.53	40.77
Last Layer	All-Heads	68.83	72.35
Single-Head	69.12	72.88
Odd-Heads	68.76	72.19
Even-Heads	68.74	72.11
Table 4: Results after applying NC-Mask to the encoder. "All" refers to applying NC-Mask to every head and layer in the encoder. Single, Odd and Even-Heads refer to the last layer head masking with NC-Mask.
To determine which part of the encoder should our masking scheme be applied to, we thoroughly investigated the effects of NC-Mask on the heads and layers of the encoder. The result is shown in Ta-
ble 4, where "Original" is the original performance of our baseline under our setting.
We first applied NC-Mask to all the heads and layers (All) of the encoder. Our initial assumption was that if a model could simply learn to attend to useful, number-related context, the model would easily leverage such information for more accurate numerical reasoning. However, the performance drops drastically (-31.53 in F1), suggesting that neglecting the roles of heads and layers of the encoder is detrimental to numerical reasoning over text. This result is also in correspondence with previous works (Rogers et al., 2020; Jo and Myaeng, 2020), where the roles of heads and layers are already defined during the pre-training of the model.
Based on the result, we then applied NC-Mask to the heads of the encoder's last layer since the last hidden states are the ones used by the decoder and span extraction module to generate the answer. The result implies that imbuing NC-Mask to induce the learning of number-related context is important. However, as the results of "Single-Head" and the other heads in the last layer suggest that considering each head's role is critical to model acquiring such contextual knowledge beneficial to its numerical reasoning capability.
Number of Heads (k)	EM	F1
No New Layer	68.75	72.30
0	68.54	72.11
1	68.68	72.27
2	69.22	72.87
4	69.21	72.86
6	69.52	73.00
12	69.96	73.59
5.4 The Extra Layer for Masked Interaction
Applying NC-Mask to the last layer replaces one of the pre-trained heads. Here, we see that the loss of encoder's original linguistic capability is inevitable to acquire the additional numerical reasoning skill. Thus, to retain the original textual reasoning capability of the encoder and provide additional numerical-contextual knowledge, we add an extra layer with NC-Mask on top of the encoder. Here, we experiment with the number of heads  $(k)$  in an effort to figure out the optimal number of heads the model needs to acquire the most accurate numerical reasoning capability.
Our initial assumption was twofold: (i) just like in the last layer's case, a single head would result in the best performance, or (ii) there would be some "sweet spot" in which the model shows the optimal performance. On the contrary to our expectations, the results shown in Table 5 display an entirely different pattern from the ones in Table 4.
From the result of  $k = 0$ , we confirm that the improvement in model performance does not arise from the naïve addition of an extra self-attention layer, since it rather exhibits a drop in performance. With the incremental addition of the NC-Mask, we evidence a proportional increase in performance, which reaches the peak when the maximum number of heads ( $k = 12$ ) is masked. Our interpretation to such disparity in masking patterns is that the extra self-attention layer is not a pre-trained layer unlike the last layer of the encoder; meaning, while the heads of the last layer have pre-defined roles, the extra layer is a randomly initialized parameter with no head-wise roles defined. This implies that the NC-Masked layer acted as a single, giant "head" that induced useful numerical-contextual knowl
Table 5: Model performance after applying NC-Mask on  $k$  different heads of the layer on top of the encoder of NC-BERT.  $k = 0$  means a vanilla Transformer encoder layer without the NC-Mask.
Model	Entity	Type	Other
Numbers (Original)	21.61	35.83	42.56
Numbers (NC-BERT)	47.14	52.86	-
Table 6: Changes (in %) in the attention patterns from the numbers in passages to different relation types within the encoder. NC-BERT exhibits amplified attention magnitude in all the three relation types compared to its original counterpart.
Recall@K	GenBERT	NC-BERT
1	0.0049	0.0081
2	0.0103	0.0171
5	0.0295	0.0434
10	0.0689	0.0840
20	0.1492	0.1513
50	0.2691	0.2340
Table 7: Evaluation result of the Recall@  $K$  from the number to entity and type-defining words. The entity and type-defining tokens are treated as the ground truth labels, and the  $K$  represents the number of top-  $K$  retrieved tokens using the corresponding number embeddings.
edge into the encoder, which in turn improved the model's numerical reasoning capability.
5.5 Interaction Between Numbers and Contextual Knowledge
In Table 6, we provide an attention pattern analysis to show that with our masking scheme, the number representations readily acquire the relevant contextual knowledge. The numbers shown in the table represent the attention scores from the number tokens in the passage to entity tokens, number-type tokens, and all the other tokens in the passage. For Numbers (Original), we have normalized the attention scores over all the heads in the last layer of the encoder, whereas the Numbers (NC-BERT) list the normalized attention scores over all the heads of the extra layer with the NC-Mask, where every head ( $k = 12$ ) is masked as in our final model architecture. The results testify that the masking scheme successfully increases the amount of interaction between numbers and their related contextual knowledge (e.g., entity and type-defining words), which in turn led to the improved reasoning accuracy of our model.
On top of the increased interaction between numbers and related contextual knowledge, in Table 7,
Question&Answer	Passage	GenBERT	NC-BERT
Q: How many active military personnel and reserve are in the Croatian Armed Forces?	The total number of active military personnel in the Croatian Armed Forces stands at 14,506 and 6,000 reserves working in various service branches of the armed forces. In May 2016, Armed Forces had ...	14,506 + ? = 14,506	14,506 + 6,000 = 20,506
Q: How many more Macedonians were there compared to Albanians according to the 2002 census?	Skopje, as the Republic of Macedonia as a whole, is characterised by a large ethnic diversity ... According to the 2002 census, Macedonians were the largest ethnic group in Skopje, with 338,358 inhabitants, or 66.75% of the population. Then came Albanians with 103,891 inhabitants (20.49%), ...	338,358 - ? = 328,337	338,358 - 103,891 = 234,467
Table 8: The case study of the DROP dataset. The table juxtaposes the model prediction cases for GenBERT and NC-BERT to qualitatively illustrate the differences between the reasoning of the two models. The calculated answers indicate that NC-BERT is able to relate numbers with their corresponding entities and type-words, leading to improved numerical reasoning accuracy.
we provide the cosine similarity retrieval result of entity and type-defining output representations using the number embeddings in terms of  $\mathrm{Recall}@\bar{K}$ . With the two kinds of contextual knowledge as the ground truth tokens, we calculate the recall for the top- $K$  tokens retrieved by the number embeddings in the encoder output. The result in Table 7 shows the increased similarity between the numbers and the corresponding entity and type representations, suggesting that our masking scheme successfully increases the similarity and thus the interaction between useful contextual knowledge and the numbers in text.
For an intuitive understanding of the effect of our proposed method, we provide a case study on the DROP dataset in Table 8. As the cases suggest, our model is better able to relate numbers in text with their pertinent entities (e.g., Croatian Armed Forces) and type words (e.g., reserves), which serve the model with useful entity-number and type-number information that lead to improved numerical reasoning capability of our model. In contrast, the baseline fails to relate the numbers needed for the calculation, which results in wrong answers.
This work proposes a novel attention masking scheme, NC-Mask, to relieve question answering (QA) models of the language model (LM) encoder's over-reliance on parametric knowledge and improve the numerical reasoning accuracy and robustness. Our analyses and empirical results provide strong evidence that BERT, a commonly used encoder in QA models, needs to employ the extra
attention channels to leverage the number-related contextual knowledge for robust numerical reasoning instead of entirely relying on the inherent self-attention mechanism. By additionally adopting a numeracy-inducing regularization term, our work also shows that the proposed masking scheme and regularization are complementary, and retaining numeracy is essential for accurate numerical calculation. Future efforts should focus on increasing the scale of models with the masking scheme, since masked, attention-constrained layers appear to more positively contribute to model's reasoning capability than the addition of fully self-attentive layers.
This work was supported by Institute for Information communications Technology Planning Evaluation(IITP) grant funded by the Korea government(MSIT) (No. 2013-2-00131, Development of Knowledge Evolutionary WiseQA Platform Technology for Human Knowledge Augmented Services) and (No. 2021-0-02155, Developing Context/Number Embedding based Numerical Reasoning).
Hadeel Al-Negheimish, Pranava Madhyastha, and Alessandra Russo. 2021. Numerical reasoning in machine reading comprehension tasks: are we there yet? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9643-9649, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Daniel Andor, Luheng He, Kenton Lee, and Emily Pittler. 2019. Giving bert a calculator: Finding operations
and arguments with reading comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5949-5954.
Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? . In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 610-623, New York, NY, USA. Association for Computing Machinery.
Kunlong Chen, Weidi Xu, Xingyi Cheng, Zou Xiaochuan, Yuyu Zhang, Le Song, Taifeng Wang, Yuan Qi, and Wei Chu. 2020a. Question directed graph attention network for numerical reasoning over text. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6759-6768.
Xinyun Chen, Chen Liang, Adams Wei Yu, Denny Zhou, Dawn Song, and Quoc V. Le. 2020b. Neural symbolic reader: Scalable integration of distributed and symbolic representations for reading comprehension. In International Conference on Learning Representations.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378.
Mor Geva, Ankit Gupta, and Jonathan Berant. 2020. Injecting numerical reasoning skills into language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 946-958.
Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, and Matt Gardner. 2019. Neural module networks for reasoning over text. In International Conference on Learning Representations.
Jae-young Jo and Sung-Hyon Myaeng. 2020. Roles and utilization of attention heads in transformer-based neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3404-3417, Online. Association for Computational Linguistics.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. Billion-scale similarity search with gpus. arXiv preprint arXiv:1702.08734.
Jeonghwan Kim, Giwon Hong, Kyung-min Kim, Junmo Kang, and Sung-Hyon Myaeng. 2021. Have you seen that number? investigating extrapolation in question answering models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7031-7037, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. *Hurdles to progress in long-form question answering.* In *Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, pages 4940–4957, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7052-7063, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Kuntal Kumar Pal and Chitta Baral. 2021. Investigating numeracy learning ability of a text-to-text transfer model. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3095-3101, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations.
Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, and Zhiyuan Liu. 2019. Numnet: Machine reading comprehension with numerical reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2474-2484.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in bertology: What we know about how bert works. Transactions of the Association for Computational Linguistics, 8:842-866.
Amrita Saha, Shafiq Joty, and Steven CH Hoi. 2021. Weakly supervised neuro-symbolic module networks for numerical reasoning. arXiv preprint arXiv:2101.11802.
Dhanasekar Sundararaman, Shijing Si, Vivek Subramanian, Guoyin Wang, Devamanyu Hazarika, and Lawrence Carin. 2020. Methods for numeracy-preserving word embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4742-4753, Online. Association for Computational Linguistics.
Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020. oLMpics-on what language model pre-training captures. Transactions of the Association for Computational Linguistics, 8:743-758.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, page 6000-6010, Red Hook, NY, USA. Curran Associates Inc.
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. Do NLP models know numbers? probing numeracy in embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5307-5315, Hong Kong, China. Association for Computational Linguistics.
A.1 Hyperparameters of the Model
In this section, we provide the important hyperparameters used during the training of our model.
Hyperparameter	GenBERT	NC-BERT
Batch size	16	16
Hidden size	768	768
Max. Sequence length	512	512
Learning rate	3e-5	3e-5
Optimizer	AdamW	AdamW
Seed	42	42
Approx. runtime	50 hrs	51.5 hrs
Table 9: Hyperparameters of the two models of this work. The hyperparameters are set so the two models are on an equal footing for fair comparison.
A.2 LsDF: Least Significant Digit First Generation
Our final model, NC-BERT, employs the least significant digit first (LSDF) generation scheme during the fine-tuning of the model. The intuition behind the LsDF generation is simple considering the basic rules of addition. When the digits of
lower significance add up with their sum greater than or equal to 10, then a carry of 1 occurs which is then passed on to the subsequent significant digit for addition. Such sequence of carries can only happen when the values of lower significance add up in advance. LSDF incorporates this elementary arithmetic rule to the generation of number answers, simply by reversing the order of the number sequence answer (e.g.,  $127 \rightarrow 721$ ). This digit-position reversing acts as an additional schematic alteration to our encoder-decoder generative architecture, which turns out to benefit the numerical reasoning capability of the model slightly. Our results also imply that considering the intuitive arithmetic calculation steps is important in numerical reasoning.