SlowGuess/ACL_2020_OCR huggingface dataset:
For pre-training on ND, we generated 1M examples for training and 10K for validation. For TD, we generated 2.5M examples for training and 10K for validation. For both synthetic datasets, we used the GENBERT model loss,  $\mathbf{L}_{\mathrm{model}}$ , from §3. To ensure that the model does not lose its language understanding abilities, we employ a multi-task setup, and include a standard masked LM objective from BERT. Specifically, given a masked token sequence  $\langle \mathbf{m} \rangle$ , we compute the contextualized representations,  $\mathbf{L}_{\mathrm{enc}}$  and pass them through a feedforward network  $\mathbf{F}\mathbf{F}_{\mathrm{mlm}}$ . For each masked index  $i$ , it outputs the probability  $p(a_i \mid i, \langle \mathbf{m} \rangle)$  of the original token  $a_i$ . The MLM loss is computed as
$$
\mathbf {L} _ {\mathrm {m l m}} (\langle \mathbf {m} \rangle) = \operatorname {m e a n} _ {i \in \text {m a s k e d}} - \log (p (a _ {i} \mid i, \langle \mathbf {m} \rangle)).
$$
Details about the MLM data are in  $\S A.3$
During training, we sample mini-batches from the respective datasets, and minimize the weighted sum of the losses. Concretely, while pre-training on ND and TD, we sample mini-batches  $X_{\mathrm{ND}}$ ,  $X_{\mathrm{TD}}$  and  $X_{\mathrm{MLM}}$  and optimize the objective
$$
\mathbf {L} _ {\text {m o d e l}} \left(X _ {\mathrm {N D}}\right) + \mathbf {L} _ {\text {m o d e l}} \left(X _ {\mathrm {T D}}\right) + \lambda \cdot \mathbf {L} _ {\text {m l m}} \left(X _ {\text {M L M}}\right).
$$
Figure 4: Progression of eval accuracy (EM) of GENBERT, for different pre-training settings listed in §5.1.
We now evaluate our two pre-training steps and their applicability for numerical reasoning tasks. We consider the following variants, aiming to investigate the contributions of ND and TD, the importance of MLM loss, and techniques like DT and RS. In all cases, we initialize GENBERT with BERT-base-uncased, use DT and RS, and include the MLM loss, except where noted:
GENBERT+ND: trained on numerical data.
GENBERT+ND-LM: trained on ND without the additional MLM loss.
GENBERT+ND-LM-DT: trained on ND using wordpiece tokenization, without the MLM loss.
GENBERT+ND-LM-RS: trained on ND without MLM loss and random shift (RS).
GENBERT+TD: trained on textual data (TD).
GENBERT $_{+\mathrm{ND} + \mathrm{TD}}$ : GENBERT $_{+\mathrm{ND}}$  trained on both ND and TD.
5.1 Pre-training Performance
We first ask whether the pre-training procedure allows GENBERT to absorb the intended numerical skills. We observe that across various settings (ND, TD, ND+TD), GENBERT consistently achieves more than  $96\%$  accuracy in predicting the correct solution for both ND and TD. Thus, we conclude that indeed a pre-trained LM can learn the designed skills from generated data.
Figure 4 shows the learning curves of GENBERT for the different variants. Note that in ND-LM-DT the model does not learn to solve the numerical data task. This demonstrates the utility of using DT over conventional wordpieces. The lower sample complexity in the case of  $\mathrm{ND + TD}$  compared to the only-TD can be attributed to the fact that ND and TD share some numeric skills and hence a model already trained on ND converges faster on TD compared to GENBERT.
5.2 Numerical Reasoning Performance
After successfully injecting GENBERT with numeric skills, we test GENBERT guided by the following questions:
(a) Are the injected skills robust and generalize to NRoT datasets like DROP?
(b) Are the new skills learned at the expense of the model's ability to understand language?
(c) Can the pre-trained weights be used with architectures other than GENBERT?
For (a), we fine-tune GENBERT on DROP and further evaluate on MWP in a zero-shot setup. For (b), we evaluate GENBERT on a RC task that does not involve numerical reasoning, namely, SQUAD (Rajpurkar et al., 2016). For (c), we use GENBERT encoder as a drop-in replacement for BERT on two other architectures.
Results on DROP We report results of GENBERT initialized by BERT-base and leave pretraining a larger model for future work. We compare GENBERT to MTMSN (Hu et al., 2019) initialized with BERT-base, as MTMSN initialized with BERT-large is a state-of-the-art model on DROP. $^{1}$
Table 4 presents fine-tuning results on DROP. Without pre-training, GENBERT performs poorly compared to current state of the art models like MTMSN, reporting an EM of only 46.1. Pretraining on each of the numerical data (ND) and textual data (TD) improves performance dramatically to 64.7 EM and 64.4 EM, respectively. Moreover, pre-training on both ND and TD leads to a performance of 68.8 EM, on par with MTMSN's 68.2 EM. This demonstrates that the skills that GENBERT learns from ND and TD are complementary. In addition, the lower performance of  $\mathrm{GENBERT}_{+\mathrm{ND - LM}}$  and  $\mathrm{GENBERT}_{+\mathrm{ND - LM - RS}}$
Development	Test
EM	F1	EM	F1
GENBERT	46.1	49.3	-	-
GENBERT+ND-LM-RS	61.5	65.4	-	-
GENBERT+ND-LM	63.8	67.2	-	-
GENBERT+ND	64.7	68.2	-	-
GENBERT+TD	64.4	67.8	-	-
GENBERT+ND+TD	68.8	72.3	68.6	72.4
NABERT+	63.0	66.0	61.6	65.1
MTMSNBASE	68.2	72.8	-	-
Table 4: Performance of GENBERT and comparable models on the development and test sets of DROP.
number	span	date	spans
GENBERT	42.3	67.3	47.5	21.1
GENBERT+ND	70.5	71.0	54.5	24.2
GENBERT+TD	69.2	72.6	55.2	22.0
GENBERT+ND+TD	75.2	74.5	56.4	24.2
NABERT+	67.8	69.2	39.8	22.4
MTMSNBASE	75.0	71.3	44.2	53.4
Table 5:  ${\mathrm{F}}_{1}$  scores on DROP development per answer type.
shows the importance of including the MLM loss and the utility of RS for short inputs.
Breaking down performance by answer type (Table 5) highlights several points. First, pre-training on ND and TD improves performance mostly due to number answer types, as expected. Second,  $\mathrm{GENBERT}_{+\mathrm{ND} + \mathrm{TD}}$  outperforms  $\mathrm{MTMSN_{BASE}}$  on questions whose answer is a span. We argue a probable cause for this are span questions that require performing a numerical computation internally, as explained in §2. Third,  $\mathrm{MTMSN_{BASE}}$  substantially outperforms GENBERT on questions whose answer is a list of non-contiguous spans. This is expected, as MTMSN has a specialized head and procedure for handling such questions, while build on a simpler and more standard RC architecture.
Generalization to MWP (zero-shot) The MAWPS repository is a collection of math word problem (MWP) datasets (Koncel-Kedziorski et al., 2016). To test the models on skills they were trained on, we picked datasets with addition and subtraction problems, and filtered out examples with other operations (e.g., multiplication and division). All models that were fine-tuned on DROP were evaluated in a zero-shot setup on 395 examples from ADDSUB (Hosseini et al., 2014), 321 from SOP (Roy et al., 2015), and 305 from SEQ (Koncel-Kedziorski et al., 2015).
Results are shown in Table 6. Overall,  $\mathrm{GENBERT}_{+\mathrm{ND} + \mathrm{TD}}$  dramatically improves performance compared to GENBERT.  $\mathrm{GENBERT}_{+\mathrm{ND}}$  performs much better than  $\mathrm{GENBERT}_{+\mathrm{TD}}$ , demonstrating the utility of ND when the context is short.
<sup>1</sup> Per ACL policy, we compare to models that were made public 3 months prior to submission.
ADDSUB	SOP	SEQ
GENBERT	2	1.2	1.3
GENBERT+ND	22.8	26.5	23
GENBERT+TD	10.4	21.5	12.1
GENBERT+ND+TD	22.8	28.3	22.3
NABERT+	19.2	19.6	17.4
MTMSNBASE	32.2	28	32.5
Figure 5: Breakdown of model accuracy (EM) by the number of terms in the arithmetic expression, for the MWP datasets ADDSUB, SOP and SEQ.
Last, MTMSN outperforms  $\mathrm{GENBERT}_{+\mathrm{ND} + \mathrm{TD}}$ . However, MTMSN uses a specialized architecture for addition and subtraction, suitable when calculations are done outside of the model. GENBERT, on the other hand, is a general-purpose generative model, that can also return span answers when the computation is done internally.
Next, we break down performance by the number of terms in the arithmetic expression (Figure 5). The plot shows that all models struggle to generalize to more complex problems, and completely fail when the calculation involves more than 3 terms. Interestingly, the drop in performance of  $\mathrm{GENBERT}_{+\mathrm{ND} + \mathrm{TD}}$  between 2 and 3 terms is significantly smaller than that of  $\mathrm{GENBERT}_{+\mathrm{ND}}$  and  $\mathrm{GENBERT}_{+\mathrm{TD}}$ . This suggests that both ND and TD are useful for improving robustness.
Error analysis To understand the limitations of our method, we analyze the errors of  $\mathrm{GENBERT}_{+\mathrm{ND} + \mathrm{TD}}$  on the development set of DROP, excluding questions with a multi-span answer which are not supported by the model. We sample 100 random examples for which  $\mathrm{GENBERT}_{+\mathrm{ND} + \mathrm{TD}}$  fails to predict the correct answer and manually analyze the types of questions and mistakes done by the model.
We find that in almost half of the cases (43%), the example requires reasoning skills that are either not covered by the pre-training tasks (e.g. sorting), or not numerical. Another common case (23%) is inaccurate predictions, such as spans that are too
Table 6: EM on MWP datasets.
EM	F1
BERT	81.1	88.6
GENBERT+ND-LM	78.1	85.8
GENBERT+ND	80.7	88.1
GENBERT+TD	80.7	88.2
GENBERT+ND+TD	81.3	88.6
Table 7: Performance on SQuAD v1 development set. Scores for BERT are using wordpiece tokenization.
long and numbers with partial digit match to the gold answer. We note that many of these errors can be addressed by extending the pre-training tasks to cover additional numerical skills and a larger number range. We leave such extensions for future work. Further details and example failure cases are provided in §A.5.
5.3 Reading Comprehension Performance
Having shown that our models successfully learned to perform NRoT, we investigate if this improvement comes at the expense of performance on RC datasets. We initialize the RC model from Devlin et al. (2019) with GENBERT weights (encoder only) and fine-tune it on SQUAD v1. As shown in Table 7, the performance of  $\mathrm{GENBERT}_{+\mathrm{ND} + \mathrm{TD}}$  is almost identical to the original BERT. Moreover,  $\mathrm{GENBERT}_{+\mathrm{ND - LM}}$  reported a loss of 3 EM points highlighting the importance of using the MLM loss.
5.4 GENBERT With Other Architectures
To further establish the utility of GENBERT, we used the weights of  $\mathrm{GENBERT}_{+\mathrm{ND} + \mathrm{TD}}$  to initialize the encoder of NABERT+ and MS-TAG, a recent multi-span tagging model of Efrat et al. (2019). Fine-tuning on DROP shows an improvement of  $\sim 2$  EM points compared to the originally reported performance:  $63.0\rightarrow 65.1$  EM for NABERT+, and  $67.3\to 69.3$  EM for MS-TAG. This shows that GENBERT can be used as a drop-in replacement for BERT, when numerical reasoning is needed.
To summarize, we have empirically shown that one can inject numerical reasoning skills into a pre-trained LM, resulting in good performance on DROP, generalization to MWP, while maintaining high performance on standard RC datasets. Moreover, the resulting weights can be used for initializing numerical reasoning models.
Most NRoT models designed for DROP are extractive QA models augmented with specialized modules (§2). Two recent work (Andor et al., 2019;
Chen et al., 2020) take a more symbolic approach and output a symbolic program augmented with operations over text. In our work, numerical computations are latent and performed internally by the model.
A related line of work has been analyzing the mathematical reasoning abilities of neural models over text (Wallace et al., 2019; Rozen et al., 2019; Ravichander et al., 2019), and on arithmetic problems (Saxton et al., 2019; Amini et al., 2019; Lample and Charton, 2020).
Designing pre-training tasks to teach LMs additional skills has been applied by Huang et al. (2019), who designed cross-lingual pre-training tasks to teach better mappings between languages, and Lee et al. (2019), who introduced the Inverse Cloze Task to pre-train an information retriever.
Large pre-trained LMs lack high-level skills such as numerical reasoning. Consequently, current models that perform numerical reasoning over a pretrained LM resorted to customized modules with limited flexibility. In this work, we propose a general method for injecting additional skills into LMs, assuming automatic data generation is possible. We apply our approach to the task of numerical reasoning over text, using a general-purpose model called GENBERT, and a simple framework for generating large amounts of synthetic examples. Our experiments demonstrate the effectiveness of our method, showing that GENBERT successfully learns the numerical skills, and performs on par with state-of-the-art NROT models of the same size.
We thank Daniel Andor and Thang Luong for helpful discussions, and Shimi Salant for constructive suggestions. This research was partially supported by The Israel Science Foundation grant 942/16, The Yandex Initiative for Machine Learning, and the European Research Council (ERC) under the European Union Horizons 2020 research and innovation programme (grant ERC DELPHI 802800).
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In North American Association for Com
putational Linguistics (NAACL), pages 2357-2367, Minneapolis, Minnesota.
Daniel Andor, Emily Pittler, Kenton Lee, and Luheng He. 2019. Giving bert a calculator: Finding operations and arguments with reading comprehension. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
Xinyun Chen, Chen Liang, Adams Wei Yu, Denny Zhou, Dawn Song, and Quoc V. Le. 2020. Neural symbolic reader: Scalable integration of distributed and symbolic representations for reading comprehension. In International Conference on Learning Representations (ICLR).
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In *North American Association for Computational Linguistics (NAACL)*, pages 4171–4186, Minneapolis, Minnesota.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In North American Association for Computational Linguistics (NAACL).
Avia Efrat, Elad Segal, and Mor Shoham. 2019. Tag-based multi-span extraction in reading comprehension. arXiv preprint arXiv:1909.13375.
Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415.
John Hewitt and Christopher D Manning. 2019. A structural probe for finding syntax in word representations. In North American Association for Computational Linguistics (NAACL), pages 4129-4138.
M. J. Hosseini, H. Hajishirzi, O. Etzioni, and N. Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. In Empirical Methods in Natural Language Processing (EMNLP), pages 523-533.
Minghao Hu, Yuxing Peng, Zhen Huang, and Dongsheng Li. 2019. A multi-type multi-span network for reading comprehension that requires discrete reasoning. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou. 2019. Unicoder: A universal language encoder by pretraining with multiple cross-lingual tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Aishwarya Kamath and Rajarshi Das. 2019. A survey on semantic parsing. Automated Knowledge Base Construction (AKBC).
Jambay Kinley and Raymond Lin. 2019. NABERT+: Improving Numerical Reasoning in Reading Comprehension.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-3526.
Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equations. In Transactions of the Association for Computational Linguistics (TACL), volume 3, pages 585-597. MIT Press.
Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. MAwPS: A math word problem repository. In Association for Computational Linguistics (ACL), pages 1152-1157, San Diego, California.
Guillaume Lample and FranÅgois Charton. 2020. Deep learning for symbolic mathematics. In International Conference on Learning Representations (ICLR).
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Association for Computational Linguistics (ACL).
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer. 2018. Deep contextualized word representations. In North American Association for Computational Linguistics (NAACL).
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller, and Sebastian Riedel. 2019. Language models as knowledge bases? In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP).
Abhilasha Ravichander, Aakanksha Naik, Carolyn Rose, and Eduard Hovy. 2019. EQUATE: A benchmark evaluation framework for quantitative reasoning in natural language inference. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 349-361, Hong Kong, China. Association for Computational Linguistics.
S. Roy, T. Vieira, and D. Roth. 2015. Reasoning about quantities in natural language. Transactions of the Association for Computational Linguistics (TACL), 1.
Subhro Roy and Dan Roth. 2015. Solving general arithmetic word problems. In Empirical Methods in Natural Language Processing (EMNLP), pages 1743-1752.
Ohad Rozen, Vered Shwartz, Roee Aharoni, and Ido Dagan. 2019. Diversify your datasets: Analyzing generalization via controlled variance in adversarial datasets. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 196-205, Hong Kong, China. Association for Computational Linguistics.
David Saxton, Edward Grefenstette, Felix Hill, and Pushmeet Kohli. 2019. Analysing mathematical reasoning abilities of neural models. In International Conference on Learning Representations (ICLR).
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems (NIPS), pages 5998-6008.
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. Do nlp models know numbers? probing numeracy in embeddings. In Empirical Methods in Natural Language Processing (EMNLP).
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
A.1 Synthetic Numerical Data Generation
We briefly describe the numerical templates, providing the details missing from Table 2. In all cases, integers are sampled from  $\{0,\dots ,20K\}$ , and split into disjoint train and development sets to assure generalization.
- signed float combination : Random signed combinations of up to 4 floats. Floats are sampled from the set of floats with two decimal places.
- min/max/avg: We sample 2-4 floats and apply a min, max, avg, operation by sampling a word from the set  $\mathcal{O} = \{\text{"longest", "last", "highest", "largest", "most", "shortest", "first", "smallest", "lowest", "least", "average"}\}$ .
- arg max, arg min: We sample word-float pairs, where words are sampled from  $\mathcal{W}$ : words in the NLTK Words Corpus² having at most 2 word-pieces, and floats are sampled as above.
- date max/min: Same as min/max/avg above, but for dates. Dates are sampled from  $\mathcal{D}$ : the set of dates until Sep 2019. The operator word is sampled from  $\mathcal{DSUP} = \{\text{"last", "latest", "most recent", "youngest", "first", "earliest", "oldest", "least recent"}\}$  and mapped to min or max.
- date difference : This teaches our model to perform date arithmetic in days, months and years.
- percentage : We teach our model to perform  $100 - x$  operations in the context of percentages. Given a number of arguments, we sample a percentage split using a flat Dirichlet distribution.
A.2 Synthetic Textual Data Generation
A.2.1 Sentence template extraction
To extract sentence templates, we abstract the text of math word problems from the corpus published by Hosseini et al. (2014). Going over examples, we split the problem text into sentences<sup>3</sup>, and abstract the tokens of each sentence independently. Tokens are abstracted according to the framework into numbers (NUM), verb categories (VERB), entities (ENT), containers (CONT) and attributes (ATTR).
To have a better control over the generation process, we extend the framework of Hosseini et al. (2014) to support two container types - agent (AGT) and environment (ENV). Agents are objects which actively collect and drop entities, for example a person or an organization. Environments are passive containers, such as places or time periods. In addi
tion, we introduce two-level containers to express inclusion relation between containers. For instance, if 3 submarines anchor near the city of Devonport, then they also anchor near the country of England.
The 12 most common extracted sentence templates, which were used for generating synthetic data, are provided in Table 8.
A.2.2 Template instantiation
Sentence templates are instantiated with a small vocabulary, that map categories into words. In this work, we construct two domain-specific small-world vocabularies, about history and the National Football League. The vocabularies are available in a json format in https://github.com/ag1988/injecting_numeracy.
A.2.3 Question templates
The 13 question templates for 7 different skills are provided in Table 9.
A.3 Data for Masked LM task
For creating the training data for the masked LM task (§ 5.1) we took the pages from English Wikipedia whose lowercased title containing a string in {season, economy, demographics, conquest, war, battle, uprising, rebellion, insurgency, conflict, crisis, revolution, military history, mutiny, regiment, revolt, geography, raids, insurrection, invasion, feud, siege, campaign, expedition, succession, coup, university}. This resulted in 156K full pages. In the remaining pages, paras with  $< 15$  numbers were discarded. Pages were tokenized using DT (§ 3) and chunked into 512-token sequences. Following Devlin et al. (2019), each token was masked with probability 0.15 with no more than 65 masks per sample. This gave us 0.7M samples.
A.4 Experimental Setup
For all our experiments, we used an older version of Hugging Face's Transformers library (Wolf et al., 2019) and provide our training hyperparameters in Table 10.
A.5 GENBERT+ND+TD Error Analysis
Table 11 summarizes the main failure types of GEN-BERT $_{+\mathrm{ND} + \mathrm{TD}}$  on 100 random examples from the development set of DROP, excluding questions with a multi-span answer.
2 https://www.nltk.org/
3 Using the Spacy library http://spacy.io/