SlowGuess/NAACL_2021_OCR huggingface dataset:
Discrete Latent Exponent (DExp) is another potentially multimodal distribution (Spokoyny and Berg-Kirkpatrick, 2020) where the model parameterizes a multinomial distribution for the exponent (similar to MCC) and uses it to sample an exponent  $e$ , which then acts as a latent variable for emitting the mean  $\mu$  of a Gaussian (standard deviation fixed at 0.05). This Gaussian is finally used to sample the output number  $z \sim N(\mu, 0.05)$ .
GMM Another attempt to circumvent the unimodal Gaussians or point estimates is to learn a Gaussian mixture model. Spithourakis and Riedel (2018) learn a mixture of  $K$  Gaussians by pretraining their means  $(\mu_i)$  and variances  $(\sigma_i^2)$  over the training corpus with Expectation Maximization algorithms, while the mixing weights  $\pi_i$  are derived from the model. Next, to sample a single number from the GMM probability mass function  $q(u) = \sum_{i=1}^{K} \pi_i N(u; \mu_i; \sigma_i)$ , the authors first sample the precision (number of decimal places) from yet another Gaussian and use that to discretize the probability mass function into equal sized bins, over which the probabilities are summed. If the sampled precision is, say 2, then the probability of emitting a number 3.14 is given by  $\int_{3.135}^{3.145} q(u) du$ . This likelihood estimate is used to train a causal language model.
Spokoyny and Berg-Kirkpatrick (2020)'s GMM implementation is slightly different: it alters the last inference step by sampling directly from the mixture of Gaussians, as they did with Log Laplace, Flow Laplace, and DExp.
GMM-prototype by Jiang et al. (2020) similarly pretrains (with EM/hard-EM) the mean, the variances, but also the mixture weights  $\pi_{i}\mathrm{s}$  of a GMM over the training corpus. They then learn  $K$  prototype embeddings  $e_i\mathrm{s}$  corresponding to the  $K$  Gaussians. When encoding a new numeral  $n$ , its (input) embedding is calculated as:  $E(n) = \sum_{i=1}^{K} w_i \cdot e_i$ , where the weights are induced from the GMM:
$$
w _ {i} = P (Z = i | U = n) = \frac {\pi_ {i} N (n ; \mu_ {i} ; \sigma_ {i})}{\sum_ {j = 1} ^ {K} \pi_ {j} N (n ; \mu_ {j} ; \sigma_ {j})}
$$
Thus the difference between GMM and GMM-prototypes is that after fixing mean and standard deviations of the Gaussian mixtures, in GMM the model learns to predict the mixture weights  $\pi_{i}$  for each individual number prediction, whereas in GMM-prototype,  $\pi_{i}$ 's are frozen and the model learns prototype embeddings  $e_i$ 's. Note that proto
type embeddings are encoder-only. To decode numbers, the authors implement weight-sharing across input and output embeddings, similar to how word vectors are trained (Mikolov et al., 2013), i.e., finding out which of the numerals in the corpus has the closest embedding.
SOM-prototype GMM-prototype, in effect, merely use the mixture of Gaussians to infer prototypes and to get the weights  $w_{i}$ 's. Jiang et al. (2020) tried another variant by identifying prototype numerals with Self Organizing Maps (Kohonen, 1990) and by defining the weights as:  $w_{i} = |g(x_{i}) - g(n)|^{-1}$  where  $x_{i}$  is the  $i$ th prototype,  $n$  is the number to be encoded, and  $g$  is a log-based squashing function.
Having organized the landscape of numeracy tasks and methods, we now present some key results for each numeracy task in NLP from previously published experiments over a subset of the described number representations:
Abstract Probes Word Embeddings vastly outperform random embedding baselines on abstract probes such as numeration, magnitude comparison, and sorting (Wallace et al., 2019; Naik et al., 2019). DICE, Value and Log Value embeddings excel at these probes, which makes intuitive sense given that they explicitly encode the numbers' magnitude - although Value embeddings do not easily extrapolate to larger numbers, possibly due to instability in training. The best number encoders with respect to these probes were found to be DigitCNNs, and character-tokenized models, e.g., ELMo, in general outperform subword ones, e.g., BERT (Wallace et al., 2019).
Arithmetic GPT-3 (Brown et al., 2020) performs extremely well at zero shot simple arithmetic, as long as the number of digits in the operands are low. The tokenization scheme could be the cause for limited extrapolation, since language models get better at arithmetic when numbers are tokenized at the digit/character level (Nogueira et al., 2021; Wallace et al., 2019). For arithmetic word problems, state of the art solvers rely on predicting an equation, which is then filled in with specific numeric values from the question (Patel et al., 2021), altogether bypassing the need for encoding numbers into embeddings.
Masked Language Modelling Zhang et al. (2020) show that BERT pretrained over datasets
where numbers are in scientific notation (Num-BERT) converges to the same loss as BERT on masked language modelling objective, and scores nearly the same on GLUE language understanding benchmarks. For (causal) numeric language modelling, Spithourakis and Riedel (2018) show that Gaussian Mixture Models are the best decoders. For (masked) numeric language modelling, Spokoyny and Berg-Kirkpatrick (2020) show that modelling the mantissa in scientific notation may be an overkill, since exponent embeddings alone outperform DigitRNN-sci over financial news and scientific articles.
Measurement Estimation Zhang et al. (2020) train a regression probe to predict measurements of objects over the CLS embeddings of BERT/NumBERT. Given a template-lexicalized sentence such as "the dog is heavy," the model must predict the weight of a typical dog, against ground truth from the Distribution over Quantities dataset (Elazar et al., 2019). They find that NumBERT is a better text encoder than BERT for measurement estimation, the only difference between them being the notation used by the respective pretraining corpora. They also experiment with two number decoders: MCC (multi-class classification) and RGR (regression / Log Value embedding). MCC performs better when trying to predict Distributions over Quantities - perhaps due to the ground truth resembling the predicted gaussians - but not on VerbPhysics - where the ground truth is less noisy. Lastly, even static word embeddings like GloVe have been shown to contain enough knowledge of measurement estimates to contrast two objects, e.g., classifying whether a car is bigger/heavier/fasster than a ball (Goel et al., 2019).
Exact Facts BERT and RoBERTa capture limited numerical commonsense, evident over NunnerSense (Lin et al., 2020) sentences such as a tricycle has [MASK] wheels, with the answer choices limited to the integers 0-10. Results can be further improved by finetuning over a Wikipedia-extracted dataset of numeric information. Mishra et al. (2020) find commonsense question answering to be one of the hardest among their Numbergame challenge, using the NumNetv2 model (Ran et al., 2019) which is commonly used for DROP question answering. Both of these experiments evaluate on exact match metrics, hence it remains to be seen if representing approximate magnitudes yields benefit in modelling numeric facts.
5 Recommendations
Based on the above results, we now synthesize key insights into a set of directed takeaways to guide practitioners' design of number representations:
Rule of thumb for string-based methods? Scientific notation is superior to decimal notation (Zhang et al., 2020) since models can learn to attend mostly to the exponent embedding rather than the mantissa (Spokoyny and Berg-Kirkpatrick, 2020). Character level tokenization outperforms subword level (Nogueira et al., 2021; Wallace et al., 2019; Geva et al., 2020). Pooled representations (DigitRNN, DigitCNN) lack a controlled study with unpooled ones (NumBERT, GenBERT) which makes it hard to proclaim a winner among the two.
Rule of thumb for real-based methods? Log scale is preferred over linear scale (Zhang et al., 2020; Jiang et al., 2020; Wallace et al., 2019; Spokoyny and Berg-Kirkpatrick, 2020), which makes intuitive sense but lacks as rigorous a study as has been undertaken in the cognitive science community (Feigenson et al., 2004). Regarding discretization, Zhang et al. (2020) show that binning (dense cross entropy loss) works better than continuous value prediction (MAE loss) on datasets where ground truth distributions are available. Lastly, modeling continuous predictions is notoriously hard for large ranges (Wallace et al., 2019) but Spithourakis and Riedel (2018) offer a way of binning such distributions by picking a precision level.
Encoding vs Decoding numbers? In our simplified discussions above, we avoid differentiating between methods for encoding and decoding numbers. Value Embedding, for instance, can be used to encode numbers (projecting scalars onto vector space) as well as to decode numbers (collapsing a vector into a scalar). On the other hand, manually-designed encoders like DICE are not easily reversible into decoding methods. Even with reversible methods, the encoders and decoders must usually be independently parameterized, unlike the input and output word embeddings which often share weights (Press and Wolf, 2016). Prototype embeddings by Jiang et al. (2020) are an exception, which share input/output embeddings for a fixed vocabulary of numbers.
Can we mix-and-match multiple methods? Given the wide range of number representations, an obvious next step is to try an ensemble of embeddings. Spokoyny and Berg-Kirkpatrick (2020)
show that for encoding numbers, exponent embeddings added to DigitRNN (scientific notation) embeddings barely outperforms the exponent embeddings alone. Similar experiments with a mix of real and string methods are yet to be seen.
Which methods for which tasks? Based on our taxonomy of tasks in Table 1, abstract tasks are good early probes for the grounded ones, e.g., finetuning GenBERT (Geva et al., 2020) on simple arithmetic helps it do well on downstream question answering, and the high scores of DICE (Sundararaman et al., 2020) on numeration and magnitude comparison are an indicator of similar boosts on (numeric) language modelling. With respect to granularity, real-based methods work well for approximate tasks such as measurement estimation and language modeling (Zhang et al., 2020; Spokoyny and Berg-Kirkpatrick, 2020) but not for exact tasks like arithmetic word problems or commonsense. DigitRNNs are broad-purpose number encoders, whereas distribution modeling methods like DExp are effective at decoding numbers.
6 Vision for Unified Numeracy in NLP
Numeracy is a core system of human intelligence (Kinzler and Spelke, 2007). Teaching numeracy to students works best when taught holistically, while less effective teachers deal with areas of mathematics discretely (Askew and Askew, 1997). While the NLP community genuinely strives to improve language models' numeric skills, not all aspects of numeracy have been sufficiently targeted. It is evident from the sparsity in Table 2 that the community is far from achieving, a holistic solution to numeracy. In this section, we outline our vision for such a unified solution, in the form of three prerequisites to consider for numerical NLU:
Evaluation. The first step towards a holistic solution to numeracy requires a benchmark covering its different subtasks. Aggregated leaderboards in NLP like GLUE (Wang et al., 2018) and SuperGLUE (Wang et al., 2019) have incentivized research on natural language understanding, with scores categorized into semantic, syntactic, logical, and background knowledge.
An analogous leaderboard could be constructed to evaluate models on numeric reasoning tasks, categorized according to the skills evaluated, e.g., exact vs approximate granularity, or abstract vs grounded numeracy. Numbergame (Mishra et al., 2020) is one such aggregation focusing on exact
numeracy benchmarks, as evaluated by F1 and exact match scores in a reading comprehension setup. Both Numbergame and our own list of tasks (Section 2.2) are preliminary attempts at teasing apart the different aspects of numeracy. We encourage researchers to extend and refine such taxonomies.
A suite of numeracy tasks, matched with evaluations of their respective numerical skills, can enable testing model generalization from one skill to another. Some progress has already been made in this transfer learning setup, e.g., GenBERT (Geva et al., 2020), finetuned on a synthetic dataset of arithmetic problems, is found to score higher on DROP QA. Similarly, DICE (Sundararaman et al., 2020), optimized for numeration, improves score on Numeracy600K order-of-magnitude prediction task. Going forward, we need several such studies, ideally for each pair of tasks to see whether some numeracy skills help models generalize to others.
Design Principles. Number representations vary based on design trade-offs between inductive biases and data-driven variance. The default BERT setup, with subword tokenization and lookup embeddings, occupies the variance end of the spectrum, allowing freedom in representing numbers. Value embeddings and DICE encodings, on the other hand, are closer to the bias end of the spectrum, since the inductive bias of continuity on the number line constrains the learning space. It is important to identify where on the bias-variance scale any representation stands, for a fair comparison.
Following parallel work in cognitive science, the community could explore whether exact and approximate numeracy require two specialized modules (Feigenson et al., 2004) or could be handled with a single representation (Cordes et al., 2001).
Model designers must also make a choice on coverage: whether to target a broad or a narrow range of numbers to be represented. Multi-class classification (Zhang et al., 2020) over a fixed number of bins, restricts the range of numbers expressed, as do DICE embeddings (Sundararaman et al., 2020). Value embeddings are continuous and theoretically unrestricted, but must practically be capped for bug-free training. On the other hand, string-based representations could always fall back to subword/char-level token embeddings to represent not only floats but also irrational  $(\sqrt{2})$  and complex  $(1 + 2t)$  numbers. Roy et al. (2015) introduced the Quantity-Value Representation format to allow closed and open ranges alongside scalar point numbers.
Broader Impact. Numbers are ubiquitous in natural language and are easily identified, at least in numeral forms. But they are by no means the only class of ordered concepts required for natural language understanding. Successful number representations can inspire work on incorporating more continuous domains into natural language processing systems. For instance, gradable adjectives like good, great, amazing, etc. are arguably on some cardinal scale, which can be mapped using value embeddings or Gaussian mixture models (Sharp et al., 2018; de Marneffe et al., 2010). Days of the week (Mon-Sun) and months of an year (Jan-Dec) form periodic patterns which can be modeled with sinusoidal functions (Martinez et al., 2020).
Lastly, numeracy is essential for natural language understanding. Consider the sentence: "Programmers earn  $200,000 versus$ 100,000 for researchers." An intelligent agent with numeracy skills would identify that  $100k is half of$ 200k, that $100k possibly denotes annual salary, and infer that higher salaries lead to higher standards of living. In short, it was able to learn something about the two concepts programmers and researchers, by crossing the continuous semantic space of numbers! The agent could now make use of this knowledge in a number-free situation, e.g., the mask in "He could not afford a car for several years after earning a CS degree because she took a job as a [MASK]" might better be filled with the word researcher, than with programmer. A key goal of imparting numeracy to NLP models is to help them understand more about the world, using numbers.
This paper summarizes and contextualizes recent work on numeracy in NLP. We propose the first taxonomy of tasks and methods concerning text-centric numeric reasoning. We highlight key takeaways from the several experiments in literature, along with caveats and scope for confirming some of the observed trends. We present a case for lack of a holistic solution to numeracy in NLP, and put forward a set of aspects to consider when working towards one. We draw the following two major conclusions from our study: (1) the default subword segmentation with lookup embeddings used to represent words is clearly suboptimal for numbers (2) there are several unanswered research questions on the level of specificity, coverage, and inductive bias needed to holistically solve numeracy.
This work was funded by the Defense Advanced Research Projects Agency with award N660011924033. We would like to thank the countless suggestions we accumulated during preliminary presentations at MLSS 2020, WeCNLP 2020, and GSS 2020, as well as over email correspondences with Biplav Srivastava, Antoine Bosselut, and Harsh Agarwal. We would like to thank the anonymous NAACL 2021 reviewers (particularly #3) for pointing out blind spots in our submission, which we have tried our best to rectify.
This work revolves around the Hindu-Arabic Numeral system and English number words, which are not the only number systems still in use today. We encourage follow-up work to take these systems into consideration, on the lines of Johnson et al. (2020) and Nefedov (2020).
Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2357-2367, Minneapolis, Minnesota. Association for Computational Linguistics.
Daniel Andor, Luheng He, Kenton Lee, and Emily Pitler. 2019. Giving BERT a calculator: Finding operations and arguments with reading comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5947-5952, Hong Kong, China. Association for Computational Linguistics.
Mike Askew and Mike Askew. 1997. *Effective teachers of numeracy*. King's College London London.
Satwik Bhattachamishra, Arkil Patel, and Navin Goyal. 2020. On the computational power of transformers and its implications in sequence modeling. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 455-475, Online. Association for Computational Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss,
Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Sarah E Bullard, Deborah Fein, Mary Kay Gleeson, Nita Tischer, Robert L Mapou, and Edith Kaplan. 2004. The biber cognitive estimation test. Archives of clinical neuropsychology, 19(6):835-846.
Kushal Chawla, Gale Lucas, Jonathan Gratch, and Jonathan May. 2020. Bert in negotiations: Early prediction of buyer-seller negotiation outcomes. arXiv preprint arXiv:2004.02363.
Chung-Chi Chen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2020. Numclaim: Investor's fine-grained claim detection. In Proceedings of the 29th ACM International Conference on Information and Knowledge Management, CIKM '20, page 1973-1976, New York, NY, USA. Association for Computing Machinery.
Chung-Chi Chen, Hen-Hsen Huang, Yow-Ting Shiue, and Hsin-Hsi Chen. 2018. Numeral understanding in financial tweets for fine-grained crowd-based forecasting. In 2018 IEEE/WIC/ACM International Conference on Web Intelligence (WI), pages 136-143. IEEE.
Chung-Chi Chen, Hen-Hsen Huang, Hiroya Takamura, and Hsin-Hsi Chen. 2019. Numeracy-600K: Learning numeracy for detecting exaggerated information in market comments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6307-6313, Florence, Italy. Association for Computational Linguistics.
Peter Clark, Oren Etzioni, Daniel Khashabi, Tushar Khot, Bhavana Dalvi Mishra, Kyle Richardson, Ashish Sabharwal, Carissa Schoenick, Oyvind Tafjord, Niket Tandon, et al. 2019. From 'f'to'a' on the ny regents science exams: An overview of the aristo project. arXiv preprint arXiv:1909.01958.
Sara Cordes, Rochel Gelman, Charles R Gallistel, and John Whalen. 2001. Variability signatures distinguish verbal from nonverbal counting for both large and small numbers. Psychonomic bulletin & review, 8(4):698-707.
Marie-Catherine de Marneffe, Christopher D. Manning, and Christopher Potts. 2010. "was it good? it was provocative." learning the meaning of scalar adjectives. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 167-176, Uppsala, Sweden. Association for Computational Linguistics.
Stanislas Dehaene. 2011. The number sense: How the mind creates mathematics. OUP USA.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378, Minneapolis, Minnesota. Association for Computational Linguistics.
Abhijeet Dubey, Lakshya Kumar, Arpan Somani, Aditya Joshi, and Pushpak Bhattacharyya. 2019. "when numbers matter!": Detecting sarcasm in numerical portions of text. In Proceedings of the Tenth Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 72-80, Minneapolis, USA. Association for Computational Linguistics.
Yanai Elazar and Yoav Goldberg. 2019. Where's my head? Definition, data set, and models for numeric fused-head identification and resolution. Transactions of the Association for Computational Linguistics, 7:519-535.
Yanai Elazar, Abhijit Mahabal, Deepak Ramachandran, Tania Bedrax-Weiss, and Dan Roth. 2019. How large are lions? inducing distributions over quantitative attributes. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3973-3983, Florence, Italy. Association for Computational Linguistics.
Lisa Feigenson, Stanislas Dehaene, and Elizabeth Spelke. 2004. Core systems of number. Trends in cognitive sciences, 8(7):307-314.
Maxwell Forbes and Yejin Choi. 2017. Verb physics: Relative physical knowledge of actions and objects. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 266-276, Vancouver, Canada. Association for Computational Linguistics.
Mor Geva, Ankit Gupta, and Jonathan Berant. 2020. Injecting numerical reasoning skills into language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 946-958, Online. Association for Computational Linguistics.
Pranav Goel, Shi Feng, and Jordan Boyd-Graber. 2019. How pre-trained word representations capture commonsense physical comparisons. In Proceedings