SlowGuess/NAACL_2021_OCR huggingface dataset:
Avijit Thawani and Jay Pujara and Filip Ilievski and Pedro Szekely
{thawani,jpujara,ilievski,pszekely}@isi.edu
NLP systems rarely give special consideration to numbers found in text. This starkly contrasts with the consensus in neuroscience that, in the brain, numbers are represented differently from words. We arrange recent NLP work on numeracy into a comprehensive taxonomy of tasks and methods. We break down the subjective notion of numeracy into 7 subtasks, arranged along two dimensions: granularity (exact vs approximate) and units (abstract vs grounded). We analyze the myriad representational choices made by over a dozen previously published number encoders and decoders. We synthesize best practices for representing numbers in text and articulate a vision for holistic numeracy in NLP, comprised of design trade-offs and a unified evaluation.
Numbers are an integral part of text. To understand a simple sentence like I woke up at 11, we need not just literacy but also numeracy. We must decode the string 11 to the quantity 11 and infer 11 to denote a time of the day, probably 11 a.m. We need commonsense to reason that 11 a.m. is quite late in the morning. This interpretation of 11 is strongly contextual, as I earn $11 per month evokes different units and value expectations. Note how the semantics remains the same for both sentences if 11 was replaced by 10, i.e., the context is tolerant to some variability.
Numbers are everywhere. Reasoning with quantities and counts is crucial to understanding the world. Evolutionary learning has given numerical cognition skills to several animals, including human beings (Dehaene, 2011). Our ancient ancestors furthered numeracy by developing multiple number systems, similar to but independent from the evolution of languages. Numeracy is an essential skill for language understanding, since numbers are often interspersed in text: the 6 million pages in English Wikipedia have over 150 million numbers.
Numbers are neglected. In NLP, however, numbers are either filtered out explicitly during preprocessing (Graff et al., 2003), or treated the same as words, often collapsing them into an UNK token. Subword tokenization approaches like BPE (Sennrich et al., 2016) and WordPiece (Wu et al., 2016) instead retain numbers, but split them into arbitrary tokens, for example 1234 might be split into two tokens as 12-34 or 123-4 or 1-234.
Recent work has shown that these are suboptimal number representations (Wallace et al., 2019; Zhang et al., 2020). On the DROP Question Answering benchmark, BERT performs five times worse when the answer is a number instead of a span of text (Dua et al., 2019). Relatively simple strategies like switching from subword to char-level tokenization (Geva et al., 2020), or from decimal to scientific notation (Zhang et al., 2020) already boost performance. Such results warrant a deeper study into the best number representations.
Numbers are important. Given the ubiquity of numbers and their fundamental differences with words, enabling NLP systems to represent them effectively is beneficial for domains like scientific articles (Spithourakis and Riedel, 2018) and financial documents (Chen et al., 2019; Jiang et al., 2020). Number understanding is also useful to detect sarcasm (Dubey et al., 2019) and to model dialogues involving price negotiations (Chawla et al., 2020).
Recent NLP progress towards numeracy has been sporadic but encouraging. In this paper, we survey prior work and highlight the kind of numeracy targeted (e.g., arithmetic, measurement, numeration) as well as the kind of representation used (e.g., value embeddings, DigitRNNs). We provide the first NLP-centric taxonomy of numeracy tasks (Section 2) and of number representations (Section 3) for the reader to succinctly comprehend the challenge posed by numeracy. We synthesize key takeaways (Section 5) and propose a unifying vision for future research (Section 6).
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644-656 June 6-11, 2021. ©2021 Association for Computational Linguistics
Benchmarking or Probing Tasks	Downstream Applications
Abstract	Grounded
Exact	Simple Arithmetic (2+3=5)	AWP (2 balls + 3 balls = 5 balls),
Exact Facts (birds have two legs)	Question Answering,
Science Problems
Approx	Numeration ('2' = 2.0),
Magnitude ('2' < '5')	Measurement (dogs weigh 50 lbs),
Numerical Language Modeling	Sarcasm Detection,
Numeral Categorization
Table 1: Seven numeracy tasks, arranged along the axes of (rows) granularity - exact vs approximate, and (columns) units - abstract vs grounded. We also list downstream applications requiring a similar granularity of numeracy.
2 Tasks
There are several different aspects of numeracy. The DROP dataset alone offers a wide variety of numeric reasoning questions such as retrieval-based (How many yards did Brady run?), count-based (How many goals were scored? given a comprehension describing multiple goals), and simple arithmetic (How many years after event 1 did event 2 occur? given dates of both events). Besides downstream applications, there have also been probing experiments to evaluate whether NLP models can decode numbers from strings (e.g., 19 to 19.0), or estimate quantities (e.g., how tall are lions?).
Such a diverse range of abilities are usually all referred to collectively as numeracy, which gives rise to confusion. We limit this abuse of terminology and provide a neat taxonomy for arranging the different tasks proposed under numeracy.
2.1 Our Taxonomy of Tasks
Drawing from work in cognitive science (Feigenson et al., 2004), we propose the following two dimensions to organize tasks within numeracy:
1. Granularity: whether the encoding of the number is (1) exact, e.g., birds have two legs, or (2) approximate, e.g., Jon is about 180 cms tall.
2. Units: whether the numbers are (1) abstract, e.g.,  $2 + 3 = 5$ , or (2) grounded, e.g., 2 apples + 3 apples = 5 apples. While abstract mathematical tasks are easy to probe and create artificial datasets for, numbers grounded in units are challenging since they need to be understood in the context of words.
2.2 Survey of Existing Tasks
We now describe 7 numeracy tasks, arranged according to our taxonomy in Table 1, as well as downstream tasks (right-most column in the table).
Simple Arithmetic is the task of addition, subtraction, etc. over numbers alone. It is convenient
to create synthetic datasets involving such math operations for both masked (Geva et al., 2020) and causal language models (GPT-3 Brown et al. 2020).
Numeration or Decoding refers to the task of mapping a string form to its numeric value, e.g., 19 to 19.0. Within NLP, this task is set up as a linear regressor probe over a (frozen) representation of the string. Numeration has been probed for in static word embeddings (Naik et al., 2019), contextualized language models (Wallace et al., 2019), and multilingual number words, e.g., nineteen or dix-neuf (Johnson et al., 2020).
Magnitude Comparison is the ability to tell which of two (or more) numbers is larger. For language models, this has been probed in an argmax setup (choose the largest of five numbers) as well as a binary classification task, e.g., given 23 and 32, pick the label 1 to indicate that  $32 > 23$  (Naik et al., 2019; Wallace et al., 2019).
Arithmetic Word Problems (AWP) are the grounded version of simple arithmetic that we find in school textbooks, e.g., Mary had two cookies. She gave one away. How many does she have left? There exist several NLP datasets on math word problems (Amini et al., 2019; Saxton et al., 2019; Roy and Roth, 2015; Hendrycks et al., 2021).
Exact Facts in the context of numeracy involves commonsense knowledge such as dice have 6 faces or birds have two legs. An approximate sense of quantity would be of little help here since assertions like dice have 5 faces or birds have three legs are factually incorrect. Two recent datasets for numeric commonsense facts are Numbergame (Mishra et al., 2020) and NumerSense (Lin et al., 2020).
Measurement Estimation is a task in psychology in which subjects are asked to approximately guess measures of objects along certain dimensions, e.g., number of seeds in a watermelon or weight of a telephone (Bullard et al., 2004). VerbPhysics (Forbes and Choi, 2017) is a benchmark of binary
comparisons between physical attributes of various objects, e.g., ball  $<_{size}$  tiger. DoQ (Elazar et al., 2019) is a web-extracted dataset of Distributions over Quantities, which can be used as a benchmark for language models' measurement estimation abilities (Zhang et al., 2020). Lastly, MC-TACO (Zhou et al., 2020) is a collection of temporal-specific measurement estimates, e.g., going for a vacation spans a few days/weeks.
Numerical Language Modeling in its literal sense is not a task but a setup, analogous to masked/causal language modeling for words. Other tasks could be modeled as numeric language modeling, e.g., arithmetic  $(2 + 3 = [MASK])$  and measurement estimation (lions weigh [MASK] pounds). In practice, numerical language modeling refers to the task of making numeric predictions for completing unlabelled, naturally occurring text.
Word predictions in language modeling are typically evaluated with classification metrics such as accuracy or perplexity. Numeric predictions, on the other hand, are evaluated with regression metrics such as mean absolute error, root mean squared error, or their log and percentage variants (Spokoyny and Berg-Kirkpatrick, 2020). Spithourakis and Riedel (2018) also propose an Adjusted Perplexity metric to cancel the effect of the out-of-vocabulary rate on the perplexity of numeric tokens.
Downstream Applications for numeracy are abound. Dubey et al. (2019) detect sarcasm in tweets based on numbers. Chen et al. (2020) identify claims in financial documents using alternative number representations and the auxiliary task of numeral understanding or categorization (Chen et al., 2018). Similarly, simple arithmetic and math word problems serve as auxiliary tasks for GenBERT (Geva et al., 2020) towards improving its score on the DROP QA benchmark.
2.3 Other Numeracy Tasks
Here, we describe foundational numeracy-related tasks that cut across our taxonomy of tasks:
(Numeric) Paraphrasing is what we call the task of identifying one-to-one correspondences between different surface forms of the same number. Twelve is the same as '12', also referred to as a dozen. This task cuts across all the tasks we discussed, since the same number, expressed in several different ways, should be nevertheless identified by an NLP model before any subsequent reasoning. Similar to how WordNet (Miller, 1995) provides
a huge list of synonyms, numeric paraphrases can be obtained by libraries<sup>1</sup> which convert numerals to words, words to numerals, etc. One could also envision this as a learning task given a large enough corpus, such as the NumGen dataset (Williams and Power, 2010) containing 2000 fact-aligned numeric expressions over 110 articles.
Quantity Entailment tasks (Ravichander et al., 2019; Roy et al., 2015), analogous to Natural Language Inference, require understanding of not equivalence (as in paraphrasing) but deeper relations like entailment and contradiction, e.g., the premise he was 16 yrs old entails the hypothesis he was a teenager. On similar lines, Mishra et al. (2020) modify the QuaRel dataset (Tafjord et al., 2019) to force models to perform quantity entailment, e.g., dog1 is light, dog2 is heavy is replaced with dog1 weighs 70 lbs, dog2 weighs 90 lbs.
Numeral Understanding is the task of categorizing numbers into percentages, prices, dates, times, quantities, etc. and their respective subcategories (Chen et al., 2018).
Fused-Head Resolution for numbers is essential to ground them when the context is implicit. For example, the sentence I woke up at 11 has a.m. or o'clock as the fused head to be resolved (Elazar and Goldberg, 2019).
Counting is the task of keeping track of discrete instances of some object. When kids count a set of objects, they quickly learn to keep a track, say on their fingers, but struggle with realizing the Cardinal Principle, i.e., the last counter value denotes the number of entities being considered (Wynn, 1990). Similarly, LSTMs (Suzgun et al., 2019) and transformers (Bhattamishra et al., 2020) have been shown to possess counting skills but in order to answer counting questions, they must also learn to map the counts to number words or numerals. Counting tasks have been proposed in computer vision (Testolin et al., 2020) as well as in NLP (Postma et al., 2018; Talmor et al., 2020).
Domain-specific tasks require background knowledge in addition to exact mathematical skills. Numbergame (Mishra et al., 2020) includes questions on Physics (find the distance travelled in 2 hrs by a train moving at  $50\mathrm{mph}$ ) and Chemistry (find the mass percentage of  $H$  in C6H6). Project Aristo (Clark et al., 2019) solves elementary and high school science problems, which often involve numeric reasoning.
Example: https://pypi.org/project/num2words/
Analogous to our taxonomy of subtasks in the previous section, here we attempt to arrange the wide variety of alternative number representations proposed in recent literature. We limit our analysis to methods of encoding (numbers  $\rightarrow$  embeddings) and/or decoding (embeddings  $\rightarrow$  numbers) numbers. We do not discuss, for example, methods that use symbolic reasoning (Andor et al., 2019) or modify activation functions to enhance numeracy (Trask et al., 2018).
A typical example of the base architecture could be BERT (Devlin et al., 2019), the workhorse of modern NLP. We assume that there exists an independent parallel process of mapping words into embeddings, such as subword tokenization followed by lookup embeddings in BERT.
3.1 Our Taxonomy
We look at two kinds of representations: string-based and real-based. Real-based representations perform some computation involving the numerical value of the number. The string-based representations instead see numbers in their surface forms; they must assign arbitrary token IDs and look up their embeddings to feed into the architecture.
3.1.1 String Based
By default, language models treat numbers as strings, the same as words. However, within string representations, one could tweak simple changes:
Notation: The number 80 could be written in Hindu-Arabic numerals (80), Roman numerals (LXXX), scientific notation (8e1), English words (eighty), or with base 20 as in French (quatrevings). Nogueira et al. (2021) exclusively study the effect of many such notation choices in language models, on the task of simple arithmetic.
Tokenization: Word level tokenizations are ineffective for numbers, since they are likely to map most numbers to an UNK token, except for a few commonly occurring ones (e.g., 1, 2, 5, 10, 100). Other possibilities are subword tokenizations like BPE and WordPiece, as well as character (or digit) level tokenizations.
Pooling: The pooling dimension of variation springs up after analyzing the effect of tokenization. With subword and character level tokenizations, a single number may now correspond to multiple tokens, e.g., 100 segmented into  $10 - 0$  or  $1 - 0 - 0$ . Prior work (Spithourakis and Riedel, 2018) has ar
gued for using RNNs or CNNs to instead pool the embeddings of these tokens into a single embedding before feeding to the language model. The default way that language models see numbers are the same as words, hence no pooling is applied.
3.1.2 Real Based
Real-based number encoders can be expressed as  $f: \mathbb{R} \to \mathbb{R}^d$  whereas decoders can be expressed as  $g: \mathbb{R}^d \to \mathbb{R}$ . Real-based methods proposed in literature can vary on account of direction (whether they encode, decode or both), scale (linear vs log), and discretization (binning vs continuous valued).
Direction: Some proposed methods are encoder-only, e.g., DICE (Sundararaman et al., 2020), while some can be decoder-only, e.g., those requiring sampling from a parameterized distribution (Spokoyny and Berg-Kirkpatrick, 2020).
Scale: Inspired by cognitive science literature (Dehaene, 2011), several methods have attempted to model numbers in the log (instead of linear) scale, i.e., to perform mathematical operations on the logarithm of the number to be represented. The first operation in a log-scaled  $f$  is  $\log(\cdot)$  and the last operation in a log-scaled  $g$  is  $\exp(\cdot)$ . We discuss more scales in the following subsection, such as the stabilized log scale (Jiang et al., 2020) and the learned scale/flow (Spokoyny and Berg-Kirkpatrick, 2020).
Discretization: Training continuous value functions for a large range of numbers turns out to be practically infeasible (Wallace et al., 2019). Some real-based methods first bin numbers before learning embeddings for each bin. These bins could be on the linear scale (0-10, 10-20, 20-30, ...) or the log scale (0.01-0.1, 0.1-1, 1-10, ...), and the lookup embeddings can be learnt by the regular cross entropy (Chen et al., 2020) or dense cross entropy (Zhang et al., 2020).
3.2 Survey of Existing Methods
Having established dimensions of variance of number representations, we describe some key string-based and real-based methods used in prior work. Table 2 depicts these methods as individual rows, with the first three columns showing their position in our taxonomy ( $\S$  3.1). The last seven columns correspond to the seven tasks ( $\S$  2.2), with each cell denoting a representative work that introduces it.
Arith	Exact Facts	AWP	Num	Approximate Mag	Meas	\(LM^N\)
String-Based	Notation	Tokenization	Pooling
Word Vectors	Decimal	Word	NA	W+19	W+19	W+19	G+19	J+20
Contextualized	Decimal	Subword	No	W+19	L+20	G+20	W+19	W+19	Z+20	SR18
GenBERT	Decimal	Char	No	G+20	G+20
NumBERT	Scientific	Subword	No	Z+20	Z+20
DigitRNN/CNN	Decimal	Char	Yes	W+19	W+19	W+19	SR18
DigitRNN-sci	Scientific	Char	RNN	BS20
Exponent	Scientific	Word	NA	BS20
Real-Based	Scale	Direction	Binning
DICE	Linear	Enc-only	No	S+20	S+20	S+20
Value	Linear	Both	No	W+19	W+19	W+19
Log Value	Log	Both	No	W+19	W+19	W+19	Z+20
MCC	Log	Dec-only	Yes	Z+20
Log Laplace	Log	Dec-only	No	BS20
Flow Laplace	Learn	Dec-only	No	BS20
DExp	Log	Dec-only	No	BS20
GMM	Linear	Dec-only	Both**	SR18
GMM-proto	Linear	Enc-only*	No	J+20	J+20	J+20	J+20
SOM-proto	Log	Enc-only*	No	J+20	J+20	J+20	J+20
Table 2: An overview of numeracy in NLP: Each row is a method (§3.2), arranged as per our taxonomy (§3.1) split by string and real, further branching into three dimensions each. The last seven columns correspond to the seven subtasks of numeracy (§2.2), split by Exact and Approximate granularity (§2.1). The cells point to representative (not exhaustive) works that have experimented with a given method (row) on a given task (column). Notes: Prototype* is encoder-only but reuses embeddings for the decoder (Jiang et al., 2020). GMM** has been discretized (Spithourakis and Riedel, 2018) as well as continuous valued (Spokoyny and Berg-Kirkpatrick, 2020).
3.2.1 String-based methods
Word Vectors & Contextualized Embeddings Word2vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014), ELMo (Peters et al., 2018), and BERT (Devlin et al., 2019) have been probed as baselines against several contending methods.
GenBERT Geva et al. (2020) present GenBERT, a question answering model with pretrained BERT serving as both its encoder and decoder. GenBERT tokenizes numbers at the digit level, and is finetuned on auxiliary tasks of arithmetic word problems and simple arithmetic.
NumBERT Zhang et al. (2020) pretrain BERT from scratch over a modified dataset such that all numbers have been converted into scientific notation, i.e., 314.1 is expressed as 3141[EXP]2). NumBERT hence follows a scientific notation, subword tokenization, and no pooling.
DigitRNN, DigitCNN Spithourakis and Riedel (2018) and Wallace et al. (2019) experimented with pooling of digit embeddings into a single embedding representing the full number. Both used RNNs as well as CNNs for pooling.
DigitRNN-sci & Exponent (Embedding) Spokoyny and Berg-Kirkpatrick (2020) use a scientific notation variant of DigitRNNs (which
we refer to as DigitRNN-sci in Table 2), as well as a simpler alternative: exponent embedding. The latter merely learns a lookup embedding for the exponent, completely ignoring the mantissa.
3.2.2 Real-based methods
DICE Determinisitic Independent-of-Corpus Embeddings (Sundararaman et al., 2020) is an attempt to handcraft number encoder  ${}^{3}f$  so as to preserve the relative magnitude between two numerals and their embeddings. Given two scalars  $i$  and  $j$  ,and their embeddings  $f\left( i\right)$  and  $f\left( j\right)$  ,the cosine distance between  $f\left( i\right)$  and  $f\left( j\right)$  is intended to monotonically increase/decrease with the Euclidean distance between  $i$  and  $j$  . DICE is offered as not only a deterministic encoding but also as an auxiliary loss function for softly training number embeddings alongside,say,SQuAD (Rajpurkar et al., 2016)
Value Embedding The most intuitive parameterized encoder for real numbers is one that feeds the scalar magnitude of the number through a shallow neural network. The converse of value embedding is to learn a shallow neural network mapping  $g: \mathbb{R}^d \to \mathbb{R}$ . This decoder is simply the probe used for decoding/numeration task.
The idea of projecting number magnitudes into
$^{2}$ Pooling as described in § 3.1.1.
3Number encoder-decoder as defined in  $\S 3.1.2$
an NLP model that otherwise inputs only lookup embeddings may appear flawed. But Vaswani et al. (2017) have (rather successfully) encoded positional information into transformers using both learned embeddings (similar to Value) and fixed ones (similar to DICE).
Log Value Wallace et al. (2019) also experiment with a log-scaled value encoder in addition to the one on a linear scale. Zhang et al. (2020) experiment with a log value decoder for measurement estimation, which they call the RGR (regress) method. Log scaling has a neuroscientific inspiration since observations of human (and animal) understanding of numbers is better modelled by a log-scale representation (Dehaene, 2011).
Log Laplace In contrast to the point estimate output of the RGR decoder, models can also be used to parameterize a distribution over numbers. Such a formulation is helpful when estimating approximate quantities. Vectors representing some context can be used to parameterize, say, the mean and variance of a Gaussian or Laplace distribution. Spokoyny and Berg-Kirkpatrick (2020) instead transform the space being modeled by parameterizing the location parameter of a Log-Laplace distribution  $L(X,1)$  where  $X$  is the context representation of unmasked tokens, in a masked (numerical) language modelling setup. When inferring or decoding a number, they sample a point  $z \sim L(X,1)$  and exponentiate it, such that the output is  $\exp(z)$ .
Flow Laplace The expressivity of number decoders can be expanded or contracted by merely parameterizing a different distribution. Spokoyny and Berg-Kirkpatrick (2020) propose a more expressive decoder where instead of the log scale, the model learns its own density mapping. After sampling  $z \sim L(X, 1)$ , the output is transformed to  $\frac{exp\left(\frac{z - a}{b}\right)}{c}$ , where  $a$ ,  $b$ , and  $c$  are also parameters emitted by the same model.
MCC or multi-class classification is another number decoder which outputs a distribution, but a discrete one: over log-scaled bins of numbers, e.g., 1-10, 10-100, and so on (Zhang et al., 2020). Previously described decoders either output a point estimate or a unimodal distribution, thus failing to hedge its predictions for a multimodal ground truth. Given a masked number prediction problem We went to the restaurant at [MASK] p.m., MCC is better equipped to estimate two peaks: one around lunch time (say, 1-2 p.m.) and another around dinner (say, 7-9 p.m.).