SlowGuess/NAACL_2022_OCR huggingface dataset:
Daniel Spokoyny
dspokoyn@cs.cmu.edu
Zhao Jin
z3jin@ucsd.edu
Physical measurements constitute a large portion of numbers in academic papers, engineering reports, and web tables. Current benchmarks fall short of properly evaluating numeracy of pretrained language models on measurements, hindering research on developing new methods and applying them to numerical tasks. To that end, we introduce a novel task, Masked Measurement Prediction (MMP), where a model learns to reconstruct a number together with its associated unit given masked text. MMP is useful for both training new numerically informed models as well as evaluating numeracy of existing systems. To address this task, we introduce a new Generative Masked Measurement (GeMM) model that jointly learns to predict numbers along with their units. We perform fine-grained analyses comparing our model with various ablations and baselines. We use linear probing of traditional pretrained transformer models (RoBERTa) to show that they significantly underperform jointly trained number-unit models, highlighting the difficulty of this new task and the benefits of our proposed pre-training approach. We hope this framework accelerates progress towards building more robust numerical reasoning systems in the future. $^{1}$
Many natural language processing tasks require a deep understanding of numbers – for example, reading comprehension (Ran et al., 2019), textual entailment (Sammons et al., 2010; Roy, 2017) and hybrid table tasks such as fact-verification (Chen et al., 2020) or question answering (Chen et al., 2021). Masked number prediction (MNP) is a popular pretraining objective to imbue language models with numerical understanding and evaluate existing models for their numerical capacity.
ilylee@ucsd.edu
Figure 1: We present the Masked Measurement Prediction (MMP) task where the model predicts the dimension, unit and real-valued number. We also show the model architecture of Generative Masked Measurement model (GeMM), the model we propose to perform MMP. We display the fixed operations used during unit conversion in yellow. In black, we show the different components of the model's prediction.
As an example of MNP, given the sentence "Cats have [#NUM] paws." a model learns to predict the number 4. While appropriate for numerical commonsense, MNP is deficient when it is used to predict measurements. Measurements, such as 2 meters or 13.2 square miles, are a special class of particularly common numbers in text that have a well-defined and typed system of units. Given a simple question: "How long did Alex Honnold climb for?", a single number alone is an insufficient answer since it is meaningless without the unit. Answers like 1000 meters or 4 hours could both suffice.
Current MNP systems do not jointly reason about numbers with units. It is reasonable to expect that pretrained models like BERT could leverage information of units directly as text without
<sup>1</sup>We will release our trained models and data-splits upon acceptance on Github.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 17 - 29
any special treatment. However, in preliminary experiments we find that this yields poor numerical abilities (see Appendix B). Furthermore, including units as text directly raise more questions: should we evaluate using all units (meters, feet, inches)? Should we equally weight across the units? Current models have no opinion about which unit is appropriate because they are not required to make unit predictions during training. Together, this indicates that current training objectives do not capture sufficient representations of measurements and that a direct application of MNP to evaluate numeracy of measurements is ill-suited.
To address these shortcomings, we propose the more challenging task of Masked Measurement Prediction (MMP) along with a new model. In this task, a model must reconstruct both the number together with the correct unit. In Figure 1 we show how in a MMP setting our model generates a dimension ("Length"), a number in metric log-space ("3.00"), the unit ("feet") and then uses the conversion factor ("3.28") to deterministically output the full measurement ("3280 feet"). This example illustrates a key distinction in that our model is flexible and can generate non-metric measurements (feet) but evaluates numerical prediction in canonical units (meters).
MMP is useful for two reasons: 1) as a way to train models to give them better numeracy 2) as a new kind of evaluation that allows for a much more fine-grained analysis of reasoning over numerical quantities. The task of measurement estimation decouples the different aspects of numeracy allowing for a more interpretable and thorough analysis of numerical reasoning. We introduce a new evaluation benchmark for MMP based on WikiConvert (WiCo) (Thawani et al., 2021a), a large scale dataset of English Wikipedia sentences with ground truth measurement annotations. We compare the performance of our models on their ability to accurately predict the dimension, unit, and value of a measurement. We employ a large pretrained transformer model as our textual encoder and examine the performance of different discriminative, generative, and latent variable models along with several ablations. Our contributions are as follows:
- We introduce a novel challenging task  $MMP$  for pretraining and evaluating numeracy.
Figure 2: GeMM as a graphical model. The broken arrows represent a deterministic unit conversion. Examples of unit values and their corresponding dimension values are also shown.
- We show that linear probing of existing pretrained models on MMP significantly underperforms fully finetuned models.
- We train a model that reasons jointly about numbers and units which predicts numbers 8.1 times more accurately than the probed pretrained models.
- We find our best performing generative model outperforms human annotators on two evaluations, achieving  $7.4 - 7.8\%$  better dimension accuracy and  $33.5 - 39.9\%$  better unit accuracy. Furthermore, this model predicts a number closer to ground truth than our annotators  $66.2 - 78.8\%$  of the time.
Furthermore there are numerous applications of better measurement prediction and unit reconstruction such as in table to text generation (Moosavi et al., 2021), answering numerical queries (Sarawagi and Chakrabarti, 2014; Ho et al., 2019) or for improving e-commerce product search(Arici et al., 2021). We hope that Masked Measurement Prediction becomes a standard benchmarking tool from which we can gain insight how to best incorporate new numeracy modeling techniques as well as evaluate existing models.
2.1 Background + Notation
The International System of Units (SI) defines seven fundamental dimensions (Length, Time, Mass, etc.) and seven corresponding base SI units (meters, seconds, kilograms, etc.). The SI system is the most widely adopted measurement standard
2Our metric of choice described in Equation 2 is invariant to the specific choice of canonical unit i.e., log-mae in meters is equal to log-mae in feet.
and is used internationally in domains such as commerce, finance, logistics, and science. We designate  $\mathcal{D}$  to be the set of composite dimensions obtained from (and including) the fundamental dimensions. Values of  $\mathcal{D}$  include velocity and power. We let  $\mathcal{U}$  be the set of all units: the various ways to describe dimensions. For example, units of Length include meters and miles. Each training example consists of a real number  $y$ , a dimension  $d \in \mathcal{D}$ , a unit  $u \in \mathcal{U}$ , and the remainder of the sentence  $S$ . In MMP, our task is to predict  $y$ ,  $d$ , and  $u$  given only  $S$ . In the next sections we describe our generative model designed for MMP followed by the ablations we consider.
Measurements have complex semantic meanings, shaped by many standards, particular instruments, and natural world phenomena. Consider a text concerning rainfall. From a dimensional analysis perspective, the units inches per year (in/y) and meters per second  $(m / s)$  share the same dimension velocity. However, mentioning in/y usually implies that the text is discussing total rainfall in a region. Likewise, the use of  $m / s$  suggests that the text is examining the speed of falling rain droplets. To capture this complexity, we consider a generative model that learns the joint distribution of the number, dimension, and unit.
We now describe the generative process of our full model. To start, conditioned on  $S$ , our model samples a discrete dimension variable  $D$ . Then conditioned on the sampled dimension, our model samples a discrete unit variable  $U$  compatible with the dimension. For example, conditioned on the dimension velocity our model will output a distribution over the units of velocity such as [miles per hour; meters per second, inches per year] as opposed to all of  $\mathcal{U}$ . We then separately predict a distribution on the canonicalized measurement,  $\bar{Y}$ , which is the numerical quantity represented in a base canonical (metric) unit like meters. During inference time, we use the highest scoring dimension and unit and choose the proper conversion factor to deterministically produce the final number  $y$  represented in the predicted unit. We refer to this Generative Masked Measurement model as GeMM, where the joint  $p(D,Y,U|S)$  is given by the following equation:
$$
p (\boldsymbol {D} | \boldsymbol {S}) \times p (\boldsymbol {U} | \boldsymbol {D}, \boldsymbol {S}) \times p (\boldsymbol {Y} | \boldsymbol {S})
$$
We show the graphical model of GeMM in Figure
2. We also consider,  $\text{GeMM} \boxed{\mathrm{U - Y}}$ , a slight variant where we have a direct dependence between the unit and number prediction with a joint equal to:
$$
p (\boldsymbol {D} | \boldsymbol {S}) \times p (\boldsymbol {U} | \boldsymbol {D}, \boldsymbol {S}) \times p (\boldsymbol {Y} | \boldsymbol {U}, \boldsymbol {S})
$$
2.3 Discrete Latent Dimension Model
We also consider an unsupervised generative model which treats the dimension as a discrete latent variable. We use the same number of dimension classes  $|\mathcal{D}|$  and train to maximize the log-likelihood of the observed  $\pmb{Y}$ . We refer to this model as Lat-Dim and is characterized by:
$$
p (\boldsymbol {Y} | \boldsymbol {S}) = \sum_ {D} p (\boldsymbol {D} | \boldsymbol {S}) \times p (\boldsymbol {Y} | \boldsymbol {D}, \boldsymbol {S})
$$
To evaluate this model we build a contingency matrix of the predicted classes and using a linear solver find the best mapping between our predicted and true dimensions. We can then apply this mapping to the model predictions and calculate classification metrics for dimension prediction.
2.4 Model Ablations
We also consider several model ablations of  $GeMM$ . Our first ablation is  $GeMM\text{-Y}\text{-U}$  which models  $p(\pmb{D}|\pmb{S})$ . The second,  $GeMM\text{-Y}$ , learns the distribution  $p(\pmb{U},\pmb{D}|\pmb{S}) = p(\pmb{D}|\pmb{S})\times p(\pmb{U}|\pmb{D},\pmb{S})$ . The third,  $GeMM\text{-U}$ , models  $p(\pmb{Y},\pmb{D}|\pmb{S}) = p(\pmb{D}|\pmb{S})\times p(\pmb{Y}|\pmb{D},\pmb{S})$ . Our final ablation is  $GeMM\text{-U}\text{-D}$  which learns  $P(\pmb{Y}|\pmb{S})$  directly.
2.5 Model Architectures
For our textual encoder, we use the Huggingface Transformers (Wolf et al., 2020; Liu et al., 2019) implementation of RoBERTa, a pretrained 12-layer transformer. We refer to this text encoder as  $T$  such that given a sentence  $S$ , our model outputs a 768-dimensional vector  $h_T$ . We use a single linear layer,  $W_S \in \mathbb{R}^{768 \times M}$ , to project  $h_T$  to  $h$  and treat the dimension  $M$  as a hyper-parameter. To form a distribution over the real number line  $\mathbb{R}$  we use a Log-Laplace model, a competitive model used in the numeracy literature (Spokoyny and Berg-Kirkpatrick, 2020; Thawani et al., 2021a; Zhang et al., 2020). This is equivalent to  $L_1$  regression in log-space and yields the following loss function where  $Y$  and  $Y^*$  are predicted and ground truth numbers, respectively:
$$
\log P (\boldsymbol {Y} | \boldsymbol {S}) = \left| \log \boldsymbol {Y} ^ {*} - \log \boldsymbol {Y} \right| + \log \left| \frac {1}{\boldsymbol {Y}} \right| \tag {1}
$$
Split	Examples	Max #	Min #
All	919,237	5.5E+36	1E-06
Train	728,629	5.5E+36	1E-06
Val	91,110	4.4E+14	1.2E-06
Test	91,092	1.6E+21	1.8E-06
As shown in Figure 1, we project  $h$  with a linear layer  $W_{D} \in \mathbb{R}^{M \times |\mathcal{D}|}$  to obtain a distribution over  $D$ . We then use a separate linear layer,  $W_{U} \in \mathbb{R}^{M \times |\mathcal{U}|}$ , to project  $h$  and obtain a distribution over  $U$ . To predict  $\bar{Y}$ , we project  $h$  with a linear layer  $W_{Y}$ . In the case of GeMM, we let  $W_{Y} \in \mathbb{R}^{M \times |\mathcal{D}|}$  in order to parameterize a mean of a Log-Laplace distribution for each dimension in  $D$ . For GeMM [U-Y], we set  $W_{Y} \in \mathbb{R}^{M \times |\mathcal{U}|}$  to output the mean of a Log-Laplace distribution for each unit in  $U$  and the remaining models, we set  $W_{Y} \in \mathbb{R}^{M \times 1}$  resulting in a single mean of a Log-Laplace distribution. For training, we use cross-entropy loss for the dimension and unit distributions, and the loss from the equation above for number prediction.
We train and evaluate our models on WiCo (Thawani et al., 2021a), a dataset of English Wikipedia sentences where the number and unit in each sentence are human-annotated. We canonicalize the units and map each to a single dimension. For example both feet per second and miles per hour map to velocity. We show the distribution of all measurements and lengths in Figure 3. The resulting dataset consists of 919,237 sentences with annotated (number, unit, dimension) triples. We provide more details on the data in Appendix A.
We train all models using a batch size of 200 and use the AdamW (Loshchilov and Hutter, 2019) optimizer with a learning rate of  $1e^{-4}$  and a linear warm-up schedule of 500 steps. We use the “ $\star$ ” symbol to indicate that we freeze the transformer parameters for training. For all frozen models we use a log frequency weighted cross-entropy due to the highly imbalanced classes as well as a higher
Table 1: Summary statistics for Wiki-Convert. The median number of characters and tokens per example is 106 and 33, respectively.
Model	10-shot	40-shot	70-shot	100-shot
GeMM-Y-U	15.5	50.0	52.5	53.4
GeMM-Y-U	42.5	51.2	57.6	60.5
Majority	14.3	14.3	14.3	14.3
Table 2: Results (measured by F1 ↑) of our few-shot experiment on dimension classification (probing  $p(D|S)$ ).  $x$ -shot implies the model is trained on  $x$  labeled examples per dimension. GeMM Y-U indicates an ablation of GeMM where  $Y$  and  $U$  are not modeled. * indicates the model's parameters are frozen during training.
Model	10-shot	40-shot	70-shot	100-shot
GeMM U-D	1.94	1.82	1.72	1.75
GeMM U-D	1.70	1.56	1.43	1.41
Median	1.99	1.99	1.99	1.99
Table 3: Results (log-mae  $\downarrow$  ) of our few-shot experiment on number prediction (probing  $p\left( {Y \mid  S}\right)$  ).
learning rate of  $1e^{-3}$ . We employ early stopping with a patience of five epochs on validation score.
To evaluate the performance of our models, we report the macro averaged F1 score for dimension and unit prediction and log-mae to evaluate number prediction. We define log-mae in Equation 2 where  $Y$  is the predicted number and  $Y^{*}$  is the ground truth number. As a simple baseline for dimension and unit prediction, we employ majority class voting. For number prediction we use the median of all the numbers in the training set.
$$
\log \text {m a e} = \frac {1}{| \mathcal {D} _ {\text {t e s t}} |} \sum_ {\mathcal {D} _ {\text {t e s t}}} \left| \log_ {1 0} Y ^ {*} - \log_ {1 0} Y \right| \tag {2}
$$
4.1 Few-Shot
To study the degree to which current pretrained models capture different aspects of numeracy, we consider the following few-shot experiment. We sample a balanced dataset of dimensions where each class gets 10, 40, 70, or 100 labeled examples. We train GeMM-Y-U and GeMM-U-D on the few-shot task where the pretrained text encoder  $T$  parameters are frozen and compare their performance against full fine-tuning. Due to the high variance of GeMM-Y-U, we report the average of three random seeds. In Table 2 and Table 3 we show results of GeMM-Y-U and GeMM-U-D respectively.
Although performance improves with more data, the frozen models significantly underperform their
Figure 3: Histograms of WiCo numbers binned by base-10 exponent. All numbers are canonicalized to their SI form. Left: All numbers labeled by dimension. Right: Numbers in the length dimension labeled by unit.
Model	Probing Type	Val	Test
Majority	-	33.1	33.1
GeMM*	p(D|S)	69.1	67.5
GeMM-Y-U	p(D|S)	88.0	86.8
GeMM-Y	p(D|S)	87.0	87.3
GeMM-U	p(D|S)	87.2	86.6
Lat-Dim	p(D|S)	9.0	9.1
GeMM	p(D|S)	87.4	87.0
GeMM-U-Y	p(D|S)	86.4	86.1
Table 4: Results (F1 ↑) for dimension prediction conditioned on  $S$  only. GeMM U-Y indicates a variant of GeMM where  $\bar{Y}$  is dependent on  $U$  (in addition to  $S$ ).
Model	Probing Type	Val	Test
GeMM-U	p(D|Y,S)	95.5	95.7
GeMM U-Y	p(D|Y,S)	96.4	96.6
unfrozen counterparts across all dataset sizes. For example, in the 100-shot dataset, the frozen model shows 7.1 lower F1 and 0.34 higher log-mae. These results suggest that current pretrained transformers do not capture numeracy to a large extent.
4.2 Dimension Prediction
We train our models and their ablations on the full dataset and measure their performance on dimension prediction. In Table 4, we show the results of dimension prediction conditioned on  $S$ . We observe that the performance gap between the frozen and unfrozen GeMM grows to 19.5 F1 on the test
Table 5: Results (F1 ↑) for dimension prediction conditioned on  $\bar{Y}$  and  $S$ .
Model	Probing Type	Val	Test
Majority	-	8.9	9.0
GeMM※	p(U|D,S)	29.8	29.8
GeMM -Y	p(U|D,S)	52.9	51.7
GeMM	p(U|D,S)	51.5	54.9
GeMM U-Y	p(U|D,S)	49.3	47.8
Table 6: Results (F1 ↑) on unit prediction conditioned on the true dimension and text. Ablations are above the double horizontal line.
split despite training on 3 orders of magnitude more training data than the few-shot setting.
By using Bayes' rule, we perform dimension prediction conditioned on both  $S$  and  $\bar{Y}$  and show our results in Table 5. We observe that both models show improved dimension prediction ability when supplied with the number with GeMM U-X reaching 96.6 F1 score, an effective error rate reduction of 75%.
4.3 Unit Prediction
We show the unit prediction performance of our models in Table 6. The strongest performing model for unit prediction was GeMM with a F1 score of 54.9. Again, the frozen GeMM produced a 25.1 lower F1 score than its unfrozen counterpart.
We note that even though the F1 scores on unit prediction are much lower than dimension prediction, they are still significantly better than the majority baseline. Although one can freely substitute a unit with one in the same dimensional class, we tend to be more systematic and choose units that allow for more straightforward human readability or reflect the actual instruments used for measure
Model	Probing Type	Val	Test
Median	-	1.98	1.97
GeMM*	p(Ŷ|S)	1.377	1.370
GeMM-U-D	p(Ŷ|S)	0.529	0.531
GeMM-U	p(Ŷ|D, S)	0.468	0.469
p(Ŷ, D|S)	0.517	0.518
Lat-Dim	p(Ŷ, D|S)	0.545	0.546
GeMM	p(Ŷ|S)	0.517	0.515
GeMM-U-Y	p(Ŷ|U, D, S)	0.401	0.401
p(Ŷ, U, D|S)	0.526	0.526
Table 7: Results (log-mae ↓) for number prediction conditioned on  $S$ . In the second row of GeMM [U], we select the highest scoring  $d^{*} \in D$  and predict  $y$  conditioned on  $d^{*}$  and  $S$ . In the second row of GeMM [U-Y], we select the highest scoring  $u^{*} \in U$  and  $d^{*} \in D$  and predict  $y$  conditioned on  $u^{*}$ ,  $d^{*}$ , and  $S$ . For Lat-Dim, we sum over the latent variable  $D$  to predict  $y$  conditioned on  $S$ .
ment. As a result, we gravitate towards regularities that models can learn to recognize. The converse of this is also interesting as it suggests that the expressed units imply more semantic meaning than what is captured in the standardized measurement.
4.4 Number Prediction
We show the number prediction performance of our models in Table 7. Consistent with our previous experiments, all models outperform  $GeMM^{\text{※}}$ . Furthermore, we observe that not modeling  $U$  and  $D$  (as is the case in  $GeMM \boxed{U} \boxed{D}$ ) increases log-mae, i.e., results in worse numerical prediction. While competitive with  $GeMM$  and its variants on number prediction, Lat-Dim cannot predict dimensions with the same efficacy (Table 4).
We also experiment with the setting where  $\text{GeMM-U}$  conditionally generates the number for a particular dimension. In this setting,  $\text{GeMM-U}$  improves  $\log$ -mae to 0.469. Extending this setting further, we condition  $\text{GeMM-U-Y}$  on both a unit and a dimension to produce the best  $\log$ -mae among our models: 0.401.
We now revisit our original motivating example: "Alex Honnold climbed for [NUM] [UNIT)". Assume we want to know the distance of a climb. To do this, we condition  $\text{GeMM} \overline{\text{U·Y}}$  on  $D = \text{length}$  and  $U = \text{feet}$ . If, on the other hand, we want to know the duration of a climb, we change the conditioning to  $D = \text{time}$  and  $U = \text{hours}$ . Now, if we
want to know the length of Alex Honnold's climbing career, we condition  $\text{GeMM} \boxed{\mathrm{U - Y}}$  on  $D = \text{time}$  and  $U = \text{years}$ . These examples illustrate the flexibility of  $\text{GeMM} \boxed{\mathrm{U - Y}}$  and the importance of jointly modeling numbers, units, and dimensions.
4.5 Quantitative Analysis
4.5.1 Dimensions and Unit
In Figure 4a we visualize a confusion matrix of dimension predictions by  $\text{GeMM}$  U-Y. The low accuracy for electric charge and temperature is attributed to a mislabeling in the dataset. For mass, we find many ambiguous situations where either mass or length are appropriate. See the first row of Table 10 for such an example.
Thus far, we have treated dimensions as distinct classes with no relationships. However, dimensions are compositions of the seven fundamental dimensions. Therefore, dimensions that share fundamental dimensions are more similar than those that do not. To quantify this similarity, we can treat dimensions as a vector where each element represents the exponent of a fundamental dimension. Then to measure the similarity of two dimensions, we take their Manhattan distance. To illustrate, assume there exist only two fundamental dimensions: Length and Time. Let speed = (1, -1) and length = (1, 0) where the first element represents Length and the second represents Time. The Manhattan distance between speed and length is equal to one. In Figure 5, we visualize the Manhattan distance between the predictions of GeMM U-Y and ground truth. We observe that there is generally an inverse relationship between error count and the distance of the errors. This observation suggests that our model has learned that some dimensions are more similar than others. This suggestion is reinforced by Figure 4a where misclassifications tend to have small distances from the true dimension. For example, velocity is most often misclassified as length. For unit prediction, we find that most mistakes occur substituting units with ones that have similar magnitudes like feet for meters or kilometers for miles.
4.5.2 Numeracy
In Table 8, we show log-mae by dimension as predicted by GeMM U-Y. We note that errors are not uniform across dimensions, predicting areas is 2.2
Sentences with mislabeled Celsius as Coulombs, which may due to wrong annotation between  $^\circ \mathrm{C}$  and C. Also observed by Elazar et al. (2019)
(b)
Figure 4: Confusion matrices for predictions by GeMM U-Y over the validation split. Left 4a: Dimension prediction. Most misclassified dimensions are similar to their ground truth counterparts in terms of Manhattan distance. Right 4b: Unit prediction for examples that share the length dimension. Most misclassified units of length share similar magnitudes to their ground truth units.
Length	Area	Velocity	Mass	Power
0.37	0.54	0.19	0.55	0.27