SlowGuess/NAACL_2022_OCR huggingface dataset:
Figure 5: Manhattan distance between true and predicted dimensions by GeMM U-Y. We treat dimensions as vectors whose elements are the exponents of the fundamental dimensions that compose a given dimension. Note that the y-axis is in log-scale.
Table 8:  $\log$  -mae  $\downarrow$  by dimension. It is harder to predict numbers of Area and Mass than other dimensions.
Model	Human	Model > Human
D	U	D	U	Y
Tech Ann.	96.7	86.2	88.9	46.3	78.8
AMT Ann.	96.7	77.0	89.3	43.5	66.2
Table 9: Dimension and unit prediction accuracy of our human evaluation experiment. GeMM U-Y outperformed the human annotators in both evaluations. Tech Ann. is over a balanced set of 90 sentences labeled by Technical Annotators. AMT Ann. is over a balanced set of 2,122 sentences annotated by AMT Annotators. The final column shows the model predicted a number closer to ground truth in  $66.2 - 78.8\%$  of the cases.
times harder velocities. We also observe that the magnitudes of errors seem to be positively correlated with the variances observed in Figure 3.
4.5.3 Human Evaluation
We perform two evaluations of GeMM U-Y against human annotators. In the first evaluation, we compare against the combined effort of three Technical Annotators on a balanced set of 90 sentences randomly sampled from the test set. The annotators worked together to predict the missing dimensions,
#	Text	True	GeMM	USY	Prediction	Human Prediction
Dim	Unit	Num	Dim	Unit	Num	Dim	Unit	Num
1	Hope is gaff rigged, 'V'-bottomed and has an [#NUM] [UNIT] centerboard.	Mass	pounds	385.6	Length	feet	2.97	Length	meter	50
2	Some have been running for over 50 years, each covering about [#NUM] [UNIT].	Velocity	miles year	0.10	Area	sqkm	2.09E+10	Area	sqmi	2.59E+07
3	Another medium-sized corvid, the [#NUM] [UNIT] Eurasian magpie (Pica pica) is also amongst the most widely reported secondary prey species for goshawks there.	Mass	grams	0.22	Mass	grams	0.05	Mass	grams	0.2
4	The twin cylinder, liquid-cooled, in-line two-stroke, [#NUM] [UNIT] Rotax 582 has also been used.	Power	horse-power	47725	Power	horse-power	39248	Power	horse-power	45000
5	Chrysothamnus may grow up to a [#NUM] [UNIT] tall shrub or subshrub, usually with woody stem bases	Length	cms	1.2	Length	meters	1.147	Length	meters	1
6	Kurt Busch was the fastest in the first practice session with a time of 21.372 seconds and a speed of [#NUM] [UNIT].	Velocity	miles hour	75.1	Velocity	miles hour	63.584	Velocity	meters second	10
Table 10: Instances of the MMP task performed during our human evaluation experiment, all numbers are in SI units. In ex. 1, both the model and humans predict the incorrect dimension length instead of mass. The preceding sentence of ex. 2 references "trains" leading both to incorrectly predict area instead of velocity. In ex. 6 the model predicts the speed of the NASCAR driver Kurt Busch's car whereas the humans had mistaken him for a runner.
units, and accurate measurement estimates. Examples of sentences and annotations shown in Table 10.
In the second evaluation, we compare against Amazon Mechanical Turk (AMT) Annotators on a balanced set of 2,122 sentences randomly sampled from the test set. We show the results for both evaluations in Table 9.
In both evaluations, the model outperforms the human annotators on every task. For dimension prediction, the model led by 7.4-7.8 percentage points. Of the sentences where the dimension was correctly annotated, the model led by 33.5-39.9 percentage points on unit prediction. For sentences where both the model and human correctly predicted the dimension, the model predicted a number closer to ground truth  $66.2 - 78.8\%$  of the time.
4.6 Qualitative Analysis
4.6.1 Semantic Head Embeddings
In Figure 6 we plot the t-SNE embeddings of the sentences'  $h$ , the output of our text encoder. We label each  $h$  with the masked measurement's true dimension, unit and exponent of the number. In 6a we observe that most embeddings labeled by their true dimension tend to form tight clusters. In 6b we filter to only show embeddings that share the Length dimension and label them by their units. We find that clusters are organized by the relative magnitudes of their units: large (Kilometers, miles), medium (feet, meters), and small (millimeters, inches, centimeters). Further we see that yards appear close to other imperial units of feet and
miles. Finally, in 6c when embeddings are binned by the exponent of their values we observe that the left to right direction appears to capture the increasing magnitude of a number.
5.1 Numeracy
Multiple works have probed word embeddings like word2vec, GloVe, FastText (Naik et al., 2019) and contextual embeddings from models like BERT (Wallace et al., 2019; Zhang et al., 2020) or T5 (Pal and Baral, 2021) on a variety of numerical tasks like sorting, numeration, magnitude prediction, and common sense (Lin et al., 2020). Several works have targeted numeracy pretraining using left to right language models (Spithourakis and Riedel, 2018), CNN and RNN based models (Chen et al., 2019), pretrained transformers (Spokoyny and Berg-Kirkpatrick, 2020; Jin et al., 2021), for an overview (Thawani et al., 2021b).
Incorporating synthetic mathematical data augmentations (Geva et al., 2020) has improved question answering while numerical pretraining has been shown to lower masked language modelling perplexity (Thawani et al., 2021a). Either directly or indirectly units have been involved in providing more interpretable explanation of quantities (Chaganty and Liang, 2016), solving Fermi problems (Kalyan et al., 2021) and resolving numeric Fused-Heads (Elazar and Goldberg, 2019).
(b)
Figure 6: t-SNE visualizations of semantic head embeddings labeled by (left 6a) dimension, (middle 6b) units of length, and (right 6c) number exponent bin. Middle: we observe a clustering of imperial units: feet, yards, miles. Right: we show two directions where magnitudes of length and area measurements increase in value.
5.1.1 Numeracy Benchmarks
Several numeracy benchmarks have been proposed like quantitative reasoning in natural language entailment (Ravichander et al., 2019) and synthetic measurement estimation (Jin et al., 2021). The closest benchmark to our work is the Distribution over Quantities dataset (DoQ) introduced by Elazar et al. (2019). A rule-based method was combined with simple heuristics to build DoQ resulting in its high-coverage albeit also higher noise. Although, WiCo is smaller, it has much higher fidelity since it utilizes a feature used by editors of Wikipedia to automatically convert quantities into different units. Further, WiCo provides the whole sentence as context as opposed to triplets of words. Zhang et al. (2020) use artificial templates to probe models on DoQ and find little difference between numerically pretrained and frozen embeddings such as ELMo. In contrast, our findings show there is a significant gap on WiCo between fully finetuned models and their frozen counterparts.
The pretrained RoBERTa model we use in experiments was likely pretrained on data that included WiCo. Thus, it is reasonable to be concerned about inflated test performance. That said, the task we consider is distinct from the self-supervised task used to pretrain RoBERTa (i.e. masked word classification vs. masked number regression). Further, our experiments on directly probing RoBERTa to predict masked numbers and units showed poor performance – indicating, perhaps, that even if RoBERTa's pre-training set did include WiCo, RoBERTa did not memorize aspects of our test set relevant to masked number prediction, partially mitigating these concerns.
The human evaluation studies we conducted are
a quite limited 'guesstimating' task. The human annotators were not allowed to use any external information from searching the internet or looking up answers in knowledge-bases. Their total average completion time per question was 33 seconds. Furthermore, many annotators may not have strong intuition about measurements with unfamiliar and uncommon unit types. For these reasons it is not surprising that our models outperform the human annotators in this limited experiment. However, these human evaluation studies do help calibrate the difficulty of the MMP task on WiCo.
In this work we propose Masked Measurement Prediction, a new task that requires models to jointly predict masked numbers and units in running text. We motivate this task as an important extension of existing masked number-only prediction tasks that addresses their limitations and allows for better evaluation of numeracy in NLP models. In our study, we show that probing of traditional pretrained transformers exposes a gap in their understanding of contextualized quantities. Through careful quantitative and qualitative analysis of our new model, which directly reasons about underlying units and dimensions, we find that it is possible to learn good representations of measurements. For future work we aim to extend this dataset to cover more existing standardized units from organizations such as UNECE. We hope our MMP task encourages research into further development of better numeracy methodologies.
4United Nations Economic Commission for Europe
Tarik Arici, Kushal Kumar, Hayreddin Ceker, K K Saladi, and Ismail B. Tutar. 2021. Solving price per unit problem around the world: Formulating fact extraction as question answering. In KDD TrueFact Workshop.
Arun Chaganty and Percy Liang. 2016. How much is 131 million dollars? putting numbers in perspective with compositional descriptions. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 578-587, Berlin, Germany. Association for Computational Linguistics.
Chung-Chi Chen, Hen-Hsen Huang, Hiroya Takamura, and Hsin-Hsi Chen. 2019. Numeracy-600K: Learning numeracy for detecting exaggerated information in market comments. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6307-6313, Florence, Italy. Association for Computational Linguistics.
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020. Tabfact: A large-scale dataset for table-based fact verification. In International Conference on Learning Representations (ICLR), Addis Ababa, Ethiopia.
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matthew I. Beane, Ting-Hao Kenneth Huang, Bryan R. Routledge, and William Yang Wang. 2021. Finqa: A dataset of numerical reasoning over financial data. In EMNLP.
Yanai Elazar and Yoav Goldberg. 2019. Where's my head? definition, data set, and models for numeric fused-head identification and resolution. Transactions of the Association for Computational Linguistics, 7:519-535.
Yanai Elazar, A. Mahabal, Deepak Ramachandran, Tania Bedrax-Weiss, and Dan Roth. 2019. How large are lions? inducing distributions over quantitative attributes. In ACL.
Mor Geva, Ankit Gupta, and Jonathan Berant. 2020. Injecting numerical reasoning skills into language models. In ACL.
Vinh Thinh Ho, Yusra Ibrahim, Koninika Pal, Klaus Berberich, and Gerhard Weikum. 2019. Qsearch: Answering quantity queries from text. In SEMWEB.
Zhihua Jin, Xin Jiang, Xingbo Wang, Qun Liu, Yong Wang, Xiaozhe Ren, and Huamin Qu. 2021. Numgpt: Improving numeracy ability of generative pre-trained models. ArXiv, abs/2109.03137.
A. Kalyan, Abhinav Kumar, Arjun Chandrasekaran, Ashish Sabharwal, and Peter Clark. 2021. How much coffee was consumed during emnlp 2019? fermi problems: A new reasoning challenge for ai. ArXiv, abs/2110.14207.
Bill Yuchen Lin, Seyeon Lee, Rahul Khanna, and Xi-ang Ren. 2020. Birds have four legs?! numersense: Probing numerical commonsense knowledge of pretrained language models. ArXiv, abs/2005.00683.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR.
Nafise Sadat Moosavi, Andreas Rücklé, Dan Roth, and Iryna Gurevych. 2021. Learning to reason for text generation from scientific tables. arXiv preprint arXiv:2104.08296.
Aakanksha Naik, Abhilasha Ravichander, Carolyn Rose, and Eduard Hovy. 2019. Exploring numeracy in word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3374-3380, Florence, Italy. Association for Computational Linguistics.
Kuntal Kumar Pal and Chitta Baral. 2021. Investigating numeracy learning ability of a text-to-text transfer model. ArXiv, abs/2109.04672.
Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, and Zhiyuan Liu. 2019. NumNet: Machine reading comprehension with numerical reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2474-2484, Hong Kong, China. Association for Computational Linguistics.
Abhilasha Ravichander, Aakanksha Naik, Carolyn Rose, and Eduard Hovy. 2019. Equate: A benchmark evaluation framework for quantitative reasoning in natural language inference. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 349-361.
Subhro Roy. 2017. Reasoning about quantities in natural language. Ph.D. thesis, University of Illinois at Urbana-Champaign.
Mark Sammons, V.G. Vinod Vydiswaran, and Dan Roth. 2010. "ask not what textual entailment can do for you...". In ACL.
Sunita Sarawagi and Soumen Chakrabarti. 2014. Open-domain quantity queries on web tables: annotation, response, and consensus models. Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining.
Georgios P Spithourakis and Sebastian Riedel. 2018. Numeracy for language models: Evaluating and improving their ability to predict numbers. arXiv preprint arXiv:1805.08154.
Daniel Spokoyny and Taylor Berg-Kirkpatrick. 2020. An empirical investigation of contextualized number prediction. In EMNLP.
Avijit Thawani, Jay Pujara, and Filip Ilievski. 2021a. Numeracy enhances the literacy of language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6960-6967, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Avijit Thawani, Jay Pujara, Filip Ilievski, and Pedro Szekely. 2021b. Representing numbers in NLP: a survey and a vision. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644-656, Online. Association for Computational Linguistics.
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. Do nlp models know numbers? probing numeracy in embeddings. In Empirical Methods in Natural Language Processing.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Xikun Zhang, Deepak Ramachandran, Ian Tenney, Yanai Elazar, and Dan Roth. 2020. Do language embeddings capture scales? In FINDINGS.
We train and evaluate our models on Wiki-Convert (WiCo) (Thawani et al., 2021a), a dataset of English Wikipedia sentences where the number and unit in each sentence are human-annotated. The built-in template in Wikipedia can ensure the text contains numbers and units. For example,  $\{\{\mathrm{convert}|2|\mathrm{km}|\mathrm{mi}\}\}$  displays as 2 kilometres  $(1.2\mathrm{mi})$ . By searching within Wikipedia articles for the use of this template, the authors of WiCo automatically extract human-annotated numbers. To perform unit canonicalization, we use Pint  $^5$  whenever the mapping is unambiguous. In the ambiguous case, we manually inspect the sentence and perform the mapping. For example, we map the unit sqmi in WiCo to square miles to let pint perform unit
Figure 7: log-mae  $\downarrow$  by units of length. Predicting numbers for small magnitude units is easier than predicting numbers for their larger counterparts.
canonicalization. Table 10 shows examples of the extended dataset. The original dataset contains 924,473 sentences. The median sentence length is 106 characters, with 29,597 sentences has a length shorter than 20 characters. We provide statistics of the data in Table 1. For preprocessing we exclude sentences which have more than 64 tokens to have efficient computing memory or where the number is negative for simplicity. According to Thawani et al. (2021a) WiCo, "... has been extracted from Wikipedia dumps, which are licensed under the GNU Free Documentation License (GFDL) and the Creative Commons Attribution-Share-Alike 3.0 License." Thawani et al. (2021a) constructed WiCo with the intent that it be used to further numeracy NLP research. Our use of WiCo is aligned with its authors' goals.
B MLM Preliminary Unit Probe
We perform a preliminary unit probe shown in Table 11. The model predicts vastly different numbers when conditioned on different units. We observe a mean of 3086.8 and a standard deviation of 5820 for all the converted metric output.
We train our model GeMM U-Y on a single Nvidia GeForce RTX 2080 Ti for 4 hours and 14 minutes with a total parameter of 124,696,538.
C.1 Quantitative Analysis
In Figure 7, we show log-mae is relatively small for small magnitude units, which means predicting
5Pint: https://github.com/hgrecco/pint
Input: [UNIT]	m	km	ft	mi	yd	in	meters	kilometers	feet	miles	yards	inches	-
Output	200	10	200	2	100	1	200	20	20	2	50	3	-
Conversion factor	1	1000	0.3048	1609.34	0.9144	0.0254	1	1000	0.3048	1609.34	0.9144	0.0254	-
Metric Output	200.0	10000.0	60.96	3218.68	91.44	0.0254	200.0	20000.0	6.096	3218.68	45.72	0.0762	-
Mean (Metric Output)	-	3086.8 m
std (Metric Output)	-	5820 m
Table 11: Example outputs for Alex Honnold climbed for [MASK] [UNIT].
numbers for small magnitude units is easier than predicting numbers for their larger counterparts.
In Figure 4, we show confusion matrices of dimension and unit predictions by GeMM U-X.
D Human Annotators
D.1 Evaluation 1
The Technical Annotators have diverse scientific backgrounds ranging from chemistry, earth sciences, and computer science. One annotator is a native Chinese speaker, and two are native English speakers.
D.2 Evaluation 2
In Figure 8 we show the instructions provided along with the interface we designed for our MMP task. While the workers' geographic location were not provided to us by Mechanical Turk, we aimed to compensate the workers above the US federal minimum wage of  $7.25. We paid workers$ 0.15 per annotation with an average completion time of 33 seconds. This equates to an hourly rate of $12.80 after Mechanical Turk fees. Other demographic information is only provided by Mechanical Turk for an extra fee.
E Ethical Considerations
Like any system that makes predictions, those made by GeMM are not necessarily accurate and may be used by malicious actors to generate fake information to mislead their audience. Additionally, GeMM is an extension of RoBERTa and therefore inherits the biases learned during the training of RoBERTa. Our work focuses exclusively on English and Arabic numerals. As noted by Thawani et al. (2021a), the units in WiCo are heavily biased towards European and American units as they are over-represented in English Wikipedia.
Labeling Instructions
Instructions: For each sentence please give your best estimate for the number in the units. Do not look things up, certain questions are ambiguous and that's okay. Really important th. number will be interpreted in the units that you select! For number please just input the dig and decimals points without any spaces or commas.
1. 'My car weights [#NUM][UNIT].' Answer: Dimension=Mass, Unit=ton, Value=1
2. My brother is [#NUM][UNIT] tall. Answer: Dimension=Length, Unit=ft, Value=5.8
3. 'My house is [#NUM][UNIT] large.' Answer: Dimension=area, Unit=sqft, Value=1200.41
Figure 8: Left: Instructions for labeling task. Right: we show the interface used by the labelers
My building is [#NUM][UNIT] tall.
Please Guess the Dimension
Length[1]
Mass[2]
Area[3]
Velocity[4]
Power[5]
Please Guess the Number
and the Units
Quick Filter
meters  $(\mathfrak{m})^{[8]}$
miles (mi)[9]
centimeters  $(\mathsf{cm})^{[0]}$
kilometers  $(\mathrm{km})^{[q]}$
millimeters (mm)[w]
$\checkmark$  feet (ft)[e]
inches (in)[t]
yards (yd)[a]