CQE: A Comprehensive Quantity Extractor
Authors: Satya Almasian
Vivian Kazakova
Philipp Göldner
Michael Gertz
Venue: EMNLP 2023 Main (2023)
Keywords: quantities
quantity extraction
information extraction

Abstract: Quantities are essential in documents to describe factual information. They are ubiquitous in application domains such as finance, business, medicine, and science in general. Compared to other information extraction approaches, interestingly only a few works exist that describe methods for a proper extraction and representation of quantities in text.
 In this paper, we present such a comprehensive quantity extraction framework from text data. It efficiently detects combinations of values and units, the behavior of a quantity (e.g., rising or falling), and the concept a quantity is associated with. Our framework makes use of dependency parsing and a dictionary of units, and it provides for a proper normalization and standardization of detected quantities. Using a novel dataset for evaluation, we show that our open source framework outperforms other systems and -- to the best of our knowledge -- is the first to detect concepts associated with identified quantities. The code and data underlying our framework are available at https://github.com/vivkaz/CQE.

CQE: A Comprehensive Quantity Extractor
Satya Almasian; Vivian Kazakova', Philipp Göldner and Michael Gertz Institute of Computer Science, Heidelberg University, Germany {almasian, gertz Jeentreainoo uni uni-heiielbergg.... de [vivian. kazakova, goeldner uni uni-heideeeerr......mooeneaee de
Abstract
and Chakrabarti, 2014; Banerjee et al., 2009; Maiya et al., 2015). Consequently, their performance is measured based on the downstream task, and the quality of the extractor, despite its contribution to the final result, is not separately evaluated. There- fore, when in need of quantity extractor, one has to resort to number of open source packages, with- out a benchmark quantity extraction is rarely the main objective, the capabilities of the available systems and their def- inition of quantity vary based on the downstream task. As result, the context information about a quantity is reduced to the essentials of each sys- tem. Most systems consider quantity number with a measurable and metric unit (Foppiano et al., 2019). However, outside of scientific domains any noun phrase describing value is a potential unit, e.g.. "5 bananas". Moreover, a more meaningful representation of quantities should include their be- haviour and associated concepts. For example, the sentence "DAX fell 2% and S&P gained more than 2%" the value/unit pair (2, percentage) indi- cates two different quantities in association with different concepts, DAX and S&P, with opposite behaviours, decreasing and increasing. These sub- tleties not captured by simplified models. In this puper, we present a comprehensive quantity extraction (CQB) framework. Our system is capa- ble of extracting standardized values, physical and non-physical units, changes or trends in values, and concepts associated with detected values. Further- more, we introduce NewsQuant, new benchmark dataset for quantity extraction, carefully sclected from diverse set of news articles in the categories of economics, sports, technology, cars, science, and companies. Our system outperforms other libraries and extends on their capabilities to extract concepts associated with values. Our software and data are publicly available. By introducing a strong base- line and novel dataset, we aim to motivate further research and development in this field.
Quantities are essential in documents to de- scribe factual information. They are ubiqui- tous in application domains such finance, business, medicine, and science in general. Compared to other information extraction ap- proaches, interestingly only few works exist that describe methods for proper extraction and representation of quantities in text.
performance guarantee. Since
In this paper, we present such comprehen- sive quantity extraction framework from text data. It efficiently detects combinations of values and wnits, the behavior of quantity (e.g., rising or falling), and the concept quan- tity is associated with. Our framework makes use of dependency parsing and a dictionary of units, and it provides for proper normaliza- tion and standardization of detected quantities. Using a novel dataset for evaluation, we show that our open source framework outperforms other systems and to the best of our knowl- edge - is the first to detect concepts associ- ated with identified quantities. The code and data underlying our framework are available at https://git thub. com/ vivaaz/CQE.
1 Introduction
Quantities are the main tool for conveying factual and accurate information. News articles are filled with social and financial trends, and technical doc- uments use measurable values to report their find- ings. Despite their significance, comprehensive system for quantity extraction and evaluation framework compare the performance of such systems are not yet at hand. In the literature, a few works directly study quantity extraction, but their focus is limited to physical and science do- mains (Foppiano et al., 2019). Quantity extraction is often part of a larger system, where identifica- tion of quantities is required to improve numeri- cal understanding in retrieval textual entailment tasks (Roy et al., 2015; Li et al., 2021; Sarawagi "These authors contributed equally to this work.

---

2 Related Work
tion module. QFinder (Almasian et al., 2022) uses Quantulum3 in a similar way to demonstrate quan- tity search on news articles, but does not comment on the extractor's performance.
In literature, quantity extraction is mainly com- ponent of a larger system for textual entailment or search. The only work that solcly focuses on quantity extraction is Grobid-quantities (Foppiano et al., 2019), which uses three Conditional Random Field models in cascade to find value/unit pairs and to determine their relation, where the units are limited the scientific domain, a.k.a. SI units. (Roy et al., 2015)'s s definition of a quantity is closer to ours and is based on Forbus' theory (Forbus, 1984). A quantity is a (value, unit, change) triplet, and noun-based units are also considered. Extrac- tion is performed as step in their pipeline for quantity reasoning in terms of textual entailment. Although they only evaluate on textual entailment, the extractor is released as part of the CogComp natural language processing libraries, under the name Illinois Quantifier."
Another system that indirectly considers concepts is Xart (Berrahou et al., 2017). where instances of n-ary relations containing numerical values and unit are extracted and concepts are argument in these relations. However, the concepts are limited to a domain ontology with specific concepts of a given application domain.
A number of other works utilize quantity extrac- tion as part of their system. MQSearch (Maiya et al., 2015) extracts quantities with set of regu- lar expressions for search engine on numerical information. Qsearch (Ho et al., 2019) is another quantity search system, based quantity facts ex- tracted with the Illinois Quantifier. The works by (Banerjee et al., 2009; Sarawagi and Chakrabarti, 2014) 4) focus on scoring quantity intervals in census data and tables.
Two prominent open source libraries for quantity extraction are (a) Recognizers- Text (Huang al..
2017: Chen et al., 2023) from Microsoft and (b) Quantulum3. Recognizers- Text uses regular ex- pressions for the resolution of numerical and tem- poral entities in ten languages. The system has sep- arate models for the extraction of value/unit pairs for percentages, age, currencies, dimensions, and temperatures and is limited to only these quantity types. Moreover, it cannot proactively distinguish the type of quantity for extraction and the user has manually select the correct model.
3 Extraction of Quantities
In the following, we describe our quantity represen- tation model and detail our extraction technique.
3.1 Quantity Representation
In general, anything that has count is measur- able is considered a quantity. We extend upon the definition by (Roy al., 2015) to include concepts and represent a quantity by a tuple (v, u, ch, en) with the following components:
Quantulum3 uses regular expression to extract quantities and dictionary of units for normaliza- tion. For units with similar surface forms, a classi- fier based on Glove embeddings (Pennington et al., 2014) is used for disambiguation, e.g.. "pound" as weight currency.
1. Value (r): real number or a range of values, describing a magnitude, multitude, or duration, e.g., "the car accelerates from 0 to 72 km/h", , has range of (0, 72) and, "the car accelerated to 72 km/h" has single value come in different magnitudes. often denoted by prefixes, and sometimes containing fractions, e.g.. "He carns 10k curos" + v 10000, or "I "1/5 th his earnings"- + u = 0.2.
72. Values
Recognizers- Text is used in the work of (Li et al., 2021) to demonstrate quantity search, where the results are visualized in the form of tables or charts. They define quantity facts as triplets of (related, value & unit, time). Related is the quantitative re- lated information, close to our definition of concept. However, it is not part of their quantity model but rather extracted separately using rules. They utilize the quantity facts for the visualization of results but do not evaluate their system or the quantity extrac-
2. Unit (u): A noun phrase defining the atomic unit of measure. Units are either part of predefined set of known scientific and monetary types, or in more general case, are noun phrases that refer to the multitude of an object, e.g.. "2 apples" u = apple (Rijgersberg al., 2013). The prede- fined set corresponds either to (a) scientific units for measurement of physical attributes (c.g.. "2km" has the scientific unit (u = kilometre)),
phttps://giie htttts/s//giituub..m///CCCoooo htmm/Cogoom//cocooppeeo tree/ master/quantif Last accessed: October 16, 2023
hattps://githhr homm/nielstron/quasee accessed: October 16, 2023
Last

---

(b) currencies, as the unit of money (e.g.. "10k curos" refers currency). Predefined units can have many textual or symbolic surface forms, e.g., "euro", "EUR", or "E", and their normalization is a daunting task. Sometimes the surface forms coincide with other units, re- sulting in ambiguity that can only be resolved by knowing the context, c.g., "She weighs 50 pounds" is measure of weight (u pound- mass) and not a currency.
(v 0.4, u percenlage, ch down, en (German, DAX))
(0 0.1, percentage,
ch up, en (CAC40, France)),
3.2.1 Pre-processing
The pre-processing stage includes the removal of unnecessary punctuations, e.g., "m.p.h" - "mph", the addition of helper tokens, and other text clean- ing steps. An example of a helper token is placing minus in front of negative values for easy detection in other steps. These steps are done prior depen- dency parsing and POS tagging to improve their performance. Numerals that do not fit the definition of a quantity, such as phone numbers and dates, are detected with regular expressions and disregarded in further steps.
3. Change (ch): The modifier of the quantity value, describing how the value is changing, e.g.. "roughly 35$" is describing an approxi- mation. (Roy et al., 2015) introduce four cate- gories for change: (equal), ~ (approximate), (more than), and < (less than). These cate- gories are mainly describing the bounds for quantity. We extend this definition by account- ing for trends and add two more categories: up and down for increasing and decreasing trends, c.g. "DAX fell 2%" indicates a downward trend (ch = down), while "He weighs more than 50kg" is indicating bound (ch = ">)).
3.2.2 Tokenization
We perform a custom task word tokeniza- tion. Our tokenizer is aware of separator patterns in values and units and avoids between- splitting. For example, in the sentence "A beetle goes from to 80 km/h in seconds.", a normal tokenizer would split km/h -> (km, /, h) but we will keep the unit token intact. Another example is a numeri- cal token containing punctuations, e.g., F-3 where naive tokenization changes the value.
4. Concept (en): Concepts are subjects describing or relating to a value. A quantity mentioned in a text is either measuring property of a phc- nomenon, e.g., "height of the Eiffel Tower", in which case the phenomenon and the property are the concepts, or an action has been made, involving a quantity, e.g.. "Google hired 100 people", in which case the actor is what the quantity referring to. In the phrase "DAX fell 2% the quantity is measuring the worth of en = DAX or in "The BMW Group is invest- ing a total of $200 million" the investment is being made by en = BMW Group. Some- times a concept is distributed in different parts of sentence, e.g., "The iPhone 11 has 64GB of storage. + concept may may not be present, e.g., "200 people were at the concert" has no concept.
3.2.3 Value, Unit, and Change Detection
The tokenized text is matched against a of rules based on a dependency parsing tree and POS tags, set of 61 rules was created based on patterns observed in financial data and scientific documents and by studying previous work (Maiya et al., 2015; Huang et al., 2017). A comprehensive list of all rules can be found in the repository of our project. The rules are designed to find tokens associated with value, unit, and change.
iPhone 11, storage. A
Value/unit pairs are often sets of numbers and nouns, numbers and symbols, or number and adjec- tives in various sentence structures. For ranges, the rules become more complex, as lower and upper bounds need to be identified using relational key- words such as "from... to" or "between".
3.2 Quantity Extraction
Similar to previous work, we observed that quanti- ties often follow a recurring pattern. But instead of relying on regular expressions, we take advantuge of linguistic properties and dependency parsing. The input of our system is sentence, and the out- put is a list of detected quantities.
Changes are often adjectives or verbs that have a direct relation to number and modify its value. Sometimes symbols before a number are also an indication of change, e.g., "~ 10" describes an general, there are six change categories. ~ for approximate equality, = for exact equality, > for greater than bounds, < for less than
approximation.
Example 1: "In Europe, German DAX fell 0.4 pc, while the CAC40 in France gained 0.1." results in

---

bounds, up denoting an increasing or upward trend, and down for decreasing or downward trend. As an example of the extraction, we look at value, unit and change detection for the two quantities in Example 1. Note that in this stage the surface forms are detected and not normalized values, c.g.,
Keywords, such as for, of, at or by before or after a value point a potential concept. For example, "with carbon levels at 1200 parts per million" results in en = (carbon, levels). The noun and pronouns before and after such keywords are potential concepts.
"pe" versus "percentage".
The NOUN rule detects the surface form for the first value/unit pair, (0.4, pe). Here, the value has NUM as a POS-tag and is the immediate syntactic dependent of the unit token, which is a noun or proper noun.
2.
The entire subtree of dependencies with number (value) as one of the Icaf nodes is inspected find the closest verb related to the number. If no verb is found, then the verb con- nected to the ROOT is selected. The nominal subject of the verb is considered as the con- cept. In Example I, both "German DAX" and "CAC40 in France" are the nominal subjects of the closest verbs to the values in the text.
The LONELY NUM rule detects the value/umit pair for the second quantity, namely (0.1, None). If all other rules fail to find a value/unit pair, this rule detects the number with the POS-tag NUM. QUANTMOD DIRECT NUM detects the change, by looking at the verb or adjective directly before NUM tokens. Here, "fell" is trigger word for downward trend. For Example I, we thus have two extracted triplets with value, unit, and change.
Sometimes values occur in a relative clause that modifics the nominal, c.g., "maximum in- vestment per person, which is 50000" + cn (mariium, investment, per, person). In such case, the noun phrase before the relative clause is the concept, since the rela- tive clause is describing it.
(c- = 0.4, pc, ch- ffel))
(e = 0.1, u = None, ch = gained),
4. the numerical value in a sentence is not associated with the nominal of the sentence, then it is mostly likely related to the object. Therefore, the direct object of the verb is also a candidate, e.g., "She gave me raise of SIk", where "raise" is the direct object of the verb.
More examples are given In Appendix A.I. If no unit is detected for a quantity, its context is checked for the possibility of shared units. For the quantity (v = 0.1, u Nonc, ch gained) in Example 1, "percentage" is the derived unit, al- though not mentioned in the text. Shared units often occur in similarly structured sub-clauses or after connector words such as "and", "while", or "whereas"" The similarity between two sub-clauses is computed using the Levenshtein ratio between the structure of clauses. The structure is repre- sented by POS-tags, e.g., "German DAX fell 0.4 p "JJ NNP VBD CD NN" and "the CAC40 in France gained 0.1" " "DT NNP IN NNP VBD CD". This ratio is between 0 and 100, where larger values indicate higher similarity. If connector words are present and the ratio is larger than 60, the unitless quantity is assigned the unit of the other sub-clause, e.g., None becomes pc.
5. Finally, if the concept is not found in the pre- vious steps, and there a single noun the sentence, the noun is tagged as the concept, e.g., "a beetle that can go from 0 to 80 km/h in about 8 seconds, + en = (beetle).
From the list of candidate tokens for con- cepts, tokens previously associated with units and values are filtered and stopwords are re- moved, e.g.. "CAC40 in France" results in en (CAC10, France). Generally, a concept is repre- sented list of tokens.
3.2.5 Normalization and Standardization
Finally, the candidate values are filtered by logical rules to avoid false detection of non-quantities, e.g.. in "S&P 500", 500 is not quantity.
The final stage is the normalization of units and changes using dictionaries and standardization of values. The units dictionary is set of 531 units, their surface forms and symbols gathered from the Quantulum3 library, dictionary pro- vided by Unified Code for Units of Measure (UCUM) (Lefrançois and Zimmermann, 2018), and
3.2.4 Concept Detection
Concepts are detected in one of the following five
ways, ordered by priority:

---

list of units from Wikipedia, An example of an entry in this dictionary for "curo" is:
Various trigger words or symbols for bounds and trends are managed in the changes dic- tionary, where detected tokens for change are mapped to one of the allowed categories ~,= <, up, down. For example, the entry for equality is "=": [ "exactly", "just", "equals", "ootalling",="""
{ ("euro":
"surfaces": ["Euro", "Euros", "euro",
"euros"],
"symbols": ["EUR", "eur", c]}
The detected token span of a unit is normalized by matching against the different surface forms and symbols in the dictionary. The normalized form is the key of the dictionary and is added to the output, e.g., "curo" in the example above or "cm" giving "centimetre". The normalization makes the compar- ison of different units easier. Note that conversions between metric units is not supported. For example, "centimetre" kept as the final representation and not converted to "metre".
Evaluation
CQE is compared against Illinois Quantifier (IIQ), Quantulum3 (Q3), Recognizers Text (R- Txt), Gorbid-quantitiss (Grbd) and GPT-3 with few-shot learning (Brown al., 2020). From here on, the abbreviations are used to refer to the re- spective system. We first compare the function- ality of the models, then describe our benchmark dataset and compare the models on precision, re- call and Fl-score for quantity extraction. Finally, the unit disambiguation module is evaluated on custom-made dataset against Q3. Our evalu- ation code and datasets are available at https: //gi thub. com/
If the detected surface form is shared across multi- ple units, the unit is ambiguous and requires further normalization based the context. Since language models are great capturing contextual informa- tion, for this purpose, we train BERT-based clas- sifier (Devlin et al., 2019). There are 18 ambiguous surface forms in our unit dictionary, and for each a separate classifier is trained that allows to dis- tinguish among units based on the context. If an ambiguous surface form is detected by the system, the relevant classifier is used to find the correct normalized unit.
4.1 Comparison of Functionality
Table compares the functionality of the models in terms of different types of values, units, and changes, as well as normalization techniques. IIIQ is the only baseline that is able to detect changes in values but in limited setting that does not consider upward or downward trends. III performs normalization for currencies, however, scientific units are not normalized. Furthermore, it fails to detect fractional values and ranges. After our approach (CQE), Q3 has the most func- tionality and is the only model that correctly detects ranges and shared units and performs unit disam- biguation. On the other hand, Q3 disregards noun- based units, and although it is capable of detecting wide range of value types, makes incorrect de- tections of non non-quanntin non-quantitative values.
Compound units are also detected and normalized independently. For example, "kV/em" results in "kilovolt per centimetre', where VkV" and "em" are normalized based on separate dictionary entries. If no valid match in the dictionary exists, the sur- face form is tagged as noun unit and lemmatized, e.g., "10 students" gives = student. In some cases, the adjective before a noun is also part of the unit, e.g., "two residential suites" results in u residential suite.
The value dictionary contains the necessary information to standardize values to real numbers. More specifically, it contains surface forms for prefixes and suffixes of scales, e.g., "B: billion" or "n: nano", spelled out numbers in textual format, e.g., "fourty-two: 42", fractions in textual format, e.g., "half: 1/2", and scientific exponents, e.g., "102: 100'. This combination is used to convert values decimal format. Scientific notations with exponent and mantissa are converted to decimal values, "23F2 > 23
R-Txt has dedicated models for certain quantity types but fails to detect other types in the text, ig- noring ranges, scientific notation, and noun-based wnits. The unit normalization is limited to the quan- tity types and lacks disambiguation.
Grbd model's major shortcoming is the lack of value standardization. where fractions such "1/3" and scaled values like "2 billion" are not standard- ized correctly. The system is limited scientific units, and unit normalization works differently than another system, where the scientific units are con-
'tt http:://ewwwwiippee wikipedia. www.iii..gg/wwkiii. Convert/liste unnits Last accessed: April 17, 2023

---

Table Comparison functionality for various extractors.
Feature
Example
CQE
III
R-TxI
Q3
Grbd
Value
5k (5k)
Standardization
euros (5000)
/
Negative Values
(-5)
X
/
Fractions
1/3 the population (0.33)
/
X
/
V
Range
40-60 km/h (40-60)
/
X
X
/
J
Non-quantities
iPhone 11 (-)
X
X
X
V
Scientific Notation
1.9 10a (190)
X
X
X
Unit
Imm (mm)
V
Unit normalization
Imm (millimetre)
X
AEKIKK
X
Unit disambiguation
10 pound (sterling or mass?)
XX
X
Noun Units
200 people (people)
X
Shared Units
about $9 (both dollar)
5
Change
more than 100 (>)
5
X
X
X
Trends
DAX fell 2% (down)
X
X
X
x
Concept
AAPI. rose 2% (AAPL)
/
X
X
X
Table 2: Statistics of the number of sentences, quantities, and sentences with and without quantities in the NewsQuant and R-Txt datasets.
Dataset
#sent
#quantity
#sent with quantity
#sent w/o quantity
NewsQuant
590
904
475
115
R-Txt-currencies
180
255
178
R-Txt-dimension
93
121
77
14
R-Txt-temperature
34
34
R-Txt-age
19
22
18
verted the base unit, and values are also scaled accordingly. For example, "Imm" is converted to (0,001, metre). GPT-3 has lot of variability in the output and does not provide concrete and sta- ble functionality like the models discussed this section. Therefore, it is not further considered in this comparison.
have perfect agreement. A concept is span of tokens in the text and does have standardized representation, therefore, Cohen Kappa coefficient cannot be used. Instead, we report Krippendorff's alpha (Krippendorff, 2004), with the value of 0.79. In total, the annotators completely agreed on all elements for 62% of the annotations.
We additionally evaluate four datasets available in the repository of R-Txt for age, dimension, temper- ature, and c These datasets contain only unit/value pairs. The original datasets only con- tained tags for certain quantity type and would ig- nore other types, giving the R-Txi model an advan- tage. For example, in the R-Txt-currencies, only the currencies were annotated, and other quantities were ignored. We added extra annotations for all other types of quantities for fair comparison. For example, in the sentence want earn $10000 in 3 years" from the currency dataset, where only "$1 $10000" was annotated, we add "3 years". Statis- tics the number of sentences and quantities for each dataset are shown in Table The NewsQuant
4.2 NewsQuant Dataset
For qualitative comparison, we introduce new evaluation resource called NewsQuant, consisting of 590 sentences from news articles in the domains of economics, sports, technology, cars, science, and compunies. To the best of our knowledge, this the first comprehensive evaluation set in- troduced for quantity extraction. Each sentence is tagged with one or more quantities containing value, unit, change, and concept and is annotated by the two first authors of the paper. Inter-annotator agreements computed separately for value, unit, change, and concept between the two first authors on a subset of 20 samples. For the first three, the Cohen Kappa coefficient (Cohen, 1960) with val- ues of 1.0, 0.92, and 0.85 is reported. Value detec- tion a simpler task for humans and annotators
httts:///gttubb..m/iicreerr/ com/nicrosoft/
thuni Last accessed: Octoher 16, 2023

---

dataset is the largest dataset for this task containing over 900 quantities of various types. NewsQuant designed to test for the functionalities mentioned in Tablel and includes negative examples with non- quantity numerals.
davinci-003 model from the GPT-3 AP1 with a sequence length of 512, temperature of 0.5, and no frequency or presence penalty. For more details, refer to Appendix A.2. We are aware that with extensive fine-tuning and more training examples GPT-3 values are likely to improve. However, the purpose of this paper is neither prompt engineer- ing nor designing training data for GPT-3, and the few-short learning should suffice for baseline.
4.3 Disambiguation Dataset
To train unit disambiguation system, dataset of 18 ambiguous surface forms is created using ChatGPT For each ambiguous surface form, at least 100 examples are generated, and the final training dataset consists of 1,835 sentences with various context information. For more challenging surface forms, more samples are generated. For the list of ambiguous surface forms and the number of samples for each class, refer to Appendix A.3. A test dataset is generated in the same manner using ChatGPT, consisting of 180 samples, 10 samples per surface form. For more information on the dataset creation, please see Appendix A.4.
4.5 Analysis of Results
All the models are compared on precision, re- call, and Fl-score for the detection of value, unit, change, and concept. Disambiguation systems are also compared regarding precision, recall, and Fl-score of unit classification. Permutation re- sampling is used to test for significant improve- ments in Fl-scores (Riezler and Maxwell, 2005), which statistically more coherent in comparison to the commonly paired bootstrap sampling (Koehn, 2004). Results denoted with mark highly signifi- cant improvements over the best rest-performing base- line with a p-value < 0.01.
4.4
Implementation
CQE is implemented in Python 3.10. For de- pendency parsing, part-of-speech tagging, and the matching of rules SpaCy 3.0.9f is used. The unit disambiguation module, with BERT-based classi- for
4.5.1 NewsQuant:
Table 3 shows the result on the NewsQuant dataset. Since Q3, Grbd, and R-Txt do not detect changes, respective entries
fiers, is trained using spacy-transforme smooth intergeneration with other SpaCy modules. Parsers were created to align the output format of different baselines so that the differences in output representation do not affect the evaluation. For in- stance, for IIIQ, we normalize the scientific units and account for differences in the representation of ranges in Q3. If value is detected by baseline but not standardized or unit is not normalized to the form present in the dataset, applied for a unified output. These steps do not hurt the performance of the baseline models but rather align their output to the format of the benchmark dataset. For more details refer to Appendix A.2. Moreover, to keep up with the recent trends in NLP and the lack of a baseline for concept detec- tion, we introduce GPT-3 baseline. The GPT-3 model is prompted to tag quantities with 10 exam- ples for few-shot learning. Prompts and examples are available in our repository. We use the text-
left empty. CQE beats all baselines in each category by significant mar- gin, where most of the errors are due to incorrect extraction of the dependency parsing tree and part- of-speech tagging.
The second best model, Q3, scores highly for value detection, but misses all the noun base units and tends to overgeneralize tokens to units where none exist, e.g., in "0.1 percent 5884", Q3, detects "at" as percent per ampere-turn. Q3 makes mistakes different currencies and their normalization. We attribute this to their incomplete unit dictionary. R-Txi works well for the quantity types with ded- icated models, but all the other quantities are ig- nored or misclassified. One has to manually select quantity type for the R-Txt, therefore, we ran all the available model types each sentence, where any detected quantity is forced into the available model types, resulting in miss-classifications. III has trouble with compound units, e-g., "$2.1 per gallon" and tends to tag the word after a value as a unit, e.g., in "women aged 25 to 54 grew by 1 1%,, grew by is the detected unit. Although III is
is
'https://chat.opemt com/ Last accessed: October 16, 2023
/tttps://spacy.tttt io/ Last accessed: Octoher 16, 2023
httpps:/sppacy. hwt.spacy.i//uniiiii ioo/uiiverse/projectt
"https://platee http://plaa.omm.ppaa..... I Last accessed: Octo-
spacy-transformers Last accessed: October 16, 2023
her 16, 2023

---

Table 3: Precision, recall, and Fi-score for detection value, unit and change on NewsQuant.
Value
Value+Unit
Value+Change
Model
P
R
F1
P
R
F1
P
R
F1
CQE
92.0
91.9
85.6
85.5
88.2
88.1
Q3
65.0
83.3
73.0
42.1
53,9
47.2
III
50.6
66.0
57.3
32.8
42.8
37.1
44.2
57.6
50.0
R-Txt
59.7
82.2
69.1
29.6
40.7
34.2
Grbd
58.8
53.1
55.8
37.4
33.7
35.5
GPT-3
72.1
69.1
70.6
60.3
57.9
59.1
53.1
50.9
51.9
Table 4: Precision, recall and Fl-score for detection of value and unit R-Txt Datasets.
currency
dimension
temperature
age
Model
Detect
P
F1
Fl
F1
FI
CQE
82 6
85.9
84.2
85.5
87.6
86.5
94.3
97.1
95.7
91.3
95.5
93.3 3
Q3
69.2
84.7
76.2
76.9
93.4
84.3
91.7
97.1
94.3
91.3
95.5
93.3
IIIQ
Value
65.5
70.6
67.9
65.3
77.7
70.9
88.9
94.1
91.4
65.4
77.3
70.8
R-Txt
67.4
91.8
77.7
73.6
90.1
81.0
91.9
100.0
95.8
77.8
95.5
85.7
Grbd
46.6
35.3
40.2
75.8
59.5
66.7
84.0
61.8
71.2
60.0
27.3
37.5
GPT-3
50.5
54,9
52.6
80.2
80.2
80.2
93.5
85.3
89.2
92.3
54.5
68.6
78.1
79.6'
78.2
79.2
92.8
95.5
93.3
Q3
Value
29.5
36.1
32.5
56.5
68.6
61.9
61.1
76.5
74.3
82.6
86.4
84.4
IIIQ
+Unit
41.8
41.6
45.1
43.4
52.1
47.5
30.6
32.4
31.4
42.3
50.0
45.8
R-Txt
46.7
63.5
53.8
44.6
54.5
49.1
91.9
100.0
95.8
70.4
86.4
77.6
Grbd
24.9
18.8
21.4
44.2
34.7
38.9
32.0
23.5
27.1
40.0
18.2
25.0
GPT33
40.8
44.3
42.5
65.3
65.3
65.3
45.2
41.2
43.1
92.3
$4.5
68.6
Table Relaxed and strict matching, precision, recall and Fi-score for concept detection the NewsQuant.
subset of units, where percentages and compound units are mainly ignored.
GPT-3 achieves score close to Q3 for the de- tection of units and values and close to III for changes. Nevertheless, due to extreme hallucina- tion, extensive post-processing of the output is re- quired for evaluation, e.g., many of the values ex- tracted were not actual numbers and units were not normalized. Moreover, GPT-3 often confuses value suffixes with units, e.g., "billion" "million" and, despite the normalization prompt, fails normal- ize units and required manual normalization for most detections.
Relaxed Match
Striet Match
Model
R
F1
P
R
CQF
76.2
76.1
76.1
57.0
57.0
57.0
GPT-3
55.9
53.7
54.8
26.3
25.2
25.7
Table 6: Weighted micro-average precision, recall and F1-score on the unit disambiguation dataset.
Model
R
F1
CQE
89.9
89.4
88.1
Q3
57.33
57.78
54.46
4.5.2 R-Txt Dataset:
supposed to normalize currencies, practice the normalization is limited and often currency sym- bols are not normalized. Moreover, trends are ig- nored by IIIQ, and the model is biased to predict cquality (=) for most changes, and other changes are rare.
Evaluation results on the four quantity types of the R-Txt dataset are shown in Table 4, where our model once again outperforms all baselines on value+unir detection for all categories except for temperature. Nevertheless, for temperature, the R-Txt improvement
CQE is not statistically significant. The small size of the age and temper- ature dataset results in inconsistent significance testing. The closeness of value detection between models is due to the structure of the dataset. Most
The Grdb model detects the correct surface form for values in most cases, however, due to unstable standardization many standardized values are in- correct. Unit normalization is limited to a small

---

values have the surface form of a decimal, and the diversity of types like ranges, fractions, and non- quantities negligible. For more details on the error analysis and common mistakes each model on NewsQuant and R-Txt, see Appendix A.6.
5 Conclusion and Ongoing Work
In this paper, we introduced CQE, comprehen- sive quantity extractor for unstructured text. Our system is not only significantly outperforming re- lated methods as well as a GPT-3 neural model for the detection of values, unils and chunges but also introduces the novel task of concept detection. Fur- thermore, we present the first benchmark dataset for the comprehensive evaluation of quantity ex- traction and make our code and data available to the community. We are currently extending the extractor by improving the quality of edge cases and looking at the compatibility of our rule set to other application domains, e.g., medical text.
4.5.3 Concept Detection:
Finally, concept detection is evaluated on the NewsQuant dataset. Results shown in Table 5. Following the approach of UzZaman et al. (UzZa- man et al., 2013) for evaluation, strict and relaxed matches are compared. A strict match an exact to- ken match between the source and target, whereas a relaxed match counted when there is an over- lap between the systems and ground truth token spans. Based on the scores we observe that concept detection is harder in comparison to value unit detection. Even GPT-3 struggles with accurate pre- dictions. Our algorithm for concept detection is limited common cases and does not take into ac- count the full complexity of human language, leav- ing room for improvement in future work. More- over, in many cases, the concept is implicit and hard to distinguish even for human annotators, In general, our approach is more recall-oriented, as we keep any potential candidate from the concept detection step in the final result set, trying to cap- ture as many concepts as possible. Hence, there is a big gap between partial and complete matches. However, since the method rule-based, rules can be adjusted to be restrictive and frecus.
Limitations
Despite an extensive effort to account for most common cases, CQE is still mainly rule-based ap- proach, requiring manual feature engineering and rule- writing for unseen cases. This issue is more prominent in the case of concept extraction, where the order in which we apply the rules has di- rect impact on correct extractions. If the rule with higher priority finds a candidate, the rules further down the list are ignored. Although for humans identifying the correct rule to use is easy by con- sidering context and sentence formulation, such delicate difference in language is not casily cap- tured in rule-based systems. Moreover, CQE relies heavily on correct dependency parsing and POS tagging, and any error on the initial extraction prop- agates through the entire system. Consequently, even changes in the versions of the SpaCy model used for dependency parsing and POS tagging can produce slightly varying results.
4.5.4 Unit Disambiguation:
CQE is compared against Q3 (the only other sys- tems with disambiguation capabilities) in Table 6. Since the normalization of units is not consistent in the GPT-3 model and requires manual normal- ization, GPT-3 is left out of this study. All 18 clas- sifiers are evaluated within a single system. The results are averaged by weighting the score of each class label by the number of true instances when calculating the average. CQE significantly outper- forms Q3 on all metrics, and it is easily expendable to surface forms and units by adding new classifier. Since the training data is generated using ChatGPT, new classifier can be trained using our paradigm and data generation steps, as shown in Appendix A.4. For a detailed evaluation of each class, see Appendix A.5.
References
Satya Almasian, Milena Bruseva, and Michael Gertz. 2022. Qfinder: A framework for quantity-centric ranking. In SIGIR '22: The 45th International ACM SIGIR Conference on Research and Development Information Retrieval, Madrid, Spain, July 11 15, 2022, pages 3272 $277.AC.
Somnath Banerjee, Soumen Chakrabarti, and Ganesh Ramakrishnan. 2009. Learning Rank for Quan- tity Consensus Queries. In Proceedings of the 32nd Annual International ACM SIGIR Conference on Re- search and Development in Information Retrieval, SIGIR 2009, Boston, MA, USA. July 19-23, 2009. pages 243-250. ACM.

---

Soumia Lilia Berrahou, Patrice Buche, Juliette Dibie, and Mathieu Roche. 2017. Xart: Discovery of cor- related arguments of n-ary relations in text. Expert Syst. Appl., 73:115-124
Philipp Koehn. 2004, Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Nat- ural Language Processing, EMNLP 2004, meet- ing of SIGDAT, a Special Interest Group of the ACL, held in conjunction with ACI. 2004, 25-26 July 2004, Barcelona, Spain, pages 388-395. ACI..
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sustry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemer, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020. December 6-12, 2020, virtual.
Klaus Krippendorff. 2004. Content Analysis: An Intro- duction to Its Methodology (2nd ed.). Sage publica- tions.
Maxime Lefrançois and Antoine Zimmermann. 2018. The unified code for units of measure in RDF: cdt: ucum and other UCUM datatypes. In The Seman- tic Web: ESWC 2018 Satellite Events ESWC 2018 Satellite Events, Heraklion, Crete, Greece, June 3- 7, 2018, Revised Selected Papers, volume 11155 of Lecture Notes in Computer Science, pages 196-201. Springer.
Tongliang Li, Lei Fang, Jian-Guang Lou, Zhoujun Li, and Dongmei Zhang. 2021. AnaSearch: Extract, Re- trieve and Visualize Structured Results from Unstruc- tured Text for Analytical Queries. In WSDM '21, The Fourteenth ACM International Conference on Web Search and Data Mining, Virtual Event, Israel, March 8-12, 2021, pages 906-909. ACM.
Sanxing Chen, Yongqiang Chen, and Börje F Karlsson. 2023. Dataset and baseline system for multi-lingual extraction and normalization of temporal and numer- ical expressions. arXiv preprini arXiv:2303. 18103.
Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Mea- surement, 20(1))37-466
Arun S. Maiya, Dale Visser, and Andrew Wan. 2015. Mining Mcasured Information from Text. In Proceed- ings of the 38th International SIGIR Conference on Research and Development in Information Retrieval, pages 899-902. ACM.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Papers), J, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Confer- ence on Empirical Methods in Natural Language Pro- cessing (EMNLP), pages 1543, Doha, Qatar. Association for Computational Linguistics.
Luca Foppiano, Laurent Romary, Masashi Ishii, and Mikiko Tanifuji. 2019. Automatic identification and normalisation of physical measurements in scientific literature. In Proceedings of the ACM Symposium on Document Engineering 2019, Berlin, Germany, September 23-26, 2019. pages 24:1-2.:4. ACM.
Stefan Riezler and John T. Maxwell. 2005. On some pitfalls in automatic evaluation and significance test- ing for MT. Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Meusures for Machine Translation and/or Summarization, pages 57 64, Ann Arbor, Michigan. Association for Com- putational Linguistics.
Kenneth D. Forbus. 1984. Qualitative process theory. Artif. Intell.. 24(1-3):85-168
Hajo Rijgersberg, Mark van Assem, and Jan L.. Top. 2013. Ontology of units of measure and related con- cepts. Semantic Web, 4(1):3-13.
Vinh Thinh Ho, Yusra Ibrahim, Koninika Pal, Klaus Berberich, and Gerhard Weikum. 2019. Qsearch: Answering quantity queries from text. In The Seman- tie Web ISWC 2019- 18th International Semantic Web Conference, Auckland, New Zealand, October 26-30, 2019, Proceedings, Part I, volume 11778 of Lecture Notes in Computer Science, pages 237-257. Springer.
Subhro Roy. Tim Vieira, and Dan Roth. 2015. Reason- ing about quantities in natural language. Transac- tions of the Association for Computational Linguis- rics, 3:1-13.
Sunita Sarawagi and Soumen Chakrabarti. 2014. Open- domain Quantity Queries on Web Tables: Annotation. Response, and Consensus Models. In The 20th ACM SIGKDD International Conference Discovery and Data Mining, KDD '14, New York, NY, USA August 24 27, 2014, pages 711-720. ACM.
WenHao Huang. Zijia Lin, Chris McConnell, and Börje F. Karlsson. 2017. Recognizers-Text Recog- nition and Resolution of Numbers, Units, and Date/time Entities Expressed Across Multiple Lan- guages.
Knowledge

---

Naushad UzZaman, Hector Llorens, Leon Derezynski. James Allen, Marc Verhagen, and James Pustejovsky. 2013. Semeval-2013 task I: Tempeval-3: Evaluating Time Expressions. Events, and Temporal Relations. In Second Joint Conference on Lexical and Compu- kational Semantics (* SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013). pages 1-9.

---

A Appendix
A.2 GPT-3 and Few-shot Learning
A.1 Value, Unit and Change Detection Rule
To tag sentences using GPT-3, we use the few- shot learning paradigm by prompting the model to tag quantities and units in the text, given 10 dis- tinct examples. GPT-3 is mainly advertised as agnostic, few-shot learner, and we have not performed extensive fine-tuning. With the 10 ex- amples, we aim to account for a varicty of outputs, e.g., compound units, when no quantity is present, noun-based units, and prefixes for scaling the mag- nitude of value. Our full prompt is as follows, where the quantities with order of change, value, unit surface form, unit, concept. The unit surface form is used in post- processing if GPT-3 is not able to normalize the unit.
this section, we provide two additional examples for value, unit, and change detection and describe the logic behind a few other rules.
Example 2: "The Meged field has produced in the past about million barrels of oil, but its last well was capped due technical problems that have not been resolved."
NUM_NUM detects the compound number of million, where number, is the child of million, a noun, in the dependency tree.
output in numbered list,
QUANTMOD DIRECT NUM detects the re- lation between the adjective "about" to the value which is later identified as the change.
Tag quantities and units in the texts:
NOUN_ CUN NUM_ADD ADGPRIGHT_LT NOUN finds a noun or proper noun that has a number as child in the dependency tree. If there prepositions in the children of the noun, they are also considered part of the unit. In this case, [million, barrels, of, oil] are de- tected using this rule.
Sentence: Woot is selling refurbished, unlocked iPhone XR phones with 64GB of storage for about $330.
Answer:
1. =, 1.64, GB, gigabyte, storage
2. -, 330, $, dollar, iPhone XR phones
Sentence: The chain operates more than 600 supermarkets and less than 800 convenience stores.
The naming of the rules is preserved in the repository. From the combination of all rules, the candidate tokens [1, million] for value, [barrels, of, oil] for unit and [about] for change are extracted.
Answer:
1. >, 600, supermarkets, supermarkets, chain
Example 3: "They have $3500 month mortgage and two kids in private school."
2. <, 800, convenience stores, convenience stores, chain
NUM SYMBSY matches symbol followed by number. In this case, $3500 is detected.
Sentence: The spacecraft, which is about the size of school bus, flew into Dimorphos at a speed of about 4.1 miles per second, that's roughly 14,760 miles per hour (23, 760 kilometers per hour).
NOUN_NUM QUANT finds a number with noun or adverb as its head in the dependency tree. Here, have [mortgage, 3500, $, month).
Answer:
finds compound units with "per", "a" or "an" in between, c.g.. [3500, month, a]
1. -, 4.1, miles per second, mile per second, spacecraft
2. 14760, miles per hour, mile per hour, spacecraft
NOUN_NUM detects noun that has num- ber as a child, c.g., [kids, two].
3. 23760, kilometers per hour, kilometer per hour, spacecraft
The mentioned rules contribute to ex- traction of two candidate quantities: [S. 3500, a, month, mortgage], [two, kids]].
Sentence: And overnight dogecoin fell from
0.317 to 0. 308 a 2.8 percent drop.
Answer:

---

1. =, 1.0.317-0. 308, -, dogecoin =, 2.8, percent, percentage, dogecoin Sentence: This about minus 387 Fahrenheit (minus 233 Celsius).
Table Ambiguous surface forms, units associated with them and the number of samples in the training for each surface form and unit pair.
Surface
Units
# samples
:nswer:
1. -387, Fahrenheit, Fahrenheit,
cent, celsius
144
2. -233, Celsius, Celsius,
chinese yuan, japanese yen
100
kn
croatian kuna, knot
116
Sentence: WhatsApp more than 2 billion users send fewer than 100bn messages a day.
P
point, penny
149
R
south african rand, roentgen barn, bit foot, minute foot, minute inch, second inch, second celsius, coulomb fahrenheit, farad kiloton, byte, bel poise, pixel
100
b
127
Answer:
104
3, 2000000000, users, users, WhatsApp
104
2. <, 100000000000 messages, messages,
112
users
112
C
116
Sentence: This includes colors between red and blue wavelengths ranging between 390 and 700 nm.
100
kt =, 390-700, nm, nanometer, wavelengths
100
B
107
P
102
Answer:
dram pound
armenian dram, dram
180
pound sterling, pound-mas mass acre, year
131
113
Sentence: You don't t have a two-year bachelor's s degree or a six to eight-year phd degree.
number of samples per surface form and associated units for each surface form are shown in Table 7.
Answer:
1. =, 2, year, year, bachelors degree =, 6-8, year, year, phd degree
A.4 Disambiguation Prompts
To generate the dataset for disambiguation, ex- perimented with multiple prompts, using ChatGPT. The aim was to create training/test data in JSON- format, where the sentences are not duplicates or too simple. For this purpose, two sentences were formulated (one for each unit, in each surface form) and are used input examples of different contexts. The prompt explicitly asks for JSON format output and 20 samples, due the sequence length limi- tation ChatGPT. The final prompt is as follows, where UNIT1 and UNIT1 are replaced with different units with the shared surface and "SURFACE_FORM" denotes the ambiguous surface form:
Sentence: The price CO2 and fuel consumption are not clear.
Answer:
No quantities or units
Sentence: (sentence}
Answer:
[sentence] is replaced with the query sentence to be tagged. Nevertheless, the output of GPT-3 is not consistent and requires extreme post-procett post-processing. The post-processing includes cleaning the pre- dicted values to only include numbers, normaliza- tion of the units even if the unit is miss-spelled, e.g., "celsiu" instead of "celsius", "ppb" to "parts- per-billion", or "E" to "curo".
Create training set of 20 samples, for "UNIT1" " and "UNIT2", where in the text the surface form of the unit is always "SURFACE_FORM" but the unit is different. Output in JSON format as follows:
A.3 Ambiguous Surface Forms
("text": "Sentence 1", "unit": "UNIT1" }, ("text": "Sentence
In our unit dictionary, we encountered 18 ambigu- ous surface forms with different normalized units and collected least 100 samples for each. This list is not comprchensive and different scientific domains, more ambiguous cases might occur. The
"unit": "UNIT2" }}
The test dataset is created in the same manner. For certain units, multiple generations were required to get more complex sentences. In such cases, we

---

Table Error analysis of different extraction systems.
Mistake
Systems
Trouble detecting lemperature types, e.g.. celsius and fahrenheit are both denoted as degree.
Q3. IIIQ, GPT-3, Grbd
Dollar types are not identified, e-g., "hong kong dollar" and "new zealand dollar" dollar. Unit normalization does not work for the majority of the times.
Q3, IIIQ Grbd
IIQQ GPT-3, Grbd
Bias towards predicting for changes.
IIQQ GPT-3
Cryptecurencciss and rare ones are not recognized, c-g. Bitcoin or Markka. Sports units not recognized, e.g., pPg. rpg. apg.
Q3
Q3
Temporal values mistaken quantities, e.g., 2 pm. Compound units are rarely found, e.g., kph. Units in short sentences Problematic distinction between "year" and "year of age". Units are confused with concepts, e.g., building rate of S80 per - "sq m" as concept. recall
Txt-R, GPT-3, Grdb
Txt-R, GPT-3
recognized, e.g., rmb 10 usd 20.
CQE
CQE, Q3, GPT-3
GPT-3
limited quantity types.
Txt-R, Grhd
Delection of concepts where none exist.
CQE
Problem with correct standardization of values.
Grdh
Multi word compound units and most percentages are ignored.
Grdb
Unable to correctly distinguish different temperature units.
Grdb
specifically asked for sentences that do not start with "the" and are more complex. After each gen- cration, all examples were checked by the authors of the paper. Faulty samples with wrong units were removed. In some cases, surface forms were manu- ally altered to match the specifications of the task,
Since baseline models have various definitions of quantities and different unit dictionaries, it is ex- pected that their output representation varies al- though in many cases they refer to the same quan- tity. This is most prominent for the units, as differ- ent unit dictionaries or lack of unit normalization results multiple surface forms for a single unit. For example, the unit "celsius" in the benchmark dataset is detected as "degree celsius" " by Q3, "c" by Txt-R, "c", celsiu" und"iel by IIIQ, "cel- siu", "degrees celsiu" and "degree celsiu" by GPT- 3, "dege" and "degC" by Grbd. As demonstrated in the example. GPT-3, Grbd, and IIIQ not only have a single different name for the unit celsius but multiple surface forms. Although one can argue that multiple representations for the is pointing to a lack of normalization altogether, we chose to map these variations to single unified format, such that they are all pointing to the same unit. The only exception is for currencies in IIIQ, since their work explicitly claims that unit normalization is performed for currencies. For the entire pipeline and post-proe post-processing steps, refer to the evaluation repository.
A.5 Disambiguation per Class
A detailed evaluation of the disambiguation dataset is shown in Table 9, where precision, recall, and Fl-score are computed separately for each class. present
For each surface form, 10 examples in the test dataset. We noticed that distinguish- ing between "Japanese yen" and "Chinese yuan partially difficult for the BERT-based classi- fier since both them are currencies and used in similar contexts. Another difficult distinction is be- tween "penny" and "point", since monetary values and the stock market point unit are used in similar contexts. In comparison, in Q3 certain units almost never predicted, hence the multiple zeros in the evaluation results.
A.6 Error Analysis on NewsQuant and Txt-R Datasets
We analyzed the incorrect detection for all the mod- els and the common mistakes. Except for the points discussed in the main part of the paper, Table 8 provides overview of the remaining common mistakes and systems associated with them.
A.7 Output Post-processing
The goal of the post-prue proscesocsssing is to achieve a uni- fied representation, close to the benchmark dataset.

---

Table 9: Precision, recall and Fl-score for unit disam- biguation per class.
CQE
Q3
Class
P
F1
P
R
F1
knot
100
100
100
16.67
100
28.57
Toentgen
100
100
100
44.44
80
57. 14
barn
100
80
88.89
100
80
88.89
Japanese yen
60
100
75
100
100
100
inch
83.33
100
90.91
77.78
70
0.7368
Armenian dram
100
60
75
0
Chinese yuan
100
100
byte
66.67
cent
83,33
100
90.91
0
Croatian kuna
100
100
100
0
0
year
100
100
100
0
0
poise
100
100
100
100
100
100
south african rand
100
100
100
100
60
75
minute
80
80
80
60
90
72
bit
83.33
100
90.91
40
44.44
bel
80
100
88.89
100
kiloton
100
100
100
100
second
100
88.89
40
53.33
coulomb
100
100
100
100
100
100
dram
71.43
100
83.33
0
0
point
55,56
100
71.43
0
0
fahrenheit
100
100
100
100
100
100
celsius
100
94.74
0
pixel
pound-mass
90.91
pound sterling
100
100
100
100
80
88.89
foot
80
80
100
50
66.67
penny
100
20
33.33
0
0
farad
100
80
88.89
80
80
80
100
100
100
0


Reviews:
[{"review_id": "H3WlCwjnpv", "reviewer": "Reviewer_5wNU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper lays a strong mathematical foundation for the proposed methodology, ensuring its reproducibility. The experimental design and results are well-documented, enabling a clear understanding to the reader. Moreover, the introduction of a new benchmark dataset is a significant contribution to the field.", "reasons_to_reject": "While the paper presents a sound framework, including a visual representation of the proposed methodology would have been beneficial. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a novel framework for quantity extraction from unstructured text, utilising dependency parsing and a dictionary of units. The key contributions include a novel methodology for detecting concepts associated with identified quantities and introducing a new benchmark dataset for evaluating quantity extraction methods.", "excitement": 5, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YT60pDkdPR", "reviewer": "Reviewer_H5PX", "summary": "", "questions": "No specific question to ask.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work is of a refreshing return to ingenious solutions to problems seemly simple, but that can elude state of the art solutions. The paper is well written and easy to follow. The developed work is sound and the originality resides in narrowing the application to a single problem and solving it with a high precision and recall compared to baselines.\n", "reasons_to_reject": "The options and decisions taken could have been more explored and analyzed in greater detail.\n", "questions_to_authors": "No specific question to ask.", "paper_topic_and_main_contributions": "This paper describes an interesting solution for a seemly simple problem with a dedicated approach to extract quantities, units, modifiers, and concepts from textual sources.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "As far as I could tell, the presented references are enough.\n", "Typos_Grammar_Style_and_Presentation_Improvements": "As far as I could tell, there were no typos or grammar issues.\n"}, "extra_text": {}}, {"review_id": "7W3TjPW5w5", "reviewer": "Reviewer_vmbW", "summary": "", "questions": "A.\tPlease define “helper tokens”.\n\nB.\tCould you explain why the method computes the similarity between subclauses by comparing sequences instead of parsed trees? (section 3.2.3)\n\nC.\tLine 366-367: “the conversion between metric units is not supported”. Is the conversion of other units, such as volumes (ml, l) supported?\n\nD.\tWhy has Grobid-quantities method cited in the Related Work section (line 87) not been compared?\n\nE.\tAre units expressed by Greek letters handled by CQE?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the detailed comments, clarifying questions, and pointers to useful related work. The related work section will be adjusted accordingly. \n\nBefore answering the individual questions, first some general statements: \n\nWe were somewhat surprised by the low reproducibility score since both the code and the dataset were submitted alongside the paper (please refer to the zip file in supplementary material on open review platform). The other reviewers were quite happy with the access and reproducibility.\n\nWe understand the concern you raised regarding the comparison of models with different outputs. However, claiming that a \"comparison should be done on the minimal common subset\" is somewhat misleading. We argue that one should not design a benchmark to match the minimum requirement of the systems that came before, but rather design a comprehensive benchmark, encompassing essential information about a quantity even if previous systems are not able to extract all of it. To this end, the aim of post-processing is not to mimic the CQE results but rather to extract information in the standardized way of the benchmark.  Without the post-processing, models that do not perform unit normalization could have not been compared at all. \n\nThis also brings us to the comparison against the  R-Txt model. In the same way that lexical based systems like BM25 are still used as baseline against large language models with semantic understanding, one should not degrade and limit a model to a lower standard because another system cannot compete. The aim of this paper, beyond introducing a novel system, is to point out the lack of a benchmark and consolidate definitions and evaluation systems for quantity extraction, and the R-Txt model with such vast limitations is a good demonstration of the importance of this task.\n\nWe also appreciate your feedback regarding the clarification of the notation and agree that we should elaborate more on it. \nIn simple terms, a quantity mentioned in the text is either measuring a property of a phenomenon, e.g., ''height of the Eiffel Tower\", in which case the phenomenon and the property are the concepts, or an action has been made involving a quantity, e.g, \"Google hired 100 people\", in which case the actor is what the quantity is referring to. \nOur definition of change aligns directly with IllQ, which, to quote directly from their paper, says “A change specifies how the parameter is changing, e.g., increasing.” \n\nIn the following, we will address the questions raised:\nA. CQE relies heavily on POS tagging and dependency parsing performed by the SpaCy library, where certain patterns in text result in incorrect extractions. For example, with negative values such as  -100, the token \"-\" is treated as a punctuation separate from the value. To this end, during pre-processing, we change \"-\" to minus for a correct extraction.\n\nB. Investigating the parsing tree is also a viable way of looking at subclauses. However, looking at part-of-speech tags provide a fast and effective method in our case. We did not investigate alternatives. \n\nC. While metric units can be converted based on conversion factors, conversions are not performed and are not the focus of this study. The reason for this decision is two-fold: 1) Converting all units to a base unit would remove the important unit granularities that are specific to certain domains. 2) Inter-unit conversion is achievable by libraries or a dictionary of scaling factors and is not a novel contribution of the system. \n\nD. Very good point and one we should have mentioned in the paper. Upon the original submission of the paper, we were not able to run the code from the Gorbid quantities. However, the repository was updated two months ago and we now performed additional evaluations, which we will include in the paper. We include the results in the following:\n\nNewsQuant: \n\n\tP\tR\tF1\tmodel\ttype\n\t58.8\t53.1\t55.8\tGorbid\tvalue\n\t37.4\t33.7\t35.5\tGorbid\tvalue+unit\n\nR-Txt-age:\n\n\tP\tR\tF1\tmodel\ttype\n\t60.0\t27.3\t37.5\tGorbid\tvalue\n\t40.0\t18.2\t25.0\tGorbid\tvalue+unit\n\nR-Txt-temperature:\n\n\tP\tR\tF1\tmodel\ttype\n\t84.0\t61.8\t71.2\tGorbid\tvalue\n\t32.0\t23.5\t27.1\tGorbid\tvalue+unit\n\nR-Txt-dimension:\n\n\n\tP\tR\tF1\tmodel\ttype\n\t75.8\t59.5\t66.7\tGorbid\tvalue\n\t44.2\t34.7\t38.9\tGorbid\tvalue+unit\n\nR-Txt-currency:\n\n\tP\tR\tF1\tmodel\ttype\n\t46.6\t35.3\t40.2\tGorbid\tvalue\n\t24.9\t18.8\t21.4\tGorbid\tvalue+unit\n\nComparing the result with the tables in the paper shows that Gorbid quantities performs on par with other baselines. The model has a particular problem with the detection of currencies and percentages in the text. The unit and value normalization has different meanings in their system, the authors do not normalize the name of the units to a unified representation, they rather convert metric units to a base unit (as indicated in the answer to C) and scale the values accordingly. \n\nE. Yes, they are.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The extraction of quantities, units, and the entity measured is a difficult problem that has many applications in science, technics, and the general domain. There are relatively few methods and corpora available. This paper describes a new work for the English language that will be made available on GitHub. \n\nThe method is compared to four state-of-the-art methods that it outperforms. The paper is globally clear and convincing. The experiments are detailed. The result analysis section and the A6 appendix provide relevant hypotheses and examples.", "reasons_to_reject": "The notion of ‘concept’ is insufficiently described and formalized. Concepts are defined as “either properties being measured or entities that the value is referring to or is acting upon”, which is vague. The CQE method is designed “to capture as many concepts as possible.” This has consequences for the manual annotation of the NewsQuant corpus where the inter-annotator agreement is low, and for the comparison of CQE and GPT3 (Table 5), which is based on a questionable reference.\n\nMore generally, a more formal definition would make the evaluation of the method and, furthermore, its comparison with other methods more robust. It would also broaden the scope of application. As a source of inspiration, the authors could look at SSN and SOSA that distinguish between the measured property and the entity (or part of entity) observed and spatiotemporal conditions.\n\nThe same remark applies to ‘change’. IllQ achieves lower performances for Value+change extraction. How much could it be due to a different understanding of what ‘change’ is? The definition of IllQ for the ‘change’ entity would be needed here.\n\nIt is standard to provide the annotators with written guidelines, along with the corpus distribution. It would improve the annotation quality and reusability.\n\nThe reader understands that the outputs of the methods are not directly comparable, and some post-processing is applied to unify them that add information to the direct output. This is not a correct procedure, whatever is the post-processing is. The comparison should be done on the minimal common subset instead of “improving” the existing methods to mimic the CQE results.\n\nMore precisely, the comparison of units extracted by IllQ and Q3 and by CQE as described in section 4.4 Implementation seems unfair and must be clarified. The reader understands that the outputs of the three methods are not directly comparable, and some post-processing is applied to normalize and unify them. The paper must provide the performance score of the extraction of units of the three methods without alteration for a fair comparison. The post-processing needed for comparing the normalized extracted units must be detailed and show that no bias is introduced. The mention “Unit normalization does not work for the majority of the times” for IllQ and GPT3 should be replaced by “Unit normalization not available” to be consistent with Table 1 (line 10).\n\nAnother postprocessing is added to R-Txt model (Line 478-479) that adds extra annotations for all types of quantities. This makes R-Txt and CQE methods not comparable. The evaluation should have been done for the only quantities that R-Txt can predict.", "questions_to_authors": "A.\tPlease define “helper tokens”.\n\nB.\tCould you explain why the method computes the similarity between subclauses by comparing sequences instead of parsed trees? (section 3.2.3)\n\nC.\tLine 366-367: “the conversion between metric units is not supported”. Is the conversion of other units, such as volumes (ml, l) supported?\n\nD.\tWhy has Grobid-quantities method cited in the Related Work section (line 87) not been compared?\n\nE.\tAre units expressed by Greek letters handled by CQE?", "paper_topic_and_main_contributions": "The paper presents a new method called CQE for the extraction of mentions of quantity, unit, and concept in textual documents. CQE consists of a set of 61 syntax-based rules and a BERT-based classifier for unit disambiguation. The classifier is trained with examples automatically generated using ChatGPT. \nThe paper also presents NewsQuant, which is a new corpus manually annotated with quantity, unit, and concept mentions where quantity and units are normalized.\nCQE is compared to three other quantity extractors and GPT3. The experiments show that CQE outperforms them on NewsQuant.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "NA"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "About the formalization of units and linked concepts, these three papers could be useful to look at.\n\nRijgersberg, H., Van Assem, M., & Top, J. (2013). Ontology of units of measure and related concepts. Semantic Web, 4(1), 3-13.\nJanowicz, K., Haller, A., Cox, S. J., Le Phuoc, D., & Lefrançois, M. (2019). SOSA: A lightweight ontology for sensors, observations, samples, and actuators. Journal of Web Semantics, 56, 1-10.\n\nCompton, M., Barnaghi, P., Bermudez, L., Garcia-Castro, R., Corcho, O., Cox, S., ... & Taylor, K. (2012). The SSN ontology of the W3C semantic sensor network incubator group. Journal of Web Semantics, 17, 25-32.\n\nAbout the extraction of units, measures, and entities, the reference [Berrahou et al., 2017] may interesting to look at.\n\nBerrahou, S. L., Buche, P., Dibie, J., & Roche, M. (2017). Xart: Discovery of correlated arguments of n-ary relations in text. Expert Systems with Applications, 73, 115-124."}}]

Author rebuttal:

Meta-review: This paper introduces a novel framework for extracting quantities from unstructured text using dependency parsing and a unit dictionary. It also presents a new benchmark dataset for evaluating quantity extraction methods. Its soundness and excitement are generally well-supported by reviewers because of the simplicity and insight provided by this work. Two major areas of improvements are brought by one of our reviewers. (1) The definitions of "concept" and "change" are important for researchers and annotators, which are not articulated very clearly in the manuscript; (2) the use of post-processing to convert output from existing works may lead to unfair comparisons so it's suggested that the authors include a thorough explanation of the post-processing steps.
Decision: Accept-Main