Multi-Source (Pre-)Training for Cross-Domain Measurement, Unit and Context Extraction
Year: 2023

Abstract:
We present a cross-domain approach for automated measurement and context extraction based on pre-trained language models. We construct a multi-source, multi-domain corpus and train an end-to-end extraction pipeline. We then apply multi-source task-adaptive pre-training and fine-tuning to benchmark the cross-domain generalization capability of our model. Further, we conceptualize and apply a task-specific error analysis and derive insights for future work. Our results suggest that multi-source training leads to the best overall results, while singlesource training yields the best results for the respective individual domain. While our setup is successful at extracting quantity values and units, more research is needed to improve the extraction of contextual entities. We make the cross-domain corpus used in this work available online1.

text: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, pages 1–25 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: Numeric components such as counts, measurements and are crucial information for researchers across various disciplines. An automatic system for the extraction of numerical measurements (e.g. 10 %) and related "contextual" information such as measurands (e.g., concentration) and measured entities (e.g., chemical solution) for scientific literature can aid in the efficient construction of knowledge bases (Court and Cole, 2018) for downstream research tasks.
Ideally, the system should be able to handle multiple subject domains or even unseen domains, as relying on multiple specialized systems is inefficient and sometimes infeasible: For instance, each specialized model requires dedicated training and deployment resources. Further, the target-domain cannot always be known at inference time.
1https://github.com/liy140/ multidomain-measextract-corpus
Related Work. Most existing work tackle domain-specific measurement extraction problems. In the bio-medical field alone, there exist several publications that specialize in one particular subfield and text genre, for instance food laboratory tests (Kang and Kayaalp, 2013), narrative radiology reports (Sevenster et al., 2015) or diabetes test trials (Hao et al., 2016). Many topic-specific systems are not designed as sole measurement extractors, additionally offering comprehensive knowledge base construction or text processing functionalities (Swain and Cole, 2016; Dieb et al., 2015; Epp et al., 2021; Lentschat et al., 2020; Friedrich et al., 2020). Due to their specialization, it is difficult to apply them to novel domains without considerable adaption effort.
A few domain-independent systems have been developed, but they either offer limited context extraction capabilities (Soumia Lilia Berrahou et al., 2013; Mündler, 2021) or lack concrete definitions of the extracted contextual entity types (Foppiano et al., 2019; Hundman and Mattmann, 2017). Moreover, for these systems, no study of cross-domain generalization capabilities has been performed.
Harper et al. (2021)’s SemEval Task represents a key milestone for the progress of domainindependent measurement extraction research. 25 teams participated in the competition, thereby introducing the task, which has before mostly been solved through traditional NLP methods, to a wide range of recent architectures, among others also the now dominantly used pre-trained language model architecture (Devlin et al., 2019). Moreover, Harper et al. (2021) define the task in a domainagnostic manner and provide an annotated multidomain measurement extraction corpus with data sourced from the OA-STM Corpus (Elsevier Labs, 2015). However, due to its small data size (295 paragraphs), the corpus is not sufficient on its own for studying cross-generalization effects.
1
Contributions. To address the research gaps mentioned above, i.e. the need for cross-domain generalization as an evaluation criterion for the measurement and context extraction task, we aim to build a cross-domain measurement, unit and context extraction system. We make the following contributions:
• To facilitate multi-domain training, we expand the corpus published by Harper et al. (2021), creating a multi-domain, multi-source corpus for measurement, unit, and context extraction including two additional source domains. • We construct an end-to-end model pipeline based on pre-trained language models (Devlin et al., 2019) and achieve state-of-the-art performance comparable to the first placed MeasEval team (Davletov et al., 2021). • We study the effect of (a) adaptive intermediate pre-training (Gururangan et al., 2020) and (b) multi-source fine-tuning (Zhao et al., 2020) on cross-domain generalization. For (a) we apply full intermediate pre-training and adapter-based pre-training (Hung et al., 2021; Houlsby et al., 2019) using a curated multidomain task-adaptive pre-training corpus (Gururangan et al., 2020). For (b), we experiment with different pooled combinations of finetuning domains. • Finally, we carry out a task-specialized error analysis using entity-level analysis methods adapted from Fu et al. (2020) to determine concrete error sources for well-grounded model improvement.
In the following sections, we explicate our corpus construction approach (§2), the model architecture (§3), and domain adaption methods (§4). Finally, we present our experimental results (§5) coupled with the error analysis (§6) and a concluding discussion (§7).

heading: 2 A New Multi-Domain Corpus for Measurement Extraction
text: In this section we describe the creation of a multidomain corpus for measurement and context extraction. This will enable the investigation of crossdomain prediction performance.

heading: 2.1 Data Model and Source Corpora
text: The first step in corpus creation for measurement extraction is to decide on a data model that relates objects to be measured, values, and their con-
text. We adapt the data model and terminology as proposed by Harper et al. (2021), excluding the "Qualifier" and "Modifier" classes to increase the candidate pool for corpus expansion.
Figure 1 presents the resulting adapted data model for our multi-source corpus: A Quantity (Q) is made up of one or more numeric Values (V) and optionally a Unit (U). A MeasuredEntity (ME) is the object, event or phenomenon, whose quantifiable property is measured. This would be the MeasuredProperty (MP), i.e., the measurand that can be attributed to the measured object. Figure 2 shows the annotation of an example sentence according to the presented data model. The extended data model definition can be found in Appendix A.
For selecting the source corpora, we compiled a candidate pool consisting of three datasets from related work and one additional patent dataset created by the chemical company BASF SE. We then evaluated candidate datasets with respect to compatibility with the data model. Table 1 presents the evaluation summary. As such the resultant corpus is comprised of the MeasEval corpus (Harper et al., 2021), the Battery Material Patents (BM) dataset as well as the Material Science Procedural (MSP) corpus (Mysore et al., 2019).

heading: 2.2 Data Processing and Annotation
text: We apply several processing steps to normalize each source corpus with respect to the measurement extraction data model of Figure 1.
MeasEval Corpus. For the MeasEval corpus, in order to accommodate the limited input length of most pre-trained models, we split the paragraphs into sentences using spaCy2. We deal with particularities of scientific language, e.g., bibliographic references and abbreviations by applying custom segmentation rules.
BM Corpus. The BM dataset describes and classifies information regarding entities and properties of battery materials from patent claims. The original annotations specify the patent type of a claim (e.g., material claim vs. process claim) as well as phrase-level entity and relation information across 15 entity types and 13 relation types (e.g., stirrer elements, complexants, main metals). The entity types Value, Unit and Property can be directly mapped to entities defined in our data model, i.e. Quantity value (Q), Unit (U) and MeasuredProperty (MP) respectively. By contrast, there are multiple source entity types that can be mapped to the MeasuredEntity (ME) class. These are parsed through graph traversal: we follow the relations that are connected to Value entities, we thereby find their respective U, MP and ME. We save each claim separately and do not apply additional segmentation measures to preserve the unique structure of the patent style.
MSP Corpus. The MSP Corpus comprises 230 articles describing material synthesis procedures (MIT Open Source License, Mysore et al. 2019). Although the annotation scheme is comparable to the Battery Materials dataset, including entity types such as Material, Operation, Amount-Unit, Synthesis-Apparatus, Number etc., a more complex mapping would be required to cover the various semantic structures present in this dataset. For this reason, we opted to manually re-annotate the data instead of relying on entirely automatic re-labeling. To reduce the annotation effort, automatic labels
2https://spacy.io
using the original annotation data were created. As with the BM corpus, we follow all entities related to the entity Number and map them to our defined labels (see B).
The annotation process involved four non-native annotators from different scientific backgrounds and mixed genders.
We drafted a separate annotation guideline that (i) explained the task according to the MeasEval annotation guidelines and (ii) introduced datasetspecific instructions (Appendix C). We re-annotate all samples of the validation articles (89 sentences) and test articles (129 sentences), and a subset of the training articles (860 sentences) to limit the annotation effort. We use the evaluation split for NER provided by Mysore et al. (2019).
An inter-annotator-agreement (IAA) study validated the reproducibility of our guidelines, producing substantial agreement scores. This is described further in Appendix D.
Final Corpus. The final multi-domain, multisource corpus consists of the normalized version of the three corpora described above, totaling 2,389 sentences from scientific articles (MeasEval), 1,077 sentences from material synthesis procedures and 352 battery material patent claims (not segmented). An overview of the final corpus is given in Appendix E.

heading: 3 Extraction Architecture
text: We now describe our model setup which is designed to extract the entity and relation types described in the previous section.
We model the extraction as a two-step pipeline made up of two token-classification models, which we coin as Task 1 and Task 2. We first extract all Quantities (Task 1) and then simultaneously predict U, ME and MP (Task 2) based on each extracted Quantity. This cascading setup resolves the relation extraction problem of assigning the
context entities to the correct quantity span, as the data model allows for a deterministic, rule-based assignment of the relations between U, ME and MP (see Gangwar et al. (2021); Davletov et al. (2021)).
Figure 3 shows the extraction flow based on an example sentence: The information from the first task is input into the second task through special tokens [Q] and [/Q] which we wrap around the identified Q spans (see also Gangwar et al. (2021); Davletov et al. (2021)). For each identified Q, an enriched prediction sample is created, thereby allowing for overlapping entities and conditioning the unit and context entity extractor on one Q at a time. For Q extraction we use binary IO-tags (Liu et al., 2021). For Task 2 we use the BIO-tagging scheme. To accommodate the tokens [Q] and [/Q] which signal the identified Q spans from Task 1, we add them to the models’ vocabulary as special tokens, extending the embedding size by two. For training, we use cross-entropy loss over all classes and train Task 1 and Task 2 separately.
A drawback of this simple architecture is the fact that it cannot enforce the 1:1 relationships prescribed by the data model, since it is possible to predict more than one ME or MP. Further, we set the input sequence to the size of a single sentence to account for the one-sentence annotation window of the MSP and Battery Materials dataset.

heading: 4 Domain Adaption and Generalization
text: We experiment with a) adaptive pre-training and b) multi-source fine-tuning. Figure 4 summarizes the applied methods and resulting model configurations. With the exception of the training setting
with all sources, all shown configurations are applied to the models of both tasks.
Adaptive Pre-Training. This setup comprises a combination of pre-trained base models and intermediate pre-training: We use BERTBASE3 (Devlin et al., 2019) as the baseline model representing the canonical text domain, and SciBERT 4 (Beltagy et al., 2019), which we expect to be more closely related to the domains of our measurement extraction corpus, because it was pre-trained from scratch on scientific articles.
We create intermediately pre-trained variants for each of the two BERT-models using task-adaptive pre-training (TAPT) (Gururangan et al., 2020): we continue pre-training of the models on the unlabeled (training) data of our measurement extraction corpus. Thereby we aim to bring the models closer to the target domains of the task and induce increased task performance compared to the base models. We also apply adapter-based (Houlsby et al., 2019; Pfeiffer et al., 2020) intermediate pretraining to compare full TAPT against to a more
3https://huggingface.co/bert-base-uncased 4https://huggingface.co/allenai/scibert_scivocab_uncased
parameter efficient approach (Kim et al., 2021). As pre-training data, we create a "curated" (Gururangan et al., 2020) multi-source corpus comprised of the pooled data from all three training datasets. We enhance it with unlabled data from the same datasets to further increase the corpus size. As such we add all articles of the OA-STM dataset (Elsevier Labs, 2015), the source on which the MeasEval annotations are based. We remove paragraphs which appear in the test and validation splits of the MeasEval data through fuzzy string matching 5. Further, we add 1,128 Battery Materials claims which were excluded from the measurement extraction corpus due to lack of Quantity spans, and include the rest of the MSP data that was not re-annotated. This resulted in a pre-training corpus of approximately 630k words.
Multi-Source Fine-Tuning. To investigate the impact of multi-domain training, we also employ three experimental setups that are applied in the fine-tuning stage of model training. In each setup a model is trained on one, two, or all domains and evaluated on all applicable target domains. The first set-up single-source uses only a single data set. To build our multi-source corpora we pool multiple data sources from related tasks (Aue and Gamon, 2005; Zhao et al., 2020): As such, the second set-up duo-source uses the concatenation of two source datasets, e.g., BM + MeasEval, and the third set-up all sources uses all corpora. Due to considerable discrepancies between the Quantity annotation logic of the BM dataset and the other two datasets, no all sources setup was applied to Task 1.

heading: 5 Experiments
text: We perform various experiments investigating the generalization capabilities of our system depending on data selection and domain adaption techniques. The implementation details can be found in Appendix G.

heading: 5.1 Evaluation and Scoring.
text: For comparing the predicted outputs to gold spans, we use the competition evaluation script provided by the MeasEval authors (Harper et al., 2021), which is designed to jointly evaluate all sub-tasks by matching predicted Quantities to gold Quantities whilst taking into account the relationships
5https://github.com/seatgeek/thefuzz
between their respective contextual entities. To benchmark against the MeasEval competition results we report the competition metric Overlap F1 (see Harper et al. 2021). For all other results, we report the traditional token-based strict F1, which is frequently used to evaluate NER and sequencetagging tasks (cf. Fried et al. 2019; Swain and Cole 2016). To this end, we adapt the MeasEval evaluation script by including nervaluate’s 6 strict F1 implementation.

heading: 5.2 Results.
text: Table 2 shows a summary of the experiment scores evaluated on the entire multi-source corpus for Task 1 and Task 2. We observe that the models are rather
6https://github.com/MantisAI/nervaluate
accurate for the Q and U classes, while extraction performance for the contextual entities MP and ME is much lower. Below we study the results in more detail with regards to the influence of cross-domain fine-tuning and adaptive pre-training measures, and perform a dedicated analysis for the end-to-end performance of two pipeline compositions. The full result tables can be found in Appendix H.
Adaptive Pre-training Experiments. To study the effect of adaptive pre-training, we analyse the average F1 score by model type and task-adaptive pre-training setup (No PT vs. Full PT vs. Adapter PT): SciBERT vs. BERT: For both Task 1 and Task 2, we observe that both pre-trained and base SciBERT models achieve higher scores than their BERT counterparts (Figure 5, Figure 6). Full PT vs. No PT: When comparing different intermediate pre-training setups, we see no systematic patterns comparing Full PT and No PT performance. This is especially true for Task 2, where the average performance is quasi identical. Thus, we hypothesize that TAPT with limited pre-training data such as in our case is not sufficient for the more complex token classification task. Adapter PT: From Table 2 we see that adapter PT is often comparable or worse to Full PT and No PT. The performance drop is especially pronounced for the more challenging ME and MP class, while it remains comparable for the Q and U class.
Multi-Source Fine-Tuning Experiments. For the investigation of multi-source fine-tuning, we analyse the average F1 score by source domain and task (Table 4, Table 5): Cross-domain vs. in-domain: We observe that cross-domain fine-tuning is beneficial for overall generalization, while in-domain fine-tuning results in better in-domain performance. An exception to that is the rather specialized, low-resource BM domain. Its best Task 2 performance is achieved in the MSP+MeasEval →BM cross-domain setup. Similarly, leaving BM out from multi-source training setups results in higher overall scores, suggesting that overly specialized training signals negatively impact generalization.
End-To-End Evaluation. Table 3 shows the resulting E2E performance for selected model configurations, which allows us to assess the error propagation of the the cascading task flow.
We apply two separate Task 1 models for the multi-source setup, as we have not trained a model using all three datasets for Task 1 due to diverging Quantity annotation styles between BM and the other two data sources. We observe that the best overall end-to-end performance is achieved in the multi-source scenario, caused by superior performance on the MSP and BM domains. For MeasEval, we notice that the Unit extraction scores remain relatively high given the 0.2 drop in Quantity extraction. Remarkably, we see that Unit extraction works better in the MeasEval→MSP setup than in the in-domain setup.

heading: 5.3 Comparison with MeasEval Leaderboard
text: In Table 6 we compare a single-source and a multisource end-to-end setup against the highest-ranking team of the MeasEval competition. All models were selected based on the best strict F1 MeasEval target domain performance (as opposed to the overall performance) of the development data.
Our single-source setup performs on par with the winning team from Davletov et al. (2021), showing superior scores for the Q and U classes, comparable scores for the ME class, and inferior scores for the MP class and the relation classes. As such we achieve competitive results with an arguably simpler model setup: Davletov et al. (2021)’s quantity extraction model is based on an ensemble of multiple LUKE models and entity-aware self-attention (Yamada et al., 2020). Further, they use XLMRoBERTa-large for unit and context span extrac-
T2 classes: U, MP, ME Target domain
tion and apply multi-task learning with parallel task-specific layers for each entity type. Moreover, we work with a smaller input context of one single sentence, while the winning team applies a data augmentation technique, increasing the available context (Davletov et al., 2021). However, we point out that their model learns far more entity types at the same time (seven in total), as we only work with a subset of the MeasEval task definition.

heading: 6 Error Analysis
text: To better understand the challenges of the task and deficiencies of our system, we perform an indepth error analysis. We analyze error sources on a fine-grained entity level. To prevent the leakage of test data knowledge, we apply all error analysis methods on the development portion
Model Q U ME MP HQ HP O 1st place MeasEval Davletov et al. (2021) 0.861 0.722 0.437 0.467 0.482 0.318 0.551 Single-source setup (T1: MeasEval+SciBERT+Full PT; T2: MeasEval+SciBERT+No PT) 0.877 0.885 0.432 0.437 0.465 0.307 0.550 Multi-source setup (T1: MSP+MeasEval+SciBERT+Full PT; T2: All sources+SciBERT+No PT) 0.876 0.864 0.404 0.440 0.46 0.27 0.533
Table 6: Benchmarking against MeasEval leaderboard’s top team, scores correspond to MeasEval’s competition scoring overlap F1.
of the corpus using our best development model setup (Task 1: BM+SciBERT+Full PT & DuoSource(MSP+MeasEval)+Full PT; Task 2: All Sources+SciBERT+No PT). Due to the relatively high scores for the Unit class, we focus the analysis on the Q, ME and MP classes.
Entity Data Attributes. To detect model weaknesses related to the properties of entity spans, we draw on the notion of data attributes as defined by Fu et al. (2020): These are "[...] values which characterize the properties of an entity that may be correlated with the NER performance." (p. 6059). These values can be related to characteristics of the entity’s surface string (e.g., entity length) or its surrounding context (e.g., sentence length). We analyse the following attributes:7:
• Entity length (eLen): The number of tokens in an entity. • Sentence entity density (eDen): The number of entities in a sentence divided by the sentence length. Thus, paragraphs with multiple measurements and associated contextual entities will have a higher entity density than paragraphs with a single measurement. • Gold quantity distance (qDist): The character-level span distance of the gold entity to its associated gold Q. qDist only applies to
7We use Huggingface’s BertTokenizerFast based on SciBERT vocabulary for tokenization based attributes https: //huggingface.co/docs/transformers/model_doc/bert
the classes ME and MP. We have filtered out all cross-sentence entities for this attribute.
Analysis. In our approach, we first calculate the described data attributes. For (partial) matches and missing predictions we base the calculation on the gold entity span, for spurious predictions we base the it on the predicted span. Then, we average the attributes by match type, i.e., match, partial (match), missing and spurious, to allow the comparison of attribute averages between matches and errors. The last three columns of Table 7 show the distribution of match types by domain and entity class. We see that the number of matches is especially high for the Q class, while the number of errors is especially high in the ME class of the MeasEval and BM domains. The remaining columns show the grouped attribute averages by domain, entity class and match type. The bar charts indicate the relative magnitude of an attribute mean within one domain. We make the following observations:
• eLen: Partial matches occur particularly for longer entities. For instance, we observe that tokens such as brackets or dashes within a longer entity are often broken up and predicted with different labels (e.g., ”deionized (DI) water” is broken into ”deionized”, ”DI) water”). Further, spurious predictions are always relatively short, often shorter than the average eLen of matches. For MSP the missing MEs have a high eLen, suggesting that the model has difficulties extracting longer phrases. The same phenomenon holds for the MPs of the BM domain. • eDen: For both MeasEval and MSP spurious Qs are predicted for sentences with a lower entity density. Further, we observe that both missing MPs and MEs of the MeasEval data
and missing Qs of the BM data appear in sentences with higher entity density. This implies that the model may ’overlook’ entities when many potential entities are in one area. • qDist: The model mainly struggles with long range dependencies for the ME class: qDist shows the distance of the supposed gold ME to its root Q by match type. We see that our setup is good at predicting MEs that are close to their root Q, as qDist is rather small for matches. However, for higher qDist MEs the match type is often missing or spurious. This means that the model does not predict an ME at all (missing), or predicts a spurious one, probably closer to the root Quantity. This issue does not apply to MPs, as their qDist is much lower on average.

heading: 7 Conclusion
text: We have applied pre-trained language models to end-to-end measurement, unit and context extraction. While our setup exhibits good extraction performance for Quantities and Units, more research has to be done to improve the extraction of contextual entities. We have identified long-range dependencies of MEs as a particular error source.
In terms of cross-domain generalization and multi-source training, multi-source training produced the best overall results, while single-source training often yielded the best results for the respective target domain. An exception to this was the small BM dataset, for which we observed the best unit and context extraction performance in the cross-domain prediction setting. This is an indicator for domain generalization, especially for low-resource domains. However, this needs to be confirmed in additional experiments with a dataset comprising even more domains.
When comparing adaptive pre-training methods, the most consistent performance driver was the use of the SciBERT base model instead of the BERT base model. Further, we found adapter-based intermediate pre-training to be worse in most cases for both model types and tasks, which may be due to the task complexity. This theory is affirmed by the fact that we saw better adapter-based pre-training results for the simpler Quantity extraction.
Finally, we found non-conclusive results for the comparison of no pre-training versus full pretraining. The instability of results may be due to the limited size of our pre-training data, or the effect of catastrophic forgetting. Future work with a larger pre-training corpus may give clearer insights into this case.

heading: Limitations and Ethical Considerations
text: We would like to discuss the following limitations and ethical considerations:
In this paper, we investigated the cross-domain extraction performance based on a multi-source corpus. Our working assumption is that this corpus represents enough variety to support such a claim. However, we point out that the corpus is biased towards English scientific and patent language, as well as the chemical / material science subject domain. Further, we remark that the subjects distribution itself is biased towards the BM and MSP datasets as the the more varied MeasEval dataset only contains few examples for each of its 10 subjects. Consequently, a balanced corpus should have a more even distribution of both subject domains and language domains by increasing the size of the currently underrepresented domains and ideally including data from more than only the English language.
Further, despite having substantial IAA scores for the re-annotation of the MSP corpus, we often perceived the task as difficult and ambiguous and felt the limitations of only having two contextual entities, instead of the three as proposed by Harper et al. (2021). Yet, the low IAA score (0.334) for the excluded Qualifier entity suggests that including it may not have eased the task. Hence, it may be valuable to further the study of how the measurement extraction problem can be modelled to resolve some of the ambiguities for context extraction.
Finally, while we tried to stay as closely to the original annotation guidelines as proposed by Harper et al. (2021) as possible (with the exception
of the two cases explicated in Appendix C, there is a high likelihood of annotation drift. The reannotators of the MSP corpus were not involved in the original MeasEval annotation procedure and it is possible that the interpretation of the annotation guidelines was slightly different at places than the authors have originally intended. Our adaption of the annotation guidelines can be found at the end of this paper (Appendix J)

heading: Appendix

heading: A Extended Description of the Data Model
text: Below we explicate each of the entity types, relations and their associated cardinalities of our data model which are largely based on Harper et al. (2021)’s definitions.
1. Entities define what is to be extracted: • Quantity (Q): A quantity is made up of
a) one or more numeric values or counts signifying amounts or measurements and optionally b) a Unit (U) indicating the magnitude of the values. According to the MeasEval annotation guidelines, values and units are annotated in one span where possible. Contiguous values of a range or a list belong to the same Quantity span (e.g. "Possible beverage sizes are 200, 300 or 400 ml"). • MeasuredEntity (ME: A measured entity is the object, event or phenomenon, whose quantifiable property is measured. • MeasuredProperty (MP): A measured property is a quantifiable property of the MeasuredEntity, i.e., the measurand that can be attributed to the measured object.
2. Relations define how many entities can be extracted and how they relate to each other:
• HasQuantity (HQ): This relationship links the context entities to their respective quantities. This relation can be drawn from a MeasuredEntity to the Quantity, if no associated MeasuredProperty exists. Otherwise, it is drawn from the MeasuredProperty to the Quantity. The cardinalities of this relationship show that there can be at most one HasQuantity relation for any Quantity span, whereas any MeasuredProperty or MeasuredEntity can be linked to multiple Quantity spans. Consequently, there can be at most one MeasuredProperty and one MeasuredEntity linked to any Quantity span. Consider the sentence "The book was 600 pages long and weighed 0.5 kg.". Here, the MeasuredEntity "book" can be linked to two Quantity spans. • HasProperty (HP): This relation shows which MeasuredProperties can be attributed to a MeasuredEntity. While there

heading: B MSP automatic label suggestions for re-annotation

heading: C Annotation Guidelines for the Re-Annotation of the MSP dataset
text: We provide the complete annotation guidelines for the re-annotation of the MSP dataset as at the end of this document (see Appendix (J).
Below we explicate specific re-annotation guidelines, diverging from the original measurement extraction guidelines provided by Harper et al. (2021).
Specific re-annotation guidelines Over the course of the annotation procedure, the annotators have agreed on additional guiding principles to better capture the relationships between measurements and entities in the context of experiments as described by the synthesis procedures:
Material-centered annotation As a general rule, we prioritize the annotation of experimental participants over other attributes of the experimental procedure. In the sentence "The mixture of elements were heated in evacuated quartz ampoules at 1220 K [...]", we would annotate the "mixture of elements" as the MeasuredEntity of the Quantity "1220 K" as opposed to the "evacuated quartz ampoules".
Experimental conditions Temperatures, times or rates specify the conditions under which experimental operations are performed. We annotate the activity for which the conditions apply as the MeasuredProperty and experiment participants which are worked on under
these conditions as the MeasuredEntity (Table 9).
Transformations Experimental procedures often describe transformations of the MeasuredEntities before a measurable operation occurs. It is often not possible to pin-point one particular noun phrase that represents the entity to which the operation is being applied. Thus, we annotate all prior steps that are relevant for the operation as the MeasuredEntity. (Fig. 10)

heading: Quantity MeasuredProperty MeasuredEntity
text: Although these guidelines deviate from the original MeasEval annotation guidelines, we believe that these rules are appropriate exceptions to accommodate the nature of experimental procedures, as these rules promote more information regarding measurements to be extracted. This goes in the direction of the "multiple hypothesis hypothesis" proposed by the authors of the MeasEval task, wherein they postulate that different interpretations of contextual information can be useful in different downstream applications (Harper et al., 2021).
D Inter-annotator-agreement study
We conducted an IAA study for the re-annotation of the MSP dataset which spanned five rounds. For the annotation procedure we used the annotation tool prodigy 8. After each round, the IAA was
8https://prodi.gy/
analyzed both through comparing the agreement score and the annotations themselves. The final annotation was chosen by selecting the annotation on which most annotators agreed. When there was no agreement, a discussion with all annotators decided either on the solution that adhered most closely to the existing guidelines or an amendment to the guidelines. As agreement measures, we calculate Krippendorff’s Alpha coefficient (Krippendorff, 2004). To ensure comparability, we follow the same implementation steps as the MeasEval authors and calculate the disagreement on the charlevel using the python package simpledorff 9. Under that assumption, each character in an annotation sample is treated as a "markable" entity with its own label. Figure 7 shows the development of
agreement for the annotation of MeasuredEntity and MeasuredProperty over the five rounds The scores for the Unit and Value entities agreement were always near 1.0 and thus excluded from the analysis. The dip in agreement in round three was mainly due to a conflicting understanding of the supposed span length of MeasuredProperties. Having resolved this conflict, a "substantial" (Viera and Garrett, 2005) agreement > 0.67 could be achieved in round four and reproduced in round five.
Although there were a number of ambiguous cases, the structure and content of experimental descriptions is mostly simple and formulaic. This is reflected in the moderate to high IAA scores compared to the MeasEval IAA, where the scores for both ME (0.55) and MP (0.64) are lower.

heading: E Corpus Overview
text: Table 11 details the main characteristics of each dataset.

heading: F Domain similarity
text: Following the approach of Gururangan et al. (2020), we investigate the domain similarity of our datasets
9https://github.com/LightTag/simpledorff
MeasEval MSP Battery Materials
# Sentences/claims 1250, 415, 724 860, 89, 128 194, 86, 72 #Unigrams 38,897 17,062 16,788 #Unique unigrams 9,029 4,024 1,382 Ratio 23% 24% 8%
Quantity Total ents 882, 281, 499 1671, 195, 201 278, 118, 102 Unique/total ents 0.83, 0.89, 0.81 0.5, 0.72, 0.71 0.62, 0.69, 0.78
Measured Entity
Total ents 875, 273, 499 1669, 193, 199 278, 118, 102 Unique/total ents 0.7, 0.6, 0.7 0.42, 0.61, 0.67 0.36, 0.36, 0.55 Example ents ’cells’, ’electrons’,
’samples’, ’soil’ mixture’, ’solution’, ’reaction’, ’V2O5’
’secondary particles’, ’lithium metal oxide powder’,
’precursor’
Measured Property
Total entities 563, 179, 330 1379, 145, 157 263, 118, 99 Unique/total ents 0.7, 0.61, 0.71 0.28, 0.41, 0.48 0.2, 0.34, 0.31 Example ents ’n’, ’depth’, ’p’,
’odds ratios’, ’ratio’, dissolved’, ’dried’, ’calcined’, ’heated’
particle size distribution’, ’tap density’, ’sodium level’,
’average particle size’,
Table 11: Main characteristics of the datasets by data split (train, val, test)
by studying their vocabulary overlap. The vocabulary overlap is based on the ratio of shared unigrams which we gather by tokenizing the texts with scispacy 10.
The matrices in Figure 8 show the resulting vocabulary overlap of the three datasets. They highlight the similarity between the MSP and BM dataset, which is especially pronounced in the comparison of the 500 most common unigrams with an overlap of 89%. All in all, we assume that the MSP and BM corpus share the most similarity, followed by MeasEval and MSP and MeasEval and BM.
G Implementation Details
Below we lay out our implementation details for pre-training, and fine-tuning of the model setup.
Table 12 describes our computational infrastructure. We intermediately pre-train and fine-tune our base-models BERT-base-uncased
10https://allenai.github.io/scispacy/; en_core_sci_lg model
(bert-base-uncased) and SciBERT-uncased (allenai/scibert_scivocab_uncased) which both have a 12 hidden layers with a hidden size of 768.
Adaptive Pre-training Full intermediate pretraining was carried out using the masked language modeling script provided by the Huggingface Transformers Library 11. For adapter-based pre-training we use AdapterHub (Pfeiffer et al., 2020) as well as their script for masked language modeling 12. The hyperparameters are given in Table 13. Adapter config and reduction factor are adapter pre-training exclusive parameters. Except for the parameters in the table we use the default values provided by the script. For adaptive pretraining we did performed no systematic hyperparameter search, so there might be more optimal parameter settings.
Hyperparameter search for fine-tuning Hyperparameter tuning was performed using the ray.tune optimization framework for scalable hyperparameter tuning 13. The tuning details are shown in Tables 14. The training and validation loops are implemented with pytorch-lightning 14, a research framework built on pytorch 15. We train the models for Task 1 and Task 2 independently from each other, meaning that we train and tune our Task 2 models based on gold Quantities instead of prediction outputs from a Task 1 model. This is done by pre-enriching the Task 2 training sequences with
11https://github.com/huggingface/transformers/blob/main/ examples/pytorch/language-modeling/run_mlm.py
12https://github.com/adapter-hub/adapter-transformers/ blob/master/examples/pytorch/language-modeling/run_mlm. py
13https://docs.ray.io/en/latest/tune/index.html 14https://pytorch-lightning.readthedocs.io/en/stable/ 15https://pytorch.org/docs/stable/index.html
special tokens ([Q] and [Q]) based on gold Quantity spans which simulates a perfect Task 1 performance. For future work, it might be also interesting to train and tune on the end-to-end pipeline. We optimize the models based the development strict F1 score, which is calculated by comparing predicted and gold BIO-tag sequences. We find that most models have their best parameter setting at a learning rate of 1e-05 or 5e-5 and a batch size of 16 or 32. Only the adapter-based models benefit from larger training rates of 1e-4 or 2e-4.

heading: H Test Results Tables
text: Table 15 shows the complete results table on the test data of Task 1. Table 16 shows the complete results table on the test data of Task 2. All scores refer to the strict F1, not overlap F1.

heading: I Development Result Tables
text: Tables 17, 18, and 19 show the Task 1, Task 2 and end-to-end results on the development portion of the corpus. All scores refer to the strict F1, not overlap F1.

heading: J Re-Annotation guidelines for the Material Synthesis Procedural Text Corpus
text: Scientific knowledge is published and achieved in the form of unstructured texts. Numeric components in the form of counts, measurements and units (e.g. 500mg) and their contexts (e.g. Ibuprofen, dosage) are often crucial information for researchers across all domains. The goal of this annotation task is to prepare data for an end-to-end pipeline, which is able to extract quantities, units, measured objects and properties from texts, as well as the semantic relationships between each other. Section A) of this document describes how the entity and relation labels can be defined in a general setting. Section B) will provide guidance for the annotation of a specific dataset, the Materials Science Procedural Text Corpus by Mysore et al..

heading: A) General task guidelines
text: The entity and relation labels are described in the following table. They are a subset of the SemEval 2021 Task 8, MeasEval Basic Annotation Set.
• Number (N)
– DefinitionA numeric value or a count signifying an amount or measurement and contiguous specifiers (e.g. >, ˜). This is the root entity in each sample, i.e. other entities must always be able to directly refer to a number. Numeric values which do not signify a quantifiable amount (e.g. page numbers, citations, mathematical formulas) are not annotated.
– Example The patient weighted ˜100 pounds and was prescribed an Ibuprofen dosage of 500 mg.
• Unit (U)
– Definition The unit linked to the Number. To be annotated if available.
– Example The sick patient weighted 100 pounds and was prescribed an Ibuprofen dosage of 500 mg.
• measuredEntity (mE)
– Definition "A required (if possible) span that has a given [Number + Unit] either as its direct value or indirectly via a MeasuredProperty. Every Quantity should
ideally be associated with a MeasuredEntity. If no relevant information appears in the text, the Number can be standalone, but can have no other relationships. A MeasuredEntity can be related to either a MeasuredProperty by a HasProperty relationship, or to a Quantity by a HasQuantity relationship." (cited from SemEval 2021 Task 8 annotation guidelines, "Quantity" reference replaced with "Number"). This label describes the concept that is being quantified by the number (and the unit). In most cases the measuredEntity consists of one or more noun phrases (and their specifiers if they are in a contiguous span).
– Example The sick patient weighted ˜100 pounds and was prescribed an Ibuprofen dosage of 500 mg.
• measuredProperty (mP)
– Definition "An optional span associated with both a MeasuredEntity and a [Number]. Not every [Number] will be associated with a MeasuredProperty. A MeasuredProperty must be related from a MeasuredEntity by a HasProperty relationship, and must be related to a Quantity through the HasQuantity relationship." (cited from SemEval 2021 Task 8 annotation guidelines, "Quantity" reference replaced with "Number"). The measuredProperty can be interpreted as the "quantity-denoting target-word" of the number (definition from FrameNet). As such is it often a quantifiable specifier or attribute of the measuredEntity (e.g. volume, concentration, temperature etc.), but can also encompass longer target phrases.
– Example The patient weighted ˜100 pounds and was prescribed an Ibuprofen dosage of 500 mg.
Graph representation: The entity labels their relations can be depicted in a graph. This can be especially helpful when identifying the measured entity and measured property or verifiying one’s annotations.
Case 1: (N, U) <- hasQuantity <- (mP) <- hasProperty <- (mE)
Case 2: (N, U) <- hasQuantity <- (mE)
Each data sample must contain at least a Number. The other labels are only to be annotated if they are contained in the text. Below are a few hints and rules for the general annotation task. Quantity and the Number label will be used synonymously.

heading: A.1) Multi-class classification
text: Multi-entity classifications are possible, i.e. a measuredEntity for one Number can be a measuredProperty for another. This can be the case, because classification is always performed from the perspective of the root Number. For the same reason there can be measuredEntities or measuredProperties containing numbers (that are not the root number of the annotation sample). Examples
"The lowest input of odd nitrogen corresponds to 3.5-6.1 (x10-4) wt.% N accumulated over 3 byr and mixed into 1.5- 2.6 m, of soil."
N U mE mP 3.5-6.1 (x10-4) wt.% N lowest input of odd nitrogen
N U mE mP 3 byr lowest input of odd nitrogen accumulated
N U mE mP 1.5-2.6 m soil

heading: A.2) Span extent
text: We annotate measuredEntities and measuredProperties as completely as possible, i.e. using the longest coherent and informative text span. However, we do not annotate copula (e.g. were, have been etc.) prepositions or articles at the beginning or end of a span. Examples
"The earth surface temperatures have risen by 0.5 °C compared to baseline levels."
N U mE mP 0.5 °C earth surface temperatures risen
A.3) Duplicate measuredEntities mentions Some sentences will have multiple mentions of the same measuredEntity. We annotate the span that is closest to its root Number. Example
"The O2/N ratio was measured with the aforementioned machinery (O2/N = 2.8)."
N U mE mP 2.8 O2/N ratio

heading: A.4) Part-whole relationships
text: Fractions and percentages often describe partwhole relationships, where the fraction or percentage describe a partial characteristic of a bigger whole. For annotation, we mark the whole as the measuredEntity and the part as the measuredProperty. Examples
"The hamburger consisted of 30% patty and 10% cheese."
N U mE mP 30 % hamburger patty
N U mE mP 10 % hamburger cheese
"Steam activation was carried out by heating an amount of sample in a flow of 10% water vapor."
N U mE mP 10 % flow water vapor
Part-whole relationships can also be described without the use of fractions or percentages:
"The patty of the hamburger was 200g."
N U mE mP 200 g hamburger patty
Graph representation: hamburger -> hasProperty -> patty -> hasQuantity -> 200g

heading: A.5) Non-noun measuredProperties
text: MeasuredProperties can also be verbs or adjectives. To test whether a verb can be a measuredProperty, one can reformulate the sentence using the nominalized verb form valditate with the graph representation to check if all relations can be applied correctly. Examples
"The earth surface temperatures have risen by 0.5 °C compared to baseline levels."
Reformulated: There has been a rise of earth surface temperature by 0.5 °C compared to baseline levels.
"The patient weighed 100 pounds."
Reformulated: The weight of the patient is 100 pounds.

heading: A.6) Hints for the Unit entity
text: Ratios (e.g. weight ratio) and pH values are not considered units. Instead, they are labeled as measuredProperties.

heading: B) Specific guidelines for Mysore et al.’s Materials Science Procedural (MSP) Text Corpus
text: Originally, the MSP Corpus contains annotations regarding the materials, operations and conditions of experiments in materials science. To expand the existing MeasEval Dataset, we need to adapt these annotations to the above-introduced entities and labels. For this, the dataset was automatically processed beforehand using mapping rules for each preexisting label (e.g. all materials where labeled as measuredEntities). However, these automatically created labels are often incorrect and must be adjusted which is the main annotation task here.
Characteristics of the data: Each data sample is pre-labeled with at least a number and in most cases suggestions for the Unit, measuredProperty and measuredEntity are given. One data sample is created for each Number and its related measuredEntities and measuredProperties. Hence, sentences with multiple quantities and related contexts will yield as many data samples as there are Numbers in the text. Due to the specificity of this corpus, some additional rules apply. They are listed below.

heading: B.1) Number specifiers
text: Symbols and textual specifiers of Numbers are often not included in the label suggestion. Therefore, we must expand the Number-span to also contain these specifiers. Example: In the first example the suggested number would be "The patient weighted ˜ [100] pounds...", we would then extend the span to "The patient weighted [˜100] pounds...".
B.2) Removing irrelevant entities and adjusting spans Sometimes the there will be suggested entities, that are not related to the root Number. These false suggestions must be removed. Further, to adhere to the rule for maximum span annotation we adjust spans for measuredEntities and measuredProperties which can be extended. Examples
"The gel was ground to powders and then calcined at 400 °C in a muffle furnace under air atmosphere."
Suggested:
N U mE mP 400 °C muffle furnace, air calcinated
Corrected:
N U mE mP 400 °C powders calcinated
"The as-synthesized zeolites were calcined at 580 degC for 4 h under a flow of air."
Suggested measuredEntity = zeolites Corrected measuredEntity = as-synthesized zeolites

heading: B.3) Experiment procedures
text: A large fraction of this corpus’ Numbers describe experimental conditions, e.g. how long a solution was stirred. As a result, the quantities can often not be linked to an explicitly measured object, but only to the object that is being experimented on. Therefore, we mark these objects as measuredEntities and the experimental circumstances as measuredProperties. If the measuredEntity is explicitly given, we mark that as measuredEntity instead of the object that is impacted by the experiment. Examples
"The obtained sample was washed with absolute ethanol, and then dried at 60 °C for 10h."
N U mE mP 60 °C obtained sample dried
N U mE mP 10 h obtained sample dried
"The solution was modified by dissolving it in 10 wt% ethanol."
N U mE mP 10 wt% ethanol

heading: B.3.1) Experiment operations
text: We only mark procedural operations (e.g. added or dissolved) as the measuredProperty of a Number and a measuredEntity, if the measuredEntity is the main participant of the operation. Examples
"The solution was modified by dissolving it in 10 wt% ethanol."
In the example above we do not mark dissolving as the mP, because ethanol is not the component that is being dissolved.
"500 g of the sample was dissolved in 10 ml NaCl solution."
N U mE mP 500 g sample dissolved
N U mE mP 10 ml NaCL solution
Here the sample is the entity that is being dissolved, thus we can mark ’dissolved’ as its measuredProperty. Be careful that the operation marked as the measuredProperty has a proper relation to the Number span.
"The composite was ground, pressed and sintered at 300 °C."
N U mE mP 300 °C composite sintered
In this example, sintered is the operation which directly related to the temperature measure, whereas the other operations do not have a measuredProperty (e.g. what the pressure of the pressing was or how granular the grounding was).
"Copper (99,99%) was purchased from Sigma-Aldrich."
In this case ’purchased’ is not the target-word of the 99,99%, as it represents a purity measure and not an amount that was purchased. Thus, there is no measuredProperty in this sentence.
B.3.2) MeasuredProperty operations span Operations are often specified by additional descriptors, that are contiguous to the operation or in a separate span. In most cases we only annotate the operation as the measuredProperty, because the descriptors are semantically dependent (the so-called ’oblique nominal’) on the operation phrase, which is difficult to express within our annotation scheme. Examples
"The chemical was heated at 300 °C under constant airflow."
N U mE mP 300 °C chemical heated
Here, "under constant airflow" is dependent on "heated". We would need additional labels to capture these kind of multi-level relations, which exceeds the scope of this annotation scheme. In the case, that a contiguous span with multiple properties could be annotated, we proceed in the same manner and only annotate the highest level to stay consistent.
"The chemical was dried in air at 300 °C."
N U mE mP 300 °C chemical dried
Exception: Preceding adverbial and adjectival modifiers
We can add such descriptors which occur prior to the operation to the operation measuredProperty span, as they almost exclusively occur together contiguously, thus ensuring consistent annotations.
"The chemical was under magnetic stirring for 2 h."
N U mE mP 2 h chemical magnetic stirring

heading: B.3.3) Properties of operations
text: When attributes or specifiers of an operation are given we try to mark the main experiment participant as the measuredProperty as opposed to the operation itself. Example
"The chemical was calcinated at 300 °C with a heating rate of 10 °C per minute."
N U mE mP 10 °C per minute chemical heating rate

heading: B.4) Ambivalent relations
text: Some experimental descriptions do not explicitly name the measuredEntity, but e.g. only the result of the experimental operation. If this is the case and an experimental operation is also in the sentence, we can mark the experimental operation as the measuredProperty. Examples
"The enhanced form was obtained by calcination at 220 °C under a flow of air."
N U mE mP 220 °C calcination under a flow of air
This sentence does not mention the object that is being calcinated. Therefore, we annotate the operation as the measuredEntity. Note that we can also annotate "under a flow of air" as a measuredProperty here, because it can be directly linked to "calcination" and "220 °C" without being dependent on another measuredProperty.
"NH4OH solution was slowly added until the pH was 10."
N U mE mP 10 pH
This sentence does not mention, whose pH becomes 10. Because pH is not a unit, we can mark it as the measuredEntity.

heading: B.4.1) Coreferences
text: If the measuredEntity is mentioned as a coreference, but not explicitly, we annotate the coreference as measuredEntity. Example
"Finally, it was filtered, washed with water and ethanol, and vacuum-dried at 70 °C."
N U mE mP 70 °C it vacuum-dried

heading: B.4.2) Transformation of the measuredEntity
text: Synthesis procedures often describe transformations of the measuredEntities before a measurable operation occurs. It is often not possible to pinpoint one particular noun phrase that represents the entity to which the operation is being applied. Instead, we annotate all prior steps that are relevant for the operation as the measuredEntity. Examples
"To prepare C3N4-Pd composites, the as-prepared g-C3N4 was added into 100 mL ethanol and was sonicated for 2 h to obtain thin g-C3N4 nanosheets."
N U mE mP 2 h as-prepared g-C3N4 was added into 100 mL ethanol sonicated
N U mE mP 100 mL ethanol
B.5) Dealing with nested information in brackets Sometimes additional information about a measuredEntity is given in brackets. We only annotate measuredProperties that are directly related to both the Number and the measuredEntity. Examples
"20g of gold (99.99% purity) were ground."
N U mE mP 20 g gold ground
N U mE mP 99.99 % gold purity
"In a typical process, NiCl2*6H2O (0.173 g) was dissolved in a solution."
N U mE mP 0.173 g NiCl2*6H2O dissolved

heading: B.6) MeasuredEntity for Ratios
text: Ratios explain "how many times one number contains another" (Wiki). This should also be expressed in the measuredEntity of a ratio. If the two concepts described by the ratio are explicitly mentioned, annotate them (either in a contiguous span if possible, and separately if not). Examples
"At a weight ratio of 1:1, the MWCNT@MPC composite was mixed with sublimed sulfur."
N U mE mP 1:1 MWCNT@MPC composite was mixed with sublimed sulfur weight ratio

heading: B.7) Abbreviations
text: N U mE mP 100 ml Hydrochloric acid (HCl added
We try to include abbreviations into the entity span, if possible. Example
"Hydrochloric acid (HCl, 100 ml) was added to the mixture."

heading: Future Work
text: For future work, it might be also interesting to train and tune on the end-to-end pipeline. We optimize the models based the development strict F1 score, which is calculated by comparing predicted and gold BIO-tag sequences. We find that most models have their best parameter setting at a learning rate of 1e-05 or 5e-5 and a batch size of 16 or 32. Only the adapter-based models benefit from larger training rates of 1e-4 or 2e-4