architectures in almost all labels with an average F1 score of 99.23%. Average precision and recall are 99.13% and 99.33%, respectively. We noticed that both <base>, <pow> and <time> have lower f1-score. While <base> and <pow> require more training data, <time> expressions may overlap with <number> suggesting more contextual information should be introduced.

<table><caption>Table 6.6: Evaluation scores (%) of the Values ML model with holdout set. Support (Supp) indicates the number of labels in the training data. Values in bold indicate the highest score. P: precision, R: recall.</caption><thead><tr><th rowspan="2">Label</th><th colspan="3">CRF</th><th colspan="3">BidLSTM_CRF</th><th colspan="3">BidLSTM_CRF FEATURES</th><th colspan="3">SciBERT</th><th rowspan="2">Supp</th></tr><tr><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th></tr></thead><tbody><tr><td>&lt;alpha&gt;</td><td>98.06</td><td>96.03</td><td>92.02</td><td>97.67</td><td><strong>99.53</strong></td><td>98.58</td><td>97.82</td><td><strong>99.53</strong></td><td>98.66</td><td><strong>98.59</strong></td><td><strong>99.53</strong></td><td><strong>99.05</strong></td><td>126</td></tr><tr><td>&lt;base&gt;</td><td><strong>99.91</strong></td><td>92.31</td><td><strong>96.00</strong></td><td>96.92</td><td>92.31</td><td>94.52</td><td>96.92</td><td>93.85</td><td>95.32</td><td>90.40</td><td><strong>98.46</strong></td><td>92.88</td><td>13</td></tr><tr><td>&lt;number&gt;</td><td>97.50</td><td><strong>99.88</strong></td><td>98.36</td><td>99.24</td><td>99.34</td><td>99.29</td><td>99.21</td><td>99.38</td><td>99.30</td><td>99.48</td><td>99.31</td><td><strong>99.40</strong></td><td>811</td></tr><tr><td>&lt;pow&gt;</td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td>92.92</td><td>92.31</td><td>92.47</td><td>90.28</td><td>93.85</td><td>91.90</td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td><strong>100.00</strong></td><td>13</td></tr><tr><td>All (micro avg)</td><td>95.79</td><td>99.27</td><td>97.50</td><td>98.90</td><td>99.17</td><td>99.03</td><td>98.86</td><td>99.25</td><td>99.05</td><td><strong>99.13</strong></td><td><strong>99.33</strong></td><td><strong>99.23</strong></td><td></td></tr></tbody></table>

## 6.4 Conclusions

Our approach to property and conditions extraction emerges as a comprehensive strategy to address the diverse intricacies across different domains. The development of Grobid-quantities underscores our commitment to providing a versatile solution for extracting properties in a flexible way across diverse materials science domains. Our combined approach balances specificity and adaptability, enabling us to navigate the complexities of diverse domains while maintaining a robust foundation. As we continue to refine and enhance our methodologies, we anticipate further advancements in our ability to extract, analyse, and interpret material properties, thus contributing to the broader scientific understanding and application of materials science principles.

This contribution was published in the paper "Automatic identification and normalisation of physical quantities from scientific literature" [119].
# Chapter 7

Large scale collection of
experimental data from
scientific literature: SuperCon²
Database

## 7.1 Introduction

This Chapter presents the database we constructed through an automatic process that leverages the work presented in this dissertation. This is the main contribution of our work: SuperCon² Database. We processed 37770 research articles in PDF format belonging to the category *cond-mat.supr-cond* in ArXiv<fnref n="1" />. We obtained 40324 records of superconductor materials with their respective properties and conditions, including Tc, applied pressure measurement methods. We presented a novel data flow that ingests PDF documents (Chatpter 3), combining traditional ML architectures with a novel process that extracts materials and properties (Chapters 4 and 6) and construct a database automatically. This work represents a novel approach to TDM for material informatics. Compared with other methods, our work does not need to be adapted to the data source, whether ArXiv, BioRXiv, ChemRXiv, or any proprietary publisher article.

SuperCon² Database is an efficient, parallel flow of data ingestion and not a replacement for SuperCon. Considering the strict quality requirements that SuperCon

<footnote n="1">¹https://arxiv.org/archive/cond-mat</footnote>
must meet and the fact that automatic processes are imperfect, the data must be validated before being accepted in SuperCon. Compared to the original SuperCon construction, SuperCon<fnref n="2" /> Database was collected in a few days. The updated data model counted new properties: 2052 triplets with applied pressure (*material-Tc-pressure*) and 3602 records with an explicit measurement method (*material-Tc-measurement method*).

In this chapter, we discuss how SuperCon<fnref n="2" /> is built, including the technical information about the data format and the processing.

## 7.2 Database construction

The ingestion process (Figure 7.1) is designed using an Extract-Aggregate approach, implemented through Grobid-superconductors (see chapters 3 and 4).

<figure><img src="image_23.png" /><figcaption>Figure 7.1: Ingestion process. The numbers between parentheses represent the order in which each operation is performed.</figcaption></figure>

Storage is implemented using MongoDB <fnref n="2" />, an open-source document database, and the collections are designed as different stages in the processing:

* the **binary** collection contains the original PDF documents

* the **documents** collection contains the structured document

<footnote n="2">²https://www.mongodb.com</footnote>
* **tabular** collection stores the summarised records, and finally
* **logger** contains detailed information on the status of the process of each document, including possible errors. More details are given in Section 8.3.3.
* **training_data** Used to collect training examples in and discussed in Section 8.2.3.

The "Extraction Task" takes the PDF documents in input and transforms them into a rich representation document that includes text passages (sentences, paragraphs), annotations, and tokens as JSON files, which we will refer to as *structured documents* (Figure 7.2).

Each passage comprises the following attributes: the text, the type (whether a sentence or paragraph), the main section: header, body, and annexe, and the subsections: title, abstract, paragraph, and caption. Furthermore, a passage contains the list of spans representing the extracted entities and a list of layout tokens. The spans are characterised by text, type, attributes, a unique identifier, and other internal information (e.g. from which ML model the entity was extracted). The attributes are stored as a key-value and mainly contain information extracted by the material parser, such as the chemical formula, the structured composition, the material class, etc. The layout tokens contain low-level information coming from the PDF document: font size, font face, superscript, subscript, bold, italic, and coordinates within the PDF document. The coordinates are pairs of "x" and "y" numbers that are used to build annotation "boxes" to encapsulate each token independently (see Figure 8.4 in the following chapter as an example).

The "Aggregation Task" takes the structured document as input and reduces it to a table format where each row (referred to as a *summarised record*, or, *record*) pivots around the relation materials-Tc and attaches additional elements to it. The number of aggregated records can increase when large entities are extracted and may contain condensed information referring to multiple materials. For example, the raw material "Zn and CU doping La Fe B" will be aggregated as two records (doping: Zn, formula: La Fe B) and (doping: Cu, formula: La Fe B).

In the following listing, we illustrate an example of an aggregated record: lines 2-21 contain the record information such as material name, Tc, pressure, etc. Lines 22-34 represent the span list, which is the original annotation information. They are used to link the aggregated records back to the original structure document information. Line 35, 'hash' is a unique signature of the original document using the first 10 characters of the MD5 hash function on the binary content. We use this information to link the original document, the structured document, and the summarised records. 'type', 'timestamp' and 'status' are internal workflow information
<figure><img src="image_24.png" /><figcaption>Figure 7.2: Example of the information from one single entity from a passage extracted in the "Extraction Task". The different structured information is highlighted: links, attributes, PDF entities coordinate (to visualise annotations on the PDF document), and annotations references within the passage (to visualise annotations on text).</figcaption></figure>
discussed in the next chapter. The rest of the lines contain the bibliographic data (39-42).

<table><caption>Listing 7.1: Example of record related to the FeSe material after the aggregation.</caption><tr><td>1</td><td>{</td></tr><tr><td>2</td><td>_id: ObjectId("63dcae91e4d716dd10dd5a7d"),</td></tr><tr><td>3</td><td>"rawMaterial": "FeSe",</td></tr><tr><td>4</td><td>"materialId": "1432944924",</td></tr><tr><td>5</td><td>"name": null,</td></tr><tr><td>6</td><td>"formula": "FeSe",</td></tr><tr><td>7</td><td>"doping": null,</td></tr><tr><td>8</td><td>"shape": null,</td></tr><tr><td>9</td><td>"materialClass": "Chalcogenides, Iron-chalcogenides",</td></tr><tr><td>10</td><td>"fabrication": null,</td></tr><tr><td>11</td><td>"substrate": null,</td></tr><tr><td>12</td><td>"variables": null,</td></tr><tr><td>13</td><td>"criticalTemperature": "13K",</td></tr><tr><td>14</td><td>"criticalTemperatureId": "1905530427",</td></tr><tr><td>15</td><td>"measurementMethod": "",</td></tr><tr><td>16</td><td>"measurementMethodId": "",</td></tr><tr><td>17</td><td>"appliedPressure": null,</td></tr><tr><td>18</td><td>"appliedPressureId": null,</td></tr><tr><td>19</td><td>"section": "body",</td></tr><tr><td>20</td><td>"subsection": "paragraph",</td></tr><tr><td>21</td><td>"sentence": "The 11 type superconductor FeSe exhibits<br/>↔ superconductivity with T c = 13K [2] and its T c reaches 37K<br/>↔ under high pressure (4-6 GPa) [3,4].",</td></tr><tr><td>22</td><td>"spans": [</td></tr><tr><td>23</td><td>{</td></tr><tr><td>24</td><td>"id": "1432944924",</td></tr><tr><td>25</td><td>"text": "FeSe",</td></tr><tr><td>26</td><td>"type": "&lt;material&gt;",</td></tr><tr><td>27</td><td>"linkable": false,</td></tr><tr><td>28</td><td>"offset_start": 27,</td></tr><tr><td>29</td><td>"offset_end": 31,</td></tr><tr><td>30</td><td>"token_start": 0,</td></tr><tr><td>31</td><td>"token_end": 0</td></tr><tr><td>32</td><td>},</td></tr><tr><td>33</td><td>[...]</td></tr><tr><td>34</td><td>],</td></tr><tr><td>35</td><td>"hash": "f70a71214f",</td></tr><tr><td>36</td><td>"type": "automatic",</td></tr></table>
37 "timestamp": ISODate("2022-11-24T09:02:53.256Z"),
38 "status": "new",
39 "title": "Evidence of Inhomogeneous Superconductivity in FeTe1-
    ↪→ xSexby Scotch-Tape Method",
40 "doi": "10.1143/jpsj.81.113707",
41 "authors": "Hiroyuki Okazaki, Tohru Watanabe, Takahide Yamaguchi,
    ↪→ Yasuna Kawasaki, Keita Deguchi, Satoshi Demura, Toshinori
    ↪→ Ozaki, Saleem. J. Denholme, Yoshikazu Mizuguchi, Hiroyuki
    ↪→ Takeya, Yoshihiko Takano",
42 "publisher": "Physical Society of Japan",
43 "journal": "Journal of the Physical Society of Japan",
44 "year": 2012
45 }

## 7.3 Results

Compared to the original SuperCon, SuperCon² was collected in a few days. There is no information about how SuperCon was constructed, only that it started in 1987 [7]. When we built the new process to obtain SuperCon², following domain-expert guidance, we considered two pieces of information: “applied pressure” and “measurement method” that have not been collected systematically in SuperCon. “Applied pressure” is the pressure applied to make the material a superconductor and has gained attention because it can radically change the physical structure of a material. On the other hand, the “measurement method” is how scientists measured the superconducting transition temperature T_c and can be used to semantically recognise multiple T_c obtained from the same material or sample. In particular, identifying calculated properties is fundamental when providing data for predictions. These properties were present only for a minority of records in SuperCon and, we hypothesise, were not considered in the initial design. The fact that applied pressure was recorded in different database fields suggests that such action was more a curator’s decision than a methodological change. The resulting SuperCon² counted 2052 triplets with applied pressure (*material-T_c-pressure*), and 3602 records with an explicit measurement method (*material-T_c-measurement method*). The SuperCon² schema is discussed in 7.3 with examples in Table 7.2.

Our TDM process is limited to text, whereas the manual process focuses on plots and tables. This limitation reduces the density of the collected data, as observed by the amount of information (33,000 records) extracted from only 7227 articles. Such additions should be built as separate projects. However, the automatic pro-
cess cannot be used without manual validation, given the high-quality constraints of SuperCon. This allows combining the TDM process with data curation in a homogeneous flow described in Chapter 8.

<table><caption>Table 7.1: Comparison in volumes from Supercon and SuperCon².</caption><thead><tr><th>Category</th><th>SuperCon</th><th>SuperCon<sup>2</sup></th></tr></thead><tbody><tr><td>Size (records)</td><td>~33000</td><td>40324</td></tr><tr><td>Size (papers)</td><td>7227</td><td>37700</td></tr><tr><td># records with applied pressure</td><td>6</td><td>2052</td></tr><tr><td># records with measurement method</td><td>600</td><td>3602</td></tr><tr><td>Process</td><td>Manual</td><td>Automatic</td></tr><tr><td>Time for process</td><td>N/A</td><td>A few days</td></tr><tr><td>Scope</td><td>Text, plots, tables</td><td>Text</td></tr></tbody></table>

## 7.4 Conclusions

SuperCon² Database represents the result of our effort to build an automatic TDM process to extract experimental data automatically. As a consequence, this is the main contribution of this dissertation. Applying techniques and methodologies described in the previous chapters allowed us to collect a large database in a few days. SuperCon² Database was composed of 40324 records of superconductor materials and properties, including additional properties such as the applied pressure and the T_c measurement method, which were not systematically collected in SuperCon. This database is available in various formats at https://github.com/lfoppiano/supercon.
<table><caption>Table 7.2: Summary and description of the SuperCon² schema. “Internal information” is technical information not accessible to the users.</caption><thead><tr><th>Field name</th><th>Description</th><th>Examples</th></tr></thead><tbody><tr><td colspan="3"><em>Material information</em></td></tr><tr><td>Raw material</td><td>The material or sample as it appears in the text</td><td></td></tr><tr><td>Name</td><td>Canonical name of a material</td><td>PCCO, PCO, Metal diboride, hydrogen, carbon</td></tr><tr><td>Formula</td><td>Material expressed as chemical formula. This includes also formulas with stoichiometric variables</td><td>Pr<sub>1.869</sub>Ce<sub>0.131</sub>CuO<sub>4</sub> - δ, MgB<sub>2</sub>, La<sub>2-x</sub>Sr<sub>x</sub>CuO<sub>4</sub></td></tr><tr><td>Doping</td><td>Doping ratio and doping materials that might be adjoined to the material</td><td>Overdoped, underdoped, optimally doped, bulk, pure, 1% Zn, Zn (from Zn-doped XYZ)</td></tr><tr><td>Shape</td><td>The shape of the material or the sample</td><td>Single crystal, polycrystal, wire, powder, film</td></tr><tr><td>Variables</td><td>Variables that can be substituted in the formula</td><td>x = 0, RE=Ln,St</td></tr><tr><td>Class</td><td>Material classification according to the domain-experts taxonomy</td><td>cuprates, oxides, and alloys</td></tr><tr><td>Fabrication</td><td>All the information that does not belong to any of the previous tags</td><td>Intercalated, synthesized by MBE method, electron-doped, hole-doped</td></tr><tr><td>Substrate</td><td>Substrate material described in the raw material</td><td>PCCO films onto Pr<sub>2</sub>CuO<sub>4</sub>(PCO)/SrTiO<sub>3</sub></td></tr><tr><td colspan="3"><em>Properties</em></td></tr><tr><td>Critical Temperature</td><td>Superconducting critical temperature</td><td></td></tr><tr><td>Applied Pressure</td><td>Pressure applied when measuring the superconducting critical temperature</td><td></td></tr><tr><td>Measurement Method</td><td>Method for measurement of the superconducting critical temperature</td><td>Magnetic susceptibility, specific heat, calculation, prediction, resistivity</td></tr><tr><td colspan="3"><em>Document bibliographic information</em></td></tr><tr><td>Section</td><td>The main body section of the paper</td><td>Header, body, annex</td></tr><tr><td>Subsection</td><td>The secondary segmentation area of the paper</td><td>Paragraph, table caption, figure caption, title, abstract</td></tr><tr><td>Authors, Title, DOI, Publisher, Journal, Year</td><td colspan="2">Bibliographic information of the document</td></tr><tr><td colspan="3"><em>Internal information</em></td></tr><tr><td>Hash, Timestamp</td><td colspan="2">Hash calculated on the binary content of the original PDF document and the timestamp when the document was processed.</td></tr></tbody></table>
# Chapter 8

## Reducing the impact of manual curation

### 8.1 Introduction

The SuperCon database was built manually from 1987 [7] by the National Institute for Materials Science (NIMS) in Japan, and it is considered a reliable source of experimental data on superconductors [9, 135–137]. However, the updates of SuperCon have become increasingly challenging due to the high publication rate. In previous chapters of this dissertation, we have described our approach to creating a reliable automatic process that allows the rapid collection of experimental data from scientific literature (Chapter 3, 4, 6). We created “SuperCon² Database”, a comprehensive database of superconductors containing around 40000 entries within an operational duration of just a few days (Chapter 7). Matching the level of quality seen in SuperCon while simultaneously automating the extraction of organised data can be achieved with a properly designed curation process. We use the term *curation* to describe the overall process of reviewing and validating database records, while *correction* refers to the specific action of altering the values of one or more properties within an individual record. When writing this article, we are unaware of any other curation tool focusing on structured databases of extracted information. Several tools for data annotation, such as Inception [138] and Doccano [139], concentrate on text labelling and classification.

In this contribution, we designed and developed a workflow with a user interface, “SuperCon² Interface”, crafted to produce structured data of superior quality and efficiency to the one obtained by the “traditional” manual approach consisting of
reading documents and noting records, usually on an Excel file.

This contribution comprises three main novelties:

* We developed a workflow and a user interface that allow the curation of a machine-collected database. We demonstrate that using it for data correction resulted in higher quality than the “traditional” (manual) approach.

* We devise an anomaly detection process for incoming data with a lower rejection rate (false positive rate) from domain experts.

* We propose a mechanism that selects training data based on corrected records, and we demonstrate that such selections are rapidly improving the ML models.

## 8.2 Curation workflow

The curation of the SuperCon² Database acts as a workflow where user actions result in database records state transitions (Figure 8.1). Allowed manual actions include a) *mark as valid* (validation) when a record is considered correct or corrected by someone else. When a record is not valid, users can: b) *mark as invalid* when considered “potentially” invalid (or the curator is not confident), c) perform *manual correction* to update it according to the information from the original PDF document, and d) *remove* the record when it was not supposed to be extracted.

In addition to manual operations by users, this workflow also supports automatic actions: “anomaly detection” for pre-screening records (Section 8.2.2) and the “training data collector” for accumulating training data to improve ML models (Section 8.2.3).

Although only the most recent version of a record can be viewed in this system, the correction history is recorded (Section 8.3.3).

### 8.2.1 Workflow control

The “curation status” determines the workflow state (Section 8.2.1), the user action, and the error type (Section 8.2.1).
<figure><img src="image_25.png" /><figcaption>Figure 8.1: Schema of the curation workflow. Each node has type and status properties (Section 8.2.1). Each edge indicates one action. The workflow starts on the left side of the figure. The new records begin with “Automatic, New”. Changes of state are triggered by automatic (Section 8.2.2) or manual operations (update, mark as valid, etc.. Section 8.3.1) and results in changes of the properties in the node. Each combination of property values identifies each state. “(*)” indicates a transition for which the training data are collected (Section 8.2.3)</figcaption></figure>
## Curation status

The curation status (Figure 8.1) is defined by *type* of action, manual or automatic, and *status*, which can assume the following values:

* **new**: default status when creating a new record.

* **curated**: the record has been amended manually.

* **validated**: the record was manually marked as valid.

* **invalid**: the record is wrong or inappropriate for the situation (e.g., $T_m$ or $T_{curie}$ extracted as superconducting critical temperature).

* **obsolete**: the record has been updated, and the updated values are stored in a new record (internal status<fnref n="1" />).

* **removed**: the record has been removed by a curator (internal status).

## Error types

We first introduced the error types in Section 4.3.4. However, this work extended its scope to consider data curation and anomaly detection. The additional error type values are as follows:

* **Composition resolution**: The exact composition cannot be resolved (e.g., the stoichiometric values cannot be resolved).

* **Value resolution**: The extracted formula contains variables that cannot be resolved, even after reading the paper. This includes when data is from tables

* **Anomaly detection**: Anomaly detection has modified the data, facilitating their retrieval from the interface.

* **Curation amends**: The curator updates the data, which does not present issues due to the automatic system.

Users are required to select one *Error Type* at every record update or removal. This information is stored in the “original” record and can differ at every modification.

<footnote n="1">¹“internal status” indicates that their records should be hidden in the interface</footnote>
### 8.2.2 Anomaly detection

Anomaly detection is identifying unusual events or patterns in the data. In our context, this means pinpointing data that is greatly different from the expected values. This post-process was introduced in a limited scope to draw attention to certain cases during the curation.

The anomaly detection uses a rule-based approach and marks any record that matches the following conditions

* the extracted $T_c$ is greater than room temperature (273 K), negative, or contains invalid characters and cannot be parsed (e.g. "41")

* the chemical formula cannot be processed by an ensemble composition parser that combines Pymatgen [27], and text2chem [16]

* the extracted applied pressure cannot be parsed or falls outside the range 0 - 250 GPa.

Records identified as anomalies have *status* "invalid" and *error type* "anomaly detection" for easy identification. Since this process may find false positives, its output requires validation from curators. For example, in specific contexts, $T_c$ values above room temperature or applied pressure up to 500 GPa may be valid in researchers' hypotheses, calculations or simulated predictions.

We ran the anomaly detection on the full SuperCon² Database (40324 records [24]). The anomaly detection identified 1506 records with invalid $T_c$, 5021 records with an incomplete chemical formula, 304 with invalid applied pressure, and 1440 materials linked to multiple $T_c$ values. Further analysis and cross-references with contrasting information may be added in future.

### 8.2.3 Automatic training data collector

The curation process is a valuable endeavour demanding significant knowledge and human effort. To maximise the use of this time for collecting as much information as possible. We integrated an automatic procedure in the curation process that, for every correction, accumulates the related data examples that can be used to improve the underlying ML models.
## Training data collection

In the event of a correction (update, removal) in a database record, this process retrieves the corresponding raw data: the text passage, the recognised entities (spans), and the layout tokens information. This information is sufficient to be exported as training examples, which can be examined and corrected, and feedback to the ML model.

In detail, the process performs the following actions:

* The updated record is prepared and stored.
* The raw data originating the updated record is identified. First, the corresponding structured document is retrieved from the document collection using the document identifier (the hash). Then, the exact text passage in the structured document is located using a unique ID assigned to each material in the database records.
* If the raw data has already been collected, it is skipped. This is when multiple records of the same text passage are corrected.
* Otherwise, the raw information comprising the text string, the spans, and the layout tokens are collected and saved in a separate collection.
* The data collected is then sufficient to generate workable instances in different output formats and the related feature files.

## Training data management

We designed a specific page of the interface (Section 8.3) to manage the collected data (Figure 8.2) in which each row corresponds to a training example composed by the decorated text showing the identified entities, the document identifier, and the status. The users can examine the data, delete it, send it to the annotation tool to be corrected, and then export it. We integrated our interface with Label-studio [3] to correct the collected training examples. Label-studio is an open-source, python-based, modern interface supporting many TDM tasks (NER, topic modelling, image recognition, etc.).
<figure><img src="image_26.png" /><figcaption>Figure 8.2: Screenshot of the training data management page in the SuperCon² interface. Each row contains one potential training data example. Each example comprises a sentence and its extracted entities (highlighted in colour) with potential annotation mistakes that need to be corrected using an external tool: we used Label-Studio [3]. The column “Status” indicates whether the example has been sent to the external tool.</figcaption></figure>

## 8.3 Curation interface

The workflow is operated through the user interface, which offers several key features to facilitate the data curation process (Figure 8.1). It provides a comprehensive view of materials and their properties as a table that includes search, filter, and sorting functionality (Figure 8.3). The detailed schema, including examples, is reported in our previous work [24].

During the curation process, it is often necessary to switch back and forth between the database record and the related context in the paper (the related paragraph or sentence). Our interface provides a viewer for individual documents, which visualises a table with the extracted records in the same window and the original PDF document decorated with annotations that identify the extracted materials and properties (Figure 8.4).

### 8.3.1 Manual curation approach

In this section, we discuss our strategy regarding manual curation, which is still indispensable for developing high-quality structures.
<figure><img src="image_27.png" /><figcaption>Figure 8.3: Screenshot of SuperCon² interface showing the database. Each row corresponds to one material-Tc pair. On the top, there are searches by attribute, sorting and other filtering operations. Curation controls (mark as valid, update, etc.) are on the right (last column). Records are grouped by document with alternating light yellow and white.</figcaption></figure>

<figure><img src="image_28.png" /><figcaption>Figure 8.4: PDF document viewer showing an annotated document. The table on top is linked through the annotated entities. The user can navigate from the record to the exact point in the PDF, with a pointer (the red bulb light) identifying the context of the examined entities.</figcaption></figure>
We selected curators from superconductor domain experts to certify sufficient data quality. Nevertheless, as confirmed from our experiment in Section 8.4.3, each individual's experience may impact the final result. We followed two principles to guarantee robustness in the curation process. First, we built solid curation documentation as a form of example-driven guidelines with an iterative approach we first introduced in [4]. Then, we used a double-round validation approach, in which the data was initially corrected by one person and validated in a second round by a different individual.

### 8.3.2 Curation guidelines

The guidelines consist mainly of the general principles and the correction rules with examples of solutions. The guidelines are designed to provide general information applied to corrections and essential explanations containing illustrations for a faster understanding (e.g. the meaning of the colours of the annotations). Differently from our previous work [4], these guidelines are divided into examples for different scenarios based on the error types mentioned in Section 8.2.1. Each example described the initial record, its context, the expected corrected record and a brief explanation, as illustrated in Figure 8.5.

### 8.3.3 Curation and processing logs

The Supercon² interface gives access to information regarding the ingestion (processing log) and the curation process (curation log). The processing log is filled up when the new data is ingested, and it allows us to explain why certain documents haven't been processed (Figure 8.6 top). For example, sometimes documents fail because they don't contain any text (image PDF documents) or are too big (more than 100 pages).

The curation log provides a view of what, when and how a record has been corrected (Figure 8.6 bottom).

## 8.4 Results and evaluation

This section illustrates the experiments we have run to evaluate our work. The evaluation is composed of three sets of results. The anomaly detection rejection rate (Section 8.4.1) indicates how many anomalies were rejected by curators after
<figure><img src="image_29.png" /><figcaption>Figure 8.5: Sample curation sheet from the curation guidelines. The sheet is composed of the following information: a) Sample input data: a screenshot of the record from the “SuperCon² Interface”, b) *Context* represented by the related part of the annotated document referring to the record in exams. c) The *Motivation*, describing the issue, d) the *Action* to be taken, and the *Expected output*.</figcaption></figure>

<figure><img src="image_30.png" /><figcaption>Figure 8.6: Top: *Processing log*, showing the output of each ingestion operation and the outcome with the detailed error that may have occurred. Bottom: *Correction log*, indicating each record, the number of updates, and the date/time of the last updates. Clicking on the “Record id” shows the latest record values.</figcaption></figure>
validation. Then, we demonstrate that the training data automatically selected contributed to improving the ML model with a small set of examples (Section 8.4.2). Finally, we evaluated the quality of the data extraction using the interface (and the semi-automatic TDM process) against the classical method of reading the PDF articles and noting the experimental information in an Excel file. In Section 8.4.3, we find that using the interface improves the quality of the curated data by reducing missing experimental data.

## 8.4.1 Anomaly detection rejection rate

We evaluated the anomaly detection by observing the “rejection rate”, which consists of the number of detected anomalies rejected by human validation. Running the anomaly detection on a database subset with 667 records found 17 anomalies in $T_c$, one anomaly in applied pressure, and 16 in the chemical formulas. Curators examined each reported record and rejected 4 (23%) anomalies in $T_c$, six anomalies (37%) in chemical formulas and 0 anomalies in applied pressure. This indicates an appropriately low rate of false positives, although a study with a larger dataset might be necessary.

## 8.4.2 Training data generation

We selected around 400 records in the Supercon² Database that were marked as invalid by the anomaly detection process, and we corrected them following the curation guidelines (Section 8.3.2). Then, we examined the corresponding training data corrected by the interface (Section 8.2.3) and obtained a set of 352 training data examples for our ML models. We call the obtained dataset *curation* to be distinguished from the original SuperMat dataset, referred to as *base*.

We prepared our experiment using SciBERT [67] that we fine-tuned for our downstream task as in [24]. We trained five models we evaluated using a fixed holdout dataset from SuperMat, averaging the results to smooth out the fluctuations. We use the DeLFT (Deep Learning For Text) [110] library for training, evaluating, and managing the prediction models. A model can be trained with two different strategies:

1. *“from scratch”*: when the model is initialised randomly. We denote this strategy with an (*s*).

2. *“incremental”*: when the initial model weights are taken from an existing model. We denote this strategy with an (*i*).
The latter can be seen as a way to “continue” the training from a specific checkpoint. We thus define three different training protocols:

1. **base(s)**: using the *base* dataset and training from scratch (s).

2. **(base+curation)(s)**: using both the *base* and *curation* datasets and training from scratch (s).

3. **base(s)+(base+curation)(i)**: Using the *base* dataset to train from scratch (s), and then continuing the training with the *curation* dataset (i).

We merge “curation” with the base dataset because the curation dataset is tiny compared to “base”, and we want to avoid catastrophic forgetting [140] or overfitting.

Table 8.1: F1-score from the evaluation of the fine-tuned SciBERT models. The training is performed with three different approaches. The *base* dataset is the original dataset described in [4], and the *curation* dataset is automatically collected based on the database corrections by the interface and manually corrected. *s* indicate “training from scratch”, while *i* indicate “incremental training”. The evaluation uses the same holdout dataset from SuperMat [4]. The results are averaged over 5 runs of train and evaluation.

<table><thead><tr><th></th><th>base(s)</th><th>(base+curation)(s)</th><th>base(s)+(base+curation)(i)</th></tr></thead><tbody><tr><td>Nb total examples</td><td>16902</td><td>17254</td><td>16902(s), 17254 (i)</td></tr><tr><td>&lt;class&gt;</td><td>70.41</td><td><strong>73.02</strong></td><td>71.86</td></tr><tr><td>&lt;material&gt;</td><td>79.37</td><td>80.09</td><td><strong>80.37</strong></td></tr><tr><td>&lt;me_method&gt;</td><td>66.72</td><td>66.57</td><td><strong>66.95</strong></td></tr><tr><td>&lt;pressure&gt;</td><td>46.43</td><td><strong>48.42</strong></td><td>47.23</td></tr><tr><td>&lt;tc&gt;</td><td>80.13</td><td><strong>80.92</strong></td><td>80.34</td></tr><tr><td>&lt;tcValue&gt;</td><td>78.29</td><td>78.41</td><td><strong>79.73</strong></td></tr><tr><td>All (micro avg.)</td><td>76.67</td><td>77.44</td><td><strong>77.48</strong></td></tr><tr><td>Δ avg. w/ baseline</td><td>-</td><td>+0.77</td><td>+0.81</td></tr></tbody></table>

The trained models are then tested using a fixed holdout dataset that we designed in our previous work [24], and the evaluation scores are shown in Table 8.1.

This experiment demonstrates that with only 352 examples (2% of the Super-Mat dataset) comprising 1846 additional entities (11% of the entities from the SuperMat dataset) (Table 8.2), we obtain an improvement of F1-score from 76.67%<fnref n="2" />

<footnote n="2">²In our previous work [24] we reported 77.03% F1-score. There is a slight decrease in absolute scores between DeLFT 0.2.8 and DeLFT 0.3.0. One cause may be using different hyperparameters</footnote>
to values between 77.44% (+0.77) and 77.48% (+0.81) for (base+curation)(s) and base(s)+(base+curation)(i), respectively.

<table><caption>Table 8.2: Data support, the number of entities for each label in each dataset used for evaluating the ML models. The *base* dataset is the original dataset described in [4], and the *curation* dataset is automatically collected based on the database corrections by the interface and manually corrected.</caption><thead><tr><th></th><th>base</th><th>base+curation</th><th>Δ</th></tr></thead><tbody><tr><td>&lt;class&gt;</td><td>1646</td><td>1732</td><td>86</td></tr><tr><td>&lt;material&gt;</td><td>6943</td><td>7580</td><td>637</td></tr><tr><td>&lt;me_method&gt;</td><td>1883</td><td>1934</td><td>51</td></tr><tr><td>&lt;pressure&gt;</td><td>274</td><td>361</td><td>87</td></tr><tr><td>&lt;tc&gt;</td><td>3741</td><td>4269</td><td>528</td></tr><tr><td>&lt;tcValue&gt;</td><td>1099</td><td>1556</td><td>457</td></tr><tr><td>Total</td><td>15586</td><td>17432</td><td>1846</td></tr></tbody></table>

This experiment gives interesting insight relative to the positive impact of selecting the training data. However, there are some limitations: the *curation* dataset is small compared to the *base* dataset. This issue could be verified by correcting all the available training data, repeating this experiment, and studying the interpolation between the size of the two datasets and the obtained evaluation scores. A second limitation is that the hyperparameters we chose for our model, particularly the learning rate and batch size, could still be better tuned to obtain better results with the second and third training protocols.

### 8.4.3 Data quality

We experimented with evaluating the effectiveness and accuracy of data curation using two methods: a) the user interface (*interface*), and b) the “traditional” manual approach consisting of reading PDF documents and populating an Excel file (*PDF documents*).

We selected a dataset of 15 papers, which we assigned to three curators — a senior researcher (SD), a PhD student (PS), and a master's student (MS). Each curator received ten papers: half to be corrected with the *interface* and half with the *PDF Document* method. Overall, each pair of curators had five papers in common, which they had to process using opposite methods. For instance, if curator A receives

<footnote>in version 0.3.0, such as batch size and learning rate. However, the most probable cause could be the impact of using the Huggingface tokenizers library which is suffering from quality issues https://github.com/kermitt2/delft/issues/150.</footnote>
paper 1 to be corrected with the *interface*, curator B, who gets the same paper 1,
will correct it with the *PDF document* method. After curation, a fourth individual
manually reviewed the curated content. The raw data is available in Tables 8.3
and 8.4.

We evaluated the curation considering a double perspective: time and correct-
ness. Time was calculated as the accumulated minutes required using each method.
Correctness was assessed using standard measures such as precision, recall, and the
F1-score. Precision measures the accuracy of the extracted information, while recall
assesses the ability to capture all expected information. F1-Score is a harmonic
means of precision and recall.

Discussion

Overall, both methods required the same accumulated time: 185 minutes using the *interface* and 184 minutes using the *PDF Document* method. Not all the curators were familiar with the *interface* method when the experiment was conducted. Although they had access to the user documentation, they had to get acquainted with the user interface, thus the accumulated 185 minutes included such activities.

We examined the quality of the extracted data, and we observed an improvement
of +5.55% in precision and a substantial +46.69% in recall when using the *interface*
as compared with the *PDF Document* method (Table 8.5). The F1-score improved
by 39.35%.

The disparity in experience significantly influenced curation accuracy, particu-
larly regarding high-level skills. Senior researchers consistently achieved an average
F1-Score approximately 13% higher than other curators (see Table 8.6). Further-
more, we observed a modest improvement between master’s and PhD students.
These findings may suggest that for large-scale projects, employing master students
instead of PhD students may be a more cost-effective choice. Thus, using only a
few senior researchers for the second round of validation (Section 8.3.1). However, a
new experiment with more people should be conducted to confirm this hypothesis.

Finally, the collected data suggest that all three curators had overall more cor-
rected results using the interface as illustrated in Table 8.7.

The results of this experiment confirmed that our curation interface and work-
flow significantly improved the quality of the extracted data, with an astonishing
improvement in recall, thus preventing curators from overlooking important infor-
mation.
<table><caption>Table 8.3: Timetable recording the time spent for each of the 15 articles. Each row indicates the time and the event (Start, Finish) from each of the curators: Master Student (MD), PhD Student (PD), and Senior Researcher (SR). Duration is expressed in minutes.</caption><thead><tr><th>Time</th><th>Event</th><th>Document ID</th><th>Curator</th><th>Duration (mins)</th></tr></thead><tbody><tr><td>14:40</td><td>Start</td><td>02bf1b3db9</td><td>PS</td><td>0</td></tr><tr><td>14:49</td><td>Finish</td><td>02bf1b3db9</td><td>PS</td><td>9</td></tr><tr><td>14:53</td><td>Start</td><td>00b50fc0a8</td><td>PS</td><td>0</td></tr><tr><td>14:58</td><td>Finish</td><td>00b50fc0a8</td><td>PS</td><td>5</td></tr><tr><td>14:37</td><td>Start</td><td>0aa1b3161f</td><td>MS</td><td>0</td></tr><tr><td>14:50</td><td>Start</td><td>0454e07f64</td><td>SR</td><td>0</td></tr><tr><td>14:58</td><td>Finish</td><td>0454e07f64</td><td>SR</td><td>8</td></tr><tr><td>15:01</td><td>Start</td><td>02cbc58819</td><td>PS</td><td>0</td></tr><tr><td>15:06</td><td>Start</td><td>00c32076f4</td><td>SR</td><td>0</td></tr><tr><td>15:07</td><td>Finish</td><td>0aa1b3161f</td><td>MS</td><td>30</td></tr><tr><td>15:08</td><td>Finish</td><td>02cbc58819</td><td>PS</td><td>7</td></tr><tr><td>15:08</td><td>Start</td><td>044939701d</td><td>PS</td><td>0</td></tr><tr><td>15:12</td><td>Start</td><td>0021fd339f</td><td>MS</td><td>0</td></tr><tr><td>15:15</td><td>Finish</td><td>00c32076f4</td><td>SR</td><td>9</td></tr><tr><td>15:17</td><td>Finish</td><td>044939701d</td><td>PS</td><td>9</td></tr><tr><td>15:17</td><td>Start</td><td>08e1cb8f4f</td><td>PS</td><td>0</td></tr><tr><td>15:20</td><td>Start</td><td>0c7d3163ea</td><td>SR</td><td>0</td></tr><tr><td>15:31</td><td>Finish</td><td>08e1cb8f4f</td><td>PS</td><td>14</td></tr><tr><td>15:32</td><td>Finish</td><td>0021fd339f</td><td>MS</td><td>20</td></tr><tr><td>15:32</td><td>Start</td><td>039105663f</td><td>MS</td><td>0</td></tr><tr><td>15:37</td><td>Finish</td><td>0c7d3163ea</td><td>SR</td><td>17</td></tr><tr><td>15:53</td><td>Finish</td><td>039105663f</td><td>MS</td><td>21</td></tr><tr><td>15:55</td><td>Start</td><td>02c4f00127</td><td>MS</td><td>0</td></tr><tr><td>15:58</td><td>Start</td><td>0454e07f64</td><td>PS</td><td>0</td></tr><tr><td>16:02</td><td>Start</td><td>0da5febabf</td><td>SR</td><td>0</td></tr><tr><td>16:08</td><td>Finish</td><td>0454e07f64</td><td>PS</td><td>10</td></tr><tr><td>16:09</td><td>Finish</td><td>02c4f00127</td><td>MS</td><td>14</td></tr><tr><td>16:11</td><td>Finish</td><td>0da5febabf</td><td>SR</td><td>9</td></tr><tr><td>16:11</td><td>Start</td><td>0012333581</td><td>SR</td><td>0</td></tr><tr><td>16:12</td><td>Start</td><td>00c32076f4</td><td>PS</td><td>0</td></tr><tr><td>16:18</td><td>Start</td><td>021c413172</td><td>MS</td><td>0</td></tr><tr><td>16:22</td><td>Finish</td><td>00c32076f4</td><td>PS</td><td>10</td></tr><tr><td>16:23</td><td>Start</td><td>0c7d3163ea</td><td>PS</td><td>0</td></tr><tr><td>16:30</td><td>Finish</td><td>0012333581</td><td>SR</td><td>19</td></tr><tr><td>16:32</td><td>Finish</td><td>021c413172</td><td>MS</td><td>14</td></tr><tr><td>16:37</td><td>Start</td><td>02bf1b3db9</td><td>MS</td><td>0</td></tr><tr><td>16:38</td><td>Finish</td><td>0c7d3163ea</td><td>PS</td><td>15</td></tr><tr><td>17:32</td><td>Finish</td><td>0021fd339f</td><td>SR</td><td>12</td></tr><tr><td>17:34</td><td>Start</td><td>039105663f</td><td>SR</td><td>0</td></tr><tr><td>17:55</td><td>Finish</td><td>039105663f</td><td>SR</td><td>21</td></tr><tr><td>17:56</td><td>Start</td><td>02c4f00127</td><td>SR</td><td>0</td></tr><tr><td>18:00</td><td>Finish</td><td>02c4f00127</td><td>SR</td><td>4</td></tr><tr><td>18:00</td><td>Start</td><td>021c413172</td><td>SR</td><td>0</td></tr><tr><td>18:09</td><td>Finish</td><td>021c413172</td><td>SR</td><td>9</td></tr></tbody></table>
<table><caption>Table 8.4: Evaluation scores obtained for each document and method (I: Interface, P: PDF) combination. TP: True positive, FP: False positive, FN: False negative. P: Precision, R: Recall, F1: F1-score</caption><thead><tr><th>Document ID</th><th># pages</th><th>Method</th><th># TP</th><th># FP</th><th># FN</th><th>P</th><th>R</th><th>F1</th></tr></thead><tbody><tr><td colspan="9">Senior Researcher (SR)</td></tr><tr><td>0454e07f64</td><td>4</td><td>I</td><td>6</td><td>0</td><td>0</td><td>100.00</td><td>100.00</td><td>100.00</td></tr><tr><td>00c32076f4</td><td>13</td><td>P</td><td>8</td><td>0</td><td>0</td><td>100.00</td><td>100.00</td><td>100.00</td></tr><tr><td>0c7d3163ea</td><td>9</td><td>I</td><td>13</td><td>1</td><td>0</td><td>92.86</td><td>100.00</td><td>96.30</td></tr><tr><td>0da5febabf</td><td>11</td><td>P</td><td>8</td><td>0</td><td>1</td><td>100.00</td><td>88.89</td><td>94.12</td></tr><tr><td>0012333581</td><td>13</td><td>I</td><td>11</td><td>0</td><td>0</td><td>100.00</td><td>100.00</td><td>100.00</td></tr><tr><td>0aa1b3161f</td><td>5</td><td>I</td><td>9</td><td>0</td><td>1</td><td>100.00</td><td>90.00</td><td>94.74</td></tr><tr><td>0021fd339f</td><td>14</td><td>P</td><td>4</td><td>0</td><td>8</td><td>100.00</td><td>33.33</td><td>50.00</td></tr><tr><td>039105663f</td><td>9</td><td>I</td><td>11</td><td>1</td><td>0</td><td>91.67</td><td>100.00</td><td>95.65</td></tr><tr><td>02c4f00127</td><td>13</td><td>P</td><td>0</td><td>0</td><td>3</td><td>100.00</td><td>0.00</td><td>0.00</td></tr><tr><td>021c413172</td><td>5</td><td>I</td><td>15</td><td>0</td><td>0</td><td>100.00</td><td>100.00</td><td>100.00</td></tr><tr><td colspan="9">PhD Student (PS)</td></tr><tr><td>02bf1b3db9</td><td>7</td><td>I</td><td>5</td><td>0</td><td>2</td><td>100.00</td><td>71.43</td><td>83.33</td></tr><tr><td>00b50fc0a8</td><td>11</td><td>P</td><td>2</td><td>0</td><td>7</td><td>100.00</td><td>22.22</td><td>36.36</td></tr><tr><td>02cbc58819</td><td>4</td><td>I</td><td>4</td><td>0</td><td>3</td><td>100.00</td><td>57.14</td><td>72.73</td></tr><tr><td>044939701d</td><td>12</td><td>P</td><td>4</td><td>0</td><td>2</td><td>100.00</td><td>66.67</td><td>80.00</td></tr><tr><td>08e1cb8f4f</td><td>16</td><td>I</td><td>5</td><td>1</td><td>1</td><td>83.33</td><td>85.71</td><td>84.51</td></tr><tr><td>0454e07f64</td><td>4</td><td>P</td><td>0</td><td>1</td><td>5</td><td>0.00</td><td>16.67</td><td>0.00</td></tr><tr><td>00c32076f4</td><td>13</td><td>I</td><td>8</td><td>0</td><td>0</td><td>100.00</td><td>100.00</td><td>100.00</td></tr><tr><td>0c7d3163ea</td><td>9</td><td>P</td><td>9</td><td>0</td><td>5</td><td>100.00</td><td>64.29</td><td>78.26</td></tr><tr><td>0da5febabf</td><td>11</td><td>I</td><td>9</td><td>0</td><td>0</td><td>100.00</td><td>100.00</td><td>100.00</td></tr><tr><td>0012333581</td><td>13</td><td>P</td><td>4</td><td>4</td><td>3</td><td>50.00</td><td>72.73</td><td>59.26</td></tr><tr><td colspan="9">Master Student (MS)</td></tr><tr><td>0aa1b3161f</td><td>5</td><td>P</td><td>1</td><td>0</td><td>9</td><td>100.00</td><td>10.00</td><td>18.18</td></tr><tr><td>0021fd339f</td><td>14</td><td>I</td><td>12</td><td>3</td><td>3</td><td>80.00</td><td>100.00</td><td>88.89</td></tr><tr><td>039105663f</td><td>9</td><td>P</td><td>4</td><td>1</td><td>7</td><td>80.00</td><td>41.67</td><td>54.79</td></tr><tr><td>02c4f00127</td><td>13</td><td>I</td><td>3</td><td>1</td><td>1</td><td>75.00</td><td>100.00</td><td>85.71</td></tr><tr><td>021c413172</td><td>5</td><td>P</td><td>7</td><td>1</td><td>7</td><td>87.50</td><td>53.33</td><td>66.27</td></tr><tr><td>02bf1b3db9</td><td>7</td><td>P</td><td>2</td><td>0</td><td>5</td><td>100.00</td><td>28.57</td><td>44.44</td></tr><tr><td>00b50fc0a8</td><td>11</td><td>I</td><td>7</td><td>2</td><td>0</td><td>77.78</td><td>100.00</td><td>87.50</td></tr><tr><td>02cbc58819</td><td>4</td><td>P</td><td>5</td><td>0</td><td>2</td><td>100.00</td><td>71.43</td><td>83.33</td></tr><tr><td>044939701d</td><td>12</td><td>I</td><td>5</td><td>0</td><td>1</td><td>100.00</td><td>83.33</td><td>90.91</td></tr><tr><td>08e1cb8f4f</td><td>16</td><td>P</td><td>1</td><td>0</td><td>6</td><td>100.00</td><td>14.29</td><td>25.00</td></tr></tbody></table>
<table><caption>Table 8.5: Evaluation scores (P: precision, R: recall, F1: F1-score) between the curation using the SuperCon² interface (Interface) and the traditional method of reading the PDF document (PDF document).</caption><thead><tr><th>Method</th><th>P (%)</th><th>R (%)</th><th>F1 (%)</th><th># docs</th></tr></thead><tbody><tr><td>PDF document</td><td>87.83</td><td>45.61</td><td>52.67</td><td>15</td></tr><tr><td>Interface</td><td><strong>93.38</strong></td><td><strong>92.51</strong></td><td><strong>92.02</strong></td><td>15</td></tr></tbody></table>

<table><caption>Table 8.6: Evaluation scores (P: precision, R: recall, F1: F1-score) aggregated by experience (MS: master student, PD: PhD student, SR: senior researcher). Each person corrected ten documents.</caption><thead><tr><th>Experience</th><th>P (%)</th><th>R (%)</th><th>F1 (%)</th><th># docs</th><th># pages</th></tr></thead><tbody><tr><td>MS</td><td>90.03</td><td>60.26</td><td>64.50</td><td>10</td><td>96</td></tr><tr><td>PD</td><td>83.33</td><td>65.69</td><td>69.45</td><td>10</td><td>100</td></tr><tr><td>SR</td><td><strong>98.45</strong></td><td><strong>81.22</strong></td><td><strong>83.08</strong></td><td>10</td><td>96</td></tr></tbody></table>

<table><caption>Table 8.7: Evaluation scores (P: precision, R: recall, F1: F1-score) listed by experience (MS: master student, PD: PhD student, SR: senior researcher), and method (PDF document, Interface).</caption><thead><tr><th>Experience</th><th>Method</th><th>P (%)</th><th>R (%)</th><th>F1 (%)</th><th># docs</th><th># pages</th></tr></thead><tbody><tr><td rowspan="2">MS</td><td>PDF Document</td><td>94.58</td><td>36.55</td><td>48.67</td><td>6</td><td>46</td></tr><tr><td>Interface</td><td>83.19</td><td>95.83</td><td>88.25</td><td>4</td><td>50</td></tr><tr><td rowspan="2">PD</td><td>PDF Document</td><td>70.00</td><td>48.51</td><td>50.78</td><td>5</td><td>49</td></tr><tr><td>Interface</td><td>96.67</td><td>82.86</td><td>88.11</td><td>5</td><td>51</td></tr><tr><td rowspan="2">SR</td><td>PDF Document</td><td><strong>100.00</strong></td><td>55.56</td><td>61.03</td><td>4</td><td>51</td></tr><tr><td>Interface</td><td>97.42</td><td><strong>98.33</strong></td><td><strong>97.78</strong></td><td>6</td><td>45</td></tr></tbody></table>
## 8.5 Code availability

This work is available at https://github.com/lfoppiano/supercon2. The repository contains the code of the SuperCon² interface, the curation workflow, and the ingestion processes for harvesting the SuperCon² Database of materials and properties. The guidelines are accessible at https://supercon2.readthedocs.io.

## 8.6 Conclusions

In this contribution, we presented a semi-automatic "staging area" to validate efficiently new experimental records automatically collected from superconductor research articles (SuperCon² Database [24]) before they are ingested into the existing, manually-build database of superconductors, SuperCon [7]. The system provides a curation workflow and a user interface (SuperCon² Interface) tailored to efficiently support domain experts in data correction and validation with fast context switching and an enhanced PDF viewer. Under the hood, the workflow ran "anomaly detection" to automatically identify outliers and a "training data collector" based on human corrections to efficiently accumulate training data to be feedback to the ML model. Compared with the traditional manual approach of reading PDF documents and extracting information from an Excel file, SuperCon² significantly improves the quality of curation by approximately 6% and +47% for precision and recall, respectively.
# Chapter 9

## Conclusion

We propose an end-to-end pipeline for extracting material information from the scientific literature to improve the efficiency and quality of materials databases. This work describes the end-to-end construction of a comprehensive TDM process to extract databases of experimental data automatically. This work represents a novel application of information extraction in materials science. Our effort resulted in the creation of SuperCon² Database (Chapter 7), consisting of 40324 materials and properties records collected from 37000 articles.

In Chapter 3 we describe the automatic system that reads PDF documents, extracts relevant information (Chapter 4) and stores them in a tabular format where each entry represents a material and its related properties ($T_c$, applied pressure, measurement methods, etc.) using a combination with Grobid-quantities: a general system for identification and standardisation of physical quantities and measurements, described in Chapter 6. ML models have been trained and evaluated using SuperMat (Chapter 5), a dataset we developed with domain experts that provides annotations and relations between entities in 164 scientific documents from superconductor research. Material expressions are carefully managed with a specific material parser that combines different methods to decompose mixed information (doping, shape, formula, name, etc.). In Chapter 8, we describe a staging area using the obtained "SuperCon² Database" with machine-collected entities. Using a user interface, "SuperCon² Interface", we allow domain experts to examine and correct the data by accessing the exact location in the original PDF document decorated with the extracted information. Our interface significantly improves the curation quality by increasing precision and recall by approximately 6% and 47%, respectively.

This work can be expanded to permanent magnetic materials, spintronics, and
thermoelectric research domains. The SuperMat dataset can be expanded to support more properties, such as current, or by adding documents from related domains, for example, magneto-caloric materials research. Finally, with the advent of Large Language Models (LLM), we can replace the relation extraction rule-based algorithm with integration with fine-tuned LLM models, which we have demonstrated promising capabilities in material-properties interaction [25].
Bibliography

[1] G Gajda, A Morawski, K Rogacki, T Cetner, A J Zaleski, K Buchkov, E Nazarova, N Balchev, M S A Hossain, R Diduszko, K Gruszka, P Przyshupski, Ł Fajfrowski, and D Gajda. Ag-doped FeSe$_{0.94}$ polycrystalline samples obtained through hot isostatic pressing with improved grain connectivity. *Superconductor Science and Technology*, 29(9):095002, jul 2016.

[2] Yusuke Shibata, Shintaro Nomura, Ryosuke Ishiguro, Hiromi Kashiwaya, Satoshi Kashiwaya, Yusuke Nago, and Hideaki Takayanagi. Magnetic field imaging of a tungsten carbide film by scanning nano-SQUID microscope. *Superconductor Science and Technology*, 29(10):104004, aug 2016.

[3] Maxim Tkachenko, Mikhail Malyuk, Andrey Holmanyuk, and Nikolai Liubimov. Label Studio: Data labeling software, 2020-2022. Open source software available from https://github.com/heartexlabs/label-studio.

[4] Luca Foppiano, Sae Dieb, Akira Suzuki, Pedro Baptista de Castro, Suguru Iwasaki, Azusa Uzuki, Miren Garbine Esparza Echevarria, Yan Meng, Kensei Terashima, Laurent Romary, Yoshihiko Takano, and Masashi Ishii. Supermat: construction of a linked annotated dataset from superconductors-related publications. *Science and Technology of Advanced Materials: Methods*, 1(1):34–44, 2021.

[5] Evgeny Blokhin and Pierre Villars. The pauling file project and materials platform for data science: From big data toward materials genome. 2018.

[6] Yukari Katsura, Masaya Kumagai, Takushi Kodani, Mitsunori Kaneshige, Yuki Ando, Sakiko Gunji, Yoji Imai, Hideyasu Ouchi, Kazuki Tobita, Kaoru Kimura, and Koji Tsuda. Data-driven analysis of electron relaxation times in pbte-type thermoelectric materials. *Science and Technology of Advanced Materials*, 20(1):511–520, 2019.

[7] Masashi Ishii and Koichi Sakamoto. Structuring superconductor data with ontology: reproducing historical datasets as knowledge bases. *Science and Technology of Advanced Materials: Methods*, 3(1):2223051, 2023.
[8] Timo Sommer, Roland Willa, Jörg Schmalian, and Pascal Friederich. 3dsc-a new dataset of superconductors including crystal structures. *arXiv preprint arXiv:2212.06071*, 2022.

[9] Valentin Stanev, Corey Oses, A. Kusne, Efrain Rodriguez, Johnpierre Paglione, Stefano Curtarolo, and I. Takeuchi. Machine learning modeling of superconducting critical temperature. *npj Computational Materials*, 4, 092017.

[10] Thanh Dung Le, Rita Noumeir, Huu Luong Quach, Ji Hyung Kim, Jung Ho Kim, and Ho Min Kim. Critical temperature prediction for a superconductor: A variational bayesian neural network approach. *IEEE Transactions on Applied Superconductivity*, 30(4):1-5, 2020.

[11] James J. Hamlin. Superconductivity near room temperature. *Nature*, 569:491-492, 2019.

[12] Luca Foppiano, M. Dieb Thaer, Akira Suzuki, and Masashi Ishii. Proposal for Automatic Extraction Framework of Superconductors Related Information from Scientific Literature. In *Letters and Technology News, vol. 119, no. 66, SC2019-1 (no.66)*, volume 119, pages 1-5, Tsukuba, May 2019. ISSN: 2432-6380.

[13] Duff Johnson. Pdf statistics – the universe of electronic documents, 2018-05-14.

[14] Callum J Court and Jacqueline M Cole. Auto-generated materials database of curie and n el temperatures via semi-supervised relationship extraction. *Scientific Data*, 5:180111, 12 2018.

[15] Callum J. Court and Jacqueline M. Cole. Magnetic and superconducting phase diagrams and transition temperatures predicted using text mining and machine learning. *npj Computational Materials*, 6(1):18, March 2020.

[16] Olga Kononova, Haoyan Huo, Tanjin He, Ziqin Rong, Tiago Botari, Wenhao Sun, Vahe Tshitoyan, and Gerbrand Ceder. Text-mined dataset of inorganic materials synthesis recipes. *Scientific Data*, 6(1):203, October 2019.

[17] GROBID contributors. Grobid (generation of bibliographic data). https://github.com/kermitt2/grobid, 2008 — 2019. swh:1:dir:6a298c1b2008913d62e01e5bc967510500f80710.

[18] Mikael Laakso, Patrik Welling, Helena Bukvova, Linus Nyman, Bo-Christer Björk, and Turid Hedlund. The development of open access journal publishing from 1993 to 2009. *PLOS ONE*, 6(6):1-10, 06 2011.
[19] Mikiko Tanifuji, Asahiko Matsuda, and Hideki Yoshikawa. Materials data platform - a fair system for data-driven materials science. In *2019 8th International Congress on Advanced Applied Informatics (IIAI-AAI)*, pages 1021-1022, 2019.

[20] Kyosuke Yamaguchi, Ryoji Asahi, and Yutaka Sasaki. SC-CoMIcs: A superconductivity corpus for materials informatics. In *Proceedings of the 12th Language Resources and Evaluation Conference*, pages 6753-6760, Marseille, France, May 2020. European Language Resources Association.

[21] John Lafferty, Andrew McCallum, and Fernando CN Pereira. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. 2001.

[22] Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, K. Kawakami, and Chris Dyer. Neural architectures for named entity recognition. *ArXiv*, abs/1603.01360:260-270, June 2016.

[23] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. *arXiv preprint arXiv:1810.04805*, pages 4171-4186, June 2018.

[24] Luca Foppiano, Pedro Baptista Castro, Pedro Ortiz Suarez, Yoshihiko Takano, Kensei Terashima, and Masashi Ishii. Automatic extraction of materials and properties from superconductors scientific literature. *Science and Technology of Advanced Materials: Methods*, 3(1):2153633, 2023.

[25] Luca Foppiano, Guillaume Lambard, Toshiyuki Amagasa, and Masashi Ishii. Mining experimental data from materials science literature with large language models. *arXiv preprint arXiv:2401.11052*, 2024.

[26] Sae Dieb, Luca Foppiano, Kensei Terashima, Pedro Baptista de CASTRO, Yoshihiko Takano, and Masashi Ishii. Superconductor research papers clustering using weighted annotated information. In *Proceedings of the National Conference of the Japanese Society for Artificial Intelligence, 36th (2022)*, pages 1S5IS2a03-1S5IS2a03. Japan Society for Artificial Intelligence, 2022.

[27] Shyue Ping Ong, William Davidson Richards, Anubhav Jain, Geoffroy Hautier, Michael Kocher, Shreyas Cholia, Dan Gunter, Vincent L. Chevrier, Kristin A. Persson, and Gerbrand Ceder. Python materials genomics pymatgen: A robust open-source python library for materials analysis. *Computational Materials Science*, 68:314-319, 2 2013.

[28] Kento Mitsui, Yutaka Sasaki, and Ryoji Asahi. Automatic knowledge acquisition from superconductivity information in literature. *Science and Technology of Advanced Materials: Methods*, 3(1):2206532, 2023.
[29] Sang-Hoon Park, Baekjun Kim, Sihoon Choi, Peter D. W. Boyd, Berend Smit, and Jihan Kim. Text mining metal-organic framework papers. *Journal of Chemical Information and Modeling*, 2018.

[30] Kevin Cruse, Amalie Trewartha, Sanghoon Lee, Zheren Wang, Haoyan Huo, Tanjin He, Olga Kononova, Anubhav Jain, and Gerbrand Ceder. Text-mined dataset of gold nanoparticle synthesis procedures, morphologies, and size entities. 2022.

[31] Satanu Ghosh and Kun Lu. Band gap information extraction from materials science literature – a pilot study. *Aslib Journal of Information Management*, 2022.

[32] K. Choudhary and M. L. Kelley. Chemnlp: a natural language processing based library for materials chemistry text data. 2022.

[33] John A. Keith, Valentin Vassilev-Galindo, Bingqing Cheng, Stefan Chmiela, Michael Gastegger, Klaus-Robert Müller, and Alexandre Tkatchenko. Combining machine learning and computational chemistry for predictive insights into chemical systems. *Chemical Reviews*, 2021.

[34] Filipe Rodrigues, Francisco C. Pereira, and Bernardete Ribeiro. Sequence labeling with multiple annotators. *Machine Learning*, 2013.

[35] Buzhou Tang, Ying Feng, Xiaolong Wang, Yonghui Wu, Yaoyun Zhang, Min Jiang, Jingqi Wang, and Hua Xu. A comparison of conditional random fields and structured support vector machines for chemical entity recognition in biomedical literature. *Journal of Cheminformatics*, 2015.

[36] Bernhard E Boser, Isabelle M Guyon, and Vladimir N Vapnik. A training algorithm for optimal margin classifiers. In *Proceedings of the fifth annual workshop on Computational learning theory*, pages 144–152, 1992.

[37] Usman Naseem, Matloob Khushi, Shah Khalid Khan, Kamran Shaukat, and Mohammad Ali Moni. A comparative analysis of active learning for biomedical text mining. *Applied System Innovation*, 2021.

[38] Xi Yang, Jiang Bian, Yan Gong, William R. Hogan, and Yonghui Wu. Madex: A system for detecting medications, adverse drug events, and their relations from clinical notes. *Drug Safety*, 2019.

[39] Alex Graves, Abdelrahman Mohamed, and Geoffrey E. Hinton. Speech recognition with deep recurrent neural networks. 2013.

[40] Youhui Tian. Artificial intelligence image recognition method based on convolutional neural network algorithm. *Ieee Access*, 2020.
[41] Hasim Sak, Andrew W Senior, and Françoise Beaufays. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. 2014.

[42] K Cho, B Van Merriënboer, C Gulcehre, et al. Learning phrase representations using rnn encoder-decoder for statistical machine translation, ‖ in proceedings of the 2014 conference on empirical methods in natural language processing (emnlp), 2014. *Qatar1724–1734*, 2001.

[43] D. Kim. Research on text classification based on deep neural network. *International Journal of Communication Networks and Information Security (IJC-NIS)*, 14:100–113, 2022.

[44] S. Lyu and J. Liu. Combine convolution with recurrent networks for text classification. 2020.

[45] J. Li, C. Wang, and R. Cai. Channel attention convolutional recurrent neural network on street view symbol recognition. *Highlights in Science, Engineering and Technology*, 9:390–397, 2022.

[46] T. Siswantining, S. Pratama, and D. Sarwinda. Spratama model for indonesian paraphrase detection using bidirectional long short-term memory and bidirectional gated recurrent unit. *Media Statistika*, 15:129–138, 2023.

[47] Zhiheng Huang, Wei Xu, and Kai Yu. Bidirectional lstm-crf models for sequence tagging. *arXiv preprint arXiv:1508.01991*, 2015.

[48] T. Mikolov, K. Chen, G. Corrado, and J. Dean. Efficient estimation of word representations in vector space. 2013.

[49] Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. Fasttext.zip: Compressing text classification models. *arXiv preprint arXiv:1612.03651*, 2016.

[50] J. Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation. *Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)*, 2014.

[51] M. J. Peters, M. E. Neumann, M. Iyyer, M. Gardner, C. M. Clark, K. Lee, and L. Zettlemoyer. Deep contextualized word representations. *Proceedings of the 2018 Conference of the North American Chapter Of the Association for Computational Linguistics: Hu*, 2018.

[52] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. 2017.
[53] M. Joshi, O. Levy, L. Zettlemoyer, and D. S. Weld. Bert for coreference resolution: baselines and analysis. *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conferen*, 2019.

[54] J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. *Bioinformatics*, 36:1234–1240, 2019.

[55] R. Joshi, P. Goel, and R. Joshi. Deep learning for hindi text classification: a comparison. *Intelligent Human Computer Interaction*, pages 94–101, 2020.

[56] Joon-Young Jung. Dg-based spo tuple recognition using self-attention m-bi-lstm. *Etri Journal*, 2021.

[57] Walid Hafiane, Joël Legrand, Yannick Toussaint, and Adrien Coulet. Experiments on transfer learning architectures for biomedical relation extraction. 2020.

[58] Young Min Kim and Tae Hoon Lee. Korean clinical entity recognition from diagnosis text using bert. *BMC Medical Informatics and Decision Making*, 2020.

[59] Hu Ng, Glenn Jun Weng Chia, Timothy Tzen Vun Yap, and Vik Tor Goh. Modelling sentiments based on objectivity and subjectivity with self-attention mechanisms. *F1000research*, 2022.

[60] S Shreyashree, Pramod Sunagar, S Rajarajeswari, and Anita Kanavalli. Bert-based hybrid rnn model for multi-class text classification to study the effect of pre-trained word embeddings. *International Journal of Advanced Computer Science and Applications*, 2022.

[61] Mohamed Taha, Hala H. Zayed, Marina Azer, and Mahmoud Gadallah. Automated covid-19 misinformation checking system using encoder representation with deep learning models. *Iaes International Journal of Artificial Intelligence (Ij-Ai)*, 2023.

[62] Ramchandra Joshi. Evaluation of deep learning models for hostility detection in hindi text. 2021.

[63] Abhishek Velankar, Patil Hrushikesh, Amol Gore, Shubham Salunke, and Joshi Raviraj. Hate and offensive speech detection in hindi and marathi. 2021.

[64] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[65] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019.

[66] G. Lample and A. Conneau. Cross-lingual language model pretraining. 2019.

[67] Iz Beltagy, Kyle Lo, and Arman Cohan. SciBERT: A pretrained language model for scientific text. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pages 3615–3620, Hong Kong, China, November 2019. Association for Computational Linguistics.

[68] M. Yasunaga, J. Leskovec, and P. Liang. Linkbert: pretraining language models with document links. 2022.

[69] Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing, 2020.

[70] Zhi Hong, Aswathy Ajith, Gregory Pauloski, Eamon Duede, Carl Malamud, Roger Magoulas, Kyle Chard, and Ian Foster. Scholarbert: Bigger is not always better, 2022.

[71] P. Su and K. Vijay-Shanker. Investigation of improving the pre-training and fine-tuning of bert model for biomedical relation extraction. 2021.

[72] S. Pranav, R. A. Chitteth, K. Christopher, G. Sonkakshi, P. L. Prerana, H. Lauren, and R. Rampi. A general-purpose material property data extraction pipeline from large polymer corpora using natural language processing. *npj Comput Mater*, 9, 2023.

[73] T. D. Huan, A. Mannodi-Kanakkithodi, V. Sharma, G. Pilania, and R. Ramprasad. A polymer dataset for accelerated property prediction and design. *Sci Data*, 3, 2016.

[74] S. Huang and J. M. Cole. Batterybert: a pretrained language model for battery database enhancement. *Journal of Chemical Information and Modeling*, 62:6365–6377, 2022.

[75] Jiang Guo, A. Santiago Ibanez-Lopez, Hanyu Gao, Victor Quach, Connor W. Coley, Klavs F. Jensen, and Regina Barzilay. Automated chemical reaction extraction from scientific literature. *Journal of Chemical Information and Modeling*, 62(9):2035–2045, 2022. PMID: 34115937.
[76] Hidir Aras, René Hackl-Sommer, Michael Schwantner, and Mustafa Sofean. Applications and challenges of text mining with patents. In *IPaMin@ KONVENS*, 2014.

[77] A. S. Maiya, D. W. Visser, and A. C. Wan. Mining measured information from text. *Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval*, 2015.

[78] Milan Agatonovic, Niraj Aswani, Kalina Bontcheva, Hamish Cunningham, Thomas Heitz, Yaoyong Li, Ian Roberts, and Valentin Tablan. Large-scale, parallel automatic patent annotation. In *Proceedings of the 1st ACM workshop on Patent information retrieval*, pages 1–8. ACM, 2008.

[79] Yury Hetsevich and Alena Skopinava. Processing of quantitative expressions with measurement units in the nominative, genitive, and accusative cases for belarusian and russian. In *Text, Speech and Dialogue: 17th International Conference, TSD 2014, Brno, Czech Republic, September 8-12, 2014. Proceedings* 17, pages 101–107. Springer, 2014.

[80] Soumia Lilia Berrahou, Patrice Buche, Juliette Dibie-Barthélemy, and Mathieu Roche. How to extract unit of measure in scientific documents?.

[81] George H John and Pat Langley. Estimating continuous distributions in bayesian classifiers. *arXiv preprint arXiv:1302.4964*, 2013.

[82] Yanna Shen Kang and Mehmet Kayaalp. Extracting laboratory test information from biomedical text. *Journal of pathology informatics*, 4:23–23, August 2013.

[83] Thaer M Dieb, Masaharu Yoshioka, Shinjiro Hara, and Marcus C Newton. Framework for automatic information extraction from research papers on nanocrystal devices. *Beilstein journal of nanotechnology*, 6(1):1872–1882, 2015.

[84] Kyle Hundman and Chris A Mattmann. Measurement context extraction from text: Discovering opportunities and gaps in earth science. In *Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining*. ACM, 2017.

[85] T. Hao, H. Liu, and C. Weng. Valx: a system for extracting and structuring numeric lab test comparison statements from text. *Methods Inf Med*, 55:266–275, 2016.

[86] S. Roy, T. Vieira, and D. Roth. Reasoning about quantities in natural language. *Transactions of the Association for Computational Linguistics*, 3:1–13, 2015.
[87] C. Taha and W. C. K. I. Identifying the units of measurement in tabular data. 2021.

[88] Vinh Thinh Ho, Koninika Pal, and Gerhard Weikum. Qute: Answering quantity queries from web tables. In *Proceedings of the 2021 International Conference on Management of Data*, SIGMOD '21, page 2740–2744, New York, NY, USA, 2021. Association for Computing Machinery.

[89] V. T. Ho, K. Pal, S. Razniewski, K. Berberich, and G. Weikum. Extracting contextualized quantity facts from web tables. *Proceedings of the Web Conference 2021*, 2021.

[90] Dominika Tkaczyk, Paweł Szostek, Mateusz Fedoryszak, Piotr Jan Dendek, and Łukasz Bolikowski. Cermine: automatic extraction of structured metadata from scientific literature. *International Journal on Document Analysis and Recognition (IJDAR)*, 18:317–335, 2015.

[91] Isaac G Councill, C Lee Giles, and Min-Yen Kan. Parscit: an open-source crf reference string parsing package. In *LREC*, volume 8, pages 661–667, 2008.

[92] Dominika Tkaczyk, Andrew Collins, Paraic Sheridan, and Jöran Beel. Evaluation and comparison of open source bibliographic reference parsers: A business use case. *CoRR*, abs/1802.01168, 2018.

[93] Grobid Astro contributors. Grobid-astro: A machine learning software for extracting astronomical entities from scholarly documents. https://github.com/kermitt2/grobid-astro, 2017–2022.

[94] Mohamed Khemakhem, Luca Foppiano, and Laurent Romary. Automatic Extraction of TEI Structures in Digitized Lexical Resources using Conditional Random Fields. In *electronic lexicography, eLex 2017*, Leiden, Netherlands, September 2017.

[95] GROBID Contributors. software-mentions: Grobid module to recognize in textual documents and pdf any mentions of software. https://github.com/Impactstory/software-mentions, 2018 — 2019. [Online; accessed 18-April-2019].

[96] Phil Gooch and Kris Jack. How well does mendeley's metadata extraction work? https://krisjack.wordpress.com/2015/03/12/how-well-does-mendeleys-metadata-extraction-work/.

[97] Josh M Nicholson, Milo Mordaunt, Patrice Lopez, Ashish Uppala, Domenic Rosati, Neves P Rodrigues, Peter Grabitz, and Sean C Rife. Scite: A smart citation index that displays the context of citations and classifies their intent using deep learning. *Quantitative Science Studies*, 2(3):882–898, 2021.
[98] Nipun Sadvilkar and Mark Neumann. Pysbd: Pragmatic sentence boundary disambiguation, 2020.

[99] Edward Loper and Steven Bird. Nltk: The natural language toolkit. In *Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics - Volume 1*, ETMTNLP '02, page 63–70, USA, 2002. Association for Computational Linguistics.

[100] Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. Scispacy: Fast and robust models for biomedical natural language processing. 2019.

[101] Dan Gillick. Sentence boundary detection and the problem with the u.s. In *Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers*, NAACL-Short '09, page 241–244, USA, 2009. Association for Computational Linguistics.

[102] Jonathon Read, Rebecca Dridan, Stephan Oepen, and Lars Jørgen Solberg. Sentence boundary detection: A long solved problem? *Computational Linguistics*, pages 985–994, December 2012.

[103] Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. https://github.com/explosion/spaCy, 2017. To appear.

[104] Beyond Language Understanding Bling Team, Microsoft. Blingfire : A lightning fast finite state machine and regular expression manipulation library. Github: BlingFire sbd repo, 2020.

[105] M. Jiang, Y. Chen, M. Liu, S. T. Rosenbloom, S. Mani, J. C. Denny, and H. Xu. A study of machine-learning-based approaches to extract clinical entities and their assertions from discharge summaries. *Journal of the American Medical Informatics Association*, 18:601–606, 2011.

[106] B. Tang, Y. Wu, M. Jiang, and H. Xu. Recognizing clinical entities in hospital discharge summaries using structural support vector machines with word representation features. *BMC Medical Informatics and Decision Making*, 13, 2013.

[107] T. Isazawa and J. M. Cole. Single model for organic and inorganic chemical named entity recognition in chemdataextractor. *Journal of Chemical Information and Modeling*, 62:1207–1213, 2022.
[108] Patrice Lopez, Caifan Du, Johanna Cohoon, Karthik Ram, and James Howison. *Mining Software Entities in Scientific Literature: Document-Level NER for an Extremely Imbalance and Large-Scale Task*, page 3986–3995. Association for Computing Machinery, New York, NY, USA, 2021.

[109] Thomas Lavergne, Olivier Cappé, and François Yvon. Practical very large scale CRFs. In *Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL)*, pages 504–513. Association for Computational Linguistics, July 2010.

[110] DeLFT contributors. Delft. https://github.com/kermitt2/delft, 2018. [Online; accessed 16-May-2019].

[111] Matthew C. Swain and Jacqueline M. Cole. Chemdataextractor: A toolkit for automated extraction of chemical information from the scientific literature. *Journal of Chemical Information and Modeling*, 56(10):1894–1904, 2016. PMID: 27669338.

[112] Masashi Yoshikawa, Hiroshi Noji, and Yuji Matsumoto. A* ccg parsing with a supertag and dependency factored model. In *Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 277–287. Association for Computational Linguistics, 2017.

[113] Aryeh Tiktinsky, Yoav Goldberg, and Reut Tsarfaty. pybart: Evidence-based syntactic transformations for ie. In *ACL*, 2020.

[114] Swabha Swayamdipta, Sam Thomson, Chris Dyer, and Noah A. Smith. Frame-Semantic Parsing with Softmax-Margin Segmental RNNs and a Syntactic Scaffold. *arXiv preprint arXiv:1706.09528*, 2017.

[115] Junru Zhou and Hai Zhao. Head-driven phrase structure grammar parsing on Penn treebank. In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, Florence, Italy, July 2019. Association for Computational Linguistics.

[116] Yankai Lin, Suhung Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. Neural relation extraction with selective attention over instances. 2016.

[117] Vishnu Hariharan and M. Anand Kumar. Relation extraction using convolutional neural networks. 2019.

[118] Hiroyuki Oka, Atsushi Yoshizawa, Hiroyuki Shindo, Yuji Matsumoto, and Masashi Ishii. Automatic extraction of solvent-names on polymer's solubility from academic articles. In *The 66th JSAP Spring Meeting 2019*, Tokyo, Japan, March 2019. Japan Society of Applied Physics.
[119] Luca Foppiano, Laurent Romary, Masashi Ishii, and Mikiko Tanifuji. Automatic identification and normalisation of physical measurements in scientific literature. In *Proceedings of the ACM Symposium on Document Engineering 2019*, DocEng '19, New York, NY, USA, 2019. Association for Computing Machinery.

[120] Clarissa F. D. Carneiro, Victor G. S. Queiroz, Thiago C. Moulin, Carlos A. M. Carvalho, Clarissa B. Haas, Danielle Rayêe, David E. Henshall, Evandro A. De-Souza, Felippe E. Amorim, Flávia Z. Boos, Gerson D. Guercio, Igor R. Costa, Karina L. Hajdu, Lieve van Egmond, Martin Modrák, Pedro B. Tan, Richard J. Abdill, Steven J. Burgess, Sylvia F. S. Guerra, Vanessa T. Bortoluzzi, and Olavo B. Amaral. Comparing quality of reporting between preprints and peer-reviewed articles in the biomedical literature. *Research Integrity and Peer Review*, 5(1):16, December 2020.

[121] Klaus Krippendorff. Reliability in Content Analysis: Some Common Misconceptions and Recommendations. *Human Communication Research*, 30(3):411–433, 01 2006.

[122] Antonia Zapf, Stefanie Castell, Lars Morawietz, and André Karch. Measuring inter-rater reliability for nominal data – which coefficients and confidence intervals are appropriate? *BMC Medical Research Methodology*, 16(1):93, August 2016.

[123] Jan-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. The inception platform: Machine-assisted and knowledge-oriented interactive annotation. In *Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations*, pages 5–9. Association for Computational Linguistics, June 2018.

[124] Richard Eckart de Castilho, Éva Mújdricza-Maydt, Seid Muhie Yimam, Silvana Hartmann, Iryna Gurevych, Anette Frank, and Chris Biemann. A web-based tool for the integrated annotation of semantic and syntactic structures. In *Proceedings of the Workshop on Language Technology Resources and Tools for Digital Humanities (LT4DH)*, pages 76–84, Osaka, Japan, December 2016. The COLING 2016 Organizing Committee.

[125] Karën Fort and Benoît Sagot. Influence of pre-annotation on POS-tagged corpus development. In *Proceedings of the Fourth Linguistic Annotation Workshop*, pages 56–63, Uppsala, Sweden, July 2010. Association for Computational Linguistics.

[126] Aurélie Névéol, Rezarta Islamaj Dogan, and Zhiyong Lu. Semi-automatic semantic annotation of pubmed queries: A study on quality, efficiency, satisfaction. *Journal of biomedical informatics*, 44 2:310–8, 2011.
[127] Todd Lingren, Louise Deleger, Katalin Molnar, Haijun Zhai, Jareen Meinzen-Derr, Megan Kaiser, Laura Stoutenborough, Qi Li, and Imre Solti. Evaluating the impact of pre-annotation on annotation speed and potential bias: natural language processing gold standard development for clinical named entity recognition in clinical trial announcements. *Journal of the American Medical Informatics Association : JAMIA*, 21(3):406–413, 2014.

[128] James Pustejovsky and Amber Stubbs. *Natural Language Annotation for Machine Learning: A guide to corpus-building for applications.* ” O’Reilly Media, Inc.”, 2012.

[129] Thaer M. Dieb, Masaharu Yoshioka, and Shinjiro Hara. Nadev: An annotated corpus to support information extraction from research papers on nanocrystal devices. *Journal of Information Processing*, 24:554–564, jan 2016.

[130] Martin Krallinger, Obdulia Rabal, Florian Leitner, Miguel Vazquez, David Salgado, Zhiyong Lu, Robert Leaman, Yanan Lu, Dong-Hong Ji, Daniel M. Lowe, Roger A. Sayle, Riza Theresa Batista-Navarro, Rafal Rak, Torsten Huber, Tim Rocktäschel, Sérgio Matos, David Campos, Buzhou Tang, Hua Xu, Tsendsuren Munkhdalai, Keun Ho Ryu, S. V. Ramanan, P. Senthil Nathan, Slavko Zitnik, Marko Bajec, Lutz Weber, Matthias Irmer, Saber A. Akhondi, Jan A. Kors, Shuo Xu, Xin An, Utpal Kumar Sikdar, Asif Ekbal, Masaharu Yoshioka, Thaer M. Dieb, Miji Choi, Karin M. Verspoor, Madian Khabsa, C. Lee Giles, Hongfang Liu, K. E. Ravikumar, Andre Lamurias, Francisco M. Couto, Hong-Jie Dai, Richard Tzong-Han Tsai, Caglar Ata, Tolga Can, Anabel Usie, Rui Alves, Isabel Segura-Bedmar, Paloma Martínez, Julen Oyarzábal, and Alfonso Valencia. The chemdner corpus of chemicals and drugs and its annotation principles. *Journal of Cheminformatics*, 7:S2 – S2, 2015.

[131] Christian M. Meyer, Margot Mieskes, Christian Stab, and Iryna Gurevych. DKPro agreement: An open-source Java library for measuring inter-rater agreement. In *Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: System Demonstrations*, pages 105–109, Dublin, Ireland, August 2014. Dublin City University and Association for Computational Linguistics.

[132] Skopinava AM and Lobanov BM. Processing of quantitative expressions with units of measurement in scientific texts as applied to belarusian and russian text-to-speech synthesis. 2013.

[133] Units of measurement. https://github.com/unitsofmeasurement.

[134] Luca Foppiano, Akira Suzuki, Thaer M Dieb, Masashi Ishii, and Mikiko Tan-ifuji. Leveraging segmentation of physical units through a newly open source
corpus. In *JSAP Annual Meetings Extended Abstracts The 80th JSAP Autumn Meeting 2019*, pages 4097–4097. The Japan Society of Applied Physics, 2019.

[135] B Roter and SV Dordevic. Predicting new superconductors and their critical temperatures using machine learning. *Physica C: Superconductivity and its applications*, 575:1353689, 2020.

[136] Huan Tran and Tuoc N Vu. Machine-learning approach for discovery of conventional superconductors. *arXiv preprint arXiv:2211.03265*, 2022.

[137] Tomohiko Konno, Hodaka Kurokawa, Fuyuki Nabeshima, Yuki Sakishita, Ryo Ogawa, Iwao Hosako, and Atsutaka Maeda. Deep learning model for finding new superconductors. *Physical Review B*, 103(1):014509, 2021.

[138] Jan-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. The INCEPITION platform: Machine-assisted and knowledge-oriented interactive annotation. In *Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations*, pages 5–9, Santa Fe, New Mexico, 2018.

[139] Hiroki Nakayama, Takahiro Kubo, Junya Kamura, Yasufumi Taniguchi, and Xu Liang. doccano: Text annotation tool for human, 2018. Software available from https://github.com/doccano/doccano.

[140] James Kirkpatrick, Razvan Pascanu, Neil C. Rabinowitz, Joel Veness, Guil-laume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. *CoRR*, abs/1612.00796, 2016.
# List of Publications

## Refereed Journal Paper

1. Luca Foppiano, Pedro Baptista Castro, Pedro Ortiz Suarez, Kensei Terashima, Yoshihiko Takano & Masashi Ishii (2023) Automatic extraction of materials and properties from superconductors scientific literature, Science and Technology of Advanced Materials: Methods, 3:1, DOI: https://doi.org/10.1080/27660400.2022.2153633

2. Luca Foppiano, Sae Dieb, Akira Suzuki, Pedro Baptista de Castro, Suguru Iwasaki, Azusa Uzuki, Miren Garbine Esparza Echevarria, Yan Meng, Kensei Terashima, Laurent Romary, Yoshihiko Takano & Masashi Ishii (2021) SuperMat: construction of a linked annotated dataset from superconductors-related publications, Science and Technology of Advanced Materials: Methods, 1:1, 34-44, DOI: https://doi.org/10.1080/27660400.2021.1918396

3. Luca Foppiano, Tomoya Mato, Kensei Terashima, Pedro Ortiz Suarez, Taku Tou, Chikako Sakai, Wei-Sheng Wang, Toshiyuki Amagasa, Yoshihiko Takano & Masashi Ishii (2023) Semi-automatic staging area for high-quality structured data extraction from scientific literature, Science and Technology of Advanced Materials: Methods, 3:1, DOI: https://doi.org/10.1080/27660400.2023.2286219

## Refereed Conference Papers

1. Luca Foppiano, Laurent Romary, Masashi Ishii, and Mikiko Tanifuji. 2019. Automatic Identification and Normalisation of Physical Measurements in Scientific Literature. In Proceedings of the ACM Symposium on Document Engineering 2019 (DocEng ’19). Association for Computing Machinery, New York, NY, USA, Article 24, 1–4. https://doi.org/10.1145/3342558.3345411