<figure><img src="image_1.png" /></figure>

HAL
open science

Automated Extraction and Curation of Materials
Information from Scientific Literature

Luca Foppiano

<figure><img src="image_2.png" /></figure>

To cite this version:

Luca Foppiano. Automated Extraction and Curation of Materials Information from Scientific Literature. Document and Text Processing. University of Tsukuba, 2023. English. NNT: tel-04465467

HAL Id: tel-04465467

https://hal.science/tel-04465467

Submitted on 9 Apr 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_3.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Automated Extraction and Curation of Materials
Information from Scientific Literature

March 2024

Luca Foppiano
Automated Extraction and Curation of Materials
Information from Scientific Literature

Graduate School of Science and Technology
Degree Programs in Systems and Information Engineering
University of Tsukuba

March 2024

Luca Foppiano
Abstract

The scientific literature, which contains vast human knowledge, is rapidly ex-
panding, posing challenges in organising and retrieving information. The use of
big data techniques aids in uncovering patterns and making predictions and has
long been applied in chemistry and biology. While materials science has fallen be-
hind, a new discipline called Materials Informatics (MI) has emerged in recent years,
thanks to large collaborative projects such as the Material Project. MI is a disci-
pline that leverages computational power to accelerate research in Materials Science
by employing techniques like Density Functional Theory (DFT) computations and
Machine Learning (ML). Despite advances, the limited availability of experimental
datasets, such as SuperCon or the Pauling File, hinders progress. SuperCon is a
database for superconductor materials constructed manually by Japan's National
Institute for Materials Science (NIMS). NIMS faces challenges updating SuperCon
manually due to high publication rates and scarcity of skilled labour force. Auto-
mated processes are needed to extract data from new publications promptly, and
manual curation, which is prone to errors, requires careful tools and feature selection.

In this dissertation, we propose an end-to-end pipeline for extracting material
information from the scientific literature to improve the efficiency and quality of ma-
terial databases. Our work aims to combine automated tasks and efficient tooling
to reduce the dependency on human intelligence to the minimum. The automatic
extraction pipeline processes scientific documents in PDF format and combines ML-
based models for recognising complex material-related expressions. Material ex-
pressions are further processed by a specialised "Material parser" that decomposes
granular information such as name, formulas, doping, shape, etc. The extraction of
properties and conditions (e.g., 3K, 24 GPa, 12 atm) is performed by exploiting a
general parser for extracting measurements of physical quantities: Grobid-quantities.
Grobid-quantities support the identification and normalisation of measurements to
the International System (SI) base units and allow the pipeline to support properties
and conditions from various domains flexibly. Furthermore, to obtain the necessary
high-quality training data for our ML-based processes, we developed SuperMat, a
dataset of 164 superconductor articles for evaluation and training. Superconductor
researchers constructed and validated the dataset and provided a structure contain-
ing annotated entities and relations. We demonstrate the efficacy of our pipeline by
processing a large set of scientific articles from the Arixv repository and collecting
a database with over 40000 extracted material-properties records. Finally, we pro-
posed a curation workflow to validate the extracted data by combining an enhanced
PDF viewer and a comprehensive interface. In general, using such a tool improved
the quality of the extracted data with an increased precision by 6% and recall by 47%
compared to the traditional manual approach of reading the plain PDF document
and writing the data in an Excel file. The models, dataset, and interface developed
in this work will help increase the automation and accuracy of the processing of
materials databases such as SuperCon from the scientific literature.
<figure><img src="image_4.png" /></figure>
Acknowledgements

I express my deepest gratitude to all those who have contributed directly or indirectly to the completion of this dissertation, as their support and encouragement have been invaluable throughout this academic journey. First and foremost, I extend my sincere appreciation to my advisor, Professor Toshiyuki Amagasa, for his unwavering guidance, patience, and expertise. Their insightful feedback and constructive criticism have played a pivotal role in shaping the content and direction of this research. I am grateful to the members of my dissertation committee, Professor Hiroyuki Kitagawa, Professor Takehito Utsuro, Associate Professor Takashi Inui, and Dr. Masashi Ishii, for their time, expertise and valuable insights. Their diverse perspectives enriched the quality of my work and provided me with a broader understanding of the subject matter.

I want to express my heartfelt gratitude to Dr. Masashi Ishii for his invaluable support and guidance during my time at NIMS (National Institute for Materials Science). This research has greatly benefited from fruitful discussions and collaboration with Sae Dieb and Guillaume Lambard. I owe my ability to steer my career towards academic research in computer science and machine learning to the invaluable guidance, assistance, and support of Dr. Patrice Lopez. For over a decade, Dr. Lopez has been a constant source of inspiration, providing me with the necessary tools and motivation to pursue my academic endeavours. I express my gratitude to Dr. Laurent Romary for his assistance during my time at the National Institute for Research in Digital Science and Technology (INRIA), in Paris. Additionally, I thank Dr. Mikiko Tanifuji for her support during my remarkable journey to Japan.

I have been fortunate to have many individuals who have positively impacted my career, life, and accomplishments. Unfortunately, I am unable to mention each of them individually due to space constraints.
# Contents

<table><tr><td>Abstract</td><td>i</td></tr><tr><td>Acknowledgements</td><td>ii</td></tr><tr><td>List of Figures</td><td>ix</td></tr><tr><td>List of Tables</td><td>xii</td></tr><tr><td>1 Introduction</td><td>1</td></tr><tr><td>1.1 Motivation</td><td>4</td></tr><tr><td>1.2 Problem definition</td><td>6</td></tr><tr><td>1.3 Contributions</td><td>6</td></tr><tr><td>1.3.1 Extraction from the full-text body of PDF documents</td><td>7</td></tr><tr><td>1.3.2 ML-based extraction of materials-related expressions from text</td><td>7</td></tr><tr><td>Identification of complex material sequences</td><td>8</td></tr><tr><td>Construction of SuperMat: an annotated and linked dataset of superconductors research papers</td><td>9</td></tr><tr><td>Parsing and normalising materials sequences</td><td>9</td></tr><tr><td>1.3.3 Extraction of properties and conditions as measurements of physical quantities</td><td>10</td></tr></table>
<table>
  <tbody>
    <tr>
      <td>1.3.4 Large scale collection of experimental data from scientific literature: SuperCon<sup>2</sup> Database</td>
      <td>10</td>
    </tr>
    <tr>
      <td>1.3.5 Reducing the impact of manual curation</td>
      <td>11</td>
    </tr>
    <tr>
      <td><b>2 Related work</b></td>
      <td><b>12</b></td>
    </tr>
    <tr>
      <td>2.1 Text and data mining in materials science</td>
      <td>12</td>
    </tr>
    <tr>
      <td>2.2 Machine Learning</td>
      <td>15</td>
    </tr>
    <tr>
      <td>2.3 Transformers</td>
      <td>17</td>
    </tr>
    <tr>
      <td>2.4 Materials science BERT-based pre-trained transformers</td>
      <td>19</td>
    </tr>
    <tr>
      <td>2.5 Extraction of quantified properties and measurements</td>
      <td>21</td>
    </tr>
    <tr>
      <td><b>3 Extraction from the full-text body of PDF documents</b></td>
      <td><b>25</b></td>
    </tr>
    <tr>
      <td>3.1 Introduction</td>
      <td>25</td>
    </tr>
    <tr>
      <td>3.2 Proposed approach</td>
      <td>26</td>
    </tr>
    <tr>
      <td>3.2.1 Publisher's agreement for source data</td>
      <td>27</td>
    </tr>
    <tr>
      <td>3.2.2 Abstract versus full-text</td>
      <td>27</td>
    </tr>
    <tr>
      <td>3.3 TDM pipeline</td>
      <td>28</td>
    </tr>
    <tr>
      <td>3.3.1 Document structuring and segmentation</td>
      <td>28</td>
    </tr>
    <tr>
      <td>3.3.2 Sentence segmentation</td>
      <td>29</td>
    </tr>
    <tr>
      <td>3.4 Conclusions</td>
      <td>30</td>
    </tr>
    <tr>
      <td><b>4 ML-based extraction of materials-related expressions from text</b></td>
      <td><b>32</b></td>
    </tr>
    <tr>
      <td>4.1 Introduction</td>
      <td>32</td>
    </tr>
    <tr>
      <td>4.2 Identification of complex materials sequences</td>
      <td>33</td>
    </tr>
    <tr>
      <td>4.2.1 High-quality training data</td>
      <td>33</td>
    </tr>
  </tbody>
</table>
<table><tr><td>4.2.2</td><td>Positive sampling</td><td>34</td></tr><tr><td>4.2.3</td><td>ML architectures</td><td>34</td></tr><tr><td>4.2.4</td><td>Two-levels approach</td><td>35</td></tr><tr><td></td><td>Superconductor parser</td><td>35</td></tr><tr><td></td><td>Material parser</td><td>37</td></tr><tr><td>4.2.5</td><td>Post-processing</td><td>37</td></tr><tr><td>4.2.6</td><td>Extraction of relation from materials-related entities</td><td>37</td></tr><tr><td>4.3</td><td>Results</td><td>39</td></tr><tr><td>4.3.1</td><td>Identification of materials-related entities from text</td><td>39</td></tr><tr><td></td><td>Experimental settings</td><td>39</td></tr><tr><td></td><td>Results</td><td>40</td></tr><tr><td>4.3.2</td><td>Material parser segmentation model</td><td>42</td></tr><tr><td></td><td>Experimental settings</td><td>42</td></tr><tr><td></td><td>Results</td><td>42</td></tr><tr><td>4.3.3</td><td>RE evaluation</td><td>43</td></tr><tr><td>4.3.4</td><td>End to end evaluation</td><td>43</td></tr><tr><td>4.4</td><td>Conclusions</td><td>45</td></tr></table>

5 SuperMat: Construction of a linked annotated dataset from superconductors-related publications 53

<table><tr><td>5.1</td><td>Introduction</td><td>53</td></tr><tr><td>5.2</td><td>Content acquisition</td><td>54</td></tr><tr><td>5.3</td><td>Preliminary annotation study</td><td>54</td></tr><tr><td>5.4</td><td>Tag-set design</td><td>55</td></tr></table>
<table>
  <tbody>
    <tr>
      <td>5.4.1</td>
      <td>Top-level Entities</td>
      <td>55</td>
    </tr>
    <tr>
      <td>5.4.2</td>
      <td>Level-2 entities</td>
      <td>58</td>
    </tr>
    <tr>
      <td>5.4.3</td>
      <td>Relations</td>
      <td>59</td>
    </tr>
    <tr>
      <td>5.4.4</td>
      <td>Annotation guidelines</td>
      <td>60</td>
    </tr>
    <tr>
      <td>5.5</td>
      <td>Annotation support tools</td>
      <td>60</td>
    </tr>
    <tr>
      <td>5.5.1</td>
      <td>Web-based collaborative annotation tool: INCEpTION</td>
      <td>60</td>
    </tr>
    <tr>
      <td>5.5.2</td>
      <td>Annotation suggestions</td>
      <td>61</td>
    </tr>
    <tr>
      <td>5.5.3</td>
      <td>Automatic corpus analysis</td>
      <td>61</td>
    </tr>
    <tr>
      <td>5.6</td>
      <td>Annotation process</td>
      <td>63</td>
    </tr>
    <tr>
      <td>5.7</td>
      <td>Data transformation</td>
      <td>64</td>
    </tr>
    <tr>
      <td>5.8</td>
      <td>Data Record</td>
      <td>65</td>
    </tr>
    <tr>
      <td>5.9</td>
      <td>Practical applications</td>
      <td>68</td>
    </tr>
    <tr>
      <td>5.10</td>
      <td>Technical Validation</td>
      <td>69</td>
    </tr>
    <tr>
      <td>5.11</td>
      <td>Data Availability</td>
      <td>71</td>
    </tr>
    <tr>
      <td>5.12</td>
      <td>Conclusions</td>
      <td>72</td>
    </tr>
    <tr>
      <td>6</td>
      <td>Extraction of properties and conditions as measurements of physi-cal quantities</td>
      <td>73</td>
    </tr>
    <tr>
      <td>6.1</td>
      <td>Introduction</td>
      <td>73</td>
    </tr>
    <tr>
      <td>6.2</td>
      <td>Proposed solution</td>
      <td>74</td>
    </tr>
    <tr>
      <td>6.2.1</td>
      <td>Data model</td>
      <td>74</td>
    </tr>
    <tr>
      <td>6.2.2</td>
      <td>Tokenisation</td>
      <td>75</td>
    </tr>
    <tr>
      <td>6.2.3</td>
      <td>Extraction</td>
      <td>75</td>
    </tr>
    <tr>
      <td>6.2.4</td>
      <td>Normalisation</td>
      <td>78</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>6.3</td>
      <td>Evaluation and results</td>
      <td>78</td>
    </tr>
    <tr>
      <td>6.4</td>
      <td>Conclusions</td>
      <td>80</td>
    </tr>
    <tr>
      <td>7</td>
      <td>Large scale collection of experimental data from scientific literature: SuperCon<sup>2</sup> Database</td>
      <td>81</td>
    </tr>
    <tr>
      <td>7.1</td>
      <td>Introduction</td>
      <td>81</td>
    </tr>
    <tr>
      <td>7.2</td>
      <td>Database construction</td>
      <td>82</td>
    </tr>
    <tr>
      <td>7.3</td>
      <td>Results</td>
      <td>86</td>
    </tr>
    <tr>
      <td>7.4</td>
      <td>Conclusions</td>
      <td>87</td>
    </tr>
    <tr>
      <td>8</td>
      <td>Reducing the impact of manual curation</td>
      <td>89</td>
    </tr>
    <tr>
      <td>8.1</td>
      <td>Introduction</td>
      <td>89</td>
    </tr>
    <tr>
      <td>8.2</td>
      <td>Curation workflow</td>
      <td>90</td>
    </tr>
    <tr>
      <td>8.2.1</td>
      <td>Workflow control</td>
      <td>90</td>
    </tr>
    <tr>
      <td></td>
      <td>Curation status</td>
      <td>92</td>
    </tr>
    <tr>
      <td></td>
      <td>Error types</td>
      <td>92</td>
    </tr>
    <tr>
      <td>8.2.2</td>
      <td>Anomaly detection</td>
      <td>93</td>
    </tr>
    <tr>
      <td>8.2.3</td>
      <td>Automatic training data collector</td>
      <td>93</td>
    </tr>
    <tr>
      <td></td>
      <td>Training data collection</td>
      <td>94</td>
    </tr>
    <tr>
      <td></td>
      <td>Training data management</td>
      <td>94</td>
    </tr>
    <tr>
      <td>8.3</td>
      <td>Curation interface</td>
      <td>95</td>
    </tr>
    <tr>
      <td>8.3.1</td>
      <td>Manual curation approach</td>
      <td>95</td>
    </tr>
    <tr>
      <td>8.3.2</td>
      <td>Curation guidelines</td>
      <td>97</td>
    </tr>
    <tr>
      <td>8.3.3</td>
      <td>Curation and processing logs</td>
      <td>97</td>
    </tr>
    <tr>
      <td>8.4</td>
      <td>Results and evaluation</td>
      <td>97</td>
    </tr>
  </tbody>
</table>
<table><tr><td>8.4.1</td><td>Anomaly detection rejection rate</td><td>99</td></tr><tr><td>8.4.2</td><td>Training data generation</td><td>99</td></tr><tr><td>8.4.3</td><td>Data quality</td><td>101</td></tr><tr><td></td><td>Discussion</td><td>102</td></tr><tr><td>8.5</td><td>Code availability</td><td>106</td></tr><tr><td>8.6</td><td>Conclusions</td><td>106</td></tr><tr><td>9</td><td>Conclusion</td><td>107</td></tr><tr><td></td><td>Bibliography</td><td>109</td></tr><tr><td></td><td>List of Publications</td><td>123</td></tr></table>
List of Figures

<table><tr><td>3.1</td><td>Processing pipeline for extracting superconductors materials and properties.</td><td>28</td></tr><tr><td>3.2</td><td>Grobid-superconductors extraction processes (bibliographic information, superconductor entity extraction and sentence segmentation) within the Grobid cascade data flow.</td><td>29</td></tr><tr><td>4.1</td><td>2-level architecture for solving the NER task. The white rectangles indicate the extracted information (described in Tables 4.2 and 4.3).</td><td>36</td></tr><tr><td>4.2</td><td>Holdout/training set distribution for (a) general metrics and (b) entity labels; entities and unique entities indicate the number of labelled entities with and without value duplicates, respectively, and positive examples (+) and negative examples (-) indicate the number of sentences with at least one entity and with no entities, respectively.</td><td>40</td></tr><tr><td>4.3</td><td>Holdout “out-of-domain” rates. The entities from the holdout set that are also in the training set are “in-domain”, and the entities that are not in the training set are “out-of-domain”.</td><td>40</td></tr><tr><td>4.4</td><td>Examples taken from two sources [1, 2] of results from three different architectures: CRF, BidLSTM_CRF and, SciBERT. The boxes annotating the text represent the extracted entities (material are indicated in light blue, T<sub>c</sub> in green, and T<sub>c</sub> expressions in yellow).</td><td>41</td></tr><tr><td>4.5</td><td>Holdout/training set for the Material ML model: (a) general metrics and (b) entity labels.</td><td>42</td></tr></table>
<table>
  <tbody>
    <tr>
      <td>4.6</td>
      <td>Holdout "out-of-domain" rates for the Material ML model. The entities from the holdout set that are also in the training set are the in-domain, and the entities that are not in the training set are the out-of-domain.</td>
      <td>43</td>
    </tr>
    <tr>
      <td>4.7</td>
      <td>Error types in the context of the data flow.</td>
      <td>44</td>
    </tr>
    <tr>
      <td>4.8</td>
      <td>Error type distribution in the E2EE of the 500-papers dataset.</td>
      <td>45</td>
    </tr>
    <tr>
      <td>5.1</td>
      <td>Example in the annotated corpus. Excerpt from © 2009 The Physical Society of Japan (J. Phys. Soc. Jpn. 78, 123707)</td>
      <td>56</td>
    </tr>
    <tr>
      <td>5.2</td>
      <td>Annotation workflow. Different colours illustrate the involvement of each group at each step of the workflow.</td>
      <td>63</td>
    </tr>
    <tr>
      <td>5.3</td>
      <td>Summary of the data transformation flows.</td>
      <td>64</td>
    </tr>
    <tr>
      <td>5.4</td>
      <td>Distribution of paper in the dataset by (a) publisher, and (b) year of publication.</td>
      <td>66</td>
    </tr>
    <tr>
      <td>5.5</td>
      <td>INCEPTION curation interface. Excerpt from © 2004 The Physical Society of Japan (J. Phys. Soc. Jpn. 73, 1655-1656)</td>
      <td>71</td>
    </tr>
    <tr>
      <td>6.1</td>
      <td>Schema of the data model, from the data parsing and normalisation point of view.</td>
      <td>75</td>
    </tr>
    <tr>
      <td>6.2</td>
      <td>The cascade approach of the applied parsers. The Quantities parser recognises values and units passed to Values and Units parsers for further extraction.</td>
      <td>76</td>
    </tr>
    <tr>
      <td>7.1</td>
      <td>Ingestion process. The numbers between parentheses represent the order in which each operation is performed.</td>
      <td>82</td>
    </tr>
    <tr>
      <td>7.2</td>
      <td>Example of the information from one single entity from a passage extracted in the "Extraction Task". The different structured information is highlighted: links, attributes, PDF entities coordinate (to visualise annotations on the PDF document), and annotations references within the passage (to visualise annotations on text).</td>
      <td>84</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>8.1</td>
      <td>Schema of the curation workflow. Each node has type and status properties (Section 8.2.1). Each edge indicates one action. The workflow starts on the left side of the figure. The new records begin with “Automatic, New”. Changes of state are triggered by automatic (Section 8.2.2) or manual operations (update, mark as valid, etc.). Section 8.3.1) and results in changes of the properties in the node. Each combination of property values identifies each state. “(*)” indicates a transition for which the training data are collected (Section 8.2.3)</td>
      <td>91</td>
    </tr>
    <tr>
      <td>8.2</td>
      <td>Screenshot of the training data management page in the SuperCon<sup>2</sup> interface. Each row contains one potential training data example. Each example comprises a sentence and its extracted entities (highlighted in colour) with potential annotation mistakes that need to be corrected using an external tool: we used Label-Studio [3]. The column “Status” indicates whether the example has been sent to the external tool.</td>
      <td>95</td>
    </tr>
    <tr>
      <td>8.3</td>
      <td>Screenshot of SuperCon<sup>2</sup> interface showing the database. Each row corresponds to one material-T<sub>c</sub> pair. On the top, there are searches by attribute, sorting and other filtering operations. Curation controls (mark as valid, update, etc.) are on the right (last column). Records are grouped by document with alternating light yellow and white.</td>
      <td>96</td>
    </tr>
    <tr>
      <td>8.4</td>
      <td>PDF document viewer showing an annotated document. The table on top is linked through the annotated entities. The user can navigate from the record to the exact point in the PDF, with a pointer (the red bulb light) identifying the context of the examined entities.</td>
      <td>96</td>
    </tr>
    <tr>
      <td>8.5</td>
      <td>Sample curation sheet from the curation guidelines. The sheet is composed of the following information: a) Sample input data: a screenshot of the record from the “SuperCon<sup>2</sup> Interface”, b) Context represented by the related part of the annotated document referring to the record in exams. c) The Motivation, describing the issue, d) the Action to be taken, and the Expected output.</td>
      <td>98</td>
    </tr>
    <tr>
      <td>8.6</td>
      <td>Top: Processing log, showing the output of each ingestion operation and the outcome with the detailed error that may have occurred. Bottom: Correction log, indicating each record, the number of updates, and the date/time of the last updates. Clicking on the “Record id” shows the latest record values.</td>
      <td>98</td>
    </tr>
  </tbody>
</table>
List of Tables

<table><tr><td>3.1</td><td>Results from cross-validation for sentence-based and paragraphs-based text. Measurements are micro average. P: Precision, R: Recall, and F1: F1-score.</td><td>30</td></tr><tr><td>4.1</td><td>Summary of the features used in the superconductors and material ML models. All under Architecture indicate only BidLSTM_CRF FEATURES and CRF.</td><td>46</td></tr><tr><td>4.2</td><td>Entities extracted by the superconductors parser.</td><td>47</td></tr><tr><td>4.3</td><td>Entities extracted by the material parser.</td><td>47</td></tr><tr><td>4.4</td><td>Evaluation scores (%) for the Superconductor ML model in the four architectures. For the DL architecture, the results are averaged over five runs. Support (Supp) indicates the number of labels in the training data. Values in bold indicate the highest score. P: precision, R: recall.</td><td>48</td></tr><tr><td>4.5</td><td>Holdout/Training set distribution (%) between training and holdout sets for the Superconductor ML model. Positive examples indicate the number of sentences with at least one entity, and negative examples the number of sentences with no entities.</td><td>49</td></tr><tr><td>4.6</td><td>Holdout/Training set distribution (%) between training and holdout sets on different labels for the Superconductors ML model.</td><td>49</td></tr><tr><td>4.7</td><td>Holdout/Training set distribution (%) training and holdout sets for the Material ML model.</td><td>49</td></tr><tr><td>4.8</td><td>Holdout/Training set distribution (%) training and holdout sets on different labels for the Material ML model.</td><td>50</td></tr></table>
<table>
  <tbody>
    <tr>
      <td>4.9</td>
      <td>Evaluation scores (%) of the Material ML model with holdout set. Support (Supp) indicates the number of labels in the training data. Values in bold indicate the highest score. P: precision, R: recall.</td>
      <td>51</td>
    </tr>
    <tr>
      <td>4.10</td>
      <td>Evaluation scores for the Linking. Support (Supp) indicates the number of labels in the training data. Values in bold indicate the highest score. P: precision, R: recall.</td>
      <td>52</td>
    </tr>
    <tr>
      <td>4.11</td>
      <td>Summary of the E2EE evaluation scores. Support indicates the number of labels in the training data.</td>
      <td>52</td>
    </tr>
    <tr>
      <td>5.1</td>
      <td>Summary of the IAA for each annotation iteration.</td>
      <td>55</td>
    </tr>
    <tr>
      <td>5.2</td>
      <td>Statistical overview of the dataset. Level-2 entities are considered only for materials Relations<sub>is</sub> indicates the number of relations within the same sentence (intra-sentence). Relations<sub>es</sub> indicate the number of relations from different paragraphs (extra-sentence).</td>
      <td>62</td>
    </tr>
    <tr>
      <td>5.3</td>
      <td>Inconsistencies resulting from human mistakes.</td>
      <td>62</td>
    </tr>
    <tr>
      <td>5.4</td>
      <td>Inconsistencies resulting from the overlapping of &lt;material&gt; and &lt;class&gt; labels.</td>
      <td>62</td>
    </tr>
    <tr>
      <td>5.5</td>
      <td>Average IAA between the annotated and validated documents</td>
      <td>69</td>
    </tr>
    <tr>
      <td>5.6</td>
      <td>Calculated IAA for annotations produced by domain experts, non-domain experts, and novices compared to the validated version. Annotations from domain experts are cross validated.</td>
      <td>70</td>
    </tr>
    <tr>
      <td>6.1</td>
      <td>Labels description for the Quantities parser. The values referred to by the label are highlighted in bold.</td>
      <td>76</td>
    </tr>
    <tr>
      <td>6.2</td>
      <td>Labels description for the Units parser. In bold are highlighted specific examples.</td>
      <td>77</td>
    </tr>
    <tr>
      <td>6.3</td>
      <td>Labels description for the Values parser. In bold are highlighted specific examples.</td>
      <td>78</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>6.4</td>
      <td>Evaluation scores (%) of the Quantities ML model with holdout set. Support (Supp) indicates the number of labels in the training data. Values in bold indicate the highest score. P: precision, R: recall. Deep learning results are averaged over five independent runs of training and evaluation.</td>
      <td>79</td>
    </tr>
    <tr>
      <td>6.5</td>
      <td>Evaluation scores (%) of the Units ML model with holdout set. Support (Supp) indicates the number of labels in the training data. Values in bold indicate the highest score. P: precision, R: recall.</td>
      <td>79</td>
    </tr>
    <tr>
      <td>6.6</td>
      <td>Evaluation scores (%) of the Values ML model with holdout set. Support (Supp) indicates the number of labels in the training data. Values in bold indicate the highest score. P: precision, R: recall.</td>
      <td>80</td>
    </tr>
    <tr>
      <td>7.1</td>
      <td>Comparison in volumes from Supercon and SuperCon<sup>2</sup>.</td>
      <td>87</td>
    </tr>
    <tr>
      <td>7.2</td>
      <td>Summary and description of the SuperCon<sup>2</sup> schema. "Internal information" is technical information not accessible to the users.</td>
      <td>88</td>
    </tr>
    <tr>
      <td>8.1</td>
      <td>F1-score from the evaluation of the fine-tuned SciBERT models. The training is performed with three different approaches. The base dataset is the original dataset described in [4], and the curation dataset is automatically collected based on the database corrections by the interface and manually corrected. s indicate "training from scratch", while i indicate "incremental training". The evaluation uses the same holdout dataset from SuperMat [4]. The results are averaged over 5 runs of train and evaluation.</td>
      <td>100</td>
    </tr>
    <tr>
      <td>8.2</td>
      <td>Data support, the number of entities for each label in each dataset used for evaluating the ML models. The base dataset is the original dataset described in [4], and the curation dataset is automatically collected based on the database corrections by the interface and manually corrected.</td>
      <td>101</td>
    </tr>
    <tr>
      <td>8.3</td>
      <td>Timetable recording the time spent for each of the 15 articles. Each row indicates the time and the event (Start, Finish) from each of the curators: Master Student (MD), PhD Student (PD), and Senior Researcher (SR). Duration is expressed in minutes.</td>
      <td>103</td>
    </tr>
    <tr>
      <td>8.4</td>
      <td>Evaluation scores obtained for each document and method (I: Interface, P: PDF) combination. TP: True positive, FP: False positive, FN: False negative. P: Precision, R: Recall, F1: F1-score</td>
      <td>104</td>
    </tr>
  </tbody>
</table>
<table>
  <tbody>
    <tr>
      <td>8.5</td>
      <td>Evaluation scores (P: precision, R: recall, F1: F1-score) between the curation using the SuperCon<sup>2</sup> interface (Interface) and the traditional method of reading the PDF document (PDF document).</td>
      <td>105</td>
    </tr>
    <tr>
      <td>8.6</td>
      <td>Evaluation scores (P: precision, R: recall, F1: F1-score) aggregated by experience (MS: master student, PD: PhD student, SR: senior researcher). Each person corrected ten documents.</td>
      <td>105</td>
    </tr>
    <tr>
      <td>8.7</td>
      <td>Evaluation scores (P: precision, R: recall, F1: F1-score) listed by experience (MS: master student, PD: PhD student, SR: senior researcher), and method (PDF document, Interface).</td>
      <td>105</td>
    </tr>
  </tbody>
</table>
# Chapter 1

# Introduction

Materials science is a multidisciplinary field at the intersection of physics, chemistry, and engineering, dedicated to understanding, designing, and manipulating materials for many real applications. Central to this discipline is the exploration of the structure, properties, and performance of materials, ranging from metals and ceramics to polymers and composites. By delving into the microscopic and atomic scales, materials scientists seek to uncover the fundamental principles that govern a material’s behaviour and tailor its properties to meet specific technological needs. This field plays a critical role in advancing technology and innovation, as discoveries in materials science often lead to the development of new materials with enhanced functionalities, improved performance, and novel applications across industries.

Historically, breakthroughs in materials science often resulted from chance discoveries and unexpected observations, a process driven by serendipity and relying on trial and error. However, with the advent of advanced computational tools, machine learning, and big data analytics, materials informatics (MI) has emerged as a transformative methodology. MI is a multidisciplinary field that leverages computational methods, data science, and informatics techniques to accelerate the discovery and design of new materials, identify patterns, and predict material properties with unprecedented accuracy. MI expedites the discovery of novel materials and enhances our understanding of complex relationships between structure, composition, and performance. This approach allows for more efficient and systematic exploration of the vast material space, potentially leading to breakthroughs in electronics, energy, healthcare, and beyond.

In comparison of MI with well-established fields like computational chemistry and biology, we can discern the unique contributions of each discipline and the
collective progression towards harnessing computational power for transformative advancements in science and technology. Computational chemistry is the earliest among the three, with its roots tracing back to the mid-20th century. The development of computational chemistry gained momentum with the advent of digital computers, and it has evolved significantly over the decades. Computational chemistry involves the application of theoretical methods and simulations to study the structure, properties, and behaviour of molecules, making it a well-established and mature field. Computational biology, on the other hand, gained prominence later, particularly with the explosion of biological data in the post-genomic era. The field began to flourish in the late twentieth century and has since grown rapidly. Computational biology encompasses a broad range of techniques, including bioinformatics, molecular dynamics simulations, and systems biology, to model and analyse biological processes at various levels of complexity. Materials informatics is a more recent entrant compared to computational chemistry and biology. Although the idea of using informatics approaches for material discovery has been around for some time, the field has gained significant traction over the past few decades, especially with the rise of advanced computing capabilities and the availability of large material databases.

Materials Informatics (MI) is founded on two key techniques: Density Functional Theory (DFT) computations and a data-driven approach using Machine Learning (ML). DFT computations, a core aspect of quantum mechanics simulations, delve into a material's electronic structure and properties at the atomic and molecular levels, providing a precise understanding that complements experimental data. Conversely, the data-driven approach utilises ML algorithms, from regression models to neural networks, to analyse extensive material datasets. These algorithms decode patterns and correlations, enabling researchers to predict the properties of new materials or optimise existing ones. This combined approach, which integrates quantum-level insights with data-driven predictive modelling, accelerates the materials discovery process, facilitating the efficient exploration of diverse materials spaces.

Data-driven methods have significantly contributed to the design of new materials across various sub-domains, such as magneto-caloric, thermoelectrics, and superconductor materials. Despite the impactful role of these methods, a notable challenge lies in the limited availability of experimental datasets. Currently, the primary sources of such datasets in inorganic materials are Pauling File [5], Starrydata [6], and SuperCon [7]. The use of these databases highlights the need to broaden and diversify experimental data, improving the effectiveness and applicability of data-driven approaches in materials design. The augmentation of these datasets is crucial to advancing the field, promoting innovation, and expanding the scope of materials discovery through data-driven methodologies. SuperCon is the
standard superconductor materials database developed and manually maintained by
Japan's National Institute for Materials Science (NIMS) since 1987. It consists of
records of materials reported in scientific experiments and a large set of measured
properties.

A superconductor is a material that exhibits zero electrical resistance and ex-
pulses magnetic fields from its interior (Meissner effect) when cooled below a critical
temperature. This phenomenon, known as superconductivity, occurs in various ma-
terials, with many requiring extremely low temperatures, often close to absolute zero,
to manifest this unique behaviour. The critical temperature at which superconduc-
tivity emerges varies depending on the material. Superconductors have numerous
applications, notably in the field of electrical engineering. They are used to create
powerful electromagnets for applications such as magnetic resonance imaging (MRI)
machines, particle accelerators, and magnetic levitation systems. Superconductors
also play a crucial role in developing high-speed, energy-efficient power transmis-
sion lines since they can carry large currents without any loss. The potential for
quantum computing and more efficient electronic devices is another area where su-
perconductors are actively being explored, showcasing the interdisciplinary nature
of their applications in physics, materials science, and technology.

SuperCon hosts about 32,000 superconductor material records and a complex schema with about 200 properties. The data available in SuperCon is both sparse and heterogeneous. Some records include properties that are not directly relevant to superconductivity, while others lack important properties that have not been consistently gathered [8]. For example, the pressure applied to obtain superconductivity was reported in only 16 records; however, its studies and experiments date back to the 1970s, but it became more popular only at the beginning of the 21st century. Nevertheless, SuperCon has been widely recognised for its quality and has been the data source for many attempts to design models that can predict T<sub>c</sub> [9–11].

Using ML to predict T~c~ has some criticality that needs to be considered. The models have an intrinsic applicability domain, meaning predictions are limited to the patterns and trends encountered in the training set. This can lead to significant selection bias, making the models ineffective when applied to materials constituted by different compounds (or belonging to different classes). The variety of training data to be used should contain a relevant amount of non-superconductors materials to avoid training a model biased toward the assumption that a T~c~ always exists and renders it ineffective when applied to new materials. The definition of non-superconductivity is tricky: if no superconducting critical temperature has been found, it does not mean one does not exist. It might be outside the range of current technological instruments. This leads to a conundrum when delving into the data: ignoring compounds with no reported T~c~ and maybe disregarding a potentially im-
portant part of the dataset for the sake of simplicity. Simplifying too much could
lead to an incomplete understanding of the factors that determine superconductiv-
ity and limit the ability to predict T~c~ accurately. Generally, in addition to these
considerations, several SuperCon-related works attempted to integrate complemen-
tary data from other datasets. However, at the time of writing, only [8] has aimed
to enhance SuperCon. They successfully enhanced 9150 records with crystal struc-
tures from experimental results stored in the Inorganic Crystal Structure Database
(ICSD). This recent work indicates that the SuperCon dataset continues to play a
pivotal role in the landscape of superconductor research.

1.1 Motivation

The number of publications in superconductor research remained stable in the last
10 years<fnref n="1" />. It is reasonable to expect that future breakthroughs in superconductor
research will be achieved through materials informatics, given its growing impor-
tance, and for that to happen, a necessary condition will be to have an updated
SuperCon.

Two main challenges that SuperCon is facing motivate our work. The manual
approach employed for compiling SuperCon is proving to be increasingly demanding
in keeping up with the influx of new publications on time. The reliance on manual
labour for dataset curation becomes a growing obstacle, as it demands specialised
skills that are not readily available in the market. As the field of superconductor
research evolves, the need for more efficient and scalable methods becomes appar-
ent. An automatic process is necessary to streamline and improve experimental data
collection in SuperCon. The outline of the process comprises the ingestion of com-
plete documents or text and the identification of materials and relative properties.
In a small-scale preliminary assessment, we analysed the problem, identified several
challenges to overcome, and proposed a data extraction framework [12].

The expressions that identify materials are complex and lengthy, and their iden-
tification in the scientific literature poses several challenges. Furthermore, the strict
definition of "material" cannot be expressed in a synthetic sentence valid for the
whole discipline. There are differences between subdomains in materials science
that make this process highly dependent on domain experts. A material may be
expressed in a multitude of partially overlapping definitions: as a single material, a
single sample, a family, or a class. Alternative approaches to naming exist, includ-
ing the use of commercial names or sample designations, often arbitrarily chosen

<footnote n="1">¹analysed on arXiv statistics on cond-mat category https://info.arxiv.org/about/reports/
submission_category_by_year.html</footnote>
by researchers (Sample 1, Paris-sample, etc.). Chemical formulas can be presented as stoichiometric expressions within these three categories, with possibilities such as “Y-111” or “Ba(Fe1-xCox)2As2”. Examples of standard names include Oxygen or Magnesium Diboride, while some adopt abbreviated forms like Yttrium Barium Copper Oxide (YBCO). These varied strategies provide a range of options for nomenclature within the domains of chemistry and materials science, while the conventional method primarily involves the use of chemical formulas. A class of materials refers to a broader category that shares certain fundamental characteristics or properties. Materials in the same class might be characterised by common structural features, electronic configurations, or other common characteristics that make them suitable for investigation. A family usually implies a more specific grouping that shares a closer genetic or compositional relationship. A set of materials with similar chemical compositions, crystal structures, or electronic configurations. A sample is just an arbitrary name chosen by the researchers that can overlap any of the previous definitions.

To make matters more complicated, all these definitions are not strictly enforced and may fluctuate from one laboratory to another. Authors may talk about "high-T<sub>c</sub> cuprates", referring to materials in the class of cuprates that also have high-T<sub>c</sub>. This fallacious definition does not clearly describe which materials are included: "How high should T<sub>c</sub> be to be considered 'high'?". The various interpretations that words, terms, and symbols can have in different sub-domains can cause significant confusion. For instance, the abbreviation "TC" or "T<sub>c</sub>" can refer to either "Temperature Curie" or "superconducting critical temperature", depending on the specific context. However, authors may utilise "T<sub>C</sub>" (C uppercase) to indicate the critical temperature of superconductors. When extracting text, this convention is essentially the same as referring to "Temperature Curie." Resolving this ambiguity is only possible through careful examination of the context.

Manual curation remains a necessary component, but it does not guarantee the complete elimination of errors. [8] identified more than 10000 duplicate records within the SuperCon dataset, revealing instances where certain properties were inadequately populated. This underscores the inherent challenges and limitations associated with relying solely on manual efforts for data curation, emphasising the imperative to adopt more robust and automated strategies to enhance the accuracy and comprehensiveness of the SuperCon dataset. There is a need for a curation-centric user interface and tools that allow the reduction of the manual bottleneck to a few minor actions, leaving the tedious tasks of identifying main data in scientific publications to automatic processes.
## 1.2 Problem definition

NIMS encounters various difficulties updating SuperCon to incorporate the latest research findings due to the growing number of publications every year and the scarcity of a highly qualified workforce. Employing high-skilled and educated individuals who have to work on a task that offers minimal rewards requires a large monetary commitment. To expedite the extraction of information from scientific papers, it is necessary to develop an automated process. Collaborating closely with domain experts in the field is essential to address and overcome the potential pitfalls and challenges that may arise, as they can offer validation and valuable guidance. Despite the potential challenges arising from different work methodologies, this collaboration allows sharing of best practices between people with different backgrounds, leading to a comprehensive and unified solution.

## 1.3 Contributions

In the following section, we discuss the various contributions of this dissertation, highlighting their main relevancy. We created a novel flow compared to other works that support PDF documents as input data sources. This method enables us to establish a unified interface for all domains and publishers and the capability to retrieve the full-text body (Section 1.3.1). Moreover, the scientific community widely adopts the PDF format as the "de facto" standard for publishing research papers [13]. We rely on ML for its resistance to noise and ability to generalise well on unseen examples. Nevertheless, due to the scarcity of high-quality training data (Section 1.3.2), we constructed SuperMat (Section 1.3.2). SuperMat is a novel dataset of 164 articles containing annotations and relations between entities. SuperMat is a valuable addition to the field of MI as it helps address the limited availability of structured data sources. In the realm of materials science, the information extracted is modelled as a triplet that contains material, properties, and conditions. Both properties and conditions can be expressed as measurements of physical quantities that are relatively uniform in most disciplines. However, while the conditions are constant between domains, the properties are highly domain-dependent. As part of our contributions, we co-developed with Patrice Lopez a separate module related to Grobid based on ML to extract measurements of physical quantities (Section 1.3.3). We then combined all of these techniques to perform a large-scale extraction from documents obtained by the ArXiv repository. The automatically extracted database, SuperCon² (Section 1.3.4), is the main contribution of our work, presenting a novel set of properties: "applied pressure" and "measurement methods". Domain experts consider these properties promising complementary information to breakthroughs
in discovering new superconductor materials. The database obtained addresses a
critical requirement of NIMS to streamline the extraction of experimental data from
scientific documents. Finally, to improve the quality and reduce the impact of man-
ual curation, we developed a user interface that supports domain experts in data
curation with a set of advanced functionalities, which we demonstrate substantially
improve the quality of the curation output.

1.3.1 Extraction from the full-text body of PDF documents

Unlike other works [14–16] that were predominantly based on web scraping or
through the publisher API, we designed a novel data flow for the extraction of
material-related information from PDF documents using an existing open-source li-
brary, Grobid (Generation of bibliographic data) [17]. While web scraping is rarely
allowed (it breaks the usage agreement) or is limited to public metadata informa-
tion (e.g., abstract, authors, title), API access is often hindered by the necessity
of agreements with publishers, making the gathered content difficult to share and
challenging to replicate. In both cases, each publisher carries its own limitations and
data formats and must be treated separately: mining data directly from PDF docu-
ments enables us to emancipate ourselves from the data source type. The increasing
abundance of open access literature [18] provides means to access large-scale sci-
entific literature (e.g., ArXiv (www.arxiv.org), ChemXiv (https://chemrxiv.org/),
NIMS Material Data Repository (MDR) [19] (https://mdr.nims.go.jp)). We can
then access all information from the scientific documents, including full-text body,
tables, and figures. Our work focuses only on text, leaving figures and tables to
separate projects. Accessing the body differs us from other approaches [20], which
are limited to short and synthetic abstracts. The body contains more information
that includes experimental results in related works. However, it is acknowledged
that abstracts, being synthetic and possessing a simpler structure, also play a role
in data extraction. This contribution is discussed in detail in Chapter 3.

1.3.2 ML-based extraction of materials-related expressions from
text

The scope of this work is to create databases of experimental data that are reported
in the scientific literature. The structure representing this information can be com-
plex and strongly depends on each subdomain’s characteristics and conventions.
However, we can define the basic data model for experimental data as consisting
of three main constructs: material expressions, properties, and conditions. Mate-
rial expressions are a relatively loose concept that strongly depends on the domain.
On the other hand, properties and conditions can be decomposed into expressions of
measurements of physical units. This contribution comprises three main components
that are discussed in Sections 1.3.2, 1.3.3, and 1.3.2.

Identification of complex material sequences

The field of materials science encompasses various disciplines, including chemistry, physics, and engineering. Material expressions within this field are typically lengthy and intricate sequences of characters. Materials can be represented by chemical formulas, which may include additional information such as the material’s shape (e.g., crystal, single crystal, wire) and doping details (e.g., 2% Zn-doped). The identification and extraction of these entities is extremely challenging. We use machine learning (ML) techniques to implement the named entity recognition (NER) system. This approach offers several advantages, such as context awareness, robustness to noise, and better generalisation to unseen examples compared to rule-based methods. Additionally, we discuss the evaluation of three different architectures: Conditional Random Fields (CRF) [21], Recurrent Neural Networks (RNN) based on Bidirectional chains of LSTM units [22], and Bidirectional Encoder Representation for Transformers (BERT) [23] models.

The novelty of our approach can be summarised in four main components. First, the need for high-quality training data and the lack of resources in material informatics was raised to construct a new dataset called SuperMat (Section 1.3.2) that contains annotated documents from superconductor research. This contribution is discussed in detail in Chapter 5. Second, we wanted to reduce the risk of entities with low probability being overlooked. Our ML training strategy was designed to build models that are geared toward recall rather than precision (Section 4.2). Third, we train and evaluate three different architectures and select the best-performing combination for each model. This flexibility is convenient since each architecture has its strengths and weaknesses. Finally, we needed to extract a large set of entity types; therefore, we devised a two-layer approach where the second layer was specialised to segment only the material entities (Section 1.3.2). The original text is first processed, and the main entities are extracted: materials and classes, properties ($T_c$), and parametric conditions (applied pressure, measurement methods). Then, the material entities are further processed by a specialised material parser, where different modifiers such as doping, shape, substrate, chemical formula, and name are identified. Additionally, we incorporate the use of characteristics consisting of text patterns (such as uppercase and lowercase) and layout details (such as superscript, subscript, bold, and italic) that are exclusively extracted from the PDF structure.
Construction of SuperMat: an annotated and linked dataset of superconductors research papers

One major limitation of the available MI resources is the scarcity of datasets that combine materials, conditions, properties, and their relations. At present, there are only a limited number of datasets that provide a high-quality unified approach for machine learning models, taking into account the complex and inconsistent terminology common in superconductor research. To address this limitation, we constructed SuperMat, which also serves as the basis for defining a methodology which involves iterative collection and correction of data as a collaborative effort with domain experts. This work contributes to this effort by expanding the dataset for MI. SuperMat comprises, at the time of writing, 164 articles from superconductor research annotated with 2-layers annotations: six types of top-level entities related by three types of relations between entities, and a secondary layer where identified materials are further decomposed in granular element (name, formula, doping, etc.). This dataset has already been used for practical applications such as ML training [24], Large Language Models (LLM) evaluation [25], and weighted clustering [26]. Chapter 5 is centred on the methodology for creating the dataset due to its extensive and intricate nature, which necessitated a detailed and comprehensive explanation.

Parsing and normalising materials sequences

The need to standardise material expressions arises because materials can contain
a combination of different types of information, such as chemical formulas, doping,
substrate, etc. For example, the expression “La x Fe 1-x O 7 (x=0, 0.1)” represents
a formula with substitutions, where x can take values of 0 or 0.1. Other examples
are the expressions “La-doped Fe O7” and “2% La-doped Fe O7”, which combine
the material with the presence or the amount of doping. It should be noted that
sometimes only the doping ratio is given, as in the case of “x = 0.1”. Related
tools such as text2chem [16] and PyMatGen [27] exist; however, they have a low
tolerance to noise that limits them to clean formulas. For example, these tools
would not correctly parse a non-stoichiometric formula such as “La x Fe 1-x O 7
(x=0, 0.1)”. We built a comprehensive ”Material parser” that uses an ML model
to identify specific information such as formula, name, and doping. We combine
the sequence with text2chem and PyMatGen, as mentioned above. As a result,
our parser is tolerant of noisy input and can parse and convert intricate and long
material expressions to a clean, structured form. This contribution is discussed in
Chapter 4.
### 1.3.3 Extraction of properties and conditions as measurements of physical quantities

Properties and conditions within the realm of science and engineering are often expressed through measurements of physical quantities. These measurements serve as a universal language, providing a standardised way of conveying information about various aspects of the physical world. Whether it is the length of an object, the temperature of a substance, or the intensity of a force, these measurements encapsulate essential characteristics that form the basis for scientific understanding and technological advancements. Measurement of physical quantities forms the backbone of scientific inquiry, fostering a collective language that transcends boundaries and enabling collaborative efforts in the pursuit of knowledge and innovation. Employing a different array of tools for material-related expression identification and property/conditions analysis enhances adaptability towards new requirements or different sub-domains. Although measurements may seem like consolidated information, their handling can be difficult. This is because measurement units are often used in different ways across various disciplines. Additionally, different systems of measurement units, such as using miles instead of meters or ATM instead of Pascal, may be employed for human comprehension but can cause confusion for machines. A well-known example highlighting this issue is the explosion of the Mars Climate Orbiter, which occurred due to unit conversion errors. In this contribution, we present Grobid-quantities, a Grobid module specialised in the identification, extraction, and standardisation of physical quantities and measurement. At the time of development, Grobid-quantities was the first ML-based open-source project aiming to cover multiple disciplines (from biology to materials science) and systems of units (SI base, SI extended, imperial, etc.). This contribution is discussed in detail in Chapter 6.

### 1.3.4 Large scale collection of experimental data from scientific literature: SuperCon² Database

The contributions discussed above are combined in a comprehensive extraction pipeline called Grobid-superconductors. We demonstrated a significant advance in data extraction efficiency and scalability by creating the SuperCon² Database, collecting 40324 records in a few days, indicating a substantial increase in the amount of data processed. The original SuperCon database, cultivated for two decades, contained approximately 33000 elements. SuperCon² Database serves as an important intermediate step in collecting experimental data related to superconductors, bridging the gap between the original SuperCon database and the new data. The tool's effectiveness is evident not only in its ability to match the scope of the orig-
final database but also in its ability to surpass it by capturing detailed information
such as applied pressure and measurement methods. Once curated and validated,
the data within SuperCon² can seamlessly enhance the SuperCon database, leading
to continuous improvements in the accessibility and depth of the stored scientific
information. This contribution is detailed in Chapter 7.

1.3.5 Reducing the impact of manual curation

When using automatically collected data, training machine learning models or other
data-driven processes must be done carefully. Automated processes pose significant
risks due to the inherent potential for inaccuracies and errors. We have designed
a robust solution to address this need that features a staging area ("SuperCon²
Database") that is accessed through a curation workflow with a user-friendly inter-
face. The interface offers functions to control the content of the database records,
triggering the state transition to the underlying workflow. Domain experts required
that the underlying data be organised so that any update is incrementally stored
and the history of changes in each record is accessible. Our system provides an
enhanced visualisation of the original PDF document, allowing users to inspect and
cross-reference the data efficiently. Furthermore, our solution incorporates ML capa-
bilities to improve the automated data collection process continuously. The system
accumulates training data based on corrections made during the validation process,
assuring relevant refinement of the underlying ML models. In this work, we exper-
imented and demonstrated that our workflow and interface can increase the recall
of curated information from 45% to 92% and the F1 score from 52% to 92%. The
experiments and the description of the curation interface are listed in Chapter 8.
# Chapter 2

# Related work

This chapter is divided into different contributions and aspects of this research. The chapter is organised from specific (text mining in materials science) to generic (NER, machine learning) works.

## 2.1 Text and data mining in materials science

[14] propose a method to extract Curie and Néel temperatures from scientific papers. The authors initially considered using the original ChemDataExtractor for this task. Nevertheless, they observed that the rule-based approach reduced precision and recall due to the complexity of the scientific text. As a solution, they enhanced ChemDataExtractor by incorporating a semi-supervised relationship extraction algorithm. This algorithm learns a typical pattern using a modified version of the snowball library. Typical patterns are stored and used to cluster new extracted patterns by distance similarity. The authors claim they extracted 39822 records, consisting of 11340 Néel and 28482 Curie temperature records.

[15] propose another work where they process 74000 web-scraped scientific journal articles and build a database of 20,400 magnetic and superconducting phase transition temperature records and their associated chemical compound names. The NLP pipeline operates as follows: text and tables from target journals were analysed to identify material formulas and temperatures. Subsequently, the identified entities were standardised, temperatures converted to Kelvin, and specific rules resolved doping issues.

SC-CoMIcs [20] describes a new corpus called SC-CoMIcs (SuperConductivity
Corpus for Materials Informatics) tailored for the text mining of superconducting materials. The corpus consists of 1000 manually annotated abstracts related to superconductivity with seven named entity categories—characterisation, process, property, material, element, doping, and value. The IAA (Inter Annotator Agreement) was around 75-85%, similar to other materials science corpora. Experiments using SciBERT for recognising named entities achieved a 77% F1 score comparable to human agreement. The learning curves indicate that the corpus size is sufficient, although some categories could benefit from more data. The corpus was used to build a word search tool based on word vectors, demonstrating its utility for retrieving relevant terms by category. This paper shows the potential of text mining to extract critical information from the superconductivity literature. The corpus provides a valuable resource for developing more capable natural language processing systems for material informatics.

After our work, [28] described a similar pipeline to extract superconductivity information from abstracts from the scientific literature. It uses the SC-CoMIcs [20] corpus of 1000 annotated abstracts on superconductivity to train NER and RE models. The author reports that they extracted material compositions, transition temperatures, doping information, and process details from 48,565 abstracts obtained by querying the Elsevier Scopus API (Application Programming Interface) for a total of more than 43,000 superconducting materials and 24,000 transition temperatures. They follow a similar approach to our work; first, they extract entities and find the relations. However, the abstract limits their extraction density, which has only synthetic information. Considering their search query "supercond * AND tc OR transition temp", they obtain around one superconductor material per abstract.

There are several other works on TDM on experimental information from materials science; the most relevant are discussed in the following paragraphs.

[16] describes a text mining system to extract and segment synthesis recipes from scientific articles. The articles are obtained through web scraping, and the synthesis paragraphs are identified through classification. Subsequently, NER is implemented using RNN (specifically, BidLSTM + CRF) in the identified paragraph. This helps to identify the various components of the recipe, and the precursor, operation, and synthesis conditions are extracted for a total of 19,488 solid-state synthesis reactions. This work was limited to articles after 2000 due to the lack of available HTML/HTM versions for older articles.

[29] proposes a text mining pipeline to extract metal-organic frameworks from scientific papers. In this paper, the authors focus on two main properties, surface area (SA) and pore volume (PV), mainly because they are the main properties related to the absorption properties of MOPs, are very commonly described in papers,
and have a very distinctive signature such as the units used to measure the quantity. The source data was obtained in HTML format, and after parsing, cleaning, and tokenising the text, the tokens were classified into five types using lexicons from the Cambridge structural databases. Moreover, they employ specific terms to recognise appropriate units used unconventionally. Using a rule-based algorithm, the resulting classified tokens were combined (the material was connected to the corresponding attributes). On a sample dataset, the algorithm achieves 90% and 88.8% precision in extracting surface area and pore volume data, respectively. When tested on a larger dataset of 2315 MOF papers, the precision drops to 73.2% for surface area and 85.1% for pore volume. Errors occur due to complex sentence structures and incorrect identification of MOF names.

[30] introduces a nanoparticle-related dataset of 5,154 records extracted from 4.9 million materials science papers containing synthesis recipes and morphological information. The authors successfully extracted 7,608 experimental and 12,519 characterisation paragraphs with compounds, amounts, synthesis actions, sizes, shapes, etc. Limitations include the inability to distinguish targets from other morphologies and the lack of order information for seed-mediated syntheses.

[31] develop an automated approach to extract band-gap values and relate them to chemical compounds in the titles and abstracts of articles. The authors extended ChemDataExtractor with a new BandGapParser to identify band-gap information. On a sample of 415 papers, the system achieved 51.32% correct, 36.62% partially, and 12.04% incorrect extractions. Errors are due to incorrect extraction of chemical entities, band-gap values, and referring entities to values. The approach was applied to 11,939 articles, extracting 10,608 band gap values for 10,292 compounds. The limitation of this study is the evaluation that was performed by reviewing the extracted information rather than comparing it with the ground truth.

[32] presents ChemNLP, a natural language processing (NLP) library and web application for analysing material chemistry text data. It uses the publicly available arXiv dataset of 1.8 million scholarly articles collected over 34 years. The authors analyse publication trends, author names, taxonomy categories, and word frequencies in titles and abstracts. An interactive web application was built to search for articles containing specific chemical elements and compounds. As a demonstration, ChemNLP was applied to identify new superconducting materials by comparing the arXiv data set with a DFT-based superconductor database. Machine learning techniques such as TF-IDF vectors, t-SNE clustering, and classification algorithms were used to categorise "cond-mat" arXiv articles with 80% accuracy. ChemNLP provides an open dataset and tools to apply NLP techniques to the materials science literature for knowledge discovery.
## 2.2 Machine Learning

Machine learning (ML) can extract valuable implicit knowledge from data [33], providing fresh insights into existing problems. The application of machine learning (ML) in intricate fields like chemistry requires the use of reliable and accurate data. However, it is essential to note that the results obtained from ML models may not always be attributed solely to the domain knowledge incorporated, as there could be other concealed factors within the data itself [33].

Conditional Random Fields (CRF) [21] have been widely used in various ML tasks such as sequence labelling and NER [34]. CRFs have been particularly effective in situations where multiple annotator-embedded label sequences are available, but there is no actual ground truth, as they provide a probabilistic approach to sequence labelling [34]. Furthermore, CRFs have been compared with other machine learning models [35] such as Support Vector Machines (SVMs) [36] and Structured Support Vector Machines (SSVMs) in tasks such as chemical entity recognition, demonstrating their versatility and effectiveness [35, 37, 38].

The transition from CRFs to RNNs represents a shift from probabilistic graphical models to deep learning architectures. While CRFs effectively capture dependencies between input features and make predictions based on them, RNNs excel in capturing temporal dependencies in sequential data due to their recurrent nature. This transition is particularly relevant in tasks where the sequential nature of the data is crucial, such as speech recognition and acoustic event detection [39, 40]. The use of recurrent neural networks (RNNs) has gained importance thanks to Long Short-Term Memory (LSTM) [41] and Gated Recurrent Units (GRU) [42] for text processing tasks: classification and recognition. [43–46] highlight the effectiveness of RNNs in processing sequential data, showcasing their superiority over traditional methods in natural language processing tasks. Furthermore, the combination of RNNs, particularly bidirectional LSTM recurrent neural networks (BidLSTM), with CRFs has been explored for tasks such as word segmentation and morpheme segmentation, highlighting the adaptability of RNNs in sequence labelling [47].

Particularly relevant is the work [22], which proposes a new neural architecture for NER with an architecture based on a bidirectional set of LSTM chains. Bi-directional indicates that the text is computed from left to right (forward LSTM) and from right to left (backward LSTM). Such an architecture can incorporate information usually captured by hand-crafted features or gazetteers. Such architecture takes in input two types of word representation: character-based word representation, calculated from the supervised corpus, and unsupervised word representation, learnt from unsupervised corpora. The unsupervised word representations discussed were word2vec [48], but they can also be used for more recent ones: fastText [49],
GloVe [50].

[51] proposes a novel deep contextualised word representation that addresses two key aspects: a) capturing complex characteristics of word usage and b) incorporating the polysemy of words within linguistic contexts. The proposed approach calculates word representations based on the entire sequence. The network architecture consists of two bidirectional LSTM layers with dimensions 4096 x 512. These layers work in a complementary manner, with one predicting the next token from left to right based on the preceding tokens and the other predicting from right to left. The study also highlights several significant findings: a) instead of solely outputting the last layer, averaging the weights from all layers with a coefficient improves performance in downstream tasks, b) lower layers capture syntactic information while higher layers capture context-dependent aspects (semantic), and c) incorporating ELMo embeddings in both the input and output enhance tasks that utilise the attention layer after the RNN, but does not provide benefits for other task types.

The transition from LSTM-based RNN to transformers marks a significant change in natural language processing. The introduction of Bidirectional Encoder Representations from Transformers (BERT) [23] revolutionised language representation models by leveraging the power of transformers. Transformers, as proposed by [23], replaced RNNs with self-attention, demonstrating the potential of this approach in language processing tasks [52]. BERT, based on bidirectional transformers, offers a simple yet robust architecture that allows efficient pre-training and fine-tuning with minimal task-specific modifications, leading to state-of-the-art performance across a wide range of language processing tasks such as question answering, language inference, and named entity recognition [23, 53]. The bidirectional nature of BERT enables it to capture contextual information from both preceding and subsequent words, which is crucial to understanding the meaning of a word in a given sentence, thus outperforming traditional LSTM-based models [54, 55]. Furthermore, BERT’s ability to handle long-range dependencies and its superior performance in sequence labelling tasks, such as NER and RE, has been demonstrated in various studies [56–58]. Furthermore, the effectiveness of BERT in sentiment analysis, misinformation check and text classification tasks has been highlighted, showcasing its versatility and robustness in different NLP applications [59–61]. However, it is essential to note that while BERT has shown superiority in various language processing tasks, there are instances where CNN and LSTM models also perform competitively with BERT-based models [62, 63]. In general, the advantages of BERT-based architectures lie in their conceptual simplicity, empirical power, contextual understanding, and superior performance in diverse NLP tasks, making them a crucial advance in NLP.

In the following sections, we first discuss all essential BERT-related work and
then the BERT-flavoured transformers related to materials science.

## 2.3 Transformers

[23] introduce BERT (Bidirectional Encoder Representation Transformer). While pre-trained language models have been discussed in previous work [64] and can be exploited in two main ways: feature-based or fine-tuning. BERT aims to improve the fine-tuning approach and to solve the limitation of existing left-to-right language models, where each token can attend only to previous tokens in the self-attention layers [52] of the Transformers architecture. BERT introduces an MLM (Masked Language Model) pre-training objective, where it randomly masks tokens in training examples and tries to predict them using the full context. This enables using both left and right contexts and pre-trains a fully Bidirectional Transformer. Using the original transformer architecture [52], they define BERT<sub>BASE</sub> with L=12, H=768 and A=12 (110M parameters) and BERT<sub>LARGE</sub> as L=24, H=1024 and A=16 (340M parameters) with (L=layers, H=hidden layers, and A=attention heads). The authors define the input as a sequence that can be a natural sentence, a paragraph, or a pair of sentences. They used a special separator [CLS] to start the sequence and convey the result of the final hidden state. For separating two sentences from a pair, they encode the separator with [SEP] and provide sentence embedding, which states whether a token belongs to sentence A or B. Each token is encoded using position embeddings calculated using WordPiece using a dictionary of 30000 tokens vocabulary. The sum of the vocabulary token, the sentence token, and the position embeddings represent the input representation for each token. The authors defined two training objectives: Masked LM (MLM) and Next-Sentence Prediction (NSP). The MLM is prepared by substituting 15% of the token's positions at random: 80% of the time with [MASK], 10% of the time with a random token, and 10% of the time do not substitute. The reason is to provide the model with enough randomness so that the model does not always learn to replace the tokens with [MASK]. The NSP aims to identify whether a sentence follows another sentence. This is justified by the need to perform tasks such as answering questions in which relations between sentences are determined. BERT outperforms the present architectures on most NLP tasks and benchmarks such as GLUE, MNLI and SQUAD.

After BERT was released, a multitude of flavoured models were released. We analyse the most important ones in the following paragraphs.

[65] reproduces the results obtained in BERT, revises the original architecture and proposes RoBERTa (Robust Optimised BERT Architecture). This revised BERT implementation outperforms BERT on downstream tasks such as GLUE,
RACE, and SQUAD benchmarks. The main modifications are summarised as follows. Instead of masking tokens statically only once during preprocessing when the data are duplicated ten times, they apply different masking at each instance of the same example. Dynamic masking helps for less than 1 point percentage. They suggest that changing how the input data is prepared and removing the NSP loss objective improves the performance. In particular, providing complete sentences from the same or different documents without NSP matches the original implementation of providing a pair of segments with NSP. Furthermore, results are even better when only sentences from the same documents are provided, and the NSP is removed. NSP removal was discussed in recent work [66]. Lastly, they propose training for a longer time, and supplying more data can improve the results: the original BERT was trained on 1M steps with a batch size of 250, while the RoBERTa was trained for fewer steps, 31000 using a much larger batch size of 8000 examples. This setting requires a more scalable approach because the required GPU memory is more significant. With these changes, RoBERTa can outperform BERT on most language understanding benchmarks such as SQUAD, GLUE, and RACE.

[67] propose a scientific version of BERT called SciBERT. SciBERT was trained on a random sample of 1.4M articles from Semantic Scholar (data set was not published): 18% from computer science and 82% from the biomedical domain. SciBERT was evaluated against BERT and outperformed BERT. Unlike BioBERT [54], SciBERT was trained using a different tokeniser (SentencePiece instead of WordPiece). Most importantly, the tokeniser was re-trained from scratch using scientific text. Another exciting aspect is that BioBERT was trained from a model initialised with the weight from BERT, trained for a longer time, and on more data overall. However, the limited coverage of the BERT tokeniser on scientific data penalised the performance on various downstream tasks.

[68] introduces a new approach in pre-training a BERT model in two flavours: LinkERT and BioLinkBERT on the general text and biomedical text, respectively. The authors introduce a novel approach which includes text from linked documents, and an additional objective function is to classify the type of link. The pre-training data is aggregated as follows: a) Each example is structured as a pair of text sequences as in the original BERT. b) The pairs are aggregated by randomly selecting the second segment (when available) from the same or a related document. For the LinkBERT, this only works for the Wikipedia corpus, exploiting the links between Wikipedia pages. For BioLinkBERT, the authors select sequences from related documents in the citation graph. The authors replace the NSP objective with the DRP (Document Relation Prediction), which aims to classify the relation type between the two segments, giving three possible classes: contiguous, random, and linked. Evaluation metrics calculated on QA (extractive question answering), GLUE and SQUAD outperform BERT. BioLinkBERT outperforms PubMedBERT [69] on most
NLP tasks.

[69] pre-trained a biomedical-based model called PubMedBERT and evaluated it using a newly assembled dataset called BLURB (Biomedical Language Understanding & Reasoning Benchmark). The vocabulary was trained using a BPE (Byte-Pair Encoding) approach with a length of 30522 and trained on text from PubMed abstracts: 14 million abstracts, 3.2b words. The fact that they only used abstracts could be a significant limitation on the type of information used in the pre-training. They are the first to consider the concepts of "in-domain" (in their case, biomedical text) and "out-of-domain" (everything else). Their strong claim is that domain-specific pre-training from scratch can be superior to mixed-domain pre-training, which contrasts with [70]. From the perspective of vocabulary, it is demonstrated by comparing scores from BioBERT [54] and SciBERT [67] that vocabulary specificity improves performance in tasks applied to scientific text.

[71] tested biomedical-trained BERT models on relation extraction in biomedical data. They demonstrate that downstream tasks on overlapping data used in pre-training bias the results for RE. In classification, they indicate that using only the information in the "CLS" token can be improved by adding complementary information from other layers.

[70] demonstrate that training with a multidisciplinary corpus gives better results than a transformer based on a domain-specific dataset. In addition, they also found that larger models do not always perform better. However, these results might be due to other issues they have introduced with their pre-training process. Previous work by [66] has suggested a similar concept applied to translation tasks, where including cross-training information and, in particular, the addition of data in different languages not only boosts the performance in translation but also helps the performance in non-translation tasks.

## 2.4 Materials science BERT-based pre-trained transformers

This section analyses the pre-trained BERT-based transformers using materials science text. We provide a comparison evaluation of all of these transformers as a comparative measurement.

[72] propose a material-property extraction for polymers based on a fine-tuned BERT model on 2.4 million abstracts. Their NER system achieves higher results than other material-based pre-trained BERT models. Properties are connected to
materials using a heuristic approach to find the closer entities. With their system, they extracted more than 300000 records of polymers and properties from the 2.4 million abstracts. However, the article does not provide information on data contamination between pre-training and evaluation: we do not know whether there is any overlap between the evaluation datasets, PolymerAbstract [73] and the training set of 768 abstracts. Moreover, whether their model can generalise properly is not indicated, providing out-of-domain information of their evaluation dataset compared with the training set.

[74] is another specialised set of specialised BERT models that focus on scientific articles related to batteries. In this work, the authors experiment with different approaches for fine-tuning, continuing a generic BERT training (batteryBERT), a SciBERT training (batterySciBERT), or training from scratch a new model based only on text from battery research (batteryOnlyBERT). The different vocabularies provide the main differences, where batteryBERT and batterySciBERT are constrained by the original vocabulary. The batteryOnlyBERT vocabulary was trained from scratch and could model the terminology used in the battery research articles more accurately. In document classification, batterySciBERT obtained the highest score. SQuad evaluation (extractive Q&A dataset) surprisingly showed better scores with batteryBERT. This work demonstrated the importance of having a solid base with general text and scientific information to provide a model that can generalise and adapt to multiple situations where it could be used.

[75] proposes a two-stage deep learning framework for extracting chemical formulas and applying a "Role labelling" in the chemical reaction description. Following a standard schema Extract-Link, they first extract all references to chemical compounds and, subsequently, label each compound with its role in the reaction description. They introduce a chemical-focussing pre-trained BERT flavour followed by a task-adaptive encoder (ChemRxnBERT) that provides a role labelling method. The authors created a small dataset for fine-tuning chemical reaction annotations composed of relevant paragraphs containing chemical reactions extracted from articles in various chemistry journals. Role labelling was implemented in cascade by providing the primary material between special tokens ([P], [P]), which are then linked to the other extracted entities. Since the sequence is shallow, it can link only one material with the rest of the reaction. Therefore, running the process numerous times is needed if multiple products are present in the reaction paragraph.
## 2.5 Extraction of quantified properties and measurements

NER of physical measurement has a fundamental application in scientific texts and is relevant in other disciplines, including humanistic or social science.

Attempts have been made to extract measurements from text using many approaches. At the time of this contribution, we identified the following related work.

[76] built a tool for the extraction of quantities based on Apache UIMA (Unstructured Information Management Architecture) in combination with pattern matching that circles a Finite-State Automata (FSA). The authors claim that UIMA and FSA are faster and better suited to process a large quantity of text. Their work supports multiple constructs: values, intervals, and enumerations. Units are normalised to the International System of Standard (SI). Units are loaded from a configuration file that includes the normalised format. In the same work, they also combine keyword extraction and text segmentation, which are exciting challenges in the analysis of patent text. The authors used Quantalyze<fnref n="1" />, a commercial tool designed to process patents, for comparison. They reported that Quantlyze had limited unit support.

[77] propose a search engine called MQSearch. Their work proposes a rule-based extractor for quantities and units that include the measured object. The extractor models each measurement using the five-tuple (sign, number, error, scientific notation, units), where only numbers and units are mandatory. The data flow comprises four phases: preprocessing to mitigate noisy and incorrect character extraction. Then, to recognise units, they expanded an ontology of units from the OBO Foundry with additional information from external sources and defined one associated rule for each unit. The ontology is exploited to recognise the measured object. Quantities are extracted similarly with a set of regular expressions. The last phase is post-processing, which is used to discard possible wrong or invalid values. They also present a SoLR-based search engine that allows for a search using the extracted units, values, and measured objects.

[78] built an extractor for patents using GATE (General Architecture for Text Engineering). Although GATE provides plugins for machine learning, they implemented their extraction using a lookup to a gazetteer and a database containing the transformation rules between one unit and another. The gazetteer was built from a set of units published by the GNU (Gnu's not Unix) Foundation and comprises 30000 units. In addition, they also recognise references within the text and patent information (e.g. patent number, country code). They built the annotation rules

<footnote n="1">¹https://www.quantalyze.com/</footnote>
using the JAPE language, which is structured as a matching rule and action to
perform.

Although most works are based on English, some contributions focus on specific
languages: [79] investigated issues applied to Russian-derived languages (Russian and
Belarusian). The authors make a consistent analysis of the challenges of Russian-
derived languages. They proposed a solution based on finite-state automation with
350 graphs running on top of the NooJ<fnref n="2" /> linguistic processor. Grammar covers three
of the six grammar constructs (genitive, accusative, and nominative). The evaluation
was carried out on a mixed corpus of 100,000 words and resulted in an F1 score of
82%.

[80] combine the use of an Ontological and Terminological Resource (OTR) with the use of ML. The OTR is used as a reference and can be updated with new units that are not extracted correctly or entirely. They aim to use a modified string-matching function to extract measurements and units. Thus, the main problem they try to solve is to reduce the search space using string-matching functions. They proposed a two-step process. First, they use a classifier to determine whether a sentence contains a unit. The classifier is built on a bag of words where the words are counted using three methods. TF, TF/IDF and BM25. The model is then built with three different algorithms, which are then compared: Naive Bayes [81], decision three (J48), support vector machines (SVM) and Discriminative Multinominal Naive Bayes (DMNB). In the second step, they match potential candidates in the OTR and use the string function to select the item that is more likely to represent the extracted one. Based on the similarity, they defined two thresholds that they can consider as variants of an existing or new unit to enrich the OTR and improve future recognition.

[82] describe another rule-based solution in which they outperform the respective ML-based system. They aim to extract targeted information from laboratory test results from diagnostic devices examined by the US Food and Drug Administration (FDA). The authors developed a symbolic information extraction (SIE) system for extracting four types of entities: analytes (substance considered), specimens (where the analyte is measured), units of measure of the analyte, and detection limits of the diagnostic device in the exam. The SIE is based on a combination of rules and dictionaries. First, the candidates are extracted and then ranked to select the most plausible ones from the set. The units of measures targeted in this work are only a subset of all units and are specific to this particular subdomain in biology. They evaluated their SIE against three probabilistic learning approaches: CRF, SVM, and HMM. SIE outperforms ML-based models, except for the unit of measures where the CRF obtained the best scores.

<footnote n="2">²https://nooj.univ-fcomte.fr/</footnote>
More related to materials science [83] proposes integrating relevant units of measurement when composing a small data set (392 sentences) to develop nanocrystal devices. The data set focuses on identifying several properties, including the property value and common units in the domain. x This work also leverages ML using a set of CRF engines, focussing only on a few entity types applied in a cascade.

[84] describes an integrated measurement extraction that focuses on the scientific literature in earth science. They developed a novel method that extracts context-related information from measurements. Instead of creating their measurement extraction tool, they evaluated three reuse options. Quantalyze showed low recall in both the extracted measurement and the measured object and had technical limitations in integration, such as a lack of REST API. [78], discussed previously, was discarded due to the need to maintain the rules. Therefore, they chose our tool, Grobid-quantities, which had the advantage of requiring only labelled data and providing acceptable precision, recall, and f-score results.

[85] extracts numeric laboratory test expressions from clinical trial eligibility criteria texts. They encode information using the TimeML specification language TimeML. The system presented Valx extracts: a) numeric values and units using regular expressions, b) using a hybrid approach with a knowledge base, they extract variables referred to as quantities. They cover different text representations (e.g., BMI, body mass index, etc.). Valx supports the association of multiple values to the same variable. For example, "40 and 60 years" associate both values with "years". Their tool supports normalisation to convert conventional units to international units.

[86] describe an approach to formalisation of quantities and measurements that aims at extracting information from free text and inferring complex reasoning that includes the measured entities and their quantity. The authors introduce QVR (Quantity Value Representation) with three constituents: Value (which includes values and ranges), units (which characterise the value), and change (which indicates whether there is a modification in the value, e.g. increases). The quantities are extracted using a Semi-CRF and a bank of classifiers. [86] uses four types of features that are calculated for each token (and a window including the three previous and following tokens): a) classification using a lexicon to identify whether it appears as a number, unit, etc., b) determining whether it contains a digit, all digits, etc., c) POS (part of Speech). Units are extracted by assuming they are adjoined to the numeric values. This work reserves a detailed discussion on the entailment of quantities. Given a quantity value and a text, it is a 3-way decision problem: a) "entail" when the quantity value is supported in the context, b) "contradicts" when the context, instead, does not support the quantity-value, the Q-V is different, c) no relation. The Quantity entailment aims to clarify whether the context confirms units and if the way
they are described can be equivalent (e.g. two couples, four people). The quantity
entailment is relevant for scientific text comprehension, especially for explaining
experimental results concisely and containing multiple relative comparisons within
the same paragraph, making it hard to understand.

The works of [87] and [88] focused on tables.

[88] describes a comprehensive query system that uses previously extracted quantities, units, and context to answer questions containing measurement conditions. Their data model is described as triples of the form (entity, quantity, context), where the context gives more information and proof to quantity and entity. The triplet is computed offline in several steps: first, the quantity is extracted based on [86], which we have described previously. Entities and quantities are referenced using the Yago knowledge graph. Different columns are related to their previous work [89], and the extracted data is used to create an index with conceptualised quantities. This idea is particularly relevant to our Grobid-quantity work because very few works currently use normalised quantities to search by values and intervals.

[87] assumes that the table columns are already identified and focus on covering unconventional unit names (e.g. LTS as Litres), considering that the columns should have the same normalised unit. The authors evaluate and compare their system (PUC) regarding currencies, data storage, length, mass, and volumes. For example, Grobid-quantities do not support currencies well because they hardly appear in scientific papers. At the moment of writing, Grobid-quantities does not support extraction from tables; however, since there is a plan to improve the table recognition in the Grobid library, this future work could be implemented within the same framework.
# Chapter 3

## Extraction from the full-text body of PDF documents

### 3.1 Introduction

PDF is a widely accepted format for sharing and presenting documents, as it ensures that the layout and formatting of the document remain consistent across different devices and operating systems. This is particularly important in scientific publications, where complex figures, tables, and equations must be accurately represented. The PDF format is designed for documents intended for printing, as they can embed fonts and high-quality images, and scientific journals traditionally required high-resolution print-ready files for publication. PDF files can be signed digitally, providing security and authenticity for scientific publications. This is important to ensure the integrity and trustworthiness of research findings, particularly in fields where reproducibility and transparency are paramount. Although isolating trends in the Internet jungle is challenging, PDF documents account for about 85% of the Common Crawl dataset<fnref n="1" /> and interest has been growing<fnref n="2" />. However, it is essential to note that while PDF has been widely used in scientific publications, emerging trends and technologies challenge its status as a "de facto" format. For example, the rise of open-access publishing has led to adopting HTML and XML formats for online articles, allowing greater interactivity, accessibility, and searchability of scientific content. However, those formats are penalised by the lack of standardised approaches, leading to a jungle of flavours of the same original format (e.g., XML-JATS format is one of the most widely used).

<footnote n="1">¹https://commoncrawl.github.io/cc-crawl-statistics/plots/mimetypes</footnote>

<footnote n="2">²https://trends.google.com/trends/explore?date=2000-01-01%202024-01-19&q=PDF</footnote>
## 3.2 Proposed approach

Given the critical role of PDF documents in disseminating scientific knowledge, supporting them natively is a necessary step for building effective TDM processes. Our approach uses the Grobid library (Generation of Bibliographic Data) [17], which can parse and structure text from PDF documents, such as scientific publications or patents. Grobid implements a set of pre-trained models supporting different architectures, including Conditional Random Field (CRF) [21], Recurrent Neural Networks [22], and transformer-based [23]. Other tools for document structuring, such as Cermine [90] and ParsCit [91] have been outperformed by the Grobid performances of its out-of-the-box models [92]. Grobid has been successfully extended to support several domain-specific problems, for example, recognition of astronomical entities [93], segmentation of dictionaries [94], software mention [95]. Among the various open-source tools available, at the time of writing, it is still actively developed and used in several large-scale research repositories, such as Mendeley [96], ResearchGate (https://researchgate.com), Scite.ai (https://www.scite.ai) [97].

Grobid offers several advantages. a) It does not require one to make multiple agreements with different publishers to obtain source documents. A growing number of open-access pre-print repositories offer a viable alternative for gathering input data. b) PDF support in Grobid allows one to focus on processing a single format instead of dealing with several XML flavours. c) It is integrated with PDFAlto (https://github.com/kermitt2/pdfalto), a specialised tool for converting PDF to XML, which mitigates extraction issues such as invalid character encoding and incorrectly ordered text flow. PDFAlto supports the resolution of embedded fonts and layout variants such as multi-column. d) The Grobid internal data model *LayoutTokens* access information in PDF documents at a low level for each token: style (italic, bold, superscript, and subscript), font (font type, font size), and coordinates within the visual document. e) By parsing PDF format natively, Grobid gives access to full text, figures, and tables and thus does not limit our work to public information such as abstracts. Finally, f) Grobid includes high-quality, out-of-the-box pre-trained machine learning models.

This and the following contributions are combined in Grobid-superconductors, a novel application of the Grobid library as a text mining pipeline for scientific publications in materials science. Our approach differs from other related works in several aspects. 1) Instead of abstracts [28], we support extraction from full-text body. 2) We are not required to reach an agreement with publishers [16]. 3) We focus on processing a single format instead of dealing with several XML flavours [16] 4) We avoid web scraping [15] and other processes that require addressing each website separately and may require regularly updating the harvester to follow potential website
changes. 5) We use low-level layout details as attributes in our ML models that
handle intricate representations of materials. For instance, including superscripts or
subscripts (which may not always be present in XML-formatted documents provided
by publishers) can be intuitively advantageous in identifying chemical formulas.

**3.2.1 Publisher's agreement for source data**

Obtaining scientific articles through publisher APIs (Application Programming In-
terface) or web scraping can be inconvenient for several reasons. Many publishers
control access to their API, demanding subscriptions or pay-per-view models, lim-
iting accessibility for those without institutional access or financial means. API
limitations, such as the number of requests allowed within a specific time frame, can
slow down data retrieval, especially with many articles. The lack of standardisation
in metadata and content formatting further complicates the extraction process. As
mentioned, each publisher provides its own "JATS-flavoured standard", making it
challenging to create a standardised data set from multiple sources. However, web
scraping does not guarantee data format stability and may breach the publisher's
terms of service, leading to legal and ethical complications. Open access resources,
including institutional repositories and pre-print servers, offer a wealth of scien-
tific literature at a reduced cost and in a consistent PDF format that represents
a convenient means to access and process articles from these sources due to their
accessibility and lack of legal constraints.

**3.2.2 Abstract versus full-text**

The scientific paper’s abstract, a concise summary located at the beginning of the
document, serves as a quick overview that summarises the research’s objectives,
methods, results, and conclusions in 150 to 250 words. Researchers frequently em-
ploy abstracts to assess the relevance of a paper to their work and for streamlined
database searches. In contrast, the full text of a scientific article delves into compre-
hensive details, spanning sections such as introduction, methodology, results, dis-
cussion, and conclusions. This exhaustive account is crucial for in-depth literature
reviews, detailed analysis of methods, data review, and evaluation of the validity
of the findings. Taking into account the perspective of TDM, both abstracts and
full-texts play distinct roles. Abstracts, due to their succinct nature, are suitable
for large-scale automated analyses, offering rapid insights into trends, topics, or key-
words across many papers. However, full texts are amenable to more in-depth TDM,
allowing for the detailed extraction of information and revealing nuanced relation-
ships between concepts, methodologies, and findings. Although previous studies [20]
have used abstracts, our research focuses on extracting complex information from scientific documents using the entire full-text.

## 3.3 TDM pipeline

Our TDM pipeline is built as a Grobid module called Grobid-superconductors. Grobid-superconductors is structured as a three-step process illustrated in Figure 3.1. The input is a PDF document converted to an internal model composed of text passages, tokens, and features. Those three elements are then used to perform Named Entities Recognition (NER), which are then linked by a Relation Extraction (RE) process. The output is a structured object serialised in JSON format.

<figure><img src="image_5.png" /><figcaption>Figure 3.1: Processing pipeline for extracting superconductors materials and properties.</figcaption></figure>

### 3.3.1 Document structuring and segmentation

The initial stage of our procedure involves using Grobid and PDFAlto to convert the PDF document. This results in segmenting raw data into a basic structure consisting of header, body, and annexes using the *Segmentation ML model*. These three structures are then parsed by a second Grobid model *Fulltext ML model* that identifies paragraphs, section heads, and references for body and annexes. We have created a custom process that analyses the structure and generates an internal model based on a list of text statements, tokens, and features. For example, the content of the tables is excluded, but the table captions are retained. In the process, we can also assign *section* and *subsection* information as the results of the Grobid first level: header, body, and annexe, and the second level (paragraph, table or figure caption, abstract, title) structures.

Header structures are parsed with the *Header ML model* to identify bibliographic data. This aggregated version only includes information that is relevant to our TDM
process. Our process selects a subset of bibliographic information from the header: title, authors, DOI, publisher, journal, and year of publication, and we consolidate them via Grobid to match the publisher's quality (even by processing the "pre-print version" of the publication).

<figure><img src="image_6.png" /><figcaption>Figure 3.2: Grobid-superconductors extraction processes (bibliographic information, superconductor entity extraction and sentence segmentation) within the Grobid cascade data flow.</figcaption></figure>

### 3.3.2 Sentence segmentation

An additional query concerning natural language processing (NLP) pertains to the choice between employing sentence-based or paragraph-based text. While paragraphs can be extracted as part of the PDF document's layout, obtaining sentences requires an additional step through a sentence segmenter. Sentences are typically shorter, which offers advantages in processing with large DL models. During training, sentences use less memory and allow us to train models with a larger "batch size", which has been shown to improve efficiency and obtain better results [65].

We used sentence-based segmentation in Grobid-superconductors after performing small-scale preliminary experiments on our task. For the Named Entities Recognition (NER) task, we trained and evaluated a sequence labelling model for each approach (paragraph and sentence based) in four annotated documents (3/1 document partition for training/evaluation) from SuperMat, a dataset that we built [4] and is described in Chapter 5. As indicated in Table 3.1, the F1 score increased by
<table><caption>Table 3.1: Results from cross-validation for sentence-based and paragraphs-based text. Measurements are micro average. P: Precision, R: Recall, and F1: F1-score.</caption><thead><tr><th>Label</th><th>P</th><th>R</th><th>F1</th></tr></thead><tbody><tr><td>Paragraph-based</td><td>44.44</td><td>27.21</td><td>33.76</td></tr><tr><td>Sentence-based</td><td>48.41</td><td>50.00</td><td>51.70</td></tr></tbody></table>

17.94% points when using sentence-based text.

For the RE task, we found in a previous study [12] that the sentence-based approach would favour higher precision, while the paragraph-based approach would result in greater recall.

Since Grobid supports partitioning documents into paragraphs, a sentence segmenter is necessary. The problem has been long investigated [98–104]; nonetheless, there are still challenges in segmenting paragraphs from scientific text: reference markers (also called *reference callouts*), formulas, and other constructs may contain periods and mislead algorithms. Thanks to Grobid and the ability to recognise reference markers and formulas, we use the collected ones from the text as features to improve paragraph segmentation in sentences: segmentation is aborted if the end of a sentence falls within the boundaries of one of these markers. For example, a sentence such as "[...]The evaluation from (2021, Foppiano et al.) offers a solid validation for our method" containing a reference in the form "(2021, Foppiano et al.)". It may be mistakenly segmented in the middle token "[...] et al.", and the new sentence starts with ") offers [...]".

## 3.4 Conclusions

In conclusion, the application of innovative techniques, exemplified by Grobid, marks a significant milestone in materials science. This effort represents a novelty for integrating such methods within the domain, showcasing the potential for transformative advancements. By sidestepping the complexities and expenses associated with signing agreements with scientific publishers and the intricate landscape of XML formats, this approach not only streamlines the dissemination process but also liberates researchers from the often daunting challenges posed by conventional publishing models.

In the evolving landscape of scientific dissemination, where alternative means gain traction, it is noteworthy that the PDF format remains resilient as the "de
facto” standard. The decision to publish through open access aligns with the trend toward greater accessibility and openness in scholarly communication. The increasing availability of open-access articles further reinforces the utility and relevance of Grobid, positioning it favourably as a valuable tool in the arsenal of researchers seeking efficient and cost-effective methods for sharing scientific knowledge in the materials science domain.

In this contribution, we introduced Grobid-superconductors, and we described the component responsible for reading and structuring PDF documents, preparing the data to be processed by our novel specialised models, which are discussed in Chapters 4 and 6.

This contribution was published in the paper "Automatic extraction of materials and properties from scientific literature" [24].
# Chapter 4

## ML-based extraction of materials-related expressions from text

### 4.1 Introduction

This chapter discusses the Grobid-superconductors component responsible for data extraction through an intricate set of ML models encapsulated in data parsers for extracting material-related entities from scientific text. Identifying named entities in the field of materials science poses a formidable challenge, particularly due to the intricate nature of material expressions that are composed of complex and extensive sequences requiring meticulous attention. The definition of materials remains elusive in this domain, lacking a clear-cut delineation (refer to Chapter 1). Moreover, the boundaries within materials science are loose, subject to variations based on specific subdomains, and researchers focus on distinct aspects or phenomena associated with the materials. Material expressions exhibit significant variability within superconductor research, incorporating various types of information within a single sequence. This can range from the chemical formula and doping ratio ("Zn-doped", "2% Cu-doped", for instance) to the shape of the material (e.g., single crystal, wire, polycrystalline), as exemplified by expressions such as `single crystal La x Fe 1 x 0 7` (with x = 0.1, and 0.2). Analysing non-stoichiometric formulas, in particular, presents a complex task, as these necessitate resolution with substitution values scattered throughout the paper.

The complexity inherent in deciphering such material expressions has propelled
our exploration into the realm of Machine Learning (ML) as a means to model these
intricate boundaries. ML has demonstrated its efficacy in various domains, including
bioinformatics and chemistry [105–107], often surpassing rule-based or lexicon-based
approaches by achieving a delicate balance between flexibility and simplicity. How-
ever, it is essential to acknowledge that the success of ML is contingent upon the
availability of data. This facet presents a notable challenge in material informatics
compared to other domains. This challenge, in turn, has led us to create a novel
dataset of annotated and linked text of scientific articles from superconductor re-
search. In Chapter 5, we list the characteristics at the time of writing (number
of articles, number of entities, etc.) and the methodology we used to construct it.
Moreover, the notation incorporates supplementary text formatting choices, such
as subscript and superscript, italic and bold. When these formatting options are
converted to text, they are usually standardised to a regular font size, resulting in
the omission of information. To tackle this problem, Grobid has been crucial in pre-
serving these data by utilising PDF layout information, as discussed in Chapter 3.
This approach improves the accuracy and comprehensiveness of our identification of
named entities in materials science.

We propose a novel solution that is organised as a two-stage process: initially, the text that is selected from various structures extracted by Grobid (Chapter 3) we pass them to a set of parsers that apply Named Entity Recognition (NER) and extract significant entities. Next, the extracted entities are processed in pairs by a relation extraction (RE) process, and results are collected.

## 4.2 Identification of complex materials sequences

The process of identifying complex material sequences is a recently developed method
that comprises four novel aspects.

4.2.1 High-quality training data

We have chosen to use machine learning for NER tasks due to the various advan-
tages we have previously outlined. Machine learning techniques are particularly
well-suited for extracting complex structures, although they require high-quality
training data. We encountered a scarcity of such data, prompting us to create a
new dataset. This project was an interdisciplinary collaboration between computer
scientists and materials scientists. Domain experts specialising in superconductor
materials provided guidance and technically validated the dataset. In Chapter 5 of
this manuscript, we provide a comprehensive description of the data model and a
detailed account of the dataset construction process, which we refer to as SuperMat.
The primary objective of Chapter 5 is to shed light on the specific characteristics
of the domain, including the types of data, providing examples and discussing their
relationships.

4.2.2 Positive sampling

The SuperMat training data is prepared using a new strategy to improve the model’s
ability to identify entities with low probabilities, also known as ”positive sampling”.
This approach prioritises recall over precision, aiming to capture as many entities
as possible, even if it means sacrificing precision scores. Although some unrelated
entities may be extracted, they are likely to be filtered out in the subsequent step of
RE, which was developed as rule-based and thus more conservative. This decision
is based on expecting the final result to undergo human validation. RE with a
high recall may negatively impact efficiency, as the humans validating the data
would need to spend significant time removing numerous incorrect records. On the
contrary, extracting a few high-precision records will require less time to be ready for
use. From a holistic perspective, the recall-precision bias will be balanced between
the two processes, as demonstrated in the end-to-end evaluation results, which will
be further discussed in Section 4.3.4. Positive sampling is applied by removing
examples that do not contain any entities (negative examples, Figure 4.2a). This
approach provided an improvement of 2% in both precision and recall compared to
the result without sampling. We tested additional sampling approaches, called active
and random sampling, that target highly imbalanced datasets [108]. We tested them
using ratios of negative examples of 0.1, 0.25, 0.5 and 1.0. However, they did not
provide stable evidence suggesting any improvement, considering that our dataset
was balanced.

4.2.3 ML architectures

To determine the most effective architecture for our task, we assess several tradi-
tional machine learning (ML) methods supported by Grobid through Wapiti [109]
and the DL library DeLFT [110]. The architectures we consider include the linear
CRF (CRF) [21], Bidirectional LSTM with CRF (BidLSTM_CRF) [22], Features
Bidirectional LSTM with CRF (BidLSTM_CRF_FEATURES) [22], and a fine-tuned
BERT-based transformer. For our specific task, we prefer to use the SciBERT [67]
encoder with a CRF as the activation layer (SciBERT). Unlike BERT, SciBERT
has been trained on the scientific text and has demonstrated superior performance
in various NLP tasks, including sequence labelling, classification, and question an-
swering [67]. It is worth noting that the CRF and BidLSTM.CRF_FEATURES architectures utilise features summarised in Table 4.1.

### 4.2.4 Two-levels approach

To handle a diverse range of entity types, we must address the complexity of the information structures. Since most entities are structured around material names, we have implemented a two-level strategy to manage them effectively. In this approach, after processing the text with the first level ML models in the "Superconductor parser" (Section 4.2.4), a dedicated ML-based "Material parser" for process the material entities (Section 4.2.4). While previous studies such as [15, 16, 83] have mainly focused on extracting formulas, we aim to extract a larger sequence and then deal with material segmentation separately. By adopting this approach, we can address the problems in separate instances, thus reducing their complexity. This parser can handle noisy data and be applied to various fields (Section 4.2.4).

The schema illustrated in Figure 4.1 comprises three main components: the "Superconductor parser", the "Material parser" and the "Post-processing".

#### Superconductor parser

The "superconductor parser" extracts first-level entities (Table 4.2) by aggregating the resulting entities from two ML models: "Superconductors ML model" and "Quantities ML model". The "Quantities ML model" is reused from a separate Grobid module to extract measurements and physical quantities (Chapter 6) and is limited in scope to target only entities of temperatures and pressures. Overlapping entities of temperatures and pressures are merged, exact duplicates are removed, and the largest entities (in terms of string length) are preserved. The "Superconductors ML model" was trained with SuperMat (Chapter 5), whose schema is summarised in Table 4.2.

Entities of type `<material>`, which may contain heterogeneous mixed information, are passed to the "Material parser", which aggregates ML and rule-based methods.
<figure><img src="image_7.png" /><figcaption>Figure 4.1: 2-level architecture for solving the NER task. The white rectangles indicate the extracted information (described in Tables 4.2 and 4.3).</figcaption></figure>
## Material parser

The Material parser is an expression of different approaches. First, the entity is passed through a novel Material ML model to segment and identify its content (Table 4.3). Then, different processes are applied depending on which information is available:

* Formulas are decomposed into a structured composition. We identify each element-stoichiometry pair (e.g., “O”: 7.0) using text2chem [16] and PyMatgen [27]; if only the material name is available, we lookup its formula (e.g., hydrogen to H),

* Using heuristics, we classify the formula by assigning multiple classes as they are understood by superconductor researchers, for example, cuprate, oxides, alloys, etc.

* Using the variables and values extracted, we substitute them into partial formulas. For example, in La 4 Fe 2 A 1-x 0 7 (A=Mg,Co; x=0.1,0.2), we substitute A and x using their parsed values, and applying permutations, we obtain four resolved formulas: La 4 Fe 2 Mg 0.9 0 7, La 4 Fe 2 Mg 0.8 0 7, La 4 Fe 2 Co 0.9 0 7 and La 4 Fe 2 Co 0.8 0 7.

### 4.2.5 Post-processing

Finally, after all entities are extracted, the post-processing aggregates different mentions of the same materials using the parsed formulas at the document-level. For example, formula with partial substitutions such as La 2 Fe 1-x 0 7 (x = 0.1, 0.2) will be aggregated with materials like La 2 Fe 0.9 0 7 appearing in other sections of the same document.

### 4.2.6 Extraction of relation from materials-related entities

Relation extraction (RE) links materials and their corresponding properties. We explored different approaches, such as the use of dependency parsing [112–115] or machine learning [116,117]. Unfortunately, these methods were developed or trained using text from news articles, and their performances on scientific text were unsatisfactory. Another aspect is that our dataset was too small, in terms of the number of relations, to train any of these ML methods successfully. In particular, working with dependency parsers was extremely difficult due to their inability to districate in
such complex writing. Compensating such poor performances resulted in the need
to write over-complicated rules.

We developed a rule-based algorithm that links together pairs of entities:

* **material-tcValue**: The link between a material and its corresponding T<sub>c</sub>.

* **tcValue-pressure**: The link between T<sub>c</sub> and its related critical pressure.

* **me_method-tcValue**: The link between T<sub>c</sub> and its corresponding measurement method.

Entities of type <tcValue> are pre-processed through a classifier that establishes
whether or not the temperatures refer to the superconductivity. This excludes tem-
peratures referring to irrelevant properties (e.g., annealing, transition, Curie) that
might be incorrectly extracted upstream. This rule-based classifier combines the
extracted entities of T<sub>c</sub> expressions (label <tc>) with a set of predefined standard
terms. If a temperature is not considered a T<sub>c</sub>, it is excluded from the list of possible
linking candidates.

Two scenarios are considered. First, if entities to be linked in the sentence
are only two, they are linked automatically, else further rules are applied. If the
word “respectively” appears in the sentence, we apply “order-linking”. For example,
consider the following sentence:

P-or Ba-122 and Co-doped Ba-122 have lower T<sub>c</sub>'s of about 30 K and 24 K, respectively, which makes helium-free operation questionable.

It contains the word “respectively”, and by applying “order-linking”, P-or *Ba122* is
assigned to 30 *K* and Co-doped *Ba-122* to 24 *K*.

Suppose the word “respectively” does not appear in the sentence. In that case,
we apply “distance linking” by defining the distance measurement *d* as a value
calculated as the number of characters between the centroid of each entity, measured
in the number of characters. Entities surrounded by parentheses are expanded to
cover the "whole parentheses", and their centroid is updated. As an example, in the
sentence

We tested two materials, MgB2 (T<sub>c</sub> = 39 K) and FeSe (T<sub>c</sub> = 16 K).

39 K is closer to FeSe ($d$=10) than to MgB2 ($d$=11). In this example, however,
both temperature entities would be expanded to their containing parenthesis (e.g.
"39 K" to "(Tc = 39 K)". In this case, the centre of the entity "39 K" is shifted to the left, from the initial value of 38 to 35, and the distance from MgB2 is reduced from d=11 to d=8. As a result, the MgB2 entity is correctly linked to "39 K".

Distance calculation is also adjusted with the addition of “penalties” by doubling
the calculated distance when specific keywords or punctuations (“, “, “;”, “and”,
“but’, “while’, “whereas’, “which”, “although”) appear between two entities because
they represent a logical separation of predicates [118]. In the above example, the
distance between 39 K and FeSe would be doubled (d=20), and the link would not
be made.

4.3 Results

4.3.1 Identification of materials-related entities from text

Experimental settings

We used SuperMat (Chapter 5) for training and evaluation. The holdout set evaluation uses a fixed part of a dataset for validation. The dataset was divided into a training set corresponding to 132 documents (76%) and a holdout set comprising 32 documents (24%).

Selection must be performed to reproduce the same distribution of entities from
the original dataset. We assembled the holdout set using stratified sampling and
adjusting manually, ensuring it had a similar ratio of examples, entities and unique
entities. The remaining was used as the training set (Figure 4.2a). Maintaining
the same rate for the distribution of entity types between the two sets was more
challenging. On average, we obtained approximately 15-18% labels of each type in
the holdout set (Figure 4.2b), except for the <material> label (23%).

We defined the “out-of-domain” ratio as the number of unique entities from
the holdout set that were not in the training set. The holdout set “out-of-domain”
ratio was, on average, around 72%, which challenges the model generalisation (every
100 entities in the holdout set, 72 were never seen before during training). Most
labels had an “out-of-domain” ratio above 50% (Figure 4.3); <material>, the most
important label, had the highest ratio (82%) while <me_method> and <pressure>
have the lowest (25% and 33%). The low ratio of <me_method> can be explained by
their low entity variability (11.44%).
<figure><img src="image_8.png" /><figcaption>Figure 4.2: Holdout/training set distribution for (a) general metrics and (b) entity labels; entities and unique entities indicate the number of labelled entities with and without value duplicates, respectively, and positive examples (+) and negative examples (-) indicate the number of sentences with at least one entity and with no entities, respectively.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Figure 4.3: Holdout “out-of-domain” rates. The entities from the holdout set that are also in the training set are “in-domain”, and the entities that are not in the training set are “out-of-domain”.</figcaption></figure>

## Results

We evaluated each ML model by repeating the training and evaluation process five times (except for the CRF architecture, which is entirely predictable). Although the training and evaluation datasets are fixed, the way the training examples are provided to the model is randomly shuffled at each step, adding some entropy to the training, which is visible in the evaluation scores. The results are then averaged as reported in Table 4.4.

SciBERT obtained the best results with an F1 of 77.03% and a recall of around 80.69% (Table 4.4). The features did not improve RNN models: BidLSTM_CRF and BidLSTM_CRF_FEATURES resulted in the same F1 score. This result is a surprise because features such as superscript/subscript were expected to be determinants for recognising material sequences.
The <pressure> label had the lowest performance scores in all architectures. We believe that 274 training examples are not sufficiently large. Pressure expressions depend on the context and the property they refer to, so in many occurrences, they may refer to different types of pressure (e.g., annealing pressure). The label with the highest score was <material>, with F1 values of 80.77% and 78.06% for SciBERT and BidLSTM_CRF, respectively. In addition, <material> had the highest “out-of-domain” ratio in the holdout set (greater than 75%, Figure 4.3) and the highest “label variability” (the ratio between unique entities and total entities, about 42%), which suggests that the model recognises correctly materials that have not been “seen” during the training. On the other hand, the <me_method> label, which has lower “label variability” (around 11%) and a low “out-of-domain” ratio, had an F1 score of 66.56% with SciBERT and 65.92% with BidLSTM_CRF. For <tc>, the CRF outperformed the other architectures (F1 score of 83.96%), especially SciBERT (78.35%). This outcome can be explained by the extremely low variability (12.69%) of entities labelled as <tc>.

<figure><img src="image_10.png" /><figcaption>Figure 4.4: Examples taken from two sources [1, 2] of results from three different architectures: CRF, BidLSTM_CRF and, SciBERT. The boxes annotating the text represent the extracted entities (material are indicated in light blue, T<sub>c</sub> in green, and T<sub>c</sub> expressions in yellow).</figcaption></figure>

SciBERT shows good generalisation capacity for unseen examples or examples
appearing in different contexts. For example, in Figure 4.4a, only SciBERT correctly
extracts “above 100K”, while CRF misses it entirely and BidLSTM_CRF misses
“above”. In the training data, “above 100K” is not present, but “below 100K”
and “~100K” are present, and several other entities contain the token “above”, and
SciBERT can understand that the token “above” is relevant to the temperature. In a
second example (Figure 4.4b), only SciBERT can correctly extract “W-C nanowire”,
which is not contained in the SuperMat training data. Unfortunately, we cannot
check whether “above 100K” or “W-C nanowire” are also present in the dataset
used in the pre-train of SciBERT by their authors [67] because the data are not
available.
4.3.2 Material parser segmentation model

We trained the Material ML model using the second layer annotation set from Super-
Mat comprising entities of type <material>. The resulting entities are segmented
into eight pieces of information illustrated in Table 4.3. We trained the same archi-
tecture in the same way as the “Superconductors ML model”.

Experimental settings

In this model, we created an independent holdout set because manual annotation is
performed on smaller chunks of text and requires less effort than annotating entire
sentences. We used material data extracted from a dataset of 500 documents (“500-
papers”) from three publishers: *American Institute of Physics* (AIP), *American
Physical Society* (APS) and *Institute of Physics* (IOP) [12].

<figure><img src="image_11.png" /><figcaption>Figure 4.5: Holdout/training set for the Material ML model: (a) general metrics and (b) entity labels.</figcaption></figure>

The resulting holdout set has an average coverage greater than 25% (Figure 4.5)
and an average “outside the domain” ratio of 83.93% (Figure 4.6).

Results

SciBERT obtained the best results, with F1 at 84.15% (Table 4.9). The inclusion of features in the BidLSTM_CRF architecture only improved the results by less than 1% (from 83.13 to 83.76%). The label <fabrication> did not perform well in any architecture, most likely because it is too generic (Table 4.3), and the content is too heterogeneous. Another label, <substrate> has only one-third of the training examples of <fabrication> but obtained results that were three times higher with
<figure><img src="image_12.png" /><figcaption>Figure 4.6: Holdout “out-of-domain” rates for the Material ML model. The entities from the holdout set that are also in the training set are the in-domain, and the entities that are not in the training set are the out-of-domain.</figcaption></figure>

SciBERT, suggesting that `<fabrication>` should be split into separate and more homogeneous labels.

### 4.3.3 RE evaluation

This rule-based linking was evaluated using the linked entities from SuperMat [4] (Table 4.10) and is divided considering each link type. The F1 score for the material-tcValue relation was about 80% with a precision of 88.40%. tcValue-pressure F1 score was 3% lower than material-tcValue considering much less data available (support was 118 compared with 726).

### 4.3.4 End to end evaluation

End-to-end evaluation (E2EE) measures the system's capacity from the PDF documents until the final linked results. We limited the scope of the E2EE to the triplet 'material-T_c-pressure', which, at the moment, is the backbone upon which the database is built. We performed the E2EE on the "500-papers" dataset where we manually examined the resulting database as follows: 1) we marked invalid records and 2) we identified the cause of failure from a predefined set of five error types (Figure 4.7):

* **From table:** the extracted text is wrongly extracted from a table. Although table content is ignored, the error rate from the Grobid library is still relevant due to the lack of training data.
* **Material-related NER**: entities are either not, wrongly, or partially recognised.

* **Properties NER**: quantity entities (pressure, temperature) are not correctly extracted. We measured this error separately to identify the failure that could be shared with the Quantity ML model.

* **Tc classification**: the temperature is wrongly classified as superconducting Tc.

* **RE**: given the initial steps were performed correctly, the resulting entities are not linked correctly.

<figure><img src="image_13.png" /><figcaption>Figure 4.7: *Error types* in the context of the data flow.</figcaption></figure>

The E2EE scores are summarised in Table 4.11. The recall is omitted from the table because it is less relevant. However, we estimated that the recall is between 55% and 60%, considering a sample of articles. The precision score (micro average) was 72.60% for all the subsections, although the precision of figure captions (59.28%) and unknown subsections (57.14%) were lower than those of the other subsections (> 70%). The ‘unknown’ subsections indicate that the extracted text’s structure was not well identified by Grobid, but it was nonetheless aggregated. The overall score increases to 73% when excluding unknown subsections, 75.24% when excluding figure captions, and 79.14% when excluding both. Excluding these two subsections will not affect the amount of text because they account for less than 20% of the total number of subsections.

The error types are summarised in Figure 4.8. The most common failures originate from T_c classification (40%), RE (32%), and Material-related NER (20%). The most common T_c classification failures are incorrect recognition of 1) relative values of T_c (e.g., 1 K higher than material X); 2) values indicating the transition temperature width (ΔT_c); 3) temperature values that are not T_c, for example, material synthesis temperatures (T), other critical transition temperatures that are not superconducting (e.g., T_Curie); and 4) values of temperature at which there is no superconductivity (e.g., “at 70 K there is no superconductivity”). “RE errors”
<figure><img src="image_14.png" /><figcaption>Figure 4.8: Error type distribution in the E2EE of the 500-papers dataset.</figcaption></figure>

occur mainly when the text compares relative values of T~c~ using materials as the basis for comparison (e.g., “The T~c~ = 38 K is similar to the one of MgB~2~”). Finally, “Material-related NER” issues mainly originate from: 1) implicit mention of the base material when experimented using different “substrates” combination, and 2) mismatches between <material> and <class> which, by definition, overlap (see the SuperMat entities analysis in Section 5.5.3).

## 4.4 Conclusions

This work introduces our proposed method for automatically extracting material-related expressions. This method is an essential component of our project, which aims to create a database of experimental data sourced from scientific literature. We introduced four novelties: a high-quality dataset, SuperMat. The data was prepared using positive sampling, which increases recall in the output models. Each model was trained and evaluated among several architectures, of which the best performing were selected. Finally, we designed a 2-level process where a dedicated model parsed the material entities. We applied this specialised process in an unexplored area, where extracting information is challenging due to the intricate nature of the field and the specific attributes of the entities involved.

The contribution described in this chapter has been published in the paper "Automatic extraction of materials and properties from scientific literature" [24] and partially (more details in Chapter 6) "Automatic identification and normalisation of physical quantities from scientific literature" [119].
<table><caption>Table 4.1: Summary of the features used in the *superconductors* and *material* ML models. All under Architecture indicate only BidLSTM.CRF FEATURES and CRF.</caption><thead><tr><th>#</th><th>Feature</th><th>Model</th><th>Architecture</th></tr></thead><tbody><tr><td>1</td><td>current token</td><td>all</td><td>all</td></tr><tr><td>2</td><td>current token lower cased</td><td>all</td><td>all</td></tr><tr><td>3-6</td><td>(four features) current token, prefix characters 1 to 4</td><td>all</td><td>CRF</td></tr><tr><td>7-10</td><td>(four features) current token, suffix characters 1 to 4</td><td>all</td><td>CRF</td></tr><tr><td>11</td><td>information about capitalisation: first character (INITCAP), all characters (ALLCAPS), none (NOCAPS)</td><td>all</td><td>all</td></tr><tr><td>12</td><td>digits content: all (ALLDIGIT), some digits (CONTAINDIGIT), no digits (NODIGIT)</td><td>all</td><td>all</td></tr><tr><td>13</td><td>(boolean) the token is composed of a single character punctuation information and normalisation to placeholders: no punctuation (NOPUNCT), open or end brackets (OPENBRACKET, ENDBRACKET), various punctuation (DOT, COMMA, HYPHEN, QUOTE), open or close quotes (OPENQUOTE, ENDQUOTE), anything else (PUNCT)</td><td>all</td><td>all</td></tr><tr><td>15</td><td>Shadow the numbers</td><td>all</td><td>CRF</td></tr><tr><td>16</td><td>Shadow any characters: “x” for lowercase, “X” for uppercase, “d” for digits</td><td>all</td><td>CRF</td></tr><tr><td>17</td><td>As the previous but compressed</td><td>all</td><td>CRF</td></tr><tr><td>18</td><td>Font name</td><td>superconductors</td><td>all</td></tr><tr><td>19</td><td>Font size</td><td>superconductors</td><td>all</td></tr><tr><td>20</td><td>Font style: standard (BASELINE), superscript (SUPERSCRIPT) or subscript (SUBSCRIPT)</td><td>superconductors</td><td>all</td></tr><tr><td>21</td><td>(boolean) if the token style is bold</td><td>superconductors</td><td>all</td></tr><tr><td>22</td><td>(boolean) if the token style is italic</td><td>superconductors</td><td>all</td></tr><tr><td>23</td><td>(boolean) the token is identified as a chemical compound by DataExtractor [11]</td><td>superconductors</td><td>all</td></tr></tbody></table>
<table><caption>Table 4.2: Entities extracted by the superconductors parser.</caption><thead><tr><th>Entity (tag)</th><th>Description</th></tr></thead><tbody><tr><td>Material (&lt;material&gt;)</td><td>Materials and samples names, formulas (including non-stochiometric formulas), substitution variables of values and elements, shape, doping, and substrate</td></tr><tr><td>Class (&lt;class&gt;)</td><td>Groups of materials having similar characteristics or common strategic compounds that define their nature</td></tr><tr><td>T<sub>c</sub> value (&lt;tcValue&gt;)</td><td>The value of the superconductor critical temperature</td></tr><tr><td>T<sub>c</sub> expressions (&lt;tc&gt;)</td><td>Expressions in the text that provides information about the phenomenon of superconductivity related to a value, interval or variation of T<sub>c</sub></td></tr><tr><td>Measurement method (&lt;me_method&gt;)</td><td>Technique used to measure or calculate the presence of superconductivity</td></tr><tr><td>Applied pressure (&lt;pressure&gt;)</td><td>Applied pressure when superconductivity is recorded</td></tr></tbody></table>

<table><caption>Table 4.3: Entities extracted by the material parser.</caption><thead><tr><th>Entity (tag)</th><th>Description</th></tr></thead><tbody><tr><td>Name (&lt;name&gt;)</td><td>The canonical name of a material (e.g., hydrogen, PCCO, carbon)</td></tr><tr><td>Formula (&lt;formula&gt;)</td><td>Chemical formula of the material (e.g., Pr1.869Ce0.131Cu0 4-, MgB2, La 2-x Sr x Cu0 4)</td></tr><tr><td>Doping (&lt;doping&gt;)</td><td>Doping ratio and doping materials that are added to the material name (e.g., Zn-doped, 2% Zn-doped)</td></tr><tr><td>Shape (&lt;shape&gt;)</td><td>shape of the material (e.g. single crystal, polycrystalline, thin film, powder, film)</td></tr><tr><td>Substitution variables (&lt;variable&gt;)</td><td>Variables that can be substituted in the formula.</td></tr><tr><td>Substitution values (&lt;value&gt;)</td><td>Values expressed in the doping.</td></tr><tr><td>Substrate (&lt;substrate&gt;)</td><td>Substrates as defined in the material name</td></tr><tr><td>Fabrication (&lt;fabrication&gt;)</td><td>Additional information that does not belong to any of the previous tags (e.g., intercalated, electron-doped)</td></tr></tbody></table>
<table><caption>Table 4.4: Evaluation scores (%) for the Superconductor ML model in the four architectures. For the DL architecture, the results are averaged over five runs. Support (Supp) indicates the number of labels in the training data. Values in bold indicate the highest score. P: precision, R: recall.</caption><thead><tr><th rowspan="2">Label</th><th colspan="2">CRF</th><th colspan="3">BidLSTM_CRF</th><th colspan="3">BidLSTM_CRF FEATURES</th><th colspan="3">SciBERT</th><th rowspan="2">Supp</th></tr><tr><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th></tr></thead><tbody><tr><td>&lt;class&gt;</td><td>79.74</td><td>66.79</td><td>72.69</td><td>79.01</td><td>72.62</td><td><strong>75.66</strong></td><td>77.84</td><td>72.40</td><td>74.97</td><td>72.95</td><td>75.28</td><td>74.09</td><td>1646</td></tr><tr><td>&lt;material&gt;</td><td>79</td><td>72.15</td><td>75.42</td><td>79.25</td><td>76.94</td><td>78.06</td><td>81.07</td><td>75.10</td><td>77.94</td><td>80.15</td><td>81.42</td><td><strong>80.77</strong></td><td>6943</td></tr><tr><td>&lt;me_method&gt;</td><td>60.25</td><td>68.73</td><td>64.21</td><td>56.41</td><td>79.49</td><td>65.92</td><td>55.86</td><td>80.45</td><td>65.90</td><td>56.26</td><td>81.52</td><td><strong>66.56</strong></td><td>1883</td></tr><tr><td>&lt;pressure&gt;</td><td>46.15</td><td>29.27</td><td>35.82</td><td>49.45</td><td>58.05</td><td>52.53</td><td>50.25</td><td>60.49</td><td><strong>54.36</strong></td><td>41.72</td><td>52.68</td><td>46.51</td><td>274</td></tr><tr><td>&lt;tc&gt;</td><td>84.36</td><td>83.57</td><td><strong>83.96</strong></td><td>78.61</td><td>82.54</td><td>80.48</td><td>79.19</td><td>82.07</td><td>80.60</td><td>74.46</td><td>82.66</td><td>78.35</td><td>3741</td></tr><tr><td>&lt;tcValue&gt;</td><td>69.8</td><td>66.24</td><td>67.97</td><td>70.36</td><td>75.16</td><td>72.67</td><td>68.95</td><td>76.56</td><td>72.52</td><td>70.90</td><td>79.74</td><td><strong>75.06</strong></td><td>1099</td></tr><tr><td>All (micro avg)</td><td>76.88</td><td>72.77</td><td>74.77</td><td>74.59</td><td>77.67</td><td>76.09</td><td><strong>75.17</strong></td><td>76.79</td><td>75.96</td><td>73.69</td><td><strong>80.69</strong></td><td><strong>77.03</strong></td><td></td></tr></tbody></table>
<table><caption>Table 4.5: Holdout/Training set distribution (%) between training and holdout sets for the Superconductor ML model. Positive examples indicate the number of sentences with at least one entity, and negative examples the number of sentences with no entities.</caption><thead><tr><th></th><th>training</th><th>holdout</th><th>% holdout/training</th></tr></thead><tbody><tr><td>documents</td><td>132</td><td>32</td><td>24.24%</td></tr><tr><td>examples</td><td>16902</td><td>3905</td><td>23.10%</td></tr><tr><td>entities</td><td>15586</td><td>3112</td><td>19.97%</td></tr><tr><td>unique entities</td><td>6699</td><td>1372</td><td>20.48%</td></tr><tr><td>positive examples</td><td>8380</td><td>1776</td><td>21.19%</td></tr><tr><td>negative examples</td><td>8522</td><td>2129</td><td>24.98%</td></tr></tbody></table>

<table><caption>Table 4.6: Holdout/Training set distribution (%) between training and holdout sets on different labels for the Superconductors ML model.</caption><thead><tr><th>label</th><th>training</th><th>holdout</th><th>% holdout/training</th></tr></thead><tbody><tr><td>&lt;tc&gt;</td><td>3741</td><td>639</td><td>17.08%</td></tr><tr><td>&lt;class&gt;</td><td>1646</td><td>271</td><td>16.46%</td></tr><tr><td>&lt;material&gt;</td><td>6943</td><td>1649</td><td>23.75%</td></tr><tr><td>&lt;me_method&gt;</td><td>1883</td><td>355</td><td>18.85%</td></tr><tr><td>&lt;tcValue&gt;</td><td>1099</td><td>157</td><td>14.29%</td></tr><tr><td>&lt;pressure&gt;</td><td>274</td><td>41</td><td>14.96%</td></tr></tbody></table>

<table><caption>Table 4.7: Holdout/Training set distribution (%) training and holdout sets for the Material ML model.</caption><thead><tr><th></th><th>training</th><th>holdout</th><th>% holdout/training</th></tr></thead><tbody><tr><td>examples</td><td>13648</td><td>5728</td><td>41.97%</td></tr><tr><td>entities</td><td>4512</td><td>2817</td><td>62.43%</td></tr><tr><td>unique entities</td><td>9268</td><td>3292</td><td>35.52%</td></tr></tbody></table>
<table><caption>Table 4.8: Holdout/Training set distribution (%) training and holdout sets on different labels for the Material ML model.</caption><thead><tr><th>label</th><th>training</th><th>holdout</th><th>% holdout/training</th></tr></thead><tbody><tr><td>&lt;fabrication&gt;</td><td>94</td><td>44</td><td>46.81%</td></tr><tr><td>&lt;formula&gt;</td><td>6301</td><td>2569</td><td>40.77%</td></tr><tr><td>&lt;shape&gt;</td><td>809</td><td>841</td><td>103.96%</td></tr><tr><td>&lt;substrate&gt;</td><td>32</td><td>148</td><td>462.50%</td></tr><tr><td>&lt;name&gt;</td><td>1930</td><td>949</td><td>49.17%</td></tr><tr><td>&lt;variable&gt;</td><td>1795</td><td>449</td><td>25.01%</td></tr><tr><td>&lt;value&gt;</td><td>1895</td><td>463</td><td>24.43%</td></tr><tr><td>&lt;doping&gt;</td><td>792</td><td>265</td><td>33.46%</td></tr></tbody></table>
<table><caption>Table 4.9: Evaluation scores (%) of the Material ML model with holdout set. Support (Supp) indicates the number of labels in the training data. Values in bold indicate the highest score. P: precision, R: recall.</caption><thead><tr><th rowspan="2">Label</th><th colspan="2">CRF</th><th colspan="3">BidLSTM_CRF</th><th colspan="3">BidLSTM_CRF FEATURES</th><th colspan="3">SciBERT</th><th rowspan="2">Supp</th></tr><tr><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th></tr></thead><tbody><tr><td>&lt;doping&gt;</td><td>60.41</td><td>55.85</td><td>58.04</td><td>67.98</td><td>62.42</td><td>64.95</td><td>69.00</td><td>62.34</td><td><strong>65.43</strong></td><td>63.58</td><td>62.79</td><td>63.16</td><td>792</td></tr><tr><td>&lt;fabrication&gt;</td><td>40.00</td><td>4.55</td><td>8.16</td><td>23.61</td><td>5.91</td><td>9.24</td><td>37.33</td><td>9.09</td><td>14.48</td><td>22.51</td><td>13.18</td><td><strong>16.52</strong></td><td>94</td></tr><tr><td>&lt;formula&gt;</td><td>80.81</td><td>82.29</td><td>81.54</td><td>82.59</td><td>84.14</td><td>83.35</td><td>83.83</td><td>85.14</td><td>84.47</td><td>84.53</td><td>86.56</td><td><strong>85.53</strong></td><td>6301</td></tr><tr><td>&lt;name&gt;</td><td>72.2</td><td>63.75</td><td>67.71</td><td>76.29</td><td>78.76</td><td>77.43</td><td>74.51</td><td>80.38</td><td>77.33</td><td>77.18</td><td>81.86</td><td><strong>79.44</strong></td><td>1930</td></tr><tr><td>&lt;shape&gt;</td><td>90.89</td><td>92.51</td><td>91.69</td><td>90.93</td><td>95.79</td><td><strong>93.29</strong></td><td>90.33</td><td>95.74</td><td>92.96</td><td>89.67</td><td>97.20</td><td>93.28</td><td>809</td></tr><tr><td>&lt;substrate&gt;</td><td>37.04</td><td>6.76</td><td>11.43</td><td>54.31</td><td>32.43</td><td>40.44</td><td>60.08</td><td>33.38</td><td>42.82</td><td>56.32</td><td>41.22</td><td><strong>47.59</strong></td><td>32</td></tr><tr><td>&lt;value&gt;</td><td>80.21</td><td>83.15</td><td>81.65</td><td>84.81</td><td>89.33</td><td>86.99</td><td>85.16</td><td>90.15</td><td><strong>87.58</strong></td><td>83.14</td><td>85.92</td><td>84.50</td><td>1895</td></tr><tr><td>&lt;variable&gt;</td><td>96.85</td><td>95.98</td><td>96.41</td><td>95.19</td><td>97.77</td><td>96.46</td><td>96.32</td><td>97.90</td><td><strong>97.10</strong></td><td>96.22</td><td>96.52</td><td>96.37</td><td>1795</td></tr><tr><td>All (micro avg)</td><td>81.15</td><td>78.09</td><td>79.59</td><td>82.76</td><td>83.50</td><td>83.13</td><td><strong>83.20</strong></td><td>84.33</td><td>83.76</td><td>83.11</td><td><strong>85.23</strong></td><td><strong>84.15</strong></td><td></td></tr></tbody></table>
<table><caption>Table 4.10: Evaluation scores for the Linking. Support (Supp) indicates the number of labels in the training data. Values in bold indicate the highest score. P: precision, R: recall.</caption><thead><tr><th>Relationship type</th><th>P</th><th>R</th><th>F1-</th><th>Supp</th></tr></thead><tbody><tr><td>material-tcValue</td><td>88.40</td><td>74.52</td><td>80.87</td><td>726</td></tr><tr><td>tcValue-pressure</td><td>85.71</td><td>71.52</td><td>77.98</td><td>118</td></tr><tr><td>me_method-tcValue</td><td>62.28</td><td>65.74</td><td>63.96</td><td>151</td></tr></tbody></table>

<table><caption>Table 4.11: Summary of the E2EE evaluation scores. Support indicates the number of labels in the training data.</caption><thead><tr><th>Subsection</th><th>Precision</th><th>Support</th></tr></thead><tbody><tr><td>Title</td><td>100</td><td>2</td></tr><tr><td>Abstract</td><td>80.32</td><td>61</td></tr><tr><td>Paragraph</td><td>75.2</td><td>623</td></tr><tr><td>Figure captions</td><td>59.28</td><td>140</td></tr><tr><td>Unknown</td><td>57.14</td><td>21</td></tr><tr><td>Micro avg.</td><td>72.60</td><td>847</td></tr><tr><td>Micro avg. (excl. figures)</td><td>75.24</td><td>707</td></tr><tr><td>Micro avg. (excl. unknown sections)</td><td>73.00</td><td>603</td></tr><tr><td>Micro avg. (excl. figures and unknown sections)</td><td>79.14</td><td>657</td></tr></tbody></table>
# Chapter 5

## SuperMat: Construction of a linked annotated dataset from superconductors-related publications

### 5.1 Introduction

In this contribution, we introduce SuperMat (Superconductor Materials), an annotated corpus of linked data derived from scientific publications on superconductors, which comprises 164 articles, 18698 entities, and 1524 links that are characterised into two layers. The first layer contains six categories: the names, classes, and properties of materials; links to their respective superconducting critical temperature ($T_c$); and parametric conditions such as applied pressure or measurement methods. The second layer is dedicated to material entities and contains eight classes: formula, name, doping ratio, etc. The construction of SuperMat resulted from a fruitful collaboration between computer scientists and material scientists, and its high quality is ensured through validation by domain experts. The quality of the annotation guidelines was ensured by a satisfactory Inter Annotator Agreement (IAA) between the different annotators. SuperMat includes the dataset, annotation guidelines, and annotation support tools that use automatic suggestions to help minimise human errors.
## 5.2 Content acquisition

SuperMat originates from PDF documents of scientific articles related to superconductor research. The original documents were collected from the following sources: (a) the Open Access (OA) version of peer-reviewed articles referenced in the SuperCon database records; (b) articles provided by domain experts containing suitable items and potential links of material names, $T_c$ values, measurement methods, and pressures; (c) articles from "condensed matter" category of arXiv (https://arxiv.org/archive/cond-mat) selected using the search terms of "superconductor", "critical temperature", and "superconductivity".

Pre-print versions of peer-reviewed articles were obtained using a bibliographic data lookup service called biblio-glutton (https://github.com/kermitt2/biblio-glutton) that aggregates data from various sources: the Crossref (https://www crossref. org/) bibliographic database, the unPaywall (http://unpaywall.org) service, the PubMed Central repository (https://pubmed.ncbi.nlm.nih.gov/), and mappings to other databases. We queried biblio-glutton using the bibliographic data of each article referenced in Supercon and subsequently downloaded the pre-print article associated with the retrieved record, if available. Although the published version may differ from the pre-print version of a document, the differences measured by comparing pre-print and peer-reviewed articles in biology [120] measured objective differences to be around 5%

## 5.3 Preliminary annotation study

A preliminary annotation study was carried out to assess the effort required from the annotators to reach an acceptable Inter Annotation Agreement (IAA >0.7). We annotated two randomly selected OA papers by using a preliminary version of the guidelines with a limited tag-set of four labels: `<material>`, `<tc>` (expression describing the presence or absence of superconductivity), `<tcValue>` (value of $T_c$), and `<doping>` (amounts of substitution, such as stoichiometric values, usually expressed as functions of x or y). The process was iterated multiple times. Each iteration ended with computing the IAA using Krippendorff's alpha coefficient [121, 122], while annotators discussed the disagreements and updated the guidelines.

Based on the results in Table 5.1, IAA reached a satisfactory level of around 0.9 after the third iteration. Although the average IAA reached 0.7 on three of the four labels in the second iteration, the average agreement was unsatisfactory. When analysing the disagreement, we noticed that the low score in the `<doping>` label was
<table><caption>Table 5.1: Summary of the IAA for each annotation iteration.</caption><thead><tr><th>Iteration #</th><th>IAA</th><th>IAA by label</th></tr></thead><tbody><tr><td rowspan="4">1</td><td rowspan="4">0.45</td><td>&lt;material&gt; 0.45</td></tr><tr><td>&lt;tc&gt; 0.56</td></tr><tr><td>&lt;tcValue&gt; 0.50</td></tr><tr><td>&lt;doping&gt; 0.21</td></tr><tr><td rowspan="4">2</td><td rowspan="4">0.65</td><td>&lt;material&gt; 0.75</td></tr><tr><td>&lt;tc&gt; 0.85</td></tr><tr><td>&lt;tcValue&gt; 0.85</td></tr><tr><td>&lt;doping&gt; 0.39</td></tr><tr><td rowspan="4">3</td><td rowspan="4">0.89</td><td>&lt;material&gt; 0.89</td></tr><tr><td>&lt;tc&gt; 0.91</td></tr><tr><td>&lt;tcValue&gt; 0.88</td></tr><tr><td>&lt;doping&gt; 0.94</td></tr></tbody></table>

caused by a heavy overlap with the <material> label, which required a more precise definition in the guidelines.

Based on this preliminary study, the following changes were implemented. (a) The label <doping> was merged under the <material> because, even with detailed documentation, it was too difficult for humans to annotate them consistently. (b) Three more labels were added: measurement methods and pressure (described as parametric conditions in relation to T_c) and class of materials.

## 5.4 Tag-set design

The tag set (also referred to as *labels*) represents the classes of entities and the type of relations between them, which were designed to be extracted from the text. There are two layers of tag sets: the first layer identifies entities in the text (Figure 5.1), and the second layer defines the parts within material expressions that characterise the different elements.

### 5.4.1 Top-level Entities

Entities (also referred to as Named Entities, mentions, or surface forms) are chunks of texts that represent information of interest, as follows:
<figure><img src="image_15.png" /><figcaption>Figure 5.1: Example in the annotated corpus. Excerpt from © 2009 The Physical Society of Japan (J. Phys. Soc. Jpn. 78, 123707)</figcaption></figure>
**Class** (tag: <class>) represents a group of materials defined by certain characteristics. Superconducting materials can be classified according to different criteria, such as composition and magnetic properties. Among publications collected for this study, the domain experts identified three types of classes based on: (a) the composition and crystal structure, (b) material phenomena (e.g. "I-type" and "II-type superconductivity", "BCS superconductors", "nematic", and "conventional/unconventional superconductivity"), and (c) high/low Tc value (e.g. "high-tc" superconductors).

In this work, we only considered the (a) classes, mainly because the material composition and crystal structure do not change with time. For example, a cuprate from 1998 is still called a cuprate today. In comparison, many material phenomena used for (b) are not robust enough and can be biased by the viewpoint of the author(s) or research group, or the measurement methods. Finally, the definition of "high-tc" superconductors (c) is completely relative; i.e., with the progress of research, materials once considered "high-tc" might not be so anymore.

**Material** (tag: <material>) identifies the name of one or more materials. This label is used to collect the following types of information:

* Chemical formula indicating the material by its general or non-stochiometric formula (e.g. LaFe<sub>1-x</sub>O<sub>7</sub>, WB<sub>2</sub>),

* Compositional name (e.g. magnesium diboride) or abbreviations (e.g. YBCO),

* The material's shape (e.g. wire, powder, thin film) or form of material (e.g. single/polycrystal),

* Modification by a dopant (Zn-doped, Si-doped) or by percentage of doping (2%-doped). We also considered qualitative expressions such as *overdoped*, *lightly doped*, and *pure* as valid information,

* Substrate information (e.g. grown on MgO(100) film) when it was adjacent to the material name or formula in the text,

* Additional information about the sample when it was adjacent to the material name or formula in the text (e.g. as-grown, untwinned, single-layer).

**Superconducting critical temperature** (tag: <tc>) identifies expressions related to the phenomenon of superconductivity. Any temperature mentioned in the text is not necessarily the T<sub>c</sub>. Rather, it could refer to the temperature for other processes/events, such as annealing/sintering temperature, specific measurements, and
structural changes. This label identifies the presence or absence of superconductivity at a given temperature. In addition, modifiers of this information (increasing/decreasing $T_c$) are also retained.

**Superconducting critical temperature value** (tag: `<tcValue》`) represents the temperature at which the superconducting phenomenon occurs. It can be defined by different experimental criteria, such as the onset, mid-point of resistivity drop, or zero resistivity. This value also considers boundary conditions, such as the *onset of superconductivity* and *zero resistance*.

**Applied pressure** (tag: `<pressure》`) indicates the applied pressure corresponding to a measured $T_c$.

**Measurement method** (tag: `<me_method》`) indicates the method used to measure or calculate the presence of superconductivity. Here, we considered the following categories: resistivity, magnetic susceptibility, specific heat, and theoretical calculations.

### 5.4.2 Level-2 entities

Extracted materials entities are then segmented according to the Level-2 tagset, illustrated below. The examples are shown as text entities, without font variations such as superscript or subscript.

**Material name** (tag: `<name》`) represent the canonical name of a material. Examples: PCCO, PCO, metal diboride, oxygen, carbon

**Formula** (tag: `<formula》`) identify the material as expressed from the chemical formula. Examples: Pr1.869Ce0.131Cu0 4-δ, MgB2, La 2-x Sr x CuO 4

**Doping** (tag: `<doping》`) identify the doping ratio and doping materials that are adjoined to the material name. Examples: overdoped, underdoped, optimally doped, bulk, pure, Zn-doped, Zn concentration, 1% Zn
**Shape** `<shape>` Identify the shape of the material. Examples: single crystal, polycrystalline, thin film, powder, films

**Variables** `<variable>` identify the variables that can be substituted in the formula. Examples:

* La x Fe 1-x 0.7 with x < 3, variable: x

* RE x Fe 1-x 0.7 with RE=La,Fe, and 1 < x < 3, variable: x, RE

* La x Fe y 0.7 with x = 1,2,3, and y = 2,3,4, variable: x, y

**Values** (tag: `<value>`) identify the values expressed in the stoichiometric doping.
Examples:

* La x Fe 1-x 0.7 with x < 3, value: <3

* La x Fe 1-x 0.7 with 1 < x < 3, value: 1<, <3

* La x Fe 1-x 0.7 with x = 1,2,3, and 4, value: 1, 2, 3, and 4

**Substrate** : (tag: `<substrate>`) identify the substrates as defined in the material name. Examples: PCCO films onto Pr 2 CuO 4 (PCO)/SrTiO 3, substrate: Pr 2 CuO 4 (PCO)/SrTiO 3

**Fabrication** (tag: `<fabrication>`) represents all the various information that does not belong to any of the previous tags. Examples: cointercalated, intercalated, synthesized by MBE method, electron-doped, hole-doped

### 5.4.3 Relations

The relations connect entities of materials or samples to their corresponding properties, conditions, and results. The connecting links are non-directional, and each entity has no restrictions on the number of links. The relation types have been introduced in Section 4.2.6.
#### 5.4.4 Annotation guidelines

Annotation guidelines include the principles and the rules that describe what constitutes all desired information for the SuperMat dataset and how to annotate it. They include detailed descriptions of the specific rules that have been defined for each type of information to be annotated, with one or more definitions and examples illustrating what to annotate in different cases, exceptions, and references. We used an online system to track the discussions and decisions when a question or a comment was raised and provided a link to such issues in the respective description or example. In addition, the guidelines include *linking rules* that provide information on how to connect the entities in a relationship correctly. The guidelines were built using a dynamic markup language (called RestructuredText) and stored in a git (https://git-scm.com/) version control system repository. We deployed them as HTML files via the web, which were updated automatically after each modification. They can be accessed at https://supermat.readthedocs.io.

## 5.5 Annotation support tools

The task of annotating documents is tedious and requires both attention and subject knowledge from the annotators. Annotation support tools aim to maximise the efficiency of annotators and minimise human mistakes. They comprise a web-based collaborative annotation tool, automatic annotation suggestions, and automatic corpus analysis.

### 5.5.1 Web-based collaborative annotation tool: INCEpTION

The annotation tool is the platform for creating, correcting and linking annotations. After evaluating several tools, we selected INCEpTION [123,124], a web-based multi-user platform for machine-assisted rapid dataset annotation construction. INCEpTION provides supportive functionalities that include:

* Multi-layer annotation sheets allow different annotation schemas over the same documents,

* Two annotation steps: annotation consists of manually correcting pre-imported documents, while curation allows another user to validate the annotations (Figure 5.5).
* On-the-fly automatic suggestions based on active learning and string matching (Figure 5.5),
* Bulk annotation corrections, and
* Being open-source (Apache 2.0 license), and under active development at the time of this paper (https://inception-project.github.io/).

### 5.5.2 Annotation suggestions

Previous works have demonstrated that annotation suggestions improve the quality of the output [125–127]. We provide two types of annotation suggestions. (i) *Machine-based annotated data* that were assigned to the documents before loading into the annotation tool. Here, we use a machine learning (ML)-based system from a previously implemented prototype [12] to support our tag-set. (ii). *Active learning recommendations* provided by INCEPSION are assigned on the fly based on previous annotations. The active-learning recommendations are less precise since they aim to increase the recall, and therefore they need to be explicitly accepted by the annotator.

### 5.5.3 Automatic corpus analysis

Automatic corpus analysis is a set of scripts designed to run after the validation step. These scripts automatically find inconsistencies in the links and entities while extracting the statistics of the corpus. We calculated the inconsistencies by examining every annotated entity and computing the frequency of the same text being annotated with different labels. The script outputs a summary table by visualising each annotation value and its labels and frequencies. We visually inspected this table because the reported inconsistencies can be either obvious mistakes (Table 5.4) or arise from ambiguities (Table 5.3); therefore, their context should be verified.

Although the links are conceptually non-directed, we have defined a practical convention to maintain consistency. For example, *material-tcValue* is always represented as a link between `<tcValue>` and `<material>` entities. The script also computes the statistics (Table 5.2) for the number of entities (total, unique, by class), the number of links (total, intra- and inter-paragraph, between paragraphs), and other statistical information.
<table><caption>Table 5.2: Statistical overview of the dataset. Level-2 entities are considered only for materials Relations<sub>is</sub> indicates the number of relations within the same sentence (intra-sentence). Relations<sub>es</sub> indicate the number of relations from different paragraphs (extra-sentence).</caption><thead><tr><th></th><th>Files</th><th>Paragraphs</th><th>Sentences</th><th>Tokens</th></tr></thead><tbody><tr><th>Documents</th><td>164</td><td>2800</td><td>20807</td><td>1284569</td></tr><tr><th>Level-1<br>Entities</th><td>18698</td><td>Unique entities</td><td>8071</td><td>6</td></tr><tr><th>Relations</th><td>Relations<br>1524</td><td>Relations<sub>is</sub><br>985</td><td></td><td>Relations<sub>es</sub><br>539</td></tr><tr><th>Level-2<br>Entities</th><td>Examples<br>9268</td><td>Entities<br>13648</td><td>Unique entities<br>4512</td><td>Labels<br>8</td></tr></tbody></table>

<table><caption>Table 5.3: Inconsistencies resulting from human mistakes.</caption><thead><tr><th>Text</th><th>Label 1</th><th>#</th><th>Label 2</th><th>#</th></tr></thead><tbody><tr><td>superconducting transition</td><td>&lt;material&gt;</td><td>1</td><td>&lt;tc&gt;</td><td>61</td></tr><tr><td>NCCO</td><td>&lt;material&gt;</td><td>14</td><td>&lt;tc&gt;</td><td>1</td></tr><tr><td>superconducting transition temperatures</td><td>&lt;material&gt;</td><td>1</td><td>&lt;tc&gt;</td><td>11</td></tr><tr><td>occurrence of superconductivity</td><td>&lt;material&gt;</td><td>1</td><td>&lt;tc&gt;</td><td>1</td></tr></tbody></table>

<table><caption>Table 5.4: Inconsistencies resulting from the overlapping of &lt;material&gt; and &lt;class&gt; labels.</caption><thead><tr><th>Text</th><th>Label 1</th><th>#</th><th>Label 2</th><th>#</th></tr></thead><tbody><tr><td>LiFeAs</td><td>&lt;material&gt;</td><td>89</td><td>&lt;class&gt;</td><td>1</td></tr><tr><td>Bi-2212</td><td>&lt;material&gt;</td><td>34</td><td>&lt;class&gt;</td><td>1</td></tr><tr><td>cobalt oxide</td><td>&lt;material&gt;</td><td>89</td><td>&lt;class&gt;</td><td>1</td></tr><tr><td>RE-123</td><td>&lt;material&gt;</td><td>34</td><td>&lt;class&gt;</td><td>1</td></tr></tbody></table>
## 5.6 Annotation process

The annotation workflow (Figure 5.2) was designed following the *MATTER* (Model, Annotate, Train, Test, Evaluate, and Revise) schema [128] and other related work [129, 130]. The workflow is composed of five steps (Figure 5.2): *data-preparation, correction, validation, testing and evaluation, revision*. This workflow involves three main actors: the automatic process, computer scientists, and the domain experts.

<figure><img src="image_16.png" /><figcaption>Figure 5.2: Annotation workflow. Different colours illustrate the involvement of each group at each step of the workflow.</figcaption></figure>

The first step of the annotation process involves preparing the machine-based annotated data from the source PDF documents. The PDF files are converted to an XML-based format, and annotation is automatically applied, followed by four more steps:

* Annotation: The human annotator can select a document and manually add, remove, or modify each entity based on the rules defined in the guidelines. Once the annotation is complete, the document is marked "ready" for validation.

* Validation by domain experts: Annotations from different users are validated and merged into a final document (Figure 5.5). The domain expert ("curator") can compare the different annotated versions, select the best annotation combinations, or add new ones. This step ensures that the annotations are cross-checked and domain experts validate the document.
* Automatic consistency checks and statistical analysis: This step aims to discover obvious mistakes such as mislabelling or incorrect linking. A sequence labelling model is trained and evaluated using 10-fold cross-validation. The evaluation provides precision, recall, and f-score metrics for all the labels. The resulting model produces machine-based annotated data in the following iteration.

* Review: Retrospective analysis of the past iteration, where unclear cases are discussed and documented in the annotation guidelines.

## 5.7 Data transformation

There are two processes of data transformation (Figure 5.3): (a) from the source document (PDF) to the dataset format representation (XML-based), and (b) from the dataset format representation to the annotation tool exchange formats (https://inception-project.github.io/releases/0.16.1/docs/user-guide.html#sect_format) and vice-versa.

<figure><img src="image_17.png" /><figcaption>Figure 5.3: Summary of the data transformation flows.</figcaption></figure>
* **PDF to XML-based**: This step converts the PDF source document to the dataset format representation in XML following the Text Encoding Initiative (TEI, https://tei-c.org/) format guidelines. Such transformation is performed by Grobid as described in Chapter 3. The resulting structured document is then encoded in XML as described below.

* **XML to the annotation tool exchange formats**: We transform our XML-formatted data into an INCEP TIONS compatible import format, such as the Webanno TSV 3.2 (https://inception-project.github.io/releases/0.17.0/docs/user-guide.html#sect formats webannotsv3), and vice-versa using a set of Python scripts. The Webanno TSV 3.2 format is an extension of the CONLL (https://www.signll.org/conll/) format, with additions to the header and column representation.

## 5.8 Data Record

The dataset is composed of 164 PDF documents. Although we used the pre-print version of most of the articles to comply with copyright restrictions, the dataset cannot be distributed and is not publicly available in our repository. The three leading publishers represented in the corpus are the American Physical Society (APS), Elsevier, and IOP Publishing (Figure 5.4a). Figure 5.4b illustrates the distribution by publication date. We summarise SuperMat’s content in Table 5.2, with the statistics of documents, entities, and links given separately. In particular, this dataset contains 16052 (7166 unique) entities spread over six labels and 1398 links.

Each document is encoded according to the XML TEI guidelines, a rich format for document representation. We have carried out no specific customisation to remain fully compliant with the general TEI schema. A TEI document has two main parts: the header (within the `<teiHeader>` tags) containing all the document metadata and the body (within the section delimited by the `<text>` tag). The transformed data has the following structure:

```xml
<TEI xml:lang="en" xmlns="http://www.tei-c.org/ns/1.0">
  <teiHeader>
    <fileDesc>
      <titleStmt>
        <title>[...]</title>
      </titleStmt>
      <publicationStmt>
        <publisher>[...]</publisher>
      </publisher>
    </fileDesc>
  </teiHeader>
</TEI>
```
<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /><figcaption>Figure 5.4: Distribution of paper in the dataset by (a) publisher, and (b) year of publication.</figcaption></figure>
```xml
</publicationStmt>
</fileDesc>
<encodingDesc/>
<abstract>
<p>[...]</p>
<ab type="keywords">[...]</ab>
</abstract>
<profileDesc>
</profileDesc>
</teiHeader>
<text>
<body>
<p>[...]</p>
<ab type="tableCaption">[...]</ab>
<p>[...]</p>
<ab type="figureCaption">[...]</ab>
</body>
</text>
</TEI>
```

We transformed the source documents into these TEI-compliant structures using a simplified representation for specific content types. The general objective is to flatten the content into a generic structure where priority is given to the annotations. For instance, the keywords section, which groups together the key terms defined by the author(s) of the paper, is encoded using the generic tag `<ab type="keywords">` as free text instead of the dedicated `<keywords>` element that would typically be part of the header. For both the abstract and the article body, the text is segmented in paragraphs (by means of the `<p>` element). The text is annotated with the generic `<rs>` (referencing string) element adorned with three attributes: `@type` (the entity type), `@corresp` (to provide a link to another annotation such as from material to T<sub>c</sub>), and `@xml:id` (to uniquely identify the annotation for referencing or RE purposes).

Because only the captions of tables and figures are retained from the original source, a simplified encoding was defined by means of the `<ab>` element characterised by a `@type` attribute; that is, `<ab type="figureCaption">` for figure captions and `<ab type="tableCaption">` for table captions. Here is an example:

```xml
<p>
  The electron-doped high-<rs type="tc">transition-
  temperature</rs> (<rs type="tc">Tc</rs>) <rs
  type="class">iron-based pnictide</rs>
</p>
```
superconductor <rs type="material"
xml:id="m6">LaFeAsO1-xHx</rs> has a unique
phase diagram: Superconducting (SC) double domes are
sandwiched by antiferromagnetic phases at ambient
pressure and they turn into a single dome with
a maximum <rs type="tc">Tc</rs> that
<rs type="tcValue" xml:id="m7"
corresp="#m6,#9">exceeds 45K</rs>
at a pressure of <rs type="pressure"
corresp="#m7">3.0 GPa</rs>.
[...]
</p>

In the above snippet, the entities "3.0 GPa", "exceed 45K" and "LaFeAsO1-xHx" are linked together via the pairs @corresp, @xml:id. This schema supports multiple annotations to any part of the document. For example, the entity exceed 45K has a second link with the corresponding identifier (#9) to an annotation outside this paragraph.

5.9 Practical applications

The dataset was used in the following practical applications:

* Evaluation tasks: SuperMat has been used for evaluation of Large Language Models (LLM) in mining experimental data. In particular, NER and RE were tested and applied to the materials science literature [25]. The dataset not being publicly available counted in a lower chance of being used for pre-training the LLM.

* Automatic extraction: As discussed in Chapter 4, we used this dataset to develop a system for information extraction for superconducting materials.

* Weighted clustering was implemented using SuperMat annotations to identify research papers that discuss specific information of interest in different categories of superconducting materials. The annotations were used to bias the clustering algorithm towards entity similarity to achieve a more targeted clustering toward a specific type of information [26].
## 5.10 Technical Validation

The following measures were employed to ensure the creation of a high-quality dataset:

* Each document was revised and validated by domain experts,

* The workflow begins by assigning machine-based annotated data. This has improved the annotation task over several aspects: time consumption, error rate, and annotation agreement [125–127].

* On-the-fly automatic annotation recommendations, which provide fresh suggestions based on online decisions made by the annotators.

* The annotators have rapid access to changes in the annotation guidelines.

* The discussions were documented and linked in the guidelines.

* Reviews are discussed and approved collaboratively between domain experts and other annotators.

<table><caption>Table 5.5: Average IAA between the annotated and validated documents</caption><thead><tr><th>Label</th><th>Average</th></tr></thead><tbody><tr><td>&lt;material&gt;</td><td>0.956</td></tr><tr><td>&lt;me_method&gt;</td><td>0.887</td></tr><tr><td>&lt;pressure&gt;</td><td>0.723</td></tr><tr><td>&lt;class&gt;</td><td>0.925</td></tr><tr><td>&lt;tcValue&gt;</td><td>0.863</td></tr><tr><td>&lt;tc&gt;</td><td>0.831</td></tr><tr><td>Micro avg.</td><td>0.911</td></tr></tbody></table>

These guidelines are a vital piece of this work since they contain knowledge accumulated from these activities. However, measuring the completeness of the guidelines is challenging. Assuming that the documents validated by domain experts represent the ground truth, we conducted IAA analysis between different annotators against the ground truth, using the Krippendorf's Alpha metric [121]. Table 5.5 shows the average IAA which is satisfying with a value of approximately 0.9. The highest score is obtained in the &lt;material&gt; entities, while the lowest one is obtained in &lt;pressure&gt;, which appears less frequently in the papers. The disagreement in &lt;tcValue&gt; can appear to be too low as compared with other labels such as
<class>, which is, at first look, more ambiguous. We analysed the different cases and identified three reasons why this happens. First, <tcValue> may depend heavily on the context that requires more human attention, and it is therefore more prone to errors. Second, our suggestions system is challenged in its ability to disambiguate critical temperatures from other temperature data, leading to incorrect or invalid suggestions. Finally, the presence of mathematical symbols (e.g. "~", "<", and ">") or other modifiers ("up to", "exceeds", etc.) before the <tcValue> could generate small disagreements that accumulate in the average score.

<table><caption>Table 5.6: Calculated IAA for annotations produced by domain experts, non-domain experts, and novices compared to the validated version. Annotations from domain experts are cross validated.</caption><thead><tr><th>Label</th><th>Domain experts</th><th>Non-domain experts</th><th>Novices</th></tr></thead><tbody><tr><td>&lt;material&gt;</td><td>0.969</td><td>0.950</td><td>0.924</td></tr><tr><td>&lt;me_method&gt;</td><td>0.890</td><td>0.862</td><td>0.901</td></tr><tr><td>&lt;pressure&gt;</td><td>0.836</td><td>0.741</td><td>0.746</td></tr><tr><td>&lt;class&gt;</td><td>0.990</td><td>0.836</td><td>0.899</td></tr><tr><td>&lt;tcValue&gt;</td><td>0.895</td><td>0.734</td><td>0.841</td></tr><tr><td>&lt;tc&gt;</td><td>0.874</td><td>0.776</td><td>0.830</td></tr><tr><td>All labels</td><td>0.940</td><td>0.882</td><td>0.896</td></tr><tr><td># paragraphs</td><td>1066</td><td>1648</td><td>325</td></tr></tbody></table>

To more precisely isolate the impact of the guidelines, we grouped the IAA results by level of domain experience. Table 5.6 displays the IAA between the validated data and the data corrected by (a) domain experts (researchers who conduct superconducting development experiments), (b) non-domain-experts (researchers with no experience with superconducting materials), and (c) novices (students in materials science with limited domain experience). Obviously, the domain experts have the highest agreement and the IAA value (around 0.95) is 0.06 higher on average than that of non-domain experts. Thus, superconducting materials is a complex domain that requires knowledge in materials science to produce high-quality data, while crowdsourcing initiatives such as the Amazon Mechanical Turk might not work well.

Furthermore, we measured the reliability of the guidelines by observing how quickly novices could reach a satisfying agreement with the validation of the domain experts, without any previous training on the guidelines. From Table 5.6, the novices can attain high IAA results by only using the guidelines and our annotation support tools. The average difference in agreement with domain experts (around 0.05) indicates that the guidelines are precise and complete, and that the annotations tools
offer sufficient support.

## 5.11 Data Availability

The references of the original papers and their bibliographical information, the annotation guidelines sources, and the developed code are available at the GitHub repository https://github.com/lfoppiano/SuperMat. The guidelines are freely accessible at https://supermat.readthedocs.io. The data transformation scripts were written in Python and can be run from the command line. They require Beautiful-Soup (https://www.crummy.com/software/BeautifulSoup/), an open-source library for parsing XML and HTML formats. The data analysis scripts were developed as Jupyter notebooks (https://jupyter.org/), which can easily output results and graphs in the browser. The open-source annotation tool is INCEpTION (https://inception-project.github.io/). The OA content was acquired using biblio-glutton (https://www.github.com/kermitt2/biblio-glutton). We computed the IAA using the Java library DkPro statistics (https://dkpro.github.io/dkpro-statistics/) [131].

<figure><img src="image_20.png" /><figcaption>Figure 5.5: INCEpTION curation interface. Excerpt from © 2004 The Physical Society of Japan (J. Phys. Soc. Jpn. 73, 1655-1656)</figcaption></figure>
5.12 Conclusions

This chapter presents the process of creating an annotated linked dataset for scien-
tific publications on the development of superconductors. SuperMat aims to create
a reliable infrastructure for improving text and data mining processes in the field of
superconductor materials. We annotated 164 full-text articles using a two-layer set
of annotations. The first layer contains six categories: the names, classes, and prop-
erties of materials; links to their respective superconducting critical temperature
(T_c); and parametric conditions such as applied pressure or measurement methods.
The second layer is dedicated to material entities and contains eight classes: for-
mula, name, doping ratio, etc. Experts in the field have validated the dataset, which
consists of 18698 top-layer entities, with 1524 links between materials, properties,
and conditions. This dataset is the building block for the contribution to material
extraction (Chapter 4) presented in this dissertation.

This contribution was published in the article "SuperMat: Construction of a
linked annotated dataset from superconductors-related publications" [4].
Chapter 6

Extraction of properties and
conditions as measurements of
physical quantities

6.1 Introduction

In our approach to extracting material properties and conditions, we have opted for diverse components to accommodate the intricacies of different domains. For instance, we delve into critical temperature (Tc), applied pressure, and other pertinent factors in handling superconductor articles. Similarly, we focus on properties like coercivity and remanence when dealing with magnetic materials. This tailored approach is essential to capture domain-specific nuances effectively. However, we also recognise the necessity for a more generalised method alongside this specialised handling. Here, we prioritise a generic approach centred on extracting properties expressed through measurable physical quantities. This dual strategy ensures versatility in our material analysis framework while maintaining precision in capturing specific material behaviours across various domains.

For this task, we developed a Grobid module called Grobid-quantities that aims to provide a general solution to extracting quantities and physical measurements from the scientific literature. Grobid-quantities has been used in practical applications in other domains such as Earth observation data analysis [84] It combines ML-based techniques and lexicon-based memory that aims to cover most of the existing units, unlike other work [76]. Most other approaches lack the generalisation to an extensive corpus or deal mainly with specific languages. [78] addressed identi-
fying the numeric properties of patents using GATE (General Architecture for Text Engineering). [132] investigated issues applied to Russian-derived languages. [80] described an attempt to recognise units by looking up terms from an ontology, using ML combined with pattern matching and string metrics. At the time of development, other ML-based approaches were limited to certain fields [83] or domain [82]. We incorporate a normalisation process to convert measurements to the base units of the international system (SI), a crucial step in harmonising data from various documents, disciplines and domains.

## 6.2 Proposed solution

The system accepts input in text or PDF files similar to the approach described in Chapter 3. The extracted content is processed in three steps: (a) tokenisation, (b) extraction and parsing of measurements, and (c) normalisation of quantities.

### 6.2.1 Data model

The data model (Figure 6.1) was based on the concept of *Measurement*, which links an object or a substance with one or more *quantities*. We defined four *Measurements* types: (a) atomic, in case of a single measurement (e.g., 10 grams). (b) interval (*from 3 to 5 km*) and (c) range (100±4 mm) for continuous values, and (d) a list of discrete values. A *Quantity* links the quantitative value and the unit.

Since data extracted from PDF documents unavoidably present irregular tokens from incorrect UTF-8 encoding or missing fonts, we designed this model to allow partial results. The *Value* and *Unit* entities allow three different representations (Figure 6.1): *Raw* as appear in the input, ** unifies the value into the numerical expression, and the unit with its properties (system, type). Finally, *Normalised* contains the transformed unit and values to the SI system. *Value* object supports four types of representations: numeric (2, 1000), alphabetic (two, thousand), scientific notation (3·10⁵), and time, which is also an expression of measurement. Units objects are organised following the SI, which allows representing units as products of simpler compounds (e.g. m/s to m·s⁻¹) further decomposed as triples (prefix, base and power).
<figure><img src="image_21.png" /><figcaption>Figure 6.1: Schema of the data model, from the data parsing and normalisation point of view.</figcaption></figure>

### 6.2.2 Tokenisation

This process splits the input data into tokens. Grobid-quantities uses two-phase tokenisation: (1) first, it splits by punctuation marks, then (2) each resulting token is re-tokenised to separate adjacent digits and alphanumeric characters. Given the example 25m^2, first returns a list [25m, ^, 2] and then recursively divides 25m as [25, m] resulting in [25, m, ^, 2].

### 6.2.3 Extraction

The tokens are passed through three ML models following the 2-level approach (Section 4.2.4): first, the *Quantities* parser determines the appropriate unit and value tags. The results are further processed by the respective *Units* and *Values* parsers as illustrated in Figure 6.2.

Table 6.1 describes the labels predicted by *Quantities* parser. Note that to reconstruct complex structured objects from the flat sequence generated by the engine, additional labels are necessary (such as `<unitLeft>`, `<unitRight>`, for units).

Previous work presented extensive use of databases or ontologies. In our solution, we used a similar approach. We created a list of units (in English, French, and German) with their characteristics: system (SI base, SI derived, imperial, etc.) and
<figure><img src="image_22.png" /><figcaption>Figure 6.2: The cascade approach of the applied parsers. The Quantities parser recognises values and units passed to Values and Units parsers for further extraction.</figcaption></figure>

<table><caption>Table 6.1: Labels description for the Quantities parser. The values referred to by the label are highlighted in bold.</caption><thead><tr><th>Label</th><th>Description</th><th>Example</th></tr></thead><tbody><tr><td>&lt;valueAtomic&gt;</td><td>value of an atomic quantity</td><td>2 m</td></tr><tr><td>&lt;valueLeast&gt;</td><td>least value in an interval</td><td>from 2 m</td></tr><tr><td>&lt;valueMost&gt;</td><td>max value in an interval</td><td>up to 7 m</td></tr><tr><td>&lt;valueBase&gt;</td><td>base value in a range</td><td>20 ± 7 m</td></tr><tr><td>&lt;valueRange&gt;</td><td>range value in a range</td><td>20 ± 7 m</td></tr><tr><td>&lt;valueList&gt;</td><td>list of quantities</td><td>2, 3 and 10 m</td></tr><tr><td>&lt;unitLeft&gt;</td><td>left-attached unit</td><td>pH 2</td></tr><tr><td>&lt;unitRight&gt;</td><td>right-attached unit</td><td>2 m</td></tr><tr><td>&lt;other&gt;</td><td>everything else</td><td>-</td></tr></tbody></table>
type (volume, length, etc.) and their representations: notations ($m^3$, $m^{-3}$), lemmas (cubic meter, cubic metre) and inflexions (cubic meters, cubic metres). We made this list available through the *Unit Lexicon*, which offers unit lookups by properties (such as notation, lemma, and inflexion). A second gazetteer was created to allow the transformation of alphabetic values into numeric ones (for example, 21 to 21).

Features in the *Quantities* model are generated from preceding and following tokens, presence of capital, and digits. Orthogonal features are obtained through the *Unit Lexicon*, like a *Boolean* indicating whether a token is a known unit. Typographical information (format, fonts, subscript and superscript) are ignored.

The *Units* parser works at the character level and uses the *Unit Lexicon* to highlight known units or prefixes. The input tokens are parsed and transformed into a product of triples (prefix, base, power) as shown in Table 6.2. For example, Kg / mm², corresponds to Kg · mm⁻² and becomes [(K, g, 1), (m, m, -2)] as a product of triples. We define "simple units" as all the units that can be decomposed in a single triplet and complex units when they are decomposed in multiple triplets.

<table><caption>Table 6.2: Labels description for the Units parser. In bold are highlighted specific examples.</caption><thead><tr><td>Label</td><td>Description</td><td>Example</td></tr></thead><tbody><tr><td>&lt;prefix&gt;</td><td>prefix of the unit</td><td>km<sup>2</sup></td></tr><tr><td>&lt;base&gt;</td><td>unit base</td><td>km<sup>2</sup></td></tr><tr><td>&lt;pow&gt;</td><td>unit power</td><td>km<sup>2</sup></td></tr><tr><td>&lt;other&gt;</td><td>everything else</td><td>-</td></tr></tbody></table>

We then use the structured triples to fetch the corresponding information (system, type) from the *Unit Lexicon* and attach them to the resulting object. This implementation processes the unit characters in right-to-left order. Priority modifiers, such as parentheses, are ignored. They are generally not frequent in unit expressions and require a more complex logic.

In parallel, the *Values* parser unifies the format of the identified values into numerical formats. It supports four types: numeric, alphabetic, scientific notation, and time expression (see Table 6.3). Different techniques are applied for each type: alphabetic expressions are looked up in the word-to-number gazetteer, and scientific notation is parsed and calculated mathematically. Time expressions are further segmented using the Grobid built-in "Date" model.
<table><caption>Table 6.3: Labels description for the Values parser. In bold are highlighted specific examples.</caption><thead><tr><th>Label</th><th>Description</th><th>Example</th></tr></thead><tbody><tr><td>&lt;number&gt;</td><td>numeric value / coefficient</td><td>2.5 &middot; 10<sup>5</sup></td></tr><tr><td>&lt;alpha&gt;</td><td>alphabetic value</td><td>twenty</td></tr><tr><td>&lt;time&gt;</td><td>time expression</td><td>in 1970-01-02</td></tr><tr><td>&lt;base&gt;</td><td>base in scientific notation</td><td>2.5 &middot; 10<sup>5</sup></td></tr><tr><td>&lt;pow&gt;</td><td>exponent in scientific notation</td><td>2.5 &middot; 10<sup>5</sup></td></tr><tr><td>&lt;other&gt;</td><td>everything else</td><td>-</td></tr></tbody></table>

### 6.2.4 Normalisation

The measurements extracted are transformed to the base SI unit (grams to kg, Celsius to Kelvin, etc.). We used an external Java library called Units of Measurement [133], which provides a set of standard interfaces and implementations to handle units and quantities safely. Manipulating measurements with transformations often leads to common errors due to wrong rounding and approximations.

## 6.3 Evaluation and results

We trained and evaluated our system's models using a dataset based on 32 scientific publications (English, Open Access (OA)) and three patents (with translation in English, French, and German) randomly selected from different domains such as medicine, robotics, astronomy, and physiology. Training data was generated automatically and then corrected and cross-checked by three annotators. The dataset was then divided into a training set (26 documents) and an evaluation set (9 documents). We used the holdout dataset partition to evaluate each model and produce precision, recall, and f1 scores, as summarised in Tables 6.4, 6.5, and 6.6. As presented in Chapter 4, we report the results of the three architectures introduced in Section 4.2.3: CRF, RNN (with and without layout features), and transformers, fine-tuning a pre-trained SciBERT [67].

The best results of the Quantities ML models were achieved by the SciBERT model, for which we recorded an F1 micro-average of 85.14% with precision and recall of 86.24% and 83.96%, respectively. SciBERT outperforms the CRF model by 8% F1-score. We confirmed the same observation reported in Chapter 4, that layout orthogonal features do not contribute to helping the model to generalise better. In
<table><caption>Table 6.4: Evaluation scores (%) of the Quantities ML model with holdout set. Support (Supp) indicates the number of labels in the training data. Values in bold indicate the highest score. P: precision, R: recall. Deep learning results are averaged over five independent runs of training and evaluation.</caption><thead><tr><th rowspan="2">Label</th><th colspan="3">CRF</th><th colspan="3">BidLSTM.CRF</th><th colspan="3">BidLSTM.CRF FEATURES</th><th colspan="3">SciBERT</th><th rowspan="2">Supp</th></tr><tr><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th></tr></thead><tbody><tr><td>&lt;unitLeft&gt;</td><td>88.74</td><td>83.19</td><td>85.87</td><td>88.56</td><td>92.07</td><td>90.28</td><td>88.91</td><td><b>92.20</b></td><td>90.53</td><td><b>93.99</b></td><td>90.30</td><td><b>92.11</b></td><td>464</td></tr><tr><td>&lt;unitRight&gt;</td><td><b>30.77</b></td><td>30.77</td><td>30.77</td><td>24.75</td><td><b>30.77</b></td><td>27.42</td><td>21.73</td><td>30.77</td><td>25.41</td><td>21.84</td><td><b>36.92</b></td><td><b>27.44</b></td><td>13</td></tr><tr><td>&lt;valueAtomic&gt;</td><td>76.29</td><td>78.66</td><td>77.46</td><td>78.14</td><td>86.06</td><td>81.90</td><td>78.21</td><td>86.20</td><td>82.01</td><td><b>84.50</b></td><td><b>88.19</b></td><td><b>86.31</b></td><td>581</td></tr><tr><td>&lt;valueBase&gt;</td><td>84.62</td><td>62.86</td><td>72.13</td><td>83.51</td><td>94.86</td><td>88.61</td><td>83.36</td><td><b>97.14</b></td><td>89.72</td><td><b>100.00</b></td><td>90.86</td><td><b>95.20</b></td><td>35</td></tr><tr><td>&lt;valueLeast&gt;</td><td>77.68</td><td>69.05</td><td>73.11</td><td><b>82.14</b></td><td>60.63</td><td>69.67</td><td>80.73</td><td>60.63</td><td>69.12</td><td>81.09</td><td><b>71.59</b></td><td><b>76.04</b></td><td>126</td></tr><tr><td>&lt;valueList&gt;</td><td>45.45</td><td>18.87</td><td>26.67</td><td>62.15</td><td>10.19</td><td>17.34</td><td><b>73.33</b></td><td>8.68</td><td>15.33</td><td>64.12</td><td><b>43.78</b></td><td><b>51.64</b></td><td>53</td></tr><tr><td>&lt;valueMost&gt;</td><td>71.62</td><td>54.64</td><td>61.99</td><td>77.64</td><td>68.25</td><td>72.61</td><td>77.25</td><td><b>70.31</b></td><td>73.58</td><td><b>81.52</b></td><td>67.42</td><td><b>73.71</b></td><td>97</td></tr><tr><td>&lt;valueRange&gt;</td><td><b>100.00</b></td><td>97.14</td><td>98.55</td><td>96.72</td><td><b>100.00</b></td><td><b>98.32</b></td><td>94.05</td><td>98.86</td><td>96.38</td><td>99.39</td><td>91.43</td><td>95.24</td><td>35</td></tr><tr><td>All (micro avg.)</td><td>80.08</td><td>75</td><td><b>77.45</b></td><td>81.81</td><td>81.73</td><td>81.76</td><td>81.76</td><td>81.94</td><td>81.85</td><td><b>86.24</b></td><td><b>83.96</b></td><td><b>85.08</b></td><td></td></tr></tbody></table>

general, SciBERT performs better than any other model in most labels. The CRF model better recognises only the label `<valueRange>`. The support information also suggests that `<list>` and `<unitRight>` require more training examples.

Unit expressions are generally short (1-3 characters) and have lower variability, meaning each label tends to have more duplicates than in the training datasets of other models. For example, the expressions 1% and 2% have two different values (1, 2) and the same unit (%), which would appear twice. For this reason, we evaluated the unit ML model with a dataset of 2000 annotated units called UniSCor (Units Segmentation Corpus) [134]. After annotating each unit following the schema discussed in Table 6.2, the resulting corpus contains approximately 700 simple and 1300 complex units. As illustrated in Table 6.5) the overall scores for the Units ML model are lower because the evaluation corpus is larger than the training corpus and contains a substantial percentage of complex units, while in the training dataset, the complex units are less frequent. Table 6.5 shows that features impact positively, and both architectures using them (BidLSTM_CRF FEATURES and the CRF) obtain the best scores.

<table><caption>Table 6.5: Evaluation scores (%) of the Units ML model with holdout set. Support (Supp) indicates the number of labels in the training data. Values in bold indicate the highest score. P: precision, R: recall.</caption><thead><tr><th rowspan="2">Label</th><th colspan="3">CRF</th><th colspan="3">BidLSTM.CRF</th><th colspan="3">BidLSTM.CRF FEATURES</th><th colspan="3">SciBERT</th><th rowspan="2">Supp</th></tr><tr><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th><th>P</th><th>R</th><th>F1</th></tr></thead><tbody><tr><td>&lt;base&gt;</td><td><b>80.57</b></td><td><b>82.34</b></td><td><b>81.45</b></td><td>56.01</td><td>50.34</td><td>53.02</td><td>59.98</td><td>56.33</td><td>58.09</td><td>61.41</td><td>57.08</td><td>59.16</td><td>3228</td></tr><tr><td>&lt;pow&gt;</td><td>72.65</td><td><b>74.45</b></td><td>73.54</td><td>93.70</td><td>62.38</td><td>74.88</td><td><b>93.71</b></td><td>68.40</td><td><b>78.94</b></td><td>91.24</td><td>64.60</td><td>75.60</td><td>1773</td></tr><tr><td>&lt;prefix&gt;</td><td>93.8</td><td>84.69</td><td><b>89.02</b></td><td>80.31</td><td>85.25</td><td>82.54</td><td>83.21</td><td>83.58</td><td>83.35</td><td>82.10</td><td>85.30</td><td>83.62</td><td>1287</td></tr><tr><td>All (micro avg.)</td><td><b>80.73</b></td><td>80.6</td><td>80.66</td><td>70.19</td><td>60.88</td><td>65.20</td><td>73.03</td><td><b>65.31</b></td><td><b>68.94</b></td><td>73.02</td><td>64.97</td><td>68.76</td><td></td></tr></tbody></table>

Table 6.6 shows the Value ML model scores. SciBERT outperforms the other