# Mining experimental data from Materials Science literature with Large Language Models: an evaluation study

Luca Foppiano, Guillaume Lambard, Toshiyuki Amagasa, Masashi Ishii

► To cite this version:

Luca Foppiano, Guillaume Lambard, Toshiyuki Amagasa, Masashi Ishii. Mining experimental data from Materials Science literature with Large Language Models: an evaluation study. Science and Technology of Advanced Materials: Methods, 2024, 10.1080/27660400.2024.2356506. [https://doi.org/10.1080/27660400.2024.2356502](https://doi.org/10.1080/27660400.2024.2356502)

HAL Id: hal-04407016

https://hal.science/hal-04407016v2

Submitted on 9 Apr 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# Mining experimental data from Materials Science literature with Large Language Models: an evaluation study

Luca Foppiano<sup>a,b</sup>, Guillaume Lambard<sup>c</sup>, Toshiyuki Amagasa<sup>b</sup>, Masashi Ishii<sup>c</sup>

## ARTICLE HISTORY

Compiled April 5, 2024

## ABSTRACT

This study is dedicated to assessing the capabilities of large language models (LLMs) such as GPT-3.5-Turbo, GPT-4, and GPT-4-Turbo in extracting structured information from scientific documents in materials science. To this end, we primarily focus on two critical tasks of information extraction: (i) a named entity recognition (NER) of studied materials and physical properties and (ii) a relation extraction (RE) between these entities. Due to the evident lack of datasets within Materials Informatics (MI), we evaluated using SuperMat, based on superconductor research, and MeasEval, a generic measurement evaluation corpus. The performance of LLMs in executing these tasks is benchmarked against traditional models based on the BERT architecture and rule-based approaches (baseline). We introduce a novel methodology for the comparative analysis of intricate material expressions, emphasising the standardisation of chemical formulas to tackle the complexities inherent in materials science information assessment. For NER, LLMs fail to outperform the baseline with zero-shot prompting and exhibit only limited improvement with few-shot prompting. However, a GPT-3.5-Turbo fine-tuned with the appropriate strategy for RE outperforms all models, including the baseline. Without any fine-tuning, GPT-4 and GPT-4-Turbo display remarkable reasoning and relationship extraction capabilities after being provided with merely a couple of examples, surpassing the baseline. Overall, the results suggest that although LLMs demonstrate relevant reasoning skills in connecting concepts, specialised models are currently a better choice for tasks requiring extracting complex domain-specific entities like materials. These insights provide initial guidance applicable to other materials science sub-domains in future work.

## 1. Introduction

Mining experimental data from literature has become increasingly popular in materials science due to the vast amount of information available and the need to accelerate material discovery using data-driven techniques. Data for machine learning in materials science is often sourced from published papers, material databases, laboratory experiments, or first-principles calculations [1]. The introduction of big data in materials

<footnote><sup>a</sup>Materials Modeling Group, Center for Basic Research on Materials, National Institute for Materials Science, Ibaraki-ken, Tsukuba-shi, 1-1 Namiki, 305-0044, Japan</footnote>

<footnote><sup>b</sup>Knowledge and Data Engineering, Centre for Computational Sciences, University of Tsukuba, JP</footnote>

<footnote><sup>c</sup>Data-driven Materials Design Group, Center for Basic Research on Materials, National Institute for Materials Science, Ibaraki-ken, Tsukuba-shi, 1-1 Namiki, 305-0044, Japan</footnote>
research has shifted from traditional random techniques to more efficient, data-driven methods. Data mining of computational screening libraries has been shown to identify different classes of strong CO₂-binding sites, enabling materials to exhibit specific properties even in wet flue gases [2]. Machine learning techniques have been employed for high-entropy alloy discovery, focusing on probabilistic models and artificial neural networks [3]. However, the use of advanced machine learning algorithms in experimental materials science is limited by the lack of sufficiently large and diverse datasets amenable to data mining [4]. A present central tenet of data-driven materials discovery is that with a sufficiently large volume of accumulated data and suitable data-driven techniques, designing a new material could be more efficient and rational [5]. The materials science field is moving away from traditional manual, serial, and human-intensive work towards automated, parallel, and iterative processes driven by artificial intelligence, simulation, and experimental automation [6,7]. But, materials science literature is a vast source of knowledge that remains relatively unexplored with data mining techniques [8], especially for the reason that materials science data come in diverse forms such as unstructured textual content and structured tables and graphs, adding complexity to the extraction process. As a result, nowadays, many projects still depend on manual data extraction. While extensive structured databases contain accumulated experimental data [9], they remain limited in number and highly costly due to the amount of human labour involved [10].

Additionally, addressing issues related to the quality and meaning of materials science data often demands a curation step assisted by a sub-domain knowledge frequently specific to the approached sub-field of materials science, e.g. polymers, metal-organic frameworks, high-entropy alloys, etc., with their own physical and chemical phenomena, methods and protocols, terminology and jargon. For instance, the classification of superconductors can be complex and sometimes arbitrary, blending compound-based classes like cuprates [11] and iron-based [12] materials with unconventional classes like heavy fermions [13]. The classification of superconductors can also be based on phenomena such as the Meissner effect, which describes how superconductors expel magnetic fields [14]. Superconductors can be divided into two classes according to how this breakdown occurs, the so-called type-I and type-II superconductors. As these classifications are not mutually exclusive certain materials could potentially fall into multiple categories, for example, a material can be both a cuprate and a type-II superconductor, the classification of superconductors is a complex task that demands an extensive knowledge of the developments and current state-of-the-arts. Moreover, substantial confusion may occur due to the cross-domain polysemy of used words, terms and symbols. In different sub-domains, the same term can take on specific nuances or meanings unique to a given sub-domain. This phenomenon is common in language and can lead to misunderstandings if the context of the sub-domain is unclear. For instance, the acronym "TC" or "Tc" will be employed for denoting a "Temperature Curie" or a "superconducting critical temperature", respectively. These sub-domain-specific conventions pose a significant challenge when attempting to create structured datasets across various sub-domains effectively.

Meanwhile, the advent of large language models (LLMs) has inaugurated a new technological era marked by extraordinary potential. These models not only excel in linking diverse concepts but also in engaging in sophisticated conversational reasoning [15–18]. In comparison, rule-based approaches are simpler and faster (tokens per second); however, they are time-intensive to fine-tune and have weak generalisation capabilities, as new rules should be clarified on a case-by-case basis. Small Language Models (SLMs), e.g. BERT-based models, are more specific to the task on which they are pre-trained.
The data size used for pre-training LLMs is usually high enough to contain a high diversity of contexts, thus necessitating fewer examples at the fine-tuning stage than SLMs models.

LLMs offer the possibility of integrating large corpus of textual data at training, with often the ability to ingest large textual inputs at inference with a context window ranging from 4,096 to 128,000 tokens for GPT-3.5-Turbo and GPT-4-Turbo [19], respectively (at the time of this writing). Differently, BERT-based encoders are limited to only 512 tokens, whereas 1,000 tokens are about 750 English words. The size of BERT models does not allow them to sustain their contextual memory after fine-tuning. LLMs possess the capacity to be fine-tuned and retain the contextual knowledge from pre-training, which gives them an advantage in terms of generalisation to other datasets. Finally, the interaction with LLMs via prompts, i.e., tailored instructions, changes the construction paradigm of programmatic solutions, making them more accessible, flexible, and suitable to human operators. Nevertheless, the actual capabilities of LLMs in reasoning, understanding, and recognition are still constantly evolving and being evaluated.

Previous studies in Information Extraction (IE) have shown evidence of LLMs proficiency in general tasks, presenting a valuable opportunity to develop more flexible Text and Data Mining (TDM) processes. Still, they fall short in areas where specific knowledge is required [20]. In particular, LLMs are on par with SLMs in most of the discriminative tasks such as named entity recognition (NER), relation extraction (RE) and event detection (ED) in general domains [21], in history [22], and biology [23]. Other works testing chemistry capabilities found that GPT-4 understands various aspects of chemistry, including chemical compounds [24]; however, its knowledge is general and lacks methods for learning through retrieving recent literature [25].

Therefore, this study assesses LLMs' ability to comprehend, manipulate, and reason with complex information that demands substantial background knowledge, as in materials science.

The objectives of this work can be summarised with the two following questions:

* Q1: *How effectively can LLMs extract materials science-related information?*

* Q2: *To what extent can LLMs use reasoning to relate complex concepts?*

We first classify the fundamental components of the materials science knowledge directed towards designing novel materials with functional properties into two main entity classes: material and property expressions. Properties, e.g., a critical temperature of 4 K, are expressed using measurements of physical quantities. They exhibit a structured format, including modifiers (e.g., "between", "less than", "approximately", or symbols such as ">" or "~"), values, and units, with a wide range of potential values. In contrast, material definitions are conceptually loose and often depend on the specific domain. They may require a substantial amount of accompanying text for a comprehensive description, encompassing details, e.g., compositional ratios, doping agent and ratio, synthesis protocol, process, and additional adjunct information. From a fundamental compositional standpoint, materials are defined by their chemical formula. However, in practice, authors in literature may frequently employ substantives such as commercial names, well-known terms, or crafted designations to describe samples, all of which streamline information in their research papers. Nonetheless, conveying such definitions can unambiguously be challenging.

To address Q1, we evaluate the LLM's performance on NER tasks related to materials and properties extraction. For each task, we choose a pertinent dataset and analyse the performance of each LLM.

Named Entity Recognition (NER) [26], alternatively referred to as named entity
identification or entity extraction, stands as a pivotal component within information extraction. Its primary objective is to pinpoint and categorise named entities within unstructured text, assigning them to predefined categories such as individual names, organisations, geographic locations, medical codes, temporal expressions, quantities, monetary values, percentages, and more. The process of identifying entities aligns closely with sequence labelling tasks, wherein a string of text undergoes analysis, and each token within it (basic unit of text processing, typically a word or a sequence of characters that is treated as a single unit) is designated to one of the pre-established categories. For instance, these categories may include material, doping, condition, or property, among others.

We address Q2 by assessing the capability to establish connections between a predefined set of entities and extract relationships within a given context. Extracting relations between entities is a foundational undertaking in NLP. It entails discerning connections or associations among entities referenced within textual data. For instance, in biomedical research, relationship extraction might involve identifying the association between specific genes and diseases mentioned in scientific literature.

In both cases, we compare the outcomes against a baseline determined by scores (Precision, Recall and F1-score) achieved on the same datasets by either a BERT-based encoder or a rule-based algorithm we have developed in a previous work [27, 28]. Our requirement for the models to be capable of generating output in a valid JSON (JavaScript Object Notation) format is part of our efforts to extract structured databases (Section 2.1.1).

The evaluation of generative models brings an additional complexity. Traditional SLM implementations for solving NER tasks are based on sequence labelling algorithms. They classify each token in the input stream with a limited number of labels, returning a sequence that fits the original input (same number of tokens and structure). Evaluating their performance against expected datasets involves a straightforward comparison of values. Soft-matching techniques can be employed to overlook minor discrepancies. However, with generative models, the output tokens may be structured in ways that significantly differ from the original input sequence. In more general scenarios, semantic models that compare the vectorised representations of two sequences can be utilised [29]. Nevertheless, when dealing with concepts like material expressions, a specialised approach is needed. As an illustration, the terms "solar cell" and "solar cells" represent identical concepts. Yet, the materials denoted by "Ca" (Calcium) and "Cr" (Chromium) are entirely distinct, highlighting a difference of just one letter between the two examples. For this reason, we introduce a novel evaluation method for material names, which involves normalising materials to their chemical formulas before conducting a pairwise comparison of each element. This approach provides a more meaningful and context-aware assessment of the model's performance.

We summarise our contributions as follows:

* We designed and ran a benchmark for LLMs on information extraction, particularly NER of materials and properties. This contribution addresses Q1.
* We evaluated LLMs on RE on entities in the context of materials science to address Q2.
* We propose a novel approach for evaluating Information Extraction tasks applied to materials entities which leverage "formula matching" via pairwise element comparison.
## 2. Method

We chose three OpenAI LLM models reported with their specific names for performing API calls: GPT-3.5-Turbo (gpt-3.5-turbo-0611), GPT-4 (gpt-4), and GPT-4-Turbo (gpt-4-0611-preview). The consideration of open-source LLMs has been deferred to future work due to their limited capability to generate output in a valid JSON format (Section 2.1.1, necessitating a more in-depth investigation).

Our evaluation uses different strategies: zero-shot prompting, few-shot prompting, and fine-tuning (or instruction-learning). Few-shot prompting refers to the model's ability to adapt and perform a new task with minimal examples or prompts. In contrast, zero-shot prompting denotes the model's capability to generalise to tasks it has not been explicitly trained on, emphasising transfer learning within the language domain. Finally, fine-tuning involves adjusting the parameters of a pre-trained model on a specific task or domain using a smaller, task-specific dataset to enhance its performance for that particular application.

We selected two datasets for evaluation: MeasEval [30], a SemEval 2021 task of extracting counts, measurements, and related context from scientific documents and SuperMat, an annotated and linked dataset of research papers on superconductors [31]. SuperMat contains both materials and properties and, for copyright reasons, is not publicly distributed. This reduces the risk that its annotations had been used during the pre-training of any of the LLMs.

Baseline scores were established using a SciBERT-based [32] encoder and RE rule-based algorithm [27] for material-related extractions. Grobid-quantities [28] served as the baseline for NER on properties extraction evaluated against MeasEval.

Evaluation scores, encompassing Precision, TP/(TP + FP), Recall, TP/(TP + FN)), and F1-score, 2 Precision × Recall/(Precision + Recall), were derived from pairwise comparisons between predicted and expected entities, where TP, FP and FN are the true positive, false positive and false negative instances, respectively. Precision gauges accuracy, recall assesses information capture, and F1-Score is their harmonic mean.

The evaluations condense average F1 scores and their standard deviation over three extraction runs. The raw tables with all detailed scores are provided in Appendix A.

### 2.1. Named Entities Recognition

The NER task consists of identifying relevant entities: materials, expressed through a multitude of expressions [31], or properties, expressed as measurements of physical quantities [28].

We calculated the evaluation scores using four different matching approaches. However, we will present only the most relevant to the task (leaving the complete tables<fnref n="1" /> in Appendix A):

* **strict**: Exact matching

* **soft**: Matching using Ratcliff/Obershelp [33] with a threshold at 0.9²

* **Sentence BERT**: Comparison using semantic similarity of sequences using Sentence BERT with a cross-encoder [29], applying a threshold set at 0.9²

* **formula matching**: Our novel method compares material expressions via formula normalisation and element-by-element exact matching.

<footnote n="1">¹The calculation of micro average provides a measure independent of the distribution of the extracted entities over the different documents.
²The threshold is fixed to a value yielding more than 90% similarity.</footnote>
Prompts for interacting with LLMs are defined by two components: system and user prompts. The system prompt is the initial instruction guiding the model's output generation, defining the task or information sought. In contrast, the user prompt is the user's input, specifying their request and shaping the model's response.

The system prompt below was fixed across all tasks. It was specifically crafted to prevent the creation of non-existing facts and favour standardised answers (e.g., "I don't know," "None," etc.) in case of inability to respond.

<caption>**Listing 1** Generic system prompt common to all requests</caption>

Use the following pieces of context to answer the user's question.
If you don't know the answer, just say that you don't know, don't try to
make up an answer.
{text}

The users' prompts for NER with zero-shot prompting were described including the definitions and examples from the SuperMat<fnref n="3" /> and MeasEval<fnref n="4" /> annotations guidelines, respectively.

Below are the user prompt templates used for both materials and properties extraction:

<caption>**Listing 2** User prompt designed for extracting materials and properties. The entity descriptions are separated by dashed lines ("------").</caption>

What are the superconductor materials mentioned in the text?
Only provide the mention of the materials. Avoid repetition.

The material can be expressed as follows:
- chemical formula with variables not substituted, like La(1-x)Fe(x),
- chemical formula with substitution variables like Zr 5 X 3 (X = Sb, Pb
  ↔, Sn, Ge, Si and Al)
- with complete or partial abbreviations like (TMTSF) 2 PF 6,
- doping rates are represented as variables (x, y or other letters)
  ↔ appearing in the material names. These values can be used to
  ↔ complement the material variables (e.g. LaFex01-x).
- doping rates as percentages, like 4% Hdoped sample or 14% Cu doped
  ↔ sample
- material chemical form with no variables e.g. LaFe03NaCl2 where the
  ↔ doping rates are included in the name
- chemical substitution or replacements, like (A is a random variable,
  ↔ can be any symbol): A = Ni, Cu, A = Ni, Ni substituted (which
  ↔ means A = Ni)
- chemical substitution with doping ratio, like (A is a random variable,
  ↔ can be any symbol): A = Ni and x = 0.2

If you don't know the answer, just say you don't know, don't try to make
↔ up an answer.
------

<footnote n="3">³https://supermat.readthedocs.io</footnote>

<footnote n="4">⁴https://github.com/harperco/MeasEval/tree/main/annotationGuidelines#basic-annotation-set</footnote>
Quantity is either a Count, consisting of a value, or a Measurement, consisting of a value and usually a unit. A Quantity can additionally
→ include optional Modifiers like tolerances.

Include relevant text that indicates the application of a modifier, such
→ as "between", "less than", "approximately",
or symbols such as ">" or "~" if they are contiguous with the span.
→ Ignore them if they are separated by additional text.

Example: "The soda can's volume was 355 ml", the quantity is "355 ml".

Extract all the Quantities in the text.

Then, we applied a few-shot prompting technique by incorporating in the users' prompt template above a set of suggestions extracted from the text (see Listing 3 below) using the respective SLMs based on the fine-tuned SciBERT-encoder for materials and properties, i.e., grobid-superconductors [27] and grobid-quantities [28], respectively. Also, as these suggestions originate from another model, they may not be entirely accurate; hence, we emphasised in the prompts that they only serve as examples or hints that the LLMs may ignore.

<caption>**Listing 3** Few-shot prompting modified prompt template.</caption>

[...]
Here are some examples appearing in the text: {hints}
[...]

### 2.1.1. Output format

For all tasks, we required the output to be formatted using a valid JSON document. We justify this decision for three main reasons: a) The responses need to be machine-readable so that the de-serialisation from JSON to objects in many programming languages becomes a trivial operation (e.g., Python, JavaScript). b) The JSON schema can be defined through a documented format regardless of the programming language or platform. Finally, c) the JSON format is an open standard that can be used by anyone and does not require reinventing the wheel by re-implementing any transformation steps from scratch.

The JSON output was obtained by adding formatting instructions in the user's prompt based on the expected output data model, for which different concepts were described differently (e.g., properties are described as a value and an optional unit). We used the implementation provided by the LangChain library<fnref n="5" /> of which one example is illustrated as follows.

<caption>**Listing 4** Example of formatting instruction to a valid JSON format</caption>

The output should be formatted as a JSON instance that conforms to the
→ JSON schema below.

As an example, for the schema {"properties": {"foo": {"title": "Foo", "description": "a list of strings", "type": "array", "items": {"type": "string"}}}, "required": ["foo"]}

<footnote n="5">⁵https://github.com/langchain-ai/langchain</footnote>
the object {"foo": ["bar", "baz"]} is a well-formatted instance of the
→ schema. The object {"properties": {"foo": ["bar", "baz"]}} is not
→ well-formatted.

Here is the output schema:

'''
{
  "properties": {
    "material": {
      "title": "Material",
      "description": "
        → Material or sample name, chemical formula, acronym. Include
        → everything that describes the material.", "type": "string"},
      "material_extra_info": {
        "title": "Material Extra Info",
        "description": "Additional information about the material",
        "type": "string"
      }
    }
  }
}
''''

### 2.1.2. *Formula matching*

Matching materials poses challenges with generative models, while encoder and sequence labelling models maintain unchanged the output from the input sequences. Therefore, evaluating generative models can be complex due to potentially divergent yet semantically equivalent output sequences. Previous works [34] resort to manual evaluation due to these challenges. Notably, as of the time of writing, no specialised approach tailored for material expressions existed. Utilising Sentence BERT, trained on general text, does not ensure accurate material embeddings, raising concerns about the meaningfulness of final matches. To address issues arising from variable sets and to enhance evaluation precision, we propose a novel method named *formula_match*, involving element-by-element pairwise comparisons on normalised formulas for extracted material denominations.

This approach extends strict matching and is activated only when the two input strings differ. In such instances, as depicted in Figure 1, the material expressions slated for comparison undergo normalisation to their formulas using a material parser developed in our prior work [27] (Figure 1 top). The material parser is adept at handling noisy material expressions and strives to parse them effectively. The anticipated output includes a structured representation with the chemical formula presented as a raw string and a dictionary detailing elements and their respective amounts. Subsequently, these structures are compared element by element, as depicted in Figure 1 bottom. The summarised evaluation scores described in Section 3.4 are calculated using the formula matching.

Evaluation and discussion of this method are detailed in Section 3.2.

## 2.2. *Relation Extraction*

The baseline is established by a rule-based algorithm from our previous work [27], which was evaluated with SuperMat and for which we report the aggregated result in Section 2.2.

The prompts are designed by providing a list of entities and requesting the LLM to group them based on their relation. Unlike the NER task, the LLM is expected to reuse information passed in the prompt to compose the response: non-matching information is considered incorrect. The summarised scores in Section 3.5 are obtained with strict matching.
The previous considerations remain relevant for both system and user prompts, with the task description reiterated in each prompt.

<caption>**Listing 5** System prompt for RE modified by emphasising the tasks</caption>

```
You are a useful assistant, who knows about materials science, physics,
→ chemistry and engineering.

You will be asked to compute relation extraction given a text and lists
→ of entities.

If you are not sure, don't try to make up your answer, just answer "None
→ ".
```

We add specific rules to avoid creating invalid groups of relations and to ignore responses containing entities not supplied in the user prompt or empty relation blocks.

The prompt for few-shot prompting was assembled by injecting three examples listed between the dashed lines ("----------") in the zero-shot prompt:

<caption>**Listing 6** Few-shot prompting for extracting relations from lists of entities</caption>

```
Given a text between triple quotes and a list of entities, find the
→ relations between entities of different classes:
"
"
{text}
"
"
{entities}

Use the following examples separated by "--------" to learn the task:
--------
text 1: The researchers of Mg have discovered that MgB2 and MgB3 are
→ superconducting at 29-31 K at ambient pressure.

entities 1:
materials: MgB2, Mg, MgB3
tcs: 29-31 K
pressure: ambient pressure

Result 1:
material: MgB2,
tc: 29-31K,
pressure: ambient pressure:

material: MgB3,
tc: 29-31K,
pressure: ambient pressure:

--------

Text 2: We are studying the material La 3 A 2 Ge 2 (A = Ir, Rh). The
→ critical temperature T C = 4.7 K discovered for La 3 Ir 2 Ge 2 in
→ this work is by about 1.2 K higher than that found for La 3 Rh 2
→ Ge 2.

entities 2:
```
materials: La 3 A 2 Ge 2 (A = Ir, Rh), La 3 Ir 2 Ge 2, La 3 Rh 2 Ge 2
tcs: 4.7 K, 1.2 K

Result 2:

material: La 3 Ir 2 Ge 2
tc: 4.7 K

Text 3: The experimental discovery of the high-temperature
→ superconducting state in the compressed hydrogen and sulfur
→ systems H2S (TC = 150 K for p = 150 GPa) and H3S (TC = 203 K for
→ p = 150 GPa)

entities 3:

materials: H2S, H3S
tcs: 150 K, 203 K
pressures: 150 GPa, 150 GPa

Result 3:

material: H2S,
tc: 4.7 K,
pressure: 150 GPa

material: H3S,
tc: 150 K,
pressure: 150 GPa

Apply strictly the following rules:

- if material is not specified, ignore the relation block,

- if tc is not specified in absolute values, ignore the relation
  → block

2.2.1. *Shuffled* vs *non-shuffled* evaluation

The list of entities supplied to the Language Model (LLM) might be derived based on their order of appearance, creating a scenario where a model generating relations sequentially may achieve an inflated score that does not accurately reflect its relational inference capabilities. To address this, we evaluate each model for RE using two strategies: a *non-shuffled evaluation*, where entities are presented in the order they appear in the original document, and a *shuffled evaluation*, where entities are randomly rearranged before being introduced to the prompt.

## 2.3. Consideration about the fine-tuning

We fine-tuned the GPT-3.5-Turbo model using the OpenAI platform, which ingested training and testing data and generated a new model in a few hours. At the time of writing this article, the fine-tuning of GPT-4 and GPT-4-Turbo is not available. All fine-tuned models were trained using the default parameters selected by the OpenAI
platform.

Table 1 illustrates the dimension of each dataset. The fine-tuned model for properties extraction was trained using the "grobid-quantities dataset" [28] because MeasEval did not contain enough examples for a consistent and unbiased evaluation.

The primary challenge encountered when employing a fine-tuned model was to achieve a valid, machine-readable JSON format. Therefore, we formatted the training data with an expected output in valid JSON format. However, the obtained fine-tuned model struggled to produce valid JSON in its output, leading us to hypothesise that this limitation might be attributed to a shortage of training examples. To address this, we modified our training data expected output from JSON to a pseudo format structured with spaces and break-lines, facilitating simpler handling by the model. The subsequent example illustrates the expected output for a RE task:

<caption>**Listing 7** Example format of the expected answer for the RE task</caption>

```
material: mat1, tc: 22K,
material: mat2, tc: 24K, pressure: 2GPa
```

We followed the same approach for fine-tuning the model for the NER task:

<caption>**Listing 8** Example format of the expected answer for the NER task</caption>

```
materials:
- material1
- material2
- material3
```

Using this technique, we could fine-tune a model and shape its behaviour to answer conversationally. Then, we used the GPT-3.5-Turbo base model to transform the response into JSON format.

To fine-tune the model for the RE task, we introduced the sorting variability in the entity lists provided in the prompt (Section 2.2). This approach does not modify the size of the data set and reduces the possibility that the model learns to aggregate entities in the order they appear in the document. This is the default approach we define as "FT.base" compared to others. In Section 3.5.1, we discuss the impact of two additional strategies for preparing the fine-tuning data. First, "FT document order" keeps the lists of entities as they appear in the document. For example, the made-up sentence "*The two materials MgB2 and MgB3 showed Tc of 39K and 40K, respectively*" will lead to two lists of entities "MgB2, MgB3" and "39K and 40K" which could be assigned in order (MgB2, 39K) and (MgB3, 40K). Intuitively, this leads to poor performance, as we see when evaluating with shuffling conditions (Section 3.5). The second strategy, "FT.augmented", is to augment the size of the dataset, generating multiple training records with a further shuffled entity list for each example in "FT.base". The data used with this strategy is roughly double that of "FT.base" (Table 1). We expect this strategy to obtain similar or better results than "FT.base".

## 3. Results and discussions

In this section, we present and discuss the formula matching and the aggregated results of our evaluations for the LLMs. The completed raw results are available in the Appendix 5.
## 3.1. Limitation of this study

In this paper, we aim to estimate how well LLMs work in tasks related to materials science. Due to the lack of clean datasets covering the entire materials science domain, we used a dataset that focuses on superconductor material. While our goal is to propose a methodology, we are aware that our results need to be verified empirically in other materials science sub-domains in future works. The following intuitions support our hypothesis: for material NER, we expect that the forms on which materials are presented in other domains would have similar expressions to the ones used in superconductor research, considering that chemical formulas, sample names, and commercial names would unlikely be very different between domains. Furthermore, the properties, expressed as measurement and physical quantities, are common to all domains; although the statistical distribution could be different, we don't expect dramatic differences within materials science. On the other hand, RE tasks surely require more datasets that focus both on different domains and different flavours of the same task. As an example, the MatSCIRE [35] dataset, which covers battery-related research, proposes a structure that challenges the relation extraction only between two entities (binary extraction) with the addition of the type of relation which could be inferred by the properties being extracted. In conclusion, we will remand the generalisation for further work.

## 3.2. Formula matching

We evaluated the formula matching to measure two main pieces of information: the gain in the F1-score, and the correctness, as the number of invalid new matches, of the gain. We compared the formula matching with the strict matching because a) it is simple to reproduce and understand visually, and b) the formula matching is built on top of strict matching. We would have more difficulties explaining matches provided by soft matching or SentenceBERT.

We examined the GPT-3.5-Turbo NER extraction (discussed in Section 3.5). 107 out of the 1402 expected records matched correctly using strict matching (P: 22.5%, R: 13.64%, F1: 17.01%). Applying formula matching on the mismatching records, we obtained an additional 176 matches (P: 61.12%, R: 36.00%, F1: 45.31%), for a total gain in F1-score of 28.3 (+266%). For the new 176 records that the formula matching was identifying, we manually examined each pair finding 5 incorrect matches, which corresponds to an error rate of 2.5%.

Most of the mismatches in the strict matching caught up by the formula matching were due to missing adjoined information. The LLMs were not able to include information about doping or shape in the response (e.g. hole-doped La 2-x Sr x CuO 4 was not matching with La 2-x Sr x CuO 4). In other cases, the formula was different by formatting, like: Nd 2-x Ce x CuO 4 and La 2-x Sr x CuO 4. However, the more interesting cases were provided by element or amount substitutions such as: electron-doped infinite-layer superconductors Sr 0.9 La 0.1 Cu 1-x R x O 2 where R = Zn and Ni which was matched Sr0.9La0.1Cu1-xNixO2, or Eu 1-x K x Fe 2 As 2 samples with x = 0.35, 0.45 and 0.5 and Eu 0.5 K 0.5 Fe 2 As 2'. These two cases were particularly complicated to match because they required a deeper understanding of the formula structure.

Among the errors of the formula matching, all of them were provided by the formula which was not correctly parsed, for example in one complicated case with the substrate information: (1-x/2)La 2 O 3 /xSrCO 3 /CuO in molar ratio with x = 0.063, 0.07, 0.09, 0.10, 0.111 and 0.125 which was incorrectly matched with
the general La203.

### 3.3. NER on properties extraction

The property extraction assessment was performed using the MeasEval dataset, with the baseline established by Grobid volumes, achieving an approximately 85% score using a holdout dataset created in conjunction with the application. At the time of writing, the evaluation of grobid-quantities [28] (version 0.7.3⁶) against MeasEval yielded a score of around 59% F1-score. This disparity was anticipated, given the slightly divergent annotation strategies employed by the MeasEval developers compared to those used in developing grobid-quantities (e.g., considerations such as approximate values and other proximity expressions were not considered).

Unexpectedly, none of the models outperformed grobid-quantities in zero-shot prompting, as depicted in Figure 2. This outcome is surprising considering that a) the expression of properties lacks a specific domain constraint (aside from potential variations in frequency distribution), and b) measurements of physical quantities are likely prevalent in the extensive text corpus used to pre-train the OpenAI models.

In the realm of few-shot prompting (Figure 2), a marginal improvement was observed only for GPT-4 and GPT-4-Turbo, resulting in an F1-score gain ranging around 2%. However, this improvement is not significant. We theorise that the hints provided to the LLMs may introduce bias. When these hints are incorrect or incomplete, the LLMs struggle to guide the generation effectively, impacting the quality of the output results. Significantly, the fine-tuned model (Figure 2) shows a slight enhancement compared to zero-shot, few-shot, and the baseline. Interestingly, in this specific instance where both the baseline and fine-tuned models are trained and evaluated on the same data, the LLM demonstrates an approximate 3% increase in the F1-score.

### 3.4. NER on materials expressions extraction

The evaluation of material expressions extraction was performed using the partition of the SuperMat [31] dataset dedicated to validation, consisting of 32 articles.

In zero-shot prompting (Figure 3), both GPT-4 and GPT-4-Turbo achieved comparable F1-scores, hovering around 50%. Notably, all LLMs scored at least 10% lower than the baseline [27]. This disparity is expected, given that material expressions may involve extensive sequences and encompass multiple pieces of information not easily conveyed in the prompt. Few-shot prompting (Figure 3) yielded improved results, with GPT-3.5-Turbo and GPT-4 slightly surpassing the baseline. The introduction of hints in the prompt indeed enhances performance, but, as previously discussed, it appears to strongly influence the LLMs, not able to mitigate the impact of invalid hints that may be provided. Equally unexpected, fine-tuning did not outperform few-shot prompting. This outcome suggests that the additional training did not significantly enhance the LLMs' ability to handle material expressions.

### 3.5. Relation extraction

The evaluation of RE utilised the complete SuperMat dataset, with the results illustrated in Figure 4, comparing the effects of shuffling across different models.

<footnote n="6">⁶https://github.com/lfoppiano/grobid-quantities/releases/tag/v0.7.3</footnote>
GPT-3.5-Turbo zero-shot and few-shot prompting demonstrate a significant difference between shuffled and non-shuffled evaluation (Section 2.2.1), suggesting a sequential connection of entities without specific contextual reasoning. Notably, the fine-tuned GPT-3.5-Turbo model outperforms the baseline by approximately 15% F1-score and does not show relevant differences when the evaluation is performed under shuffling conditions.

Figure 5 specifically highlights the shuffled version of each model and extraction type. Except for GPT-3.5-Turbo, few-shot prompting shows an improvement compared to zero-shot prompting, achieved by incorporating additional examples in each prompt. GPT-4 and GPT-4-Turbo also exhibit stable results under shuffling conditions, achieving an F1-score of around 15-18% lower than fine-tuned GPT-3.5-Turbo.

### 3.5.1. *Data variability for fine-tuning*

In Section 2.3, we describe two additional ways to prepare the data for fine-tuning. As illustrated in Figure 6, the GPT-3.5-Turbo model fine-tuned with the strategy "FT document_order" showed an inability to generalise when evaluated under shuffling conditions, where the model loses around 30% in F1-score. This suggests that adding entropy (for example, by shuffling the data) should be performed as a best practice, which could result in models with larger reasoning capabilities.

When we increased the size of the dataset used in fine-tuning to almost double (Table 1), the resulting model did not improve compared to the FT base. These results confirm that in fine-tuning, size does not matter, while data variability and quality do.

## 4. Code and data availability

This work is available at https://github.com/lfoppiano/MatSci-LumEn. The repository contains the scripts and the data used for extraction and evaluation. The code of the material parser used in the formula matching is available at https://github.com/lfoppiano/material-parsers, and the service API is accessible at https://lfoppiano-material-parsers.hf.space.

## 5. Conclusion

In this study, we have proposed an evaluation framework for estimating how well LLMs perform compared with SLMs and rule-based tasks related to materials science by focusing on sub-domains such as superconductor research. The findings obtained from our work provide initial guidance applicable to other materials science sub-domains in future research.

To evaluate material extraction comparison, we proposed a novel method to parse and match formula elements by elements through an aggregated parser for materials. This new method provides a more realistic F1 score. Compared with strict matching, we obtained a gain in F1-score from 17% to 45% for GPT3.5-Turbo NER at the price of a minimal error rate (2%).

We then evaluated LLMs on two tasks: NER for materials and properties and RE for linking them. LLMs underperform significantly on NER tasks than SLMs in material and property extraction (Q1). This finding is particularly surprising considering properties since these expressions are not confined to a specific domain.
In material extraction, GPT-3.5-Turbo with fine-tuning failed to outperform the baseline, and the same holds for any model with few-shot prompting. For property extraction, GPT-4 and GPT-4-Turbo with zero-shot prompting perform on par with the baseline. GPT-3.5-Turbo with few-shot and fine-tuning, on the other hand, outperforms the baseline by a marginal increase in points. Our results suggest that, for material expressions, small specialised models remain the most accurate choice.

The scenario improves for RE (Q2). With two examples, few-shot prompting demonstrates a significant improvement over the baseline. GPT-4-Turbo exhibits enhanced reasoning capabilities compared to GPT-4 and GPT-3.5-Turbo. GPT-3.5-Turbo performs poorly in both zero-shot and few-shot prompting, showing a substantial score decrease when entities are shuffled, which aligns with previous observations. Nevertheless, fine-tuning yields scores superior to the baseline and other models, showing stability when comparing shuffled and unshuffled evaluations.

In conclusion, to answer Q2, GPT-4 and GPT-4-Turbo showcase effective reasoning capabilities for accurately relating concepts and extracting relations without fine-tuning. However, fine-tuning GPT-3.5-Turbo out yields the best results with a relatively small dataset. GPT-4-Turbo, which costs one-third of GPT-4, remains a robust choice given its reasoning capabilities. However, for Q1, for extracting complex entities such as materials, we find that training small specialised models remains a more effective approach.

## Acknowledgements

Our warmest thanks to Patrice Lopez for his continuous support and inspiration with ideas, suggestions, and fruitful discussions.

## Funding

This work was partially supported by the MEXT Programme: Data Creation and Utilisation-Type Material Research and Development Project (Digital Transformation Initiative Centre for Magnetic Materials) Grant Number JPMXP1122715503.

## Notes on Contributors

LF developed the scripts for extraction and evaluation and wrote the manuscript. GL supported the LLM evaluation and implementation and financed access to the OpenAI API. GL, TA, and MI reviewed the article. MI supervised the process and provided the budget.

## References

[1] Pengcheng Xu, Xiaobo Ji, Minjie Li, and Wencong Lu. Small data machine learning in materials science. *npj Computational Materials*, 9(1):42, March 2023.

[2] Peter G Boyd, Arunraj Chidambaram, Enrique García-Díez, Christopher P Ireland, Thomas D Daff, Richard Bounds, Andrzej Gładysiak, Pascal Schouwink, Seyed Mohamad Moosavi, M Mercedes Maroto-Valer, et al. Data-driven design of metal-organic frameworks for wet flue gas co2 capture. *Nature*, 576(7786):253–256, 2019.
[3] Ziyuan Rao, Po-Yen Tung, Ruiwen Xie, Ye Wei, Hongbin Zhang, Alberto Ferrari, TPC Klaver, Fritz Körmann, Prithiv Thoudden Sukumar, Alisson Kwiatkowski da Silva, et al. Machine learning-enabled high-entropy alloy discovery. *Science*, 378(6615):78–85, 2022.

[4] Andriy Zakutayev, Nick Wunder, Marcus Schwarting, John D Perkins, Robert White, Kristin Munch, William Tumas, and Caleb Phillips. An open experimental database for exploring inorganic materials. *Scientific data*, 5(1):1–12, 2018.

[5] Tran Doan Huan, Arun Mannodi-Kanakkithodi, Chiho Kim, Vinit Sharma, Ghanshyam Pilania, and Rampi Ramprasad. A polymer dataset for accelerated property prediction and design. *Scientific data*, 3(1):1–10, 2016.

[6] Edward O Pyzer-Knapp, Jed W Pitera, Peter WJ Staar, Seiji Takeda, Teodoro Laino, Daniel P Sanders, James Sexton, John R Smith, and Alessandro Curioni. Accelerating materials discovery using artificial intelligence, high performance computing and robotics. *npj Computational Materials*, 8(1):84, 2022.

[7] Norbert Huber, Surya R Kalidindi, Benjamin Klusemann, and Christian J Cyron. Machine learning and data mining in materials science, 2020.

[8] Gilchan Park and Line Pouchard. Advances in scientific literature mining for interpreting materials characterization. *Machine Learning: Science and Technology*, 2(4):045007, 2021.

[9] Swetha Chittam, Balakrishna Gokaraju, Zhigang Xu, Jagannathan Sankar, and Kaushik Roy. Big data mining and classification of intelligent material science data using machine learning. *Applied Sciences*, 11(18), 2021.

[10] Boyuan Ma, Xiaoyan Wei, Chuni Liu, Xiaojuan Ban, Haiyou Huang, Hao Wang, Weihua Xue, Stephen Wu, Mingfei Gao, Qing Shen, Michele Mukeshimana, Adnan Omer Abuassba, Haokai Shen, and Yanjing Su. Data augmentation in microscopic images for material data mining. *npj Computational Materials*, 6(1):125, 2020.

[11] Ivan A Parinov. *Microstructure and properties of high-temperature superconductors*. Springer Science & Business Media, 2013.

[12] Hideo Hosono, Keiichi Tanabe, Eiji Takayama-Muromachi, Hiroshi Kageyama, Shoji Yamanaka, Hiroaki Kumakura, Minoru Nohara, Hidenori Hiramatsu, and Satoru Fujitsu. Exploration of new superconductors and functional materials, and fabrication of superconducting tapes and wires of iron pnictides. *Science and Technology of Advanced Materials*, 2015.

[13] K Mydeen, Anton Jesche, K Meier-Kirchner, U Schwarz, C Geibel, H Rosner, and Michael Nicklas. Electron doping of the iron-arsenide superconductor cefeaso controlled by hydrostatic pressure. *Physical Review Letters*, 125(20):207001, 2020.

[14] John Bardeen, Leon N Cooper, and John Robert Schrieffer. Theory of superconductivity. *Physical review*, 108(5):1175, 1957.

[15] Chaoning Zhang, Chenshuang Zhang, Chenghao Li, Yu Qiao, Sheng Zheng, Sumit Kumar Dam, Mengchun Zhang, Jung Uk Kim, Seong Tae Kim, Jinwoo Choi, et al. One small step for generative ai, one giant leap for agi: A complete survey on chatgpt in aigc era. *arXiv preprint arXiv:2304.06488*, 2023.

[16] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. *arXiv preprint arXiv:2305.10601*, 2023.

[17] Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models—a critical investigation. *arXiv preprint arXiv:2305.15771*, 2023.

[18] Simeng Sun, Yang Liu, Shuohang Wang, Chenguang Zhu, and Mohit Iyyer. Pearl: Prompting large language models to plan and execute actions over long documents. *arXiv preprint arXiv:2305.14564*, 2023.

[19] OpenAI. Models. https://platform.openai.com/docs/models, 2024. [Online; accessed 04-January-2024].

[20] Jan Kocoń, Igor Cicchecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, Anna Kocoń, Bartłomiej Koptyra, Wiktoria Mieleszczenko-Kowszewicz, Piotr Miłkowski,
Marcin Oleksy, Maciej Piasecki, Łukasz Radliński, Konrad Wojtasik, Stanisław Woźniak, and Przemysław Kazienko. ChatGPT: Jack of all trades, master of none. *Information Fusion*, 99:101861, nov 2023.

[21] Yubo Ma, Yixin Cao, YongChing Hong, and Aixin Sun. Large language model is not a good few-shot information extractor, but a good reranker for hard samples! *arXiv preprint arXiv:2303.08559*, 2023.

[22] Carlos-Emiliano González-Gallardo, Emanuela Boros, Nancy Girdhar, Ahmed Hamdi, Jose G Moreno, and Antoine Doucet. Yes but.. can chatgpt identify entities in historical documents? *arXiv preprint arXiv:2303.17322*, 2023.

[23] Milad Moradi, Kathrin Blagec, Florian Haberl, and Matthias Samwald. Gpt-3 models are poor few-shot learners in the biomedical domain. *arXiv preprint arXiv:2109.02555*, 2021.

[24] Kan Hatakeyama-Sato, Naoki Yamane, Yasuhiko Igarashi, Yuta Nabae, and Teruaki Hayakawa. Prompt engineering of gpt-4 for chemical research: what can/cannot be done? *Science and Technology of Advanced Materials: Methods*, 3(1):2260300, 2023.

[25] Kan Hatakeyama-Sato, Seigo Watanabe, Naoki Yamane, Yasuhiko Igarashi, and Kenichi Oyaizu. Using gpt-4 in parameter selection of polymer informatics: improving predictive accuracy amidst data scarcity and ‘ugly duckling’ dilemma. *Digital Discovery*, 2(5):1548–1557, 2023.

[26] David Nadeau and Satoshi Sekine. A survey of named entity recognition and classification. *Lingvisticae Investigationes*, 30(1):3–26, 2007.

[27] Luca Foppiano, Pedro Castro, Pedro Suarez, Kensei Terashima, Yoshihiko Takano, and Masashi Ishii. Automatic extraction of materials and properties from superconductors scientific literature. *Science and Technology of Advanced Materials Methods*, 3, 2023.

[28] Luca Foppiano, Laurent Romary, Masashi Ishii, and Mikiko Tanifuji. Automatic identification and normalisation of physical measurements in scientific literature. In *Proceedings of the ACM Symposium on Document Engineering 2019*, DocEng ’19, New York, NY, USA, 2019. Association for Computing Machinery.

[29] Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pages 3982–3992, Hong Kong, China, November 2019. Association for Computational Linguistics.

[30] Corey Harper, Jessica Cox, Curt Kohler, Antony Scerri, Ron Daniel Jr., and Paul Groth. SemEval-2021 task 8: MeasEval – extracting counts and measurements and their related contexts. In Alexis Palmer, Nathan Schneider, Natalie Schlüter, Guy Emerson, Aurelie Herbelot, and Xiaodan Zhu, editors, *Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)*, pages 306–316, Online, August 2021. Association for Computational Linguistics.

[31] Luca Foppiano, Thaer Dieb, Akira Suzuki, Pedro Castro, Suguru Iwasaki, Asuza Uzuki, Miren Echevarria, Yan Meng, Kensei Terashima, Laurent Romary, Yoshihiko Takano, and Masashi Ishii. Supermat: construction of a linked annotated dataset from superconductors-related publications. *Science and Technology of Advanced Materials Methods*, 1:34–44, 2021.

[32] Iz Beltagy, Kyle Lo, and Arman Cohan. SciBERT: A pretrained language model for scientific text. In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)*, pages 3615–3620, Hong Kong, China, November 2019. Association for Computational Linguistics.

[33] W. Ratcliff John. Pattern matching: the gestalt approach. https://www.drdobbs.com/database/pattern-matching-the-gestalt-approach/184407970?pgno=5, 1988. [Online; accessed 04-January-2024].

[34] Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large
language model for science. *arXiv preprint arXiv:2211.09085*, 2022.

[35] Ankan Mullick, Akash Ghosh, G Sai Chaitanya, Samir Ghui, Tapas Nayak, Seung-Cheol Lee, Satadeep Bhattacharjee, and Pawan Goyal. Matscire: Leveraging pointer networks to automate entity and relation extraction for material science knowledge-base construction. *Computational Materials Science*, 233:112659, 2024.

## Figures & Tables

<figure><img src="image_2.png" /></figure>

<figure><img src="image_3.png" /><figcaption>**Figure 1.** Two materials that appear to have a very different composition are, in reality, overlapping. (Top) Summary of the Material Parser. More information is available in [27]. (Bottom) The pairwise comparison of each chemical formula is performed element-by-element.</figcaption></figure>

<table><caption>**Table 1.** Datasets and support information for fine-tuning GPT-3.5-Turbo. For each task, the data was divided into 70/30 partitions for training and testing, respectively. The testing dataset is different from the evaluation dataset.</caption><thead><tr><th>Task</th><th>Preparation strategy</th><th>Dataset</th><th># Training</th><th># Test</th></tr></thead><tbody><tr><td>NER</td><td>N/A</td><td>SuperMat</td><td>1639</td><td>703</td></tr><tr><td>NER</td><td>N/A</td><td>grobid-quantities dataset</td><td>485</td><td>208</td></tr><tr><td>RE</td><td>FT.base/FT document</td><td>SuperMat</td><td>344</td><td>148</td></tr><tr><td>RE</td><td>FT.augmented</td><td>SuperMat</td><td>695</td><td>299</td></tr></tbody></table>
<figure><img src="image_4.png" /><figcaption>**Figure 2.** Comparison scores for properties extraction using NER. The scores are the aggregations of the micro average F1 scores and are calculated using soft matching with a threshold of 0.9 similarity. The error bars are calculated over the standard deviation of three independent runs.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>**Figure 3.** Comparison scores for material extraction using NER. The metrics are the aggregations of the micro average F1-scores, calculated using formula matching. The error bars are calculated over the standard deviation of three independent runs.</figcaption></figure>
<figure><img src="image_6.png" /><figcaption>**Figure 4.** Comparison of the scores of the shuffled extraction using zero-shot prompting, few-shot prompting and the fine-tuned model for RE on materials and properties. The metrics are the aggregated micro average F1-scores calculated using strict matching. The error bars are calculated over the standard deviation of three independent runs.</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>**Figure 5.** Overview evaluation on shuffling the provided entities in RE on materials and properties. The metrics are the aggregated micro average F1-scores calculated using strict matching. The error bars are calculated over the standard deviation of three independent runs.</figcaption></figure>
<figure><img src="image_8.png" /><figcaption>**Figure 6.** Evaluation of the impact of data variability in fine-tuning GPT-3.5-Turbo. The metrics are the aggregated micro average F1-scores calculated using strict matching. The model "FT document order" was fine-tuned with the original data, where entities were taken of appearance. In "FT base", our default strategy, the entities provided to the prompt were scrambled. The error bars are calculated over the standard deviation of three independent runs.</figcaption></figure>
# Appendix A. Full evaluation results

## A.1. NER for properties extraction

### A.1.1. Zero-shot prompting

<table><caption>**Table A1.** Performance Metrics for GPT-3.5-turbo NER in properties extraction, zero-shot prompting. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Soft matching</td><td>42.73</td><td>14.49</td><td>21.64</td><td>564</td></tr><tr><td>Sentence BERT</td><td>45.39</td><td>15.39</td><td>22.99</td><td>564</td></tr><tr><td rowspan="2">Run2</td><td>Soft matching</td><td>41.81</td><td>13.35</td><td>20.24</td><td>531</td></tr><tr><td>Sentence BERT</td><td>45.01</td><td>14.37</td><td>21.79</td><td>531</td></tr><tr><td rowspan="2">Run3</td><td>Soft matching</td><td>42.86</td><td>14.61</td><td>21.79</td><td>567</td></tr><tr><td>Sentence BERT</td><td>45.86</td><td>15.63</td><td>23.32</td><td>567</td></tr></tbody></table><table><caption>Mean and Standard deviation of F1 score</caption><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Soft matching</td><td>21.22</td><td>0.85</td><td>554</td></tr><tr><td>Sentence BERT</td><td>22.7</td><td>0.80</td><td></td></tr></tbody></table>

<table><caption>**Table A2.** Performance Metrics for GPT-4 NER in properties extraction, zero-shot prompting. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Soft matching</td><td>61.43</td><td>56.70</td><td>58.97</td><td>1535</td></tr><tr><td>Sentence BERT</td><td>65.08</td><td>60.07</td><td>62.48</td><td>1535</td></tr><tr><td rowspan="2">Run2</td><td>Soft matching</td><td>63.23</td><td>57.19</td><td>60.06</td><td>1504</td></tr><tr><td>Sentence BERT</td><td>66.42</td><td>60.07</td><td>63.09</td><td>1504</td></tr><tr><td rowspan="2">Run3</td><td>Soft matching</td><td>62.83</td><td>56.52</td><td>59.51</td><td>1496</td></tr><tr><td>Sentence BERT</td><td>66.11</td><td>59.47</td><td>62.61</td><td>1496</td></tr></tbody></table><table><caption>Mean and Standard deviation of F1 score</caption><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Soft matching</td><td>59.51</td><td>0.54</td><td>1511</td></tr><tr><td>Sentence BERT</td><td>62.72</td><td>0.32</td><td></td></tr></tbody></table>
<table><caption>**Table A3.** Performance Metrics for GPT4-turbo NER in properties extraction, zero-shot prompting. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Soft matching</td><td>59.60</td><td>54.30</td><td>56.83</td><td>1515</td></tr><tr><td>Sentence BERT</td><td>62.90</td><td>57.31</td><td>59.97</td><td>1515</td></tr><tr><td rowspan="2">Run2</td><td>Soft matching</td><td>60.34</td><td>55.44</td><td>57.79</td><td>1528</td></tr><tr><td>Sentence BERT</td><td>63.87</td><td>58.69</td><td>61.17</td><td>1528</td></tr><tr><td rowspan="2">Run3</td><td>Soft matching</td><td>60.05</td><td>54.60</td><td>57.20</td><td>1512</td></tr><tr><td>Sentence BERT</td><td>63.96</td><td>58.15</td><td>60.91</td><td>1512</td></tr></tbody></table>

Mean and Standard deviation of F1 score

<table><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Soft matching</td><td>57.27</td><td>0.48</td><td>1518</td></tr><tr><td>Sentence BERT</td><td>60.68</td><td>0.63</td><td></td></tr></tbody></table>
### A.1.2. Few-shot prompting

<table><caption>**Table A4.** Performance Metrics for GPT-3.5-turbo NER in properties extraction, few-shot prompting. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Soft matching</td><td>60.98</td><td>57.79</td><td>59.34</td><td>1576</td></tr><tr><td>Sentence BERT</td><td>64.85</td><td>61.46</td><td>63.11</td><td>1576</td></tr><tr><td rowspan="2">Run2</td><td>Soft matching</td><td>60.99</td><td>57.73</td><td>59.31</td><td>1574</td></tr><tr><td>Sentence BERT</td><td>64.55</td><td>61.09</td><td>62.77</td><td>1574</td></tr><tr><td rowspan="2">Run3</td><td>Soft matching</td><td>60.72</td><td>57.55</td><td>59.09</td><td>1576</td></tr><tr><td>Sentence BERT</td><td>64.28</td><td>60.91</td><td>62.55</td><td>1576</td></tr></tbody></table>

**Mean and Standard deviation of F1 score**

<table><caption>**Table A5.** Performance Metrics for GPT-4 NER in properties extraction, few-shot prompting. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Soft matching</td><td>59.24</td><td>0.13</td><td>1575</td></tr><tr><td>Sentence BERT</td><td>62.81</td><td>0.28</td><td></td></tr></tbody></table>

<table><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Soft matching</td><td>62.01</td><td>60.85</td><td>61.43</td><td>1632</td></tr><tr><td>Sentence BERT</td><td>65.56</td><td>64.34</td><td>64.95</td><td>1632</td></tr><tr><td rowspan="2">Run2</td><td>Soft matching</td><td>62.15</td><td>60.91</td><td>61.52</td><td>1630</td></tr><tr><td>Sentence BERT</td><td>65.52</td><td>64.22</td><td>64.86</td><td>1630</td></tr><tr><td rowspan="2">Run3</td><td>Soft matching</td><td>62.41</td><td>61.09</td><td>61.74</td><td>1628</td></tr><tr><td>Sentence BERT</td><td>65.6</td><td>64.22</td><td>64.9</td><td>1628</td></tr></tbody></table>

**Mean and Standard deviation of F1 score**

<table><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Soft matching</td><td>61.56</td><td>0.16</td><td>1630</td></tr><tr><td>Sentence BERT</td><td>64.9</td><td>0.04</td><td></td></tr></tbody></table>
<table><caption>**Table A6.** Performance Metrics for GPT-4-turbo NER in properties extraction, few-shot prompting. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Soft matching</td><td>62.18</td><td>62</td><td>62.09</td><td>1652</td></tr><tr><td>Sentence BERT</td><td>65.8</td><td>65.6</td><td>65.7</td><td>1652</td></tr><tr><td rowspan="2">Run2</td><td>Soft matching</td><td>62.42</td><td>62.24</td><td>62.33</td><td>1658</td></tr><tr><td>Sentence BERT</td><td>66.1</td><td>65.9</td><td>66</td><td>1658</td></tr><tr><td rowspan="2">Run3</td><td>Soft matching</td><td>62.48</td><td>62.78</td><td>62.63</td><td>1671</td></tr><tr><td>Sentence BERT</td><td>65.95</td><td>66.27</td><td>66.11</td><td>1671</td></tr></tbody></table><table><caption>Mean and Standard deviation of F1 score</caption><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Soft matching</td><td>62.35</td><td>0.27</td><td>1660</td></tr><tr><td>Sentence BERT</td><td>65.93</td><td>0.21</td><td></td></tr></tbody></table>
### A.1.3. Fine-tuning

<table><caption>**Table A7.** Performance Metrics for fine-tuned GPT-3.5-turbo NER in properties extraction. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Soft matching</td><td>60.76</td><td>58.57</td><td>59.64</td><td>1603</td></tr><tr><td>Sentence BERT</td><td>64.38</td><td>62.06</td><td>63.2</td><td>1603</td></tr><tr><td rowspan="2">Run2</td><td>Soft matching</td><td>60.85</td><td>58.69</td><td>59.75</td><td>1604</td></tr><tr><td>Sentence BERT</td><td>64.4</td><td>62.12</td><td>63.24</td><td>1604</td></tr><tr><td rowspan="2">Run3</td><td>Soft matching</td><td>60.49</td><td>58.45</td><td>59.45</td><td>1607</td></tr><tr><td>Sentence BERT</td><td>64.09</td><td>61.94</td><td>63</td><td>1607</td></tr></tbody></table>

**Mean and Standard deviation of F1 score**

<table><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Soft matching</td><td>59.61</td><td>0.15</td><td>1604</td></tr><tr><td>Sentence BERT</td><td>63.14</td><td>0.12</td><td></td></tr></tbody></table>
## A.2. NER for Materials extraction

### A.2.1. Zero-shot

<table><caption>**Table A8.** Performance Metrics for GPT3.5-turbo NER in materials extraction, zero-shot prompting. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Sentence BERT</td><td>37.34</td><td>22.58</td><td>28.14</td><td>1617</td></tr><tr><td>Formula</td><td>59.49</td><td>35.97</td><td>44.83</td><td>1617</td></tr><tr><td rowspan="2">Run2</td><td>Sentence BERT</td><td>37.06</td><td>22.83</td><td>28.26</td><td>1641</td></tr><tr><td>Formula</td><td>59.21</td><td>36.48</td><td>45.15</td><td>1641</td></tr><tr><td rowspan="2">Run3</td><td>Sentence BERT</td><td>37.45</td><td>22.07</td><td>27.77</td><td>1587</td></tr><tr><td>Formula</td><td>59.74</td><td>35.2</td><td>44.3</td><td>1587</td></tr></tbody></table>

**Mean and Standard deviation of F1 score**

<table><caption>**Table A9.** Performance Metrics for GPT-4 NER in materials extraction, zero-shot prompting. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Sentence BERT</td><td>28.05</td><td>0.25</td><td>1615</td></tr><tr><td>Formula</td><td>44.76</td><td>0.42</td><td></td></tr></tbody></table>

<table><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Sentence BERT</td><td>49.9</td><td>31.25</td><td>38.43</td><td>1103</td></tr><tr><td>Formula</td><td>66.4</td><td>41.58</td><td>51.14</td><td>1103</td></tr><tr><td rowspan="2">Run2</td><td>Sentence BERT</td><td>49.9</td><td>30.61</td><td>37.94</td><td>1097</td></tr><tr><td>Formula</td><td>66.94</td><td>41.07</td><td>50.91</td><td>1097</td></tr><tr><td rowspan="2">Run3</td><td>Sentence BERT</td><td>49.59</td><td>30.74</td><td>37.95</td><td>1108</td></tr><tr><td>Formula</td><td>66.46</td><td>41.2</td><td>50.87</td><td>1108</td></tr></tbody></table>

**Mean and Standard deviation of F1 score**

<table><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Sentence BERT</td><td>38.10</td><td>0.28</td><td>1102</td></tr><tr><td>Formula</td><td>50.97</td><td>0.14</td><td></td></tr></tbody></table>
<table><caption>**Table A10.** Performance Metrics for GPT4-turbo NER in properties extraction, zero-shot prompting. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Sentence BERT</td><td>45.53</td><td>22.07</td><td>29.73</td><td>883</td></tr><tr><td>Formula</td><td>70.53</td><td>34.18</td><td>46.05</td><td>883</td></tr><tr><td rowspan="2">Run2</td><td>Sentence BERT</td><td>46.67</td><td>22.32</td><td>30.2</td><td>873</td></tr><tr><td>Formula</td><td>70.67</td><td>33.8</td><td>45.73</td><td>873</td></tr><tr><td rowspan="2">Run3</td><td>Sentence BERT</td><td>46.74</td><td>22.83</td><td>30.68</td><td>878</td></tr><tr><td>Formula</td><td>70.5</td><td>34.44</td><td>46.27</td><td>878</td></tr></tbody></table><table><caption>Mean and Standard deviation of F1 score</caption><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Sentence BERT</td><td>45.98</td><td>0.56</td><td>878</td></tr><tr><td>Formula</td><td>53.60</td><td>0.30</td><td></td></tr></tbody></table>
### A.2.2. Few-shot

<table><caption>**Table A11.** Performance Metrics for GPT3.5-turbo NER in materials extraction, few-shot prompting. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Sentence BERT</td><td>71.58</td><td>69.39</td><td>70.47</td><td>1887</td></tr><tr><td>Formula</td><td>75.79</td><td>73.47</td><td>74.61</td><td>1887</td></tr><tr><td rowspan="2">Run2</td><td>Sentence BERT</td><td>78.73</td><td>88.78</td><td>83.45</td><td>2495</td></tr><tr><td>Formula</td><td>78.39</td><td>88.39</td><td>83.09</td><td>2495</td></tr><tr><td rowspan="2">Run3</td><td>Sentence BERT</td><td>78.85</td><td>88.9</td><td>83.57</td><td>2448</td></tr><tr><td>Formula</td><td>78.73</td><td>88.78</td><td>83.45</td><td>2448</td></tr></tbody></table>

**Mean and Standard deviation of F1 score**

<table><caption>**Table A12.** Performance Metrics for GPT-4 NER in materials extraction, few-shot prompting. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Sentence BERT</td><td>79.16</td><td>7.5</td><td>2276</td></tr><tr><td>Formula</td><td>80.38</td><td>4.82</td><td></td></tr></tbody></table>

<table><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Sentence BERT</td><td>75.87</td><td>75</td><td>75.43</td><td>1402</td></tr><tr><td>Formula</td><td>77.16</td><td>76.28</td><td>76.72</td><td>1412</td></tr><tr><td rowspan="2">Run2</td><td>Sentence BERT</td><td>70.67</td><td>82.65</td><td>76.19</td><td>1402</td></tr><tr><td>Formula</td><td>70.99</td><td>83.04</td><td>76.54</td><td>1854</td></tr><tr><td rowspan="2">Run3</td><td>Sentence BERT</td><td>70.99</td><td>83.04</td><td>76.54</td><td>1402</td></tr><tr><td>Formula</td><td>72.09</td><td>83.67</td><td>77.45</td><td>1826</td></tr></tbody></table>

**Mean and Standard deviation of F1 score**

<table><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Sentence BERT</td><td>76.35</td><td>1.02</td><td>1402</td></tr><tr><td>Formula</td><td>76.90</td><td>0.48</td><td></td></tr></tbody></table>
<table><caption>**Table A13.** Performance Metrics for GPT4-turbo NER in materials extraction, few-shot prompting. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Sentence BERT</td><td>54.78</td><td>59.95</td><td>57.25</td><td>1735</td></tr><tr><td>Formula</td><td>60.26</td><td>65.94</td><td>62.97</td><td>1735</td></tr><tr><td rowspan="2">Run2</td><td>Sentence BERT</td><td>55.54</td><td>59.44</td><td>57.42</td><td>1707</td></tr><tr><td>Formula</td><td>60.91</td><td>65.18</td><td>62.97</td><td>1707</td></tr><tr><td rowspan="2">Run3</td><td>Sentence BERT</td><td>55.81</td><td>60.08</td><td>57.86</td><td>1707</td></tr><tr><td>Formula</td><td>61.02</td><td>65.69</td><td>63.27</td><td>1707</td></tr></tbody></table><table><caption>Mean and Standard deviation of F1 score</caption><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Sentence BERT</td><td>57.51</td><td>0.31</td><td>1716</td></tr><tr><td>Formula</td><td>63.07</td><td>0.17</td><td></td></tr></tbody></table>
### A.2.3. Fine-tuning

<table><caption>**Table A14.** Performance Metrics for the fine-tuned GPT3.5-turbo NER in materials extraction. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Sentence BERT</td><td>61.02</td><td>65.69</td><td>63.27</td><td>1429</td></tr><tr><td>Formula</td><td>61.02</td><td>65.69</td><td>63.27</td><td>1429</td></tr><tr><td rowspan="2">Run2</td><td>Sentence BERT</td><td>72.24</td><td>67.73</td><td>69.91</td><td>1429</td></tr><tr><td>Formula</td><td>80.14</td><td>75.13</td><td>77.55</td><td>1429</td></tr><tr><td rowspan="2">Run3</td><td>Sentence BERT</td><td>72.17</td><td>67.47</td><td>69.74</td><td>1432</td></tr><tr><td>Formula</td><td>80.08</td><td>74.87</td><td>77.39</td><td>1432</td></tr></tbody></table><table><caption>Mean and Standard deviation of F1 score</caption><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Sentence BERT</td><td>69.75</td><td>0.15</td><td>1430</td></tr><tr><td>Formula</td><td>77.43</td><td>0.09</td><td></td></tr></tbody></table>
### A.3. RE for Materials-Properties extraction

#### A.3.1. Zero-shot

<table><caption>**Table A15.** Performance Metrics for the GPT3.5-turbo model on RE in materials-properties extraction. In this run, the entities are used as they appear in the input data when included in the prompt. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Strict matching</td><td>79.90</td><td>55.29</td><td>65.36</td><td>791</td></tr><tr><td>Soft matching</td><td>80.15</td><td>55.47</td><td>65.56</td><td>791</td></tr><tr><td rowspan="2">Run2</td><td>Strict matching</td><td>80.13</td><td>54.33</td><td>64.75</td><td>775</td></tr><tr><td>Soft matching</td><td>80.52</td><td>54.59</td><td>65.07</td><td>775</td></tr><tr><td rowspan="2">Run3</td><td>Strict matching</td><td>80.74</td><td>55.03</td><td>65.45</td><td>779</td></tr><tr><td>Soft matching</td><td>81.00</td><td>55.21</td><td>65.66</td><td>779</td></tr></tbody></table>

**Mean and Standard deviation of F1 score**

<table><caption>**Table A16.** Performance Metrics for the GPT3.5-turbo model on RE in materials-properties extraction. In this run, we shuffle entities to make it more challenging for the LLM. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Strict matching</td><td>65.18</td><td>0.38</td><td>781</td></tr><tr><td>Soft matching</td><td>65.43</td><td>0.31</td><td></td></tr></tbody></table>

<table><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Strict matching</td><td>76.01</td><td>51.27</td><td>61.23</td><td>771</td></tr><tr><td>Soft matching</td><td>76.91</td><td>51.88</td><td>61.96</td><td>771</td></tr><tr><td rowspan="2">Run2</td><td>Strict matching</td><td>75.82</td><td>50.74</td><td>60.80</td><td>765</td></tr><tr><td>Soft matching</td><td>76.34</td><td>51.09</td><td>61.22</td><td>765</td></tr><tr><td rowspan="2">Run3</td><td>Strict matching</td><td>76.55</td><td>52.84</td><td>62.53</td><td>789</td></tr><tr><td>Soft matching</td><td>77.19</td><td>53.28</td><td>63.04</td><td>789</td></tr></tbody></table>

**Mean and Standard deviation of F1 score**

<table><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Strict matching</td><td>61.52</td><td>0.90</td><td>775</td></tr><tr><td>Soft matching</td><td>62.07</td><td>0.91</td><td></td></tr></tbody></table>
<table><caption>**Table A17.** Performance Metrics for the GPT-4 model on RE in materials-properties extraction. In this run, entities are not shuffled when added to the prompt. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Strict matching</td><td>76.87</td><td>66.58</td><td>71.35</td><td>990</td></tr><tr><td>Soft matching</td><td>77.47</td><td>67.1</td><td>71.92</td><td>990</td></tr><tr><td rowspan="2">Run2</td><td>Strict matching</td><td>76.4</td><td>67.98</td><td>71.94</td><td>1017</td></tr><tr><td>Soft matching</td><td>76.5</td><td>68.07</td><td>72.04</td><td>1017</td></tr><tr><td rowspan="2">Run3</td><td>Strict matching</td><td>76.8</td><td>68.07</td><td>72.17</td><td>1013</td></tr><tr><td>Soft matching</td><td>76.9</td><td>68.15</td><td>72.26</td><td>1013</td></tr></tbody></table>

Mean and Standard deviation of F1 score

<table><caption>**Table A18.** Performance Metrics for the GPT4 model on RE in materials-properties extraction. In this run, entities are shuffled to make it more challenging for the LLM. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Strict matching</td><td>71.82</td><td>0.42</td><td>1006</td></tr><tr><td>Soft matching</td><td>72.07</td><td>0.17</td><td></td></tr></tbody></table>

<table><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Strict matching</td><td>75.86</td><td>65.97</td><td>70.57</td><td>994</td></tr><tr><td>Soft matching</td><td>76.16</td><td>66.23</td><td>70.85</td><td>994</td></tr><tr><td rowspan="2">Run2</td><td>Strict matching</td><td>74.06</td><td>65.44</td><td>69.48</td><td>1010</td></tr><tr><td>Soft matching</td><td>74.06</td><td>65.44</td><td>69.48</td><td>1010</td></tr><tr><td rowspan="2">Run3</td><td>Strict matching</td><td>75.37</td><td>66.4</td><td>70.6</td><td>1007</td></tr><tr><td>Soft matching</td><td>75.57</td><td>66.58</td><td>70.79</td><td>1007</td></tr></tbody></table>

Mean and Standard deviation of F1 score

<table><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Strict matching</td><td>70.21</td><td>0.63</td><td>1003</td></tr><tr><td>Soft matching</td><td>70.37</td><td>0.77</td><td></td></tr></tbody></table>
<table><caption>**Table A19.** Performance Metrics for the GPT4-turbo model on RE in materials-properties extraction. In this run, entities are not shuffled when added to the prompt. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Strict matching</td><td>79.8</td><td>69.12</td><td>74.07</td><td>990</td></tr><tr><td>Soft matching</td><td>80.61</td><td>69.82</td><td>74.82</td><td>990</td></tr><tr><td rowspan="2">Run2</td><td>Strict matching</td><td>80.22</td><td>69.2</td><td>74.31</td><td>986</td></tr><tr><td>Soft matching</td><td>81.03</td><td>69.9</td><td>75.06</td><td>986</td></tr><tr><td rowspan="2">Run3</td><td>Strict matching</td><td>78.87</td><td>73.65</td><td>76.17</td><td>989</td></tr><tr><td>Soft matching</td><td>79.88</td><td>69.12</td><td>74.11</td><td>989</td></tr></tbody></table>

Mean and Standard deviation of F1 score

<table><caption>**Table A20.** Performance Metrics for the GPT4-turbo model on RE in materials-properties extraction. In this run, entities are shuffled to make it more challenging for the LLM. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Strict matching</td><td>73.85</td><td>0.60</td><td>988</td></tr><tr><td>Soft matching</td><td>74.66</td><td>0.49</td><td></td></tr></tbody></table>

<table><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Strict matching</td><td>80.04</td><td>67.37</td><td>73.16</td><td>962</td></tr><tr><td>Soft matching</td><td>80.98</td><td>68.15</td><td>74.01</td><td>962</td></tr><tr><td rowspan="2">Run2</td><td>Strict matching</td><td>80.71</td><td>67.72</td><td>73.64</td><td>959</td></tr><tr><td>Soft matching</td><td>81.44</td><td>68.33</td><td>74.31</td><td>959</td></tr><tr><td rowspan="2">Run3</td><td>Strict matching</td><td>79.27</td><td>66.58</td><td>72.37</td><td>960</td></tr><tr><td>Soft matching</td><td>80.21</td><td>66.58</td><td>72.37</td><td>960</td></tr></tbody></table>

Mean and Standard deviation of F1 score

<table><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th></th></tr></thead><tbody><tr><td>Strict matching</td><td>76.05</td><td>0.74</td><td>960</td></tr><tr><td>Soft matching</td><td>76.87</td><td>0.64</td><td></td></tr></tbody></table>
### A.3.2. Few-shot

<table><caption>**Table A21.** Performance Metrics for the GPT3.5-turbo model on RE in materials-properties extraction. In this run, entities are shuffled to make it more challenging for the LLM. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Strict matching</td><td>76.38</td><td>70.17</td><td>73.14</td><td>1050</td></tr><tr><td>Soft matching</td><td>76.95</td><td>70.69</td><td>73.69</td><td>1050</td></tr><tr><td rowspan="2">Run2</td><td>Strict matching</td><td>75.78</td><td>69.82</td><td>72.68</td><td>1053</td></tr><tr><td>Soft matching</td><td>76.35</td><td>70.34</td><td>73.22</td><td>1053</td></tr><tr><td rowspan="2">Run3</td><td>Strict matching</td><td>75.93</td><td>69.82</td><td>72.74</td><td>1051</td></tr><tr><td>Soft matching</td><td>76.78</td><td>70.6</td><td>73.56</td><td>1051</td></tr></tbody></table>

**Mean and Standard deviation of F1 score**

<table><caption>**Table A22.** Performance Metrics for the GPT3.5-turbo model on RE in materials-properties extraction. In this run, entities are shuffled to make it more challenging for the LLM. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Strict matching</td><td>72.85</td><td>0.25</td><td>1051</td></tr><tr><td>Soft matching</td><td>73.49</td><td>0.24</td><td></td></tr></tbody></table>

<table><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Strict matching</td><td>72.66</td><td>66.49</td><td>69.44</td><td>1046</td></tr><tr><td>Soft matching</td><td>73.52</td><td>67.28</td><td>70.26</td><td>1046</td></tr><tr><td rowspan="2">Run2</td><td>Strict matching</td><td>72.61</td><td>65.18</td><td>68.7</td><td>1026</td></tr><tr><td>Soft matching</td><td>74.07</td><td>66.49</td><td>70.08</td><td>1026</td></tr><tr><td rowspan="2">Run3</td><td>Strict matching</td><td>73.16</td><td>66.05</td><td>69.43</td><td>1032</td></tr><tr><td>Soft matching</td><td>74.42</td><td>67.19</td><td>70.62</td><td>1032</td></tr></tbody></table>

**Mean and Standard deviation of F1 score**

<table><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Strict matching</td><td>72.85</td><td>0.25</td><td>1051</td></tr><tr><td>Soft matching</td><td>73.49</td><td>0.24</td><td></td></tr></tbody></table>
<table><caption>**Table A23.** Performance Metrics for the GPT4 model on RE in materials-properties extraction. In this run, entities are not shuffled when added to the prompt. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Strict matching</td><td>82.41</td><td>74.19</td><td>78.08</td><td>1029</td></tr><tr><td>Soft matching</td><td>82.41</td><td>74.19</td><td>78.08</td><td>1029</td></tr><tr><td rowspan="2">Run2</td><td>Strict matching</td><td>83.32</td><td>73.84</td><td>78.29</td><td>1013</td></tr><tr><td>Soft matching</td><td>83.51</td><td>74.02</td><td>78.48</td><td>1013</td></tr><tr><td rowspan="2">Run3</td><td>Strict matching</td><td>83.5</td><td>73.93</td><td>78.42</td><td>1012</td></tr><tr><td>Soft matching</td><td>83.6</td><td>74.02</td><td>78.52</td><td>1012</td></tr></tbody></table>

Mean and Standard deviation of F1 score

<table><caption>**Table A24.** Performance Metrics for the GPT4 model on RE in materials-properties extraction. In this run, entities are shuffled to make the task more challenging for the LLM. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Strict matching</td><td>78.26</td><td>0.17</td><td>1018</td></tr><tr><td>Soft matching</td><td>78.36</td><td>0.24</td><td></td></tr></tbody></table>

<table><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Strict matching</td><td>81.19</td><td>72.88</td><td>76.81</td><td>1026</td></tr><tr><td>Soft matching</td><td>81.19</td><td>72.88</td><td>76.81</td><td>1026</td></tr><tr><td rowspan="2">Run2</td><td>Strict matching</td><td>81.19</td><td>72.88</td><td>76.81</td><td>1026</td></tr><tr><td>Soft matching</td><td>81.19</td><td>72.88</td><td>76.81</td><td>1026</td></tr><tr><td rowspan="2">Run3</td><td>Strict matching</td><td>82.77</td><td>74.37</td><td>78.34</td><td>1027</td></tr><tr><td>Soft matching</td><td>82.86</td><td>74.45</td><td>78.43</td><td>1027</td></tr></tbody></table>

Mean and Standard deviation of F1 score

<table><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Strict matching</td><td>77.46</td><td>0.78</td><td>1022</td></tr><tr><td>Soft matching</td><td>77.52</td><td>0.80</td><td></td></tr></tbody></table>
<table><caption>**Table A25.** Performance Metrics for the GPT4-turbo model on RE in materials-properties extraction. In this run, entities are not shuffled when they are added to the prompt. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Strict matching</td><td>86.48</td><td>71.65</td><td>78.37</td><td>947</td></tr><tr><td>Soft matching</td><td>87.54</td><td>72.53</td><td>79.33</td><td>947</td></tr><tr><td rowspan="2">Run2</td><td>Strict matching</td><td>86.01</td><td>71.57</td><td>78.13</td><td>951</td></tr><tr><td>Soft matching</td><td>86.01</td><td>71.57</td><td>78.13</td><td>951</td></tr><tr><td rowspan="2">Run3</td><td>Strict matching</td><td>86.67</td><td>71.13</td><td>78.14</td><td>938</td></tr><tr><td>Soft matching</td><td>87.53</td><td>71.83</td><td>78.9</td><td>938</td></tr></tbody></table>

Mean and Standard deviation of F-Score

<table><caption>**Table A26.** Performance Metrics for the GPT4-turbo model on RE in materials-properties extraction. In this run, entities are shuffled to make the task more challenging for the LLM. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Strict matching</td><td>78.21</td><td>0.13</td><td>945</td></tr><tr><td>Soft matching</td><td>79.04</td><td>0.25</td><td></td></tr></tbody></table>

<table><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Strict matching</td><td>86.59 70.08 77.47 925</td><td></td><td></td><td></td></tr><tr><td>Soft matching</td><td>87.46 70.78 78.24 925</td><td></td><td></td><td></td></tr><tr><td rowspan="2">Run2</td><td>Strict matching</td><td>87.17 70.17 77.75 920</td><td></td><td></td><td></td></tr><tr><td>Soft matching</td><td>87.93 70.78 78.43 920</td><td></td><td></td><td></td></tr><tr><td rowspan="2">Run3</td><td>Strict matching</td><td>86.15 69.12 76.7 917</td><td></td><td></td><td></td></tr><tr><td>Soft matching</td><td>87.02 69.82 77.48 917</td><td></td><td></td><td></td></tr></tbody></table>

Mean and Standard deviation of F-Score

<table><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Strict matching</td><td>77.30</td><td>0.54</td><td>920</td></tr><tr><td>Soft matching</td><td>78.05</td><td>0.50</td><td></td></tr></tbody></table>
### A.3.3. Fine-tuning

<table><caption>**Table A27.** Performance Metrics for the GPT3.5-turbo model fine-tuned using the strategy "FT.base" on RE in materials-properties extraction. In this run, entities are not shuffled when included in the prompt. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Strict matching</td><td>91.06</td><td>80.00</td><td>85.17</td><td>123</td></tr><tr><td>Soft matching</td><td>93.50</td><td>82.14</td><td>87.45</td><td>123</td></tr><tr><td rowspan="2">Run2</td><td>Strict matching</td><td>91.06</td><td>80.00</td><td>85.17</td><td>123</td></tr><tr><td>Soft matching</td><td>93.50</td><td>82.14</td><td>87.45</td><td>123</td></tr><tr><td rowspan="2">Run3</td><td>Strict matching</td><td>91.06</td><td>80.00</td><td>85.17</td><td>123</td></tr><tr><td>Soft matching</td><td>93.50</td><td>82.14</td><td>87.45</td><td>123</td></tr></tbody></table>

**Mean and Standard deviation of F1 score**

<table><caption>**Table A28.** Performance Metrics for the GPT3.5-turbo model fine-tuned using the strategy "FT.base" on RE in materials-properties extraction. In this run, entities are shuffled to make it more challenging for the LLM. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Strict matching</td><td>85.17</td><td>0</td><td>123</td></tr><tr><td>Soft matching</td><td>87.45</td><td>0</td><td></td></tr></tbody></table>

<table><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Strict matching</td><td>92.68</td><td>81.43</td><td>86.69</td><td>123</td></tr><tr><td>Soft matching</td><td>95.12</td><td>83.57</td><td>88.97</td><td>123</td></tr><tr><td rowspan="2">Run2</td><td>Strict matching</td><td>91.8</td><td>80</td><td>85.5</td><td>122</td></tr><tr><td>Soft matching</td><td>94.26</td><td>82.14</td><td>87.79</td><td>122</td></tr><tr><td rowspan="2">Run3</td><td>Strict matching</td><td>91.06</td><td>80</td><td>85.17</td><td>123</td></tr><tr><td>Soft matching</td><td>93.5</td><td>82.14</td><td>87.45</td><td>123</td></tr></tbody></table>

**Mean and Standard deviation of F1 score**

<table><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Strict matching</td><td>85.78</td><td>0.79</td><td>123</td></tr><tr><td>Soft matching</td><td>88.07</td><td>0.79</td><td></td></tr></tbody></table>
<table><caption>**Table A29.** Performance Metrics for the GPT3.5-turbo model fine-tuned using the strategy "FT document order" on RE in materials-properties extraction. In this run, entities are not shuffled when included in the prompt. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Strict matching</td><td>88.98</td><td>80.71</td><td>84.64</td><td>127</td></tr><tr><td>Soft matching</td><td>91.34</td><td>82.86</td><td>86.89</td><td>127</td></tr><tr><td rowspan="2">Run2</td><td>Strict matching</td><td>88.98</td><td>80.71</td><td>84.64</td><td>127</td></tr><tr><td>Soft matching</td><td>91.34</td><td>82.86</td><td>86.89</td><td>127</td></tr><tr><td rowspan="2">Run3</td><td>Strict matching</td><td>88.98</td><td>80.71</td><td>84.64</td><td>127</td></tr><tr><td>Soft matching</td><td>91.34</td><td>82.86</td><td>86.89</td><td>127</td></tr></tbody></table>

Mean and Standard deviation of F1 score

<table><caption>**Table A30.** Performance Metrics for the GPT3.5-turbo model fine-tuned using the strategy "FT document order" on RE in materials-properties extraction. In this run, entities are shuffled to make it more challenging for the LLM. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Strict matching</td><td>84.64</td><td>0</td><td>123</td></tr><tr><td>Soft matching</td><td>86.89</td><td>0</td><td></td></tr></tbody></table>

<table><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Strict matching</td><td>47.73</td><td>45</td><td>46.32</td><td>132</td></tr><tr><td>Soft matching</td><td>48.48</td><td>45.71</td><td>47.06</td><td>132</td></tr><tr><td rowspan="2">Run2</td><td>Strict matching</td><td>55.73</td><td>52.14</td><td>53.87</td><td>131</td></tr><tr><td>Soft matching</td><td>55.73</td><td>52.14</td><td>53.87</td><td>131</td></tr><tr><td rowspan="2">Run3</td><td>Strict matching</td><td>57.72</td><td>50.71</td><td>53.99</td><td>123</td></tr><tr><td>Soft matching</td><td>57.72</td><td>50.71</td><td>53.99</td><td>123</td></tr></tbody></table>

Mean and Standard deviation of F1 score

<table><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Strict matching</td><td>51.39</td><td>4.39</td><td>123</td></tr><tr><td>Soft matching</td><td>51.64</td><td>3.96</td><td></td></tr></tbody></table>
<table><caption>**Table A31.** Performance Metrics for the GPT3.5-turbo model fine-tuned using the strategy "FT.augmented" on RE in materials-properties extraction. In this run, entities are not shuffled when included in the prompt. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Strict matching</td><td>89.6</td><td>80</td><td>84.53</td><td>125</td></tr><tr><td>Soft matching</td><td>92</td><td>82.14</td><td>86.79</td><td>125</td></tr><tr><td rowspan="2">Run2</td><td>Strict matching</td><td>89.6</td><td>80</td><td>84.53</td><td>125</td></tr><tr><td>Soft matching</td><td>92</td><td>82.14</td><td>86.79</td><td>125</td></tr><tr><td rowspan="2">Run3</td><td>Strict matching</td><td>89.6</td><td>80</td><td>84.53</td><td>125</td></tr><tr><td>Soft matching</td><td>92</td><td>82.14</td><td>86.79</td><td>125</td></tr></tbody></table>

Mean and Standard deviation of F1 score

<table><caption>**Table A32.** Performance Metrics for the GPT3.5-turbo model fine-tuned using the strategy "FT.augmented" on RE in materials-properties extraction. In this run, entities are shuffled to make it more challenging for the LLM. P: Precision, R: Recall, F1: harmonic average of P and R, Supp: Support, number of extracted entities.</caption><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Strict matching</td><td>84.53</td><td>0</td><td>125</td></tr><tr><td>Soft matching</td><td>86.79</td><td>0</td><td></td></tr></tbody></table>

<table><thead><tr><th>Run</th><th>Matching</th><th>P</th><th>R</th><th>F1</th><th>Supp</th></tr></thead><tbody><tr><td rowspan="2">Run1</td><td>Strict matching</td><td>91.13</td><td>80.71</td><td>85.61</td><td>124</td></tr><tr><td>Soft matching</td><td>91.94</td><td>81.43</td><td>86.36</td><td>124</td></tr><tr><td rowspan="2">Run2</td><td>Strict matching</td><td>87.9</td><td>77.86</td><td>82.58</td><td>124</td></tr><tr><td>Soft matching</td><td>88.71</td><td>78.57</td><td>83.33</td><td>124</td></tr><tr><td rowspan="2">Run3</td><td>Strict matching</td><td>89.52</td><td>79.29</td><td>84.09</td><td>124</td></tr><tr><td>Soft matching</td><td>91.94</td><td>81.43</td><td>86.36</td><td>124</td></tr></tbody></table>

Mean and Standard deviation of F1 score

<table><thead><tr><th>Matching</th><th>Avg.</th><th>σ<sup>2</sup></th><th>Avg. Supp</th></tr></thead><tbody><tr><td>Strict matching</td><td>84.09</td><td>1.51</td><td>124</td></tr><tr><td>Soft matching</td><td>85.35</td><td>1.74</td><td></td></tr></tbody></table>