Venue: NAACL 2024

# PolyIE: A Dataset of Information Extraction from Polymer Material Scientific Literature

Jerry Junyang Cheung\({}^{1*}\), Yuchen Zhuang\({}^{1*}\), Yinghao Li\({}^{1}\), Pranav Shetty\({}^{2}\), Wantian Zhao\({}^{1}\), Sanjeev Grampurohit\({}^{1}\), Rampi Ramprasad\({}^{2}\), Chao Zhang\({}^{1}\)

###### Abstract

Scientific information extraction (SciIE), which aims to automatically extract information from scientific literature, is becoming more important than ever. However, there are no existing SciIE datasets for polymer materials, which is an important class of materials used ubiquitously in our daily lives. To bridge this gap, we introduce PolVIE, a new SciIE dataset for polymer materials. PolVIE is curated from \(146\) full-length polymer scholarly articles, which are annotated with different named entities (i.e., materials, properties, values, conditions) as well as their \(N\)-ary relations by domain experts. PolVIE presents several unique challenges due to diverse lexical formats of entities, ambiguity between entities, and variable-length relations. We evaluate state-of-the-art named entity extraction and relation extraction models on PolyIE, analyze their strengths and weaknesses, and highlight some difficult cases for these models. To the best of our knowledge, PolVIE is the first SciIE benchmark for polymer materials, and we hope it will lead to more research efforts from the community on this challenging task. Our code and data are available on: [https://github.com/jerry3027/PolyIE](https://github.com/jerry3027/PolyIE).

+
Footnote †: These authors contributed equally to this work.

+
Footnote †: These authors contributed equally to this work.

+
Footnote †: These authors contributed equally to this work.

## 1 Introduction

Material science literature is growing at an unprecedented rate. For example, a simple search on Google Scholar with the term "polymers" returns more than 5 million articles on polymer materials. Such literature reports valuable information on the latest advances in material science, ranging from experimental material properties to material synthesis recipes and procedures. As machine learning (ML) has achieved success in different applications of material science Butler et al. (2018); Schmidt et al. (2019), Scientific Information Extraction (SciIE) from literature for supporting various tasks is becoming increasingly important. Automatically extracting structured information about materials from massive unstructured literature data can be invaluable to understanding material properties and synthesis, as well as building data-driven ML tools for material discovery Court et al. (2021); Doan Tran et al. (2020).

While SciIE has rapidly developed in domains such as biomedical science Luan et al. (2018); Gabor et al. (2018); Jain et al. (2020); Hou et al. (2019); Jia et al. (2019); Shi et al. (2024) and chemistry He et al. (2021); Fang et al. (2021); Kim et al. (2023), it has made limited progress in the material science domain. So far, there are only a handful of datasets for material information extraction. Some earlier works use ChemDataExtractor Swain Cole (2016) to automatically generate datasets for battery materials Huang and Cole (2020) and temperatures Court and Cole (2018). More recent datasets are created manually for solid oxide fuel cells Friedrich et al. (2020) and material science synthesis procedures O'Gorman et al. (2021). However, none of these datasets cover polymer

Figure 1: An example of entity and relation annotations in PolVIE from a material science paper Shi et al. (2011), including entity mentions as well as intra-sentence and inter-sentence N-ary relations.



materials, which are an important class of organic materials that play critical and ubiquitous roles in our daily lives. Due to their versatile properties, polymer materials are being widely used in applications such as packaging, coating, energy saving, and medical applications (Wu et al., 2021). As vast amounts of information on polymer development are being reported in literature data, there is a critical need for SciIE benchmarks and tools to harvest such information from the polymer literature.

To address this gap, we construct a dataset for extracting polymer property knowledge from unstructured literature data. Our dataset PolyIE is curated from \(146\) full-length polymer scientific articles, which are annotated by domain experts with named entities (i.e., materials, properties, values, conditions) as well as the \(N\)-ary relations among them (see Figure 1). PolyIE contains \(41635\) entity mentions and \(4443\) relations in total. It covers four different application domains of polymer materials: polymer solar cells (PSC), ring-opening polymerization (RP), polymer membranes (PM), and polymers in lithium-ion batteries (LB). This diversity of content enables the training of models with enhanced generalization capabilities. To the best of our knowledge, PolyIE is the first benchmark for SciIE from full-text polymer literature.

From the natural language processing perspective, extracting information for polymers on PolyIE introduces unique challenges for both named entity recognition and relation extraction:

**Diverse Lexical Formats of Entities.** Polymer-related entities often have different schemes of nomenclature, such as IUPAC names (_e.g._, 'poly(3-hexylthiophene)'), abbreviations ('PDPPNBr'), trade names ('Styron'), common names ('ABS plastic'), and sample labels ('PE-HDPE-01'). In addition, the identification of polymers can also be achieved through the concatenation of homopolymer names with hyphens or slashes, and the inclusion of numerical values for the component ratios and molecular weights ('PVC-PS-PC-20/30/50-800000'). This diversity of nomenclature in literature poses a challenge for named entity recognition.

**Variable-length and Cross-Sentence \(N\)-ary relations.** Previous research on relation extraction has focused on either binary relations (Luan et al., 2018; Yao et al., 2019) or \(N\)-ary relations with a fixed number \(N\)(Jia et al., 2019; Jain et al., 2020; Zhuang et al., 2022). In contrast, many relations described in polymer literature are variable-length \(N\)-ary relations. This is because 1) the reported properties may be describing one or several materials; and 2) different properties can be measured under specific conditions. Furthermore, the elements in a relation tuple may span multiple sentences as shown in Figure 1.

We study seven mainstay NER, five \(N\)-ary RE models, and two LLM-based methods on PolyIE in terms of their overall performance and sample efficiency. We find that the models based on domain-specific pre-trained models (e.g., MaterialsBERT) yield better performance than other baselines. However, all the models struggle with accurately recognizing certain categories of named entities and inferring challenging varied-length \(N\)-ary relations. Moreover, our observations indicate that, under few-shot settings, the recently popular large language models (LLMs) demonstrate inferior performance than the other baselines on PolyIE, highlighting potential limitations in comprehending material science concepts.

Our main contributions are: (1) The first polymer information extraction dataset curated from \(146\) full-length articles for polymer named entity recognition and relation extraction. (2) Thorough evaluation of seven mainstream NER, five \(N\)-ary RE, and two LLM-based models on our curated dataset. (3) Analysis of the difficult cases and limitations of existing models, which we hope will enable future research on this challenging task from the NLP community.

## 2 Related Work

Material Science NLP Datasets.Earlier studies Court and Cole (2018) leverage tools such as ChemDataExtractor Swain and Cole (2016), ChemSpot Rocktaschel et al. (2012), and ChemicalTagger Hawizy et al. (2011) to perform NER annotation for dataset curation. For example, ChemDataExtractor is applied to generate datasets for Curie and Neel magnetic phase transition temperatures Court and Cole (2018) and magnetocaloric materials Court et al. (2021). Besides, people also create expert-annotated datasets Wang et al. (2021); Weston et al. (2019) for the extraction of non-value named entities (_e.g._, material and property names) and their relationships. In recent years, there has been an uptick in efforts to include numerical values in datasets for further extraction, with several studies closely related to PolyIE: Friedrich et al. (2020) annotate a corpus of 45 open 

access scholarly articles on solid oxide fuel cells, covering entity types of materials, values, and devices. Panapitiya et al. (2021) provide annotations of CHEM, VALUE, and UNIT on a set of papers on soluble materials. However, both only provide binary relations between pairs of entities, which is inadequate for describing more complex relations.

\(N\)-ary Relation Extraction.\(N\)-ary relations are size-\(N\) tuples that describe the factual relationship between \(N\) entities. In general domains, the MUC dataset Chinchor (1998) describes event participants in news articles. In the biomedical domain, the BioNLP Event Extraction Shared Task Kim et al. (2009) and PubMed dataset Jia et al. (2019) aim to extract biomedical events from biomedical text. In the machine learning domain, SciREX Viswanathan et al. (2021); Jain et al. (2020); Zhuang et al. (2022) extracts \(N\)-ary relations in terms of <Task, Dataset, Method, Metric>. Different from these works' relations, the \(N\)-ary relations in PolyIE can have a varied number of named entities, which is more flexible in describing material knowledge but meanwhile introduces new challenges to RE. The closest work to PolyIE is drug-combo Tiktinsky et al. (2022), which extracts variable-length combinations of different drugs. However, PolyIE and drug-combo are curated for two different domains, and the relations in PolyIE include numerical values.

## 3 The PolyIE Dataset

In this section, we describe the details of the PolyIE dataset. We first formulate the two information extraction tasks for polymer material literature in SS 3.1. We then describe the data preprocessing and annotation procedures in SS 3.2 and SS 3.3, and finally present the statistics and characteristics of the PolyIE dataset in SS 3.4.

### Task Definition

PolyIE is curated for studying two key information extraction tasks on polymer literature data: (1) identifying relevant named entities, and (2) composing different entities to form \(N\)-ary relations.

**Named Entity Recognition.** Named Entity Recognition (NER) is the process of locating and classifying unstructured text phrases into pre-defined entity categories such as compound names, property names, etc. Given a sentence with \(n\) tokens \({\bf S}=(w_{1},\cdots,w_{n})\), a named entity mention is a span of tokens \({\bf e}=(w_{i},\cdots,w_{j})(0\leq i\leq j\leq n)\) associated with an entity type. In PolyIE, we focus on NER for describing polymer material properties and include four important entity types: material name, property name, property value, and condition. An illustrative example can be found in Figure 1. Based on the BIO schema Li et al. (2012), NER can be formulated as a sequence labeling task of assigning a sequence of labels \({\bf y}=(y_{1},\cdots,y_{n})\), each corresponding to a token in the input sentence.

**Variable-Length \(N\)-ary Relation Extraction.** Variable-length \(N\)-ary relation extraction (RE) refers to the process of identifying and extracting relationships between multiple entity mentions where the number of entities in the relationship can vary. Formally, given a list of \(k\) context sentences \({\cal C}=(S_{1},\cdots,S_{k})\) in one paragraph, let \({\cal E}\) be the set of entities appearing in \({\cal C}\) where each entity \(e\in{\cal E}\) belongs to one of the four entity types described in the NER task. The relation extraction task aims to extract a set of \(m\) relations \({\cal R}=(r_{1},\cdots,r_{m})\) from \({\cal C}\). Each relation \(r_{i}\) is a tuple of entities \(r_{i}=(e_{1},\cdots,e_{N_{i}}),(1\leq i\leq m)\) that describe their <material, property, value, condition> relations. Here, the number of entities \(N_{i}\) can be variable in \({\cal R}\) because: 1) the property value may correspond to several materials instead of one; and 2) the condition entity may be absent. Figure 1 illustrates this variable-length \(N\)-ary RE task.

### Data Preparation

We curate PolyIE from \(146\) publicly available scientific papers, covering four different material science domains: polymer solar cells, ring-opening polymerization, polymer membranes, and lithium-ion batteries. These papers are sub-sampled from the corpus of 2.4 million material science articles described in Shetty et al. (2023). This corpus consists of papers published between 2000 to 2021 and is collected from \(7\) different material science publishers Shetty and Ramprasad (2021, 2021). Keyword-based search was used to locate papers that span multiple application domains within polymers. The resulting dataset consists of \(100\) papers describing fullerene-acceptor polymer solar cells, \(21\) papers describing ring-opening polymerization, \(20\) describing lithium-ion batteries, and \(5\) describing polymer membranes. The text of these papers is parsed from the PDF of these papers using sciDPF 

(a scientific parser based on GROBID (GRO, 2008-2023)) into utf-8 format. The incorrectly parsed units and symbols are corrected using regular expressions. Details about the regex rules used can be found in App. B.

### Data Annotation

The PolyIE dataset is annotated by two polymer science domain experts as well as three computer science graduate students who are trained by the polymer scientists. Both the NER and RE annotations are performed using the Doccano Nakayama et al. (2018) platform, which is an open-source text annotation tool that facilitates visual annotation with a Web interface. Below, we detail the annotation schemes for the NER and RE tasks.

#### Annotating Named Entities

In PolyIE, we annotate mentions of named entities for four categories: material names, property names, property values, and conditions. Each mention is a continuous text span that specifies the actual name of an entity or its abbreviation. This is done by marking the entity mention on the Doccano platform with the corresponding entity type.

**Compound Names (Material).** Compound Name entities include text spans that refer to material objects. Only chemical mentions that could be associated with a chemical structure are annotated as Compound Names. They may be specified by a particular composition formula (_e.g._, "4,9-di(2-octyldodecyl)) ANDT"), a mention of chemical names (_e.g._, "trimethyltin chloride"), or just an abbreviation (_e.g._, "PaNDTDTFBT"). General chemical nouns (_e.g._, "ionic liquids") are not considered.

**Property Names (Property).** We annotate the properties of chemical compounds as long as they can be measured qualitatively (_e.g._, "toxicity" and "crystallinity") or quantitatively (_e.g._, "open-circuit voltage", "decomposition temperature"). Corresponding abbreviations should also be annotated (_e.g._, "PCE", "HOMO level").

**Property Values (Value).** We annotate the spans that can indicate the degree of qualitative properties (_e.g._, "soluble to water") or describe numerical values with units for quantitative properties (_e.g._, "\(9.62\times 10^{-5}~{}\Omega^{-1}m^{-1}\)", "5.14 ppm").

* 3 MHz").

#### Variable-Length \(N\)-ary Relations

For RE, we annotate the \(N\)-ary relations between the named entities to capture their <Material, Property, Value, Condition> relations.

**Primary Binary Relations.** As Doccano and most other existing text annotation tools only support annotations for binary relations, we decompose the \(N\)-ary relation annotation task into simpler binary relation annotation and later aggregate them into full \(N\)-ary relations. We split an \(N\)-ary relation into multiple binary relations for annotation: Material-Material marks the relations between material names that constitute one material system; Material-Property identifies the relation between a material and its reported property name; Property-Value annotates the corresponding property name and value; and Value-Condition marks the property values measured under a specific condition.

**Transforming Binary to \(N\)-ary Relations.** We then transform all the binary relations with common involved entities to generate \(N\)-ary relations in the format of <Material, (Material), Property, Value, (Condition)>. We abandon all binary relations that cannot be combined with other binary relations, only maintaining the generated \(N\)-ary relations with \(N>2\).

#### Inter-Annotator Agreement

All documents in PolyIE are annotated by at least two annotators independently. If annotation conflicts arise across two annotators, a third annotator is then assigned to annotate the corresponding sentences independently. The final annotation is determined by majority voting.

We calculate the inter-annotator agreement in terms of Fleiss' Kappa Fleiss (1971). The Fleiss' Kappa for individual entity types is calculated by treating other entity types as negative samples. The results are shown in Table 1. The Fleiss' Kappas for Material, Property, and Value are all in the range of almost perfect agreement, while the corresponding value for Condition lies in the range of substantial agreement. For RE, we consider all annotated relations as subjects and treat categories as binary. The Fleiss' Kappa for RE is 0.67.

We also compute the average F1-score similar to Friedrich et al. (2020). The F1-score is calculated by treating one annotator as the gold standard and 

the other annotator as predicted. For the NER, spans and entity types have to exactly match. For RE, all entity mentions within the n-ary relation have to exactly match. The averaged F1-score for the NER and RE task is 0.89 and 0.84 respectively.

### Data Analysis

Table 2 shows the key statistics for our corpus. PolyIE contains \(41635\) entity mentions and \(4443\) relations in all \(146\) fully annotated polymer material science literature. We quantitively analyze some key properties of PolyIE:

**Statistics of Entities.** For all the named entity mentions, the distribution of the four entity types Material, Property, Value, and Condition are \(49.54\%\), \(31.82\%\), \(17.00\%\), and \(1.70\%\), respectively. In total, those \(41365\) mentions describe \(10890\) distinct named entities for polymer materials.

**Statistics of \(N\)-ary Relations.** Among the \(4443\) relations on PolyIE, \(86.38\%\) are \(3\)-ary; \(13.62\%\) are \(4\)-ary; and \(3.20\%\) are \(5\)-ary. Meanwhile, \(26.65\%\) of the relations are cross-sentence relations, while the rest are intra-sentence relations.

## 4 Modeling

In this section, we describe how we model the named entity recognition and \(N\)-ary relation extraction tasks on PolyIE.

**Named Entity Recognition.** We model the NER task as a sequence labeling problem and learn a neural sequence tagger, as shown in Figure 2. We study both the bi-directional LSTM-CRF (**BiLSTM-CRF**) Ma and Hovy (2016) model and BERT-based Devlin et al. (2019) NER models for neural sequence tagging. We also study the performance of GPT-3.5 and GPT-4 on NER.

In BiLSTM-CRF, the input text is passed through an embedding layer to obtain token representations. These representations are then fed into a BiLSTM layer Lample et al. (2016) to capture contextual information. The output of the BiLSTM layer is finally sent to a subsequence Conditional Random Field (CRF) layer Lafferty et al. (2001) for sequence labeling. For the pre-trained language models (PLM), we study both \({\bf BERT}_{\rm base}\)Devlin et al. (2019) and **RoBERTa**Liu et al. (2019) for NER. We also include four domain-specific BERT models: **SciBERT**Beltagy et al. (2019), **BioBERT**Lee et al. (2020), **MatSciBERT**Gupta et al. (2022), and **MaterialsBERT**Shetty et al. (2023). All the NER models in the BERT family are fine-tuned for sequence labeling, by stacking a linear layer that maps the contextual token representations into the label space. In addition, we also evaluate LLMs' abilities in marking material science concepts. Following the existing work Tang et al. (2023), we directly prompt **GPT-3.5-turbo** and **GPT-4** with few-shot exemplars to use special marks "@@" to annotate the boundaries and types of the named entities. Detailed explanations and examples of prompts are included in App. C.

**Relation Extraction.** For relation extraction, we evaluate the performances of the rule-based method, PLM-based models, and graph-based models. For the rule-based method, we leverage the assumption, **Proximity-Rule**, that relations are more likely to be formed with most proximitive entities. As illustrated in Figure 2, PLM-based models (such as **BERT-RE**) leverage the strong representation power of pre-trained language models on entities and employ simple aggregation techniques, such as concatenation and summation, to compose relation embeddings for further prediction. Example models in this category are state-of-the-art models **PURE**Zhong and Chen (2021), which inserts a special "entity marker" token around the entities in a candidate relation; and its variant **PURE

\begin{table}
\begin{tabular}{l|c c|c c} \hline Overall & Material & Property & Value & Condition \\ \hline
0.86 & 0.88 & 0.82 & 0.88 & 0.71 \\ \hline \end{tabular}
\end{table}
Table 1: Fleiss’ kappa for all annotators across all mentions and each entity type respectively.

\begin{table}
\begin{tabular}{l r r r r r} \hline  & PSC & RP & LB & PM & All \\ \hline documents & 100 & 21 & 20 & 5 & 146 \\ sentences & 9,367 & 3,120 & 3,031 & 555 & 16,073 \\ tokens & 288,142 & 91,421 & 90,381 & 15,579 & 485,523 \\ avg. tokens/doc.* & 3,201.6 & 3,102.4 & 3,227.9 & 3,115.8 & 3,325.5 \\ \hline mentions & 28,775 & 5,760 & 6,013 & 1,087 & 41,635 \\ –Material & 13,244 & 3,120 & 3,390 & 740 & 20,494 \\ –Property & 9,848 & 1,597 & 1,616 & 187 & 13,248 \\ –Value & 5,294 & 792 & 835 & 111 & 7,032 \\ –Condition & 364 & 150 & 167 & 21 & 702 \\ entities & 7,099 & 1,621 & 1,739 & 431 & 10,890 \\ avg. mentions/doc.* & 287.8 & 274.3 & 300.7 & 217.4 & 285.2 \\ \hline relations & 3,084 & 592 & 615 & 152 & 4,443 \\ –3-ary & 2,554 & 503 & 516 & 123 & 3,838 \\ –4-ary & 388 & 89 & 99 & 29 & 605 \\ –5-ary & 142 & - & - & - & 142 \\ avg. relations/doc.* & 30.8 & 28.2 & 30.8 & 30.4 & 30.4 \\ \hline \multicolumn{5}{l}{*Avg. indicates average and doc. refers to document.} \\ \end{tabular}
\end{table}
Table 2: PolyIE corpus statistics.

 



**SUM**(Tiktinsky et al., 2022), which uses embedding summation for variable-length \(N\)-ary RE. We also study graph-based methods for \(N\)-ary RE, **DigE++**(Luan et al., 2019), which constructs a dynamic span graph from the input text, with entities as nodes and relations as edges to reason over multihop relations. For models based on LLMs like **GPT-3.5-turbo** and **GPT-4**, we randomly choose a subset of examples from the training set to serve as few-shot instances. These are then directly sent to the models as prompts to facilitate the relation extraction.

## 5 Experiments

### Experimental Setup

**Evaluation Protocol.** We split the dataset into \(123\) training articles, \(27\) validation articles, and \(27\) test articles following a \(70\%/15\%/15\%\) ratio. The three sets do not have overlapping scientific documents. For NER, we report the entity-level precision, recall, and F-1 scores of each baseline for different entity categories, as well as the corresponding micro-average of these metrics. For RE, we report the precision, recall, and F-1 score.

**Hyperparameters.** For BiLSTM-CRF, we use one layer of BiLSTM layer with \(256\)-dimensional hidden states and \(128\) embedding dimensionality. For the BERT-family NER models, we stack a linear layer with a hidden size of \(128\) on the BERT architecture for token classification. For all the NER and RE models, we use early stopping on the dev set for regularization. See App. D for details.

### Main Results

**Entity Mention Extraction.** Table 3 shows the performance of different methods for the NER task on PolyIE. From the results, we make the following observations: (1) BERT-based models significantly outperform BiLSTM-CRF model with a \(14.8\%\) gain in micro average F1-score. This is because BERT-based models have been pre-trained on a large corpus of data, allowing them to possess more semantic knowledge than BiLSTM-CRF and to better understand the context. (2) Domain-specific BERT models achieve slightly better performance than the vanilla BERT due to the encoding of domain-specific knowledge. MaterialsBERT, which is fine-tuned on a corpus of materials science article abstracts, shows the best overall performance. (3) Upon comparing the performance of different entity types, we find that it is challenging for all models to discriminate Condition entities from the other categories. We hypothesize that this is because Conditions are relatively rare in the training data, and the Condition entities could resemble property value entities.

**Relation Extraction.** Table 4 shows the performance of different methods for the RE task on PolyIE, and we make the following observations: (1) Among all the models evaluated, the PURESUM model with MatSciBERT as the encoder achieves the highest F-1 score, indicating that MatSciBERT can better understand the context, and the summation operation is an appropriate aggregation method for variable-length \(N\)-ary relation extraction. (2) The rule-based approach exhibits inferior performance in comparison to most deep learning models, indicating that there are many cases that do not conform to the proximity rule, such as cross-sentence relations and parallel relations. (3) Interestingly, the BERT-RE model shows even worse performance than the rule-based method. Compared to PURE-based models, BERT-RE directly averages the embeddings of all tokens related to the relation. As tokens with similar types have similar representations, and N-ary relations are composed of certain entity-type elements, the averaging operation results in similar relation representations, ultimately leading to poor model performance. (4) As DyGIE++ is a model specifically designed for binary relation extraction, it can only determine the presence of N-ary relations by assessing the connectivity of arbitrary pairs of elements in the relationship. It thus has stricter judging criteria than the other methods, making its precision higher at the cost of lower recall.

**Analysis on LLMs.** LLMs such as GPT-3.5-turbo and GPT-4 exhibit worse performance compared to most baseline models on both NER and RE tasks. This discrepancy is likely due to the small proportion of polymer material science content in their pre-training corpus. When these models are directly prompted with few-shot examples, as opposed to being fine-tuned with training data, they receive less domain-specific information. This limitation hinders their ability to effectively understand and process concepts related to polymer material science. Potential updates on LLMs, like external tools (e.g., knowledge retriever) Shi et al. (2023); Zhuang et al. (2023) or collaborations between LLMs and smaller pre-trained language models Yu et al. (2023); Xu et al. (2023), may further

boost the performance via injecting more domain-specific knowledge. Due to the poor performance obtained under the few-shot prompting setting and the high cost when fine-tuning LLMs, we recommend fine-tuning smaller domain-specific pre-trained language models, like MaterialsBERT in Table 3 and PURE-SUM (MatSciBERT) in Table 4, to extract polymer material science entities and relations.

### Impact of Data Size

We evaluate the NER model performance as a function of the amount of training data in Figure 3. Compared to BERT-based models, the performance of the BiLSTM-CRF model is consistently inferior, with only slight changes with varying sizes of

\begin{table}
\begin{tabular}{c|c|c|c} \hline Model & Precision & Recall & F-1 Score \\ \hline Proximity-Rule & 26.49 & 30.83 & 28.50 \\ BERT-RE & 12.06 & 40.28 & 18.57 \\ DyGIE++ & 67.53 & 50.28 & 57.64 \\ PURE & 60.27 & 54.04 & 56.98 \\ PURE-SUM (SciBERT) & 42.86 & 82.50 & 56.41 \\ PURE-SUM (MatSciBERT) & 51.91 & 83.06 & 63.89 \\ \hline GPT-3.5-Turbo & 16.37 & 34.27 & 21.73 \\ GPT-4 & 37.82 & 54.16 & 44.06 \\ \hline \end{tabular}
\end{table}
Table 4: Main RE results on the test dataset, presented as Precision, Recall, and F-1 Scores in \(\%\).

Figure 3: Effect of training data size on NER task.

\begin{table}
\begin{tabular}{l|c|c|c|c} \hline \hline Model & Material & Property & Value & Condition & Micro Average \\ \hline BiLSTM-CRF & 58.9 (68.4/51.7) & 70.5 (75.4/66.2) & 73.0 (74.6/71.5) & 13.1 (36.4/8.0) & 65.8 (72.4/60.4) \\ \hline BERTbase & 83.9 (84.0/83.8) & 77.8 (81.1/74.7) & 81.3 (83.9/79.0) & 13.8 (16.2/12.0) & 80.6 (82.4/78.8) \\ RoBERTbase & 85.4 (86.4/84.4) & 76.2 (77.4/75.2) & 81.8 (83.3/80.3) & 12.5 (16.7/10.0) & 80.7 (82.0/79.4) \\ \hline SciBERT & 85.6 (87.1/84.1) & 74.6 (77.2/72.3) & 81.9 (84.6/79.4) & 11.3 (19.0/8.0) & 80.3 (82.7/78.1) \\ BioBERT & 85.1 (84.5/85.7) & 76.9 (79.3/74.6) & 82.6 (82.6/82.5) & 15.2 (16.6/14.0) & 81.0 (81.7/80.3) \\ MatSciBERT & 85.8 (84.4/87.3) & 77.4 (78.2/76.5) & 82.4 (81.9/82.8) & 11.4 (13.2/10.0) & 81.3 (81.8/11.7) \\ MaterialsBERT & 85.7 (85.2/86.3) & 77.8 (79.8/75.9) & 82.5 (82.6/82.4) & 14.4 (17.0/12.7) & 81.6 (82.2/80.9) \\ \hline GPT-3.5-Turbo & 63.7 (61.4/67.2) & 49.4 (47.5/52.5) & 59.5 (86.6/45.9) & 2.2 (17.5/1.3) & 56.4 (58.8/54.1) \\ GPT-4 & 64.7 (57.6/75.2) & 61.6 (52.2/76.6) & 74.2 (67.1/84.2) & 5.7 (8.5/4.8) & 64.5 (56.5/75.1) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Main NER results on the test dataset, presented as “F-1 Score (Precision/Recall)” in \(\%\). We offer scores under different metrics for each entity category and the overall micro-average performance. The reported score is the average of 5 distinct runs.

Figure 2: Model architecture for Named Entity Recognition (left) and \(N\)-ary Relation Extraction (right).

 

training data. This trend demonstrates the superiority of language model pre-training stage, which allows BERT-family NER models to encode relevant knowledge for the downstream task. Comparing different BERT models, MaterialsBERT consistently outperforms vanilla BERT by a slight margin, which reflects the benefit of developing domain-specific pre-trained language models.

Figure 4(a) shows the performance of the best RE model PURE-SUM as training data size varies. With more training data, the model's performance generally increases in all the metrics. However, after training on \(60\%\) data, the recall starts to decrease, while the other metrics still slightly increase. This is because the imbalance between positive and negative cases starts to influence the training, where models are more likely to predict relations as negative, making the false negative cases increase and the recall decrease. Additional details about the impact of data size on RE tasks can be seen in App. E.

### Error Analysis

We analyze the key error types of the BERT\({}_{\rm base}\) NER model by drawing its confusion matrix on the test set, as shown in Figure 5. The confusion matrix shows that the majority of entities are correctly predicted as their gold label, with the exception of Condition entities. The limited number of training samples containing Condition entities makes it difficult for the model to distinguish them from other irrelevant entities (labeled "0"). Additionally, the resemblance between Condition and Property Value entities often results in incorrect predictions between them.

For RE, Table 5 illustrates the major error types made by the PURE-SUM model, including: (1) Interweaving or parallel relations in the text present a significant challenge for models in understanding the alignment between multiple sets of entities; (2) The task of flexible-length \(N\)-ary relation extraction is challenging, and errors often occur when encountering relations that cover more entities (e.g., determining whether to include the Condition in the prediction); (3) The last type of error frequently arises when the sentence organization is atypical, including sentences written in the passive voice.

## 6 Conclusion

We have curated a new dataset PolyIE for named entity recognition and \(N\)-ary relation extraction from polymer scientific literature. PolyIE covers thousands of <Material, Property, Value, Condition> relations curated from 146 full polymer articles. We have evaluated mainstay NER and RE models on PolyIE and analyzed their perfor

Figure 4: Performances of PURE-SUM on RE task with limited training data and few-shot setting.

Figure 5: The confusion matrix of BERT on NER task.

\begin{table}
\begin{tabular}{l|p{85.4pt}} \hline Noise Types & Input Text \\ \hline Interweaving Relations & The corresponding HOMO and **LUMO** energy levels for **PIDTT-TzTz** and **PIDFT-TzTZ-TT** are (-5.24, **3.21**) and (-5.34, -3.03) eV, respectively. \\ \hline Partially Correct Relations & For example, OFETs made using a **porphyrin-diacetylene** polymer give **mobilities** of **1\textbackslash{}^{*}\textbackslash{}^{-7}\textbackslash{}^{-1}\textbackslash {}^{8}\textbackslash{}^{-1}\textbackslash{}^{8}\textbackslash{}^{-1}\textbackslash {}^{8}\textbackslash{}^{-1}\textbackslash{}^{8}\textbackslash{}^{-1}\textbackslash {}^{8}\textbackslash{}^{-1}\textbackslash{}^{8}\textbackslash{}^{-1}\textbackslash {}^{8}\textbackslash{}^{-1}\textbackslash{}^{8}\textbackslash{}^{-1}\textbackslash {}^{8}\textbackslash{}^{-1}\textbackslash{}^{8}\textbackslash{}^{-1}\textbackslash {}^{8}\textbackslash{}^{8}\textbackslash{}^{-1}\textbackslash{}^{8}\textbackslash {}^{8}\textbackslash{}^{-1}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{ 8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8} \textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8} \textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8} \textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8} \textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8} \textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8} \textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8} \textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8} \textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8} \textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8} \textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8} \textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{ }\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8} \textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{ }\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8} \textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{ }\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{ }\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{ }\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{ }\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{ }\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{ }\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{ }\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{ }\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{ }\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{ }\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{ }\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{ }\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{ }\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{ }\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{ }\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{} \textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\textbackslash{}^{8}\ 

mance and error cases. In addition, we have also tested the performance of the strongest LLMs, GPT-3.5 and GPT-4, on PolyIE. We found that even state-of-the-art models, either domain-specific pre-trained language models or most advanced LLMs, can struggle with hard NER and RE cases. Through error analysis, we found that such difficulties arise from the diverse lexical formats and ambiguity of polymer named entities and also variable-length and cross-sentence \(N\)-ary relations. Our work contributes the first polymer scientific information extraction dataset as well as insights into this dataset. We hope PolyIE will serve as a useful resource that will and attract more research efforts from the NLP community to push the boundary of this task.

## Limitations

One limitation of PolyIE is that we have annotated only the text modality of the polymer literature corpus. While tables and figures are not included in PolyIE, they are two important modalities that contain a considerable amount of information about polymer properties. It will be interesting to explore annotation schemes that can extend PolyIE to include tables and figures and enable multi-modal information extraction jointly from text, tables, and figures. In addition, PolyIE currently covers four application subdomains for polymer materials. In the future, PolyIE can benefit from including more sub-domains for polymers, as well as scientific publications for other organic materials. Such extensions will not only make PolyIE more comprehensive for studying polymer information extraction, but also allow it to be used to study cross-domain transfer of different information extraction models.

## Acknowledgements

This work was supported in part by NSF IIS-2008334, IIS-2106961, CAREER IIS-2144338, and ONR MURI N00014-17-1-2656.

## References

* L. Beltagy, K. Lo, and A. Cohan (2019)SiBERT: a pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3615-3620. External Links: Link Cited by: SS1.
* K. T. Butler, D. W. Davies, H. Cartwright, O. Isayev, and A. Walsh (2018)Machine learning for molecular and materials science. Nature559 (7715), pp. 547-555. External Links: ISSN 0037-87X5, Document, Link Cited by: SS1.
* N. A. Chinchor (1998)Overview of muc-7/met-2. Technical report. Cited by: SS1.
* C. J. Court and J. M. Cole (2018)Auto-generated materials database of curie and neel temperatures via semi-supervised relationship extraction. Scientific data5 (1), pp. 1-12. Cited by: SS1.
* C. J. Court, A. Jain, and J. M. Cole (2021)Inverse design of materials that exhibit the magnetocaloric effect by text-mining of the scientific literature and generative deep learning. Chemistry of Materials33 (18), pp. 7217-7231. External Links: ISSN 0037-87X5, Document, Link Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, USA, pp. 4171-4186. External Links: ISBN 978-3-319-142-1, Document, Link Cited by: SS1.
* H. D. Tran, C. Kim, L. Chen, A. Chandrasekaran, R. Batra, S. Venkatram, D. Kamal, J. P. Lightstone, R. Gurnani, P. Shetty, et al. (2020)Machine-learning predictions of polymer properties with polymer genome. Journal of Applied Physics128 (17), pp. 128-137. External Links: ISSN 0037-87X5, Document, Link Cited by: SS1.
* B. Fang, C. Druckenbrodt, S. A. Akhondi, I. He, T. Baldwin, and K. Verspoor (2021)Chemu-ref: a corpus for modeling anaphora resolution in the chemical domain. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1362-1375. External Links: ISSN 1536-1473, Document, Link Cited by: SS1.
* J. L. Fleiss (1971)Measuring nominal scale agreement among many raters. Psychological bulletin76 (5), pp. 378. Cited by: SS1.
* A. Friedrich, E. Adel, F. Tomazic, J. Hingerl, R. Benteau, A. Marusczyk, and L. Lange (2020)The SOFC-exp corpus and neural approaches to information extraction in the materials science domain. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 1255-1268. External Links: ISSN 0037-87X5, Document, Link Cited by: SS1.
* K. Gabor, D. Buscaldi, A. Schumann, B. QasemiZadeh, H. Zargayouna, and T. Charnois (2018)Semeval-2018 task 7: semantic relation extraction and classification in scientific papers. In SemEval, pp. 679-688.

 

Tanishqa Gupta, Mohd Zaki, NM Krishnan, et al. 2022. Matscibert: A materials domain language model for text mining and information extraction. _npj Computational Materials_, 8(1):1-11.
* Hawizy et al. [2011] Lezan Hawizy, David M Jessop, Nico Adams, and Peter Murray-Rust. 2011. Chemicaldaggerger: A tool for semantic text-mining in chemistry. _Journal of cheminformatics_, 3(1):1-13.
* He et al. [2021] Jiayuan He, Dat Quoc Nguyen, Saber A Akhondi, Christian Druckenbrodt, Camilo Thorne, Ralph Hoessel, Zubair Afzal, Zenan Zhai, Biaoyan Fang, Hiyori Yoshikawa, et al. 2021. Chemu 2020: Natural language processing methods are effective for information extraction from chemical patents. _Frontiers in Research Metrics and Analytics_, 6:654438.
* Hou et al. [2019] Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, and Debasis Ganguly. 2019. Identification of tasks, datasets, evaluation metrics, and numeric scores for scientific leaderboards construction. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5203-5213, Florence, Italy. Association for Computational Linguistics.
* Huang and Cole [2020] Shu Huang and Jacqueline M Cole. 2020. A database of battery materials auto-generated using chemdataextractor. _Scientific Data_, 7(1):1-13.
* Jain et al. [2020] Sarthak Jain, Madeleine van Zuylen, Hannaneh Hajishirzi, and Iz Beltagy. 2020. SciREX: A challenge dataset for document-level information extraction. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7506-7516, Online. Association for Computational Linguistics.
* Jia et al. [2019] Robin Jia, Cliff Wong, and Hoifung Poon. 2019. Document-level n-ary relation extraction with multi-scale representation learning. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 3693-3704, Minneapolis, Minnesota. Association for Computational Linguistics.
* Kim et al. [2009] Jin-Dong Kim, Tomoko Ohta, Sampo Pyysalo, Yoshinobu Kano, and Jun'ichi Tsuji. 2009. Overview of BioNLP'09 shared task on event extraction. In _Proceedings of the BioNLP 2009 Workshop Companion Volume for Shared Task_, pages 1-9, Boulder, Colorado. Association for Computational Linguistics.
* Kim et al. [2023] Yunsoo Kim, Hyuk Ko, Jane Lee, Hyun Young Heo, Jinyoung Yang, Sungsoe Lee, and Kyu-hwang Lee. 2023. Chemical language understanding benchmark. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 5: Industry Track)_, pages 404-411.
* Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Lafferty et al. [2001] John D Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In _ICML_.
* Lample et al. [2016] Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 260-270, San Diego, California. Association for Computational Linguistics.
* Lee et al. [2020] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. Biobert: a pre-trained biomedical language representation model for biomedical text mining. _Bioinformatics_, 36(4):1234-1240.
* Li et al. [2012] Qi Li, Haibo Li, Heng Ji, Wen Wang, Jing Zheng, and Fei Huang. 2012. Joint bilingual name tagging for parallel corpora. In _21st ACM International Conference on Information and Knowledge Management, CIKM 2012_, pages 1727-1731.
* Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Luan et al. [2018] Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 3219-3232, Brussels, Belgium. Association for Computational Linguistics.
* Luan et al. [2019] Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019. A general framework for information extraction using dynamic span graphs. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 3036-3046, Minneapolis, Minnesota. Association for Computational Linguistics.
* Ma and Hovy [2016] Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1064-1074, Berlin, Germany. Association for Computational Linguistics.
* Mosbach et al. [2021] Marius Mosbach, Maksym Andriushchenko, and Dietrich Klawow. 2021. On the stability of fine-tuning {bert}: Misconceptions, explanations, and strong baselines. In _International Conference on Learning Representations_.
* Nakayama et al. [2018] Hiroki Nakayama, Takahiro Kubo, Junya Kamura, Yasufumi Taniguchi, and Xu Liang. 2018. doccano: Text 

annotation tool for human. Software available from [https://github.com/doccano/doccano](https://github.com/doccano/doccano).
* O'Gorman et al. (2021) Tim O'Gorman, Zach Jensen, Sheshera Mysore, Kevin Huang, Rubayat Mahbub, Elsa Olivetti, and Andrew McCallum. 2021. MS-mentions: Consistently annotating entity mentions in materials science procedural text. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 1337-1352, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Panapitiya et al. (2021) Gihan Panapitiya, Fred Parks, Jonathan Sepulveda, and Emily Saldanha. 2021. Extracting material property measurement data from scientific articles. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 5393-5402, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Rocktaschel et al. (2012) Tim Rocktaschel, Michael Weidlich, and Ulf Leser. 2012. Chemspot: a hybrid system for chemical named entity recognition. _Bioinformatics_, 28(12):1633-1640.
* Schmidt et al. (2019) Jonathan Schmidt, Mario RG Marques, Silvana Botti, and Miguel AL Marques. 2019. Recent advances and applications of machine learning in solid-state materials science. _npj Computational Materials_, 5(1):1-36.
* Shetyty et al. (2023) Pranav Shetyty, Arunkumar Chitteth Rajan, Chris Kuenneth, Sonakshi Gupta, Lakshmi Prerana Panchurnati, Lauren Holm, Chao Zhang, and Rampi Ramprasad. 2023. A general-purpose material property data extraction pipeline from large polymer corpora using natural language processing. _npj Computational Materials_, 9(1):52.
* Shety and Ramprasad (2021a) Pranav Shety and Rampi Ramprasad. 2021a. Automated knowledge extraction from polymer literature using natural language processing. _Iscience_, 24(1).
* Shety and Ramprasad (2021b) Pranav Shety and Rampi Ramprasad. 2021b. Machine-guided polymer knowledge extraction using natural language processing: The example of named entity normalization. _Journal of Chemical Information and Modeling_, 61(11):5377-5385.
* Shi et al. (2011) Qinqin Shi, Haijun Fan, Yao Liu, Wenping Hu, Yongfang Li, and Xiaowei Zhan. 2011. A copolymer of benzodithiophene with tips side chains for enhanced photovoltaic performance. _Macromolecules_, 44(23):9173-9179.
* Shi et al. (2024) Wenqi Shi, Ran Xu, Yuchen Zhuang, Yue Yu, Jieyu Zhang, Hang Wu, Yuanda Zhu, Joyce C Ho, Carl Yang, and May Dongmei Wang. 2024. Ehragent: Code empowers large language models for few-shot complex tabular reasoning on electronic health records. In _ICLR 2024 Workshop on Large Language Model (LLM) Agents_.
* Shi et al. (2023) Wenqi Shi, Yuchen Zhuang, Yuanda Zhu, Henry Iwinski, Michael Wattenberger, and May Dongmei Wang. 2023. Retrieval-augmented large language models for adolescent idiopathic scoliosis patients in shared decision-making. In _Proceedings of the 14th ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics_, pages 1-10.
* Swain and Cole (2016) Matthew C Swain and Jacqueline M Cole. 2016. Chemdataextractor: a toolkit for automated extraction of chemical information from the scientific literature. _Journal of chemical information and modeling_, 56(10):1894-1904.
* Tang et al. (2023) Ruixiang Tang, Xiaotian Han, Xiaoqian Jiang, and Xia Hu. 2023. Does synthetic data generation of llms help clinical text mining? _arXiv preprint arXiv:2303.04360_.
* Tiktinsky et al. (2022) Areyh Tiktinsky, Vijay Viswanathan, Danna Niezni, Dana Meron Azagury, Yosi Shamay, Hilleel Taubbib, Tom Hope, and Yoav Goldberg. 2022. A dataset for n-ary relation extraction of drug combinations. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3190-3203, Seattle, United States. Association for Computational Linguistics.
* Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. _Journal of machine learning research_, 9(11).
* Viswanathan et al. (2021) Vijay Viswanathan, Graham Neubig, and Pengfei Liu. 2021. CitationIE: Leveraging the citation graph for scientific information extraction. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 719-731, Online. Association for Computational Linguistics.
* Wang et al. (2021) Xuan Wang, Vivian Hu, Xiangchen Song, Shweta Garg, Jinfeng Xiao, and Jiawei Han. 2021. ChemNER: Fine-grained chemistry named entity recognition with ontology-guided distant supervision. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 5227-5240, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Weston et al. (2019) Leigh Weston, Vahe Tshitoyan, John Dagdelen, Olga Kononova, Amalie Trewartha, Kristin A Persson, Gerbrand Ceder, and Anubhav Jain. 2019. Named entity recognition and normalization applied to large-scale information extraction from the materials science literature. _Journal of chemical information and modeling_, 59(9):3692-3702.
* Wu et al. (2021) Chao Wu, Lihua Chen, Ajinkya Deshmukh, Deepak Kamal, Zongze Li, Pranav Shetyty, Jieriu Zhou, Harikrishna Sahu, Huan Tran, Gregory Sotzing, et al. 2021. Dielectric polymers tolerant to electric field and temperature extremes: Integration of phenomenology, informatics, and experimental validation. _ACS Applied Materials & Interfaces_, 13(45):53416-53424 

Ran Xu, Hejie Cui, Yue Yu, Xuan Kan, Wenqi Shi, Yuchen Zhuang, Wei Jin, Joyce Ho, and Carl Yang. 2023. Knowledge-infused prompting: Assessing and advancing clinical text data generation with large language models. _arXiv preprint arXiv:2311.00287_.
* Yao et al. (2019) Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. DocRED: A large-scale document-level relation extraction dataset. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 764-777, Florence, Italy. Association for Computational Linguistics.
* Yu et al. (2023) Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. 2023. Large language model as attributed training data generator: A tale of diversity and bias. In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
* Zhong and Chen (2021) Zexuan Zhong and Danqi Chen. 2021. A frustratingly easy approach for entity and relation extraction. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 50-61, Online. Association for Computational Linguistics.
* Zhuang et al. (2022) Yuchen Zhuang, Yinghao Li, Jerry Junyang Cheung, Yue Yu, Yingjun Mou, Xiang Chen, Le Song, and Chao Zhang. 2022. Resel: N-ary relation extraction from scientific text and tables by learning to retrieve and select. _arXiv preprint arXiv:2210.14427_.
* Zhuang et al. (2023) Yuchen Zhuang, Yue Yu, Kuan Wang, Haotiian Sun, and Chao Zhang. 2023. Toolqa: A dataset for llm question answering with external tools. _arXiv preprint arXiv:2306.13304_.

## Appendix A T-SNE Visualization of Entity Embeddings

Figure 6 shows t-SNE Van der Maaten and Hinton (2008) visualization of entity embeddings generated by BERT\({}_{\rm base}\), SciBERT and MatSciBERT. Compared with all the visualization of different entity embeddings, we can observe that pre-training on a more similar domain of corpus to fine-tuning corpus will make model generate high-quality embeddings. From the figures, we can easily observe that MatSciBERT embeddings of the same entity type are more clustered than those of BERT\({}_{\rm base}\), which is also consistent with what we observe from the quantitive results.

## Appendix B Regular Expression

Regular expression is used to correct incorrectly parsed units and mathematical symbols after tokenization to ensure the quality of tokens. Details of the regular expression rules are listed below:

* Composite units that are broken down to multiple tokens are joined to form a single token. Missing caret symbols are added.
* Degree Celsius symbols are merged to a single token. Missing circles are added.
* Numbers with a following percentage sign are merged to a single token.

## Appendix C Implementation Details of LLMs

We conduct experiments on Azure OpenAI platform, with GPT-3.5-turbo and GPT-4 in 0613 version. We set the temperature as \(0\) to obtain a stable and faithful evaluation of the LLMs' results. Following the existing work Tang et al. (2023), we have 4 components in our NER prompt: general instruction, annotation guideline, output indicator, and few-shot exemplars. (1) The general instruction part specifies the objective of the LLM to mark the polymer material science entities or relations. (2) The annotation guideline is to provide additional explanation and guidelines for the LLM to follow when annotating different types of entities and relations. (3) The output indicator specifies the output format of the LLM. (4) The few-shot exemplars allow LLM to form a more cohesive understanding of previous instructions.

The NER and RE prompts are presented below.

 

entities'within=aprovidedparagraph. These entitiesincludechemical names(CN), propertynames(PN),propertyvalues(PV),andconditions(Condition).Chemicalnames,polymermaterialnamesandtheir abstractionsareentities.Polymer materialnamesmightcontain multiplechemicalnameswithinit,labelthemasasingentity. Abstractionsofpropertynamesarealsoconsideredentities.Propertyvaluescontainingboththenumberandtheunit.Torepresentrecognizednamedentitiesintheoutputtext,enclosethenwithinspecialsymbols'@',followedbythierrespectivetypes'(CN)','(PN)',or'(PV)'beforetheending'@'.Theremainingtextshouldremainunchanged\lstinline{Listing} 2: RE prompt.

```
Asaskilledlinguist,yourmissionistoanalyzeaproverdparagraphthatcontainsfourdistinctypesofentities:ChemicalNames(CN),PropertyNames(PN),PropertyValues(PV),andConditions(Condition).Eachofthesementitiesiscendwithin"@'symbols,withtherinentitytypespecifiedinfbracketsbeforetheclosing"@''.Yourobjectiveistoidentifysandextractrelationshipsamongtheseentities,andthenpresenttheminoneoftwopossibleformats:(ChemicalNames,PropertyNames,PropertyValues,Condition)or(ChemicalNames,PropertyNames,PropertyValues).Pleaseonlyestablishrelationsusingtheprovidedentities,andonlyprovidealistoffestractedrelations.Belowaresomeexamples:
```

## Appendix D Implementation Details

All the NER and RE models are trained with the Adam optimizer Kingma and Ba (2014), with different learning rate: The BiLSTM-CRF model is trained with a learning rate of \(0.005\) and batch size of \(64\); While fine-tuning the BERT-family NERmodels, we select the learning rate of \(3e-4\); For relation extraction, instances with lengths exceeding \(300\) are broken into several shorter segments, without cutting off relations, and the models are trained with a learning rate of \(2e-4\) and a batch size of \(8\). We run all the experiments \(3\) times and report the average in the tables and the average/error bar in the figures. All experiments are conducted on _CPU_: Intel(R) Core(TM) i7-5930K CPU @ 3.50GHz and _GPU_: NVIDIA GeForce RTX A5000 GPUs using python 3.8 and Pytorch 1.10.

## Appendix E Few-Shot Learning

Figure 7 shows the performance of different NER models under few-shot settings. We can see BERT-based NER models consistently outperform BiLSTM-CRF models by large margins. However, the variances of such BERT-based NER models are also much larger. This is likely due to the different quality and representativeness of the training samples and the capacity of pre-trained language models. The MatSciBERT model, for instance, has already captured a significant amount of domain knowledge during pre-training. When it is fed with critical cases during the fine-tuning stage,

Figure 6: t-SNE visualization of entity embeddings generated by BERT, SciBERT, and MatSciBERT.

Figure 7: Effect of the few-shot learning on NER task.

 

it can quickly adapt such knowledge to fine-tuning, resulting in high-quality decision boundaries on the corpus. However, if the training samples are of poor quality and not representative, the model's performance can be limited. Such instability of BERT-based fine-tuning is also observed on GLUE (Mosbach et al., 2021).

## Appendix F Impact of Negative Sampling in Training

As the RE models are trained with negative sampling, we investigate the impact of negative samples during the training process. We study three ways to create negative samples from existing relations, by corrupting entities with other irrelevant entities of the same type in the context sentences. (1) Easy: all possible random corruptions; (2) Medium: single or double element corruption; and (3) Hard: only single-element corruption. Figure 8 shows the results when training with different negative sampling policies, with a fixed \(k=10\). We find that the hard negative sampling strategy achieves superior performance, suggesting that using hard negative cases can help the model learn better decision boundaries. In Figure 8, we also evaluate the model performances when varying the number of negative samples \(k\) from \(5\) to \(20\). The trend shows that \(k=20\) achieves the best performances with all different negative sampling strategies.

## Appendix G Annotation Guidance

In this section, we will introduce the annotation guidance. There are 4 types of entities that should be annotated: Chemical Compound, Property Name, Property Value, and Condition.

### Chemical Compound

\(\bullet\) Only chemical nouns that can be associated with a specific structure should be labeled as Chemical Compounds: _e.g._, "4,9-di(2-octyldodecyl) anNDT", "trimethyltin chloride";

\(\bullet\) Abbreviation of the chemical nouns should also be labeled as Chemical Compounds as long as it can be associated with a specific structure: _e.g._, "PaNDTDFBT";

\(\bullet\) General chemical concepts (non-structural or non-specific chemical nouns), adjectives, verbs, and other terms that can not be associated directly with a chemical structure should not be annotated: _e.g._, "polymer", "conjugated polymers" should not be annotated;

\(\bullet\) Spans: Spans of Chemical Compounds should not contain leading or trailing spaces. If the abbreviation of Chemical Compound appears inside brackets, the brackets should not be included in the annotation.

### Property Name

\(\bullet\) Properties of chemical compounds should be annotated as long as they can be measured qualitatively (such as toxicity and crystallinity) or quantitatively (with a unit and a value). Property Names that occur without a corresponding value should also be annotated: _e.g._, "Hole mobility", "Open-circuit voltage", "decomposition temperature", "conductivity", "toxicity";

\(\bullet\) Abbreviations of Property Names should be annotated: "PCE", "HOMO level", "LUMO level";

\(\bullet\) Laboratory methods should not be annotated as Property Names: "Tritration", "Cyclic voltammetry" should not be annotated as Property Names;

\(\bullet\) Spans: Spans of Property Names should not contain leading or trailing spaces.

### Property Value

\(\bullet\) Both quantitative and qualitative Property Values should be annotated;

\(\bullet\) Do not annotate overly vague adjectives;

\(\bullet\) Spans of Property Values should not contain leading or trailing spaces. Property Value and its units should be contained as a single span. Ranges of Property Value should be contained as a single span.

### Condition

\(\bullet\) Only quantitative modifiers that constrain the numerical Property Value should be annotated as Conditions;

\(\bullet\) Spans of Conditions should not contain leading or trailing spaces.

The screenshots of the official annotation guidance shared with all the annotators are listed in Figure 9 and Figure 10.

Figure 8: Effect of the quality and amount of negative samples during training in \(N\)-ary relation extraction.

 



## Appendix H Top-5 Named Entities

\begin{table}
\begin{tabular}{l|p{284.9pt}} Entity Type & Top-5 Entities \\ \hline Chemical Name & PC71BM, P1, PCBM, thiophene, P3HT \\ \hline Property Name & PCE, J sc, V oc, absorption, HOMO \\ \hline Property Value & nm, 10\%, 300, 0.76 V, 0.82 V \\ \hline Condition & 5\% weight loss, in solution, red-shifted, illumination of AM1.5, at room temperature \\ \hline Overall & PCE, J sc, V oc, PC71BM, HOMO \\ \end{tabular}
\end{table}
Table 6: The top-5 entities for each category.

Figure 9: Overview of documents to annotate on the annotation platform.





## Annotate Data

For entity annotations, you first need to select the range of words from the corpus and then select the corresponding entity label for it:

To remove annotations, you can click on an existing annotation and click the little cross highlighted below:

Figure 10: Instructions on assigning pre-defined labels to named entities.