Construction and Application of Materials Knowledge Graph in Multidisciplinary Materials Science via Large Language Model
Authors: Yanpeng Ye
Jie Ren
Shaozhou Wang
Yuwei Wan
Imran Razzak
Bram Hoex
Haofen Wang
Tong Xie
Wenjie Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Large Language Model
Knowledge Graph
AI4Science
Material Science
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: Knowledge in materials science is widely dispersed across extensive scientific literature, posing significant challenges for efficient discovery and integration of new materials. Traditional methods, often reliant on costly and time-consuming experimental approaches, further complicate rapid innovation. Addressing these challenges, the integration of artificial intelligence with materials science has opened avenues for accelerating the discovery process, though it also demands precise annotation, data extraction, and traceability of information. To tackle these issues, this article introduces the Materials Knowledge Graph (MKG), which utilizes advanced natural language processing techniques, integrated with large language models to extract and systematically organize a decade's worth of high-quality research into structured triples, contains 162,605 nodes and 731,772 edges. MKG categorizes information into comprehensive labels such as Name, Formula, and Application, structured around a meticulously designed ontology, thus enhancing data usability and integration. By implementing network-based algorithms, MKG not only facilitates efficient link prediction but also significantly reduces reliance on traditional experimental methods. This structured approach not only streamlines materials research but also lays the groundwork for more sophisticated materials knowledge graphs.

Construction and Application of Materials Knowledge Graph in Multidisciplinary Materials Science via Large Language Model
Yanpeng Ye!.21, Jie Ren Shaozhou Wang"* Yuwei Wan Imran Razzak', Bram Hoex Haofeng Wang", Tong Xie?.,, and Wenjie Zhang!
'School Computer Science and Engineering, University New South Wales *GreenDDynamies
3 Department of Materials Science and Engineering, City University Hong Kong *Departeent of Linguistics and Translation, City University of Hong Kong School of Photovoltaic and Renewable Energy Engineering, University of New South Wales Coollege of Design & Innovation, Tongji University
Abstract
Knowledge materials science is widely dispersed across extensive scientific literature, posing significant challenges the efficient discovery and integration of new materials. Traditional methods, often reliant on costly and time-consuming experimental approaches, further complicate rapid innovation. Addressing these challenges, the integration of artificial intelligence with materials science has opened avenues for accelerating the discovery process, though it also demands precise annotation, data extraction, and traceability information. To tackle these issues, this article introduces the Materials Knowledge Graph (MKG), which utilizes advanced natural language processing techniques integrated with large language models to extract and systematically organize decade's 's worth of high- quality research into structured triples, contains 162,605 nodes and 731,772 edges. MKG categorizes information into comprehensive labels such as Name, Formula, and Application, structured around a meticulously designed ontology, thus enhanc- ing data usability and integration. By implementing network-based algorithms, MKG not only facilitates efficient link prediction but also significantly reduces reliance on traditional experimental methods. This structured approach not only streamlines materials research but also lays the groundwork for more sophisticated science knowledge graphs.
Introduction
In the contemporary information era, despite notable advancements, the creation and advancement of novel materials still heavily rely on traditional, cime-consuming trial-and-error methods intertwined with chemical and physical intuitions. These conventional research approaches significantly impede the life-cycle heghhppeforrance material research. Given the specialization, inherent complexity, and vast knowledge base of material science, researchers focusing on single direction often struggle to efficiently access and understand material knowledge from multidisciplinary studies. For instance, researchers in solar cell development might not fully comprehend studies related to solid-state
"Corresponding authors:
shaozhou @ greendynamics.co..a
jie. Set.zhang@unsee @unsw.ddu.un "These authors contributed equally to this work.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

batteries or organic light-emitting diodes. Yet, the electronic properties of materials across these different domains are highly related, and researchers in different domains can potentially learn from each other. To accelerate the progress of materials research, there is pressing need to efficiently integrate knowledge from various disciplines [T. However, this vital knowledge is scattered across vast array of over 10 million scientific papers, covering diverse topics and disciplines such as materials preparation and functionalization methods, advanced materials characterization techniques, and the exploration of physical, chemical, and biological properties, along with their applications in fields like electronic devices, clean energy storage and transfer, and mechanical engineering. This fragmentation of knowledge represents significant barrier to interdisciplinary collaboration and innovation. A critical gap in current research infrastructure is the lack of an effective materials science database that can consolidate this scattered knowledge, facilitating easier access and interdisciplinary integration.
Despite the existence of current databases of scientific literature such as Scopus, Web of Science, and Crossref, which offer ways to search for research papers based on specific labels, extracting useful information about material science from the vast ocean of literature remains demanding. To obtain a clearer sense of materials properties, some structured database projects such as Materials Project (2). OQMD (3). and NOMAD [ were developed. However, these databases contain many computational results obtained through techniques like Density Functional Theory (DFT) or Molecular Dynamics (MD) simulations (5). While these computational databases can provide valuable references for predicting and understanding certain materials systems, they often face discrepancies with experimental observations. Therefore, there is an urgent need within the field of materials science for a database grounded in experimental research and practical information.
Knowledge graph (KG) is structured representation of information that models the controlled vocabulary and ontological relations of topical domain as nodes and edges, enabling complex queries and insights that traditional databases cannot easily provide. The adoption of knowledge graphs offers several advantages, including enhanced data interoperability, the ability to infer new knowledge through relational data analysis, and improved data quality and consistency through structured representation [6]. (7). These features make knowledge graphs particularly valuable for integrating diverse information sources and providing unified view of domain's 's knowledge, thereby facilitating more informed decision decision-making making and discovery (8. However, the construction of knowledge graphs in specific fields always requires the participation of a large number of experts (9). This labor-intensive process not only limits the scalability of KGs but also impacts their performance and timeliness (10). With the rapid development of natural language processing (NLP), methods for extracting information from unstructured text and constructing knowledge graphs have become more efficient and accurate (II). For instance, in 2016, the Metallic Materials Knowledge Graph (MMKG)
was developed to store materials information from various web data resources [12] Knowledge
graphs tailored lithium-ion battery cathodes have been constructed, aimed identifying potential new materials candidates [13). User-friendly databases focusing on specific material types, such as Metal- OrgaalOrganic Framework Knowledge Graphs (MOF-KG), have been developed (14). Recently, a material knowledge graph, MatKG, and MatKG2, containing information on material properties,
structure, and applications, has been developed [ 115
However, these material knowledge graphs face even greater challenges. Firstly, although advance- ments in NLP technology have reduced the dependency on experts to certain extent, training data still requires extensive annotation to enhance model accuracy [16). Secondly, the construction of these knowledge graphs often involves predicting relationships between nodes to form triples, which means the entities represented in the KG are not always based on real instances [17]. This can diminish the authenticity and credibility of the KG. Additionally, this approach makes updating the knowledge graph difficult, as each new node introduced necessitates predicting its relationship with every other node, complicating the maintenance of a dynamic and accurate knowledge graph, especially in advanced fields like material science. Acknowledging these challenges, the emergence of LLMS like GPT and LLaMA represents a breakthrough, offering new solutions enhance the zero-shot method [18]. extraction, and credibility of structured information [19], (20). The fine-tuning technique of LLMS can significantly enhance their performance in specific domain text tasks through training with fewer samples (21). (22). This means improving the results of Named Entity Recognition
(NER) and Relation Extraction (RE) without requiring a large amount of labor becomes possible and was adopted in our research.
2

---

In this paper, we have achieved significant advancements in the development of Materials Knowl- edge Graph (MKG), a pioneering graph database tailored for the field of materials science. Our contributions are highlighted in three key areas: 1) We propose a method to achieve NER, RE, and entity resolution (ER) with high accuracy. Through this method, we can easily convert unstructured text into triples and retain the source information of each triplet. This method also makes updating KG very convenient. 2) We constructed the first accurate knowledge graph dedicated to materials, where researchers can easily get information about the material by querying the MKG. 3) We use well-defined label system so our KG can be easily scaled up and potentially combined with other structured databases or KGs. 4) We demonstrate similarity calculation method based on Jaccard Similarity for materials and applications.
2 Methods
Figure [ presents the elaborated pipeline of our study. The KG construction part can be divided into there tasks ontology design, knowledge extraction, and entity resolution (ER). For domain-specific KG, the design of ontology often relies on experts in the field. This work forms an effective ontology by defining and summarizing a small number of papers through LLM. The knowledge extraction task begins with the manual annotation and normalization of the initial training dataset, annotated training set is used to finetune the LLM for NERRE tasks. Simultaneously, the inference dataset is divided into ten batches, facilitating the iterative refinement process that follows. The ER tasks are conducted
using advanced NLP technologies, including CheettttaFxtrattor (23) mat2vec (24), and our expertly curated dictionary. These steps enable the integration of information from these distinct fields into unified knowledge graph and clean the extracted data. After ER, we selectively enhance the training dataset with high-quality results to improve the model's performance in subsequent iterations. The knowledge graph is finally constructed using the normalized data from the last iteration. To complete the graph and predict potential material applications, we employ both network-based algorithms and graph embeddings. This methodology provides critical insights and recommendations for researchers the materials science domain.
MKG Pipeline
Large Language Model Finetune
Knowledge Extraction
Knowledge Graph Completion
Perform Graph Completion Through Node Similarity Algorithm
Ontological design
Task Definitions
LLMaasiined
E (4.. Af (AUR)
Deternine the core attritutee
I. R)
licecilly the prkortz ity betweer core rode attrtbutes.
Task
hoce dlvilarity alpertinn. Consteer simriarity matrrall. Compare dirriirllrrty of applcaation.
Match relatiers between core
Ontology Design
Knowledoe
Extraction
Task 2 Knowledgge Extraction
Fire
small amouun of snastated (its.
retaties ietatie Un parggea (erigrett mwi)
Interdisciplinary Materials Science Knowledge Graph Led LLM
Intagadic HBk and Et licks.
Retain sourses knowladdee.
Peovide the extracted Concepts 'n IIC formet
Task 3 Entity Resolution
A strustured derabone that incegraees ancitins and sumrrric sarge between there graph database.
Entity Resolution
Gart (H, (,)
Loe LLM oper-opurce SLP taols for duta slearing and nor malalatiion Adopt targeted methods for difeeeent abtributes KG.
Gnor (H',
Knowledge Graph
neo4i
Figure 1: Pipeline of the fine-tuned LLM for knowledge graph tasks.
2.1 Data preparation and schema design
Material experts annotated nine distinct categories from the abstracts of 75 research papers, forming the training dataset for the Large Language Models (LLMs). The structure of the data reflects the structure of the KG. This structure summarizes ten articles through LLM and extracts key attributes. As illustrated in Figure 2 (a), the "Formula", "Name" and "Acronym" node serves as the central hub within the graph, linking to nodes that encapsulate its nomenclature, composition, and various attributes. Among these core attributes, the priority of the attribute pointed to by the
3

---

arrow decreases in sequence. The "Structure/Phase" node describes the material's physical state, while the "Application" node denotes its practical uses, and the "Property" node details inherent characteristics. Additionally, the "Descriptor" node provides qualitative information enhancing the contextual understanding of the material.
Furthermore, the "Application" node is extended to include "Property", "Descriptor", and "Domain" nodes, indicating the specific attributes and the broader context of the material's application. Among them. "Domain" is unique to "Application" and represents the field to which the application belongs. To maintain data provenance and traceability, each node is linked to "Digital Object Identifier (DOI)" node. This allows for source verification where querying the intersections of "DOI" node connections at both ends of relation can pinpoint the source article from which the relations were derived, ensuring data integrity within the graph structure. Figure 2 (b) is an example of MKG, from which it can be seen that the core of MKG to capture the connection and structure between materials and their applications.
(a)
(b)
source from
Domain
Property
sseuns from
fro Somaal
Descriptor
Application
Each nodes
Formula Acronym
Calan Ceands
App Bytthei' Chanteteii
Finss
Search foo
from demsin
DOI
appliication
Cheracterization
TOY propeny
Structrrl Phass Synthesis
THos property
Figure 2: This schematic represents the (a) MKG schema and (b) an example of path in MKG between the "Name" node "Copper Indium Gallium Selenide" and "Application" node "Thin Films".
Given the inherent complexity of sentences and the variability in terminologies across abstracts, normalization process was employed after the initial extraction. This normalization ensured uniform representation of entities with similar meanings. For example, terms such as "Lithium- Ion Battery" and "Li-ion batteries" were standardized to "lithium-ion battery", while phrases like "solution casting method", "solvent post-treatment method", and "solution-based deposition" were simplified to "solution-processed ensuring consistency and facilitating effective training of LLMs. The field of materials encompasses wide range of areas. At this stage, our priority is to focus on energy materials. We downloaded 150,000 abstracts of peer-reviewed research articles on energy material science, including batteries, solar cells, and catalysts, from the Web of Science. Each abstract was stored in JSON file format, structured as "DOI- text", facilitating seamless processing and analysis.
All annotated entities underwent this normalization process,
2.2 LLMs training, evaluation and inference
The training dataset, composed of compiled data, was employed to fine-tune models including LLaMA 7b, LLaMA2 7b [25). and Darwin (26). Upon obtaining high-quality results from the normalized inference, we iteratively retrain fine-tuned LLM to infer subsequent batches of data. The models underwent training over 10 epochs with a batch size of 1. Additionally, 60 abstracts were annotated to assess the LLMSs" performance. Our evaluation primarily focused on the NER capabilities of the model and preliminarily explored the RE task's 'ss ability to identify potential internal relations among relevant element sets. Moreover, since standardized entities during the data compilation phase, we also assessed the LLMs" effectiveness in standardizing the entities. Specifically, for NER, RE, and ER tasks, we employ a unified framework for evaluation based on precision, recall, and the Fl score, taking into account the instances of false positives (FP) and false negatives (fn) to quantify the performance.
For NER, a true positive correctly identified entity, while for RE, correctly identified relationship between entities, and for ER, it is correctly standardized entity according to the schema. Conversely, a false positive occurs when the model incorrectly identifies an entity, relation, or standardization, and a false negative is when the model fails to identify a correct entity, relation,

---

schema element that should have been recognized. After defining these terms, we can evaluate each task using standard precision, recall, and Fl score metrics.
After evaluation, we chose a fine-tuned L.M that demonstrates optimal performance in both NER and RE tasks iteratively infer the 150,000 abstracts. The output of inferences is organized in the "DOI-text-
format. Consequently, the fine-tuned LLM not only extracts entities but also assigns them appropriate labels, thereby accomplishing NER and RE tasks concurrently. Moreover, every entity and relation identified in the response is traceable, enhancing the integrity and utility of the data.
2.3 Entity resolution
The quality of KG is crucial for its credibility in checking and correcting the inference results before graph construction. To ensure the precision of these results, we initially employed ChemDataExtractor to identify chemical formulas and "Name-Acronym" pairs from abstracts. Subsequently, entities recognized by both the Large Language Model (LLM) and ChemDutaExtractor are embedded using the mat2vec model. We analyze their similarities to rectify core entities and ensure accurate "Name" to "Acronym" associations. Through this step, we can make the "Name" " and "Formula" different from "Acronym" in MKG. Therefore, we have named this stage "ER-NF/A" ("Entity resolution Name/Formula and Acronym"). Given the frequent mislabeling of "Name" and "Formula", we progress to the "Entity resolution Name and Formulas" (" ("ER-N/F"" phase. Here, we refine the L.M using specifically curated training set comprising 2,000 accurately labeled entities for binary classification, which evaluated against an additional set of 200 labels.
For other labels, we implement a Density-Based Spatial Clustering of Applications with Noise [7 algorithm to create the "Entity resolution expert dictionary" ("ER-ED"). This algorithm dynamically forms clusters based on vector similarity without the need to predefine the number of clusters. Each cluster is named by material science experts, leading to the creation of an expert dictionary containing approximately 600 terms related to structures, phases, applications, and more. This dictionary, along with the entities extracted by the LLM,, undergoes similarity analysis to standardize and confirm the accuracy of both entities and relations. To elevate the quality of the training dataset, we selectively integrate high-quality data from normalized inference outputs from each iteration into the training set, continuously monitoring and enhancing the performance of the fine-tuned LLM to ensure its efficacy.
2.4 Knowledge graph construction
Given collection of entities E and relations R, a knowledge graph IC E x R structured as directed multi-relational graph. It comprises triples formatted as (h, r)) E K., where h and t are entities within E. To structure the inference results into triples, we employ three labels that signify the material as the core label, with the core label serving as the head h. the names of other labels forming the relations Rt and their values acting as tails l. Within the hierarchy of core labels, "Formula" is accorded the highest priority, followed by "Name", and then "Acronym". Furthermore, each head h and tail node is linked to the DOI of the source article associated with the triplet. This setup allows us to ascertain the provenance of the relation between any two nodes by examining the intersection in their connected entities. Then we transfer the triples into MKG and store the MKG via graph database Neo4j, which also supports the subgraph matching function, where subgraph matching naturally suits the need to search for certain materials with user-input conditions. To facilitate access to the detailed information, we also make the dataset available in the RDF and CSV format for straightforward data handling.
2.5 Graph completion
The process of Graph Completion (GC) is shown in Figure 3 (a), where we perform link prediction through GC, segmented into four primary stages: graph splitting, similarity calculation, validation and evaluation, and parameter optimization. This structured approach ensures a comprehensive exploration of the link prediction capabilities. Graph splitting is meticulously performed based on the chronological assignment of the nodes. Nodes encapsulated within defined prediction window 8 are utilized as Gver, which are designated for the validation of predictions. Nodes preceding this predictive interval serve as Gm, employed to train and refine the prediction models. During the similarity calculation stage, advanced graph algorithms and embeddings are applied on Gun to
5

---

ascertain the similarity between the "Materials" and "Applications" nodes. This phase leverages an enhanced Jaccard similarity metric, partitioned into three distinct components: S(m, a), F(m, a). and T (m,, a), which collectively improve the specificity of predictions.
(a)
War
War
Weighted Sunc S(m a) - 20)(m.)) w)/FF
G (V, El, nodes, edges
Feature
Konmon Fostl/ F(n. jeepured ('act
Predh weedow
cinaa #x
Time based Relevance cep-
Compare Similarity
G, - fv yeartri sty
Temporal Errors, Structural Errors optimize
verify
Rank Similarity
G.. VveV sy
Materials around another Application
Potential link from the potential Material
Potential from the potential Material
Similarity
Similarity calculation
Materials around target Application
Potential link to the target Material
Potential link to the target Application
Material
Property
Application Predicated Interaction
Known Interaction
Figure (a)The process of MKG graph completion and (b) the schematic diagram of nodes comparison.
In validation and evaluation phase, the sorted "Material" and "Application" nodes undergo rigorous testing on Gver- The effectiveness of the predictive model is assessed by tallying the incorrect predictions, represented as En, which serves as a crucial measure of performance. The final stage, parameter optimization, involves the meticulous adjustment of the parameters S(m, a). F(m, a), and T(m, a) based on their performance on Gver- This iterative process is crucial for minimizing the count of En and enhancing the overall accuracy of the GC method. This adaptive adjustment underscores the dynamic and responsive nature of the model in refining link prediction accuracy within the graph completion framework.
worth mentioning that from a global perspective, we not only need to consider whether an "Application" may be similar to "Material" node. We also need to consider whether the "Material" currently associated with this "Application" similar to the potential "Material". Therefore, further improvement of Jaccard Similarity:

---

n
|4, n B,l |A U B;
Jmar (A, B3)
Wi
where n is the number of attributes considered, A; and B, are the specific attributes of materials A and B respectively, and w, are the weights assigned to each attribute. signifying their importance in the context of material application. As shown in Figure 3 (b), this approach mimics the investigative processes commonly employed by materials scientists when developing new materials. In determining the suitability of new materials for specific applications, researchers systematically analyze their chemical and physical properties, well as their structural characteristics, comparing these attributes the target field. This enhanced method not only accounts
to those of materials well-established for the number of overlapping attributes but also emphasizes the significance of each attribute in relation to the material's potential application. A higher m score indicates a stronger likelihood of applicability between the material in question and the target application. Thus, materials scoring high on this metric are considered promising candidates for the specified applications.
In addition to the Enhanced Jaccard Similarity, employ the TransE model (28). TransE treats entities and relations as vectors in the same embedding space. The core idea of TransE is to model relations by interpreting them as translations in the embedding space.
3 Result
To illustrate the advantages of MKG more clearly, conducted comparative analysis with
MatKG2 [15), highlighting the differences in their construction methodologies as illustrated in Figure
4
MatKG2 is built using& multi-step process for named entity recognition (NER) and relation extraction (RE) that does not retain the origin of each relation. In contrast, our approach employs an
end-to-finish methodology utilizing single large language model (LLM) designed to handle both NER and RE simultancously. This not only preserves the source of cach triple within the knowledge graph, enhancing its factual accuracy, but also demonstrates superior performance in the RE task.
(a) MatKG2
abstracts:
Named enticy recegnition:
Enily revolutio:
Rebation extraction:
Triplex: (A
ha amazs
Re2 sanolecto expoodt (M1)) facto
Baity 1102, manbers (801) Donts viles Aopo
Tasty Et
Mexsand by siceple redocc deposatina
Zesty C: (000 locta
(6.3 D1 lecupellii
(A lea parlim Di
Tasty deporiii
(A, 72 kos rystucc
(A les, chack
Dot s0o0rd
DOC: 10000
appliction
102 growaly by socticc leyar deposiiveesal aad msed
Baity: 1100, Jusk layer
Asportie
Aeprwilins prosollle
zadby parskaasclae
(b) MKG
150 Aloraaad alsttrcci::
Namned recognition and Relation estraction:
Triples:
DOC X00CC)
1007 sanodhast copoosd (16)) Doaan stara
X000
Terndas
DOE
(TIOG, bet ersctes, (01)) focre 300001
oraites arasdben, XXX31)
simpli
7X02
has syrthcos hww.sherrdddd......m.. 33333))
Dot socec
byaar depar
Seuctor:: Scatbeii
(101) (aven))
Applcatio::
(100, kot apelicatoon paroodbtee oaber
(0000)
dependiic
pormontai odll...
Tndioturral
Figure 4: Schematic comparison of MKG and MatKG2.
To demonstrate the effectiveness of this pipeline, we evaluated the performance of each LLM on each task, as shown in Table[] Darwin significantly outperforms both the LLaMA 7b and LLaMA2 7b models in the Fl scores for NER and RE tasks, suggesting that Darwin yields more effective results in text-related tasks in materials science. However, there is no marked difference in the performance of these models on ER tasks; we can say LLM has weaker ability to complete ER in our task. This may be attributed to the limited contextual memory capabilities of the LLMs, and this is why we need to apply additional ER processes. The normailzed Darwin shows the performance of our ER and normalization process, the result indicates that it not only achieves the ER task successfully, but also contributes to the NER and RE task.
To better understand the enhancements provided by each component in the FR/Normalization process, ablation studies were performed. The outcomes, detailed in Table 2 demonstrate that each technique
7

---

Table 1: Result of NER, RE, and ER through Fine-tuned LLMs.
Model
Task
Precision
Recall
FI score
NER
0.1196
0.6869
0.2036
MatBERT
RE
0.0250
0.5696
0.0479
0.0928
0.5303
0.1579
NER 0.6101
0.6216
0.6158
Llama 7b
RE
0.5305
0.5405
0,5355
ER
0.3687
0.3757
0.3722
NER
0.7419
0.7667
0.7541
Llama2 7b
RE
0.6452
0.6667
0.6557
0.4484
0.4633
0.4557
NER 0.8013
0.7935
0.7974
Darwin
0.7036
0.6968
0.7002
ER
0.4593
0.4548
0.4571
0,9520
0.9083
0.9296
Darwin (ER)
RE 0,9039
0.8625
0.8827
ER
0.9127
0.8708
0.8913
positively affects the effectiveness of ER. The results highlight that utilizing the expert dictionary (ER-ED) is the most beneficial method, enhancing accuracy comprehensively across nearly all labels that are included in the dictionary. Additionally, the improvements seen with ER-NF/A are noteworthy; this step is based the ChemDataExtraatoor effectively removes the majority of incorrect material identifications in core labels and offers even more substantial benefits to NER than to ER-ED. Furthermore, from the results of the ablation experiment, ER-N/F appears that its contribution is not as significant as the other two components, but considering the confusion between the material Formula and Name may have impact on subsequent MKG applications. ER-N/F is also indispensable part the ER process. The normalized data transformed into triples to construct the MKG, which comprises 162,605 nodes and 731,772 edges, illustrated in the schematic diagram in Figure 5
Table 2: Result of the ablation experiment in normalization.
Method
NER F1
V
RE F1
V
ER F1
V
Darwin (Normalized)
92.96
88.27
89.13
ER-N/F
92.96
83.29
-4.98
89. 13
ER-NF/A
84.01
-8.85
83.07
-5.20
84.54
-4.59
ER-ED
88.59
-4.37
80.20
-8.07
50.00
-38.83
Finally, to demonstrate the reliability and accuracy of our similarity algorithm, divided MKG into two knowledge graphs based on the paper's 's publication year. KG, with earlier years, applied similarity calculation algorithms predict potential links, while KG, with later years, is used to verify how many sets of "aaprrialAppplicctions" result shown in Figure 6 Specifically, Figure (a). (a), the grey line uses only abstracts published before the year to make predictions, and the percentage of reported predictions in the next few years displayed. The predictive capacity of the MKG is substantial; therefore, for each prediction, only use the top 200 amplerial-application pairs for validation. Looking at the results, as time progresses, the predicted materials are gradually verified. Using data from 2014, within nine years, 48.5% of the mnaterial-apllicall effectiveness and reliability of the MKG, and also lays the foundation proposing new materials in some fields. The red, blue, and green lines are the average percentage of reported links based on network similarity, Jaceard similarity directly between "Formula" and "Application" and TransE for link prediction, respectively. Network-ber based similarity has the highest accurate prediction, which
these predictions were reported in the following years. The
predictions have been validated. This outcome demonstrates the

---

(a) (b)
Figure 5: (a) Global schematic diagram of MKG; (b) Local schematic diagram of MKG.
indicates the effectiveness of simulating the thinking of real material development using algorithms. The Figure 6 (b) is example, visualizing the KG made from data before 2018 and the predicted "Material Application" reported within 5 years using network-basee algorithms.
(a)
(b)
50
Similarity Similartty
2014
Predicted Example Using Data Before 2018 and Reported in 5 Years
reported application of material
Percentage subsequently
40
P.mc-b-Si: Optical device (Passivation layer)
AIF Thin solar cells (Conductive film)
3. Ni,Fobb:: Water splitting (HER)
30
Ni(OHNZ)-Ag-ROu splitting (HER)
20
Cu,ZnSnSeg: Perovskite solar cells (HTL)
2019
LINO; Supercapacitor (Electrolyts)
10
Nb,0g: battery (Anode)
127. Nb.0g: Lithium sulfur battery (Promoter)
128. PdAg: Water splitting (OER)
Reported Material in Application after 2018
Year after prediction
Figure 6: Validation of the graph completion. (a) Percentage of reported prediction after years. (b)
Example of predicted material application using data before 2018.
Conclusion, Discussion, and Future Work
In this paper, we present novel natural language processing (NLP) pipeline for Knowledge Graph (KG) construction, designed to efficiently extract triples from unstructured scientific texts. This methodology uniquely enables the fine-tuning of Large Language Models (LLMs) using minimally annotated datasets. The fine-tuned LLMs are then utilized extract structured information from extensive corpora of unstructured text, bypassing predictive modeling to enhance the authenticity and traceability of the structured data extracted.
Utilizing this approach, we constructed a Material Knowledge Graph (MKG) that encapsulates relations between materials and their associated entities, such as properties and applications. derived from the abstracts 150,000 peer-reviewed papers. Our analysis not only confirms the effectiveness and credibility of the MKG but also enriches our understanding of the material science domain.

---

Additionally, our methodology and the resulting KG hold significant potential across various dimen- sions:
1. Currently, our original text consists of article abstracts, which may omit substantial amount of experimental methods and data. Fortunately, our method can be directly applied to full-text extraction, allowing to more accurately capture subtle distinctions within complex scientific literature.
2. The ontology design MKG has improved the precision of entity labeling in our pipeline and facilitates more accurate categorization of data, including the incorporation of intricate attributes such as synthesis conditions or material properties. This enhancement significantly increases the granularity and practical utility of the KG.
3. The adaptability of our NLP pipeline to a broad range of scientific fields suggests its potential as Foundational template for constructing domain-specific KGs beyond materials science.
4. Integrating our Material Knowledge Graph with existing KGs, such MatKG, paves the way for the creation of a more interconnected and expansive dataset. This synergy facilitates not only advanced research but also the development of innovative applications in materials science and related domains.
For future work, given the apparent predictive capacity of our method for link prediction, we plan to focus on:
1. Analyzing Historical Patterns: By incorporating additional attributes such as authorship, publi- cation year, and affiliations into our Knowledge Graph, we focus on analyzing historical "social" patterns. For example, investigating whether specific groups or institutions consistently engage in the repurposing of materials. This analysis will allow us to understand how the dynamics of material repurposing are influenced by collaborative networks or institutional trends. We will assess which materials or types of materials are more frequently repurposed and explore the potential social dynamics that drive these trends. This approach aims to provide insights into the lifecycle and innovation patterns of materials based past activities and collaborations.
2. Identifying Future Materials and Applications: After constructing a complete MKG, in-depth graph neural networks and algorithms can be developed and used to explore potential relations in the graph. By exploring the data within our enriched MKG, we will identify new potential applications for the most promising materials or explore the future trend of materials. This proactive approach seeks to unlock new uses for materials before they become mainstream, providing a competitive edge materials science. Additionally, by incorporating time dynam- ics, we aim to enhance the utility of our pipeline, allowing for time-aware graph embedding techniques for graph completion. This will enable us to explore how material properties and applications evolve over time, providing dynamic perspective the development and use of materials in various fields.
3. Studying Cluster Formation: We plan analyze the formation of clusters within the KG, which are based on the links between materials. Understanding these clusters will help reveal how materials are interconnected, which can aid in discovering the connections between different materials that were not apparent from isolated studies.
10

---

References
[1] Vinecth Venugopal and Elsa Olivetti. "MatKG: An autonomously generated knowledge graph in Material Science". In: Scientific Data 11.1 (2024), p. 217.
[2] Anubhav Jain al. "The materials project: A materials genome approach to accelerating materials innovation, APL Mater". In: APL materials (2013).
[3] James E Saal et al. "Materials design and discovery with high-throughput density functional theory: the open quantum materials database (OQMD)". In: Jom 65 (2013), pp. 1501-1509, [4] Claudia Draxl and Matthias Scheffler. "The NOMAD laboratory: from data sharing to artificial intelligence".
In: Journal of Physics: Materials 2.3 (2019), p. 036001.
[5] David Mrdjenovich et al. "Propnet: a knowledge graph for materials science". In: Matter 2.2 (2020), pp. 480.
[6] Shaoxiong Ji et al. "A Survey Knowledge Graphs: Representation, Acquisition, and Ap- plications". In: IEEE Transactions on Neural Networks and Learning Systems 33.2 (2022), pp. 494-514.
[7] Jing Zhang al. "Neural, symbolic and neural-symbolic reasoning on knowledge graphs". In:
Al Open 2 (2021), 1) pp. 14-35.
[8]
T. Mitchell et al. "Never-ending learning". In: Commun. ACM 61.5 (2018), pp. 103-115.
[9] Lingfeng Zhong et al. "A Comprehensive Survey on Automatic Knowledge Graph Construc-
tion". In: ACM Comput. Surv. 56.4 (2023).
[10] Shirui Pan et al. "Unifying Large Language Models and Knowledge Graphs: A Roadmap". In: IEEE Transactions on Knowledge and Data Engineering (2024), pp. 1-20.
[11] L. Weston et al. "Named Entity Recognition and Normalization Applied to Large-Scale Information Extraction from the Materials Science Literature". In: Journal of Chemical Information and Modeling 59.9 (2019), pp. 3692-3702.
[12] Xiaoming Zhang et al. "MMKG: An approach to generate metallic materials knowledge graph based on DBpedia and Wikipedia". In: Computer Physics Communications 211 (2017), pp. 98-112.
[13] Zhiwei Nie et al. "Automating materials exploration with a semantic knowledge graph for Li-ion battery cathodes". In: Advanced Functional Materials 32.26 (2022), p. 2201437.
[14] Yuan An et al. "Knowledge Graph Question Answering for Materials Science (KGQA4MAT): Developing Natural Language Interface for Metal-Organic Frameworks Knowledge Graph
(MOF-KG)". In: arXiv preprint arXiv:2.30. /36/ (2023).
[15] Vineeth Venugopal and Elsa Olivetti. "MatKG-2: Unveiling precise material science ontology through autonomous committees of LLMs". In: for Accelerated Materials Design NeurlPS 2023 Workshop (2023).
[16] Peng Su et al. "Using distant supervision to augment manually annotated data for relation extraction". In: PLOS ONE 14.7 (2019).
[17] Rita Т. Sousa, Sara Silva, and Catia Pesquita. "Explainable Representations for Relation Prediction in Knowledge Graphs". In: arXiv preprint arXiv:2306. /2687 (2023).
[18] Yang Tan et al. "Retrieval-Enhaaned Mutation Mastery: Augmenting Zero-Shot Prediction of Protein Language Model".. In: arXiv preprint 24..2:2410.21122 (2024).
[19] Tom Brown et al. "Language models few-shot learners". In: Advances in neural information processing systems 33 (2020), pp. 1877-1901.
[20] Hugo Touvron et al. "LLaMA: Open and Efficient Foundation Language Models". In: arXiv preprint arXiv:2302.
(2023).
[21] Tong Xie et al. "Creation of a structured solar cell material dataset and performance prediction using large language models". In: Patterns (2024).
[22] John Dagdelen et al. "Structured information extraction from scientific text with large language models". In: Nature Communications 15 (2024), 1418.
[23] Matthew C. Swain and Jacqueline M. Cole. "CheemDaaEExtrctttt" A Toolkit for Automated Extraction of Chemical Information from the Scientific Literature". In: Journal of Chemical Information and Modeling 56.10 (2016), pp. 1894-1904.
[24] Vahe Tshitoyan et al. "Unsupervised word embeddings capture latent knowledge from materials science literature". In: NATURE 571.7763 (2019), pp. 95+

---

[25] Hugo Touvron et al. "Llama 2: Open foundation and finettunee chat models". In: arXiv preprint arXiv:2307.09286 0928880 (2023).
[26] Tong Xie et al. "DARWIN Series; Domain Specific Large Language Models for Natural Science". In: arXiv preprint arXiv:2308. 13565 (2023).
[27] Martin Ester al. "A density-based algorithm for discovering clusters in large spatial databases with noise". In: kdd. Vol. 96. 34. 1996, pp. 226-231.
[28] Antoine Bordes et al. "Translating embeddings for modeling multi-relational data". In: Ad-
vances in neural information processing systems 26 (2013).
12

---

A Appendix
A.1 Validation for MKG
To validate MKG, we extract the 500 triples that are randomly extracted except the "DOI" and "Domain" nodes, which are informative information, and checked by the experts in relevant material science. After splitting these triples apart, the annotator can obtain 1000 entities and 500 relations. The report from the annotator is shown in Table 3 The labels "Application", "Structure/Phase" "Synthesis" and "Characterization" exhibit 100% accuracy in both entity and relation. This due to the rigorous normalization of entities under these categories using our expert dictionary. However, strict normalization can also lead to certain entity loss, which is difficult avoid. In contrast, the "Descriptor" and "Property", characterized by their vast diversity and broad spectrum, apply a less stringent standardization process, which leads a certain degree of imprecision in entity and relation. But the results are still satisfactory. "Name" and "Acronym" have also achieved commendable 100% entity accuracy. However, when it comes to analyzing relations, the ChemDataExtractoo encounters certain limitations. A detailed analysis of "Name" and "Acronym" misclassifications reveals that most of these wrong entities originate from "Formula", which is due to the reasonable error of LLM"'s binary classification of "Formula" and "Name" and ChemDataExttratton Fortunately, the impact of this kind of eπror is not significant and will not bring a sigmificant influence, as fundamentally, the "Name", "Formula" and "Acronym" from same source all represent the same material.
Table Human evaluation metric on randomly selected triples.
Label
Number in total
Entity disagree
% disagree
Relation disagree
% disagree
Formula
257
N/
N/A
Name
227
12
5.3
Acronym
26
3
11.5
Descriptor
127
5.5
12
9.4
Property
147
9
6.1
Application
73
0
Structure/Phase
36
0
Synthesis
50
0
Characterization
57
0
A.2 prediction result
As Figure Jillustrates, we display the materials ranked in the top five for 2023 across various domains, alongside their rankings in previous years based available data up 2023. It is important to note that these materials-application pairs were not ranked within the top five before 2017, which is why they do appear in the figure for those earlier years. Stars mark the years these materials were featured in literature for their respective applications. The figure emphasizes the significance and emerging utility of these materials by highlighting the top five in applications such as catalysts, batteries, fuel/solar cells, and other fields from 2017 to 2023, indicating their growing importance and potential to revolutionize technology in these areas. This prediction serves as an inspiration to materials researchers, accelerating progress in material development. For example, consider the material NiFeN HC/NF; not only excels in the Oxygen Evolution Reaction (OER) but has also demonstrated its capability as a catalyst for the Hydrogen Evolution Reaction (HER). This means that NiFeN HC/NF is a promising bifunctional electrocatalyst for water electrolysis. aligning with our understanding of its potential in dual-function applications.
A.3 Paramaters for LLM fine tuning
epochs: 10 train batch size (per device): eval batch size (per device): 1 gradient accumulation steps: 2 evaluation strategy: "no" save strategy: "steps" save steps: 500 save total limit: learning rate: 2c-5 weight decay: 0. warmup ratio: 0.03 Ir scheduler type: "cosine" logging steps: tf32: False
13

---

fsdp: "full shard auto wrap" fsdp transformer layer cls to wrap: "LlamaDecoderLayer token max length: 1024
(a)
Top prediction Catatyst
Rank
prediction Burtery
Mal. night.-Gn HoN---H
opiiking
W
Mix PcoyC
Sodium
Mec IRAGCAACC
Mingin
turkeyy joeat
Mat Be daped can
Med,
App: Thernal ewey stonge
mas HecHt, vao
Van Cartarioo
Vert MJ JoA
Med, 5000
Winer sptitiing (Mtr))
App Socium butieyy poodde
2017
2010
2019
2000
2001
2012
2003
Year
Year
(c)
(d)
Top prediction FuckBoalr
Guinda,
pxpomo,
Mail MayC
Vic Auind
Malt Ag,50, App: chanden
Mac Catrutllle
Orly
Cayosset,
Cuily,
2017
2015
2019
2000
2001
2002
2323
2017
2014
2013
2038
2001
2022
2022
Year
Figure 7: Top five potential materials (a) Catalyst, (b) Fuel/Solar cell, (c) Battery and (d) Other fields, predicted on all available data (2014 2023) and their rank trends in the past years (2014- related year). The stars represent the material has been reported in the related year.
A.4 Experiments compute resources
The compute resources were primarily consumed in LLM fine-tuning, which involved using 8 GPUs for 9.02 hours with 36% utilization rate, 25.98 98 service units, and 14245 MiB of storage from quota of 81920 MiB.
A.5 Data and Code
The MKG is made publicly available hhttps:///oi.orrg. 10 618.00844/mmmm www.mmn..ighhaee2255 25997 188.v3 in both RDF and CSV format. All the code and datasets for LLM fine-tuning inference. Entity Resolution and Graph Completion can be found http://gttll.ccommmaatct/E////v/ Knowledge Graph.git
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The paper's contributions and scope are reflected clearly in the abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have mentioned that strict normalization and entity resolution processes can cause the loss of some correct entities.
Guidelines:
The answer NA means that the paper has limitations, while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when the image resolution is low, or images are taken low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to addressing problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not to penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer:
[Yes]
Justification:
For the theoretical part, proof has been provided.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in the appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We have placed parameters for LLM fine-tuning, other information like code and prompt will be made public if the article is fortunate enough to be accepted.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The dataset, MKG and code have been included in the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ips.cc/ppili wttpcccpppppppppeeeeeeeeeeeeeeee httpa///ggiieee/eeeSebiiisssssss /Cc////oddiiisssseeeeeee ssiiiiiisiyyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: We have included some of the information, the remaining information will be reflected in the code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
All the the results are accompanied by error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have mentioned this.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We strictly follow the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Promoting the development of materials science and scientific advancement.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All these used in paper mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The new assets introduced in the paper well documented and the documentation provided alongside the assets
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
20


Reviews:
[{"review_id": "0rG34Jz0hC", "reviewer": "Reviewer_jF92", "summary": "The paper on constructing and applying a materials knowledge graph (MKG) in multidisciplinary materials science via a large language model (LLM) is valuable and well-written. However, it primarily focuses on application rather than strong technical contributions, with issues in experimental design, lack of non-LLM baselines for comparison, and insufficient comparison with sophisticated knowledge graph completion methods. It may be a good dataset track paper but not the main track.", "questions": "Please refer to W2 and W5", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The studied problem is of great value in the real world\n- The paper is well-written and easy to follow\n- The produced MKG could be very helpful for the computational and experimental material science", "weaknesses": "- W1: This paper is towards applications of LLMs in an engineering flavor instead of having strong technical contributions. The process includes prompt engineering, basic model fine-tuning, and human-in-the-loop entity resolution. And this paper looks like an extended application of Darwin, instead of an independent research work.\n- W2: The experimental design is flawed. In particular, why the normalization is only applied to Darwin? Is it possible that other base models + normalization can perform better? And is the normalization only working well for Darwin?\n- W3: Non-LLM baselines for NER and RE should be included for comparison.\n- W4: The modified Jaccard similarity method is claimed as a specified KG completion algorithm for material science. Therefore, the experiments should include comparisons with more sophisticated KG completion methods, instead of only comparing with TransE, which is outdated.\n- W5: Minor issues include but are not limited to: Typos in Figure 4, what is FMKG?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude for the thorough review and valuable feedback on our manuscript. Your insights are highly appreciated and have been carefully considered to enhance our work. We also want to emphasise some points in the paper to answer your question.\n\n**W1 Response:**\n\nWe would like to thank the reviewer's comment. As reviewers xvtf and is27 have noted, this paper emphasises a novel process for constructing a materials science knowledge graph. The novelty of our work lies in integrating state-of-the-art LLM to perform NERRE simultaneously and for continuous and dynamic updating of the knowledge graph, allowing for real-time integration of new research findings directly into the MKG. The challenges include the potential for LLM-generated hallucinations and biases during the knowledge extraction process, which necessitates sophisticated normalisation and entity resolution processes to maintain the integrity and credibility of the knowledge graph. Additionally, it is crucial to emphasise that this is the first knowledge graph in the materials science domain that achieves multi-level connections for complex material science knowledge, representing a significant advancement over previous models that were limited to merely binary connections. For instance, instead of linking an alloy’s composition directly to its mechanical properties, MKG includes intermediate nodes such as processing techniques and microstructural features. This allows for a deeper exploration of the complex interdependencies that affect an alloy's properties, offering an unprecedented level of detail in modelling material interactions. \n\nSolving STEM problems through LLMs may seem like an engineering flavour, but it is indeed a research direction encouraged by *NeurIPS*, such as the work of Hu et al. [1] and Lu et al. [2]. Compared with the biomedical field, the rarity of KG applications in material science is primarily due to the absence of effective knowledge graphs that can comprehensively capture the intricate data and complex relations inherent in these fields. Different from existing material science knowledge graph construction methods, such as the MatKG series, we utilise LLM for knowledge extraction, ensuring authenticity and traceability of the information. In other words, this work is not an extension of Darwin, although Darwin's performance is the best among several open-source models we compared.\n\n[1] Hu, Xiuyuan, et al. \"De novo drug design using reinforcement learning with multiple gpt agents.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[2] Lu, Pan, et al. \"Learn to explain: Multimodal reasoning via thought chains for science question answering.\" Advances in Neural Information Processing Systems 35 (2022): 2507-2521.\n\n**W2 Response:**\n\nIn this study, the application of LLMs is primarily focused on the knowledge extraction part, where we finetuned LLMs on a small set of annotated data to effectively perform entity and relation extraction. The following normalisation process involves cleaning and standardising NERRE result. Therefore, once we identified the LLM with the best performance in knowledge extraction (i.e., Darwin for this task), we concentrated on normalising its output. Theoretically, normalisation is applicable to any LLM that can successfully perform knowledge extraction. However, in our study, we opted for the best-performing model for in-depth analysis based on considerations of efficiency and clarity.\n\nWe have also revised the caption to make our ideas more straightforward and avoid misunderstandings: “*Table 1:* Comparative results of NER, RE and ER across different models using fine-tuned LLMs and non-LLMs. The Darwin model, which demonstrated the highest overall performance, was selected to showcase the effects of subsequent normalisation.”\n\n**W3 Response:**\n\nInitially, we focused on leveraging LLM due to their advanced capabilities in handling the complex semantics of scientific texts, which is critical for both NER and RE. Besides, the low labour cost required for fine-tuning is also an indispensable factor in building an automated pipeline. In contrast, identifying the relations by non-LLMs requires a large amount of accurately annotated data for training, which could not be efficient when used for a dynamic knowledge graph that includes state-of-the-art domain knowledge. The absence of non-LLM models in our initial baseline was due to this strategic focus rather than an oversight. In response to the reviewer's insightful feedback, we have incorporated MatBERT and MatKG's construction methods. The results of this expanded baseline comparison will be detailed in our next revision and we have put the result in global response for your reference.\n\n**W4 Response:**\n\nWe would like to clarify that the primary focus of our research is leveraging LLMs for KG construction, which addresses issues during the construction process rather than on graph completion itself. The purpose of employing graph completion techniques is to demonstrate the quality, credibility, and effectiveness of the KG we constructed. In other words, using both Jaccard and TransE was not intended to compare their effectiveness; instead, our goal was to explore different methods to enhance the credibility of these steps. Choosing these well-known methods ensures that the study remains approachable and comparable. We opted to modify the basic Jaccard algorithm due to the suboptimal performance of the standard similarity algorithm, as depicted in Fig 6. This modification reflects a common practice in materials science research, where researchers often select trending materials from fields aligned with their research areas as potential candidates for current applications. Our modifications, simulating this process, were minor yet crucial to better suit the needs of the MKG.\n\n**W5 Response:**\n\nWe apologise for this typo, and thanks for pointing out this problem. it was revised to “MKG”.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2HqRpIFrwf", "reviewer": "Reviewer_xvtf", "summary": "The study presents an innovative pipeline for Knowledge Graph (KG) construction, specifically designed for efficient extraction of triples from unstructured scientific texts. The methodology enables fine-tuning of Large Language Models (LLMs) with limited annotated datasets, which is then utilized to extract structured information from extensive corpora of unstructured text. The authors have constructed a Material Knowledge Graph (MKG) that captures relationships between materials and their associated entities, such as properties and applications, derived from abstracts of 150,000 peer-reviewed papers.", "questions": "Does the inclusion of DOIs affect the resource consumption such as memory or performance in any noticeable way?\n\nIt is common practice to begin with abstracts. Do the authors intend to extend their work to encompass full texts in the future?", "limitations": "The authors did addressed the limitations, and mentioned that strict normalization and entity resolution process can loss some correct entities.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The paper introduces a novel pipeline for KG construction that departs from predictive modeling, enhancing the authenticity and traceability of extracted structured data.\n\nQuality: The authors demonstrate the effectiveness and credibility of the MKG through ablation experiments and similarity analyses based on node similarity and graph embedding. The results indicate the substantial predictive capacity of the MKG, with 48.5% of 'material-application' predictions validated within nine years, which is impressive.\n\nClarity: The paper provides a clear and comprehensive explanation of the methodology, including the fine-tuning of LLMs, extraction of structured information, and construction of the MKG. Detailed results and analyses support the authors' claims.\n\nSignificance: The MKG has significant potential in extending the depth of structured information extraction, improving entity labeling precision, and adapting the pipeline to other scientific fields.", "weaknesses": "Some figure captions, like \"Fig 1(a)\" and \"Fig 3,\" are unclear. Captions would benefit from additional context.\n\nCertain acronyms, such as \"ER-NF\" on line 239 and \"NER” / “RE\" on line 80, appear before being defined.\n\nTypo on line 140: \"task\"s->task’s.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude for the thorough review and valuable feedback on our manuscript. In this text, we will address each of your comments in detail, clarifying how we have addressed or plan to address the concerns raised.\n\n**Question 1: Does the inclusion of DOIs affect the resource consumption such as memory or performance in any noticeable way?**\n\nResponse: We would like to thank the reviewer's comment. The inclusion of DOI nodes indeed increases the total number of nodes within the knowledge graph, which in turn marginally elevates memory usage. However, the increase is manageable and does not stress modern hardware. We have performed stress tests to confirm that the additional memory requirements are well within the capabilities of modern hardware systems (Actually, the entire MKG has only 70 MB when converted into RDF format).\nRegarding performance, the knowledge graph is a relational structure database that enables selective interactions between nodes. For example, during operations such as calculating the shortest paths, nodes connected by the 'Sourcefrom' relation can be selectively excluded from computations. This strategic exclusion eliminates any potential adverse impacts on the graph's performance.\nAdditionally, we have other strategies to mitigate the influence of the increase in nodes, such as storing DOI nodes and other nodes separately. The subgraph of DOI is only queried when users specifically require information related to these DOIs, ensuring efficient data management and system performance.\n\n**Question 2: It is common practice to begin with abstracts. Do the authors intend to extend their work to encompass full texts in the future?**\n\nResponse: We would like to thank the reviewer for their comment. We intend to extend our work to encompass full texts, which is an important part of our future work. In addition to broadening the scope of our analysis, we plan to refine our ontology design to include more comprehensive information. This will involve selectively extracting and standardising quantitative data from scientific articles. We also aim to implement a hierarchical approach to manage the complexity and enhance the efficiency of information extraction from full texts. For instance, we could initially categorise the text into sections such as 'Introduction,' 'Methods,' 'Results,' and 'Discussion.' This segmentation allows us to apply specific extraction techniques tailored to the information typically found in each section, streamlining the process and improving accuracy.\n\n**Weakness:**\n\nWe thank the reviewer for highlighting the issues with our figure captions and typo errors. We have made the necessary corrections throughout the manuscript. For instance, the original caption for Fig 3 is replaced by: \"**Figure 3:** The process of network-based graph completion, illustrating how nodes are categorised into Materials, Properties, and Applications. The diagram on the left shows that old and new Materials share similar Properties and similar Applications. As depicted in the centre diagram, this shared attribute implies a degree of similarity between the old and new Materials - similar characteristics and applications. Consequently, as shown in the right diagram, old Materials can potentially be utilised in new Applications.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LdEMPR0vVI", "reviewer": "Reviewer_is27", "summary": "This paper presents an innovative way on leveraging the power of Large Language Models for the construction of a Material Knowledge Graph (MKG) and link prediction. The method includes annotating few scientific articles (abstracts) related to material science which are used for training and finetuning LLMs. After that step and using additional articles and a finetuned LLM, triples are generated. Entity resolution is performed by using different Natural Language Processing (NLP) techniques such as ChemDataExtractor, mat2vec and an expert dictionary. Finally, the MKG is constructed and used for link prediction with the aid of network-based algorithms and graph embeddings. This approach is compared with another technique called MatKG2 and experiments were conducted for finding the LLM that provides the best results and for evaluating the link prediction of the MKG.", "questions": "Limitations of this work are not detailed in the paper.\n- What would you consider limitations of this work?\n- How would you ensure that MKG will be up-to-date with the state of the art in material science?", "limitations": "There are some concerns about the scalability and maintainability of MKG which are not addressed in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "•\tThis is an innovative work for the fast and automatic construction of a Material Knowledge Graph with minimal annotation that could have a broad impact in the advancements of material science. \n\n•\tIt reuses effectively new technologies such as Large Language Models and other NLP techniques.\n\n•\tIt does not require too many annotated documents or other manual tasks.", "weaknesses": "•\tMore experiments would have supported better this work. The result of this method could have been compared with some baseline experiments of simply using LLMs for the triples generation. Moreover, since this method is compared with MatKG2, it would have been useful to compare the results of precision, recall and F1 score of MatKG2 with the results of MKG. \n\n•\tWhile the work seems quite interesting it not well written and several typos and syntactic errors have been found which are detailed below. Proof-reading would have been beneficial before submission.\n\n- Line 64: “ A user-friendly databases.. “ -> “User-friendly databases..”\n\n- Line 80: Acronyms NER and RE are used but they are introduced later in the text (line 85)\n\n- Line 89: “… through query the MKG.” -> “… through querying the MKG.”\n\n- Line 93: “the elaborate workflow” -> “the elaborated workflow”\n\n- Line 98: “NERRE” -> I believe this refers to NER and RE\n\n-\tFigure 1 (b): ChemDataExactor -> ChemDataExtractor\n\n- Figure 4: FMKG -> the material knowledge graph has been referred in the whole paper MKG. The acronym FMKG is introduced only in this figure.\n\n- Line 233: “indicate”->”indicates\", “achieve”->”achieves”, “contribute”->contributes”.\n\n- Line 240: “ChemDataExactor” -> ”ChemDataExtractor”\n\n- Line 252: “The result shows in.. “ -> “The result is shown in…”\n\n• Figure 3 and network-based link prediction is not well explained in the article. \n\n• The code and Knowledge Graph is not available for further evaluation. The authors state that they will only be available if the paper gets accepted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude for the thorough review and valuable feedback on our manuscript. We will answer all your questions and concerns, clarifying how we have addressed the concerns raised.\n\n**Question 1: What would you consider limitations of this work?**\n\nResponse: We would like to thank the reviewer for their comment. The limitation of the work is that the entities and relations in the Entity Resolution task are missing. As noted in the 'Appendix,' we prioritise accuracy in our knowledge graph, and thus, we implement rigorous normalisation processes in ER. This approach may inadvertently lead to the wrongful exclusion of some correct entities. To mitigate this, we are planning for a multi-tier knowledge graph approach. In this approach, entity processing is staged across multiple levels. Initially, entities are processed with broader, less restrictive criteria. As they progress through the system, increasingly stringent criteria are applied at each level to refine and purify the entity data. Additionally, entities that are initially filtered out undergo a secondary review process using LLMs, which helps determine whether to reintegrate them or permanently exclude them. This staged, iterative process aims to balance accuracy with coverage, minimising wrongful exclusions while maintaining the integrity of our knowledge graph.\n\n**Question 2: How would you ensure that MKG will be up-to-date with the state of the art in material science?**\n\nResponse: We would like to thank the reviewer for their comment. We agree that this is a crucial aspect of our future work. To ensure that MSKG remains current with the state of the art in materials science, we have implemented several strategies for continuous updates. These include automated monitoring of new publications using predefined keywords and topics, conducted periodically, and semi-annual manual reviews by domain experts to validate and refine the graph content. Additionally, we are incorporating advancements in LLMs, particularly the development of Agents. These agents not only add new data but also critically review and correct existing information by identifying discrepancies between the updated graph and previous graph. This dual approach of addition and correction significantly enhances the accuracy and relevance of our knowledge graph.\n\n**Weakness: More experiments would have supported better this work...**\n\nResponse: We have tried to add several naive LLMs into baseline experiments, However, with fine-tuning, LLM performs better, but also cannot perform NER and RE stably according to the expected format. That's why we gave up adding simple LLM. To further elaborate on the reasons, we cited a work that can prove this viewpoint and have made modifications in the article: “We evaluated the performance of each LLM on each task…” → “We evaluated the performance of each LLM across various tasks. Xie et al. *(Tong Xie, Patterns)* have demonstrated by comparing GPT-3.5 with the fine-tuned LLaMA that the latter significantly outperforms the naive models. Even under more lenient manual evaluation conditions for GPT-3.5, a noticeable performance gap persists between it and the fine-tuned LLaMA. This evidence aligns with our findings.” Therefore, we exclude the naive LLMs from the evaluation baseline.\nThe purpose of comparing with MatKG2 is to highlight the traceability of knowledge in MKG, reflecting the optimisation of the process. Given that the relation in MatKG2 is predicted, we believe a quantitative comparison of the construction methods between MatKG2 and MKG would be unfair. Moreover, the data and code of MatKG2 is not available online. However, we fully acknowledge the reviewer's suggestion. Therefore, we have included the evaluation of MatBERT for knowledge extraction which is core of the MatKG construction, and relevant discussion in the baseline. We think this additional experiment can not only provide a comparison between the MKG and MatKG series but also provide a comparison between the LLM and non-LLM pipelines. The results of this expanded baseline comparison will be detailed in our next revision and we have put the result in global response for your reference.\n\n*Xie, Tong, et al. \"Creation of a structured solar cell material dataset and performance prediction using large language models.\" Patterns 5.5 (2024).*\n\n**Weakness: Typo and code.**\n\nResponse: We thank the reviewer for pointing out the typo. We have carefully checked the paper again to ensure that all the typos and syntactic errors have been modified. All the code and data are open-source, and the link will be added once this work is accepted.\n\n**Weakness: Extension for the Fig 3 and network-based link predication.**\n\nResponse: We would like to thank the reviewer's comment. We have replaced the original caption for Fig 3 with a new caption: \"**Fig 3:** The process of network-based graph completion, illustrating how nodes are categorised into Materials, Properties, and Applications. The diagram on the left shows that both old and new Materials share similar Properties as well as similar Applications. As depicted in the centre diagram, this shared attribute implies a degree of similarity between the old and new Materials - similar characteristics and applications. Consequently, as shown in the right diagram, old Materials can potentially be utilised in new Applications.\"\n\n**Limitation: There are some concerns about the scalability and maintainability of MKG which are not addressed in the paper.**\n\nResponse: Our current work aims to find an outstanding pipeline to construct the KG, so we have relatively simplified some content, such as focusing on the abstract of papers. In future work, the scalability and maintainability of the knowledge graph are important tasks, including but not limited to broadening the ontology, including quantitative data, integrating with existing materials science databases, and dynamically updating the knowledge graph.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DX9j2GMASD", "reviewer": "Reviewer_5h8v", "summary": "This paper introduces the Materials Knowledge Graph, a pioneering graph database designed for materials science. It leverages advanced NLP methods and LLMs to extract and organize a vast amount of high-quality research into structured triples. It streamlines the discovery process by organizing information into nodes and edges, enhancing data integration and reducing the need for traditional experiments. The MKG also employs algorithms to predict material applications, offering a significant advancement in accelerating materials research.", "questions": "1.\tHow can your methods be adapted for other scientific fields? Are there specific modifications needed?\n2.\tWhat strategies do you have for continuously updating and curating the MKG to ensure its relevance and accuracy?", "limitations": "The authors have made a commendable effort in addressing the limitations of their work. They acknowledge the dependency on manual annotation for data preparation, which could limit scalability and timeliness. \nHowever, the paper could benefit from a more detailed discussion on maintaining the currency and accuracy of the MKG as the field of materials science evolves. Continuous updates and curation will be crucial, and outlining specific strategies for this would strengthen the paper.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1.\tThis paper demonstrates a thorough and systematic methodology, including data preparation, model training, entity resolution, and graph construction, ensuring a robust and credible knowledge graph.\n2.\tThe MKG shows a cutting-edge approach to parsing and structuring vast amounts of scientific literature, offering a significant advancement in accelerating materials research.\n3.\tThe application of link prediction algorithms for predicting material applications is a robust method for identifying new potentials in the field.\n4.\tExperiments shows the effectiveness of the MKG and the employed models, enhancing the credibility and reliability of the results.", "weaknesses": "1. As the field of materials science evolves, maintaining the currency and accuracy of the MKG could become increasingly complex, requiring continuous updates and curation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude for the thorough review and valuable feedback on our manuscript. Your insights are highly appreciated and have been carefully considered to enhance our work. In this text, we will address each of your comments in detail, clarifying how we have addressed or plan to address the concerns raised.\n\n**Question 1: How can your methods be adapted for other scientific fields? Are there specific modifications needed?**\n\nResponse: We appreciate this insightful question. Constructing a knowledge graph requires a well-defined ontology and a practical knowledge extraction method. To adapt our approach to other scientific fields, one primarily needs to design the ontology of the graph. This involves defining the types of nodes and the relations between nodes, as well as annotating a dataset. About 50 data based on our training template should suffice to establish a relatively accurate domain-specific knowledge graph.\nIn the ER process, it is advisable to utilise an embedding model particularly suited to your domain. For material science, where properties and applications are deeply influenced by complex material characteristics, additional enhancements should include integrating specialised embedding models tailored to capture these detailed interactions. These models enhance ER by embedding all entities within the materials science context, thus ensuring more accurate and contextually relevant data processing. Notably, while these enhancements are specialised for materials science, they build upon standard foundational models, employing pre-trained frameworks such as word2vec - mat2vec, BERT - MatBERT. These models not only meet the intricate demands of materials science, but also retain the flexibility to be applied broadly across diverse scientific domains. This ensures that our approach, while adaptable to the unique demands of materials science, remains universally applicable. In other words, when there are no suitable domain-specific models, the base models are also applicable.\n\n**Question 2: What strategies do you have for continuously updating and curating the MKG to ensure its relevance and accuracy?**\n\nResponse: Response: We thank the reviewer for their comment. We acknowledge the critical importance of keeping the MSKG up-to-date with the latest advancements in the field. To achieve this, we have implemented several strategies for continuous updates. These include the automated monitoring of new publications using predefined keywords and topics, conducted periodically, along with semi-annual manual reviews by domain experts who validate and refine the graph's content. Additionally, we are harnessing advancements in Large Language Models, particularly the development of intelligent agents. These agents actively participate not only in adding new data but also in critically reviewing and correcting existing entries by identifying discrepancies between the updated graph and previous versions. This dual strategy of addition and correction plays a vital role in enhancing the accuracy and relevance of our knowledge graph.\n\nAdditionally, to further optimise the pipeline and reduce costs. We also focus on technological innovations that streamline the process of KG construction. Our approach includes automatically constructing an expert dictionary for entity resolution and leveraging clustering algorithms to identify central nodes and pivotal connections within the graph. This method facilitates a more automated and efficient process. Automating these crucial steps significantly reduces reliance on manual curation while ensuring that our knowledge graph remains robust and comprehensive.\n\n**Supplementary explanation:**\n\nWe greatly appreciate the reviewer's insights and comments. As mentioned by the reviewer, further enhancing the currency and accuracy of MKG is an essential goal for our future research. With sufficient currency in the knowledge extraction process, our next focus will be on improving entity resolution (ER) by replacing the machine learning model with Large Language Models (LLMs) to further improve the currency of the method. In addition, due to the existence of LLM, the amount of data required for the training set has been significantly reduced. We further use an active learning strategy to create a new training set through each round of knowledge extraction, further reducing labour costs.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our sincere appreciation to all the reviewers for your valuable feedback on our work, and we have responded to all your questions (in the corresponding rebuttal sections). We also add some supplementary experimental results in the *pdf* file, mainly to reproduce the KG construction method used by MatKG. Since this method is centered around MatBERT, we think this additional experiment can not only providea comparison between the MKG and MatKG series but also provide a comparison between the LLM and non-LLM pipelines.

Meta-review: This paper presents an approach for using LLMs to construct a Material Knowledge Graph (MKG). The review point out multiple strengths of the paper, including real-world value of the studied problem, the innovative KG construction approach, the systematic methodology used and the use of link prediction in this context. Weaknesses revolve around the paper potentially being light on technical contributions. Overall, the strengths outweigh the weaknesses and I recommend acceptance of the paper.
Decision: Accept (poster)