HoneyBee: Progressive Instruction Finetuning of Large Language Models for Materials Science
Authors: Yu Song
Santiago Miret
Huan Zhang
Bang Liu
Venue: EMNLP 2023 Findings (2023)
Keywords: materials science
LLaMa
instructions based finetuning
progressive finetuning
feedback based instructions
TL;DR: We extract domain relevant instructions from large language models (MatSci-Instruct) in a trustworthy manner to finetune HoneyBee, a large language model specialized to materials science.

Abstract: We propose an instruction-based process for trustworthy data curation in materials science (MatSci-Instruct), which we then apply to finetune a LLaMa-based language model targeted for materials science (HoneyBee). MatSci-Instruct helps alleviate the scarcity of relevant, high-quality materials science textual data available in the open literature, and HoneyBee is the first billion-parameter language model specialized to materials science.  In MatSci-Instruct we improve the trustworthiness of generated data by prompting multiple commercially available large language models for generation with an Instructor module (e.g. Chat-GPT) and verification from an independent Verifier module (e.g. Claude). Using MatSci-Instruct, we construct a dataset of multiple tasks and measure the quality of our dataset along multiple dimensions, including accuracy against known facts, relevance to materials science, as well as completeness and reasonableness of the data. Moreover, we iteratively generate more targeted instructions and instruction-data in a finetuning-evaluation-feedback loop leading to progressively better performance for our finetuned HoneyBee models. Our evaluation on the MatSci-NLP benchmark shows HoneyBee's outperformance of existing language models on materials science tasks and iterative improvement in successive stages of instruction-data refinement. We study the quality of HoneyBee's language modeling through automatic evaluation and analyze case studies to further understand the model's capabilities and limitations. Our code and relevant datasets are publicly available at https://github.com/BangLab-UdeM-Mila/NLP4MatSci-HoneyBee.

HoneyBee: Progressive Instruction Finetuning of Large Language Models for Materials Science
Yu Song', Santiago Miret Huan Zhang', Bang Liu l
University of Montreal/ / Mila Quebec AI, 'Intel Labs 'University of Waterloo bang. liu)@umontreal. craiiagg.mireteinte..cmmm com ho44zhannuwaatrrloo.co
Abstract
derstanding of novel material systems, which will be crucial for addressing contemporary societal challenges like climate change and drug discovery. The potential impact of NLP materials science is chiefly underpinned by the vast reservoir of ma- terials science knowledge contained in text-based resources, such as textbooks, scientific journals, and assorted reports. In spite of the prospective richness of materials science textual data available from diverse sources, number of challenges con- tinue to significantly hinder the effective digestion and comprehension relevant materials science textual knowledge (Song et al., 2023; Kononova et al., 2021). Some of the challenges relate to the general availability of data, while other relate to the ability to effectively process domain-specifie information, such chemical notation and data contained in figures and tables (Gupta et al., 2023). This scareity of readily accessible, high-quality text corpora suitable for efficient language model training has in turn slowed the development of com- prehensive language models capable of spanning the extensive conceptual range within the highly interdisciplinary materials science field.
We propose an instruction-basse process for trustworthy data curation in materials science (Mattcii-nnttruet, which then apply fine- tune a LLaMa-bassee based language model targeted for materials science (HoneyBee). MatSci- Instruct helps alleviate the scarcity of rele- vant, high-quality materials science textual data available in the open literature, and HoneyBee is the first billion-paranie parameter language model specialized to materials science. In MatSci- Instruct we improve the trustworthiness of gen- erated data by prompting multiple commer- cially available large language models for gen- eration with Instructor module (e.g. Chat- GPT) and verification from an independent Ver- ifier module (e.g. Claude). Using MatSci- Instruct, construct dataset of multiple tasks and measure the quality of our dataset along multiple dimensions, including accuracy against known facts, relevance to materials sci- ence, as well as completeness and reasonable- ness of the data. Moreover, we iteratively gen- erate more targeted instructions and instruction- wewpriinn-evlaaatoonneeeeee
data a leading to progressively better performance for our finetuned HoneyBee models. Our evalu- ation on the MatSci-NLP benchmark shows HoneyBee's outperformance of existing lan- guage models on materials science tasks and iterative improvement in successive stages of instruction-data refinement. We study the qual- ity HoneyBec's language modeling through studies
loop
While data availability remains an ongoing chal- lenge in applying modern NLP tools for materials science, recent advancements have led to the emer- gence of large language models (LLMs) proficient in handling general language tasks that concur- rently demonstrate substantial aptitude in areas like chemistry and materials science (Bran et al., 2023; Boiko et al., 2023). Such advancements provide the potential to harness the implicit knowledge encap- sulated in these models, which have been trained on vast text corpora spanning broad range of subjects, to generate accessible, instruction-based datasets for specialized domains like materials science.
automatic evaluation and analyze to further understand the model's capabilities and limitations. Our code and relevant datasets are publicly !vailable.'
1 Introduction
Natural language processing (NLP) holds consid- erable promise in expediting the discovery and un-
Yet, while we can generate targeted instruction- based data to make applying NLP for materials science more accessible, the quality of these in- structions requires rigorous evaluation before be-
Equal advising.
Corresponding author: Canada CIFAR AI Chair.
https://gthub..om//angg. hat.://annggaa--Uaaaaa Sh--ddMM-MilaYY P4Matšci- HoneyBcc

---

MatSci-Instruct Exumple
broader knowledge base while effectively perform- ing pertinent materials science language tasks. This paper seeks to concurrently address the pre- viously outlined challenges of trustworthy instruc- tion generation and capable, open-source language models for materials science. We propose MatSci- Instruct to generate reliable, instruction- data from large language models. This data is then used to train HoneyBee, hallion-parameter spe- cialized materials science language model based on the LLaMa architecture (Touvron et al., 2023). The key contributions of our research are as follows:
Instruction:
How the ME coupling down the 2D limit verified in the study?
Input: [..] In particular, stripy antiferromaguets often exhibit broken crystal and magnetic symmetries, which may bring about the magnetoelectric (ME) effect and enable the manipulation of intriguing properties and functionalities by electrical means. L-.] This work reports the ME effect in the 2D stripy antiferromanntic insulator C:OCI down to single layer. By measuring the lumneling resislance of CrOCl on the purumeter space of temperature, magnetic field, and applied voltage, we verified the ME coupling down to 2D limit and unraveled its mechanism. [...]
Output: The ME coupling down to the 2D limit was verifled by measuring the tunneling resistance of CrOCI on the pacameter space temperature, magnetic field, applied voltage in the study.
MatSci-Instruet- - A Two-Step Framework for Trustworthy Instruction-Datt Gener- ation: We propose a universally applicable methodology suited for instruction gen- cration in scientific domains. Mastci-Instruct generates specialized instruction-dars using two-step framework Generation and Verifi- cation. In the Generation step, an instructor model (Chat-GPT 2) creates domain-specific instruction-dail focused on materials science. During the Verification step, the instruction- data are cross-verified by separate verifier model (Claude 3) for accuracy and relevance shown by the example in Figure 1. More- over, in Section 4.1 conduct human evalu- ations that suggest good alignment of our gen- erated MatSci-Instruc Dataset with human experts across several dimensions: accuracy against known facts, relevance to matcrials science, and the completeness and reasonable- ness of the language model output.
Accuracy 90/100
Relevance 100:100
Completeness 90/100
Reusonableness
90/100
Figure I: Example instruction generated by the MatSci- Instruct process used to train the HoneyBee language model that contains general language knowledge and is specialized in materials science. The relevant text to correctly answer the instruction is highlighted. MatSci- Instruct follows structured instruction generation tem- plate and ensures instruction quality through an iterative verification loop described in Section 3.1.
ing utilized for language model training. This is particularly salient in the context of complex sci- entific applications like materials science, which encompasses wide range of subfields that together describe the properties and behavior of matter that make up materials systems. This need for trust- worthy and pertinent instructions necessitates the creation of a robust process to validate the quality of instructions for downstream applications.
HoneyBee A High-Performance LLaMa- Based Model Progressively Trained via MatSei-Instruet: Utilizing the MatSci- Instruct two-step framework, we apply a Progressive Definement-Feedbbee strategy to finetune a LLaMa model, culminating in the HoneyBee model. In our progressive finetun- ing strategy, the HoneyBee model's perfor- mance on MatSci-Instruct data guides subse- quent instruction-dan generation. This itera- tive process results in further refined instruc- tions to generate qughityquality instruction- data for finetuning, ensuring the progressive acquisition of specialized knowledge by the
Aside from data scarcity in scientific domains, another significant impediment to the application of NLP in materials science is the limited presence of specialized language models that incorporate both in-depth materials science knowledge and a robust understanding of general language. The bulk of today's s available language models for materials science are built on the BERT architecture (Gupta et al., 2022; Walker et al., 2021; Huang and Cole, 2022), whose performance in general NLP tasks has been superseded by several more advanced lan- guage model architectures in recent years (Touvron al., 2023; Scao al., 2022: Brown et al., 2020; Chung et al., 2022). This highlights the need for the development of more capable language mod- els in materials science that can accommodate a
hhttps://platfon openai. www.aa..co//ooc...... com//ocs// api-reference rehereee//eeetat
"https://docs.annhreee httproppp...cocccaapeeeeeeeeeeae com/claude//docs

---

model. We evaluate the performance of Hon- eyBee using materials science language benchmark (Song et al., 2023), and thoroughly analyze its strengths and limitations.
proposed (Mangrulkar et al.. 2022), such P- Tuning (Liu al., 2021), Prefix Tuning (Li and Liang 2021), Prompt Tuning (Lester et al.. 2021), and LoRA (Hu et al., 2021). Among these, LoRA utilizes low-rank matrix decomposition to limit the additional parameters required for fine-tuming, For data curation in specialized fields, instructions- based fine-tuning extracts detailed data directly from LLMS (Ouyang et al., 2022), reducing human annotation effort and providing scalable solutions. For example, Alpaca (Taori et al., 2023; Wang et al., 2022) exploits LLMs to generate synthetic instructions for model finetuning. However, LLM- synthesized data still suffer from data quality issues, which is especially critical for scientific domains. To address these concerns, we design a generation- verification strategy for trustworthy data generation and a progressive refinementffeedaae strategy for finetuning LLMs on specialized instructions.
2 Related Work
Large Language Models Large Language Mod- els (LLMs) have gained substantial attention from the NLP research and wider technology communi- ties due to their remarkable proficiency in language understanding and generative tasks. Pioneers like GPT-3 (Brown et al., 2020), with its 175 billion parameters, demonstrated the capacity to capture complex linguistic patterns, and subsequent mod- els like Gopher (Rae et al., 2022), GLM (Zeng et al., 2022), PaLM (Chowdhery et al., 2022), BloomZ. (Scao et al., 2022), Chincilla (Hoffmann et al., 2022), and OPT (Zhang et al., 2022) continue to drive progress. Commercial models like Chat- GPT (OpenAI, 2022) and Claude (Bai et al., 2022) further expand the landscape of performant LLMs. Compared to commercial LLMs, LLaMa (Touvron al., 2023) stands out for its greater accessibility and good performance, offering an efficient and accessible platform for domain-spectin finetuning in various domains, including materials science.
3 Method
Our work consists of two interacting components:
1) MalSci-Insiruct: a trustworthy instruction gen- eration framework for obtaining scientific textual data from LLMs; 2) HoneyBee: a materials science LLM progressively finetuned from LLaMA (Tou- vron et al., 2023) using MatSci-Instruce generated data. We connect HoneyBee to MatSci-Instruet with a refinement-feedback loop to progressively generate new data and finetune HoneyBee based on its training status shown in Figure 2.
NLP for Materials Science NLP applications within materials science are constrained by the dual shortage of openly accessible, high-quality data and high-performing language models. While strides have been made towards enhancing data availability (Song et al., 2023; Olivetti et al., 2020; Kononova et al., 2021; Gao et al., 2020), the primary focus has been on generating expert- annotated data for finetuning BERT-based models, which lack the advanced capabilities of contem- porary LLMs. For detailed review of the per- formance of various BERT models materials science language tasks, we refer the reader Song al. (2023). The prevailing scarcity of data and specialized LLMs in materials science motivates us to propose MatSei-Instruct, an instruction- based method for data creation, and HoneyBee, a special- ized LLM tailored for materials science.
3.1 MatSci-Instruet
The challenges of cost-effectively generating high- quality instruction data are not unique to materials science, but rather, pervasive across various sci- entific domains. Our proposed solution, MatSci- Instruct, is an innovative, domain-agnostic method- ology that leverages the power of large language models (LLMs) to generate specialized instruction sets for subsequent model finetuning.
Depicted in Figure 2, MarSci-Instruct employs a trifecta of distinct LLMs. The Instructor model crafts instructions using structured prompts encap- sulating topic and task details. The Verifier then evaluates these instructions against accuracy, rele- vance, completeness, and reasonableness criteria, ensuring only dependable instructions advance finetuning. Finally, the Evaluator assesses the out- put the finetuned model along similar dimen- sions as the Verifier. Poorly executed instructions
Instruction Finetuning LLMs LLMs consis- tently demonstrate substantial improvements when finetuned for specialized tasks, as seen with biomedical models like ChatDoctor (Li al., 2023) and HuaTuo (Wang al., 2023). While the large model size of LLMs poses challenge for effec- tive finetuning, several efficient methods have been

---

Instructor
Verifier
Evaluator
HonsyBos
<instruction>
<input>
<Otputt>
"Evaluate this
by
"Evaluate model
lask/tppc
Accuracy
Reesonabblenes
Accuracy
Redsorabienid
Sample arXiv
Completeness
Relevance
Completeness
MatScf-Instruct
Model espability evalustions
Accurasy of output
Ressoniblinnss of output
Progressive Instruction Fine-tuning
Compietenese autput.
Figure 2: MatSci-Instruct and HoneyBee training workflow. We start with a series of predetermined structured instruction generation prompts that contain both topic and task descriptions. The Instructor (Chat-GPT) then generates a series of instructions that are then passed through the Verfifier (Claude). The instructions that receive high scores with the Verifier are used for progressive finetuning in HoneyBee. The Evaluator (GPT-4) then evaluates HoneyBee's outputs and poor instructions that lead to bad performance subsequently regenerated from the beginning creating an instruction generation feedback loop for greater instruction quality.
are flagged for further refinement, verification, and evaluation. Ultimately, we generate 52k instruc- tions spanning content-based and open-ended tasks, some of which include empty inputs. Table shows that the number instructions gets reduced in later stages of the wwwressiv--efiieeee--feeeeeeeeee loop mainly due to greater emphasis on quality.
2. Evaluation
BeneyBee-7b-Stage-. performs infer- ence on new test data (data_testo). crafted by the Instructor, with outputs evaluated by the Evaluator.
3. Feedback Response
MatSci-Instruet Statisties
HoneyBee-70 Stage-1 generates responses for the test data (data_lesto)
Response Generation:
instructions for first stage
52,658
# open-ended instructions
9,931
# content-hased instructions
39,170
Scoring Responses: The Evaluator spots weak responses (data_ted tettt__tetool_,_ 8)).
# instructions with empty input
3,557
# instructions for subsequent stages
3,020
4. Instruction-Data Adaptation and Refinement
avg. input length (in words)
920.8
avg. instruction length (in words)
76.5
Focusing on Weaknesses: The Instruc- tor crafts more training (data_train) and test data (dala (ese__esss,,, focusing on issues identified by the Evaluator.
avg. output length (in words)
211.2
Table Statistics instruction data generated by MatSci-Instruct spanning diverse instruction types.
Finetuning Stage 2: HoneyBee-7b refines to HoneyBea-7b-SSage-2 Re-Evaluation: HoneyBee-7b-Stage-2 is tested with new test data
3.1.1 MatSci-Instruct Example
Next, we show full example of the progressive MatSci-Instruct instruction-data refinement proce- dure in the loop with Honey Bee model finetuning:
This process is repeated in an iterative feedback loop for continued refinement as shown in Figure 2.
1. Instruction- Generation and Fiinetuiing
3.1.2 Instructor Module
Data Generation: Instructor generates training data (data_traing]_ - 10)).
The Instructor module of our framework, embod- ied by ChatGPT, performs the generation of mate- rial science instruction-data. This module employs concise instruction prompt schema composed of three elements: <instruction>, <input>, and <output>. The <instruction> outlines the task using a standardized NLP task set, the <input>
Data Verification: Verifier removes low- scoring data (data_traing 21
Finetuning: LLaMa-7b becomes HoneyBee- MonneyBee-7--Stuge- 7b-Stage-l after finetuning with data from Alato_traingl 11).

---

electronic materials ca carbon, nanotubes construction materials
This method only diversifies the task prompts but also aligns the generated instruction-dat more closely with practical, domain-specific contexts. To conclude the instruction-dar data generation pro- cess, ChatGPT compiles ten representative instruc- tion prompt samples from all possible options. These samples are formatted in standardized JSON format, readily available for use in the sub- sequent steps of the MatSci- process. This approach ensures a comprehensive and diverse set of instructions, which in turn contributes to robust and adaptable language model during finetuning.
ceramics semicondustors
magnetic materials
bio-inspired materials
nanomateriall polymers
superconductors
glass smart materials
energy materials
self-healing materials
composites graphene me mettalsiomateriol optical materials thermoelectric materials
3.1.3 Verifier Module
Figure 3: Wordcloud of diverse materials science topics contained in the MatSci- Instruct dataset.
Generating high-quality instruction-des can be challenging, and the presence of low-quality data in finetuming a model can lead to misleading re- sults. To address this issue, MatSci-Instruct em- ploys two-step framework by incorporating Ver- fier model to improve the trustworthiness of gen- crated data. Specifically, we use Claude as the Verifier to ensure the quality of the instructions generated by the Instructor (Chat-GPT).
contains the relevant data, and the <output> gen- erates pertinent response to the task.
We query ChatGPT with this schema, populat- ing the <instruction> and <input> fields with a selection of 20 NLP tasks and 20 materials science subtopics shown in Figure 3, to ensure task and con- tent diversity. These selections are manually veri- fied before they are utilized in structured prompts for generating detailed finetuning instruction-data. Detailed lists of prompts and materials science top- ics are available Appendix and Appendix F. Following the schema, we engage in random sampling process, selecting five candidate topics and five tasks, then applying them to the instruction prompts for data generation. For robustness, we direct ChatGPT to flag in the <output> field any instruction that cannot be processed based solely on the <input> and <instruction>. To control task difficulty and boost diversity, occasionally limit the length of <instructions
Our evaluation based on four dimensions: ac- curacy, relevance, completeness, and reasonable- ness. Similar to the instruction-data generation, instruction verification is based on a standard set of prompts, shown in Appendix E, which include precise definitions of the evaluation criteria along with the complete instructions generated by the Instructor. Concretely, the evaluation criteria are:
The accuracy of the instruction- data evaluated by comparing with known facts or credible sources. This involves check- ing the accuracy of any claims or statements made in the text and verifying that they are supported by evidence.
Accuracy:
<output>. To enhance the diversity and robustness of the instruction-dal generation process, design in- corporates several additional strategies. One such strategy employs an open-ended task where the <input> field remains intentionally blank, allow- ing the model to generate responses without pre- defined constraints. This approach tests the gener- ative abilities of the model under uncertainty and promotes more varied outcomes. Another key strat- egy is content-based instruction-data generation. Instead of relying on predefined topics and tasks, this approach utilizes real-world materials science literature. We select a random open-access paper from the materials science category on arXiv and extract specific fragment to fill the <input> field.
The relevance of the instruction- data is assessed by determining how directly it relates to materials science. This is achieved by analyzing the text's content and ascertain- ing its applicability to the field.
Relevance:
Completeness: Completeness an essential dimension to ensure the instruction-dan com- prehensively address the given task, inclusive of all sub-questions. This involves consider- ing both depth and conciseness to ensure that the output complete and comprehensive.
Reasonableness: The reasonableness the instruction- about logical consistency.

---

This dimension ensures no evident contradic- tions exist within the generated data.
the progressive generation of new instruction data that takes into account the evaluated model's per- formance on different criteria, tasks, and topics.
The verifier module (i.e., Claude) evaluates the instruction-data based on the four dimensions men- tioned above and identifies any low-quality data that falls below a predetermined threshold. This rigorous verification ensures the use of high-quality data in model fine-tuning, thereby improving the overall efficacy and accuracy of the system. Our verification protocol is designed for modularity and extensibility. This modular design facilitates the in- corporation of additional agents into a multi-agent system, each assessing instruction-de -ata based on pre-defined criteria. The final decision on data qual- ity is then reached through a consensus mechanism, augmenting the robustness and comprehensiveness of the verification process, ensuring high-quality data for model finetuning.
Instructions leading to suboptimal performance by HoneyBee are returned to the Instructor, trig- gering the creation of more detailed and targeted instructions for future iterations. This iterative pro- cess also includes instruction evaluation by the In- structor, enabling the generation of more precise in- struction data for subsequent rounds. For instance, should HoneyBee score low on 'Completeness' for
particular instruction, we inform the Instructor of this deficiency, providing the criteria for 'Com- pleteness' . Consequently, the Instructor generates enhanced instruction-data to improve HoneyBee's completeness in responding similar tasks.
Our progressive finetuning process for the lan- guage model is based on LoRA (Hu et al., 2021), where we create and train separate set of low- rank matrices & that bypass the need for chang- ing the actual parameters of the language model Φ. Since i consists of low rank-matrices, it is signif- icantly more parameter and compute efficient for model finetuning. In our finetuming process, we assume that the Instructor + Verifier models act as the teacher model and the HoneyBee model acts the student model. In this setting, the student model will continually learn from the instruction-data and undergo testing during the learning process, allow- ing us monitor its performance in real-time. The finetuning process continues for a set number of epochs with early stopping if the student model converges to a given loss value. Next, we evaluate the response quality of the student model for any given instruction with the Evaluator. In our pro- gressive finetuning strategy, we monitor the eval- uation scores after each stage, denoted Spalssae and terminate the process when the Staleess stops yielding significant improvements. In our exper- iments in Section 4, we perform three stages of progressively finetuning both the instruction-date and the HoneyBee model parameters.
3.1.4 Evaluator Module
The Evaluator model assesses the output of the HoneyBee language model along similar evalua- tion dimensions as the Verifier, namely: accuracy, completeness, and reasonableness. We no longer consider relevance at this stage since the verifi- cation step filtered out all instructions with little relevance to materials science. In this paper, we use GPT-4 4 (OpenAI, 2023) as the Evaluator model, which provides an additional independent LLM that is different, and potentially more advanced, than the Insiructor and Verifier LLMs. The Evalu- ator also helps with the identification of poorly for- mulated instructions according to the performance of the HoneyBee model. These instructions are then passed back to the Instructor for additional iterative refinement.
3.2 HoneyBee
Upon obtaining set of trustworthy instruction data from the Verifier, tion dataset to finetune a LLaMa-based model for a specific domain. In this work, we finetune a model for materials science using a progressive finetuning technique to convert a standard LLaMM model to a specialized model in material science: HoneyBBec,
the generated instruc-
Experiments
3.2.1 Progressive Instruction Finetuning
Our experiments focus on assessing the ability of MatSci-Instruc create high-quality, trustworthy instruction-data relevant to materials science, as described in Section 1.1, along with understanding the capabilities and limitations of HoneyBee.
In our approach, as depicted in Figure 2. we harness a progressive instruction finetuning methodology that relies on a feedback loop. This loop enables
hattps://opendar http:/ppaaa..cmcm//e/eerr///////

---

4.1 MatSci-Instruct Evaluation
Model
Accuracy
Completeness
Reasonableness
Zero-Shot LLMs
A critical piece of the MatSci-Instruct pipeline is the independent verification and evaluation of the instruction-data generated by the Instructor model. Given the importance of the Verifier and Evalua- tor in ensuring the quality of the instruction-data, and the fact that understanding materials science textual data requires deep domain understanding. we conducted evaluation with human experts the trustworthiness of the instructions generated by MatSci-Instruct. In our human expert evalu- ation, we asked two graduate students majoring material science to evaluate 50 randomly se- lected instruction data along the same evaluation dimensions as the Verifier module (accuracy, rel- evance, completeness, reasonableness) Next, we conducted a verification and evaluation of the same 50 instructions using Claude and GPT-4 respec- tively. We measure agreement between the human experts and the LLMs by calculating Spearman and Pearson correlation coefficients between the scores along each of the dimensions.
Chat-GPT
92.55
98.74
99.84
Llama-7b
78.81
90.36
97.64
Llama-13b
84.22
91.22
98.33
Alpaca-7b
81.35
92.01
98.49
Alpaca-13b
86.24
92.17
98.80
Honey Bee without Verlication
HoneyBee-7b
85.42
93.24
98.49
HoneyBee-
88.76
93.99
98.93
HoneyBee with MatSci-Instruct
93.42
99.07
HB-7b-Stage2
89.99
94.84
99.64
HBB-7b-Stage3
91.95
95.78
99.90
HB-13b-Stagel
94.
94.42
99.40
HB-13b-Stage2
96.42
95.42
99.78
HB-13b-Stage3
98.11
97,00
99.89
Table 2: Evaluation results for various LLMs based on performance MatSci-Instruct data along with accu- racy, completeness, and reasonableness performed by GPT-4. HoneyBee performs better with verification and gets progressively better with each iterative stage of MatSci-Instruct approaching and exceeding the perfor- mance of Chat-GPT in the case of HoneyBee- 13b. We highlight scores that outperform Chat-GPT.
Claude Spearman
GPT4 Spearman GPT4 Pearson
Cloude Pearson
4.2 HoneyBee Task Evaluation
Comelltiinn Conficcent
We generated a test set of instruction datta with the same generation process the train set that is only introduced during model evaluation. The results in Table 2 show that HoneyBee gets progressively better with each iteration of MatSci-Instruss for both HoneyBee-7h and HoneyBee- 13b. HoneyBee without verification also outperforms LLaMA and Alpaca LLMs of equal size indicating the value of the progressive finetuming approuch on specialized materials science instruction-data HoneyBee- 13b closely matches, and in some exceeds, the evalua- tion performance of Chat-GPT which served the Instructor. Notably, HoneyBecs Bee-13b 10r more parameter efficient than GPT-3.
0.6
0.4
0.2
0.0
Accuracy
Relevence
Completeness Ressoneblenness
Figure 4: Correlation between human evaluation and LLM evaluation (Claude, GPT-4). Both Spearman and Pearson correlation coefficients consistently exceed 0.6 between both methods indicating good agreement.
4.3 HoneyBee Performance on MatSci-NLP
As shown in Figure 4, both Claude and GPT-4 had correlation coefficients higher than 0.6 for each dimension and an overall coefficient as high as 0.8 when compared to manual evaluation. This indi- cates decent level of consistency between manual and automatic evaluations for random sample of instructions, which gives us confidence in the ability of MatSci-Instruct to generate trustworthy, high-quality instructions for Honey Bee finetuning.
In addition to evaluating the performance of Hon- ByBee based on LLM assessment in Section 4.2, investigate the performance of HoneyBee on MatSci-NLP, broad benchmark of materials sci- ence NLP tasks (Song al., 2023). We study Hon- ey Bee's performance under two settings: 1. Low- data training setting as applied in the original paper by Song et al. (2023); 2. Zero-shot performance

---

Table 3: Low-resouree finetuning and zero-shot evaluation results for various HoneyBee on MatSci-NLP tasks. For low-resource finetuning, follow the method described in Song et al. (2023). HoneyBee outperforms all models across the vast majority of tasks for both low-resource finetuming and zero-shot settings. MatSci-Instruct's Progressive-Refine
method improves HoneyBee's performance each consecutive stage. We report macro-FI (top) and micro-FI (bottom) scores highlighting the best, second-best and third-best performing LLM. Honey-7b and Honey Bee- 13b outperform both ChatGPT and Claude and are generally competitive with GPT-4.
Named Entity
Relation
Argument
Paragraph
Synthesis Action
Sentence
Overall
Model
Recognition
Extraction
Classification
Clusification
Filling
Tasks)
Low-Resource Finetuning MatSci-NLP
MaiSciBERT (Gupla et 2022)
0.436
0.251
0.623
MarBERT (Walker al., 2021)
0,451
0.256
0.513
0.594
0,58)
0.792
HloneyBee- Bee 7b
0.518
0.653
0.76
Honeyyeee Bee 136
0.578
0.486
0.658
0.662
0.743
0.401
0.611
Zero-Sbot LLM Performance
LLaMM-7b (Touxron et sl., 2023)
0.042
0.160
0.279
0.064
0.013
0.042
0.218
0.013
0.087
0.010
0.064
LLaMA-136
0.057
0.109
0.042
0.233
0.039
0.079
0.138
(Touxron et sl., 2023)
0.066
0.016
0.054
0.189
0.009
0.074
0.008
0.059
Alpuca:7b (Taori et al., 2023)
0.031
0.053
0.029
0.375
0.179
0
0.139
0.141
0.018
0.037
0.009
0.294
0.129
0
0.039
0.101
Alpuca:
0.053
0.016
0.111
0.310
0.442
0.375
0.110
0.202
(Taori et al., 2023)
0.046
0.035
0.072
0.237
0.278
0.334
0.015
0.145
Chas-GPT
0.033
0.232
0.204
0.433
0.300
0.320
0.368
0.274
(OpenAI, 2022)
0.052
0.145
0,203
0.450
0.183
0.318
0.280
0.233
Clande (Bai et al., 2022)
0.033
0.232
0.195
0.442
0.280
0.329
0.393
0.276
0.048
0.143
0.169
0.467
0.177
0.326
0.305
0.234
GPT-4
0.189
0.445
0.483
0.679
0.743
0.788
0.502
0.543
(OpenAI, 2023)
0.121
0.432
0.353
0.522
0.677
0.689
0.483
0.468
Zero-Shot Honey Bee with MatSei-Instruct
HoneyBee-Thh Stgell
0.173
0.138
0.196
0,380
0.592
0.416
0,292
0,301
0.148
0.120
0.096
0,207
0.208
0.334
0.105
0.174
HoneyRee-Thh Sugge2
0.243
0.199
0.237
0.440
0,612
0.467
0.344
0.363
0.166
0.145
0.123
0.301
0.289
0.345
0.176
0.221
0.267
0.245
0.290
0.490
0.688
0.490
0.393
0.409
Heo--Beo-
0.190
0.178
0.189
0.343
0.342
0.365
0.289
0.271
0.369
0,301
0.389
0.500
0.701
0.512
0.467
0.463
Honey Boo-133-Sugell
0.256
0.224
0.265
0.379
0.378
0.402
0.334
0.320
0.391
0.367
0.437
0.576
0.765
0.557
0.508
0.514
Honey Bec- Beo-13b-Stage2
0.299
0,290
0.303
0.41
0.401
0.461
0.379
0.363
0.429
0.412
0.481
0.611
0.801
0.589
0.578
0.557
Honey Bee- 13b Stage3
0.372
0.346
0.378
0.467
0.429
0.503
0.423
0.417
on MatSci-NLP tasks shown in Table 3. MatSci- NLP contains a wide range of text data related to material science that spans wide range of NLP tasks and types of materials, including but not lim- ited to fuel cells, inorganic materials, glasses, and superconductors. For evaluation on MatSci-NL., we follow the same convention as Song al. (2023) where we report both macro-FI and micro- Fl scores in Table 3.
Low-Resource Finetuning: In our experiments. we follow the same low-resource procedure de- scribed in (Song et al., 2023) by splitting the data in MatSci-NLP into 1% training subset and 99% test subset for evaluation. Subsequently. all models, including HoneyBec are finetuned on the training subset and evaluated on the test subset of MatSci- NLP data. The results on low-resource finetun- ing in Table 3 show that both HoneyBee-7b and HoneyBee-133 perform best overall, outperform-

---

ing MatBERT (Walker al., 2021) and MatSciB- ERT (Gupta et al., 2022) among all tasks in MatSci- NLP with the exception of named entity recogni- tion. MatBERT and MatSci-BERT are both BERT models pretrained on different corpora of materials science textual data. While the domain-specifte pretraining significantly boosts the score of both models for MatSci-NLP tasks, HoneyBee shows better performance without requiring pretraining on materials science textual data. This is a signifi- cant advantage of HoneyBee and MatSci-Instruae given that large, high-quality corpora of materi- als science text are generally difficult to obtain as described in Section 2.
5 Conclusion
In this work, we introduce MatSci-Instruct, an it- erative instruction generation method for materials science, and HoneyBee, sfate-of-the-an of-thh-art large lan- guage model for materials science. To the best of our knowledge, HoneyBee the first billion- parameter scale language model that is special- ized in materials science. HoneyBee outperforms current state-of-the-an general language models (LLaMa, Alpaca) and materials science BERT- based language models (MatBERT, MatSciBERT) in various materials science NLP tasks, and note HoneyBec's performance improvement with each successive MatSci-Instruct stage. MatSci-Instruct provides a valuable framework for generating instruction-data to progressively finetune LLMs where instruction-data from an Instructor are veri- fied by Verifier before being used for finetuning. Additionally, poor instruction-data is refined based feedback from an Evaluator leading to higher quality instruction-data and model performance for the desired specialization as shown by the results in Section 4. Future work remains in augmenting materials science LLMs with external knowledge, such as known scientific facts, which can further improve an LLM's reliability and interpretability.
Zero-Shot Performance: The zero-shot perfor- mance results, where we assess performance di- rectly on the test subset, in the lower part of Table show that HoneyBee outperforms both I.LaMa and Alpaca models. Notably, HoneyBee-7b-Stagelll which corresponds to only one round of MatSci- Instruct, outperforms both LLaMa and Alpaca models for equal (7b) and larger (13b) parame- ter sizes. The data in Table 3 further confirms the results from Table 2 that show progressive improve- ment with each stage of MatSci-Instruct where both HoneyBee-7 and Honey I 3b exhibit clear im- provement in iterative stages. We also observe that model parameter size matters for zero-shot performance with 13b parameter models outper- forming 7b for HoneyBee and Alpaca, both of which are instruction finetuned models. Interest- ingly, LLaMA-7b generally outperforms LLaMa- 13b across most MatSci-NLP tasks and in the over- all score on MatSci-NLP.
Limitations
While HoneyBee outperforms current state-of-the- art methods in various materials science NL.P tasks, remains unclear how well HoneyBee would gen- eralize the tasks outside of the MatSci-NLP bench- mark and MatSci-Instruct instruction-data to solve complex materials science challenges. Such chal- lenges may include creating synthesis recipe for materials or explaining the behavior of ma- terials system based on fundamental scientific con- cepts. Materials science remains a wide-ranging and complex field with many open questions re- maining on the true capability of Honey Bee and other LLMs to understand important materials sci- ence concepts. MatSci-Instie Instruct also relies on the availability of highly performant L.MSs to serve the Instructor, Verifier and Evaluator which can be limited in their own capabilities. Furthermore, our work focuses primarily on the materials science do- main and further studies are needed to understand how applicable our methods would be to additional scientific domains.
4.4 HoneyBee - Case Study
We perform case study to further understand the capabilities and limitations of the various LLMs we studied, including HoneyBee, Alpaca, and Chat- GPT. Our case study results, with full data and text included in Appendix D, show that HoneyBee- B generally produces outputs of the same quality as Chat-GPT while other models generally produce lower quality outputs. This provides additional weight to the results in Section 4.1 indicating that HoneyBee- 13b can match the quality of Chat-GPT after multiple rounds of progressive refinement- feedback finetuning using MauScc-IIsttuct.

---

Broader Impact
Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev. Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hycontaek Lim, Barret Zoph. Alexander Spiridonov. Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee. Zongwei Zhou, Xuezhi Wang, Brennan Sacta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier- Hellsteel,,,ern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- cling with pathways.
Both HoneyBee and Inntructt can help pro- mote research on NLP for material science in apply- ing, training and evaluating LLMs for practical ap- plications in the field. The general frameworks de- scribed in this paper can also be transferred to other scientific domain, such biology, physics and chem- istry, where trustworthy textual data is required. Our research does not raise major ethical con- cerns.
Acknowlegments
This work is supported by the Mila internal funding Program P2-V1: Industry Sponsored Academic Labs (project number: 10379), the Canada CIFAR AI Chair Program, and the Canada NSERC Discov-
ery Grant 132011-2021-031155,
Hyung Won Chung, Le Hou, Shayne Longpre. Bar- ret Zoph, Yi Tay. William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghami, Siddhartha Brahma, al. 2022. Scaling cetructiionfineture language models. arXiv preprint arXiv:2210.1144: 11416.
References
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christo- pher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez. Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott John- ston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Teltenn-Lawton, Tom Con- erly. Tom Henighan, Tristan Hume, Samuel R. Bow- man, Zac satfield-Dodde Ben Mann, Dario Amodei, Nicholas Joseph. Sam McCandlish, Tom Brown, and Jared Kaplan. 2022. Constitutional ai: Harmlessness from ai feedback.
Xiang Gao, Rong Tan, and Guanghui Li. 2020. Re- search on text mining of material science based on natural language processing. In IOP conference se- ries: materials science and engineering, volume 768, page 072094. IOP Publishing.
Tanishq Gupta, Mohd Zaki, Devanshi Khatsuriya, Kausik Hira, N M Anoop Krishnan, and Mausam 2023. DiSCoMaT: Distantly supervised composition extraction from tables in materials science articles. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 13465-13483 13443, Toronto, Canada. Association for Computational Linguistics.
Tanishq Gupta, Mohd Zaki, NM Krishnan, et al. 2022. Matscibert: A materials domain language model for text mining and information extraction. npj Compu- tational Materials, 8(1):1-11.
Daniil A Boiko, Robert MacKnight, and Gabe Gomes. 2023. Emergent autonomous scientific research ca- pabilities of large language models. arXiv preprint arXiv:2304.05332.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Train- ing compute-optimal large language models. arXiv preprint arxiv:2203.15556.
Andres M. Bran, Sam Cox, Andrew D White, and Philippe Schwaller. 2023. Chemerow: Augmenting large-language models with chemistry tools. arXiv preprint 233.v:230...5376.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Shu Huang and Jacqueline M Cole. 2022. Batterybert: A pretrained language model for battery database enhancement. Journal of Chemical Information and Modeling.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton,
Olga Kononova, Tanjin He, Haoyan Huo, Amalie Tre- wartha, Elsa A Olivetti, and Gerbrand Ceder. 2021.

---

Opportunities and challenges of text mining in mate- rials research. Iscience, 24(3):102155.
Arthur Mensch, Jean-Baatinn Lespiau, Maria Tsim- poukelli, Nikolai Grigorev, Doug Fritz. Thibault Sot- tiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, lason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer. Oriol Vinyals. Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Ko- ray Kavukcuoglu, and Geoffrey Irving. 2022. Scaling language models: Methods, analysis & insights from training gopher.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.
The power of scale for parameler-efficient prompt tuning.
Xiang Lisa Li and Percy Liang. 2021.
ttniing Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joini Conference Natural Longuage Processing (Volume I: Long Papers), pages 4582- 4597, Online. Association for Computational Lin- guistics.
Teven Le Scao, Angela Fan, Christopher Akiki, El- Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: 176b- parameter open-access multilingual language model. arXiv preprint arX.::2211.05555 1.05100.
Yunxiang Li, Zihan Li, Kai Zhang, Ruilong Dan, and You Zhang. 2023, Chatdoctor: A medical chat model fine-tuned Ilama model using medical domain knowledge.
Xiao Liu, Yanan Zheng, Zhengxiao Du Ming Ding. Yujie Qian, Zhilin Yang. and Jie Tang. 2021. Gpt understands, too.
Song, Santiago Miret, and Bang Liu. 2023. Matsci- nlp: Evaluating scientific language models on ma- terials science language tasks using text-to-schema modeling. arXiv preprint arXiv:2305.08266
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, and Sayak Paul. 2022. Peft: State- of-the-art parameter-efiin efficcient fine-tuning methods. https: //gg wwt.ccoo///////e hwwph/hgggggggggggg/eeeeggeeee
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin. Percy Liang. and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction instruction-following Ilama model. https: gi thub. wttp:/cttoottoe/.ete tatsu-ttssaaaa 1ab/ stanford_alppca.
Elsa A Olivetti, Jacqueline M Cole, Edward Kim, Olga Kononova, Gerbrand Ceder, Thomas Yong-Jin Han, and Anna M Hiszpanski. 2020. Data-driven materials research enabled by natural language processing and information extraction. Applied Physies Reviews, 7(4):041317.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13977 13971.
OpenAI. 2022. openaiintroduuinnchhggg. https: / //openai. , cww/blog/chateeeeeeeoooc [Accessed 22-Jun- 2023].
OpenAL. 2023. Gpt-4 technical report.
Nicholas Walker, Amalie Trewartha, Haoyan Huo, Sanghoon Lee, Kevin Cruse, John Dagdelen, Alexan- der Dunn, Kristin Persson, Gerbrand Ceder, and Anubhav Jain. 2021. The impact of domain- domain-specifie pre- pre-training on named entity recognition tasks in ma- terials science. Available at SSRN 3950755.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedhack. Advances Neural Information Processing Systems, 35:27730-2771 277444.
Haochun Wang, Chi Liu, Nuwa Xi, Zewen Qiang, Sendong Zhao, Bing Qin, and Ting Liu. 2023. Hu- atuo: Tuning llaaa model with chinese medical knowledge.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, Eliza Rutherford, Tom Hennigan, Ja- cob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Mari- beth Rauh, Po-Sen Huang, Amelia Glaese, Jo- hannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Anto- nia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Bud- den, Fsme Sutherland, Karen Simonyan, Michela Pa- ganini, Laurent Sifre. Lena Martens, Xiang Lorraine Li, Adhigunaa Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou,
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A Smith, Daniel Khashabi, and Han- nanch Hajishirzi. 2022. Self-instruct: Aligning lan- guage model with self generated instructions. arXiv preprint arXiv:22 12. 10560.
Aohan Zeng Xiao Liu, Zhengxiao Du, Zihan Wang. Hanyu Lai, Ming Ding, Zhuoyi Yang. Yifan Xu. Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, and Jie Tang. 2022. Glm- 130b: An open bilingual pre-trained model.

---

Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022.
Opt: Open pre-trained transformer language models. arXiv preprint arXiv:205.001068.

---

Appendix
Table 5: MatSci-Instruct samples a diverse set of NLP tasks to generate instructions including general NLP tasks to main general language capabilities.
A Experimental Details
MatSci-Instruet NLP Tasks
We finetune LLaMA models with 7B and 13B pa- rameters using instructions from MatSci-Instruct. As shown in Table 2, we analyze the effect of var- ious rounds of iterative instruction feedback. For finetuning, we use the A optimizer with an initial learning rate le-4 on 2 A100 GPUs for LLaMa-7b and A100 GPUs for LLaMa-13b. We assign batch size of 4 to each GPU with gradient accumulation step of 32 and maximum sequence length of 2048.
Machine Reading Comprchension
224
Question Answering
224
Open-Ended Generation
214
Classification
203
Information Extraction
201
Relation Extraction
193
Analysis
189
Topic Modeling
188
Writing
180
Commonsense Reasoning
172
Code Interpretation
172
Event Extraction
167
Instruction Generation Details
Grammar Correction
165
Clustertin
162
Named Entity Recognition
160
Table MatSci-Instrus samples diverse of mate- rials science topic areas.
Text Simplification
153
Summarization
149
Sentiment Analysis
149
MatSci-Instruc Topies
Part-of-Speech Tagging Editing
146
Bio-inspired Materials
221
145
Self- Self-Healigg Materials
209
Magnetic Materials
195
Smart Materials
190
C Quality of Instructions for each MatSci-Instruct Stage
Metals
189
Semiconductors
188
The preset threshold for filtering low-quality data is average of 95 or above for all four di- mensions and a score of no less than 90 for each individual dimension. As shown in Table 6, after filtering, the remaining instruction data exhibits sig- nificantly improved quality, approaching full marks making the data significantly more trustworthy for finetuning specialized LLMs.
Carbon Nanotubes
184
Polymers
182
Thermoelectrie Materials
180
Optical Materials
180
Superconductors
179
Graphene
177
Glass
174
Energy Materials
166
Composites
165
D LLM Case Study
Electronic Materials
163
Construction Materials
158
This section provides the details for the case study in Section 4.4.
Ceramics
155
Nanoaterials
153
D.1 Open-ended Instruction
biomaterials
149
We analyze the response of different LLMs to the following open-ended query: Analyze the proper- lies and applications of electronic materials.
The full list of materials science topics and NL.P tasks sampled for MatSci- Instruct instruction included in Table 4 and Table 5, We sample broad range of materials science topics and NLP that generally balanced yielding a set of instructions that includes specialized materials science text as well as general language capabilities.
Chat-GPT: Electronic materials class of materials that possess unique properties that make them suitable for use electronic de- vices and technologies. These materials are characterized by their electrical conductivity,

---

Table 6: Claude evaluation scores of MatSci-Instie Instruct before and after removing low-quality instruction data
lium nitride (GaN) possess unique optical properties, ideal for use in displays, touch screens, and flat panel displays. Magnetic materials like iron, cobalt, and nickel are used in magnetic sensors, hard drives, and mag- netic memory due to their unique magnetic properties. Materials like diamond and sili- con carbide possess unique thermal properties ideal for use in high-temperature electronic de- vices. The properties of electronic materials are critical for modern electronic devices, and research is ongoing to discover new materials with even more unique properties.
Score/Stage
2
3
before 92.45
92.20
92.56
avg. Accuracy
after
100.0
100.0
100.0
before
86.01
87.18
86.96
avg. Relevance
after
100.0
100.0
100.0
before
88.80
86.13
89.33
avg. Completeness
after
99.98
99.99
99.99
before
97.75
98.21
98.16
avg. Reasonableness
after
100.0
100.0
100.0
HoneyBee-7Bi Electronic materials possess unique properties that make them suitable for various applications. They exhibit high elec- trical conductivity, low resistivity, and excel- lent thermal conductivity. These materials are commonly used in electronic devices such transistors, integrated circuits, and sensors. They also find applications in optoelectronies, energy storage devices, and flexible electron- ics. The properties of electronic materials enable efficient charge transport and facilitate the flow clectrons, making them essential components in modern technology.
optical properties, magnetic properties, and thermal properties. The electrical conductiv- ity of electronic materials is critical prop- erty for materials used in electronic devices such as transistors, diodes, and integrated cir- cuits. Semiconductors such as silicon and germanium are widely used in microproces- sors and solar cells because of their unique electrical properties. Optical properties are also important for electronic materials used in displays, touch screens, and flat panel dis- plays. Optoelectronic materials like indium oxide (ITO) and gallium nitride (GaN) pos- sess unique optical properties that make them ideal for use in these devices. Magnetic prop- erties are important for electronic materials used in magnetic sensors, hard drives, and magnetic memory. Magnetic materials like iron, cobalt, and nickel possess unique mag- netic properties that make them ideal for use in these devices. Lastly, thermal properties are important for electronic materials used in high-temperature electronic devices such as microprocessors and power electronics. Ma- terials like diamond and silicon carbide pos- sess unique thermal properties that make them ideal for use in these devices.
Alpaca-13h: Electronic materials are mate- rials that have electrical properties and can be used in a variety of applications. These materials can be classified into two categories: conductors and insulators. Conductors are ma- terials that allow the flow of electricity, while insulators materials that do not allow the flow of electricity. Electronic materials can be used in a variety of applications, such as in semiconductors, transistors, and diodes. Semi- conductors are materials that have properties that are intermediate between those of conduc- tors and insulators and are used in the manu- facture of integrated circuits. Electronic mate- rials are also used in the manufacture of solar cells, which convert light into electricity.
HoneyBee-133 Electronic materials have unique properties such as electrical conduc- tivity, optical properties, magnetic properties, and thermal properties, making them suitable for in electronic devices. Semiconductors such silicon and germanium are used in microprocessors and solar cells due to their unique electrical properties. Optoclectronic materials like indium tin oxide (ITO) and gal-
Electronic materials are materi- als that have the ability to control the flow of electrons. These materials used in vari- ety of applications, such as electronics, optics, and sensors. Examples of electronic materials include semiconductors, which are materials that can be doped to control the flow of elec- trons, and metallic materials, which are con-
Alpaca-7b:

---

ductors of electricity. Other examples include superconductors, which are materials that can conduct electricity with zero resistance, and dielectrics, which are materials that can store electricity. Electronic materials are used in variety of upplications, such as electronics, optics, and sensors.
partial fillingss of the exciton lattice, stabilized by the exciton interactions. In particular we predict supersolid phase in which a polariton condensate coexists with spatial order. " 'The replies from the different LLMs are below:
The exciton interactions are re- sponsible for stabilizing the ordered states at partial fillings of the exciton lattice, and in particular for predicting a supersolid phase in which polariton condensate coexists with spatial order.
Chat-GPT:
ChatGPT is the most comprehensive and rele- vant answer as provides a detailed analysis of the properties and applications of electronic mate- rials, covering electrical conductivity, optical prop- erties, magnetic properties, and thermal properties. Chat-GPT also gives specific examples of mate- rials and their applications in various electronic devices. HoneyBee- 13b is the second-ber booodd-betttt answer it also covers the key properties of electronic terials and their applications in various electronic devices. HoneyBee-1 13b provides specific exam- ples of materials and their unique properties that make them ideal for use in different applications. However, HoneyBee-133 output lacks specificity and does not provide enough detail to fully ad- dress the question. HoneyBee-7b gives the third- best answer it gives general overview of the unique properties of electronic materials and their applications in various electronic devices. How- ever, HoneyBeed Bee-7b's output lacks specificity and docs not provide enough detail. Alpaca- fousth-best answer as it provides a basic under- standing of electronic materials and their use in various applications but lacks detail and specificity. Alpaca-70 7b is the least comprehensive and relevant answer as it provides a limited understanding of electronic materials and their use in various applica- tions. Alpaca-7b's output does not provide enough detail or specific examples to support its claims.
Honey BoneyBee-Ile 13b: The exciton interactions are responsible for stabilizing the ordered states at partial fillings of the exciton lattice, which are intermediates between the normal state and a polariton condensate. In particular, they are responsible for the predicted supersolid phase, where polariton condensate coexists with spatial order.
The exciton interactions are responsible for stabilizing the ordered phases that mediate the transition to the polariton con- densate, including the supersolid phase.
HoneyBee-70
The exciton interactions studied in the given text are responsible for the phase transition from the normal state polariton condensate, as well as the supersolid phase
Alpaca-13h.
1oppcc-13bba 13b is the
The exciton interactions are re- sponsible for the transition from the normal state to polariton condensate, well as the stabilization of the supersolid phase
Alpaca-7b:
The answers from HoneyBee- 13b and Chat-GPT are better compared to the outputs the other mod- els. HoneyBee- 3b and Chat-GPT directly address the question by mentioning the ordered states at partial fillings of the exciton lattice, which are in- termediates between the normal state and a polari- ton condensate, and the predicted supersolid phase. The answers also use language that closely matches the language used in the original text, indicating good understanding of the material.
D.2
Content-based Instruction
Next, we analyze the response of different LLMs to the following query: What are the exciton interac- tions responsible for? with the following <input>: Inspired by the recent experimental observation of strongly coupled polaritons in a Moire heter- obilayer, we study a model of dipole-interacting excitons localized on sites of a lattice and coupled to planar cavity photons. We calculate the phase diagram of this system by computing fluctuations around the mean field and determining the stability of the excitation spectrum. We find that the tran- sition from the normal state to a polariton conden- sate is intermediated by a series of ordered states at
E
LLM Prompts
In this section we provide some of the prompts used for the different modules in Inttei--nstruet. We plan to make the full list of prompts, data and code available upon publication.

---

"Evaluate accuracy of the given text by com- paring with known facts or credible sources. This involves checking the accuracy of any claims or statements made in the text, and verifying that they are supported by evidence. The next line directly provide the text. fout- put_text) Please return score ranging from 0 to 100, with 0 being the worst and 100 be- ing the best. Please the strictest grading standard. The score should be in JSON format with field name "score". You should not output any other information or text."
cluding input, instruction and output) by sessing how fully the output addresses the instruction, including all sub-questions. Con- sider both depth and conciseness. The next 3 lines directly provide the input, instruction and output respectively. fimput_text] (instruc- tion] (output_ted) text) Please return score rang- ing from 0 to 100, with 0 being the worst and 100 being the best. Please grading standard. The score should be in ISON format with a field name of 'score'. You should not output any other information or text." "
the strictest
"Evaluate relevance of the given text by con- sidering how directly the text is related to ma- terials science. The next line directly pro- vide the text. {output_ree) [expt Please return score ranging from 100, with being the worst and 100 being the best. Please use the strictest grading standard. The score should be ISON format with field name of 'score'. You should not output any other information or text."
"Evaluate reasonableness of the given text by considering how logically consistent the con- tent is, with no obvious contradictions. The next line directly provide text. (output_tens text] The score should range from 0 to 100, with 0 being the worst and 100 being the best. Please use the strictest grading standard. The score should be in JSON format with field name of 'score". You should not output any other information or text."
"Evaluate completeness of the given text (in-


Reviews:
[{"review_id": "W1U5cO35o0", "reviewer": "Reviewer_1nqS", "summary": "", "questions": "See \"reasons to reject\". In addition, considering that the authors utilize 3 different LLMs (ChatGPT, Claude, GPT-4) to aid in the whole process, perhaps it would be useful to list all of their results in Table 2 as a reference.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your feedback and are happy to hear that you recognize our contribution with HoneyBee as the first instructions-finetuned materials science language model with a trustworthy, iterative instruction generation process leveraging different LLMs. To clarify, the general process for creating instructions-data and training HoneyBee is:\n\n**Instruction Generation and Finetuning:**\n\n1. Data Generation: ChatGPT creates training data (data_train0 [1-10]).\n2. Verification: Claude removes low-scored data (data_train0 [1 and 2]).\n3. Fine-Tuning: LLAMA-7b becomes HoneyBee-7b-stage-1 with data (data_train0 [3-10]).\n\n**Evaluation:**\n\n* Evaluation: HoneyBee-7b-stage-1 performs inference on new test data (data_test0), crafted by ChatGPT, with outputs evaluated by GPT-4.\n\n**Feedback Response:**\n\n* Response Generation: HoneyBee-7b-stage-1 generates responses for the test data (data_test0).\n* Scoring Responses: GPT-4 spots weak responses (data_test0[7 and 8]).\n\n**Instruction Adaptation and Improvement:**\n* Focusing on Weaknesses: ChatGPT crafts more training (data_train1) and test data (data_test1), focusing on issues identified by GPT-4.\n* Fine-Tuning Stage 2: HoneyBee-7b-stage-1 refines to HoneyBee-7b-stage-2.\n* Re-Evaluation: HoneyBee-7b-stage-2 is tested with new test data (data_test1).\n\n**Feedback Loop:**\n* Iterative Process: This loop continues until HoneyBee-7b reaches the desired performance.\n\nWe include relevant prompt details used for instruction generation, verification and evaluation in the Appendix along with the full prompt data available in our code and dataset, which will be released upon publication. Next we address some of the feedback and questions you asked:\n\n> What data does GPT-4 evaluate HoneyBee's output on, the full set of data generated in the current iteration, or a subset?\n\nWe use structured prompts to ask GPT-4 to evaluate the output of HoneyBee based on the evaluation standards described in Sec 3.1.2. At each stage, the Evaluator (GPT-4) will evaluate the full set of HoneyBee’s outputs on a test set based on a standard set of evaluation prompts that we plan to publish along with the code and data.\n\n> Why do we need to train HoneyBee first, then evaluate its output, and not just evaluate the original generated data?\n\nOur framework includes an iterative loop that iteratively improves the quality of the instructions and model performance as shown in Table 2 and Table 3 respectively. The feedback improves HoneyBee’s underlying modeling ability compared to the previous iterations as well as the instructor, verifier and evaluator models. Moreover, HoneyBee is an open-source model that will be released to the community at no cost, making it more accessible than closed-source LLMs.\n\n> What is the motivation or benefit of the proposed method in the paper as opposed to directly using GPT-4 and Claude as two different verifiers to filter and revise the data?\n\nIf we had GPT-4 and Claude as two consecutive verifiers, the verified data would contain a greater bias compounded by both of the models. By keeping the evaluation independent from the verification we reduce such bias and also enable the iterative refinement of instructions based on the performance of the HoneyBee model during each stage of training, which significantly improves performance as shown in Table 2 and Table 3. If GPT-4 and Claude were intermingled as verifiers, it would be more challenging to perform this iterative instruction refinement given the greater bias in the data used for finetuning.\n\n> Q：Experiment details and train/test data set generation?\n\nThe details and hyperparameters of the experiment are shown in Appendix A. The test set is auto-generated with the same generation process as the train set but only introduced during evaluation of modeling performance, which makes model evaluation more robust. Additionally, we use a public dataset MatSci-Bench to test the ability of model generalization to realistic NLP applications shown in Section 4.3. The zero-shot generalization abilities shown in Table 3 make our results pertaining to HoneyBee’s materials science knowledge particularly convincing.\n\n> Q: list results of ChatGPT, Claude, and GPT-4 in Table 2\".\n\nClaude and GPT-4 are used as a verifier and an evaluator, and hence are responsible for evaluating the performance of different models. Therefore, we cannot use them as baselines for comparison in Table 2 given that the results are based on their own output.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The first distilled data in the field of materials science, and the authors verify the validity of the data.\n2. The first instruction-tuned model specific to material science, which is based on LLaMA and LoRA.", "reasons_to_reject": "1. Poor presentation of the proposed method. From the main text of this paper, I personally cannot understand how the evaluator module works. What data does GPT-4 evaluate HoneyBee's output on, the full set of data generated in the current iteration, or a subset? Why do we need to train HoneyBee first, then evaluate its output, and not just evaluate the original generated data? What is the motivation or benefit of the proposed method in the paper as opposed to directly using GPT-4 and Claude as two different verifiers to filter and revise the data?\n2. Lack of necessary details in the experiment. For example, I'm curious how the training and test set are divided. Is the test set also auto-generated, and if so, does this introduce specific bias?", "questions_to_authors": "See \"reasons to reject\". In addition, considering that the authors utilize 3 different LLMs (ChatGPT, Claude, GPT-4) to aid in the whole process, perhaps it would be useful to list all of their results in Table 2 as a reference.", "paper_topic_and_main_contributions": "Summary: The paper introduces a framework, MatSci-Instruct, designed for data generation in materials science. This process is then employed to fine-tune a large language model, named HoneyBee, making it specialized for materials science tasks. In addition, the authors use three different LLMs as instructor, verifier, and evaluator, respectively, to assist in this entire process of data generation and student model training. Experimentally, HoneyBee is evaluated using the MatSci-NLP benchmark, where it shows superiority over existing language models for materials science tasks.\n\nMain Contributions:\n1. Introduction of MatSci-Instruct: A LLM-assisted framework for data curation in materials science. This process uses one model to instruct and generate data and another separate model to verify its accuracy and relevance.\n2. Development of HoneyBee: HoneyBee is the first billion-parameter language model that is specialized for materials science tasks.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 2 caption: Verfifier -> Verfier"}, "extra_text": {}}, {"review_id": "Jch9HBXZLH", "reviewer": "Reviewer_RMWP", "summary": "", "questions": "A. Sec 3.1.1: Do I understand correctly that the generated data from ChatGPT is the **output** for an \"instruction\" and an \"input\" rather than the instruction itself? Please clarify and consistently specify the data generated in this work. This section directly conflicts with the abstract: \"Moreover, we iteratively **generate more targeted instructions** in a finetuning-evaluation-feedback loop leading to progressively better performance for our finetuned HoneyBee models.\"\n\nB. \"We query ChatGPT with this schema, populating the <instruction> and <input> fields with a selection of 20 NLP tasks and 20 materials science subtopics shown in Figure 3, to ensure task and content diversity.\" - Please describe the following in greater detail: 1) I see that Table 5 lists the 20 tasks, where were these tasks obtained? i.e is it an existing dataset? 2) What do the second columns of Table 4 and 5 mean? 3) Where was the input fed to ChatGPT for each of the tasks obtained? 4) Was a single instruction per task used for the 20 tasks? 5) Were the instructions manually authored?\n\nC. Was there an overlap in the input data to ChatGPT and the MatSci-NLP tasks?\n\nD. Did ChatGPT, Claude, or GPT4 use any few shot examples in their prompts?\n\nE. Sec 3.1.2: \"The accuracy of the instruction data is evaluated by comparing it with known facts or credible sources\" - where are the facts and credible sources obtained from?\n\nF. Sec 3.2.1: \"Instructions leading to suboptimal performance by HoneyBee are returned to the Instructor, triggering the creation of more detailed and targeted instructions for future iterations.\" - Setting aside the confusion of question A, if the instruction is flawed and the instruction is changed to fit a poorly trained model is the instruction not likely to be contrived and unintuitive to humans?\n\nG. Sec 4.3: \"Lowdata training setting as applied in the original paper by Song et al. (2023)\" - please describe in the body of this paper the precise fine-tuning data that was used from Song et al.? Were Honeybee models fine-tuned on any additional data besides the synthetic data generated in MatSci-Instruct?\n\nH. Please consider reporting the performance of ChatGPT on the benchmark dataset of Table 3 (the metrics of Table 2 are vague), this is likely to be extremely useful to practitioners choosing to use Honeybee models vs ChatGPT. This is also likely to be further vet the effectiveness of your data generation process.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for feedback and are happy that you recognize our contribution in iterative instruction generation for training HoneyBee, the first materials science LLM. We also appreciate that you noticed that our method provided strong empirical gains over prior methods and that the paper was easy to read.  To clarify, the general process for creating instructions-data and training HoneyBee is:\n\n**Instruction Generation and Finetuning:**\n\n1. Data Generation: ChatGPT creates training data (data_train0 [1-10]).\n2. Verification: Claude removes low-scored data (data_train0 [1 and 2]).\n3. Fine-Tuning: LLAMA-7b becomes HoneyBee-7b-stage-1 with data (data_train0 [3-10]).\n\n**Evaluation:**\n\n* Evaluation: HoneyBee-7b-stage-1 performs inference on new test data (data_test0), crafted by ChatGPT, with outputs evaluated by GPT-4.\n\n**Feedback Response:**\n\n* Response Generation: HoneyBee-7b-stage-1 generates responses for the test data (data_test0).\n* Scoring Responses: GPT-4 spots weak responses (data_test0[7 and 8]).\n\n**Instruction Adaptation and Improvement:**\n* Focusing on Weaknesses: ChatGPT crafts more training (data_train1) and test data (data_test1), focusing on issues identified by GPT-4.\n* Fine-Tuning Stage 2: HoneyBee-7b-stage-1 refines to HoneyBee-7b-stage-2.\n* Re-Evaluation: HoneyBee-7b-stage-2 is tested with new test data (data_test1).\n\n**Feedback Loop:**\n* Iterative Process: This loop continues until HoneyBee-7b reaches the desired performance.\n\nWe include relevant prompt details used for instruction generation, verification and evaluation in the Appendix along with the full prompt data available in our code and dataset, which will be released upon publication. Next we address some of the feedback and questions you asked:\n\n> Reasons to Reject 1: Data generalization details.\n\nPlease see the answers to the specific questions for more details.\n\n> Reasons to Reject 2: The primary methodology of the paper is relatively straightforward and well adopted elsewhere (generate synthetic training data with a multi-billion parameter LLM, filter, fine-tune smaller LLM) - leaving the data contributed and the fine-tuned models being the primary contributions.\n\nIt is well known that LLMs suffer from hallucination problems. When the text generated by the model does not follow the original text (Faithfulness) or does not conform to the facts (Factualness), we can consider that the model exhibits hallucination problems. Therefore, our contribution also lies in proposing an iterative fine-tuning framework that uses multiple steps to eliminate low-quality, potentially hallucinated, fine-tuning data and obtain trustworthy models through progressive training, which is fundamentally different from existing instruction-tuning approaches.\n\n> Question A:\n\nYes, the \"instruction\" and \"input\" are formatted with a prompt template to ask ChatGPT to generate relevant data with the generated data being referred to as “instruction data”. We will revise our manuscript to make this clearer.\n\n> Question B: 1) I see that Table 5 lists the 20 tasks, where were these tasks obtained? i.e is it an existing dataset? 2) What do the second columns of Table 4 and 5 mean? 3) Where was the input fed to ChatGPT for each of the tasks obtained? 4) Was a single instruction per task used for the 20 tasks? 5) Were the instructions manually authored?\n\n1）Initially, we asked ChatGPT to generate 20 candidate tasks which we then downselected to the most appropriate NLP tasks related to materials science. The tasks are not directly related to an existing dataset, but were informed by a review of current relevant literature.\n\n2）The second column represents the number of instructions. We will add this to the Table captions.\n\n3)-5) As we clarify in the first part of our response, we use a manually designed prompt template to generate instruction data for which we only need to specify the input text and task when generating instruction data. The input text is randomly selected from a span of a materials science academic paper from arXiv, and the task is randomly chosen from 20 candidates. Together these templated prompts, which we describe in more detail in the Appendix, provide a scalable and repeatable manner to generate high-quality, relevant instructions data.\n\n> Question C: Was there an overlap in the input data to ChatGPT and the MatSci-NLP tasks?\n\nNo, we kept the data separate for independent evaluation of the performance of HoneyBee and other baseline models.\n\n> Question D: Did ChatGPT, Claude, or GPT4 use any few shot examples in their prompts?\n\nNo, these models already have very good zero-shot capabilities and do not require providing additional examples. Only when generating new instructions-data based on the feedback from the previous stage, we will select bad cases to put into the prompt so that the instructor can improve based on previous instructions and thereby generate higher-quality data.\n\n> Question E: Sec 3.1.2: \"The accuracy of the instruction data is evaluated by comparing it with known facts or credible sources\" - where are the facts and credible sources obtained from?\n\nWe specify in the verification and evaluation prompts: \"evaluate the instruction data by comparing it with known facts or credible sources\". As such, known facts or credible sources come from the knowledge of the large model itself. Generally speaking, ChatGPT, Claude, and GPT-4 are trained with massive samples and themselves constitute a huge knowledge base. In future work, we aim to have these facts be based on external knowledge which would make instruction verification more robust.\n\n> Question F:\n\nHere the term \"instructions\" refers to the initial output of ChatGPT (the instructor) for finetuning HoneyBee. Concretely, we use a prompt to ask chatGPT to generate instructions-data (question-answering pairs). In the case of instruction refinement, the prompt includes previous instructions along with feedback from the evaluator model (GPT-4). We will revise our manuscript to make this clearer.\n\n> Question G:\n\nIn our experiments, we follow the same low-resource procedure described in Song et al. (2023) by splitting the data in MatSci-NLP into 1% training subset and a 99% testing subset for evaluation. After that, all models, including HoneyBee, are finetuned on the training subset and evaluated on the test subset based on MatSciNLP data.\n\n> Question H:\n\n Thanks for the feedback. We have performed zero-shot experiments for Chat-GPT, Claude and GPT-4 for Table 3 (shown in the table below), which we plan to include the results in the camera-ready version if the paper is accepted.\n\n**Zero-shot setting on MatSci-NLP Benchmark:** Data shown are Micro-f1/macro-f1 for each task\n\n| Task | NER | Relation Extraction | Event Argument Extraction | Paragraph Classification | Synthesis Action Retrieval | Sentence Classification | Slot filling | Overall |\n| ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ | ------ |\n\nChatGPT |  0.063/0.052 | 0.232/0.145 | 0.204/0.203 | 0.433/0.450 | 0.300/0.183 | 0.320/0.318 | 0.368/0.280 | 0.274/0.233 |\n\nClaude | 0.063/0.048 | 0.232/0.143 | 0.195/0.169 | 0.442/0.467 | 0.280/0.177 | 0.329/0.326 | 0.393/0.305 | 0.276/0.234 |\n\nGPT-4 | 0.189/0.121 | 0.445/0.432 | 0.453/0.353 | 0.679/0.522 | 0.743/0.677 | 0.788/0.689 | 0.502/0.483 | 0.543/0.468 |\n\nFrom the results above, we can see that Honey-7b and HoneyBee-13b outperform both ChatGPT and Claude and are generally competitive with GPT-4. HoneyBee is also an open-source LLM and therefore more accessible compared to all the closed-source models (ChatGPT, Claude, GPT-4).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is easy to read.\n- The paper reports strong empirical gains over prior baselines.", "reasons_to_reject": "- The paper lacks several details about the data generation process (elaborated in questions), making it extremely hard to judge the correctness and scope of the generated data, its usefulness in future tasks, and the generalizability of the fine-tuned models.\n- The primary methodology of the paper is relatively straightforward and well adopted elsewhere (generate synthetic training data with a multi-billion parameter LLM, filter, fine-tune smaller LLM) - leaving the data contributed and the fine-tuned models being the primary contributions.", "questions_to_authors": "A. Sec 3.1.1: Do I understand correctly that the generated data from ChatGPT is the **output** for an \"instruction\" and an \"input\" rather than the instruction itself? Please clarify and consistently specify the data generated in this work. This section directly conflicts with the abstract: \"Moreover, we iteratively **generate more targeted instructions** in a finetuning-evaluation-feedback loop leading to progressively better performance for our finetuned HoneyBee models.\"\n\nB. \"We query ChatGPT with this schema, populating the <instruction> and <input> fields with a selection of 20 NLP tasks and 20 materials science subtopics shown in Figure 3, to ensure task and content diversity.\" - Please describe the following in greater detail: 1) I see that Table 5 lists the 20 tasks, where were these tasks obtained? i.e is it an existing dataset? 2) What do the second columns of Table 4 and 5 mean? 3) Where was the input fed to ChatGPT for each of the tasks obtained? 4) Was a single instruction per task used for the 20 tasks? 5) Were the instructions manually authored?\n\nC. Was there an overlap in the input data to ChatGPT and the MatSci-NLP tasks?\n\nD. Did ChatGPT, Claude, or GPT4 use any few shot examples in their prompts?\n\nE. Sec 3.1.2: \"The accuracy of the instruction data is evaluated by comparing it with known facts or credible sources\" - where are the facts and credible sources obtained from?\n\nF. Sec 3.2.1: \"Instructions leading to suboptimal performance by HoneyBee are returned to the Instructor, triggering the creation of more detailed and targeted instructions for future iterations.\" - Setting aside the confusion of question A, if the instruction is flawed and the instruction is changed to fit a poorly trained model is the instruction not likely to be contrived and unintuitive to humans?\n\nG. Sec 4.3: \"Lowdata training setting as applied in the original paper by Song et al. (2023)\" - please describe in the body of this paper the precise fine-tuning data that was used from Song et al.? Were Honeybee models fine-tuned on any additional data besides the synthetic data generated in MatSci-Instruct?\n\nH. Please consider reporting the performance of ChatGPT on the benchmark dataset of Table 3 (the metrics of Table 2 are vague), this is likely to be extremely useful to practitioners choosing to use Honeybee models vs ChatGPT. This is also likely to be further vet the effectiveness of your data generation process.", "paper_topic_and_main_contributions": "The paper presents an approach for generating training data using a variety of commercial LLMs (ChatGPT, Claude, GPT4) for fine-tuning billion parameter LLMs (based on LLaMa) for a variety of materials science NLP tasks - the generated data (and the process) is referred to as MatSci-Instruct and the finetuned LLaMa model is referred to as HoneyBee. The training data is generated by ChatGPT, verified by Claude, used to finetune a LLaMa model, the outputs of which are evaluated by GPT4. This process is repeated a few times. The paper evaluates the HoneyBee models on a benchmark of NLP tasks in materials science and demonstrates superior performance to prior methods based on small parameter BERT models.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None.", "Typos_Grammar_Style_and_Presentation_Improvements": "- MatBERT outperforms the HoneyBee model on paragraph classification micro-F1; please fix the highlighting in Table 3."}, "extra_text": {}}, {"review_id": "4OaWjkee9o", "reviewer": "Reviewer_ztYe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for feedback and are happy that you recognize the value of our contribution in creating a trustworthy instruction generation process for finetuning LLMs on materials science. We also appreciate that you recognize that HoneyBee will be a valuable contribution to the community and outperforms current methods on materials science NLP tasks, especially with iterative instruction generation. Next we address some of the feedback and questions you called out:\n\n> Q: The paper does not motivate the proposed approach and justify the design choices made.\n\nThank you for this feedback. Our primary motivation is to create a high-performing, open-source LLM for materials science (HoneyBee), which does not exist today. In order to create HoneyBee, we required trustworthy text-based data that is relevant to materials science which is not available in the public literature. To generate that data, we developed MatSci-Instruct, which creates instructions-data for finetuning a LlaMa model to arrive at HoneyBee in iterative refinement stages. We chose LlaMa as the base model given its strong performance, open-source availability and past work indicating good performance for instructions based finetuning (e.g., Alpaca). MatSci-Instruct leverages various LLMs as instructor, verifier and evaluator to minimize bias and increase trustworthiness in the instruction-data generation process with iterative refinement leading to additional performance increases. To the best of our knowledge, HoneyBee is the best performing open-source language model in materials science, even outperforming closed source LLMs, which we plan to release to the community at no cost, making it more accessible than closed-source LLMs.\n\n> Q: There are many approaches to distill knowledge from large language models (e.g. https://arxiv.org/pdf/2110.07178.pdf) for targeted domains. These approaches are not discussed or compared against. Are these approaches ineffective for scientific domains, and why?\n\nCompared with instruction finetuning, traditional methods for knowledge distillation in specific fields (such as material science) have the following drawbacks:\n*  **Poor Generalization Ability:** Knowledge distillation provides poor generalization compared to instruction-based finetuning, which unifies all tasks into the form of QA, and can cover various tasks and enhance the model's generalization ability.\n* **Instructions-Data Quality:** In our approach, we can control the instructions of fine-tuning, not only to ensure the quality of the data, but also to generate data purposefully, so as to obtain a better knowledge distillation effect.\n* **In-Context Learning:** Instruction-based fine-tuning can fully explore the in-context learning ability of large models and improve the modeling ability of language models to NLP task instructions. Supervised training can teach language models to learn to complete tasks according to instructions, so that language models can learn to follow instructions, even for tasks they have never seen before.\n\n\n> Q: The paper does not provide a detailed analysis of the computational resources required for training and fine-tuning.\n\nIn Appendix A, we outline details for the compute requirements for our experiments: Specifically, we use 2 A100 GPUs for LLaMa-7b experiments and 4 A100 GPUs for LLaMa-13b experiments. Please let us know if you would like to see any additional details.\n\n> Q: In scientific domains such as material science that are evolving and demand factuality, it might make sense to augment language models with external knowledge (i.e., rely on semi-parametric knowledge) instead of trying to pack everything in the model parameters (parametric knowledge).\n\nThank you for making this very relevant point. In a general sense, by relying on semi-parametric knowledge, language models can benefit from the scalability and flexibility of external knowledge sources, which can be updated independently and adapted to specific application domains. Yet, obtaining external knowledge and modules specifically tailored to the material science domain can be challenging. Material science is a specialized field with complex and domain-specific information that may not be readily available in structured or easily accessible formats. It therefore requires a diverse set of tools for high-quality data curation, as well as domain-specific methods to effectively augment language models with external material science knowledge or modules.\nGiven the variety of technical challenges required to properly infuse semi-parametric knowledge, we leave this research for future work. In this case, one can consider augmenting large language models with external knowledge instead of attempting to include all knowledge within the model parameters.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Two-step framework to generate trustworthy instructions in target scientific domains. Paper shows effectiveness in material science domain.\n2. The billion-parameter specialized materials science LM is a valuable resource, especially since it outperforms existing language models in materials science.\n3. The iterative approach shows that finetuning-evaluation feedback can progressively improve the quality of a domain-specialized model. \n4. Experiments indicate superiority of the proposed model over existing models in materials science. \n", "reasons_to_reject": "1. The paper does not motivate the proposed approach and justify the design choices made. \n2. There are many approaches to distill knowledge from large language models (e.g. https://arxiv.org/pdf/2110.07178.pdf) for targeted domains. These approaches are not discussed or compared against. Are these approaches ineffective for scientific domains, and why?\n3. The paper does not provide a detailed analysis of the computational resources required for training and fine-tuning. \n4. In scientific domains such as material science that are evolving and demand factuality, it might make sense to augment language models with external knowledge (i.e., rely on semi-parametric knowledge) instead of trying to pack everything in the model parameters (parametric knowledge).", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes an iterative instruction generation method for materials science that can also apply to other scientific domains. In a generation step, ChatGPT generates domain-specific instructions which are then evaluated by an independent evaluator. Additionally, an evaluator refines poor quality instructions based on feedback, leading to higher quality instructions and model performance in the targeted domain. Following this approach, they obtain a billion-parameter LLM specialized in the materials science domain. Experiments on benchmarks demonstrate the superiority of proposed model over existing language models in materials science.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes a methodology for instruction finetuning an LLM for the domain of materials science, including the release of a new benchmark for this domain. While reviewers agreed that the application area and new benchmark were interesting, they weren't very excited about the novelty of the approach itself compared with previous work, and felt that the paper was in some places unclear or that details were missing. The authors should do a careful pass to revise the manuscript for clarity and inclusion of details following the reviewers' feedback in subsequent drafts.
Decision: Accept-Findings