MatSci-NLP: Evaluating Scientific Language Models on Materials Science Language Tasks Using Text-to-Schema Modeling
Year: 2023

Abstract:
We present MatSci-NLP, a natural language benchmark for evaluating the performance of natural language processing (NLP) models on materials science text. We construct the benchmark from publicly available materials science text data to encompass seven different NLP tasks, including conventional NLP tasks like named entity recognition and relation classification, as well as NLP tasks specific to materials science, such as synthesis action retrieval which relates to creating synthesis procedures for materials. We study various BERT-based models pretrained on different scientific text corpora on MatSci-NLP to understand the impact of pretraining strategies on understanding materials science text. Given the scarcity of high-quality annotated data in the materials science domain, we perform our fine-tuning experiments with limited training data to encourage the generalize across MatSci-NLP tasks. Our experiments in this low-resource training setting show that language models pretrained on scientific text outperform BERT trained on general text. MatBERT, a model pretrained specifically on materials science journals, generally performs best for most tasks. Moreover, we propose a unified text-to-schema for multitask learning on MatSci-NLP and compare its performance with traditional fine-tuning methods. In our analysis of different training methods, we find that our proposed text-to-schema methods inspired by question-answering consistently outperform single and multitask NLP fine-tuning methods. The code and datasets are publicly available1.

heading: 1 Introduction
text: Materials science comprises an interdisciplinary scientific field that studies the behavior, properties and applications of matter that make up materials systems. As such, materials science often requires
∗ Equal contribution. † Corresponding author. Canada CIFAR AI Chair.
1https://github.com/BangLab-UdeM-Mila/ NLP4MatSci-ACL23
deep understanding of a diverse set of scientific disciplines to meaningfully further the state of the art. This interdisciplinary nature, along with the great technological impact of materials advances and growing research work at the intersection of machine learning and materials science (Miret et al.; Pilania, 2021; Choudhary et al., 2022), makes the challenge of developing and evaluating natural language processing (NLP) models on materials science text both interesting and exacting.
The vast amount of materials science knowledge stored in textual format, such as journal articles, patents and technical reports, creates a tremendous opportunity to develop and build NLP tools to create and understand advanced materials. These tools could in turn enable faster discovery, synthesis and deployment of new materials into a wide variety of application, including clean energy, sustainable manufacturing and devices.
Understanding, processing, and training language models for scientific text presents distinctive challenges that have given rise to the creation of specialized models and techniques that we review in Section 2. Additionally, evaluating models on scientific language understanding tasks, especially in materials science, often remains a laborious task given the shortness of high-quality annotated data and the lack of broad model benchmarks. As such, NLP research applied to materials science remains in the early stages with a plethora of ongoing research efforts focused on dataset creation, model training and domain specific applications.
The broader goal of this work is to enable the development of pertinent language models that can be applied to further the discovery of new material systems, and thereby get a better sense of how well language models understand the properties and behavior of existing and new materials. As such, we propose MatSci-NLP, a benchmark of various NLP tasks spanning many applications in the materials science domain described in Section 3. We utilize
3621
this benchmark to analyze the performance of various BERT-based models for MatSci-NLP tasks under distinct textual input schemas described in Section 4. Concretely, through this work we make the following research contributions:
• MatSci-NLP Benchmark: We construct the first broad benchmark for NLP in the materials science domain, spanning several different NLP tasks and materials applications. The benchmark contents are described in Section 3 with a general summary and data sources provided in Table 1. The processed datasets and code will be released after acceptance of the paper for reproducibility.
• Text-to-Schema Multitasking: We develop a set of textual input schemas inspired by question-answering settings for fine-tuning language models. We analyze the models’ performance on MatSci-NLP across those settings and conventional single and multitask fine-tuning methods. In conjunction with this analysis, we propose a new Task-Schema input format for joint multitask training that increases task performance for all fine-tuned language models.
• MatSci-NLP Analysis: We analyze the performance of various BERT-based models pretrained on different scientific and nonscientific text corpora on the MatSci-NLP benchmark. This analysis help us better understand how different pretraining strategies affect downstream tasks and find that MatBERT (Walker et al., 2021), a BERT model trained on materials science journals, generally performs best reinforcing the importance of curating high-quality pretraining corpora.
We centered our MatSci-MLP analysis on exploring the following questions:

heading: Q1 How does in-domain pretraining of language models affect the downstream performance on
text: MatSci-NLP tasks? We investigate the performance of various models pretrained on different kinds of domain-specific text including materials science, general science and general language (BERT (Devlin et al., 2018)). We find that MatBERT generally performs best and that language models pretrained on diverse scientific texts outperform a general
language BERT. Interestingly, SciBERT (Beltagy et al., 2019) often outperforms materials science language models, such as MatSciBERT (Gupta et al., 2022) and BatteryBERT (Huang and Cole, 2022).
Q2 How do in-context data schema and multitasking affect the learning efficiency in lowresource training settings? We investigate how several input schemas shown in Figure 1 that contain different kinds of information affect various domain-specific language models and propose a new Task-Schema method. Our experiments show that our proposed TaskSchema method mostly performs best across all models and that question-answering inspired schema outperform single task and multitask fine-tuning settings.

heading: 2 Background
text: The advent of powerful NLP models has enabled the analysis and generation of text-based data across a variety of domains. BERT (Devlin et al., 2018) was one of the first large-scale transformerbased models to substantially advance the state-ofthe-art by training on large amounts of unlabeled text data in a self-supervised way. The pretraining procedure was followed by task-specific finetuning, leading to impressive results on a variety of NLP task, such as named entity recognition (NER), question and answering (QA), and relation classification (Hakala and Pyysalo, 2019; Qu et al., 2019; Wu and He, 2019). A significant collection of large language models spanning millions to billions of parameters followed the success of BERT adopting a similar approach of pretraining on vast corpora of text with task-specific fine-tuning to push the state-of-the-art for in natural language processing and understanding (Raffel et al., 2020; Brown et al., 2020; Scao et al., 2022).

heading: 2.1 Scientific Language Models
text: The success of large language models on general text motivated the development of domainspecific language models pretrained on custom text data, including text in the scientific domain: SciBERT (Beltagy et al., 2019), ScholarBERT (Hong et al., 2022) and Galactica (Taylor et al., 2022) are pretrained on general corpus of scientific articles; BioBERT (Lee et al., 2020), PubMedBERT (Gu et al., 2021), BioMegatron (Shin et al., 2020) and Sci-Five (Phan et al., 2021) are pretrained on
various kinds of biomedical corpora; MatBERT (Walker et al., 2021), MatSciBERT (Gupta et al., 2022) are pretrained on materials science specific corpora; and BatteryBERT (Huang and Cole, 2022) is pretrained on a corpus focused on batteries. Concurrently, several domain-specific NLP benchmarks were established to assess language model performance on domain-specific tasks, such as QASPER (Dasigi et al., 2021) and BLURB (Gu et al., 2021) in the scientific domain, as well as PubMedQA (Jin et al., 2019), BioASQ (Balikas et al., 2015), and Biomedical Language Understanding Evaluation (BLUE) (Peng et al., 2019) in the biomedical domain.

heading: 2.2 NLP in Materials Science
text: The availability of openly accessible, high-quality corpora of materials science text data remains highly restricted in large part because data from peer-reviewed journals and scientific documents is usually subject to copyright restrictions, while open-domain data is often only available in difficult-to-process PDF formats (Olivetti et al., 2020; Kononova et al., 2021). Moreover, specialized scientific text, such as materials synthesis procedures containing chemical formulas and reaction notation, require advanced data mining techniques for effective processing (Kuniyoshi et al., 2020; Wang et al., 2022b). Given the specificity, complexity, and diversity of specialized language in scientific text, effective extraction and processing remain an active area of research with the goal of building relevant and sizeable text corpora for pretraining scientific language models (Kononova et al., 2021).
Nonetheless, materials science-specific language models, including MatBERT (Walker et al., 2021), MatSciBERT (Gupta et al., 2022), and BatteryBERT (Huang and Cole, 2022), have been trained on custom-built pretraining dataset curated by different academic research groups. The pretrained models and some of the associated fine-tuning data have been released to the public and have enabled further research, including this work.
The nature of NLP research in materials science to date has also been highly fragmented with many research works focusing on distinct tasks motivated by a given application or methodology. Common ideas among many works include the prediction and construction of synthesis routes for a variety of materials (Mahbub et al., 2020; Karpovich et al.,
2021; Kim et al., 2020), as well as the creation of novel materials for a given application (Huang and Cole, 2022; Georgescu et al., 2021; Jensen et al., 2021), both of which relate broader challenges in the field of materials science.

heading: 3 MatSci-NLP Benchmark
text: Through the creation of MatSci-NLP, we aim to bring together some of the fragmented data across multiple research works for a wide-ranging materials science NLP benchmark. As described in Section 2, the availability of sizeable, high-quality and diverse datasets remain a major obstacle in applying modern NLP to advance materials science in meaningful ways. This is primarily driven by a high cost of data labeling and the heterogeneous nature of materials science. Given those challenges, we created MatSci-NLP by unifying various publicly available, high-quality, smaller-scale datasets to form a benchmark for fine-tuning and evaluating modern NLP models for materials science applications. MatSci-NLP consists of seven NLP tasks shown in Table 1, spanning a wide range of materials categories including fuel cells (Friedrich et al., 2020), glasses (Venugopal et al., 2021), inorganic materials (Weston et al., 2019; MatSciRE, 2022), superconductors (Yamaguchi et al., 2020), and synthesis procedures pertaining to various kinds of materials (Mysore et al., 2019; Wang et al., 2022a). Some tasks in MatSci-NLP had multiple source components, meaning that the data was curated from multiple datasets (e.g. NER), while many were obtained from a single source dataset.
The data in MatSci-NLP adheres to a standard JSON-based data format with each of the samples containing relevant text, task definitions, and annotations. These can in turn be refactored into different input schemas, such as the ones shown in Figure 1 consisting of 1) Input: primary text jointly with task descriptions and instructions, and 2) Output: query and label, which we perform in our text-to-schema modeling described in Section 4. Next, we describe the tasks in MatSci-NLP in greater detail:
• Named Entity Recognition (NER): The NER task requires models to extract summarylevel information from materials science text and recognize entities including materials, descriptors, material properties, and applications amongst others. The NER task predicts the best entity type label for a given text span
si with a non-entity span containing a “null” label. MatSci-NLP contains NER task data adapted from Weston et al. (2019); Friedrich et al. (2020); Mysore et al. (2019); Yamaguchi et al. (2020).
• Relation Classification: In the relation classification task, the model predicts the most relevant relation type for a given span pair
(si, sj). MatSci-NLP contains relation classification task data adapted from Mysore et al. (2019); Yamaguchi et al. (2020); MatSciRE (2022).
• Event Argument Extraction: The event argument extraction task involves extracting event arguments and relevant argument roles. As there may be more than a single event for a given text, we specify event triggers and require the language model to extract corresponding arguments and their roles. MatSciNLP contains event argument extraction task data adapted from Mysore et al. (2019); Yamaguchi et al. (2020).
• Paragraph Classification: In the paragraph classification task adapted from Venugopal et al. (2021), the model determines whether a given paragraph pertains to glass science.
• Synthesis Action Retrieval (SAR): SAR is a materials science domain-specific task that defines eight action terms that unambiguously identify a type of synthesis action to describe a synthesis procedure. MatSci-NLP adapts SAR data from Wang et al. (2022a) to ask language models to classify word tokens into pre-defined action categories.
• Sentence Classification: In the sentence
classification task, models identify sentences that describe relevant experimental facts based on data adapted from Friedrich et al. (2020).
• Slot Filling: In the slot-filling task, models extract slot fillers from particular sentences based on a predefined set of semantically meaningful entities. In the task data adapted from Friedrich et al. (2020), each sentence describes a single experiment frame for which the model predicts the slots in that frame.
The tasks contained in MatSci-NLP were selected based on publicly available, high-quality annotated materials science textual data, as well as their relevance to applying NLP tools to materials science. Conventional NLP tasks (NER, Relation Classification, Event Argument Extraction, Paragraph Classification, Sentence Classification) enable materials science researchers to better process and understand relevant textual data. Domain specific tasks (SAR, Slot Filling) enable materials science research to solve concrete challenges, such as finding materials synthesis procedures and realworld experimental planning. In the future, we aim to augment to current set of tasks with additional data and introduce novel tasks that address materials science specific challenges with NLP tools.

heading: 4 Unified Text-to-Schema Language Modeling
text: As shown in Figure 1, a given piece of text can include multiple labels across different tasks. Given this multitask nature of the MatSci-NLP benchmark, we propose a new and unified Task-Schema multitask modeling method illustrated in Figure 2 that covers all the tasks in the MatSci-NLP dataset. Our approach centers on a unified text-to-schema modeling approach that can predict multiple tasks simultaneously through a unified format. The underlying language model architecture is made up of modular components, including a domainspecific encoder model (e.g. MatBERT, MatSciBERT, SciBERT), and a generic transformer-based decoder, each of which can be easily exchanged with different pretrained domain-specific NLP models. We fine-tune these pretrained language models and the decoder with collected tasks in MatSciNLP using the procedure described in Section 4.3.
The unified text-to-schema provides a more structured format to training and evaluating language model outputs compared to seq2seq and text-
to-text approaches (Raffel et al., 2020; Luong et al., 2015). This is particularly helpful for the tasks in MatSci-NLP given that many tasks can be reformulated as classification problems. NER and Slot Filling, for example, are classifications at the tokenlevel, while event arguments extraction entails the classification of roles of certain arguments. Without a predefined schema, the model relies entirely on unstructured natural language to provide the answer in a seq2seq manner, which significantly increases the complexity of the task and also makes it harder to evaluate performance. The structure imposed by text-to-schema method also simplifies complex tasks, such as event extraction, by enabling the language model to leverage the structure of the schema to predict the correct answer. We utilize the structure of the schema in decoding and evaluating the output of the language models, as described further in Section 4.3 in greater detail.
Moreover, our unified text-to-schema approach alleviates error propagation commonly found in multitask scenarios (Van Nguyen et al., 2022; Lu et al., 2021), enables knowledge sharing across multiple tasks and encourages the fine-tuned language model to generalize across a broader set of text-based instruction scenarios. This is supported by our results shown in Section 5.2 showing textto-schema outperforming conventional methods.

heading: 4.1 Language Model Formulation
text: The general purpose of our model is to achieve multitask learning by a mapping function (f ) between input (x), output (y), and schema (s), i.e., f(x, s) = y. Due to the multitasking nature of our setting, both inputs and outputs can originate from different tasks n, i.e. x = [xt1, xt2, ...xtn] and y = [yt1, yt2, ...ytn], all of which fit under a common schema (s). Given the presence of domain-specific materials science language, our model architecture includes a domain-specific BERT encoder and a transformer decoder. All BERT encoders and transformer decoders share the same general architecture, which relies on a self-attention mechanism: Given an input sequence of length N , we compute a set of attention scores, A = softmax(QTK/( √ dk)). Next, we compute the weighted sum of the value vectors, O = AV , where Q, K, and V are the query, key, and value matrices, and dk is the dimensionality of the key vectors.
Additionally, the transformer based decoder dif-
fer from the domain specific encoder by: 1) Applying masking based on the schema applied to ensure that it does not attend to future positions in the output sequence. 2) Applying both selfattention and encoder-decoder attention to compute attention scores that weigh the importance of different parts of the output sequence and input sequence. The output of the self-attention mechanism (O1) and the output of the encoderdecoder attention mechanism (O2) are concatenated and linearly transformed to obtain a new hidden state, H = tanh(Wo[O1; O2] + bo) with Wo and bo being the weight and biases respectively. The model then applies a softmax to H to generate the next element in the output sequence P = softmax(WpH + bp) , where P is a probability distribution over the output vocabulary.

heading: 4.2 Text-To-Schema Modeling
text: As shown in Figure 1, our schema structures the text data based on four general components: text, description, instruction options, and the predefined answer schema.
• Text specifies raw text from the literature that is given as input to the language model.
• Description describes the task for a given text
according to a predefined schema containing the task name and the task arguments.
• Instruction Options contains the core explanation related to the task with emphasis on three different types: 1) Potential choices of answers; 2) Example of an input/output pair corresponding to the task; 3) Task-Schema : our predefined answer schema illustrated in Figure 2.
• Answer describes the correct label of each task formatted as a predefined answer schema that can be automatically generated based on the data structure of the task.

heading: 4.3 Language Decoding & Evaluation
text: Evaluating the performance of the language model on MatSci-NLP requires determining if the text generated by the decoder is valid and meaningful in the context of a given task. To ensure consistency in evaluation, we apply a constrained decoding procedure consisting of two steps: 1) Filtering out invalid answers through the predefined answer schema shown in Figure 2 based on the structure of the model’s output; 2) Match the model’s prediction with the most similar valid class given by the annotation for the particular task. For example, if for the NER task shown in Figure 1 the model’s predicted token is “BaCl2 2H2O materials”, it will be matched with the NER label of “material”, which is then used as the final prediction for computing losses and evaluating performance. This approach essentially reformulates each task as a classification problem where the classes are provided based on the labels from the tasks in MatSci-NLP. We then apply a cross-entropy loss for model fine-tuning based on the matched label from the model output. The matching procedure simplifies the language modeling challenge by not requiring an exact match of the predicted tokens with the task labels. This in turns leads to a more comprehensible signal in the fine-tuning loss function.

heading: 5 Evaluation and Results
text: Our analysis focuses on the questions outlined in Section 1: 1) Studying the effectiveness of domainspecific language models as encoders, and 2) Analyzing the effect of different input schemas in resolving MatSci-NLP tasks. Concretely, we study the performance of the language models and language schema in a low resource setting where we
interval of two standard deviations. We denote the best performing encoder model and those that outperform the general language BERT according to the micro-f1 with orange shading with MatBERT and SciBERT performing best on most tasks and ScholarBERT and general language BERT generally performing worst.
perform fine-tuning on different pretrained BERT models with limited data from the MatSci-NLP benchmark. This low-resource setting makes the learning problem harder given that the model has to generalize on little amount of data. Moreover, this setting approximates model training with very limited annotated data, which is commonly found in materials science as discussed in Section 2. In our experiments, we split the data in MatSci-NLP into 1% training subset and a 99% testing subset for evaluation. None of the evaluated encoder models were exposed to the fine-tuning data in advance of our experiments and therefore have to rely on the knowledge acquired during their respective pretraining processes. We evaluate the results of our experiments using micro-F1 and macro-F1 scores of the language model predictions on the test split of the MatSci-NLP that were not exposed during fine-tuning.

heading: 5.1 How does in-domain pretraining of language models affect the downstream performance on MatSci-NLP tasks? (Q1)
text: Based on the results shown in Table 2, we can gather the following insights:
First, domain-specific pretraining affects model performance. We perform fine-tuning on various models pretrained on domain-specific corpora in a low-resource setting and observe that: i) MatBert, which was pretrained on textual data from materials science journals, generally performs best for most tasks in the MatSci-NLP benchmark with SciBERT
generally performing second best. The high performance of MatBERT suggests that materials science specific pretraining does help the language models acquire relevant materials science knowledge. Yet, the underperformance of MatSciBERT compared to MatBERT and SciBERT indicates that the curation of pretraining data does significantly affect performance. ii) The importance of the pretraining corpus is further reinforced by the difference in performance between SciBERT and ScholarBERT, both of which were trained on corpora of general scientific text, but show vastly different results. In fact, ScholarBERT underperforms all other models, including the general language BERT, for all tasks except event argument extraction where ScholarBERT performs best compared to all other models. iii) The fact that most scientific BERT models outperform BERT pretrained on general language suggests that pretraining on high-quality scientific text is beneficial for resolving tasks involving materials science text and potentially scientific texts from other domains. This notion of enhanced performance on MatSci-NLP when pretraining on scientific text is further reinforced by the performance of BioBERT by Wada et al. (2020). BioBERT outperforms BERT on most tasks even though it was trained on text from the biomedical domain that has minor overlap with the materials science domain. This strongly indicates that scientific language, regardless of the domain, has a significant distribution shift from general language that is used to pretrain common language models.
to heatmap ranging from best and worst. The concentration of red hues on right side indicates that the questionanswering inspiring schema generally outperform conventional fine-tuning method. Our proposed Task-Schema generally outperforms all other schemas across most enconder models.
Second, imbalanced datasets in MatSci-NLP skew performance metrics: We can see from Table 2 that the micro-F1 scores are significantly higher than the macro-f1 across all tasks. This indicates that the datasets used in the MatSci-NLP are consistently imbalanced, including in the binary classification tasks, and thereby push the micro-F1 higher compared to the macro-F1 score. In the case of paragraph classification, for example, the number of positive examples is 492 compared with the total number of 1500 samples. As such, only models with a micro-F1 score above 0.66 and macro-F1 above 0.5 can be considered to have semantically meaningful understanding of the task. This is even more pronounced for sentence classification where only 876/9466 ≈ 10% corresponds to one label. All models except ScholarBERT outperform a default guess of the dominant class for cases. While imbalanced datasets may approximate some realworld use cases of materials science text analysis, such as extracting specialized materials information, a highly imbalanced can be misguiding in evaluating model performance.
To alleviate the potentially negative effects of imbalanced data, we suggest three simple yet effective methods: 1) Weighted loss functions: This involves weighting the loss function to give higher weights to minority classes. Focal loss (Lin et al., 2017), for example, is a loss function that dy-
namically modulates the loss based on the prediction confidence, with greater emphasis on more difficult examples. As such, Focal loss handles class imbalance well due to the additional attention given to hard examples of the minority classes. 2) Class-balanced samplers: Deep learning frameworks, such as Pytorch, have class-balanced batch samplers that can be used to oversample minority classes within each batch during training, which can help indirectly address class imbalance. 3) Model architecture tweaks: The model architecture and its hyper-parameters can be adjusted to place greater emphasis on minority classes. For example, one can apply separate prediction heads for minority classes or tweak L2 regularization and dropout to behave differently for minority and majority classes.

heading: 5.2 How do in-context data schema and multitasking affect the learning efficiency in low-resource training settings? (Q2)
text: To assess the efficacy of the proposed textual schemas shown in Figure 1, we evaluate four different QA-inspired schemas: 1) No Explanations - here the model receives only the task description; 2) Potential Choices - here the model receives the class labels given by the task; 3) Examples - here the model receives an example of a correct answer, 4) Task-Schema - here the model receives our pro-
posed textual schema. We compare the schemas to three conventional fine-tuning methods: 1) Single Task - the traditional method to solve each task separately using the language model and a classification head; 2) Single Task Prompt - here we change the format of the task to the same QA-format as “No Explanations”, but train each task separately; 3) MMOE by Ma et al. (2018) uses multiple encoders to learn multiple hidden embeddings, which are then weighed by a task-specific gate unit and aggregated to the final hidden embedding using a weighted sum for each task. Next, a task-specific classification head outputs the label probability distribution for each task.
Based on the results shown in Table 3, we gather the following insights:
First, Text-to-Schema methods perform better for all language models. Overall, the Task-Schema method we proposed performs best across all tasks in the MatSci-NLP benchmark. The questionanswering inspired schema (“No Explanations”, “Potential Choices”, “Examples”, “Task-Schema ”) perform better than fine-tuning in a traditional single task setting, single task prompting, as well as fine-tuning using the MMOE multitask method. This holds across all models for all the tasks in MatSci-NLP showing the efficacy of structured language modeling inspired by question-answering.
Second, schema design affects model performance. The results show that both the pretrained model and the input format affect performance. This can be seen by the fact that while all scientific models outperform general language BERT using the Task-Schema method, BERT outperforms some models, mainly ScholarBERT and BioBERT, in the other text-to-schema settings and the conventional training settings. Nevertheless, BERT underperforms the stronger models (MatBERT, SciBERT, MatSciBERT) across all schema settings for all tasks in MatSci-NLP, further emphasizing the importance of domain-specific model pretraining for materials science language understanding.

heading: 6 Conclusion and Future Works
text: We proposed MatSci-NLP, the first broad benchmark on materials science language understanding tasks constructed from publicly available data. We further proposed text-to-schema multitask modeling to improve the model performance in lowresource settings. Leveraging MatSci-NLP and text-to-schema modeling, we performed an in-
depth analysis of the performance of various scientific language models and compare text-toschema language modeling methods with other input schemas, guided by (Q1) addressing the pretrained models and (Q2) addressing the textual schema. Overall, we found that the choice of pretrained models matters significantly for downstream performance on MatSci-NLP tasks and that pretrained language models on scientific text of any kind often perform better than pretrained language models on general text. MatBERT generally performed best, highlighting the benefits of pretraining with high-quality domain-specific language data. With regards to the textual schema outlined in (Q2), we found that significant improvements can be made by improving textual schema showcasing the potential of fine-tuning using structured language modeling.
The proposed encoder-decoder architecture, as well as the proposed multitask schema, could also be useful for additional domains in NLP, including both scientific and non-scientific domains. The potential for open-domain transferability of our method is due to: 1) Our multitask training method and associated schemas do not depend on any domain-specific knowledge, allowing them to be easily transferred to other domains. 2) The encoder of our proposed model architecture can be exchanged in a modular manner, which enables our model structure to be applied across multiple domains. 3) If the fine-tuning data is diverse across a wide range of domains, our method is likely to learn general language representations for opendomain multitask problems. Future work could build upon this paper by applying the model and proposed schema to different scientific domains where fine-tuning data might be sparse, such as biology, physics and chemistry. Moreover, future work can build upon the proposed schema by suggesting novel ways of modeling domain-specific or general language that lead to improvements in unified multi-task learning.
Limitations
One of the primary limitations of NLP modeling in materials science, including this work, is the low quantity of available data as discussed in Section 2. This analysis is affected by this limitation as well given that our evaluations were performed in a low-data setting within a dataset that was already limited in size. We believe that future work can improve upon this study by applying larger datasets, both in the number of samples and in the scope of tasks, to similar problem settings. The small nature of the datasets applied in this study also presents the danger that some of the models may have memorized certain answers instead of achieving a broader understanding, which could be mitigated by enlarging the datasets and making the tasks more complex.
Moreover, we did not study the generalization of NLP models beyond the materials science domain, including adjacent domains such as chemistry and physics. This targeted focus was intentional but imposes limitations on whether the proposed techniques and insights we gained from our analysis are transferable to other domains, including applying NLP models for scientific tasks outside of materials science.
Another limitation of our study is the fact that we focused on BERT-based models exclusively and did not study autoregressive models, including large language models with billions of parameters highlighted in the introduction. The primary reason for focusing on BERT-based models was the diversity of available models trained on different scientific text corpora. Large autoregressive models, on the other hand, are mostly trained on general text corpora with some notable exceptions, such as Galactica (Taylor et al., 2022). We believe that future work analyzing a greater diversity of language models, including large autoregressive models pretrained on different kinds of text, would significantly strengthen the understanding surrounding the ability of NLP models to perform text-based tasks in materials science.
While the results presented in this study indicate that domain-specific pretraining can lead to noticeable advantages in downstream performance on text-based materials science tasks, we would like to highlight the associated risks and costs of pretraining a larger set of customized language models for different domains. The heavy financial and environmental costs associated with these pretrain-
ing procedures merit careful consideration of what conditions may warrant expensive pretraining and which ones may not. When possible, we encourage future researchers to build upon existing large models to mitigate the pretraining costs.
Broader Impacts and Ethics Statement
Our MatSci-NLP benchmark can help promote the research on NLP for material science, an important and growing research field. We expect that the experience we gained from the material science domain can be transferred to other domains, such as biology, health, and chemistry. Our Textto-Schema also helps with improving NLP tasks’ performance in low-resource situations, which is a common challenge in many fields.
Our research does not raise major ethical concerns.
Acknowlegments
This work is supported by the Mila internal funding - Program P2-V1: Industry Sponsored Academic Labs (project number: 10379), the Canada CIFAR AI Chair Program, and the Canada NSERC Discovery Grant (RGPIN-2021-03115).

heading: A Experimental Details
text: We performed fine-tuning experiments using a single GPU with a learning rate was 2e-5, the hidden size of the encoders being 768, except ScholarBERT which is 1024, using the Adam (Kingma and Ba, 2014) optimizer for a max number of 20 training epochs with early stopping. All models are implemented with Python and PyTorch, and repeated five times to report the average performance. The full set of hyperparameters is available in our publicaly released code at https: //github.com/BangLab-UdeM-Mila/ NLP4MatSci-ACL23.

heading: B Additional Text-to-Schema Experiments
text: To arrive at our data presented in Table 3, we conducted experiments for all the language models across all tasks in MatSci-NLP. The results for seven tasks in MatSci-NLP are shown in subsequent tables:
• Named Entity Recognition in Table 4.
• Relation Classification in Table 5.
• Event Argument Extraction in Table 6.
• Paragraph Classification in Table 7.
• Synthesis Action Retrieval in Table 8.
• Sentence Classification in Table 9.
• Slot Filling in Table 10.
The experimental results summarized in the aforementioned tables reinforce the conclusions in our analysis of (Q2) in Section 5.2 with the text-toschema based fine-tuning method generally outperforming the conventional single and multitask methods across all tasks and all language models.
interval of two standard deviations. We highlight the best performing method.
interval of two standard deviations. We highlight the best performing method.
interval of two standard deviations. We highlight the best performing method.
interval of two standard deviations. We highlight the best performing method.
interval of two standard deviations. We highlight the best performing method.
interval of two standard deviations. We highlight the best performing method.
of two standard deviations. We highlight the best performing method.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Yes - in Section 7.
3 A2. Did you discuss any potential risks of your work? Yes - in Section 7.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Yes - in Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Yes - in Section 3.
3 B1. Did you cite the creators of artifacts you used? Yes - in Section 3.
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Yes - in Section 3.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Yes - in Section 3 and Section 7.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Yes - in Section 3.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Yes - in Section 3.
C 3 Did you run computational experiments? Yes - in Section 5.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Yes - in Appendix Section A.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Yes - in Appendix Section A.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Yes - in Section 5.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Future work could build upon this paper by applying the model and proposed schema to different scientific domains where fine-tuning data might be sparse, such as biology, physics and chemistry. Moreover, future work can build upon the proposed schema by suggesting novel ways of modeling domain-specific or general language that lead to improvements in unified multi-task learning. Limitations
One of the primary limitations of NLP modeling in materials science, including this work, is the low quantity of available data as discussed in Section 2. This analysis is affected by this limitation as well given that our evaluations were performed in a low-data setting within a dataset that was already limited in size. We believe that future work can improve upon this study by applying larger datasets, both in the number of samples and in the scope of tasks, to similar problem settings. The small nature of the datasets applied in this study also presents the danger that some of the models may have memorized certain answers instead of achieving a broader understanding, which could be mitigated by enlarging the datasets and making the tasks more complex. Moreover, we did not study the generalization of NLP models beyond the materials science domain, including adjacent domains such as chemistry and physics. This targeted focus was intentional but imposes limitations on whether the proposed techniques and insights we gained from our analysis are transferable to other domains, including applying NLP models for scientific tasks outside of materials science. Another limitation of our study is the fact that we focused on BERT-based models exclusively and did not study autoregressive models, including large language models with billions of parameters highlighted in the introduction. The primary reason for focusing on BERT-based models was the diversity of available models trained on different scientific text corpora. Large autoregressive models, on the other hand, are mostly trained on general text corpora with some notable exceptions, such as Galactica (Taylor et al. , 2022). We believe that future work analyzing a greater diversity of language models, including large autoregressive models pretrained on different kinds of text, would significantly strengthen the understanding surrounding the ability of NLP models to perform text-based tasks in materials science. While the results presented in this study indicate that domain-specific pretraining can lead to noticeable advantages in downstream performance on text-based materials science tasks, we would like to highlight the associated risks and costs of pretraining a larger set of customized language models for different domains. The heavy financial and environmental costs associated with these pretrain-
ing procedures merit careful consideration of what conditions may warrant expensive pretraining and which ones may not. When possible, we encourage future researchers to build upon existing large models to mitigate the pretraining costs