言語処理学会 第30回年次大会 発表論文集（2024年3月）

Empirical Study on Text Classiﬁcation of
Small Science Domain Datasets
Shanshan Liu1 Masashi Ishii2 Yuji Matsumoto1
1 Center for Advanced Intelligence Project, RIKEN
2 Material Database Group, MaDIS, NIMS
{shanshan.liu, yuji.matsumoto}@riken.jp
ishii.masashi@nims.go.jp

Abstract
As for the classiﬁcation of scientiﬁc texts, there are noclear states on how much data is needed for model training and what performance mainstream text classiﬁers canachieve. This paper uses the number of training papers asa variable to train the models for sentence classiﬁcation
tasks on the thermoelectric material synthesis proceduredataset and the polymer biodegradability dataset, and analyzes the results of two classiﬁers based on LLaMA-2-7b
and SciBERT-base. We aim to provide model baselines
and corresponding data requirements for reference.

1

Introduction

For information extraction tasks on scientiﬁc texts, theﬁrst important step is to obtain sentences or text blocks in
the literature that describe the target information.
In our previous work [1], we found that the performanceof text classiﬁcation1） has a huge impact on subsequent
entity recognition and relation extraction tasks. However,further performance improvement is quite challenging.
The application scenarios of scientiﬁc information extraction are mostly to extract a speciﬁc type of information(such as the material synthesis procedure) in speciﬁc domains. It is diﬃcult to collect enough labelled data toadequately train a classiﬁer, and few language models havebeen pretrained with the domain text. Additionally, there
is a lack of empirical research on this task. Advanced textclassiﬁers are often only validated on general text datasets(e.g., the sentiment of ﬁnancial news [2] or movie reviews[3]). Researchers have diﬃculty estimating data requirements and model performance.
1） ”Text classiﬁcation” in this paper refers to sentence classiﬁcation.

To remove some hindrances to further research, we report the results of two mainstream text classiﬁers on two
datasets we built – Thermoelectric Material Synthesis Procedure (TMSP) dataset and Polymer Biodegradability (PBIO) dataset.
Scientiﬁc text classiﬁers based on SciBERT [4] were
dominant before the emergence of the large language models (LLMs), and with the development of the LLMs, studies using LLaMA-2-7b [5] for text classiﬁcation have produced top-of-the-line results on general benchmarks [6].
SciBERT-base is trained on computer science and broad
biomedical domain, which has been widely used in various tasks addressing scientiﬁc texts. LLaMA-2-7b is an
open LLM trained on 2 trillion tokens of data. Its promising performance in a variety of tasks has made it a hot
topic in recent researches. Therefore, these two pretrainedmodels can be used as suitable baselines.
Our works may help to answer two questions:• How much annotated data (number of papers) isneeded for supervised training of text classiﬁer?
• How do the LLaMA-2-7b- and SciBERT-based classiﬁers perform on small scientiﬁc datasets?
Depending on the characteristics of the text, diﬀerent
amounts of training data are required. The more specialized texts demands more data and in-domain language
models. There is huge potential for supervised learning
using small datasets - 5 TMSP-type papers （around 20
positive sentences）/ 20 P-BIO-type papers（arround 330
positive sentences）can be trained to achieve acceptableperformance. Even if LLMs perform well in general textclassiﬁcation, they may not be good for scientiﬁc text.
SciBERT-based classiﬁer is a trivial but eﬀective solution
for texts in which expertise terms are more common.

― 2891 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Table 1 Statistics of Datasets
Train
Dev
Test
#Paper Avg. #Sent Avg. #Pos Sent #Paper Avg. #Sent Avg. #Pos Sent #Paper Avg. #Sent Avg. #Pos Sent
TMSP
368
163.4
5.04
93
154.6
5.0
115
164.7
5.0
P-BIO
50
189.3
16.5*
5
125.4
6.4
5
100.0
6.8
#Paper, Avg. #Sent, Avg. #Pos Sent are the number of papers, the average number of sentences per paper, and the average number of
positive sentences per paper.
* Training set includes several polymer biodegradability reviews, making the distribution diﬀerent from the Test set.
Table 2

Index Dataset

Instance examples

Label

Sentence
Polycrystalline samples of CaCu3Ti4-xRuxO12 were prepared by a solid-state reaction method.

1

TMSP

Pos

2

TMSP

Pos

3

TMSP

Neg

The product was ﬁnely ground, pressed into a pellet, and sintered in air
at 1000-1050 ◦C for 20 h.
In the solid solution of CaCu3Ti4-xRuxO12, one can observe how the localized electrons on the
Cu2+ sites become itinerant with Ru substitution.
The enzymatic degradation of the rubber polymer poly(cis-1,4-isoprene),

4

P-BIO

Pos

e.g. by the latex clearing protein Lcp1VH2 of Gordonia polyisoprenivorans VH2
has been demonstrated with latex milk or pure isoprene-rubber particles, recently.

5

P-BIO

Neg

Polyethylene degrading ability of the isolates has been assessedindividually in a synthetic media containing polyethylene as a carbon source.
Note: Bold mentions are the salient information the subsequent extraction tasks focus on. Italicized mentions have thesame expression as entities that need to be identiﬁed, but do not need to be extracted in subsequent tasks.

2

ria/fungi) that can degrade it from scientiﬁc papers. The

Methodsentences with both polymer names and microorganism

We evaluate the text classiﬁers based on LLaMA-2-7b
and SciBERT-base pretrained language models (PLMs).
To investigate the quantitative demands on the training data,
we randomly selected diﬀerent numbers of papers from thetraining set for ﬁne-tuning while keeping the validation andnames is positive.
Statistics of datasets are shown in Table 1. We show
some sample sentences of both datasets in Table 2.

3.2

Model

test sets unchanged. On the TMSP dataset, the experiments

The SciBERT-based classiﬁer consists of the SciBERT-

are carried out with [5, 10, 20, 30, 40, 50, 60, 80, 100, 120,

base2）, two MLP layers and one linear layer. The last140, 150, 160, 180, 200, 250, 300, 350] papers, while on

hidden state of the ﬁrst token - [CLS] token - is passed

the P-BIO dataset, number of papers for training is [5, 10,

to two consecutive MLP layers, and then a linear layer for

20, 30, 40, 50] respectively.

classiﬁcation. To determine the eﬀect of domain diﬀerence3

Experiment

3.1

Datasetin pre-training, we also utilized the BERT-based3）modelon P-BIO dataset. The experimental setup using BERT is
the same as for SciBERT.
As for the model using LLaMA-2-7b4）, we follow the

TMSP The Thermoelectric Materials Synthesis Proce-work of Li et al., 2023 [6]: LS-LLaMA, and used thedure dataset contains 576 papers. Materials, processes,
conditions of the process, and other relevant information

same ﬁne-tuning strategy, i.e. ﬁne-tuning with Low-Rank
Adaptation (LoRA) [7] to minimize the cross-entropy loss.

are included in the target information. The sentences de-

LS-LLaMA combines LLaMA-2-7b with a linear layer andscribing material synthesis procedures are positive.

classiﬁes the sequence by the score over the last token.

P-BIO Polymer Biodegradability dataset, a dataset con-taining 60 papers. This dataset is built to extract the relation between the polymer and the microorganism (bacte-2） https://huggingface.co/allenai/scibert scivocab
uncased3） https://huggingface.co/bert-base-uncased4） https://huggingface.co/meta-llama/Llama-2-7b-hf― 2892 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

tence. Secondly, with limited data and even fewer positive
examples, the classiﬁer may not be adequately trained.
The target sentences of TMSP contain fewer professionalterms, and the way of expressing is easy to understand. Our
experiments show that on the TMSP, both of classiﬁers canachieve an F1 of more than 80% when using 80 papers(Fig.2), and the accuracy is also above 98.7% (Fig.1). As
the number of training papers increases, they can ﬁnallyreach the highest value when using 300/250 papers (SciBERT: 83.32% F1; LLaMA-2-7b: 84.35% F1). (The wholeexperiment results is in Appendix A.1)
Figure 1

For P-BIO, a sentence in which the polymer name coex-

Accuracy on the TMSP dataset

ists with the name of the microorganism is regard as positive. Microorganism names are not common words (e.g.
Gordonia polyisoprenivorans VH2), and the sentences areoften complex and require expertise. To exceed 80% F1,
SciBERT needs 20 papers (84.85% F1, 98.00% Acc) while
40 papers are as necessary for LLaMA-2-7b (91.43% F1,
98.80% Acc). The best results contributed by SciBERTbased model with 50 papers (91.67% F1, 98.80% Acc). As
can be seen from Fig.3 that the SciBERT’s curve is still inan upward phase and the LLaMA-2-7b’s curve is oscillating, 50 papers are far from enough to get the best out ofboth models.
Figure 2

F1 score on the positive class of TMSP dataset

Considering that we are doing supervised ﬁne-tuning on
small datasets, only LLaMA-2-7b is utilized in this work.
We run 5- / 3-epoch ﬁne-tuning on SciBERT- / LLaMA2-7b-based classiﬁers and the model with the highest ac-

The results indicate that on tasks with text properties
similar to TMSP - less professional nouns, wording andsentence construction similar to general text - only 5 papers(about 100 positive examples) for training achieves an F1
above 70% and an accuracy over 98%. However, on a
narrow spectrum of text - P-BIO, achieving a 70% F1

curacy on the development set is used for testing.

score requires an in-domain PLM (SciBERT in our case)

3.3

Evaluation metricand more than 30 papers.

We report the precision (P), recall (R) and micro F1
score (F) on the positive class and the accuracy (A) for the

4.2

LLaMA-2-7b v.s SciBERT-baseEven though they are both scientiﬁc texts, the twobinary classiﬁcation.

datasets focus on two types of information, two diﬀer-

4

Results

4.1

Data requirement

ent scientiﬁc domains. Therefore, the validity of the twomodels is quite diﬀerent.
On TMSP, the two models have similar eﬀects in the

Two datasets reﬂect diﬃculties in categorising scientiﬁctexts. One is that the same expression may appear in both
positive and negative examples, such as the name of a synthetic material (e.g. CaCu3Ti4-xRuxO12) or a degradedpolymer (e.g. polyethylene) (see Table 2), which requires
the classiﬁer to capture the implication of the entire sen-

early stage, but as the data increases, LLaMA-2-7b showsoﬀ its ”large” model capabilities (84.35% F1, 98.93% Acc,
250 papers) and clearly outperforms SciBERT.
What surprises us is: The results of LLaMA-2-7b arealways signiﬁcantly lower than those of SciBERT on P-BIO
except with 5 or 40 training papers.

― 2893 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Figure 3

Accuracy and F1 score of positive classes on TMSP and P-BIO datasets

Even using the entire dataset, LLaMA-2-7b turns in lesstential of small scientiﬁc datasets. Training with 50 papers

than ideal numbers. We repeated the experiment of training

can obtain an F1 score of up to 91.67%, which is enough

LLaMA-2-7b using 50 papers three times and obtained theto promote subsequent information extraction tasks, suchsame results in ﬁrst two runs: only 2 out of 34 positive

as entity recognition and relation extraction.

examples were successfully predicted. At the third time,the model was trained well (87.88% F1, 98.40% Acc) but

4.3

still is inferior to SciBERT. We report the result of the lastSpeciﬁc domain PLM matters

The performance of BERT is close to that of LLaMA-

case in Figure 3 and Table 4.

2-7b but far from that of SciBERT in (c) and (d) of Fig.3.

We conjecture that when available data is not suﬃcient,changes in data distribution have a signiﬁcant impact on

While their pre-training data and model sizes are not in thesame order of magnitude at all, LLaMA-2-7b has only a

LLaMA-2-7b. As shown in Table 1, the use of review

slight advantage over BERT on P-BIO. This is strong ev-

papers (which lists a large number of polymers and corre-

idence that SciBERT’s strong performance on the P-BIO

sponding decomposition microorganisms) makes obvious

dataset is due to the domain of the pre-training dataset

discrepancy in the distributions between the training set

rather than its model structure. Despite the pre-training

and development set / test set. When part of data is used,

corpus containing the biomedical domain, which does not

random sampling may not select review articles, allowing

exactly overlap with the domain of P-BIO, the similarities

LLaMA-2-7b to achieve the same results as SciBERT within the textual characteristics of the two domains are suﬃ-

40 articles. Compared with SciBERT, LLaMA-2-7b is notcient to produce more desirable results. Choosing a similarrobust enough to distribution changes.

domain dataset based on text characteristics for unsuper-

It is agnostic whether LLaMA-2-7b will surpass SciB-

vised pre-training of PLMs may be a workable option if no

ERT after the training set exceeds more than 80 papers, as

big corpus is available from that domain.

it did on TMSP. The annotation dataset, however, cannot
be further expanded easily because there are few relevant

5Conclusion

studies in this domain, and fewer papers can be obtained.

We have conducted a study on the amount of data re-Annotating this type of information also requires a lot morequired to train a scientiﬁc sentence classiﬁer and the eﬀect

eﬀort from experts than generic text.

of classiﬁers with diﬀerent pretrained language models on

The weak performance of LLaMA-2-7b on P-BIO em-

two types of texts. For highly specialized texts, more train-phasizes the importance of speciﬁc domain pretraineding data is required, with 20 papers and in-domain pre-models for scientiﬁc text. In the range of 5 - 50 papers,

trained models needed to achieve results above 80% F1

SciBERT is just on par with the LLaMA-2-7b on TMSP.

score. For near-generic scientiﬁc texts, only 5 papers are

However, on P-BIO, beneﬁting from pretrained on biomed-

necessitated to exceed the qualifying line of 80% F1 score,ical domain corpus, SciBERT has reached an F1 score andand the large generalized language model performs wellaccuracy (91.71% F1, 98.80% Acc, 50 papers) that is muchwith more data. We encourage researchers to constructhigher than that on TMSP (Best: 80.79% F1, 98.88% Acc,

datasets and select base models for text classiﬁcation tasks

50 papers). SciBERT’s eﬀectiveness demonstrates the po-based on the text characteristics.

― 2894 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

6

Acknowledgement
We thank Chowdhury Mohammad Mahir Asef and Mut-sunori Uenuma for their contribution to the construction of
the Thermoelectric Material Synthesis Procedure (TMSP)
dataset.

References[1] Tatsuya Ishigaki, Yui Uehara, Shanshan Liu, Topic Goran,and Hiroya Takamura. Machine learning-based knowledge
acquisition from material science literature. 2021.
[2] The twitter ﬁnancial news dataset.
https://huggingface.co/datasets/zeroshot/
twitter-financial-news-sentiment.
[3] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang,
Christopher D. Manning, Andrew Ng, and Christopher
Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven
Bethard, editors, Proceedings of the 2013 Conferenceon Empirical Methods in Natural Language Processing, pp. 1631–1642, Seattle, Washington, USA, October

2013. Association for Computational Linguistics.
[4] Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: A pretrained language model for scientiﬁc text, 2019.
[5] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer,
Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia
Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn,
Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut
Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning
Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra,
Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan
Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams,
Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov,Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey
Edunov, and Thomas Scialom. Llama 2: Open foundationand ﬁne-tuned chat models, 2023.
[6] Zongxi Li, Xianming Li, Yuzhang Liu, Haoran Xie, Jing Li,
Fu lee Wang, Qing Li, and Xiaoqin Zhong. Label supervised
llama ﬁnetuning, 2023.
[7] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan AllenZhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.
Lora: Low-rank adaptation of large language models, 2021.

― 2895 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Table 3#Paper
5
10
20
30
40
50
60
80
100
120
140
150
160
180
200
250
300
350

Results on the TMSP dataset
Accuracy (%)
F1 score on the positive class (%)
LLaMA-2-7b SciBERT-base LLaMA-2-7b
SciBERT-base
98.07
98.40
70.53
70.38
98.54
98.56
75.83
73.88
98.59
98.70
76.84
77.72
98.83
98.75
80.04
79.01
98.92
98.75
81.59
78.47
98.79
98.88
79.68
80.79
98.78
98.92
79.93
81.18
98.87
98.88
80.99
80.90
98.91
98.78
81.06
80.99
98.94
98.85
81.91
81.87
98.95
98.78
82.48
80.96
98.95
98.93
82.26
82.19
98.82
98.92
80.25
81.48
99.01
98.88
83.65
81.24
98.93
98.89
82.28
82.14
99.01
98.93
84.35
81.83
99.03
98.96
84.32
83.32
98.98
82.78
99.04
83.66
Table 4

#Paper
5
10
20
30
40
50

Results on the P-BIO dataset
Accuracy (%)
LLaMA-2-7b SciBERT-base BERT-base
94.80
93.60
95.00
95.20
97.00
94.60
95.20
98.00
95.00
96.60
98.40
97.20
98.80

98.40

98.80
98.80

98.40

97.80

A

Appendix

A.1

Results on the TMSP dataset

F1 score on the positive class (%)
LLaMA-2-7b SciBERT-base BERT-base
56.67
20.00
46.81
50.00
73.68
34.15
47.83
84.85
44.44
70.18
88.57
77.42
91.43
91.43
88.57
87.88
91.67
84.51

Please refer to Table 3 for the experimental results on
the TMSP dataset.

A.2

Results on the P-BIO dataset

Please refer to Table 4 for the experimental results on
the TMSP dataset.

― 2896 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).