dataesr/scientific-paragraphs-categorization huggingface dataset:
Dataset description Our study focused on the COVID-19 Open Research Dataset 1  [40]  which contains scientific papers on COVID-19 and related historical coronavirus research. Throughout this study, we refer to the dataset as COVID19-MOOD-data. The COVID19-MOOD-data dataset is divided into two main corpora, respectively named Papers1 and Papers2. Papers1 contains the commercial use subset (includes PubMed Central content), while Papers2 contains the commercial use subset (includes PubMed Central content), the non-commercial use subset (includes PubMed Central content) and the custom license subset. Three data pre-processing operations are performed per corpus (Papers1, Papers2) in order to create three corpora according to the title, abstract and content: -Title represents the corpus that contains only paper titles; -Abstract represents the corpus that contains only paper abstracts; -Content represents the corpus that contains only paper contents.
The free term extraction approach The free term extraction approach seeks to ensure that users will be able to extract significant terms related to a specific domain from a given corpus. As we mentioned in Section Related work, existing tools have been proposed for term and concept extraction. We opted for the BioTex tool to support the free term extraction mode for several reasons: -BioTex was initially built for medical domain term extraction. -BioTex uses hybrid measures (linguistic and several statistical measures) for the term extraction process. -Most existing tools (e. g. Maui-indexer 2 , Topia Termextract 3 , KEA 4 , etc. ) are designed for keyword extraction within single documents, and they only function for English language documents, while BioTex is tailored for terminology extraction and supports sets of documents (corpora) and multi-language use.
Proposed combination for term extraction Based on the elements given in Sects. The free term extraction approach and The driven term extraction approach, we propose a workflow in Fig. 2  for term extraction and analysis dedicated to scientific papers. We outline this workflow according to the type of corpus, measure, and approach: -The type of corpus: as described in the data section, for a given paper, we considered three parts to build the corresponding corpora, i. e. the Title (T), Abstract (A) and Content (C); -The measures: BioTex integrates several statistical measures, each of which uses a specific strategy to compute the term score. In this case, we selected the two measures C_Value and F-TFIDF-C_M. C_Value indicates the importance of terms that appear most frequently in a document, based on the idea that the frequency of appearance of a term in the document reflects its importance in the document.
Each corpus was partitioned into smaller units to enhance scalability. The results obtained from the smaller units were then composed by computing the average ranked values. The final rank for a given term was thus equal to the average of its ranked values in all sub-corpora in which it was present. The final result gave a set of terms, listed in ascending order according to the ranking values. Table  3  shows an example of the MWT set obtained using BioTex. The Terms column contains the extracted terms, the in_umls column indicates if the corresponding term is available in the Unified Medical Language System (UMLS) Metathesaurus  [3]  or not, and rank shows the significance of the term based on statistical measures in the whole list of terms for a given corpus. In our study, we used the UMLS Metathesaurus as reference for the extracted terms as our study is linked to a biomedical terminology analysis.
Corpus representative terms In this section, we illustrate how representative terms can be extracted from different datasets. Based on the BioTex ranking measures, a term is more important than another one in a given corpus if it has a higher ranking than the other term. Figure  3  shows representative terms for the Title, Abstract and Content corpora with the corresponding statistical measures (see Tables  7  and 8  for more details). This figure highlights which terms are important in each part of the Papers. Note that the extracted terms are different for each measure and sub-corpus, but some of them are similar for both. For example, terms like public health, immune responses are extracted using both measures from the Abstract corpus. In order to quantitatively display the number of representative intersecting terms from different corpora, we show common terms between Title vs Abstract, and Title vs Content corpora for the Papers2 corpus in Fig.
Relevant term extraction from corpora for both measures This involves quantitative and qualitative analysis of the terms extracted within each corpus, while taking both measures (C_Value and F-TFIDF-C_M) into account. In other words, it consists of analysing terms obtained for both measures, i. e. terms detected at the same time, and also terms specific to each of them. The quantitative analysis aims to highlight, for each dataset, the number of terms obtained by each measure, the number of terms obtained for both measures, and which are available or not in the UMLS Metathesaurus. While the qualitative measure aims to highlight, in each case, how the terms obtained are important or not regarding the study domain. For the data representation, we take advantages of Venn Diagram  [16] , see in Appendix Fig.
Discriminant and common term extraction from corpora In this case, term analysis is performed per dataset or by jointly considering multiple corpora, i. e. between Title, Abstract and Content corpora. Appendix Fig. 11  corresponds to discriminant and common term extraction from Papers1-title, Papers1-abstract and Papers1-content. There are common terms in the overall corpus such as gene expression, virus replication, influenza virus, etc. These terms tend to be relevant in the Title, Content and Abstract corpora. Moreover, [respiratory infection, acute respiratory infection, etc. ], [innate immune response, endoplasmic reticulum, etc. ], and [nucleotide sequences, room temperature, etc. ] are discriminant terms in the Title, Abstract and Content corpora.
Conclusion In this paper we describe ITEXT-BIO, a generic methodology for biomedical term extraction. We show how it allows users to extract terms (or concepts) from different types of textual data using several combined strategies. The free term extraction approach extracts terms from corpora, while the driven term extraction approach extracts, from a corpus and a set of terms, a set of variations of these terms. We illustrate that the proposed combined strategies based on statistical measures and textual segments help efficiently extract and categorize terms (representative, discriminant and new terms) from a corpus or corpora. We also quantitatively and qualitatively analysed the extracted terms to determine those related to the study domain and those that could be considered as emerging terminology for disease monitoring.
Acknowledgements This study was partially funded by  EU  grant  874850 MOOD  and is catalogued as  MOOD 003. The contents of this publication are the sole responsibility of the authors and do not necessarily reflect the views of the  European Commission. This study was partially funded by the  French Agricultural Research Centre for International Development (CIRAD)  the  French General Directorate for Food (DGAL) , and the  SONGES Project (FEDER and Occitanie Region). The research was also supported by the  French National Research Agency (ANR)  under the Investments for the Future Program, referred to as  ANR-16-CONV-0004 ,  #DigitAg.
9933 Syncytial virus 1. 9752 Acute respiratory syndrome 1. 9961 Incubation period 1. 99325 t cell 1. 9746 95percent ci 1. 996 Health emergency 1. 9932 Infectious bronchitis 1. 9726 Ebola virus 1. 9945 Index patient 1. 9932 Sars coronavirus 1. 9723 Influenza viruses 1. 9943 Membrane rafts 1. 9931 BMC public health 1. 9701 Avian influenza 1. 9939 pcr products 1. 9929 t cells 1. 9689 Respiratory tract infections 1. 9938 2c atpase 1. 9926 Acute respiratory infection 1. 9672 Health care 1. 9925 b cell 1. 9924 Mini review 1. 9636 Hepatitis c 1. 9922 Close contact 1. 9924 Respiratory viral infections 1. 9636 Type I 1. 9918 Final dataset 1. 9922 BMC public 1. 9625 Cell line 1. 9914 3d8 scfv 1. 9921 Ebola virus disease 1. 9592 Spike protein 1. 9909 Pol ii 1. 992 Supplementary information 1. 9574 Codon usage 1.
Using these two ML algorithms we evaluate the prediction performance on the target properties, which are band gap, formation energy, and elasticity of crystalline compounds in the database of Materials Project (MP). Moreover, to improve prediction of the properties of the materials without human bias in the selection of initial atomic features for the CGCNNs, we use Atom2Vec that provides atom representation obtained in an unsupervised manner from the materials. We also perform the predictions with the different sizes of training set to investigate the data-size dependency of the predictive models. According to the amount of dataset, the use of structural information, and the ability to identify the best descriptor with its interpretability, these algorithms showed different prediction performances.
Despite growing interest in the ML methods mentioned above, the amount of data that can be directed used for the prediction is usually not enough since most of the related studies of material properties have quite a narrow scope. Therefore, only a very little portion of the published data has matched with the given specific scope. Recently, a new approach, called Sure Independent Screening Sparsifying Operator (SISSO)  (Ouyang et al 2018) , showed stable prediction performances in a relatively small dataset by identifying the intrinsic relationship which is immutable between a target property and physical quantities. The relationship among physical quantities can be expressed in a mathematical equation that is composed of several descriptors. It provides a model that can be used to predict unknown properties of materials of interest like ML has performed for the same reason, but analytic function can be obtained by SISSO, while a role among input features is completely hidden in a generated ML model.
As we mentioned above, such ML models achieved a remarkable improvement in the prediction of material properties. However, for material engineers who want to apply ML to their research but are not familiar with it, exploiting the ML models is still challenging because its performance highly depends on the amount of data to be prepared/preprocessed for predictive models, and representation of atoms in compounds for their features, and appropriate targets, which are the important material properties in its design. In this study, we investigate the difference in model performance according to the choice of atomic features-selected by empirical knowledge or by the unsupervised way (Atom2vec)-as initial input values to the CGCNN model. Assuming that the amount of data for learning a model is limited, we also examine whether SISSO could show promising prediction performance.
Atom2Vec Atom2Vec introduced in (Zhou et al 2018) is unsupervised learning of atoms from a database and provides atomic descriptors that capture well the similarities and properties of atoms in a vector space and show their enhanced effectiveness over simple empirical descriptors in ML problems for materials discovery. Atom2Vec lets machines learn atom representation from only the existence of compounds (or environments) in a materials database. To represent how atoms are bond together to form their environments, atom-environment pairs are generated from the chemical composition of a compound: Each atom is selected as a target, and the counts of all remaining atoms are represented as a corresponding environment to the target one, and of itself. For example, Bi Te 2 3generates two atom-environments pairs: Bi Te " " " 2 3", -( ) and Te Bi " " " 3 2".
-( ) Then, from a materials database, we build atom-environment matrix X whose i j , ( )entry represents the number of pairs where the ith atom and the jth environment appear together. The authors in  (Zhou et al 2018)  proposed two types of learning algorithms for Atom2Vec, one is model-free and the other model-based machines. Nevertheless, to exclude any probability model to describe connections between atom and environment, we here only consider a model-free machine. In the model-free machines, the normalization of the atom-environment matrix is applied to its row vector to overcome imbalanced atom distribution in the environment. Then, using singular-value decomposition (SVD)  (Sapper and Hinderliter 2013) , we can obtain row vectors in the subspace of the d largest singular values which determines the dimension of encoded feature vectors for atoms and is fixed as = d 20 in this work (figure  2 ).
SISSO SISSO is an algorithm that can be used to discover the intrinsic relationship which is immutable between a target property and physical quantities based on the compressed-sensing method  (Ouyang et al 2018). Since its performance is not dependent on the size of an input dataset, it has been known to provide stable results even though the training set is relatively small. In this paper, SISSO has been employed to obtain the mathematical equations composed of atomic features and to predict unknown materials' properties. We have used thirty-six kinds of physical quantities of each atom as the input features; atomic radius, atomic volume, atomic weight, and so on (see all in figure S1 in SI (available online at stacks. iop. org/MRX/8/026302/mmedia)). To find the relationship between such atomic features and material properties, SISSO investigates their combinations which means combining more than one atomic features by the given mathematical operator set which is defined as f f º + -´¸- - H , , , ,exp, log , , , 1 , 2 , 3 _1, _2 , m { | | ˆ( ) ˆˆ}[ ] ( ) where f 1 and f 2 are objects in Φ and, the superscript (m) denotes that physically meaningful operations are only allowed.
Dataset The Material Projects (MP) database, including a variety of computed properties such as crystal structure, electronic band structure, and energy, was used for training and validation of the CGCNN models: inorganic crystalline compounds  (Jain et al 2018). The MP database (pymatgen=2019. 3. 27) has 124,515 inorganic compounds with 89 elements and 227 space groups, and almost 9 out of 10 compounds are binary, ternary, and quarternary. We focused on predicting four basic properties of crystalline materials: formation energy (E f ), band gap (E g ), bulk modulus (K VRH ), and shear modulus (G VRH ). Before training the predictive models for the target properties, we prescreened the data obtained from poorly converged calculation and multivalent cathode projects in MP. Note that only binary and ternary compounds were considered in this study because the feature space in SISSO rapidly grows with the elements in a compound: The total number of binary and ternary compounds was 000 for formation energy, 000 for band gap, and 000 for bulk and shear moduli (figure  2 ).
To investigate the data-size dependency of the predictive models, a thousand of crystalline materials was randomly set apart from the whole dataset. Each model was trained with 80% of the data and then tested with 20% of the data. Note that Atom2Vec can learn the representation of atoms from a massive database, and the learned vectors can be used as general descriptors in ML problems with a different database in materials science. This fact enables us to independently learn atom representations from the whole MP database and to use the learned vectors in the CGCNN model.
CGCNN with the initial atom embeddings We compared atom vectors from Atom2Vec with the atom representation from the empirical and random features and investigated which atom vector is more effective in use for supervised learning tasks of formation energy, band gap, and elasticity predictions. For comparison, we fixed the architecture of the CGCNN models in all the dataset: three convolution layers with atom representation of length sixty-four, one fully-connected layer with one hundred twenty-eight hidden units after the pool layers. For training, Adam optimizer (Kingma and Ba 2015) with a learning rate of 0. 001 was used. Then, we fed the three different types of atom representations in the CGCNN model and examined the prediction performance for the targets, formation energy, band gap, and elasticity in inorganic compounds.
Model performances in the predictions utilizing the two atom representations are summarized in figure 3, tables 1 and 2 for mean absolute error (MAE), and R 2 score, respectively. Overall, the two different atom representations have similar R 2 scores in all target properties; the difference of the averaged R 2 scores in the atomic featurizations are less than 0. 2. The initial atom features generated by empirical featurization led to the better performance of band gap and formation energy prediction than the initial atom features from Atom2vec for both the small and whole dataset, except band gap prediction for the whole dataset. In particular, CGCNN with empirical atom representation to predict formation energy outperformed those with Atom2vec atom representation for the ternary compounds in the small dataset.
Meanwhile, CGCNN with Atom2vec achieved slightly better performances of elasticity prediction for the binary and ternary compounds in the small and whole dataset. In particular, R 2 scores of band gap prediction almost doubled for both the binary and ternary compounds. Note that the length of the initial atom vector by the unsupervised learning of atom in Atom2vec is much less than those of the empirical one. Nevertheless, the atom representation from the Atom2vec showed comparable or better performances to the empirical one, independent of the size of the database, the type of  compound, and the task. This result is line with which Zhou et al also showed that the effectiveness of Atom2vec in use for a supervised learning task for formation energy prediction of elpasolite crystals ABC 2 D 6.
Atom representation in SISSO We have employed the SISSO algorithm to predict the properties listed above by using the same dataset as that of ML. (atomic features from Mendeleev, Rung=2, 3D descriptors) (Supplementary Info. : about input atomic features, and details of running SISSO, version: SISSO 1. 0) The MAE outcomes from SISSO results are systematically larger than that of ML due to two reasons. It seems to be mostly related to the lack of structural information of the materials in the input features. Another reason is the MAEs have not been fully minimized since high accuracy setting has not been considered. Therefore, the direct comparison of the performances between SISSO and ML methods presented below (table 3) could not be fair. On the other hand, we could discuss the comparison of two different results by only SISSO: the difference due to the change of the dataset size.
For the comparison, we introduced MAE gain that means the increment of the MAEs in percentage when changing the input dataset to the small one from the whole one, and it is given by = - ḾAE MAE MAE MAE % 100. gain small whole whole ( ) In the comparison, ML prediction results show that MAE is changed dramatically depending on the size of the dataset. On the contrary, SISSO results systematically maintain the similar accuracies while the size of the dataset is reduced; especially the MAE of E f of binary and G VRH of ternary is increased slightly, even though quite a smaller data subset has been used. In the case of E g of ternary, the MAE is decreased at the smaller dataset, surprisingly. The similar trend found in the R 2 scores presented in table 4: R loss 2 of ternary compounds shows that the model performance between small and whole dataset is not too large.
However, in the case of the binary compounds in the small dataset, R 2 scores are seriously increased and its behavior is similar to those of the ML results. It seems that the intrinsic relationship between atomic features and target values has been lost in the test set of binary compounds, while the importance of the crystal geometry on the ML methods are strongly dependent on the change of the dataset size. Next, we consider only ten atomic features, while a total of thirty-six atomic features is available from the results of the additional feature engineering process described in SI. Its prediction results are presented in tables 3 and 4 for MAEs and R 2 scores, respectively. To investigate such larger feature space, additional steps have been performed for the formation energy of the binary compounds of the small dataset by using such 36 atomic features as presented in figure 4.
It allows more combinations of input features and then the possibility of highend performance of SISSO in the prediction of the target properties has been checked. Finally, the obtained R 2 score with 37 features for the formation energy is 0. 846, which is comparable to that of the state-of-the-art approach based on crystal graphs as a structural representation such as CGCNN when the data set of the restricted size (small one) was used. The advantage of the SISSO combined with the Mendeleev Python library is its high availability that does not require any pre-defined structural information for the target compounds. This allows us to skip the long time process of DFT calculations. As a result, it could be used as an alternative of CGCNN methods when the given dataset has deficient information about atomic topologies and has only compositional information of the target compounds.
Conclusions In this study, we introduced the two state-of-art ML frameworks, CGCNN and SISSO, to predict crystalline materials' properties such as band gap, formation energy, and bulk and shear moduli. CGCNN outperformed SISSO regardless of the size and type of the dataset, and its prediction tasks as other DL approaches have been done for the case in the field of materials science, recently  (Gilmer et al 2017 , Sutton et al 2018). CGCNN is also able to have the interpretability of an embedding vector in a specified structure such as perovskites. However, CGCNN needs some domain knowledge to avoid the human bias of atomic characteristics for vector embedding. Meanwhile, SISSO showed consistent results in prediction performance even in the small dataset. Also, SISSO could provide the best descriptor out of a large space of mathematical combinations of simple atomic features, and identify the relationship between atomic descriptors and property in terms of an analytical equation  (Ouyang et al 2018).
To avoid human bias on feature selection for the initial atomic descriptor, we adopted Atom2vec. The atom representation from Atom2vec showed comparable or better performances to the empirical one, regardless of the size of the database, the type of compound, and the task. This showed the effectiveness in use for the supervised learning task of materials properties over empirical atomic features. Therefore, Atom2vec for the initial atom embedding in graph-based neural networks would be a better choice for researchers who want to apply it to the general task for materials discovery than the empirical one. Although ML for materials design and discovery has been improving, it is still challenging to develop a model with high-throughput computation and precise prediction for most properties of crystalline compounds; researchers looking to introduce an ML method for this field are forced to spend a lot of time and effort in finding the appropriate model and descriptors according to the given dataset and the target properties.