dataesr/scientific-paragraphs-categorization huggingface dataset:
A. Semi-supervised deep learning The literature on semi-supervised deep learning is rich and is growing rapidly, showing significant gains in performance in recent years. One of the early methods in this class of techniques used deep network generated predictions as pseudo-labels to retrain the network with unlabeled (unknown) data points  [7]. Another group of semi-supervised methods are referred to as "multi-view training" techniques, in which multiple models are trained to generate different representations of data. For instance, tri-training [8] consists of 3 different models trained on the same dataset using bootstrap sampling. After a supervised training of every model, an unlabeled data point is added to the training set of one model only if the other two models agree on its label. Another very competitive class of semi-supervised methods can be referred to as "self ensembling" methods.
These algorithms learn to exploit robustness to stochastic perturbations caused by noise or randomness in data augmentation or model design [9],  [10]. These methods use additional loss terms for consistency regularization. For instance, in this category ladder networks [9] achieved competitive results on MNIST and CIFAR-10 by trying to reconstruct the original signal of the lower layers using the signal of the last layer and a noisy output of the hidden layers. By using consistency regularization, the method developed in [10] pushes the boundary decision function to less dense areas of the decision space and enforces mutual exclusivity of classes. The Π model [11] uses a similar idea: part of its training loss consists of the mean squared error between predictions of the same input obtained with different noise or data augmentation.
In another work  [23]  forest oriented super pixels or super voxels were used to augment a random forest classifier for 2D or 3D biomedical image segmentation. Among other techniques that can help reduce the need for large amount of labeled data for automatic medical image segmentation, we refer to transfer learning [24]-[28] and active learning [29], [30] methods, and the recent survey in  [31]. Transfer learning consists of pretraining a neural network on a large labeled dataset from a source domain, before finetuning it on a small labeled dataset from the target domain  [25] ,  [27]. Active deep learning, on the other hand, aims to choose most informative unlabeled samples in an intelligent and selective manner for annotation, thus aims to minimize expert time needed for optimal labelling.
A. Semi-Supervised Learning with Deep Embedded Clustering In the classification task setting, a convolutional neural network (CNN) aims to produce predictions y i , i = 1,. , n from a labeled dataset ℒ = x i ∈ X 1 n. By learning this mapping, the algorithm reduces the dimension of the data space X to a much smaller space, with the help of max-pooling and fully convolutional layers. The CNN, therefore, summarizes useful information from the data as the dimension of each layer decreases, and, as a result, each of its layers can be interpreted as a feature embedding of the data. A clustering algorithm, on the other hand, aims at grouping an unlabeled dataset  = x i ∈ X 1 n into k clusters represented by their mean values μ j , j = 1,. , k. Compared to supervised CNNs, clustering algorithms have the advantage of being able to produce clusters directly from the data without needing any labels; but they are, by definition, uninformed of class labeling task, and are adversely affected by the curse of dimensionality.
These algorithms, therefore, perform better as the dimension of the data space X is reduced. The main idea in this work was to combine both of these approaches, in order to take advantage of the high capacity of a deep CNN and the ability of a robust clustering algorithm based on [6] to learn without labels, while circumventing the above-mentioned problems associated with the clustering algorithms. In this framework, the CNN learns a mapping between the data X and an embedding Z of smaller dimension, and the clustering algorithm aggregates the data points into different categories from this lower-dimensional embedding. Moreover, both of these tasks, namely learning the embedding and the clusters, are performed simultaneously. Our algorithm is designed as follows: we construct a CNN such that its last layer is a fully connected layer without a softmax activation, with number of units equal to the number of classes of the dataset.
Then, as in [6], we add on top of this last fully connected layer a clustering layer similar to the t-SNE algorithm  [34]. This layer uses a t-distribution kernel to measure the similarity between an embedding z i and the mean of a cluster distribution μ j : q i, j = 1 + ‖z i -μ j ‖ 2 /α -α + 1 2 ∑ j′ 1 + ‖z i -μ j′ ‖ 2 /α -α + 1 2 (1) In this equation, q i,j can be interpreted as the probability of a data point i to belong to a cluster j, and therefore as a prediction. To compare it with the last layer of a CNN, which is usually a dense layer, this layer learns means or centroids of different clusters representing each class from the embedding of the penultimate layer. On the other hand, a dense layer learns a mapping between an embedding and the predictions, and its parameters are, therefore, the weights of this mapping.
• The MNIST dataset, consisting of 70,000 hand written digits [5], • The SVHN, consisting of around 125,000 pictures of digits from street house numbers [37], • The iSeg, consisting of 3D T1-weighted (T1w) and T2-weighted (T2w) brain MRI scans of 23 six-month old infants as part of the iSeg2017 challenge  [1]. Brain tissue segmentation in iSeg is challenging as gray matter and white matter appear with isointensity values on both T1w and T2w MRI scans around six months of age. We used the half moons data for illustration, the MNIST and SVHN as standard image classification benchmarks, and the iSeg as a medical image segmentation task. In this section, first we present the network architectures for the classification and segmentation tasks, and then describe the details of the experimental setup and results for each experiment.
C. Half Moons In this section, we show a simple demonstration of SSLDEC in action applied to the halfmoons dataset. In this test, we drew 1000 points from a two half-moons distribution using the python package Sklearn  [36] , with a noise standard deviation of 0. 1. We randomly selected 4 points from each half-moon as labeled data, and considered the rest of the points as unlabeled data. In this section only, our CNN consisted of 4 dense layers of size 10 with RELU activation function, followed by one dense layer of size 2 and a clustering layer. Figure  4  shows the decision boundary of our model after training the algorithm only with the labeled data, and after SSLDEC training; which shows that SSLDEC was able to recover the underlying distribution of the two half moons after using the unlabeled data.
This simple experiment meant to display the advantage of the proposed semi-supervised method compared to supervised-only training with limited data. Separating the two half moons using a limited amount of points is considered a hard problem and the results depend on the number and position of the labeled points. In our experiments we observed that by using a total of 8 points (4 of each label) fairly distributed over the dataset, SSLDEC could accurately find the decision boundary and achieve a high accuracy.
D. MNIST The hand-written digits database,  MNIST [5] , has been widely used as a standard benchmark to compare semi-supervised learning algorithms. This dataset consists of 70,000 small images, among which 10,000 are used as the test set. Following the standard semisupervised learning experiments with this dataset, we randomly selected 100 (and 1000) images, 10 (and 100) of each class, of the training set as labeled data, and used the rest of the training data as unlabeled samples. For this classification task, we used moderate data augmentation through small translations of one pixel along the width and the height of each image. We found, experimentally, that using this augmentation to draw different predictions from the same input generated better results than computing the prediction of a noisy input.
Following standard evaluation criteria for semi-supervised learning, we randomly selected 10 different subsets, and reported mean accuracy score on the test set along with its standard deviation reported in brackets. Figure  5  presents our results as well as the results of several recent semi-supervised methods for labeled sets of size 100 and 1000. Our method generated competitive results on this dataset, particularly outperforming [16] using a labeled dataset of size 1000. Figures  6  and 7  illustrate sample results and the performance of our proposed semisupervised method (SSLDEC) on MNIST classification. Figure  6  displays the training set along with some misclassified examples, as well as the confusion matrix on the test set. Among the misclassified images, many can be stated as ambiguous for a human annotator.
SVHN SVHN has also been used as a standard benchmark for semi-supervised methods, where 500 and 1000 samples from the training set are used as labeled data and the rest of the training data are used as unlabeled samples. Figure  8  presents our results for the SVHN dataset. Similar to the MNIST dataset, we report the mean accuracy score of 10 different experiments as well as its standard deviation, reported in brackets. These results show that our method outperformed most of the other methods tested on the same dataset, with the exception of the VAT with 1000 labeled points. Note that on MNIST our method outperformed VAT by a large margin (Figure  5 ). Also note that while our method did not perform as well as "mutual exclusivity" [10] on MNIST, it outperformed mutual exclusivity by a margin on SVHN.
We used iSeg training and test data to evaluate our semi-supervised learning method (SSLDEC) for segmentation. To train our 3D densely connected CNN with SSLDEC, we randomly selected a subset of the training dataset in three scenarios: from the training set of 10 images, we only kept one fifth of 1, 2, and 5 images to draw labeled training samples. For semi-supervised learning we used the rest of the unlabeled training and test data to improve the performance of the CNN using SSLDEC, following the training algorithm described in Section III. We tested the trained model on the 5 cases from the iSeg training set that were not used as labeled data in training as well as on iSeg test data. We compared SSLDEC results against two supervised-only settings: one using the labeled training set used in each SSLDEC trained model as defined above (i.
On official iSeg test data, by using only one-fifth of the slices of each training image as labeled data and using the rest of the training and test images as unlabeled data, we obtained Dice scores of 93. 7%, 88. 8% and 86. 3% for CSF, GM, and WM, respectively, compared to 94. 8%, 90. 6% and 88. 5% for the fully-supervised model trained on all training data. Figure  12  shows sample axial and sagittal slices of T1w and T2w images as well as segmentations of a case from the test set (the 6th case in the iSeg training data that was used as a test case in our study), comparing our semi-supervised method (SSLDEC) using one fifth of 2 images as the labeled training data, with segmentation obtained from the pseudo-labelling method and the ground truth (manual segmentation provided by challenge organizers).
For CSF segmentation our method outperformed all other models including the higher bound model which used full labeled images instead of only using one-fifth of the labels of each training image. Results of (d) our model (SSLDEC) on a case from the isointense infant brain MRI segmentation challenge (iSeg), compared to (c) the ground truth (manual segmentation) and (e) a pseudo-labelling technique. Original T1-and T2-weighted MRI images of this case are also displayed in (a) and (b) which show the difficulty in distinguishing WM and GM due to their isointensity appearance at this age. These slices were extracted from image 6 on the iSeg dataset, after training models on one fifth of 2 images only (images 1 and 2 in the iSeg dataset). These results show that the pseudo-labelling technique includes several CSF zones, which are not present in the ground truth or in the results of our semi-supervised technique (black circles).
Results: To address these limitations, we have developed BioTransformer, a freely available software package for accurate, rapid, and comprehensive in silico metabolism prediction and compound identification. BioTransformer combines a machine learning approach with a knowledge-based approach to predict small molecule metabolism in human tissues (e. g. liver tissue), the human gut as well as the environment (soil and water microbiota), via its metabolism prediction tool. A comprehensive evaluation of BioTransformer showed that it was able to outperform two stateof-the-art commercially available tools (Meteor Nexus and ADMET Predictor), with precision and recall values up to 7 times better than those obtained for Meteor Nexus or ADMET Predictor on the same sets of pharmaceuticals, pesticides, phytochemicals or endobiotics under similar or identical constraints.
Most in silico metabolism prediction tools are quite specific to certain classes of reactions or metabolic processes, such as phase I (only) or phase II (only) reactions. Some in silico metabolism predictors, such as SMARTCyp  [23, 24]  and isoCYP  [25] , are limited to predicting phase I metabolism (or a portion of phase I metabolism), while others are more comprehensive (e. g. Meteor Nexus-Lhasa Limited, UK)  [26]  and SyGMa  [27]  cover a broad range of phase I and phase II biotransformations. Some programs are commercial such as Meteor Nexus, MetabolExpert (CompuDrug, Bal Harbor, FL, USA)  [28]  and ADMET Predictor (Simulation Plus, Lancaster, CA, USA)  [29] , while others are freely available either as web services (e. g. XenoSite  [30]  or as freely accessible standalone software packages (e.
In particular: (1) very few tools predict more than the SoMs; (2) none of the tools combine phase I, II, gut microbial metabolism, promiscuous enzymatic metabolism, and environmental microbial metabolism together; (3) many tools suffer from poor performance  [37] ; (4) almost all of the tools were developed and trained on drug molecules and were not adapted for non-drug xenobiotics; (5) only a small number of tools provide predicted structures in a downloadable or shareable format, and those that do place severe restrictions on their distribution;  (6)  almost none of the existing tools are open access or open source; and (7) very few of the tools make their databases or training sets available. These limitations have slowed the development of in silico metabolism prediction software and have also restricted the field to a tiny number of applications, mainly in the pharmaceutical industry.
Addressing these limitations and extending the capabilities of in silico metabolism prediction software could lead to substantial benefits in many other scientific disciplines including, but not limited to, analytical chemistry, natural product chemistry, agricultural and nutrition science, environmental chemistry, exposomics and metabolomics. Potential applications might include the in silico expansion of chemical databases of drugs (e. g. DrugBank  [38] ), food compounds (e. g. FooDB  [39] ), phytochemicals (e. g. PhytoHub  [40] ), environmental contaminants (e. g. ContaminantDB  [41] , T3DB  [42] , the CompTox Database  [43] ), organism-specific metabolites (e. g. HMDB  [2] , ECMDB  [44] , YMDB  [45] ), and other chemicals of biological interest (e. g. ChEBI  [46] , KEGG  [47] ). In fact, a notable effort carried by Jeffryes et al.
, has led to the development of the Metabolic In silico Network Expansion (MINEs) databases. The MINE databases contain close to 600,000 metabolites from compounds derived from KEGG  [47] , EcoCyc  [48] , and YMDB  [45]. The metabolites were generated computationally using reaction rules based on the Enzyme Commission classification system  [49] , and the Biochemical Network Integrated Computational Explorer (BNICE) algorithm  [50]. Jefrryes et al. reported that 93% of the computationally generated putative metabolites starting from KEGG compounds were not found in PubChem, the largest publicly accessible chemical database. Therefore, we anticipate that in silico expansions of the aforementioned databases using BioTransformer, will lead to the discovery of new exposure biomarkers, new bioactive metabolites, and consequently to the development of better drugs and consumer products (e.
It has been specifically designed to address essentially all of the shortcomings previously identified with existing in silico metabolism prediction tools. In particular, Bio-Transformer is freely available and furthermore its databases and predictions are free to download and use. It consists of two components: a metabolism prediction tool, and a metabolite identification tool. Bio-Transformer's metabolism prediction tool (BMPT) generates predicted metabolite structures in standard electronic formats, and it provides comprehensive metabolite predictions. BMPT covers a wide range of molecular classes. In particular, BMPT combines a knowledge (or rule)-based approach with a machine learning approach to predict (1) human CYP450calyzed phase I metabolism of xenobiotics, (2) human gut microbial metabolism, (3) phase II metabolism, (4) promiscuous enzymatic metabolism, and (5) environmental microbial metabolism of endogenous and exogenous compounds.
Structure and implementation of BioTransformer BioTransformer consists of a metabolism prediction tool (BMPT), and a metabolite identification tool (BMIT). The BMPT consists of five independent prediction modules called "transformers", namely: (1) the Enzyme Commission based (EC-based) transformer, (2) the CYP450 (phase I) transformer, (3) the phase II transformer, (4) the human gut microbial transformer, and (5) the environmental microbial transformer. For the prediction of metabolites, BioTransformer implements two approaches, a rule-based or knowledge-based approach, and a machine learning approach. BioTransformer's knowledge-based system consists of three major components: (1) a biotransformation database (called MetXBi-oDB) containing detailed annotations of experimentally confirmed metabolic reactions, (2) a reaction knowledgebase containing generic biotransformation rules, preference rules, and other constraints for metabolism prediction, and (3) a reasoning engine that implements both generic and transformer-specific algorithms for metabolite prediction and selection.
The BMPT machine learning system uses a set of random forest and ensemble prediction models for the prediction of CYP450 substrate selectivity, and for the Phase II filtering of molecules. BioTransformer's Metabolite Identification Tool builds on the BMPT to identify specific metabolites using mass spectrometry (MS) data, namely accurate mass or chemical formula information. In this section, we describe the structure, content, and implementation of MetXBioDB, the knowledgebase, the reasoning engine, the CYP450 metabolism and Phase II prediction systems, and the metabolite identification tool. Figure  3  gives a brief overview of each "transformer" module, their tasks, and the type of prediction approach they employ. Additional file 2: Figure  S1  illustrates the design workflow for the aforementioned BioTransformer components.
MetXBioDB: a database of metabolites and experimentally confirmed biotransformations and biodegradations MetXBioDB is a database that consists of a manually curated collection of > 2000 experimentally confirmed biotransformations derived from the literature. It was developed to help with: (1) the design of biotransformation rules, (2) the training and validation of machine learning metabolism prediction models, and (3) the design of preference rules. Each biotransformation in MetXBioDB includes a starting reactant (structure and identifiers), a reaction product (structure and identifiers), the name or type of the enzyme catalyzing the biotransformation, the type of reaction, and one or more citations. For the purposes of this paper, a reactant is defined as a small molecule that binds to a specific enzyme and undergoes a metabolic transformation catalyzed by that enzyme.
A biotransformation describes the chemical conversion or molecular transformation of a reactant to one or more products by a specific enzyme (or enzyme class) through a defined chemical reaction. Cytochrome P450 enzymes (CYP450s) are responsible for > 90% of phase I oxidative reactions and > 75% of drug metabolism  [58] , while UDP-glucuronosyltransferases (UGTs) and sulfotransferases (SULTs) are responsible for the phase II metabolism of most xenobiotics  [59, 60]  In the gut microbiota, enzymatic reactions are mostly reductive, and are carried out by anaerobic bacteria due to the very low concentration of oxygen. The "starting" reactants in the current version (version 1. 0) of MetXBioDB primarily consist of xenobiotics such as drugs, pesticides, toxins and phytochemicals. The database also includes a small number of sterol lipids and a selected set of mammalian primary metabolites.
In assembling MetXBioDB we gathered reaction data from the existing literature (> 100 references) along with data downloaded from publicly available databases such as DrugBank  [38] , PharmGKB  [61] , XMETDB  [62] , and SuperCYP  [63]. These databases list over 1000 enzyme-substrate associations for the major CY4P50s and UDP-glucuronosyltransferases (UGTs). Along with published scientific reports, Phe-nolExplorer  [64]  and PhytoHub  [40]  were also used to compile information about the metabolism of polyphenolic compounds in the gut. The data curation process consisted of three phases including: (1) the collection of biotransformation data, (2) the creation and annotation of biotransformation objects and, (3) data validation. This process was conducted collaboratively with a small team of chemistry experts.
Additional reaction information was gathered from resources such as the SIB Bioinformatics Resource Portal (ExPASy)  [68] , the BRENDA enzyme database  [69] , various Cyc databases  [70] , the UniProt knowledgebase (UniProtKB)  [71] , the KEGG database  [47] , and enzyme nomenclature information provided by the International Union of Biochemistry and Molecular Biology (IUBMB)  [49]. The collected data was used to: (1) design, test, and validate generic reaction/transformation rules, (2) add constraints and rules that would be used by the reasoning engine, and (3) map entities from different concepts. An example of the type of concept mapping done for the reaction knowledgebase is given here: phosphatidylcholines are a chemical class, the glycerophospholipid metabolism pathway is a metabolic pathway, a human is a biosystem, therefore phosphatidylcholines are mapped to the glycerophospholipid metabolism pathway in humans.
If the reaction passed all the tests, it was added to the database; if it failed, the reaction schema was subject to one or more iterations and tests until validated. Some of the encoded reactions in the reaction knowledgebase apply to a very specific set of chemicals, and can be used to accurately predict the metabolism of compounds belonging to those classes. Such examples include the aforementioned conversion of diacyl-snglycero-3-phosphoethanolamines to diacyl-sn-glycero-3-phosphoserines, and the metabolism of several classes of lipids, which are known to follow classic primary metabolic pathways. Other reactions are so generic or nonspecific that they would lead to a high number of false predictions if applied blindly. Some examples of highly non-specific reactions include aliphatic hydroxylation, N-dealkylation, and glucuronidation, among many others.