dataesr/scientific-paragraphs-categorization huggingface dataset:
For relatively large datasets like DBHuman, that are widely available today, the CN -feaMLP model is good default method that robustly outperforms or competes with reference shallow or deep learning methods, over the four considered settings. We believe that integrating other multiple description views of molecules and proteins within the same model, is an interesting direction for future developments. Importantly, the CN chemogenomic network provides a versatile architecture to incorporate such views. In line with this idea, we now explore transfer learning as an implicit data augmentation approach, and explore its interest to enhance the performance of the proposed chemogenomic neural network CN. Indeed, transfer learning combines different tasks related to the same types of data (proteins or molecules in our case), with the aim of providing additional information or representation power brought by different prediction tasks.
Evaluation of transfer learning for chemogenomics Principles of transfer learning in chemogenomics The principle of transfer learning is to gain knowledge by solving one problem and apply this knowledge to a different but related problem. Transfer learning can take two forms: pre-training, or co-training with other prediction tasks (even unsupervised tasks) for which large datasets are available. This may allow sharing of information and improve the performances of the chemogenomics task. Co-training refers to a strategy in which several prediction tasks are trained simultaneously, while sharing parameters of the neural networks, in order to learn a richer set of features. We did not consider co-training, because this approach suffers from lack of flexibility in training, as observed in the field of Natural Language Processing  [65].
In pre-training, also called "curriculum learning", a model is first trained on a task for which a large dataset is available, called the source task. Then, the pre-trained model is fine-tuned by re-training the weights on the task of interest, called the target task, for which a smaller training set is available. Therefore, when training on the target task, the initial parameters of the encoder are not randomly chosen, but are assigned to their optimised values according to the source task. Paul et al  [66]  explored transfer learning in a chemoinformatics problem related to organic solar cell screening, but this task only screened chemicals, not (protein,molecule) pairs, as in the present study. Fig. 9  illustrates the principle of curriculum learning, in the case of a DTI prediction task with the chemogenomic neural network CN.
First, the protein sequence and molecular graph encoders are trained with molecule-and protein-specific source tasks, corresponding to the grey path in Fig. 9. Then, the pre-trained encoders are re-trained, while training the MLP pair chemogenomic prediction task cor- responding to the black path in Fig. 9. Alternatively, it is possible to "freeze", i. e. not retrain, the pre-trained molecule and protein encoders, but only train the final MLP pair with the representations learnt from the pre-trained encoders as inputs. If the source and target datasets are related "enough", this can improve predictions on the target task. In the next sections, we explore two pre-training settings: one with a formally identical source task, and another with a different source task.
Transfer learning by pre-training with a formally identical source task The results presented in the previous sections showed that our CN chemogenomic network could outperform reference deep and shallow machine-learning methods (kronSVM or NRLMF ) on large datasets like DBHuman, but not on small datasets like DBEColi. Therefore, we chose DBEColi to explore whether transfer learning by pre-training on a larger and formally identical task, i. e. prediction on the DBHuman dataset, can improve the prediction performance of CN on small datasets like DBEColi to reach those of shallow methods. This setting is slightly different from that in Fig. 9 , because the source and target tasks are formally identical: the source task of DTIs prediction on DBHuman is used to pre-train the whole CN network (protein encoder, molecule encoder and MLP pair ), before re-training the whole CN on the smaller DBEColi dataset.
We varied the number of layers and neurons in the MLP pair network, in order to search for the best archi- tecture when freezing one or the two encoders. Indeed, a fully connected neural network like MLP pair may need a relatively large width and depth to leverage the representations extracted by pre-trained frozen encoders. We pre-trained the three parts of the chemogenomic neural network (the protein sequence encoder, the Fig. 9  Sketch of the chemogenomic neural network CN of Fig. 1 , modified for curriculum learning molecular graph encoder and the prediction MLP pair net- work) on the larger DBHuman dataset before re-training on the DBEColi dataset. Interestingly, this led to a loss of performance on S 1 , with 0. 2 in AUPR score instead of 0. 4. This resulted from the protein encoder pre-trained on DBHuman.
This is not the case of the S 2 (orphan proteins) which would have required pre-training of the protein encoder, and for the most difficult S 4 (double orphan) setting. In fact, although the source and target tasks are formally identical, DBEColi contains E. coli proteins and their ligands, whereas DBHuman contains human proteins and their ligands. Therefore, the two datasets may present statistical bias in protein sequences characteristics, since the distribution of protein sequences in bacteria is expected to differ from that of human proteins. However, both datasets contain more homogeneous small organic drug-like molecules. This may explain a loss of performance when pre-training with human proteins, while pre-training only the molecular graph encoder on the DBHuman dataset improves DTIs prediction in the relevant settings ( S 1 and S 3 ).
Transfer learning by pre-training with different source tasks To further investigate the interest of transfer learning with the CN chemogenomics deep learning algorithm, we now compare the prediction performance obtained on the small target task DBEColi, when the large source task is of different although related nature. We used the PCBA dataset as source task. It contains information about 90 bio-activities for hundreds of thousands of molecules (see "Materials and methods"). PCBA appeared relevant for pre-training the molecule encoder of the chemogenomic network, because the bio-activities of molecules result, at least in part, from their overall interaction profile with proteins in the cell. More precisely, the source task used to pre-train the molecule encoder consists in simultaneous predictions of molecules activities in 90 bioassays.
We evaluated pre-training of the molecular graph encoder on PCBA by 5-fold nested cross-validation with a test and train "positive:negative" ratio of 1:5, in the four S 1 , S 2 , S 3 and S 4 settings of the DBEColi dataset, and com- pared it to pre-training the molecular graph encoder on DBHuman. In the case of pre-traning with DBHuman, only the parameters of the molecule encoder are kept as initial values for re-training on DBEColi, since we showed that pre-trained parameters as initial values for the protein encoder decreased performances. Therefore, parameters of the protein encoder and of the MLP pair networks were randomly initialised. In the case of pre-training with PCBA, the source task is to predict the PubChem assay bio-activities. The pre-training step again only involves the molecule encoder and the MLP pair , since this data- set only contain bio-activities for molecules.
3. We also studied the performance of the methods on datasets of various sizes. Indeed, although large DTIs datasets are becoming available nowadays, learning on a small but well focused dataset with respect to the problem at hand might sometimes be a better strategy. Therefore, it is interesting to assess prediction performances of the chemogenomic methods both on large and small datasets. For large datasets like DBHuman (in the range of 10,000 interactions and above), we showed that the proposed deep learning chemogenomic neural network CN outperforms state-of-the art reference shallow machinelearning methods, particularly in the important S 2 , S 3 and S 4 orphan settings, and competes with the reference deep learning FNN method that uses expert-driven descriptors as inputs. In order to evaluate whether this result would generalise to other large datasets, we ran the CN deep learning method and the NRLMF shallow method on another large dataset built from the ChEMBL database (see "Materials and methods").
The results in the S 1 , S 2 , S 3 and S 4 settings, for a single train/validation/test split are displayed in Table  2. They show that the CN method outperforms the shallow NRLMF method in two of the three key orphan settings ( S 3 and S 4 ), while performing similarly in S 1 and S 2. Although we did not conduct a full study on this larger dataset (56,000 interactions) for computational reasons, the results in Table  2  show the same trends as those obtained on the DrugBank-based DBHuman dataset: deep learning methods should be used on large datasets (in the range of 10,000 interactions or more). In particular, our study highlights that, for large datasets, the FNN algorithm with expert-based protein and molecule descriptors appears as a simple deep learning approach that provides state-of-the-art performance in many settings.
It should be considered as a good default method, and as a reference method for future chemogenomic benchmark studies. In particular, we did not combine sophisticated expert-crafted descriptors, as proposed in other studies with shallow algorithms [68-71], which leaves space for improvement and reinforces the interest of this approach. However, the proposed chemogenomic network CN offers a versatile architecture for more sophisticated strategies, such as data augmentation approaches or transfer learning, in order to enhance the representation power of protein or molecule encodings with respect to the problem at hand. For smaller datasets like DBColi (less than a few thousands of interactions), we observed that NRLMF shallow method provides the best performance for DTIs prediction. It is a good default method for large scale DTI prediction, and provides a reference method for future chemogenomics benchmark studies on small datasets.
To further explore this point, we also considered the Random Forest algorithm with proteochemometric descriptors proposed in van Westen et al. [8] , because they displayed the best performances on small datasets in specific families of proteins (i. e. not in the chemogenomics setting considered in the present paper). We tested the descriptors recommended in this study (z-scales(3) together with protein fingerprints(FP8) as protein descriptors, and ECFP as molecule descriptors) on the DBColi dataset. The prediction performances of this method are shown in Table  3 , where we also recall those of the other methods tested in the present paper. Random Forests with proteochemometric descriptors did not reach the performance of the other shallow methods, but they overtook those of FNN and CN deep learning methods.
However, we are aware that the prediction performances depend on the dataset, and that methods may rank differently on family-focused datasets used in  [8] , and on more diverse datasets DBColi dataset. All methods were trained on drug-like molecules and drugabble proteins, because the aim was drug specificity prediction. However, all tested methods readily apply to prediction of interactions between other types of molecules or proteins, because only the protein sequences and molecule structures are required. However, the methods should be re-trained, because the good default hyper-parameters provided in the present study might not be optimal for highly different datasets (for example, not druglike molecules, or very different proteins such as plant proteins). This point is particularly critical for the CN and FNN deep learning methods, because they contain more hyper-parameters than shallow methods.
In addition, the results of the present study might not extrapolate to datasets highly different from those considered here, because methods might rank in different orders on very different datasets. An important contribution of the present paper is to explore data augmentation techniques in deep learning for chemogenomics. We investigated two directions for integration of different data sources: 1. Learning with multiple views of the data: combining learnt and expert-driven features in the CN network yielded to substantial improvements, and future efforts could be devoted to the design of neural net-  works that integrate multiple views of molecules and proteins within the same model. However, in the case of small datasets like DBEColi, these performance improvements did not allow to reach those of shallow machine-learning methods.
Availability of data and materials Regarding the implementation, our code is available on GitHub at: https :// githu b. com/bplay e/NNk_DTI. The datasets supporting the conclusions of this article are available in the data repository, http://membe rs. cbio. mines -paris tech. fr/~bplay e/NNk_DTI. zip. Also, we broadly used machine learning shallow algorithms and auxiliary functions (for instance for computing performance scores) implemented in the scikit-learn library  [61] , a viral and well-maintained free software machine learning library for Python. Also, we used the Tensorflow library [72] and the Keras library [73] to implement deep learning algorithms. To implement NRLMF, we used the PyDTI python package  [23]  available at: https ://githu b. com/steph enliu 0423/PyDTI. We used the implementation of LAkernel software [74] available at: http://membe rs.
e. , seizure class "ictal" or free seizure class "interictal") using machine learning techniques such as support vector machine (SVM), k-nearest neighbor (k-NN), and linear discriminant analysis (LDA). The experimental results were based on two EEG datasets generated from the CHB-MIT database with and without overlapping process. The results obtained have shown the effectiveness of the proposed method that allows achieving a higher classification accuracy rate up to 100% and also outperforms similar state-of-the-art methods.
Various epileptic seizure detection methods have been proposed in literature, and a brief review of some techniques related to our work was conducted and introduced in the next section. Gotman and Gloor were among the first to detect seizures using EEG signals  [4]. The first machine learning approach was developed using the CHB-MIT database  [5]. Some researchers have used a large database, extracted from Freiburg and CHB-MIT databases for seizure detection, and their proposed method allows achieving a high detection performance up to 100%  [6]. Some proposed methods are based on artificial neural networks allowing achieving a perfect classification rate for detecting epileptic seizures  [7]. Some are based on support vector machine (SVM) with time-frequency features and wavelet variance features allowing achieving correct classification rates up to 100%  [8].
The plethora of EMD, DWT, and DTWCT signal decomposition methods used in different fields of analysis justifies the choice in this work. This step is followed by the extraction of the statistical characteristics to extract the necessary information with reduction of used data dimensions. The aforementioned modulators were combined with three popular machine-learning techniques to find a powerful model for detecting epileptic seizures. Therefore, this work aims to develop an EEG signal classification system for seizure detection using CHB-MIT (EEG scalp) databases, and to ensure the robustness of the proposed model in which two databases have been extracted and realized based on EEG segments with and without overlapping procedure. The results proved that the proposed model is suitable for epileptic seizure detection.
EEG database All tests and simulations were conducted based on the database collected at the Boston Children's Hospital (CHB) and used to validate the experimental results. This "CHB-MIT" database includes 23 different subsets containing EEG records of pediatric patients. It contains 182 seizures. All signals were sampled at 256 Hz per second with a resolution of 16 bits. Most files contain 23 EEG signals. The international 10-20 EEG electrodeposition and nomenclature system was used for these recordings  [5].
Database preparation The CHB-MIT database is one of the EEG databases used in recent research for epileptic seizure detection. To prove the effectiveness of the proposed method, two EEG datasets have been extracted from the CHB-MIT database. Hence, the extraction of shorter EEG segments based on no overlapping process represents the first used database, while the second based on overlapping process. The segment extraction process of these two datasets is clearly explained in this section. The performance evaluation criteria to assess the performance and the effectiveness of the proposed method and the experimental results obtained will be shown as follows. The developed seizure detection model is based on EEG segment extraction with an 8-second rectangular window. The seizure time is known and given in the description database for each patient.
The first extracted dataset contains 400 EEG segments extracted approximately from all the recorded signals in the CHB-MIT database with 200 no seizure segments ("interictal") and 200 seizure segments ("ictal"). According to a previous study  [6]  , the choice of the 8 seconds window size (with 2 048 samples) provides a more robust performance than that of shorter or longer windows. To prove the effectiveness of the proposed seizure detection model, the overlapped database has been created with an overlapping process based on the first extracted database that will be presented as follows.
Overlapping procedure While the first database was created (with 400 segments), to evaluate the performance and better prove the effectiveness of the proposed seizure detection method, another database will be created based on the first realized with an overlapping process. Hence, new segments will be created by overlapping between two segments belonging to the same EEG signal in the same class. Let suppose that: -N is the segment number of the first database already created (without overlapping) called B. N1 and N2 is the number of segments in "interictal" class B1 and "ictal" class B2 respectively, -Nc is the segment number of the second database with overlapping process called Bc. Nc1 and Nc2 are the numbers of segments in "interictal" class Bc1 and "ictal" class Bc2 respectively. Therefore, the overlap process is presented as follows: -n represents the segment samples number (2 048), i is segment index (i th segment), p is the point offset for overlap, fe is the sampling frequency 256 Hz, and sc is the segment seconds number (8 seconds).
X1 i , X2 i , X3 i , X4 i , X5 i , X6 i , X7 i , X8 i X1 i+1 , X2 i+1 , X3 i+1 , X4 i+1 , X5 i+1 , X6 i+1 , X7 i+1 , X8 i+1 i i + 1, j p = 6 Example: let suppose i and i+1 are two segments of the same class to be overlapped where and , are represent the sample of segments of and respectively, and the new obtained segment with a gap between the segment is. i = [X1 i , X2 i , X3 i , X4 i , X5 i , X6 i , X7 i , X8 i ] i+1 = [X1 i+1 , X2 i+1 , X3 i+1 , i+1 , X5 i+1 , X6 i+1 , X7 i+1 , X8 i+1 ] j = [X3 i , X4 i , X5 i , X6 i , X7 i , X8 i , X1 i+1 , X2 i+1 ] Finally, the new segment can be presented by the following expression: j = { i [ n sc × (sc -p) : n × sc ] ; i + 1 [ 1 : n sc × (sc -p) ]} (10) j = { i [ f e × (sc -p) : n × sc ] ; i + 1 [ 1 : f e × (sc -p) ]} (11) The size of new database (overlapped) will be increased compared to the first database.
N = N1 + N2 Nc = Nc1 + Nc2 Nc1 = (N1 × 2) -1 Nc2 =(N2 × 2) -1 and ; hence, and. Nc = [(N1 × 2) -1] + ((N2 × 2) -1) = 2N1 + 2N2 2 (12) Therefore, the overlapped database size will be 798 segments, with 399 segments in both classes and each length segment 8 seconds. The experimental results of the proposed approach using the two extracted databases will be presented by following the performance criteria to validate the model accuracy.
Feature selection process The sampling rates of the databases are 256 Hz; the polynomial order was chosen to be 2 on S-Golay filter and the frame length was selected to be 17, proved to be the parameter' choice  [10]. In MSPCA, a five-level decomposition was used to generate the sub-bands signals. The Kaiser rule was responsible for eliminating the number of loadings  [26]. After the artifact removal procedure with S-Golay and MSPCA, the features have been extracted from the sub-band decomposed signals using EMD, DWT, and DTCWT methods. The Daubechies wavelets of different orders (2, 3, 4, 5, and 6) are investigated for the analysis of epileptic EEGs. This family of wavelets has orthogonal property and is efficient as a filter implementation  [38]. The fourth order Daubechies wavelet is found to be most appropriate for analysis of EEG data  [6, 39].
Performance evaluation criteria To prove the robustness of the proposed method, cross-validation (CV) is a procedure mainly applied to estimate the proficiency of machine learning algorithm. CV is used to estimate how the model should work when used to make predictions of unused data at the step of preparation and model learning. In a 10 times CV, the database is randomly separated into 10 different mutually exclusive folds of the same size, where nine folds will be used to training the data and the remaining one will be used to test and predict the classification. The CV accuracy is expressed by: CV = 1 k k ∑ i=1 A i (13) A i i = 1,. , k. where represents the number of folds used, and is the accuracy measure of each, In addition, there are three performance measures to evaluate machine learning methods using true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN), which can be expressed by: Sensitivity (%) = T P T P + FN × 100 (14) Speci f icity (%) = T N T N + FP × 100 (15) Accuracy (%) = T P + T N T P + FN + T N + FP × 100 (16).
Results The previous sections have shown that the first goal in this work is to find a model to classify EEG signal segments to develop automatic seizure detection systems using the children's EEG database CHB-MIT. However, shorter EEG segments with 8 seconds have been extracted from this database that makes it possible to develop a practical and robust seizure detection system to diagnose epilepsy. Ictal record signals represent a small amount of data compared to interictal record signals. Therefore, a subsampling paradigm of the class with larger proportions (interictal) is used to circumvent the imbalance between the two classes  [40]  , in order to have better accuracy of seizure detection. In this work, segments of interictal data were randomly selected to produce a comparable and balanced number of ictal and interictal segments.
The obtained results using the overlapped database ensure the reliability and stability of this seizure detection model as illustrated in Table  2  and Fig. 6. The combination of DTCWT and MSPCA offers advantages over the DWT and EMD and keeps the same 100% accuracy performance. The important difference between EMD and wavelet-based methods lies in the process of feature extraction. IMF signals are the product of EMD methods and have the same length as the original EEG signals, which have given no reduction in the number of samples. On the other hand, the coefficients characterizing different subbands produced by wavelet-based methods (DWT and DTCWT) have a much smaller number of samples than the EEG signals input. However, the DWT suffers from certain disadvantages such as the oscillatory nature, the aliasing of the spectrum with limited directional information, and the lack of lag invariance.