dataesr/scientific-paragraphs-categorization huggingface dataset:
Introduction The current paradigm in rationalised drug design is to associate a disease to one or several molecular targets, usually proteins called target proteins, or therapeutic targets, involved in biological pathways necessary to the disease devel-In addition, we also test data augmentation techniques, namely multi-view learning and transfer learning, to enhance representation learning for chemogenomics, which is a new contribution in this field. In particular, we explore the simultaneous use of expert-based and learnt features in a differentiable fashion, with the idea that combining both types of features might increase performances. We also explore transfer learning via curriculum learning. This refers to using an additional larger dataset for pre-training the neuron network with an auxiliary task, before using it in the chemogenomic task of interest, with the aim of learning richer representations of molecules or proteins.
We tested various ways to implement this idea. Section 7 finally presents a short discussion and conclusion about the proposed methods for chemogenomics with deep learning. Regarding the implementation, our code is available on GitHub at: https : //github. com/bplaye/N N k_chemo. The code and data are available at: http : //members. cbio. mines -paristech. f r/ ∼ bplaye/N N k_chemo. zip. Also, we broadly used machine learning shallow algorithms and auxiliary functions (for instance for computing performance scores) implemented in the scikit-learn library  [8] , a viral and well-maintained free software machine learning library for Python. Also, we used the Tensorflow library  [9]  and the Keras library  [10]  to implement deep learning algorithms. To implement NRLMF, we used the PyDTI python package  [11]  available at: https : //github.
com/stephenliu0423/P yDT I. We used the implementation of LAkernel software  [12]  available at: http://members. cbio. mines-paristech. fr/ jvert/software/. 2 Pioneering works in machine-learning for chemogenomics Various chemogenomics methods have been proposed in the last decade  [13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 11]. They differ by (i) the descriptors used to encode proteins and ligands or the similarities between these objects, (ii) the ML algorithm that is used to learn the model and make the predictions. We shortly describe the principle of those the methods displaying state-of-the-art performances. Most of them are similarity-based approaches, meaning that they consider as input data the similarity between instances. They rely on the assumption that similar compounds (resp.
Note that the NRLMF approach is also generalised to orphan molecules and proteins, by computing latent representations of orphan molecules and proteins as a weighted sum of the latent representation of their neighbours. Based on the results reported in the original paper  [11] , we observe that the proposed method NRLMF outperforms the other state-of-the-art methods  [23, 22, 26, 24]  on the Yamanishi benchmark dataset  [13]. Therefore, this method is used in this study as reference method to compare the proposed deep learning approaches for chemogenomics.
Many other AGGREGAT E (l) node functions have also been proposed in the literature, but they have not been tested in the context of chemoinformatics or chemogenomics. Typically, some studies have considered to sum over edge type-specific standard convolutional layers  [53, 54, 55] , or even also proposed to update edges attributes similarly to node attributes, hoping for more expressive GNN  [56, 57, 53]. We tested this principal, i. e. taking into account bond attributes in the molecule graph, as shown in the Results section. Velickovic et al. [58]  proposed an attention mechanism to design a neighbourhood aggregation function (instead of a node aggregation function) for graph representation learning. Their GAT AGGREGAT E (l) node function resulted in performance improvements higher than those reported in other studies, which usually show very little performance increase (one or two points maximum) on graph prediction gold standard datasets comparatively to minimal GNN formulation.
In addition, this method seems particularly powerful since it aggregates neighbouring information after identifying the most suitable task-specific neighbours at each node. However, we could wonder whether the GAT AGGREGAT E (l) node would be interesting in the case of small graphs such as molecular graphs, where all atoms are supposedly relevant. Recently, Shanthamallu et al. [59]  quantitatively assessed that the distribution of attention weights for any node obtained by GAT is almost always uniform, in the case of gold standard datasets for graph representation learning. This means that the attention mechanism is not suitable in these cases, since its selects all neighbours as relevant. Therefore, we discuss and evaluate the GAT AGGREGAT E (l) node function in more details in the Results section.
Protein sequence encoder Before encoding a protein, i. e. a sequence of amino acids, with neuron network-based encoders, we need to define attributes for amino acids. An obvious solution is to use the so-called "one-hot" encoding, where amino acids of the protein sequence are encoded by a binary vector of the length 20 (one bit for each natural amino acid), containing a single one and 19 zeros. However, richer encodings are available. For instance, an amino acid can be encoded by a mutation probability profile. This position-dependent probability profile corresponds to the probability of the current amino acid to be mutated in other amino acids. It is usually calculated by running sequence alignment algorithm like PSI-BLAST  [61]  against a reference database. Additional structural or physical amino acid properties can be used to complete amino acid attribute vectors.
Deep learning for chemoinformatics Deep learning in chemoinformatics refers to prediction problems involving molecules (such as predicting solubility or toxicity), but not to the prediction of protein-ligand interactions as in chemogenomics. Therefore, chemoinformatics deep learning only involves two of the four blocks presented in Fig. 1 : the molecule encoder and the prediction block which consists in an M LP pair (i. e. FNN) in most studies. The striking arrival of deep neuron networks into the field of DTI prediction occurred during the virtual screening Merk challenge whose winners used FNN on top of expert-based features. In later studies  [68] , the winners of the challenge performed an exhaustive evaluation of the effect of hyper-parameters of FNNs to predict binding of around 140,000 compounds in 19 target assays from the PubChem database.
In that context, Ramsundar et al. [70]  used several datasets containing information about toxicity, pitfalls in virtual screening, results of biological assays, or interactions with protein targets. They reported that multitask FNN outperforms other approaches, and that adding more training data was still improving the multitask model. However, Dahl et al. [68]  showed that multitask FNN performed slightly better than singletask FNN only for some tasks. This was confirmed by Xu el al. [71]  who showed that positive transfer learning is conditioned by sharing a significant amount of input data whose labels are correlated. In general, other studies  [72, 73, 74, 75]  reported that FNN performs the best for a variety of chemoinformatics problem, but only for some metrics and datasets. Overall, it is however possible to consider that FNN, with expert-driven chemical descriptors as inputs, is a reference method in deep learning for chemoinformatics problems, to which other methods should be compared.
The authors considered the Davis and Kiba datasets, both containing selectivity assays in the kinase protein family, so that this study focuses on DTI prediction within a single protein family. The Davis dataset contains interactions between 442 proteins and 68 ligands and the Kiba dataset contains interactions between 467 targets and 52 498 drugs. They obtained better or similar performance compared to Simboost  [77]  (a gradient boosting machine-based method) and KronRLS  [16]  (analogue of KronSVM but with kernelised RLS) in terms of Mean Square Error (MSE). When "binarying" the outputs (they set the threshold of pKd value, distinguishing actives from inactives, to 7), their method performed the best, in terms of AUPR, for both datasets, but significantly better only for the KIBA dataset.
Importantly, they did not compare to FNN (with expert-driven descriptors for molecules and proteins as inputs) and to NRLMF, which led to state-of-the-art performances in many studies, as mentioned previously. In a more sophisticated approach, Tsubaki et al. [78]  considered the following blocks: (1) GNN modules to encode molecules, in which the molecular graph-level representation m (l) is learnt with an aggregation function AGGREGAT E (l) graph corresponding to the sum over the atom level representations, as in the minimal GNN defined above. (2) Protein encoding h prot is performed with a CNN on amino acid representations which learn latent representation at the amino acid-level h (aa)i. (3) The Comb module uses an attention mechanism that affects weights to learnt representation of each amino acid in the sequence.
This attention mechanism allows interpretation of predictions based on which amino acids are involved in ligand binding. This was tested by comparing the amino acids found to be critical for interaction with a molecule by the attention mechanism to those found to interact with the molecule when docking the ligand into the binding pocket of the protein. Note that this control is based on a prediction methods (docking), and not on experimental data, and that it can be performed only for proteins with known 3D structures. The authors used two datasets extracted from the DrugBank database. First, a human data set with 3,500 positive interactions between 1,000 unique molecules and 850 unique human proteins. Second, a C. Elegans dataset containing 4,000 positive interactions between 1,434 unique molecules and 2,504 unique C.
Elegans proteins. They compared the performance of their method to an approach based on SVM with fingerprint-RBF kernel (not known to be state-of-the-art comparing to the use of the Tanimoto kernel), on various test "positive:negative" ratios (1:1, 1:3 and 1:5). Indeed, much more negative than positive interactions are recorded, and considering more negative interactions in the test set correspond to a more realistic setting in the context of proteome-wide DTI prediction. For the human-specific dataset, the precision score is always higher for SVM (around 5% improvement), but conversely, the ROCAUC is always higher for their deep learning approach (from 1% to 5% improvement). Overall, the ROCAUC varied in the range of 0. 91 to 0. 97. For the recall score, SVM suffers more from an imbalanced positive:negative ratio than their deep learning approach.
Indeed, their results on the gold standard Yamanishi dataset  [13]  display similar performance than kronRLS and KBMF2K  [29]  (a probabilistic matrix factorisation approach), while these methods were shown to be outperformed by NRLMF on this dataset. Moreover, the choice of the Yamanishi dataset is questionable here. Indeed, it is relatively small, so that it is not well suited to fairly evaluate deep learning models, as will be further discussed in the Results section. Overall, we are far from the striking gap of performance obtained by deep learning-based models in the image processing and natural language processing fields. Moreover, the datasets considered in the above studies are very heterogeneous, and recent promising developments in GNN technologies have not yet been explored in chemoinformatics, and in particular, in chemogenomics.
Therefore, in the present paper (see Results section), one of the directions that we explored is related to some of the most promising directions in GNN research, to evaluate their potential to improve predictions with the chemogenomic neuron network. In the following, to investigate whether neuron networks and related recent developments can lead to significant improvements for chemogenomics, we first compare, on the two DrugBank-based datasets, the standard formulation of the chemogenomic neuron network with the state-of-the-art methods presented above. Indeed, the chemogenomic neuron network is very modular, which let significant room of improvement. Therefore, we then evaluate separately promising neuron network architectures for the molecular graph encoder, protein sequence encoder and Comb operation, in order to explore various deep learning strategies for data featurisation.
Then, we adopt another perspective to improve chemogenomic predictions with deep learning. Indeed, data featurisation and data augmentation techniques are independent levers of improvement for the chemogenomic neuron network prediction performance. Therefore, we also explore the simultaneous use of expert-based and learnt features in a differentiable fashion, with the idea that combining both types of features might increase performances. This strategy uses multiple views of the data as a way to provide "more informative" data. We also explore data augmentation thanks to transfer learning via curriculum learning. This refers to using an additional larger dataset for pre-training the neuron network with an auxiliary task, before using them it in the chemogenomic task of interest, with the aim of learning richer representations of molecules or proteins.
Dataset Several bio-activity databases are publicly available. Some provide quantitative measurements such as the IC50, EC50, Ki and potency (in particular the PubChem  [79] , ChEMBL  [80]  and BindingDB  [81]  databases), while others report binary information on activities or interactions between molecules and protein pairs, like the DrugBank  [82]  or KEGG  [83]  databases. The DrugBank database  [82]  is a widely considered bio-activity database. Although much smaller than PubChem and ChEMBL, it provides high-quality information for approved and experimental drugs along with their targets. It contains around 17,000 curated drug-target associations with high-quality standards. Therefore, we used the DrugBank database version 5. 1. 0 to build two druggable proteome-wide drug-target interaction datasets.
In the first dataset, called DBH, we only kept interactions including human proteins and their ligands, whereas in the second, called DBEC, we only kept interactions including Escherichia Coli proteins and their ligands. DBEC is composed of 592 molecules targeting a total of 314 proteins, and includes 874 protein-ligand interactions. DBH is composed of 4834 molecules targeting a total of 2561 proteins, and includes 13070 protein-ligand interactions. In both cases, recorded protein-ligand interactions constitute the positive training pairs, and the vast majority of the targets and drugs are involved in only one or two known interactions. All other protein-ligand pairs are unlabeled because no interactions were recorded for them in the database. Most of these pairs are expected not to interact, but a small number of them are in fact missing interactions.
However, as an approximation, we considered that all unlabeled pairs as negative examples. As we later examine curriculum learning to increase the performance on these DrugBank-based datasets, we also consider a PubChem-based and a protein sub-cellular localisation datasets as auxilliary datasets for neuron network pre-training. We pre-processed the PubChem datasets included in the MolNet dataset collection  [84]  (namely, the PCBA dataset) to keep the bioassays composed of a maximum of common molecules. The resulting dataset consists in a total of 439, 863 unique molecules and 90 binary classification tasks (bioassays). For each bioassay, molecules are labelled as active, inactive or unknown. When a molecule is labelled as unknown for a bioassay (because it has not been experimentally tested), the prediction error for the corresponding task and molecule is set to zero.
First, we considered four way to split our Drug-Bank-based dataset for the cross-validation scheme, each corresponding to a typical chemogenomic setting. The S 1 settings corresponds to data split at random, S 2 to the orphan ligand case (pairs in one fold only contain molecules that are absent from all other folds), S 3 to the orphan protein case (pairs in one fold only contain proteins that are absent from all other folds), and S 4 to double orphan (pairs in one fold only contain proteins and molecules that are both absent from all other folds). This approach was suggested by Pahikkala et al. paper  [87] , because it allows assessment of performance in various real-life situations in chemogenomics. Note that the folds of S 4 were built by intersecting those of S 2 and S 3 , so that there are 25 folds in the S 4 setting instead of 5.
Reference similarity-based approaches As reference methods not involving deep learning, we considered the kronSVM  [7]  and NRLMF  [11]  approaches as they led to state-of-the-art performances, in particular on the gold standard Yamanishi dataset. These methods have been shortly introduced in section 2 of the present study. For kronSVM, the regularisation hyper-parameter (often called C) of SVM was automatically optimised when assessing the performance via 5-fold nested cross-validation. We set the five hyper-parameters of NRLMF to their default value, as stated in the original paper  [11]. These two approaches use molecule and protein valid kernels. For proteins, we used the local alignment kernel  [88]  (LAkernel) that mimics the behaviour of the Smith-Waterman (SW) score. While the SW score only keeps the contribution of the best local alignment between two sequences to quantify their similarity, the LAkernel sums up the contributions of all possible local alignments.
LAkernel has been shown to overtake other protein sequence kernels on protein homology detection  [88]. This kernel compares proteins from the evolutionary point of view, but it seem relevant for drug virtual screening because a local similarity of sequence may correspond to a local similarity of binding pockets  [28]. For molecules, we used the Tanimoto similarity measure, defined as the ratio between the number of substructures shared by the two molecular graphs over the number of all substructures considered. This similarity measure is very widely used in chemoinformatics, and it is also a valid kernel  [89]. Moreover, ECFP4 fingerprints of molecules have been reported to perform the best  [90]  and are widely used for drug virtual screening. Therefore, we also considered the Tanimoto metric on the Morgan fingerprints (analogue of ECFP4) as the molecule kernel, that we computed thanks to RDKit library  [91]  in Python.
Notably, we spend little time in fine-tuning these hyper-parameters, compared to those of the chemogenomic neuron network. We set the number of neurons to 2000, 1000 and 100 for the successive three stacked fully connected layers, weight decay and dropout probability to 0 (no regularisation), the initial learning rate to 10 -3 , the batch size to 100, the learning rate decay to 0. 9, the number of epochs (number of times the algorithm sees the entire dataset) to 100 and early stopping patience to 20. The early stopping patience is the number of successive epochs, for which the performance on the validation data decreased, required to stop training.
In practice, training was always stopped early. After exploring the hyper-parameter spaces, we set, for both datasets, the number of filters to 100, the number of convolutional layer to 3, the stride to 3, the filter size to 8, the weight decay and dropout to 0, batch size to 20, initial learning rate to 10 -3 , learning rate decay factor to 0. 9 and number of neurons in the last prediction layer to 100. We found that training was not sensitive to relatively small variations of hyper-parameters. In particular, adding a small quantity of dropout (until 0. 5) leads progressively to a longer training and a loss of training performance, whereas it did not significantly improve the performance on the validation data. Furthermore, we repeated training and testing procedures of the chemogenomic neuron network three times for DBEC and two times for DBH (because the training was computationally too expensive), and kept only the best achieved performance.
Comparison of the "standard" chemogenomic neuron network to reference methods We first compared the performance of the standard chemogenomic neuron network (CN) to those of the considered state-of-the-art reference methods, namely SVM and NRLMF for machine-learning methods not involving deep learning, and FNN for deep-learning methods, as described in the Materials and Methods section. Performances are reported in Tables 1, 2, 3 and 4, displayed in the format "mean score±score standard deviation", for various ratios of positive:negative test samples. To summarise the results, we can first notice that the "best" method depends on the dataset and the setting. On the DBEC dataset, the NRLMF method performs globally better than SVM for non deep learning reference methods, and CN performs better than the reference method FNN for deep learning.
Overall, the NRLMF method performs slightly better in all cases. This could be explained by the fact that DBEC is a relatively small dataset, which favours to the NRLMF and SVM approaches not involving deep learning. On the larger DBH dataset, SVM outperformed NRLMF and CN outperformed FNN. Globally, the standard CN deep learning approach leads to the best results. It appears as a good default method that seems to be the most robust in a panel of settings. Globally, the performance reached on the DBH are 10 points higher than those reached on DBEC, which was expected since there are much more data (more information) to train the models in DBH than in DBEC. Furthermore, all methods appeared to be sensitive to an imbalanced test set. Indeed, when changing the "positive:negative" ratio in the test set from 1:1 to 1:5, all methods lost about 20% performance in AUPR on S 1 , and up to 30% on S 2 , S 3 , and S 4 , although SVM and NRLMF seem slightly more robust, as shown in Tables  1, 2 , 3 and 4.
However, we expect that a relatively large number of drug-like molecules interact with more than one protein  [93] , whereas most of the molecules in the database have one or two known targets. Therefore, a significant amount of false predicted positives are expected to correspond to real interactions, which means that the performance in the case of the 1:5 "positive:negative" ratio, the performances might be underestimated. Regarding the settings, the best scores are obtained for S 1 , the worst for S 4 , and the scores obtained in the S 2 and S 3 datasets are intermediate between those observed on S 1 and S 4. This was expected, since S 4 corresponds to the double orphan test set, where no pair in the training set contains proteins or ligands present in the tested pairs to guide the predictions, whereas the models can rely on training pairs containing either the same proteins in S 2 or the same ligands in S 3.
The loss of performance between the raw (S 1 ) and the double orphan (S 4 ) settings is about 10 points of ROCAUC and 20 points of AUPR. More importantly, the prediction on S 4 varies more depending on the test folds than in the other settings. This can be understood as a consequence of the small size of the test set in the S 4 setting (any test fold is 1/25 th of the total amount of data whereas they represent 1/5 th in the other settings), which might result in more heterogeneous test sets. Interestingly, the performance reached by all models is better for S 3 than for S , which suggests that predicting ligands for new protein targets is a more difficult task than predicting targets for new molecules. We think it is due to the bias in the data. Indeed, the DrugBank dataset from which our datasets were extracted, contains molecules from drug discovery projects that usually focused on some specific molecular scaffolds  [94]  binding to a given therapeutic target.