dataesr/scientific-paragraphs-categorization huggingface dataset:
The SVM and NRLMF methods did not benefit from an increasing number of negative samples in the train set, whereas deep learning-based methods, as expected, strongly did. More precisely, increasing the number of negative samples in the training set, up to five times the number of positive sample, increased the performance of CN and FNN by about 10% in AUPR (only 2% in ROCAUC) for the DBH dataset, and by 20% in AUPR (ten points of ROCAUC) for the smaller DBEC dataset. We also observed that CN benefits slightly less than FNN from an increasing number of negative in the training set. The performance of CN and FNN did increase between a ratio of 1:5 and 1:10 in some cases, meaning the neuron network-based models may still benefit more from more training negative samples. In the following, we tested several changes in the "standard" chemogenomic neuron network (CN) discussed in the two preliminary studies described in the current section.
None of these two modifications lead to significant performance improvement for the four settings, as reported in Table  5. Although the GAT function led to substantial improvements in gold standard datasets of graph representation learning, it did not in chemogenomics. This may be due to the the fact that molecular graphs are small, and that all atoms of molecules are informative when learning an end-to-end molecule encoding. In the case of the "wbond" function, the standard GNN may already leverage bond information based on the nodes attributes and degrees (i. e. , based on the topology of the graph). Independently, we tested several options to replace the summation in the AGGREGAT E (l) graph function and standard COM BIN E graph function of the minimal GNN, that builds the graph-level representation directly from node representations.
3,0. 6}). In the case of hierarchical pooling, that we implemented based on Ying et al. [51] , we also varied the dimension of the nodes embedding (i. e. the number of convolutional filters) and the quantity of dropout when building the soft assignment matrices. None of these modifications led to substantial performance improvements for the four settings on the DBEC dataset. Intuitively, this means that, in the case of small chemical compounds, the simple sum function has the same representation power than the max for the AGGREGAT E (l) graph function and concatenation for the COM BIN E graph function. Therefore, in the case of small molecular graphs, the sum function can capture representative elements in the graph (as well as the max function) and can learn to store information of subgraphs of different sizes (as well as the concatenation function), if these strategies are relevant for DTI prediction.
It is most likely that the self-attention mask "σ(W att • [h prot , h mol ] + b att )" produces a uniform attention weight distribution in most cases. Up to now, we only explored modifications of the neuron network architecture to encode (protein,molecule) pairs from their raw representation. We conclude that searching for more sophisticated architectures is not the right direction to improve the prediction performance of the the chemogenomic neuron network, at least as observed on our DrugBank-based benchmark dataset. In other words, since the performance does not see to be limited by the representation power of the chemogenomic neuron network architecture, the bottleneck seems to be related to data featurisation, diversity and quantity. In particular, a feed forward neuron network on top of expert-based features is highly competitive.
4a ), we explored the hyper-parameter spaces by varying the number of layers (from 1 to 3), the number of hidden units (in {10,100,200,1000,2000}) while promoting pyramidal architectures (as motivated in  [70] ) and the quantity of dropout (in {0,0. 3,0. 6,0. 9}). To rule out the possibility of an implementation error, we performed two sanity checks. First, we checked that we recover the performance obtained by an MLP with expert-based features when we set to one the dimension of the expert-based features (i. e. we silent the protein sequence and molecular graph encoders while keeping the same implementation). Similarly, we also checked that we recover the performance obtained with expert-based features while silencing M LP f ea and M LP pair in the same manner. The performance reached by such a model is significantly lower than that obtained individually by the chemogenomic neuron network and an FNN with expert-based features, on the four S 1 , S 2 , S 3 , S 4 settings of the DBEC dataset (data not shown).
4a. The performance of such a network are reported in Table  5  at the row named CN -f eaM LP. Remarkably, this approach significantly outperformed the standard chemogenomic neuron network for the S 1 setting, while performing similarly on the three others. More precisely, the best-reported performance on the S 1 setting is the one reached by the reference FNN that simply uses expert-based features as input. To further challenge this approach, we assessed the performance on the four S 1 , S 2 , S 3 and S 4 settings using a 5-fold nested cross-validation scheme, with a test and train "positive:negative" ratio of 1:5. The performance on DBEC and DBH are reported in Figure  5  next those reached by the reference methods and the standard chemogenomic network of the present study (CN). For the DBEC dataset, the CN -f eaM LP model outperforms both the standard chemogenomic neuron network and the FNN reference model (with expert-based protein and molecule descriptors as inputs) on S 1 and S 2 , and performs similarly, yet slightly better, for the S 3 and S 4.
However, the performance is still significantly lower than that of NRLMF, on this small dataset. Therefore, we conclude that the NRLMF approach should be used on relative small datasets. However, on the larger DBH dataset, the CN -f eaM LP model outperforms the standard chemogenomic neuron network model for the S 1 , but it does not overtake the FNN approach. For S 2 , S 3 and S 4 , the CN -f eaM LP approach reaches slightly better performance than both the standard chemogenomic neuron network and the reference FNN method. Overall, we conclude that, for relatively large datasets like DBH (that are widely available our-days), the CN -f eaM LP model provides the best DTI prediction performance in chemogenomic studies. Moreover, we believe that integrating other multiple description views of molecules and proteins in the same model, is an interesting domain for future developments.
Evaluation of transfer learning for chemogenomics The principle of transfer learning is to gain knowledge while solving one problem and applying this knowledge to a different but related problem. In fact, prediction of DTIs in the context of chemogenomics can already be viewed as a form of transfer learning. Indeed, we can formulate the problem of predicting DTIs at a large scale in the protein and molecule spaces as a problem of predicting ligands for each protein, while learning on all the problems of predicting ligands for all the other proteins. In this particular case, all the tasks are formally identical, since they correspond to predicting ligands for proteins. In the present section, we investigate the efficiency of transfer learning in the context of chemogenomics, using tasks that are not related to the prediction of ligands for proteins within the same dataset, but other tasks involving proteins and expected to implicitly contain valuable biological information for solving the chemogenomic problem.
In the context of deep learning, transfer learning can take two forms: pre-training, or co-training encoders on separated larger prediction tasks (even unsupervised ones). Leveraging the information in larger datasets is aimed to enhance performances on the source task. Co-training refers to the parameter sharing strategy in which we train several prediction tasks simultaneously, while task-specific neuron encoders share information in order to enable them to learn a richer set of features. We tested co-training on two prediction tasks which are directly related: one based on the "SCOPe" database  [95]  on which we classified 13. 963 proteins over seven types of 3D fold (alpha proteins, beta proteins, alpha and beta proteins, multi-domain proteins, membrane and cell surface proteins and peptides, small proteins), and one a "secondary structure" dataset  [67]  composed of 4388 proteins in which all amino acids are assigned to one of the eight possible types of secondary structures.
Thus, these two prediction tasks are explicitly related since they both characterise protein folding, but at different level. The co-training of the protein sequence encoder on the SCOPe and "secondary structure" datasets did not yield to an improvement of performance (data not shown), although these two tasks are strongly related. A similar observation was made in the Natural Language Processing field  [96]. Co-training learnable feature encoders is likely to be too restrictive and therefore, is probably efficient only in few specific cases. Therefore, we did not further consider co-training in this study. Pre-training, also called "curriculum learning", is a procedure in which the model is first trained on a large dataset (called the source dataset) to learn a rich set of features that are optimised to make predictions for source task trained on a large dataset.
A sketch of curriculum learning in the case of the chemogenomic network can be found in Fig. 6. Next, this pre-trained model is fine-tuned by re-training the weights on the smaller dataset of interest (named the target dataset). This means that, when training for the target task, the initial parameters of the encoder are not randomly chosen, but are assigned to their optimised values for the source task. In such a procedure, the model fits the large input data available for the source task, and fine-tunes its weights in a second phase to adapt to the smaller dataset of the target task. Alternatively, it is also possible to "freeze" (i. e. not retrain) the pre-trained molecule and protein encoders, and train only the final FNN that take as input the representations learnt from the pre-trained encoders.
In the case where the source and target datasets are related "enough", this could improve predictions on the target dataset. For instance, Paul et al  [97]  performed such transfer learning in a chemoinformatics problem related to organic solar cell screening (only chemicals are screened). Figure  6 : Sketch of curriculum learning with the chemogenomic neuron network. First, the learnable protein sequence and molecular graph encoders are optimised with molecule-and protein-specific source tasks (grey path). In a second phase, the pre-trained sequence and molecular graph encoders are used (either "frozen" or re-trained, to fit the chemogenomic prediction task (black path). Before discussing our results, we start with general observations about implementation of curriculum learning with the chemogenomic neuron network.
Furthermore, we varied the number of layers and neurons in the prediction sub-network in order to search for the best architecture when freezing one or the two encoders. Indeed, an MLP may need a relatively large width and depth to leverage the representations extracted by pre-trained frozen encoders. Finally, it is of general interest to explore whether transfer learning can improve the prediction performance of the standard chemogenomic network. However, we chose to focus on the DBEC dataset. Indeed, it seemed more critical to perform data augmentation on a small dataset like DBEC, and because we already showed that deep learning methods could outperform reference shallow machine-learning methods (SVM or NRLMF ) on larger datasets like DBH. Therefore, the goal was to evaluate whether transfer learning improves prediction performance of the chemogenomic neuron network on DBEC, and whether these improvements allowed this deep learning method to reach the performance of reference shallow methods.
In a first attempt, we considered to pre-train the three parts of the chemogenomic neuron network (the protein sequence encoder, the molecular graph encoder and the prediction sub-network) on the larger DBH dataset in order to use it for prediction on the smaller DBEC dataset. This setting is slightly different from that shown in Fig. 6 : here, a single source task is used to pre-train the whole CN network (protein encoder, molecule encoder and FNN), corresponding to predicting DTIs on the larger DBH dataset, and use the optimised parameters as starting values (instead of random values) to re-train the whole CN on the smaller DBEC dataset. This led to a loss of performance on S 1 (0. 2 of AUPR instead of 0. 4). Interestingly, we found that this loss of performance arouse from the protein encoder pre-trained on DBH.
Indeed, pre-training on DBH and freezing the protein sequence encoder, while training the molecular graph encoder and prediction sub-network directly on the target task with randomly initialised parameters, led to the same decrease in AUPR score. On the contrary, when pre-training and freezing the molecular graph encoder while training the protein sequence encoder and the prediction sub-network directly on the target task, we obtained similar or better performance than with direct training of the chemogenomic neuron network, as shown in Table  5  at the row named "curriculum learning". In particular, the performance obtained on S 1 and S 3 were significantly better. This means that DTI prediction with the chemogenomic neuron network may benefit from a molecular graph encoder pre-trained on larger datasets.
The two prediction tasks (DBH and DBEC) are formally identical, but their training data have different characteristics. Indeed, DBEC contains E. coli proteins and their ligands, whereas DBH contains human proteins and their ligands. Therefore, the two datasets may present critical statistical bias in protein sequences characteristics, since the distribution of protein sequences in bacteria is expected to differ from that of human proteins. However, both datasets contain small organic drug-like molecules. These remarks may explain why a loss of performance is observed when pre-training with human proteins, while pre-training only the molecular graph encoder on the DBH dataset is suitable to predict DTIs on the DBEC dataset. When we pre-trained the encoders without freezing the parameters in the second phase, and use an initial learning rate in the second phase one order of magnitude lower than the one used to train directly the chemogenomic neuron network (i.
e. initial learning rate set to 10 -4 instead of 10 -3 ), the performance was lower than that obtained when training the chemogenomic neuron network from scratch. In particular, the average AUPR is 0. 3 instead of 0. 4 on S 1. This means that encoders' weights pre-trained on DBH, in particular those of the protein encoder, are not suitable starting points for fine-tuning on DBEC. To investigate these conclusions, we selected larger and chemical and protein datasets to pre-train the encoders with source tasks that are now different in nature, although related to the target task. The PCBA dataset is an excellent choice for pre-training a graph neuron network encoder for molecules since it contains the information about a hundred of bio-activities for hundreds of thousands of molecules. This source task for molecules is to predict whether they are active or not.
To our knowledge, there is no such evident choice for protein datasets. We considered the sub-cellular localisation dataset  [67] , containing six thousand proteins labelled with one of the twelve sub-cellular localisation, called the CellLoc dataset in the following (see Materials and Methods for a description of these datasets). This source task for proteins is to predict their cell localisation. Recall that the target task is to predict DTIs in the DBEC dataset. We considered pre-training only two blocks of the chemogenomic neuron network, (i) the protein sequence encoder and (ii) the molecular graph encoder in order to use them for DTI prediction on DBEC. When pre-training the molecular graph and protein sequence encoders on the P CBA and CellLoc datasets respectively, and freezing them before fitting the FNN model for DTI prediction on the DBEC dataset, we recovered the performance reached by direct training of the standard chemogenomic neuron network on DBEC.
More precisely, we recovered a performance of 0. 4 in AUPR only with a prediction neuron sub-network with three hidden layers of 1000, 500 and 100 hidden neurons each. However, we obtained 0. 28 of AUPR with a prediction neuron sub-network made of a single hidden layer of 100 hidden units. This means that the prediction sub-network needs to be complex enough to leverage the information in the representations learnt by the pre-trained encoders. In other words, these pre-trained encoders are informative with respect to the DBEC prediction task, but not as suited as encoders trained on the DBEC prediction task itself. To study curriculum learning in more details with the chemogenomic neuron network, we evaluated pre-training of the molecular graph encoder on DBH and P CBA via 5-fold nested cross-validation with a test and train "positive:negative" ratio of 1:5 on the four S 1 , S 2 , S 3 and S 4 settings of the DBEC dataset.
In the case of pre-traning with DBH, the strategy is slightly different from that presented in Figure  6. In fact, the standard chemogenomic network is pre-trained with DBH, but only the parameters of the molecule encoder are retrained on DBEC (i. e. parameters of the protein encoder and of the FNN are randomly initialised). In the case of pre-training with P CBA, since this dataset only contain bio-activities for molecules, the pre-training step only involves the molecule encoder and the FNN, and the source task is to predict the PubChem assay bio-activities. Then, the parameters of the molecule encoder are kept for retraining the chemogenomic network on DBEC. The performance are reported in Figure  7  with those of the reference methods. The final model obtained when pre-training the molecular graph encoder on the P CBA (resp.
DBH) dataset is called CN -currP CBA (resp. CN -currDBH) in Fig. 7  Pre-training the molecular graph encoder on P CBA yielded to similar performance than direct training on DBEC for the four settings. We thus confirmed with 5-fold nested cross validation the observations previously made on a single train/validation/test split for the four settings. Interestingly, pre-training the molecular graph encoder on DBH resulted in a better model than that trained directly on the S 1 , S 2 and S 3 of the DBEC dataset (CN and CN -currDBH in Figure  7 ), as observed in the previous experiment with a single train/validation/test split. Although the proteins involved in DBH and DBEC follow different statistics, the global prediction tasks are indeed strongly related, so that transfer learning can be observed between the two tasks.
Future studies could consider larger source datasets. For instance, we could consider an unsupervised learning task as a source task such as a reconstruction task like in auto-encoders  [37, 98]  (predicting the original input from the hidden representation), or an adversarial task  [99]  (distinguishing real data from fake data, such as non-valid molecules or randomly generated protein sequences). However, fig. 7  shows that, in general, the reference methods SVM or NRLMF still led to the best performance on this small DBEC dataset, and even with transfer learning, deep learning methods did not outperform these reference machine-learning methods. raw (S In the present work, we investigated deep learning methods for chemogenomics. In particular, we studied representation learning methods for molecules and proteins based on deep learning networks in the context of drug-target interaction prediction.
To this end, we first defined the chemogenomic neuron network which consists of a feed-forward neuron network (FNN) taking as input the combination of molecular and protein learnt representations based on molecular graph and protein sequence encoders. More precisely, data representation are learnt from the raw description of the considered data, in order to maximise prediction performance. For large datasets, we show that this standard chemogenomic neuron network (with GCN as graph molecule encodre and CNN as protein sequence encoder) outperforms state-of-the art reference machine-learning methods not using deep learning, and is competitive with the reference FNN models using expert-driven descriptors as inputs, particularly in orphan settings like S 2 , S 3 and S 4 , for which we found that the chemogenomic neuron network reached the best performance.
Moreover, we investigated the latest developments in graph representation learning, and point out critical issues that question the efficiency of more complex methods to learn representations in the context of chemogenomics. Although the diversity of neuron network architectures covered in the present work, and their applications in virtual screening, are not exhaustive, we can still conclude that the current research in the neuron network architecture for encoding molecular graphs and protein sequences does not seem a promising direction to improve drug virtual screening in the chemogenomic framework. Indeed, we observed that promising modifications of the graph encoder (inspired from other fields of graph learning) to learn molecule descriptors, and of the protein sequence encoder and pairwise representation encoder, as parts of the chemogenomic neuron network, did not yield to significant improvements in prediction performance, on our DrugBank-based dataset.
Therefore, for future directions, we recommend to assign efforts in the design of neuron network that integrate multiple views of molecules and proteins in the same model. However, in the case of small datasets like DBEC, these performance improvements did not allow to reach those of reference shallow machine-learning methods (SVM or NRLMF). In contrast, we report that curriculum learning did not result in strong performance improvement on the small DBEC dataset, meaning that a promising direction to improve the chemogenomic neuron network is at first to integrate various data views, rather than to increase the amount of auxiliary data. Our study builds on and provides new insights on the chemogenomic neuron network initially proposed by several authors  [101, 78, 76]. In particular, these studies did not justify their choice of neuron network architecture via the comparison of prediction performance of various architectures, did not consider most recent developments in graph neuron networks, and did not compare to established state-of-the-art machine-learning methods (SVM or NRLMF for shallow methods, and FNN with expert-based descriptors for deep learning methods).
In addition, we used different settings (S 1 , S 2 , S 3 , S 4 ) in order to better assess the performance of the considered methods in a panel of scenarios that are encountered in practice. Indeed, real case studies usually search proteome-wide off-targets for new drugs or drugs for new therapeutic targets. They are closer to the orphan settings S 2 , S 3 , or S 4 that to the S1 setting. Therefore, measuring prediction performance only on datasets similar to S 1 overestimates the performance that can be expected in real case studies  [28]. We emphasise that FNN taking as input standard expert-based molecular fingerprints and protein sequence descriptors is a good default method for prediction of DTI on relatively large datasets like DBH, and therefore, should be considered as a reference simple method that leads to state-of-the-art performance, and to which future chemoinformatic studies should be compared.