dataesr/scientific-paragraphs-categorization huggingface dataset:
Experiment 1 Modelling. Human data for experiment 1 is based on  [28]. We trained CapsNets with the above architecture to solve a vernier offset discrimination task and classify groups of identical shapes. The loss function included a term for shape type classification, a term for vernier offset discrimination, a term for the number of shapes in the image, a term for reconstructing the input based on the network output, and a term for localizing the stimulus position in the image (see Eqs 1-6). The shape repetition, reconstruction and location loss terms are used to stabilize performance but are not essential for producing (un)crowding (see Table A in S1 Appendix). Each loss term was scaled so that none of the terms dominated the others. For the shape type classification loss, we implemented the same margin loss as in  [39].
All loss terms decreased during training (see S2 Appendix). L reconstruction ¼ X i;j ðinput The training dataset included vernier stimuli and six different shape types (Fig  7A ). Shapes were presented in groups of one, three or five shapes of the same type. The group was roughly centered in the middle of the image. We trained N = 10 networks and averaged their performance. Importantly, the network was never exposed to (un)crowding stimuli during training. Therefore, the network could not trivially learn when to (un)crowd by overfitting on the training dataset. This situation is similar for humans: they know about shapes and verniers, but their visual system has never been trained on (un)crowding stimuli. After training, we tested the vernier discrimination performance on (un)crowding stimuli (Fig 7B ), and obtained reconstructions of the input images.
In the cuboids configuration, perspective cuboids were presented to the left and to the right of the vernier (width = 58', angle of oblique lines = 135˚, length = 23. 33'). Cuboids contained the lines from the lines condition as their centermost edge. Observers were instructed to fixate a fixation cross during the trial. After each response, the screen remained blank for a maximum period of 3 s during which the observer was required to make a response on vernier offset discrimination by pressing one of two push buttons. The screen was blank for 500 ms between response and the next trial. An adaptive staircase procedure (PEST;  [67] ) was used to determine the vernier offset for which observers reached 75% correct responses. Thresholds were determined after fitting a cumulative Gaussian to the data using probit and likelihood analyses.
In order to avoid large vernier offsets, we restricted the PEST procedure to not exceed 33. 3' i. e. twice the starting value of 16. 66'. Each condition was presented in separate blocks of 80 trials. All conditions were measured twice (i. e. , 160 trials) and randomized individually for each observer. To compensate for possible learning effects, the order of conditions was reversed after each condition had been measured once. Auditory feedback was provided after incorrect or omitted responses. Modelling. To model the results of experiment 2, we trained CapsNets to solve a vernier offset discrimination task and classify verniers, cuboids, scrambled cuboids and lines. The loss function included the shape type loss, the vernier offset loss and the reconstruction loss from experiment 1 (Eqs 2, 3 and 5): L total ¼ a shape type L shape type þ a vernier offset L vernier offset þ a reconstruction L reconstruction ð7Þ The training dataset included vernier stimuli and one of three different shape types (lines, cuboids, scrambled cuboids; Fig  8A ).
The scrambled cuboids were included to make the task harder, and to prevent the network from classifying cuboids simply based on the number of pixels in the image. Similarly, the lines were randomly presented in a group of 2, 4 or 6 to prevent the network from classifying cuboids simply based on the number of pixels in the image. Both, cuboids and shuffled cuboids were always presented in groups of two facing each other. The distance between the shapes was varied randomly between one to six pixels. As before, (un)crowding stimuli were never presented during training. Therefore, the network could not trivially learn when to (un)crowd by overfitting on the training dataset. After training, we tested the vernier discrimination performance for (un)crowding stimuli (Fig 8B ) while varying the number of recurrent routing by agreement iterations from 1 to 8 iterations.
ACKNOWLEDGEMENTS We thank  Zvi Bern ,  Marco Chiodaroli ,  Lance Dixon ,  André de Gouvêa ,  Henrik Johansson ,  Ian Low ,  Radu Roiban ,  Oliver Schlotterer , and  Zhewei Yin  for a combination of helpful discussions and comments on the manuscript. JJMC and LR are supported by the  European Research Council  under  ERC-STG-639729 ,  Strategic Predictions for Quantum Field Theories.
We first propose a formulation of chemogenomics with deep learning, called the chemogenomic neural network (CN), as a feed-forward neural network taking as input the combination of molecule and protein representations learnt by molecular graph and protein sequence encoders. We show that, on large datasets, the deep learning CN model outperforms state-of-the-art shallow methods, and competes with deep methods with expert-based descriptors. However, on small datasets, shallow methods present better prediction performance than deep learning methods. Then, we evaluate data augmentation techniques, namely multi-view and transfer learning, to improve the prediction performance of the chemogenomic neural network. We conclude that a promising research direction is to integrate heterogeneous sources of data such as auxiliary tasks for which large datasets are available, or independently, multiple molecule and protein attribute views.
Outline and contributions "Reference shallow methods in machine-learning for chemogenomics" section recalls key reference shallow machine-learning methods for chemogenomics. In "Proposed chemogenomic neural network" section, we present a formal scheme, named the chemogenomic neural network (CN), for molecule and protein representation learning in the context deep learning for chemogenomics. "Related works" section shortly reviews related works in chemogenomics and QSAR with deep learning approaches. It is followed by the "Materials and methods" section. Then, in the "Results" section, we first compare the prediction performances of state-of-the-art shallow and deep machine-learning methods, which was never discussed in previous chemogenomics studies. We show that, on small datasets, the more simple and less computationally demanding shallow methods perform better than deep learning methods.
On large datasets, the proposed chemogenomic neural network (CN) with representation learning competes with state-of-the-art shallow and deep methods that use expert-based descriptors, but is not ultimately superior. These conclusions differ from those of previous works which considered specific datasets and compared deep learning approaches with baseline (i. e. not state-of-the-art) shallow methods. Furthermore, we consider data augmentation techniques, namely multi-view and transfer learning. In particular, we tested the interest of combining expertbased and learnt features. We also tested various implementations of transfer learning using additional larger datasets to pre-train the encoders, before their use in the chemogenomic task of interest. We show that multi-view and transfer learning can improve prediction performance of deep learning methods on small datasets, although not reaching the performances of shallow methods.
As another reference shallow model, we also used Matrix factorisation (MF) approaches that decompose the (protein, molecule) interaction matrix into the product of two matrices of lower ranks that operate in the two corresponding latent spaces of proteins and molecules  [22, 26]. More precisely, we used the NRLMF method developed by Liu et al  [23] , because it proved to outperform other shallow state-of-the-art methods on various datasets  [9, [18] [19] [20] 22].
Related works Direct predecessors of the proposed chemogenomic neural network are introduced below. Ozturk et al. [48]  used the following architecture for the four blocks of the chemogenomic neural network: (1) a three layers CNN on SMILES representations to encode molecules, (2) three layers CNN on protein sequences to encode proteins, (3) concatenation to combine the learnt molecule and protein encodings (4) a three layers feedforward neural network to predict the affinity of molecules for proteins. Their study was restricted to the kinases protein family, and therefore, was far from covering the protein diversity required to predict drugs unexpected targets. On this focused dataset, they obtained better or similar performance compared to Simboost  [49]  (a gradient boosting method) and KronRLS  [12]  (analogue of KronSVM but with kernelised Recursive Least Square) in terms of Mean Square Error (MSE).
When "binarising" the outputs based on a threshold in pKd value to distinguish actives from inactives, their method performed slightly better than the considered shallow methods, in terms of AUPR. However, they did not compare to the simpler architecture of FNN with expert-driven descriptors for molecules and proteins as inputs, or to state-of-the-art shallow methods such as NRLMF. In a more sophisticated approach, Tsubaki et al. [50]  considered the following blocks: (1) GNN modules to encode molecules, in which the molecular graph-level representation m (l) is learnt with an aggregation function AGGREGATE (l) graph corresponding to the sum over the atom level representations, as in the "minimal" GNN defined above. (2) Protein encoding h prot is performed with a CNN on amino acid representations which learns latent representations at the amino acid-level h (aa) i.
Importantly, they did not provide a clear comparison with state-of-the-art shallow methods, and as in  [50] , they did not either compare to a simple FNN with expert-based descriptors as inputs, which often leads to the best prediction performances, as shown in "Results" section of the present paper. Overall, the results of previous works in chemogenomics with deep learning are far from the striking gap in performance obtained by deep learning-based models in the image or natural language processing fields. They do not provide comparison to simpler deep learning methods, or to state-of-the-art shallow methods. Moreover, the datasets considered in these studies are very heterogeneous and often, do not cover the protein diversity required for drugs specificity prediction. Deep learning approaches have also been proposed in QSAR methods, such as Koutsoukas et al.
Datasets The DrugBank database  [52]  is widely used as a bio-activity database. Although much smaller than PubChem  [53]  and ChEMBL  [54] , it contains around 17000 curated drug-target associations with high-quality standards. Therefore, we used the DrugBank database version 5. 1. 0 to build two interaction datasets that span proteins over the whole druggable proteome. The first dataset, called DBHuman, keeps interactions involving human proteins and their ligands, whereas the second, called DBEColi, keeps interactions involving Escherichia coli proteins and their ligands. DBEColi is composed of 592 molecules targeting 314 proteins, and includes 874 interactions. DBHuman is composed of 4834 molecules targeting 2561 proteins, and includes 13,070 interactions. In both cases, known interactions are the positive training pairs, and most targets and drugs are involved in only one or two known interactions.
All other protein-ligand pairs are unlabelled. Most of them are expected not to interact, but a small number may be missing interactions. However, we considered unlabelled pairs as negative examples, as commonly assumed. We also used a larger dataset derived from the ChEMBL database. We recovered all drugs (according to the filter defined in ChEMBL) targeting human proteins. Drugs had on average many more targets than in the DrugBank-based datasets, and we restricted the number of targets per drug to 40, to keep manageable computational times, leading to around 56,000 interactions between about 3700 small molecules and 1700 proteins. This dataset is used to evaluate whether the main conclusions drawn on the DrugBank-based datasets that we studied in more details, may generalise to other datasets.
To provide a source task for transfer learning, we also consider a PubChem-based dataset to pre-train the molecule encoder on auxiliary tasks. More precisely, we used the MolNet dataset collection  [55]  (namely, the PCBA dataset) to build a large dataset consisting in 439, 863 molecules and 90 binary classification tasks, i. e. PubChem bioassays. For each bioassay, molecules are labelled as active, inactive or unknown. When a molecule is labelled as unknown for a bioassay because it has not been experimentally tested, the prediction error for the corresponding task and molecule is set to zero.
Evaluation procedures We evaluated the performance by 5-fold nested crossvalidation. Although we recorded the ROCAUC  [56]  and AUPR  [57]  scores for each test fold, we predominantly refer to the AUPR. Indeed, the AUPR score is considered as a more significant quality measure than the ROCAUC when negative interactions are in fact unknown interactions, as in DTI prediction, and when there are many more negative than positive samples in the test set, as in all experiments of this paper. We considered four ways to split the two Drug-Bank-based datasets for the cross-validation scheme, corresponding to different settings. The S 1 setting cor- responds to data split at random, S 2 to the orphan pro- tein case (pairs in one fold only contain proteins absent in all other folds), S 3 to the orphan ligand case (pairs in one fold only contain molecules absent in all other folds), and S 4 to double orphan (pairs in one fold only contain proteins and molecules both absent in all other folds).
Reference shallow methods As reference methods not involving deep learning, we considered the kronSVM  [5]  and NRLMF  [23]  approaches introduced in "Reference shallow methods in machine-learning forchemogenomics"section, because they led to state-of-the-art performances in various studies and datasets. For kronSVM, the regularisation hyper-parameter of SVM, often called C, was optimised by assessing the performance in a 5-fold nested cross-validation scheme. We set the five hyper-parameters of NRLMF to their default value, as recommended in the original paper  [23]. These two approaches require valid kernels for molecules and for proteins. For proteins, we used the local alignment kernel  [59]  (LAkernel) that mimics the Smith-Waterman (SW) score. LAkernel was shown to overtake other protein sequence kernels on protein homology detection  [59] , but it also proved relevant for drug virtual screening  [25].
For molecules, we used the Tanimoto similarity measure, defined as the ratio between the number of substructures shared by the two molecular graphs over the total number of considered substructures. This similarity measure is very widely used in chemoinformatics, and it is also a valid kernel  [60]. We also considered a Random Forest with proteochemometric descriptors, because this method lead to the best prediction performance on specific protein families datasets. More precisely, we used the implementation of Random Forest from scikit-learn  [61]  and considered 512-bits ECFP as molecular descriptors (extracted with the RDKit python package), and zscales(3) together with protein fingerprints(FP8) as protein descriptors (extracted with the R package), as suggested by van Westen et al. [62].
Reference deep-learning method As reference deep learning method, we considered a simple feed-forward neural network (FNN) with concatenation of expert-based numerical feature vectors as inputs for proteins and molecules, as mentioned in "Related works" section. To derive the expert-based descriptors for molecules, we extracted 1021-dimensional structural Morgan fingerprint vectors (analogues of ECFP4) with the RDKit library to extract the following features, based on Coley et al. work  [35] : "atom identity", "number of hydrogens", "number of heavy neighbours", "formal charge", "is in ring", "is aromatic", "polar surface area", "partial charge", "chirality tag". The chemical bond description contains: "bond type", "is in ring", "is aromatic", "is conjugated". For protein expert-based descriptors, we used the ProtR package to extract 1920-dimensional feature vectors corresponding to 11 protein level feature groups including amino acid composition (AAC), dipeptide composition (DC), autocorrelation descriptors, "composition, transition and distribution" descriptors (CTD), quasi-sequence-order descriptors (QSO), pseudo-amino acid composition descriptors (Pse-AAC), amphiphilic pseudo-amino acid composition descriptors (Am-Pse-AAC), topological descriptors at atomic level and total amino acid properties.
Comparison of the proposed chemogenomic neural network to reference methods We first compared the performance of the chemogenomic neural network (CN) to those of state-of-theart reference methods, namely kronSVM and NRLMF for machine-learning shallow methods, and FNN with expert-based descriptors as inputs for deep-learning methods, as described in "Materials and methods". All performances are reported in Supporting Materials, displayed in the format "mean score±score standard deviation", for various ratios of positive:negative test samples. Figures  3a, b , 4a, b display the ROCAUC and AUPR performance obtained on the DBEColi and DBHuman datasets, for a "positive:negative" test sample ratio set to 1:5. The AUPR and ROC-AUC scores obtained for other ratios are gathered in Additional file 2.
First, we observe that the best performing method depends on the dataset and on the setting. On the small DBEColi dataset (874 interactions), ROCAUC scores are very close for all methods, but the shallow NRLMF method tends to perform the best overall. For the more important AUPR score (as explained in "Materials and methods" section), the NRLMF and kronSVM shallow methods clearly outperform deep Fig. 3  Performances on DBEColi for the S1 (random split), S2 (orphan proteins in test set), S3 (orphan molecules in test set), and S4 (double orphan in test set) settings, with a positive:negative samples ratio set to 1:5. For each setting, the order from left to right in which the results are displayed is given in the legend methods. Among deep learning methods, overall, the proposed CN performs better than the FNN reference method.
Our results show that, on small datasets (a few thousand interactions, or less), shallow methods should be preferred. NRLMF appears as a good default method, although its performances in the S 2 and S 4 orphan set- tings are very close to those of the proposed CN deep method, because it is simpler and computationally efficient. On the larger DBHuman dataset (13,070 interactions), overall, the two deep learning methods CN and FNN outperform the two shallow methods NRLMF and kronSVM, both in terms of ROCAUC and AUPR. This is particularly true in the three orphan settings ( S 2 , S 3 , and S 4 ) which are important in the context of drug specific- ity prediction. The proposed CN deep learning approach globally performs sightly better than the FNN reference deep learning approach in ROCAUC, and similarly in AUPR.
Overall, for large datasets (in the range of ten thousand of interactions, or above), deep learning methods should be preferred to shallow methods, because they perform better on a wide panel of settings. This tendency will be confirmed below in the Discussion section, on the larger ChEMBL-derived dataset (56,000 interactions). Globally, the performance reached on the DBHuman are 10 points higher than those reached on DBEColi, which was expected since there are much more data, i. e. more information to train the models, in DBHuman than in DBEColi. Regarding the settings, the best scores are obtained for S 1 , the worst for S 4 , and the scores obtained for S 2 and S 3 are intermediate. Indeed, S 4 corresponds to the double orphan test set, where no pair in the training set contains proteins or ligands present in the tested pairs to guide the predictions, whereas the models can rely on training pairs containing either the same proteins in S 2 or the same ligands in S 3.
We considered (1:1, 1:2, 1:5 and 1:10) "positive:negative" samples ratios in the train set. We did not test larger proportions of negatives because it would be hardly computationally tractable, and it would correspond to strongly imbalance datasets. The kronSVM and NRLMF shallow methods did not benefit from an increasing number of negative samples in the train set, whereas deep learning-based methods strongly did between 1:1 and 1:5. More precisely, increasing the number of negative samples in the training set, up to five times the number of positive samples, increased the performance of CN and FNN by about 10% in AUPR and 2% in ROCAUC for the DBHuman dataset, and by 20% in AUPR and 10% in ROCAUC for the smaller DBEColi dataset. We also observed that CN benefits slightly less than FNN from an increasing number of negatives in the training set.
5  is significantly lower than that obtained by the CN chemogenomic neural network of Fig. 1 , and by the reference FNN with expert-based features, on the four S 1 , S 2 , S 3 , S 4 settings of the DBEColi dataset (data not shown). A possible explanation is that simultaneous training of the MLP fea and MLP pair networks with molecular graph and protein sequence encoders is an ill-conditioned non-convex optimisation problem. To address this issue, we separately pre-trained the pairs of neural networks encoding expert-based and learnt features, before re-training them together on the same training data. More precisely, the final MLP pair net- work is trained with the GNN and CNN networks alone, then with the MLP fea for proteins and molecules alone. The pre-trained GNN and CNN, and MLP fea networks are finally re-trained together, as in the architecture displayed in Fig.
5. The performances of this network for a single test/ train split are reported in Table  1  at the row named CN -feaMLP. Remarkably, this approach outperformed the CN chemogenomic neural network on the S 1 setting, while performing similarly or above on the three others. To evaluate this approach in more details, we assessed its performance on the S 1 , S 2 , S 3 and S 4 settings using a 5-fold nested cross-validation scheme, with a test and train "positive:negative" ratio of 1:5 (results of Table  1  corresponded to a single train/test split). The performance on DBEColi and DBHuman are reported in Figs. 7  and 8 , together with those of reference methods and of the CN chemogenomic network represented in Fig. 1. For the DBEColi dataset, the CN -feaMLP model outperforms both the CN and the FNN reference model on S 1 and S 2 , and performs similarly, yet slightly better, on S 3 and S 4.
However, on this small dataset, the performances remain lower than those of the NRLMF shallow method, particularly on S 1 and S 3. On the larger DBHuman dataset, the CN -feaMLP model outperforms the CN chemogenomic neural network in the S 1 setting, but it does not overtake the FNN approach. In the S 2 , S 3 and S 4 orphan settings, the CN -feaMLP approach reaches slightly better perfor- mances than both the CN chemogenomic neural network and the reference FNN method. Overall, on this larger dataset, the two shallow models, kronSVM and NRLMF, still have much lower performances than the deep learning models, and the proposed CN -feaMLP model dis- played robust performances that remained among the best achieved in the four settings. We conclude that data augmentation techniques based on multi-views of the data improve the prediction performance of the deep-learning algorithm CN.