DeepPCM: Predicting Protein-Ligand Binding using Unsupervised Learned Representations

1 INTRODUCTION . A main goal of cheminformatics in the area of drug discovery is to model the interaction of small molecules with proteins in-silico . The ability to accurately predict the binding affinity of a ligand towards a biological target without the need to conduct expensive in-vitro experiments has the potential to accelerate the drug development process by enabling early prioritization of promising drug candidates ( Cortés-Ciriano et al. , 2015 ) . A common approach is to train a machine learning algorithm to predict the binding affinity of ligands towards a certain biological target using a training set of compounds that have been experimentally measured on this target . This modality is commonly referred to as a quantitative structure-activity-relationship ( QSAR ) model ( van Westen et al. , 2011 ) . QSAR models can be broadly classified into two types : single-task QSAR models and multi-task QSAR models ( Figure 1 ) . In single-task QSAR modeling , a model is trained separately for each protein to predict a binary or continuous outcome ( binding vs not-binding or the binding affinity ) given a compound input . The machine learning model used could be anything from logistic regression to deep neural networks ( Lenselink et al. , 2017 ; Cherkasov et al. , 2014 ) .In multi-task modeling , a single model is trained to predict binding across multiple proteins simultaneously , allowing the model to take advantage of the correlations in binding activity between compounds on different targets ( Caruana , 1997 ; Yuan et al. , 2016 ) . This is done , for example , by using a neural network with multiple output nodes where each output node corresponds to a different protein . Thus , multiple outputs are predicted given a compound input ( Simões et al. , 2018 ; Dahl et al. , 2014 ) . While these methods have been employed on various protein targets , there are methodological concerns to their use ( Lima et al. , 2016 ; Mitchell , 2014 ) . Both single-task and multi-task models must be retrained from scratch if one wishes to incorporate binding data for a new protein , and both can not be used at all to make predictions on new protein targets for which experimental data is absent ( van Westen et al. , 2011 ) . An attractive solution to this problem is to also include protein information in a so called proteochemometric model ( Figure 1 ) . The additional protein information enables a PCM model to directly utilize similarities between proteins for bioactivity modeling ( Cortés-Ciriano et al. , 2015 ; van Westen et al. , 2011 ) . This leads to many potential benefits over classical QSAR single-task and multi-task modeling . First , as proteins are explicitly represented by a defined featurization , PCM models can be used to make activity predictions on proteins without pre-existing bioactivity data , which is impossible with single-task or multi-task modeling . They can also be used to model binding on proteins for which experimental data may be too limited to train an effective single-task model ( van Westen et al. , 2011 ) . Second , with an expressive protein descriptor , a model could leverage similarities and differences between proteins directly to model their binding behaviors , rather than merely using correlations found among the compounds they bind to , as in multi-task modeling , or ignoring protein relationships altogether as in single-task modeling ( Cortés-Ciriano et al. , 2015 ; van Westen et al. , 2011 ) . Consequently , PCM models have found success on a variety of protein targets , and using many different machine learning methods , including random forests and SVMs ( Ballester & Mitchell , 2010 ; Weill et al. , 2011 ; van Westen et al. , 2013 ; Shiraishi et al. , 2013 ; Cheng et al. , 2012 ) . Recent advances in the field of Deep Learning have also led to its use in QSAR single and multitask modeling as well as PCM modeling ( Lenselink et al. , 2017 ; Menden et al. , 2013 ; LeCun et al. , 2015 ; Schmidhuber , 2015 ) . Lenselink et al . ( 2017 ) compared deep learning methods against other machine learning methods for PCM , single-task QSAR , and multi-task QSAR models on a benchmark dataset , and the authors found that PCM models using deep neural networks outperform other machine learning methods as well as single-task and multi-task QSAR models . All of the aforementioned works make features for both the small molecules and the proteins based on hand-crafted feature extraction protocols . Small molecules are often represented by counting and aggregating smaller substructures , while proteins are often represented by aggregation of computed physio-chemical features of their amino-acids or by encoding the differences in aligned sequences . In many application domains of Deep Learning , recent research has shown that these methods generally work better when the input data representation is lower-level and unabstracted , allowing the model to learn hierarchical features directly rather than relying on features which are hand-crafted by humans ( LeCun et al. , 2015 ; Schmidhuber , 2015 ) . For example , this is the case in computer vision , where deep learning on pixel value features has been the state of the art for several years ( Schmidhuber , 2015 ) . In cases where the input space is too high-dimensional , and especially if there is not enough labeled data to train a model end-to-end , unsupervised representation learning is used to generate lower-dimensional embeddings , again relying on machine learning , rather than human engineering , for feature extraction . This is the case in natural language processing , as well as in video analysis ( Srivastava et al. , 2015 ; Erhan et al. , 2010 ; Mikolov et al. , 2013 ) . In this work , we follow this reasoning and utilize such unsupervised-learned embeddings to represent both the ligand and protein spaces for proteochemometric modelling . To the best of our knowl- edge , we combine , for the first time , compound and protein representations both generated by deep learning models that were pretrained on an unsupervised learning task , as inputs to a PCM model . Moreover , we believe that this is the first work which combines unsupervised-learned embeddings of biological and chemical entities simultaneously in order to solve a downstream task . 1.1 COMPOUND DESCRIPTORS . The following section describes the currently used handcrafted ligand descriptors , evaluates their properties , and introduces the ligand descriptors that we use for our model — known as CDDD descriptors — which are generated via unsupervised machine learning . Handcrafted Compound Descriptors The state-of-the-art compound descriptors that are used for a vast majority of PCM and other chemoinformatics tasks are different varieties of circular fingerprints . In binary keyed circular fingerprints , each bit refers to a specific substructure 's presence or absence in the molecule , while in counts format the bits refer to the number of occurrences of the substructure ( van Westen et al. , 2011 ; Glen et al. , 2006 ; Rogers & Hahn , 2010 ) . As the number of potential substructures is vast ( ∼232 ) , the resulting sparse set of bits is usually hashed and folded to a much smaller size ( ∼103 ) at the expense of hash and bit collisions ( Rogers & Hahn , 2010 ) . These structure-based descriptors can be augmented with physicochemical descriptors , such as DRAGON or PaDEL , along with other chemical descriptors like atom identity/type , MACCs keys , and topological indices ( Mauri et al. , 2006 ; Yap , 2011 ) . Issues Circular fingerprints contain information about only the presence or absence of certain substructures in the compound – they thus fail to capture the shape or arrangement of those substructures within the compound . Furthermore , fingerprints rely on a hashing protocol to compress the millions of different substructures that are recorded into a smaller vector – as a result , hash collisions can mean that completely different substructures can correspond to the same fingerprint bit . This also means that one can not determine which exact substructures are responsible for the model prediction . CDDD – Ligand Descriptor To avoid these issues , we use the CDDD ( Continuous and Data Driven Molecular Descriptors ) , a model for the generation of lower-dimensional representation vectors of molecules developed by Winter et al . ( 2019 ) . This model uses a recurrent autoencoder trained on the task of translating non-canonical SMILES string representations of compounds into their canonical form . After unsupervised learning on approximately 72 million compound SMILES , a given compound is represented by the 512-length bottleneck of the translation model , and thus is encoded into a 512-length vector . These embeddings have been shown to be effective on QSAR prediction and virtual screening tasks ( Winter et al. , 2019 ) . CDDD descriptors offer a unique , compact , and continuous vector representation for each compound , as opposed to fingerprints , which are non-unique , discrete , and must be hashed to be made compact . Molecules with the same substructures but differently arranged will correspond to different vectors . Additionally , these unsupervised-learned descriptors have demonstrated competitive or superior results compared to molecular fingerprints on a variety of other tasks , indicating their ability to effectively represent compound properties and behaviors ( Winter et al. , 2019 ) . A diagram can be found in Appendix section D . 1.2 PROTEIN DESCRIPTORS . Next , we describe the various methodologies used for handcrafted protein descriptors . There are many different examples of highly specialized protein feature-extraction protocols used for PCM models that operate over a narrow range of targets . We describe the broad categories , examine their drawbacks and introduce the unsupervised protein descriptor which we use , known as UniRep . Handcrafted Protein Descriptors Most commonly used protein descriptors can be described as either amino-acid-based , or sequence-based . Amino-acid-based descriptors are computed using some combination of physicochemical and structural properties of individual amino acids . For example , the commonly used Z-scale descriptors and their counterpart ProtFP descriptors are constructed by computing various amino acid physicochemical properties ( such as hydrophobicity or polarity ) , taking a PCA over these properties , and then representing each amino acid by its first few principal components ( Sandberg et al. , 1998 ; van Westen et al. , 2013 ) . These descriptors can be combined with structural information descriptors , such as normalized van der Waals volume , charge , and the secondary structure of the protein at that amino acid residue ( Lapins et al. , 2013 ) . The whole protein is then represented by aggregating the amino-acid descriptors over the whole sequence . Sequence-based descriptors are constructed by taking aligned sequences or regions and using encodings to represent the amino acid identity at a given location in the aligned region . For example , in Nabu et al . ( 2015 ) , the authors build a model that predicts the bioactivity on mutated vs wild type penicillin binding proteins based on the positions of the mutations in the sequence . This was done by aligning the protein sequences and uniquely representing the positions a mutation may have occurred at with a one-hot-encoding . Thus , from 75 potential mutation sites , a 75-length vector is returned for each protein . Other variations of this technique use motifs or encodings of multiple varying segments ( Lapinsh et al. , 2005 ) . Additionally , on proteins for which 3D-information is available , this information can be incorporated into the protein descriptor . Methods vary – in general , 3D-models , for example crystal structures or water-field maps , are first aligned . Subsequently , descriptors are built based on amino acid identity at specific positions at the binding pocket or taking a PCA over the aligned field maps ( van Westen et al. , 2013 ; Lapinsh et al. , 2005 ; Subramanian et al. , 2016 ; Kruger & Overington , 2012 ) . Issues Amino acid-based descriptors require crude aggregation or binning methods , as they must convert variable length sequences of amino acids into a constant-length vector . For example , a common strategy is to divide the sequence into equal-length segments , then average over the descriptors for each segment ( Lenselink et al. , 2017 ; van Westen et al. , 2013 ) . This leads to the undesirable property where a single amino acid insertion early in the sequence would shift all subsequent segments , thus changing every descriptor bit despite a small change to the protein itself . Additionally , motifs and functional domains have variable numbers of amino acids which are not necessarily contiguous , so averaging over fixed-length contiguous segments can fail to capture relevant features consistently . Sequence and 3D-structure based descriptors must be aligned , which restricts the usable bioactivity data to only a very small fraction of closely related proteins for which an alignment is meaningful . These descriptors also can not be applied to new regions of the protein space , where bioactivity measurements of similar proteins are unavailable . Essentially , these methods build descriptors that are explicitly based on the differences between a few proteins , which greatly limits the scope of the problems that can be approached with such descriptors , as well as the amount of data that can be leveraged to train models . These data availability issues are exacerbated when using 3D-descriptors , since 3D structures are only available for a subset of proteins . UniRep – Protein Descriptor We use UniRep developed by Alley et al . ( 2019 ) , which uses a multiplicative LSTM architecture on approximately 24 million protein amino acid sequences taken from UniRef50 ( Krause et al. , 2016 ) . The UniRep model is trained on the next-character-prediction task , where the LSTM predicts the next amino acid in the protein sequence given the previous amino acids . To generate fixed-length embeddings , the hidden states of the model that are generated during the forward pass are averaged over the sequence dimension . For implementation details , refer to Alley et al . ( 2019 ) . The pre-trained model can be used out-of-the-box , taking as input amino acid sequences and outputting embeddings of length 64 , 256 , or 1900 depending on the architecture used . For the PCM model which we propose here , we found that the 256-length embeddings performed best , and so we will use UniRep to refer to the 256-length UniRep descriptor in this paper . A diagram can be found in Appendix section E UniRep descriptors offer many advantages over their handcrafted counterparts . The descriptors are alignment-free sequence-based representations , which therefore allow the utilization of protein bioactivity assay data across families and species instead of just highly similar proteins , greatly increasing the available training data . Moreover , these descriptors do not require 3D-structure information , which is available for only a small subset of proteins . Additionally , there is no need for binning or averaging across arbitrary length contiguous chunks of sequence .

Summary: This paper tries to solve the protein-legend binding prediction problem in the computational biology field. It uses the learned embedding for protein and legend, separately, from two published papers. Then those two embeddings were inputted to another deep learning model, performing the final prediction. Tested on one dataset, it shows the proposed method can outperform the other baseline methods.