Contrastive Representation Learning for 3D Protein Structures
Authors: Pedro Hermosilla
Timo Ropinski
Venue: ICLR 2022 Submitted (2022)
Keywords: representation learning
structural bioinformatics
proteins

Abstract: Learning from 3D protein structures has gained a lot of attention in the fields of protein modeling and structural bioinformatics. Unfortunately, the number of available structures is orders of magnitude lower than the number of available protein sequences. Moreover, this number is reduced even more when only annotated protein structures are considered. This makes the training of existing models difficult and prone to overfitting. To address this limitation, we introduce a new representation learning framework for 3D protein structures. Our framework uses unsupervised contrastive learning to learn meaningful representations of protein structures making use of annotated and un-annotated proteins from the Protein Data Bank. We show how these representations can be used to directly solve different tasks in the field of structural bioinformatics, such as protein function and protein structural similarity prediction. Moreover, we show how fine-tuned networks, pre-trained with our algorithm, lead to significantly improved task performance.

Under review as a conference paper at ICLR 2022
CONTRASTIVE REPRESENTATION LEARNING FOR 3D PROTEIN STRUCTURES
Anonymous authors Paper under double-blind review
ABSTRACT
Learning from 3D protein structures has gained a lot of attention in the fields of protein modeling and structural bioinformatics. Unfortunately, the number of available structures is orders of magnitude lower than the dataset sizes com- monly used in computer vision and machine learning. Moreover, this number is reduced even further, when only annotated protein structures can be considered, which makes the training of existing models difficult and prone to overfitting. To address this challenge, we introduce a new representation learning framework for 3D protein structures. Our framework uses unsupervised contrastive learning to learn meaningful representations of protein structures, making use of annotated and unlabeled proteins from the Protein Data Bank. We show, how these representations can be used in the field of structural bioinformatics to directly solve different protein tasks, such as protein function and structural similarity prediction. Moreover, we show how fine-tuned networks, pre-trained with our algorithm, lead to significantly improved task performance.
INTRODUCTION
In recent years, learning on 3D protein structures has gained a lot of attention in the fields of protein modeling and structural bioinformatics. These neural network architectures process the 3D position of the atoms and/or amino acids in 3D space in order to make predictions of unprecedented performance, in tasks ranging from protein design (Ingraham et al., 2019;; Sttooaach et al., [020 Jing et al. (2021). over protein structure classification (Hermosilla et al., 202T), protein quality assessment (Baldassarre) 2020 Derevyanko et al. 2018), and protein function prediction (Amidi et al., 2017 Gligorjevicc
et al. ct al. 2021) just to name a few. Unfortunately, learning on the structure of proteins leads to a reduced amount of training data, compared for example to sequence learning. since 3D structures are harder to obtain and thus less prevalent. While the Protein Data Bank (PDB) (Berman et al. 2000) today contains only around 182 K macromolecular structures, the Pfam database (Mistry al. 2020 contains 47 M protein sequences. Naturally, the number of available structures decreases even further when only the structures labeled with specific property are considered. We refer to these as annotated protein structures. The SIFTS database, for example, contains around 220 K annotated enzymes from 96 K different PDB entries, and the SCOPe database contains 226 K annotated structures. These numbers are orders of magnitude lower than the data set sizes which led to the major breakthroughs in the field of deep learning. ImageNet (Russakovsky et al. 2015). for instance, contains more than 10 M annotated images. As learning on 3D protein structures cannot benefit from these large amounts of data, model sizes are limited or overfitting might occur.
In order, to take advantage of unlabeled data, researchers have, over the years, designed different algorithms, that are able to learn meaningful representations from such data without labels (Hadsell et al. (0066] Ye et al. 2019 Chen et al. 2020a). In natural language processing, next token prediction or random token masking are commonly used unsupervised training objectives, that are able to learn meaningful word representations useful for different downstream tasks (Peters et al. 2018 Devlin et al. 2019). Recently, such algorithms have been used to learn meaningful protein representations from unlabeled sequences (Alley et al., 2019), or as a pre- pre-trained method for later fine-tuning models on different downstream tasks (Rao et al.) 2019). In computer vision recently, contrastive learning has shown great performance on image classification when used to pre-train deep convolutional neural network (CNN) architectures (Chen et al. 2020a b). This pre-training objective has also been

---

Under review as a conference paper at ICLR 2022
used in the context of protein sequence representation learning by dividing sequences in amino acid "patches' (Lu et al. 2020b), or by using data augmentation techniques based on protein evolutionary information (Lu et al. (020a). Most recently, the contrastive learning framework has been applied to graph convolutional neural networks (You et al. 2020). These techniques were tested on protein spatial neighhoring graphs (graphs where edges connect neighbor amino acids in 3D space) for the binary task of classifying a protein as enzyme or not. However, these algorithms were designed for arbitrary graphs and did not take into account the underlying structure of proteins.
In this work, we introduce a contrastive learning framework for representation learning of 3D protein structures. For each unlabeled protein chain, we select random molecular sub-structures during training. We then minimize the cosine distance between the learned representations of the sub-structures sampled from the same protein, while maximizing the cosine distance between representations from different protein chains. This training objective enables us, to pre-train models on all available annotated, but more importantly also unlabeled, protein structures from the PDB. The obtained representation can later be used weight initialization strategy improve performance on different downstream tasks, such as protein structure classification and function prediction. Moreover, show how the learned protein representation is able to capture protein structural similarity and functionality, by embedding proteins from the same fold or with similar functions close together in this space.
The remainder of this paper is structured as follows. First, we provide a summary of the state-of-the- art in Section 2 Then, we introduce our framework in Section[]aate in Section 4 we describe the experiments conducted to evaluate our framework and the representations learned, and lastly, we provide a summary of our findings and possible lines of future research in Section
2 RELATED WORK
3D protein structure learning. Early work on learning from 3D protein structures used graph kernels and support vector machines to classify enzymes (Borgwardt et al. (005). Later, the advances in the fields of machine learning and computer vision brought new set of techniques to the field. Several authors represent the protein tertiary structure as a 3D density map, and process it with 3D convolutional neural network (3DCNN). Among the problems addressed with this technique, are protein quality assessment (Derevyanko et al.] (218)) protein enzyme
classification (Amidi al. (017). protein-ligand binding affinity (Ragoza et al. [2017), protein binding site prediction (Jiménez et al. (017)) and protein-protede interaction interface prediction (Townshend al. 2019). Other authors have used graph convolutional neural networks (GCNN) to learn directly from the protein spatial neighboring graph. Some of the tasks solved with these techniques, are
protein interface prediction (Fout et al. (017). function prediction (Gligorijevic et al. [202T), protein quality assessment (Baldassarre et al. 2020). and protein design (Strokach et al.) 2020). Recently, several neural network architectures, specifically designed for protein structures, have been proposed to tackle protein design challenges (Ingraham et al. 2019: Jing et al., 2021. or protein fold and function prediction (Hermosilla et al. 2021).
Protein representation learning. Protein representation learning based on protein sequences is an active area of research. Early works used similar techniques as the ones used in natural language processing to compute embeddings of groups of neighboring amino acids in sequence (Asgari & Mofrad 2015). Recently, other works have used unsupervised learning algorithms from natural language processing such as token masking or next token prediction (Peters et al.) 2018; Devlin
2019) to learn representations from protein sequences (Alley et al. 2019 Rao et al.) 2019 Min ct al. al. 2020 Strodthoff et al. 2020). This year, Lu et al. (2020b a) have suggested using contrastive learning on protein sequences, to obtain meaningful protein representation. Despite the advances in representation learning for protein sequences, representation learning for 3D protein structures mostly has relied on hand-crafted features. La et al. (2009) proposed a method to compute vector of 3D Zemiie descriptors to represent protein surfaces, which later can be used for shape retrieval. Recently, Guzenko et al. 2020) used a similar approach, to compute vector of 3D Zernike descriptors directly from the 3D density volume, which can be used later for protein shape comparison. The annual shape retrieval contest (SHREC) usually contains a protein shape retrieval track, in which methods are required to determine protein similarity from different protein surfaces (Langenfeld et al. 2011)] 2020). Some of the works presented here, make use of 3DCNNs or GCNNs to achieve this goal.
2

---

Under review as a conference paper at ICLR 2022
However, they operate on protein surfaces, and are either trained in supervised fashion the binary shape similarity task, or pre-trained on a classification task.
Contrastive learning. In 1992. Beeker & Hinton (9922) suggested training neural networks through the agreement between representations of the same image under different transformations. Later, Hadsell al. 2006) proposed to learn image representations by minimizing the distance between positive pairs and maximizing the distance between negative pairs (see Figure This idea was used in other works by sampling negative pairs from the mini-batches used during training (Ji et al. 2019) Ye et al. 2019). Recently, Chen et al. 22020a b) have shown how these methods can improve image classification performance. You al. (2020) have transferred these ideas to graphs, by proposing four different data transformations to be used during training: node dropping, edge perturbation, attribute masking, and subgraph sampling. These ideas were tested on the commonly used graph benchmark PROTEINS (Borgwardt et al. 2005), composed of only 1, 113 proteins. However, since this data set composed of spatial neighboring graphs of secondary structures, the proposed data augmentation techniques can generate graphs of unconnected chain sections. In this paper instead, we suggest using a domain- speciffc transformation strategy, that preserves the local information of protein sequences.
3 3D PROTEIN CONTRASTIVE LEARNING
3.1 PROTEIN GRAPH
In this work, the protein chain defined as graph 9 = RN, where each node represents the alpha carbon of an amino acid with its 3D coordinates, . E R"x3, being n the number of amino acids in the protein. Moreover, for each node, we store local frame composed of three orthonormal vectors describing the orientation of the amino acid wrt. the protein backbone, RR E nxx33x3, Lastly, each node has also different associated features with F E Ruxt The connectivity of the graph is stored in two different adjacency matrices, A E g and B _a Matrix A is defined as Aij = if amino acids and j are connected by a peptide bond and A, = 0 otherwise. Matrix is defined as Bij = 1 if amino acids and are at distance smaller than d in 3D space and B,j 0 otherwise
3.2 CONTRASTIVE LEARNING FRAMEWORK
Inspired by recent works in the computer vi- sion domain Ye et al. 20199; Ji et al. 2019: Chen et al. 2020a), our framework is trained by maximizing the similarity between repre- sentations from sub-structures of the same pro- tein, and minimizing the similarity between sub-structures from different proteins. More formally, given protein graph G, we sample two sub-structures G; and Gj from it. We then compute the latent representations of these sub- structures, h; and hy, using protein graph en- coder, h E(G.). Based on the findings of Chen et al. 2020a), we further project these la- tent representations into smaller latent represen- tation, 24 and 2j. using multilayer perceptron (MLP) with single hidden layer, 2, = P(h,). Lastly, the similarity between these representa- tions is computed using the cosine distance, s(2,,2)) ). In order to minimize the similarity between these representations, use the following loss function for the sub-structure G;:
Sampling
E
P
o
E
P
Figure 1: For each protein we sample random sub- structures which then encoded into two repre- sentations, and using encoders E' and P. Then, we minimize the distance between representations from the same protein and maximize the distance between representations from different proteins.
exp(s(z,, 2/)/T)
4 = -log
(1)
2N Σ k=
[epp 2x)/T)
where is temperature parameter used to improve learning from 'hard' examples, [[yi,,y)) E [0. function that evaluates to if & * and j. and N is the number of protein structures in the
3

---

Under review as a conference paper at ICLR 2022
current mrini-batch. To compute I we again Equation| 1 but exchange the role of and j. This loss has been used before in the context of representation learning (Chen et al. 2020a van den Oord et al. 2018), and as in previous work, framework does not explicitly sample negative examples but uses instead the rest of sub-structures sampled from different proteins the mini-batch as negative examples. In the following subsections, will describe the different components specific to framework designed to process protein structures,
3.3 SUB- SUB-STRUCTURE SAMPLING
As Csen et al. (2020a) demonstrated, the data transformation applied to the input, is of key importance to obtain a meaningful representation. Among the different transformations tested, the authors found out that image cropping and adding random noise to the pixel values were the transformations that resulted in more informative representations. In this work, we propose to use a Nomain-speciffc cropping strategy of the input data.
Proteins chains are composed of one or several stable aub-structures, called protein domains, which reoceur in different proteins. These sub- structures can indicate evolutionary history be- tween different proteins, as well as the func- tion carried out by the protein (Ponting Rus- sell 2002). Our sampling strategy uses the con- cept of protein sub-structures to sample for each protein two different continuous sub-structts structurees along the polypeptide chain. We achieve that, by first selecting random amino acid in the protein chain 2y E N. We then travel along the protein sequence in both directions using the adjacency matrix A while selecting each amino acid Jitt and Zi-ç in the process. This process continues until we have covered a certain percentage p of the protein chain, whereby our experiments indicate that value of p between 40% and 60% provides the best results (see Section 44.. If during this sampling we reach the end the sequence in one the directions, we continue sampling in the other direction until we have covered the desired percentage The selected amino acids compose the sub-structure that is then given as input to the graph encoder E. Figure 2 iillustrates this process. Note that, since our framework learns from unlabeled data, we do not sample specifically protein domains from the protein chain, which would require annotations. We instead sample random sub-structures that might be composed of a complete or partial domain, or, in large proteins, even span several domains. The training objective then enforces a similar representation for random sub-structures of the same protein chain, where the properties of the complete structure have to be inferred. We will show in our experiments, that these properties are able to encode similar structural classifications as the ones developed over the years by researchers, well information related to the protein function.
(a)
(b)
Our sampling strategy used during con- trastive learning. For protein chain (a), we select random amino acid (b). Then we travel along the chain in both directions until we have a certain percentage p of the sequence covered (c).
Figure 2:
3.4 PROTEIN ENCODER
The information captured by learned representation using contrastive learning strongly depends on the network architecture used to encode the input (Tschannen et al. 2020). Therefore, we define the following requirements for our protein encoder. The memory footprint of the network has to be minimal, to allow for large batch sizes (Chen et al., 2020a), all the input data has to be computed on-demand process our random sub-structures, and the operations must be able to completely differentiate protein structures. We base our protein encoder on the architecture proposed by Hermosilla al. (202T). whereby we perform several modifications fulfill our requirements. In the following paragraphs, we describe the proposed network architecture.
Convolution operation. Similar to CNNs and GCNNs where a message passing operation used to update the features of node, based on the information of the neighboring nodes, Hermosilla et al.) (022)) proposed a message- message-passing operator for 3D protein structures. This operator computed the value of new feature in node xú, by aggregating the features from all neighboring nodes 2y at a distance smaller than d in 3D space. The features from neighboring nodes were first scaled by a learned function represented by ML. with one hidden layer that received as input the edge features
4

---

Under review as a conference paper at ICLR 2022
from node pairs 2, resulting in the following operation:
E ,,
(2)
JEN(
where N (2xe) is the list of nodes at a distance smaller than d from Tú, ho is the learned kernel function, and [(, stg) is the function that computes the edge information between node x; and wij.
In the work of Hermosilla et al. (2021), ) the function f computed three distances between the two nodes: the Euclidean distance, the shortest path along covalent bonds, and the shortest path along covalent and hydrogen bonds. In this work, we use Equation ass our pressage-passingg algorithm but our function f computes instead a different set of edge features, similar the ones used by Ingraham et al. (2019)
Three values representing the vector 2 Z; described in the local frame of node T₁. O₂ R. and normalized by distance d.
t:
r: Three values representing the orientation information between the local frames O, and O,. We use the dot product between the three axes instead of other representations such as quaternions (In- |graham et al. 2019) or 6D representations (Zhou et al. 2019) since improved performance in our experiments (see Section 4).
d: One value representing the shortest path along peptide bond between the two nodes 2; and Tj. normalized by 8maz-
While the features computed by Hermosilla et al. (2021) provided good results in their experiments, they were rotational invariant and they were not able to differentiate chiral protein sub-structures. By incorporating information of direction, f, and orientation, r, in our kernel, the operations become rotation equivariant and provide the network with the tools to differentiate such structures. We denote the combination of these two modifications as Rot. Eq. in our experiments. Moreover, Hermosilla et al. (202T) also considered the shortest path along covalent and hydrogen bonds. Calculating such distance on demand can be computationally demanding, which is limiting factor in our setup, since we sample random sub-structures in each training step. Moreover, the resulting connectivity can depend on the method used compute the hydrogen bonds, and it can vary for highly disordered structures (Zhang & Sagui] 2015). Therefore, we do not include such bonds and let the network detect the necessary secondary structures directly from the protein structure.
The seven edge features computed by f rf, and ō) all have values in the range [-1, Similar to positional encoding (Vaswani et al. 2017). we further augment these inputs by applying the function g = y=1-2 2, x1. which makes all features contribute the final value of the kernel k,, even when their values are equal to zero. This feature augmentation results in 14 final input values to ko, the original 7 edges features, plus the transformed ones. We refer to those features as Add. Input in our experiments. Lastly, we weigh the final value of k. by a function o to remove discontinuities at distance d, where small displacement of neighboring node x, can make exit the receptive field. Similar to the cutoff function proposed by KKlicpera et al. (2020), the function smoothly decreases from one to zero at the edge of the receptive field, making the contributions of neighboring nodes disappear as they approach d. Our function is defined as (1 - tanh(d; 16 14))/2, where d; is the distance of the neighboring node normalized by d. We refer to this function as Smooth in our experiments.
Network architecture. The network architecture proposed by Hermosilla et al. (2021) processed each protein at the atomic level, and computed features per atom before they were pooled to amino acids. However, processing all atoms of protein significantly increases computation time and memory footprint. Therefore, in this work, we represent each amino acid by its alpha carbon instead. We use set ResNet bottleneck blocks (He et al. 2016) and pooling operations as as Hermosilla al. (2021) to compute 2048 features for each protein structure. A detailed description of the protein encoder provided in the supplementary material.
4 EXPERIMENTS
In this section, we will describe the experiments conducted to evaluate our methods, and demonstrate the resulting learned representations.
5

---

Under review as a conference paper at ICLR 2022
ECB
Figure 3: Dimensionality reduction of the protein representations using TSNE [Van der Maaten &
Hinton 2008). Left: Proteins from the Fold Classification task, training set on the left, test set on the right, color-coded based on the highest hierarchy level in the SCOPe classification system. Right: Proteins from the Enzyme Classification task, training on the left, test set on the right, color-coded based on the highest level of the Enzyme Commission number.
4.1 DATA SETS AND TASKS
Our main data set used for unsupervised learning is based on the PDB (Berman et al. 2000). We collected 476, 362 different protein chains, each composed of at least 25 of amino acids. This set of protein chains was later reduced for each task, removing chains from the set based on the available annotations. All networks were trained for 550 K training steps, resulting in 6 days of training.
Fold Classification. This data set first proposed by Hou et al. (2018) which consolidated 16, 712 proteins of 1, 195 different folds from the SCOPe 1.75 database (Murzin et al. 1955). Directly from this source, obtained the 3D structures of these proteins. The data set provides three different test sets with increasing difficulty: Protein, in which proteins of the same family are present in both test and training set; Family, where proteins from the families in the test set are not seen during training; and Superfamily, in which proteins from the same superfamily as the proteins in the test set are not seen during training. Performance is measured with overall accuracy on these test sets. For pre-training, filtered the PDB data set and removed all annotated protein chains with the same folds as the proteins in the test sets. This procedure generated one PDB data set for each test set composed of proteins with and without annotations. These data sets contain 377, 271 chains for the Superfamily test set, 313, 616 chains for the Family test set, and 324, 304 chains for the Protein test set.
Enzyme Classification. This data set was presented by Hermosilla et al. (2021) and it contains 37, 428 proteins from 384 different EC numbers. The task consists of, given 3D structure of an enzyme, predict its complete EC number, e.g. 4.2.3.1, among the 384 numbers available in the data set. The proteins in the data set split into three sets, training. validation, and testing, whereby proteins in each set do not have more than 50% of sequence similarity with proteins from the other sets. Thus, obtain 29, 215 proteins for training, 2, 562 proteins for validation, and 5, 651 for testing. Performance is measured with overall accuracy on the test set. For more details, we refer the reader to Hermosilla al. (202I). For pre-training, we remove all proteins from the PDB data set which belong to the same EC number as the 384 used in the test set, where only the first two digits are considered, e.g. EC4.1. Thus, we obtain 270, 861 protein chains for pre-training with and without EC labels..
Protein Similarity. We used the benchmark proposed by Holm 20199. This data set is composed of 140 protein domains for which similar protein chains have to be found from a set of 15, 21 More- over, they provide another set composed 176, 022 protein chains that we use to train our distance metric. In this benchmark, different similarity levels are considered based on the SCOPe classification hierarchy, Fold, Superfamily, and Family. Performance is measured with Fiar (see Appendix DD) For pre-training, we filtered the PDB data set and removed all proteins that are annotated with the same Fold as the 140 protein domains. This resulted in 432, 884 protein chains with and without annotated Fold class.
4.2 QUALITATIVE EVALUATION
First, we evaluate the representation learned by mapping the high dimensional space to 2D repre- sentation using TSNE (Van der Maaten & Hinton| 2008). Then, we color-code each point based on the SCOPe and EC number classification schemes (see Figure

---

Under review as a conference paper at ICLR 2022
Table I: Performance comparison of the different variants of pre-trained networks the Fold and Enzyme classification task. I-NN makes predictions by searching the latent space for the nearest neighbor on the training data set, SVM indicates results of support vector machine model trained on the learned protein representations, MLP presents the results of an MLP trained using our representations, Fine-tuning indicates the performance of fine-tuned protein encoder, and No pre-train indicates the results of protein encoder trained from scratch.
FOLD
ENZYME
Super.
Fam.
Protein
Avg
HHSuite TMalign
17.5
69.2 % 65.7 %
98.6
61.8 %
82.6 %
34.0 %
97.5 %
65.7 %
Kipf Welling 2017
16.8 %
21.3 %
82.8 %
40.3 %
67.3 %
Diehl (2019)
12.9 %
16.3 %
72.5 %
33.9 %
57.9 %
Derevyanko al.
2018
31.6 %
45.4 %
92.5 %
56.5 %
78.8 %
Gligonjevic et al.
202T
15.3 %
20.6 %
73.2 %
36.4 %
63.3 %
Baldassarr et
2020
23.7
32.5 %
84.4 %
46.9 %
60.8 %
Hermosilla et al.
2021
45.0 %
69.7 %
98.9 %
71.2 %
87.2 %
Ours (No pre-train)
47.6 %
70.2 %
99.2 %
72.3 %
87.2 %
Ours (I-NN)
21.3
47.5 %
87.7 %
52.2 %
46.5 %
Ours (SVM)
32.3 %
49.2 %
94.0 %
58.5 %
53.5 %
Ours (MLP)
38.6 %
69.3 %
98.4 %
68.8 %
74.3 %
Ours (Fine-tuning)
50.3 %
80.6 %
99.7 %
76.9 %
87.6 %
For the Fold Classification task, we take the training and test Protein, and color code each data point based on their class according to the SCOPe classification hierarchy. Note, that the model did not see during training any of the folds of the proteins the test set. We can see, that our representation clusters points from the same class for classes a, b, d, and g. However, points from classes e and f are spread among the other classes.
Moreover, also use the same evaluation for the Enzyme Classification task. We color code each data point based on the first number from the EC number. do not seem to form unique cluster for each EC number, data points from small clusters in the embedding all belong to the same EC class. This might be an indication, that the network did not use the higher levels of the EC number classification scheme cluster data points, but groups proteins based on other properties that are captured beyond the first digit of the EC number.
showws that, even the data points
4.3 QUANTITATIVE EVALUATION
In order to quantitatively evaluate the learned representation, we evaluate how the high dimensional space captures human-designed classifications developed over the years. As it is common practice we use the performance on downstream tasks, to measure the quality of the learned representation.
Distance-based classification. For the tasks of Fold and Enzyme Classification, we compute the latent rep- resentation h for the training and test sets. We then use the Euclidean distance in the latent space to search for the closest protein in the train set for each protein in the test set. We count a protein classified correctly if the selected closest protein in the training set is from the same class as the queried protein. Table I presents the results of this experiment the row J-NN. We can see, that even if the network did not see proteins from those folds during training, it learns a representation that can classify such folds with higher accuracy, than most of the supervised trained methods. However, for the En- zyme classification task, where the network did not
Table 2: Comparison of our distance metric
based on the learned representations for the Protein Similarity benchmark.
Fold Super.
Fam.
DaliLite DeepAlign mT'MaLign
0.38
0.83
0.96
0.28
0.78
0.97
0.13
0.55
0.91
TMaLign
0.12
0.39
0.85
Ours (Euclidean)
0.14
0.39
0.65
Ours (Cosine)
0.12
0.39
0.65
Ours (Cos. Learn)
0.35
0.55
0.63
7

---

Under review as a conference paper at ICLR 2022
enzymes of the same subelass during pre-training, the performance is lower than other methods but still competitive.
Moreover, we evaluate the learned representation on the Protein Similarity task. Here, we use the Euclidean distance between representations h and cosine distance between representations as a measure of protein similarity. We can see in Table 2 that, even if our method is not able to outperform state of the art methods such as DaliLite (Holm) 2019) and DeepAlign (Wang al. 2013). it achieves higher performance in the Fold test set than TMAlign (Zhang & Skolnick] 2005) and mTMAlign (Dong et al., 2018). which are estabbisbd methods to measure protein similarity. For the Superfamily test set it achieves the same performance as TMAlign, but worse than other methods, while in the Family test set achieves the lowest performance.
Using distance metric with our learned representations might not be optimal since we do not enforce them to represent a metric space during training. Therefore, we trained an MLP to transform our representation h into new representation z which minimizes the cosine distance between proteins of the same class and maximizes it if they are from different class. We learned three different representations 2 to determine if proteins are from the same Fold, Superfamily, or Family. Table presents the results on such experiment. We can see that this learned distance metric achieves the second-bes Fmar for the Fold task, surpasing commonly used methods such as TMAlign (Zhang & Skolnick 2005), mTllgg (Dong et al. 2018) and DeepAlign Wang et al. 2013). For the Super- family task achieves competitive performance compared to m TTMAlign (Dong et al. 2018). whilst for the Family tasks achieves the worst Fmaz. This might be an indication that our representation is able to capture the global structure of protein but struggles to capture fine details. When comparing timings for single query against the 15 K proteins, our method only takes a few seconds for the query, it just performs the subtraction/dot product between the representations, plus around four minutes for loading and encoding of the 15 K proteins. In contrast, DaliLite (Holm) 2019) requires around 15 hours and TMAlign (Zhang & Skolnick 2005) a bit less than one hour, on computer equipped with six cores.
Classifier. We also measure the quality of the representation, by training different classifiers using such protein representation as input and comparing to other learned and non-learned based baselines. Note that some of these baselines were designed for different tasks. However, this comparison allows positioning our learned representation into context with other available protein encoders. First, we train a support vector machine model (SVM) on our representation for the tasks of Fold and Enzyme classification, row SVM in Table [ As expected, we achieve higher performance than the Euclidean distance method. We also trained an MLP with a single hidden layer on these representations, row MLP in Table D This method performed better than the SVM in both tasks, and achieved similar performance to the accuracy obtained by training the protein encoder from scratch on the Fold Classification task, row No pre-train in Table T Furthermore, this classifier outperforms most of the other methods, including the commonly used T'MALign method (Zhang & Skolnick] 2005) on the fold classification task, where the TMAlign similarity is used to find for each protein the test set the most similar protein in the training set.
Fine tuning. Lastly, we measure if our pre-trained encoder can be used as a weight initialization scheme in fine-tuning setup. Table [ Presents the results of this experiment in row Fine tuning. We can see, that this setup achieves the highest accuracy in Table surpassing a randomly initialized protein encoder and obtaining new state-of-the-arr results on both data sets.
4.4 ABLATION STUDIES
In this section, we evaluate the pre-training procedure. First, we analyze how the amount of information removed from the sequence affects the learned representation. We measure it by training an MLP to classify the protein representations according to the Fold classification task. For pre- training, we use the entire PDB data set, and train the protein encoder for 180 K training steps, which results in two days of computation. From the results in Table 20% and 40% of the protein chain makes the contrastive objective too easy, and the protein encoder does not learn rich enough representation. On the other hand, removing between 60% and 80%
see that removing between
8

---

Under review as a conference paper at ICLR 2022
Table 3: Ablations the data transforma- tions used during pre-training for the Fold Classification task.
Table 4: Ablations the data transforma- tions used on the supervised setting for the Fold Classification task.
Transf.
Super.
Fam.
Protein
Avg
Transf.
Super.
Fam.
Protein
Avg
60%-80%
35.2 %
67.2 %
98.0 %
66.8 %
60%-80% 9.9 %
23,8 %
62.7 %
32.1 1 %
40%-60%
38.9 %
70.1 %
98.8 %
69.3 %
40%-60%
26.7 %
48.3 %
91.9 %
55.6 %
20%-40%
32.5 %
57.0%
97.6 %
62.4 %
20%-40%
38.9 %
63.2 %
98.0 %
66.7 %
Sub-graph
42.1 %
65.9 %
98.7 %
68.9 %
Noise
47.6 %
70.2 %
99.2 %
72.3 %
does not preserve enough information, and the performance also suffers. As can be seen, we found that removing between 40% and 60% of the protein chains achieves the best performance.
We further compare our suggested data transformation approach to the graph augmentation technique used by You et al. (2020). You et al. (2200) selected a random sub-graph on the spatial neighboring graph, thus selecting random area in 3D space. We can see in Table that while You et al. (2020) obtains good performance on the Superfamily test set, it obtains lower accuracy on the Family and Protein test set. Since the Family and Protein test sets contain proteins with higher sequence similarity to the training set than the Superfamily test set, we hypothesize that our method uses more information of the protein sequence for the representation than the sub-graph method, since the latest sees disconnected sections of the chains during training, while our method always sees a connected sub-chain. Although we acknowledge that both methods can be beneficial for different tasks, we observed that on average our method provides better performance.
Lastly, we evaluate how the proposed data transformation could affect the supervised training of the protein encoder on the Fold Classification task. We can see in Table 4 that in contrast to the unsupervised training, our data transformation technique used data augmentation reduces performance in this setup. Instead, the best accuracy is obtained by adding a small random Gaussian noise into the 3D coordinates of the alpha carbon. These results align with the obtained other contrastive learning works (Chen et al.) (2020a), where these extreme data transformation strategies hurt the supervised training instead of improving its performance.
CONCLUSIONS
In this paper, we have introduced contrastive learning for protein structures. While learning on protein structures has shown remarkable results, suffers from rather low availability of annotated data sets, which strengthens the need for unsupervised learning technologies. In this paper, we demonstrated, that by combining protein-aware data transformations with adequately adapted, state- of-the-art learning technologies, we were able to obtain a learned representation without the need for such annotated data. This is highly beneficial, since the availability of annotated 3D structures is limited, compared to sequence data. Using learned representation, protein tasks, that we achieve competitive performance when the distance between representations is used metric of protein similarity.
could show on relevant
We believe that our work is a first important step in transferring unsupervised learning methods to large-scale protein structure databases. In the future, foresee, that the learned representation can not only be used, to solve the tasks demonstrated the paper, but that it can also be helpful, to solve other protein structure problems. Protein-protein interaction prediction, for example, could be addressed using our proposed framework by finding the matching geometric patterns in the learned representations, Additionally, upon acceptance, plan to release the representations for all PDB proteins, and make our technology available, such that these representations can be updated with newly discovered proteins.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Ethan C. Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M. Church. Unified rational protein engineering with sequence-baser deep representation learning. Nature Methods, 2019.
A. Amidi, S. Amidi, D. Vlachakis, V. Megalooikonomou, N. Paragios, and E. Zacharaki. En- zyNet: enzyme classification using 3D convolutional neural networks spatial representation. 1rXiv:1707.06017, 2017.
E. Asgari and M.R.K. Mofrad. Continuous distributed representation of biological sequences for deep proteomics and genomies. PLoS ONE, 2015.
F. Baldassarre, D. M. Hurtado, A. Elofsson, and H. Azizpour. GraphQA: Protein Model Quality Assessment using Graph Convolutional Networks. Bioinformatics, 2020.
Suzanna Becker and Geoffrey E. Hinton. Self-organizing neural network that discovers surfaces in random-dot stereograms. Nature, 1992.
Helen M. Berman, John Westbrook, Zukang Feng, Gary Gilliland, T. N. Bhat, Helge Weissig, Ilya N. Shindyalov, and Philip E. Bourne. The Protein Data Bank. Nucleic Acids Research, 2000.
Karsten M. Borgwardt, Cheng Soon Ong. Stefan Schönauer, S. V. N. Vishwanathan, Alex J. Smola, and Hans-Peter Kriegel. Protein function prediction via graph kernels. Bioinformaticx, 2005.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05// 05709, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006.10029, 2020b.
Georgy Derevyanko, Sergei Grudinin, Yoshua Bengio, and Guillaume Lamoureux. Deep convolu- tional networks for quality assessment of protein folds. Bioinformatics, 34(23):4046-53, 2018.
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019.
Frederik Diehl. Edge contraction pooling for graph neural networks. arxiv: 1905. 10990, 2019.
Runze Dong, Shuo Pan, Zhenling Peng, Yang Zhang, and Jianyi Yang. mtm-align: a server for fast protein structure database search and multiple protein structure alignment. Nucleic acids research, 2018.
A. Fout, J. Byrd, B. Shariat, and A. Ben-Hur. Protein interface prediction using graph convolutional networks. In Advances in Neural Information Processing Systems 30. 2017.
V. Gligorijevic, P. D. Renfrew, T. Kosciolek, J. K. Leman, K. Cho, T. Vatanen, D. Berenberg, B. Taylor, M. Fisk, R. J. Xavier, R. Knight, and R. Bonneau. Structure-based function prediction using graph convolutional networks. Nature Communications, 2021.
Dmytro Guzenko, Stephen K. Burley, and Jose M. Duarte. Real time structural search of the protein data bank. PLOS Computational Biology, 2020.
R. Hadsell, Chopra, and Y. LeCun. Dimensionality reduction by learning an invariant mapping. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 20/6 IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2016.
Pedro Hermosilla, Marco Schäfer, Matëj Lang, Gloria Fackelmann, Pere Pau Vázquez, Barbora Kozliková, Michael Krone, Tobias Ritschel, and Timo Ropinski. Intrinsic-extrinscc convolu- tion and pooling for learning on 3d protein structures. International Conference on Learning Representations, 2021.
10

---

Under review as a conference paper at ICLR 2022
Liisa Holm. Benchmarking fold detection by DaliLite v.5. Bioinformaties, 2019.
Hou, B. Adhikari, and Cheng. Deepsf: Deep convolutional neural network for mapping protein
sequences to folds. In Proceedings of the 2018 ACM International Conference Bioinformatics, Computational Biology, and Health Informatics, 2018
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. In NeurIPS, pp. 15820-15831, 2019.
Xu Ji, João F. Henriques, and Andrea Vedaldi. Invariant information distillation for unsupervised image segmentation and clustering. International Conference on Computer Vision (ICCV), 2019.
J Jiménez, S Doemr G Martínez- and A S Rose. DeepSite: protein- protein-binding site predictor using 3D-convolutional neural networks. Bioinformatics, 2017.
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations, 2021.
Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. ICML,, 2017
Johannes Klicpera, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. Infernation Conference Learning Representations, 2020.
David La, Juan Sxquivel-Rodriguee Vishwesh Venkatraman, Bin Li, Lee Sael, Stephen Ueng. Steven Ahrendt, and Daisuke Kihara. 3D-SURFER: software for high-throughput protein surface comparison and analysis. Bioinformatics, 2009.
Florent Langenfeld, Apostolos Axenopoulos, Halim Benhabiles, Petros Daras, Andrea Giachetti, Xusi Han, Karim Hammoudi, Daisuke Kihara, Tuan M. Lai, Haiguang Liu, Mahmoud Melkemi, Stelios K. Mylonas, Genki Terashi. Yufan Wang, Feryal Windal, and Matthieu Montes. Protein Shape Retrieval Contest. In Eurographics Workshop on 3D Object Retrieval, 2019.
Florent Langenfeld. Yuxu Peng, Yu Kun Lai, Paul Rosin, Tunde Aderinwale, Genki Terashi. Charles Christoffer, Daisuke Kihara, Halim Benhabiles, Karim Hammoudi, Adnane Cabani, Feryal Windal, Mahmoud Melkemi, Andrea Giachetti, Stelios Mylonas, Apostolos Axenopoulos, Petros Daras, Ekpo Otu, Reyer Zwiggelaar, and David Hunter. Multi-domain protein shape retrieval challenge. In Silvia Biasotti, Guillaume Lavoué, and Remco Veltkamp (eds.), Eurographics Workshop 3D Object Retrieval, 2020.
A. X. Lu, A. X. Lu, and A. Moses. Evolution is all you need: Phylogenetic augmentation for contrastive learning. 15th Machine Learning in Computational Biology (MLCB), 2020a.
A. X. Lu, H. Zhang, M. Ghassemi, and A. Moses. A self-supervised contrastive learning of protein representations by mutual information maximization. 15th Machine Learning in Computational Biology (MLCB), 2020b.
Seonwoo Min, Seunghyun Park, Siwon Kim, Hyun-Soo Choi, and Sungroh Yoon. Pre-training of deep bidirectional protein sequence representations with structural information. Bioinformatics, 2020.
Jaina Mistry, Sara Chuguransky, Lowri Williams, Matloob Qureshi, Gustavo A Salazar, Erik Sonnhammer, Silvio C E Tosatto, Lisanna Paladin, Shriya Raj. Lorna J Richardson, Robert D Finn, and Alex Bateman. Pfam: The protein families database in 2021. Nucleic Acids Research, 2020.
A.G. Murzin, S.E. Brenner, T. Hubbard, and C. Chothia. SCOP: database for the investigation sequences and structures. J Molecular Biology, 1955.
structural classification of proteins
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. Annual Conference of the North American Chapter the Association for Computational Linguisties (NAACL), 2018

---

Under review as a conference paper at ICLR 2022
Chris P. Ponting and Robert R. Russell. The natural history of protein domains. Annual Review of Biophysics and Biomolecular Structure, 2002.
Matthew Ragoza, Joshua Hochuli, Elisa Idrobo, Jocelyn Sunseri, and David Ryan Koes. Pro- tein-ligand scoring with convolutional neural networks. Chemical Information and Modeling, 2017.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John Canny, Pieter Abbeel, and Yun S. Song. Evaluating protein transfer learning with tape. Advances Neural Information Processing Systems (NeurIPS), 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015.
Nils Strodthoff, Patrick Wagner, Markus Wenzel, and Wojciech Samek. Udsmprot: universal deep sequence models for protein classification, Bioinformatics, 2020.
Alexey Strokach, David Becerra, Carles Corbi- Corbi-Verge, Albert Perez-Riba, and Philip M. Kim. Fast and flexible protein design using deep graph neural networks. Cell Systems, 2020.
Raphael Townshend, Rishi Bedi, Patricia Suriana, and Ron Dror. End-to-end learning 3D protein structure for interface prediction. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Michael Tschannen, Josip Djolonga, Paul K. Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. International Conference on Learning Representations, 2020.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. 2018.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 2008
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
Sheng Wang, Jianzhu Ma, Jian Peng. and Jinbo Xu. Protein structure alignment beyond spatial proximity, Scientific reports, 2013.
Mang Ye, Xu Zhang, Pong C. Yuen, and Shih-Fu Chang. Unsupervised embedding learning via invariant and spreading instance feature. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in Neural Information Processing Systems, 2020.
Yang Zhang and Jeffrey Skolnick. Tm-align: protein structure alignment algorithm based on the tm-score. Nucleic acids research, 2005.
Yuan Zhang and Celeste Sagui. Secondary structure assignment for conformationally irregular peptides: Comparison between dssp, stride and kaksi. Journal of Molecular Graphics and Modelling, 2015.
Yi Zhou, Connelly Barnes, Lu Jingwan, Yang Jimei, and Li Hao. On the continuity of rotation representations in neural networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
12

---

Under review as a conference paper at ICLR 2022
Table 5:
Ablations the edge feature ele-
ments on the Fold Classification task.
Table 6: Ablations of the rotation representa- tion used on the Fold Classification task.
Super.
Fam.
Protein
Avg
Super.
Fam.
Protein
Avg
Baseline
39.0% %
65.6 %
98.6 %
67.7 %
Quat.
44.0 %
68.7 %
98.0 %
70.2 %
Rot. Eq.
45.5 % 69.7 %
98.9% %
71.4% %
6D
46.1 %
68.2 %
98.7 %
71.0 %
Add. Input
44.6%
67.7% %
70.3 98.7%
%
Dot Axis
45.5 %
69.7 %
98.9 %
71.4% %
Smooth 40.7 %
65.4% %
98.4%
68.2% %
Full
47.6 %
70.2% %
99.2%
72.3 %
A ADDITIONAL ABLATION STUDIES
First, we evaluate the different extensions we incorporated into the convolution operation as proposed by Hermosilla et al.) (2021) (see Table 5). Our baseline method uses as edge features the Euclidean distance and the shortest path along the sequence. We evaluate the performance improvement when we substitute the Euclidean distance by direction and orientation information as described in Section 3.4 denoted as Rot. Eq. in the table. We also evaluate how transforming the original inputs similar to positional encoding affects the resulting performance, denoted as Add. Input in the table. Furthermore, we evaluate the effect of the smoothing function applied towards the boundary of the receptive field, denoted as Smooth in 3 We can see that adding the components individually to the baseline, results in an improvement of accuracy in all cases. Moreover, when we incorporate all together in our final convolution operation, we experience even higher improvement, Full in Table [
Lastly, we evaluated the performance of the model, when changing the representation of the orien- tation features. Here, we compare Quaternions (Ingraham et al. 2019), with the 6D representation introduced by Zhou et al. (2019), and the simple dot product between the axes of the two frames. Results of this experiment while the 6D and the dot product obtain similar performance. Although the dot product is not able to represent full rotation, it obtained a slightly higher performance than the 6D and faster convergence during training. We hypothesize, that even the 6D representation is more descriptive, it uses more floats than the dot product method wrt. the rest of the inputs to the kernel.
shown in Table The worse performance is obtained by Quaternions,
B NETWORK ARCHITECTURE
Our neural network receives input the list of amino acids of the protein. Each protein then simplified several times with a pooling operation that reduces the number of amino acids by half each step. We use the same pooling operation proposed by Hermosilla al. (2021) where every two consecutive amino acids grouped into new node. The initial features defined by an embedding of 16 features for each amino acid type that is optimized together with the network parameters. These initial features are then processed by two ResNet bottleneck blocks (He et al. 2016) and then pooled to the next simplified protein representation using average pooling. This process is repeated four times until we obtain a set of features for the last simplified protein graph. The number of features used for each level are [256, 512, 1024, 2048]. The radius of the receptive field, d, used to compute the adjacency matrix B in each level are 18, 12, 16, 20J Á. Lastly, in order to obtain set of features for the complete protein structure we use an order invariant operation that aggregates the features of all nodes. In particular, we use the average of the features of all nodes. Figure provides an illustration of the proposed architecture.
C TRAINING PARAMETERS
In this subsection, we will describe the hyperparameters used on the different training setups. All methods were trained desktop pc with six cores, 32 Gb of RAM, and GeForce GTX 2080.
13

---

Under review as a conference paper at ICLR 2022
Protein graph
Amina add embeeding
Rest x2 (0) 206)
Pooling
RestVer x2 (12 512)
Pooling
ResNet x2 (16, 1024)
Pooling
Resh x2 (20), 20488)
Protein Representation
ResNel Bollleneck
Irt
Pretein Cone
Cone Ne
Come
DV
Did
Protein graph simplifiction
Figure 4: Illustration of our protein encoder. We use an amino acid embedding as our input features that are then processed by consecutive ResNet Bottleneck blocks and pooling operations. To obtain the final protein representation we use the average of the features from the remaining graph nodes.
C.I UNSUPERVISED SETUP
To train our models with the contrastive learning objective we used a latent representation of size 2048 and projected representation of size 128. We use Stochastic Gradient Descent (SGD) optimizer with an initial learning rate of 0.3 which was decreased linearly until 0.0001 after a fourth of the total number of training steps. We whole architecture. Moreover, we used a weight decay factor of le
batch size of 256 and dropout rate of 0.2 for the
C.2 FOLD AND ENZYME CLASSIFICATION
the SGD optimizer with an initial learning rate of 0.001 which was decreased after 100 epochs to 0.0001 and after 300 epochs 0.00001. We train the networks for total of 400 epochs using a batch size of To avoid overfitting we use dropout rate of 0.2 on the encoder and 0.5 on the final MLP. Moreover, to further regularize the model, we use a weight decay factor of 5e We additional augment the data by applying Gaussian noise to the 3D coordinates with standard deviation of 0.05.
Supervised: To train the models the supervised task of Fold classification
the supervise method but we use instead smaller learning rate of 0.0005 which is decreased to 0.00005 after 300 epochs. Moreover, we use linear learning rate warm-up for the first 25 epochs.
Fine-tune:
For the fine-tuning setup we use the same parameters
D PERFORMANCE METRIC DALILITE
To evaluate the performance of different methods on the DaliLite benchmark we use Fmoz as defined by Holm (2019). We sort the 15 K proteins based on distance metric to our target and use the following definition of Fmazi
F Fmax max F(n)
max 2p(n)r(n) p(n) + r((n)
2TP(m)
max n
where is the rank of the query in the ordered list, i. the index of the protein in the sorted list. For the n first results in the ordered list, we define p(n) as the precision, r(n) as the recall, TP(n) as the number of true positives pairs, and T is the number of structures in the class. We compute the final value for the test set by averaging the Foaar among the 140 test protein domains. For more details on this metric, refer the reader to Holm 2019
14


Reviews:
[{"review_id": "uexdDyN31Zw", "reviewer": "Reviewer_Ayvu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Representation learning for 3D protein structures is a broadly relevant problem. The authors proposed using contrastive learning with sub-structure sampling as the data transformation. The empirical results on fold classification and enzyme classification show that the proposed contrastive learning method is able to meaningful representations that capture certain properties of protein structures. The paper is clear and includes detailed ablation studies.\n\nOne major concern is that the learned representation from the proposed approach might only capture \"global\" properties of protein structures and might not be able to capture finer-grain features of protein structures. This might be baked in from the choice of using subgraph sampling as the data transformation. For example, if two antibodies could be structurally very similar to each other but bind to completely different antigens, and it is likely that the learned representation here won't be able to distinguish two structurally similar antibodies. Extending the method to capture finer-grain features in protein 3D structures could enable more use cases for the learned representation.\n\nWhile learned distance metrics on the protein structure representation space is an important research direction, likely how the distance should be defined would depend on the downstream application. It would be valuable for the authors to clarify when the proposed distance metric in this paper would or would not be a good fit.\n\nWhile the evaluation tasks (enzyme classification and fold classification) are clearly defined, they are not particularly well motivated and it would strengthen the paper to further explain the biological/clinical significance of these tasks. \n\nThe conclusions towards the end of the paper are not well-supported by the empirical results. For example, it is unclear how such a representation would help with solving protein-protein interaction related tasks. More context would be appreciated.\n\nQuestion: How are variable lengths handled in contrastive learning? Could it be that the representation is implicitly using length to classify fold/EC while baselines such as hhsuite or TMalign cannot make use of the length information?\n", "summary_of_the_paper": "This paper studies unsupervised contrastive learning on protein structures, using sub-structure sampling as the data transformation strategy in contrastive learning. The protein structure representation from contrastive learning is then evaluated for three tasks: fold classification, enzyme classification, and protein similarity.\n\nThe main contributions are:\n1) Demonstrating that contrastive learning with sub-structure sampling is a viable strategy for learning protein structure representations.\n2) Improving empirical results for fold classification and enzyme classification when compared to existing methods.", "summary_of_the_review": "Interesting approach. One major concern is that the learned representation from the proposed approach might only capture \"global\" properties of protein structures and might not be able to capture finer-grain features of protein structures. Proposed approach shows improvements on fold and enzyme classification tasks, but these tasks may have only limited practical utility.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bHC9eIwUAFk", "reviewer": "Reviewer_xGdi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found the abstract misleading. It talks about the wider availability of proteins sequence information compared to structural information, leading me to think that the proposed method would learn from both.  But it doesn't: it learns only from 3D sturctural information.\n\nFurthermore, a key missing component of the story is the notion of \"annotations.\"  This concept is mentioned in the abstract (\"Moreover, this number is reduced even further, when only annotated protein structures can be considered\") but never clearly defined.  A protein can be annotated in many ways, including annotations along the amino acid chains (domain boundaries, protein or DNA binding sites, active sites in enzymes, secondary structural elements, etc.) or annotations of the entire protein sequence (GO terms, KEGG pathways, SCOP annotations).  My best guess is that the authors intend to use the term \"annotation\" to refer to assignment of protein domains to a protein sequence or structure. (\"Note that, since our framework learns from unlabeled data, we do not sample specifically protein domains from the protein chain, which would require annotations.\") I do not actually know how protein domains get annotated, but I think it's mostly by sequence similarity.  If this is the case, then it's ironic that the authors are trying to avoid using these annotations during training, since they can presumably be easily generated from 3D structures.\n\nOverall, I found this to be an overly complicated model to test what seems like a straightforward hypothesis (i.e., that this contrastive learning approach is beneficial).  Much of the model seems to derive from a previously published approach (Hermosilla 2021), but with not insignifcant modifications, particularly in the protein encoder.  This approach makes it hard to keep track of what primary claim or claims are being tested here.  In particular, I am not sure if the authors are simply trying to create a state-of-the-art fold recognition engine (in which case the results in Table 2 suggest that they have failed) or simply demonstrate that their contrastive learning approach is beneficial.\n\nThe paper needs to be more careful about defining terms before using them. For example, we are told on p. 4 what a protein domain is, but the text refers to protein domains on pp. 1 and 2.\n\nI could not understand what the \"Protein\" task consists of in the SCOP superfamily setup (\"... and Protein, in which proteins of the same family are present during training.\"): what are the test sequences?\n\nI realize that the enzyme classification task has been presented previously, but I still think it should be described more clearly. I do not understand what a query is in this case, nor whether the train/test split is stratified by class.\n\nI did not find the qualitative evaluations in Section 4.2 to be informative.\n", "summary_of_the_paper": "This paper focuses on the general problem of learning a compact embedding of a protein based on its 3D structure.  The key novelty here is to use a contrastive learning procedure: \"minimize the cosine distance between the learned representations of the sub-structures sampled from the same protein, while maximizing the cosine distance between representations from different protein chains.\"", "summary_of_the_review": "This is a complicated model or set of models, coupled with results that do not clearly support the central hypothesis of the paper, namely, that using contrastive learning outperforms other methods for learning informative embeddigs of protein structures.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YW5TJavGz7u", "reviewer": "Reviewer_ncDp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The work tries to address an important proteomics problem\n2. The writing is generally clear and succinct, but can be improved in some parts (see comments).\n3. The approach is interesting and seems like a novel application of existing contrastive learning approaches, adapted for proteins\n4. There is a reasonable analysis of the learned models\n\nWeaknesses:\n1. Their approach of using contrastive learning seems ill-fitted to learn precise representations of protein structures. They consider sub-chains of proteins as examples. But substructures of the same protein can look very different and have very different functionality? Does it make sense to minimize the cosine distance between them? for instance: an alpha helix and a beta sheet from the same protein can be quite different, so if the cosine distance between these is minimized, it is not clear what the model is learning\n2. Can they restrict the similarity constraints within a single domain from a protein, maybe?\n3. The evaluation is weak and is done on very coarse labels. For instance, there are currently 1549 different folds (classes) in SCOP and the paper evaluates on 7 high-level folds (classes) that are very different. Many computational papers show results on at least 27 folds (Ding and Dubchak, 2001) or a few hundred folds.\n4. Some of the baselines that they compare against were developed for totally different problems. For ex: Baldassarre was for Protein Quality assessment and is specifically tuned for that problem and seems ill-suited for fold classification?\n5. The paper's goal is to use unlabeled protein structures that other supervised approaches can't use, but in Section 4.1 they say \"chains are removed based on annotations\". So, do they only use annotated structures?\n6. Table-2 shows that their approach does not work well on protein structural similarity prediction. \n\nOther comments:\n------------------------\n1. How would embeddings from other unsupervised representation learning approaches like TAPE, or maybe AlphaFold compare?\n2. In Table-2, TMalign has either similar or better performance. But in Table-1, TMalign does much worse on fold classification. This seems quite odd. How is it being used for both the tasks?\n3. Section 4.3: why does the method not do as well on unseen proteins subclasses for enzyme classification?\n4. Section 4.3, computational time comparisons -- Can you compare the training time taken by your approach to the setup for DaliLite and TMAlign that don't need gpus and can be parallelized easily on multicore machines to make them faster.\n5. Which data was used for reaching the 20-40%, 40-60% etc, in Section 4.4? Was it a validation set? If the test set is used to check the performance, then this will overfit for the particular task chosen.\n6. The \"network architecture\" section needs better writing to convey the ideas. It is very confusing. Elaborate using a diagram or flow chart? It's not clear how many features and what all is done to the input.\n7. How is representation at protein level obtained? Average over all amino acids? And for each amino-acid, is it 16 dimensional? So protein representation is also 16 dimensional?\n8. Why not just sample p% starting from a given amino acid? What's the point of getting a window left and right of a specific amino acid?\n9. What are the strengths of the \n10. Intro in Section 3.3, \"\"noise\" and cropping seem irrelevant to the approach being used.:,\n", "summary_of_the_paper": "This paper presents an unsupervised deep learning approach for learning representations of 3D protein structures. They use an objective function motivated by the recent contrastive learning approaches (from computer vision). They show the utility of their model in two downstream applications: protein fold classification, enzyme classification and protein similarity prediction. The main contributions are:\n1. they show how the contrastive learning framework can be used in the context of protein structures\n2. they analyze the learned representations\n3. they show the utility of the model to downstream applications\n4. they show how fine-tuning their model leads to an improved performance", "summary_of_the_review": "I recommend 'weak accept' due to the concerns raised in the above section.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0wtd5zLCiMy", "reviewer": "Reviewer_c8cK", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "positives:\n1) The paper addresses an important problem of inferring protein functions and structures using limited number of labeled data points.\n2) Using contrastive learning as an unsupervised method for protein embedding seems to be novel.\n3) Experimental evolutions are extensive and ablation studies have been conducted.\n4) The paper is well-written and understandable.\n\nnegatives:\n1) Perhaps the main concern is that the results do not justify that the learned embedding has superior advantage over simpler baselines. The embedding unfortunately falls short on the Protein data set in Table 2 compared to baselines. Also in Table 1, applying classifiers (SVM, MLP, and 1-NN) on the learned embedding does not result in a competing classifier suggesting than the learned low dimensional representation does not fully reflect the nonlinearities present in the target classification tasks. Not sure how to justify from these results the embedding is useful beyond an initialization for the Fine-Tuning procedure. \n2) The novelty of the method is also limited; the contrastive learning is directly applied to the protein problem and several details of the method including sub-structure sampling and protein encoder are borrowed from Hermosilla et al. and Ingraham et al.\n\nquestions:\n1) How sensitive are the results of the embedding to the choice of p?\n2) Is it possible to train simple classifiers (e.g., SVM, etc.) on the Protein data of Table 2 similar to Table 1?\n3) Is there any benefit towards using larger number of proteins in PFAM as the main unsupervised data set? The paper motivates the presence of millions of proteins in PFAM, however, uses couple of hundred thousands as the main data set --- this could enable better results.\n4) Can the authors comment on possible interpretability of their embedding regarding features for specific 3D (sub)structures that might emerge from their method? and this give additional edge over the competing methods beyond the quantitative scores?\n\n\n", "summary_of_the_paper": "The paper proposes an unsupervised method for learning low dimensional representation of proteins borrowing ideas from contrastive learning in the computer vision literature. Experiemnts on three data sets compares the performance of such embedding with baselines algorithms. ", "summary_of_the_review": "Overall while the problem that the paper addresses is important and the method is reasonable, unfortunately, as it stands the proposed embedding method does not yield very convincing results compared to existing baselines in downstream tasks. Added with limited novelty of the method the paper in my opinion is marginally below acceptance; although I remain open to authors discussions in the rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper they adapt unsupervised contrastive learning to the problem of representation learning for proteins from 3D structure, using sub-structure sampling for the data transformation. The reviewers have concerns that the application tasks used for evaluation are not particularly impactful tasks, and that additionally, they are likely to not require protein representations that require more nuanced information. There are also concerns about the clarity of the manuscript, and novelty of the technical approach.
Decision: Reject