Pre-training Sequence, Structure, and Surface Features for Comprehensive Protein Representation Learning
Authors: Youhan Lee
Hasun Yu
Jaemyung Lee
Jaehoon Kim
Venue: ICLR 2024 poster (2024)
Keywords: Protein representation learning
self-supervised learning
implicit neural representation
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We describe a new pre-training approach for protein representation learning using generalizable implicit neural networks on protein molecular surfaces, showing SOTA results for various tasks.

Abstract: Proteins can be represented in various ways, including their sequences, 3D structures, and surfaces. While recent studies have successfully employed sequence- or structure-based representations to address multiple tasks in protein science, there has been significant oversight in incorporating protein surface information, a critical factor for protein function. In this paper, we present a pre-training strategy that incorporates information from protein sequences, 3D structures, and surfaces to improve protein representation learning. Specifically, we utilize Implicit Neural Representations (INRs) for learning surface characteristics, and name it ProteinINR. We confirm that ProteinINR successfully reconstructs protein surfaces, and integrate this surface learning into the existing pre-training strategy of sequences and structures. Our results demonstrate that our approach can enhance performance in various downstream tasks, thereby underscoring the importance of including surface attributes in protein representation learning. These findings underline the importance of understanding protein surfaces for generating effective protein representations.

Published as a conference paper at ICLR 2024
PRE-TRAINING SEQUENCE, STRUCTURE, AND SUR- FACE FEATURES FOR COMPREHENSIVE PROTEIN REP- RESENTATION LEARNING
Youhan Lee', Hasun Yu", Jaemyung Lee", Jachoon Kim
Kakao Brain
{youhan. lee, shawn. yu, james brain, jack.braainekkkkadbaain.cco
ABSTRACT
Proteins can be represented in various ways, including their sequences, 3D struc- tures, and surfaces. While recent studies have successfully employed sequence- or structure-based representations to address multiple tasks in protein science, there has been significant oversight in incorporating protein surface information, a crit- ical factor for protein function. In this paper, present pre-training strategy that incorporates information from protein sequences, 3D structures, and surfaces to improve protein representation learning. Specifically, utilize Implicit Neu- ral Representations (INRs) for learning surface characteristics, and name it Pro- teinINR. We confirm that ProteinINR successfully reconstructs protein surfaces, and integrate this surface learning into the existing pre-training strategy of se- quences and structures. Our results demonstrate that our approach can enhance performance in various downstream tasks, thereby underscoring the importance of including surface attributes in protein representation learning. These findings underline the importance of understanding protein surfaces for generating effec- tive protein representations.
INTRODUCTION
Proteins are vital components of biological systems, executing a myriad of functions that underpin an extensive array of cellular processes and biological pathways. These intricate macromolecules have multi-faceted characteristics that can be represented through different paradigms, including but not limited to their amino acid sequences, three-dimensional (3D) structures, and the specific attributes of their surface regions. recent years, advancements in high-throughhut sequencing (HTS) technologies, cryogenic electron microscopy (cryo-EM1 and sophisticated algorithms for protein structure prediction (Jumper et al.] 2021 Back et al. 2021: Lin et al.] 2022) have led to an explosion of available protein sequence (Suzek al. 2007) and structure (Berman et al. 2000,
Varadi et al. [2022] Lin et al. (203)) data, most of which have been made publicly accessible. Lever- aging these abundant datasets, recent studies (Meier al., 2021 Zhang et al.) 2022 2023) have successfully employed machine learning models pre-trained on this data, resulting in significant
progress in tackling an array of downstream tasks in the field of protein science.
Despite these strides, there exists a notable oversight in the current landscape of protein representa- tion learning: the offen-underestimated significance of protein surface characteristics. The attributes of protein's 's surface are crucial in determining its functional properties, particularly in the context of molecular interactions like ligand binding, enzymatic catalysis, and signal transduction between molecules (Gainza et al.) 2020 Somnath et 2021). While existing works for protein representa- tion learning have focused heavily on encoding amino acid sequences and 3D structural elements, they have largely neglected the indispensable role that protein surfaces serve, thus leaving unad- dressed gap in the prevailing research. More specifically, the protein structure can be divided into the atoms comprising the backbone and the components constituting the side chains. In this context, the protein surfaces are determined by both backbone and side chain atoms. However, traditional pro- tein structure encoders typically process protein 2D graphs 3D geometric graphs that only contain
'These authors contributed equally this work

---

Published as a conference paper at ICLR 2024
Table I: Comparison of different protein encoders with and without sequence, structure, surface pre-training. Our model, ESM- GSM--GarNNt--N-MME covers three modalities, sequence, structure, and surface in both encoding and pre-training, achieving comprehensive protein representation learning.
Method
Sequence
Encoder
Structure
Encoder
Pre training
Sequence
Pre training
Structure
Surface Excoder
Pre training
Surface
Transformrr
GVP
GearNet
ESM-1b
ProiBett
DeepFRI
LM-GVP
ESM-GeanNee
GearNet-MCC
GeaNNetDDP
ESM GeaNNet-MC
ESM MMMM-GGaNNet-heee INR-MC (Ours)
alpha carbon or backbone atoms, respectively. Consequently, state-of-the-ait representations often lack consideration for side-chain information.
In response to this significant gap, our research aims to offer comprehensive solution. We pro- pose an all-encompassing pre-training strategy that incorporates information from all three essential aspects of proteins: sequences, 3D structures, and notably, surfaces. Our approach is pioneering in that it is the first to specifically target the learning of protein surface attributes, and employs cutting-edge Implicit Neural Representations (INRs) (Chen & Wang, 2022) to achieve this goal ef- fectively. This inclusive approach enables our model to enhance performance across various down- stream tasks, thereby emphasizing the importance of incorporating surface information in protein representation learning.
In summary, contributions include:
We are the first to propose pre- pre-training strategy that incorporates information from protein sequences, structures, and surfaces.
We utilize Implicit Neural Representations (INRs) an effective mechanism for learning surface characteristics of proteins.
We conduct a comprehensive comparison of the effects of pre-training on protein se- quences, structures, and surfaces, thereby demonstrating the efficacy of learning about surfaces.
RELATED WORK
2.1 PROTEIN REPRESENTATION LEARNING
Most studies in the field of protein representation learning have adopted one of three main ap- proaches: (i) focusing on protein sequences, (ii) concentrating on protein structures, or (iii) employ- ing a hybrid strategy that incorporates both sequence and structural information.
In the first approach, which focuses learning protein sequences, researchers commonly adopt the architecture of pre-trained language models from the field of Natural Language Processing (NLP), such as Transformer vaaasorrmer/aaaaanii Vaswani al. 2017). BERT (Devlin et al. 2018), and GPT (Radford et al., 2018), to effectively represent proteins by learning their amino acid sequences as if they were lan- guage EInaggar al. 2007 Meier et al. 2021 Notin et al., 2022). The second approach generally employs Graph Neural Networks (GNNs)-based architectures (Ingraham et al.) [019 Jing et al.) 2020 Hermosilla et al. 2020 Zhang et al. 2022) to capture the intricate structural features of pro- teins. In the third approach, hybrid models aim to learn from both protein sequences and structures. Notable studies, such as DeepFRI (Gligorijevćć et al. 202T) and LM-GVP (Wang et al.) (022) 2 have utilized encoders for both sequence and structural information and have pre pre-trained
sequence data. STEPS (Chen et al. 2023) and ESM- GearNet (Zhang et al. 2023) have gone a step further by also pre-training on structural information achieve enhanced performance.
2

---

Published as a conference paper at ICLR 2024
However, these methods have not taken into account the significant role of protein molecular sur- face information plays in various biological processes. Traditionally, molecular surfaces are defined using Connelly surfaces (Connolly, 1983; Sanner et al. [996) based on van der Waals (vdW) radii, represented as mesh-based structures derived from signed distance functions. Seminal work for modeling protein molecular surfaces is MaSIF (molecular surface interaction fingerprinting) (Gainza et al.) (220), which fingerprints molecular surfaces expressed as molecular meshes using pre-defined and pre-calculated physical and geometrical features. To remove the high pre-computation costs of featurization, [Sverrisson et al. (02T)) proposed dMaSIF, which showed that modeling molecular sur- faces as a point cloud with atom categories per point is competitive, [Somnath al. (222)) proposed HOLOProt, which attempted to segment the protein surface into "superpixels" for more efficient consideration of surface information and used the features conjunction with structure features in a multi-modal modeling manner. However, theoretically, molecular surfaces are continuous surfaces with infinite resolution, which existing mesh-based approaches cannot fully express. To tackle this challenge, utilize the Implicit Neural Representations (INRs) approach, a technique capable of perfectly capturing infinite resolution characteristics. Our model, called ProteinINR, understands protein molecular surface resolution independently. Furthermore, our ProteinINR model is gener- alizable INR approach, allowing us to develop single model capable of representing many protein structures. On the other hand, |Wang et al. (2023) proposed a harmonic message passing, called HMR, which considered surfaces during molecular representation learning. Compared to HOLO- Prot and HMR, which focused on the design of encoder, we use INRs as a pre-training framework in which a structure encoder is trained to extract structure features to recover molecular surface.
2.2 IMPLICIT NEURAL REPRESENTATIONS
Point cloud- (Qi et al. 2017a Thomas et al., 2019 Zhang et al.) [022], mesh-based (Sinha et al. 2016 Bagautdinov et al. 2018 Verma et al. 2018). and voxel-based (Curless & Levoy] 1996 Wu et al. 2015) [TTaarchheno et al. 2017 Zeng et al. 2017) methods have historically relied on fixed-sized coordinates or grids to represent 3D assets. Unfortunately. these approaches suffer from resolution dependency, making them insufficient for modeling or rendering high-resolution 3D assets effectively. In contrast, Implicit Neural Representations (INRs) concentrate learning parameterized functions that predict location -pcation-specific information for given arbitrary query coor- dinates by utilizing seminal methods such as decodinn (Park et al.) 2019b Mescheder et al.)
2019), Fourier features (Tancik et al.] 20200; Mildenhall et al. (221), sinusoidal activations (Sitz- mann al. 2020b), meta-learning (Tancik et al. 2021 Dupont al. 2022a by Bauer al., 2023), or transformer- based architecture (Chen & Wang 2022 The inherent differentiation gives INR the
benefit of being independent of resolution, enabling
depict scenes and objects with outstanding
precision and fidelity (Chen et al. 2021: Sajjadi et al., 2022 Jun & Nichol 2023).
Grattarola Vandergheynst (2022) proposed a generalized INR, which is the only work dedicated to the study of INR for proteins. The contributions were crucial because generalized INR expanded the use of INR to topological systems that do not possess a well-defined coordinate system. They utilized 2D graph spectral embedding to learn INR for various real-world systems in non-Euclidean domains, including proteins. Nevertheless, although the work demonstrated the capacity to general- ize across diverse systems, it necessitated the training of individual Multi-Layer Perceptron (MLP) models for each sample, hence constraining its ability to generalize across datasets. Our study pro- vides evidence that it is feasible to represent protein surfaces using the INR with the Euclidean coordinate system. Furthermore, our study contributes to the area by showcasing the feasibility of a generalizable INR model capable of representing an entire dataset with single model.
3 PRELIMINARIES
3.1 PROTEIN GRAPH
Proteins are constructed by 20 different amino acids. Their 3D structures are formed through the chemical bonds and interactions among the atoms of the amino acids and making them naturally suited for graph representation. Based on the GearNet's representation (Zhang et al., 2022), which aim to solve, a protein P is expressed
exhibits high performance for downstream tasks relational graph Gp, made up of (V, P.E.R). R). V is the set of nodes and each node presents a residue in protein and includes the amino acid residue type and 3D coordinate. E is the set of edges among
3

---

Published as a conference paper at ICLR 2024
Previriin sequerces
Bequenoe Encoder (ESM-16)
Sequence
Downdleam
Replace outpute sequence encoder
5. Pro-imio diructures
[Searhin
laxdon
Stucture
inttalize structure encoder
Oelu componeet
Surfuce Encoder (Mils)
Surfape
Figure I: An illustration our proposed strategy for pre-training sequences, structures, and surfaces to solve downstream tasks.
Coorcinates
Implicht Desoder
Transfonner Ercoder SellAAllention
High-treueeryy-wwwne
Spatial
Malecular suries surface point dloud
Protein atrushure
Lalend embeddding
Figure 2: An overview of our ProteinINR architecture. The points tokens, structure tokens, and latent tokens are calculated using high-frequency-nne aware point encoder, structure encoder (GearNet- Edge-IEConv) and Transformer encoders output parameters of an MLP using the tokens, then SDF values are obtained using the parameters for the query coordinates.
convolution layers, respectively. Points 16k resolution.
nodes with their types R. such as the edges between two residues located within certain distance on the protein sequence or 3D coordinates.
3.2 INRs
To model the surface the protein, and we utilize Signed Distance Function (SDF) to represent the surface. SDF is well-established strategy for representing 3D shapes scalar fields. The SDF is mathematical expression that assigns scalar value to a given coordinate x, expressing the distance between the spatial point and the closest point on the shape's surface follow:
F(x) =: SER.
We employ the methodology DeepSDF PPar et al. 2019a) and train model that possess con- tinuous implicit representations, which describe the F for geometric molecular surfaces. We define the inside surface as < 0 and the outside surface as 0. Following this definition, the equation F(x) = 0 implies the molecular surface boundary, specifically defining the molecular surface. In summary, we train model that encode a protein molecular surface and produce INR parameters, which imply the F.
4

---

Published as a conference paper at ICLR 2024
4 METHOD
As mentioned earlier, we aim to pre-train sequences, structures, and surfaces of proteins for better protein representation. The sequence, structure, and surface are quite different modalities and thus establishing strategies for pre-training the sequence, structure, and surface is very important. To learn a large volume of structural and sequence data, we employ the "series- fussssuu""o""" approach which has demonstrated superior performance in previous work (Zhang et al., 2023). First, pre-train the sequences on sequence encoder and use this encoding as input for the structure encoder. Then, pre-train the structure encoder on the surfaces using ProteinINR and utilize the weights of pre- trained structure encoder as initial weights for pre-training on the structures. Then, we pre-train the structure encoder on the structure through multi-view contrastive learning based on the approach from Zhang et al. (2022) to obtain the final protein representation. Our pre-training strategy can be seen as continual pre-training (Ke et al. 2022). Finally, we leverage protein representation from the pre-tra rreitrai model on three modalities to solve downstream tasks. Figure contains an illustration of our pre-training strategy.
4.1 GENERALIZABLE IMPLICIT NEURAL REPRESENTATIONS FOR PROTEIN
To effectively train protein surfaces, we employ INRs. In the early stages of INRs, a coordinate- based MLP is trained for each individual instance. However, with the increasing amount of datasets, the computational expense associated with training multiple MLPs for each individual data point has become costly. Consequently, various solutions have been proposed to develop gen- eralizable INR to accommodate an entire dataset within a single model. One notable approach, TransINR (Cen & Wang, 2022), entails leveraging Transformer architecture, particularly for INR parameter calculation based on multiple partial views of 3D objects as conditioning inputs. This technique has garnered considerable attention the field. Building upon these advancements, Pro- teinINR adopts and first extends these methodologies in the protein field. It represents an expressive and generalizable INR that can effectively capture the shapes of tens of thousands of protein in- stances within single model.
4.1.1 ENCODING PROTEIN USING POINT AND STRUCTURE ENCODER
The ProteinINR framework first encodes a certain protein instance P into a protein point set embed- ding h. ProteinINR inputs the 3D protein asset as protein point cloud P E R NN3 The variable N denotes the number of points in the point cloud of the protein molecular surface, and we ran- domly sampled 16,384 points to input the point encoder in our experiments. ProteinINR utilizes the Dual-scale Point Cloud Recognition (DSPoint) (Zhang et al. 202T) Encoder 1/ to address the com- plex and irregular nature of protein surfaces, which exhibit intricate high-frreuency features. This encoder effectively captures a given point cloud's high-frequency and low-frequency characteris- ties, demonstrating notable efficacy in the tasks that involve high-frequer freeufryy features, such point cloud segmentation. Following the process of updating point features through the DSPoint method, we downsample the points into a reduced set of AM points P E R M by utilizing the deformable Kernel Point Convolution (KPConv) networks (Thomas al. 2019). Ultimately, learnable linear transformation is implemented on the downsampled points to align the embeddings' hidden dimen- sion prior to cross attention.
RGB values are frequently utilized as characteristics for individual points in point cloud modeling. ProteinINR considers the chemical properties of protein surfaces stemming from their electrical en- vironment as chemical colors. Although MaSIF utilized a pre-computation technique to determine the chemical colors, the computational cost associated with this approach is prohibitively expen- sive. Fortunately, dMaSIF has shown that it is possible to create a comprehensive representation of chemical properties by utilizing atom category features and distances inside an end-to-end learn- ing framework. Building upon these findings, we adopt similar approach for protein point cloud chemical color representation. We integrate two essential elements into our approach, namely atom categorical embeddings and the property of Top K closest distances. Incorporating these character- isties into the point cloud encoder leads to the formation of embeddings that encompass the surface's s chemical attributes. The utilization of the encoder and chemical features mentioned ensures that Pro- teinINR represents the protein molecular surface by considering the intricate interaction between the protein's 's structural and chemical characteristics.
5

---

Published as a conference paper at ICLR 2024
The primary contribution of our study is employing INR training as pre-training technique inject the knowledge of protein surface characteristics into the protein structure encoder. In order to ac- complish this, represent imput protein as structure and incorporate protein structure encoder into the INR training process, which allows us to encode the protein structure graph Gp and generate protein structure embeddings g R Rxx where R and h are length of residues and length of hidden dimension, respectively. Finally, the embeddings h can be used for various downstream tasks. The utilization of this architectural design enables the protein structure encoder actively participate in the process of acquiring surface-aware representation learning. As a result, the structure encoder enhances its ability to comprehend and depict protein molecule surfaces comprehensively. We note the extracted point embedding p E I Mxh
4.1.2 SPATIALLY ARRANGED LATENT REPRESENTATIONS
Recently, Spatial Functa (Bauer et al. 2023 has demonstrated improvements in the quality of la- tent representations when two-dimensional spatial inductive biases are incorporated. Building upon this, we extend the concept to ptree-dimensiona protein surfaces. In ProteinINR, the latent em- beddings z Lixe with length of L are initially rearranged into a three-dimensional voxel grid E IR Rxxxxxxx, where i, j. and k are feature size, width, height, and depth of latent grid respec- tively. Following that, we implemented 3D convolutions on the reorganized embeddings, which allowed for the incorporation of spatial inductive biases inside the latent embeddings. Finally, latent embeddings are rearranged to have the original shape z E RLxe and projected through a learnable projection layer to have the feature dimension E RLxh While this approach may seem simple, the results are remarkably effective, leading to enhanced INR performance, as further elucidated in ablation study.
4.1.3 TRANSFORMER ENCODER FOR INRS
In ProteinINR, the latent representation (referred to as z) of a protein instance's surface is obtained using a transformer encoder. The initial step is the concatenation of the protein surface point cloud embedding p, structural embedding g, and latent embedding z as follows:
h Concat (p. s. s, h Mahee(Ma [MM+R+L)xh
(2)
Next, the final latent codes z are obtained through self self-attention processes where protein information propagated over all protein-related tokens and latent embeddings.
4.1.4 INR DECODER AND SDF REGRESSION
In order to strengthen the ability of ProteinINR to capture localized and fine fine-grained details of local surfaces, we utilize the decoder introduced by (Lee et al.) 2023). This decoder has demonstrated significant improvement of over 50% compared to the prior TransINR model. The improvement is achieved by introducing a locality inductive bias into the INR framework. In ProteinINR, the locality-aware INR decoder D utilizes the latent code predict the SDF for query coordi- nates x E IR Xx3 near molecular surface of N protein samples p The optimization ProteinINR is fulfilled by minimizing the L2 loss between the predicted SDF values and the corresponding SDF values for each SDF sample. Furthermore, clamping techniques are employed focus the model's attention on the specific details within the vicinity of the surface region. We used the clamp value of 0.2, as employed in the DeepSDF. More detailed steps are followed:
=DDxxxx> 2)
(3)
N K
min v.2 NK,
|elamp(s, 6) clamp(s, 8)12
(4)
4.2 PRE-TRAINING ON SEQUENCES AND STRUCTURES
To effectively learn protein representations from a large volume of structural and sequence data, employ the "series fusion" approach, which has demonstrated superior performance in previous

---

Published as a conference paper at ICLR 2024
work (Zhang et al. 2023). In the "series fusion" architecture, the output from the trained language model is fed into the structure encoder. We utilize ESM-1b (Meier et al. 2021) as the trained lan- guage model. To encode protein graph by learning their structural information, we adopt the Gedd-ee--dgge-IECoom architecture, which performs best across most tasks, as the structure encoder and then pre-train the structure encoder structures by employing the multi-view contrastive learning approach (Zhang et al.) 2022). The detailed information about the architecture and hyper- parameters of structure pre-training we used is described in Appendix A.I
5
EXPERIMENTS AND RESULTS
5.1 DATASET PREPARATION FOR PRE-TRAINING
INR training Before calculating samples of the signed distance function, we generated the zero- level surface, namely, the molecular surface, represented by the equation F(x) = in implicit representations. To accomplish this objective, Sanner et al. 1996), which is well-established triangulation software for molecular surfaces. Sub- sequently, we computed the SDF values for the points acquired by the sampling approach utilized in De. The sample points are sampled near to the molecular mesh obtained via MSMS. In that case, the SDF values are their distances from the nearest vertices point of given molecular surface mesh In this work, 500,000 points were generated for SDF training, serving as the SDF points independent of the protein point cloud imput. These points and corresponding SDF values are utilized as the target data for INR training. We train ProteinINR in 50 epochs with learning rate of 1e-4.
utilized the MSMS program (Connolly, [983]
Structure pre-training To pre-train structural information, we utilize AlphaFold Protein Structure Database version 2 (Varadi et al.] (022) pre-train the models. We protein structure prediction data for 20 species and Swiss-Prot (Boeckmann et al. 2003). In-depth details and statistics about the data we used are provided in the Appendix A.2
5.2 EXPERIMENTAL SETTINGS
Downstream tasks To quantify representation power of our proposed method, we adopt three downstream tasks. As in GearNet paper, we choose Enzyme Commission (EC) number prediction task and Gene Ontology (GO) term prediction proposed from Gligorijević et al. (202I). Fold Clas- sification (FC) suggested from Hou al. (2018) adopted as downstream evaluation as well. EC task is prediction of EC numbers of proteins which represent biomedical reactions they catalyze. GO task divided into three sub-tasks by their ontologies, biological process (BP), molecular function (MF), cellular component (CC). Each task predicts whether a protein is associated with a specific GO term. For EC and GO tasks, mar and pair-centric area under precision-recal curve (AUPR) values are calculated to measure performance. In the FC task, fold labels of proteins are classified, and mean accuracy is used to evaluate performance.
We evaluate total of seven models: i) GearNet, which is trained directly the downstream tasks with a structure module; ii) GearNet-INR, where the structure module is pre-trained on the surfaces, and then trained on the downstream task; iii) GearNet-MC, whose structure module is pre-trained on the structures by multi-view contrastive learning, and then trained on downstream tasks; iv) GearNet-INR-MCC whose structure module is pre-trained on the surfaces, subsequently on the structures, and then trained on downstream tasks; v) ASM--Geareet-ME where a sequence encoder is pre-trained, followed by pre-training on the structures; vi) Ger- where a sequence encoder is pre- pre-trained, followed by pre-training on the surfaces; vii) ESM-GearNet- INR-MC, which entails pre-trained sequence encoder, then pre-training the structure module on the surface, followed by further training on the structure, and finally training on the downstream tasks. We use ESM-lb as the seuqnece encoder and GearNet-EEgee-EECom We finetune each task with the datasets as described in Appendix A.3 The model is trained for 50 epochs on EC, 200 epochs on GO, and 300 epochs on fold classification task. We finetune and evaluate the model upon the framework proposed by GearNet (Zhang al. 2022) and all other settings for finetuning models is same except batch size. We use batch size as 16 per step (8 A.00 GPUs and 2 for each GPU) for all experiments.
as the structure encoder.

---

Published as a conference paper at ICLR 2024
Table 2: Performance downstream tasks. We compare the models with and without using the pre- trained weights from ProteinINR. We highlight the cases where performance is the best in terms of F max and AUPR for EC and GO task and mean accuracy for FC in bold. indicates scores extracted from Xu et al. 2023), which conducted different settings compared to our study.
Method
EC
GO-BP
GO-ME
GO-CC
FC
Sum
Fea.
AUPR
Fras
AUPR
Frar
AUPR
Frus
ALPR
Auc
ESM-Ib
86.9
65.4
45.2
33.2
65.9
63.0
47.7
32.4
87.4
GearNet
81.6
44.6
60.4
43.3
465.5
GearNettINR
81.4
44.7
59,9
43,0
47.5
466.1
GrarNet-MC
87.3
GeurNet-IRR-MC
86.9
49.8
51.1
498.5
BSM-GGarNNe--CC
521.9
ESN-GGeNNGI-NNR
89.0
50.8
67.8
50.6
48.9
530.3
BSM-GGoWNG-NNRMM
89.6
51.8
68.3
58.0
50.4
50.8
Original
Reconstructed Mosh
Original
Reconstructed Surface
Figure 3: Above images are the examples of reconstructed meshes and surfaces from ProteinINR for given proteins. ProteinINR preserves the intricate details of irregular protein surfaces, particularly capturing features such ring-like and hole shapes with remarkable fidelity.
5.3 EXPERIMENTAL. RESULTS
Representing protein surface shapes using ProteinINR The procedure for acquiring triangular mesh that corresponds to a specific protein using INR parameters from ProteinINR is outlined as follows. Initially, the SDFs are calculated for the vertices of voxel grid with regular size of 128. Following this, the marching cubes algorithm (Chernyaev 1995) is employed to compute the mesh. Protein surface samples reconstructed using ProteinINR R are depicted in Figure It is worth mentioning that the protein molecular surfaces exhibit significant irregularity and possess properties. Intriguingly, ProteinINR effectively preserves intricate information, even hole or ring-like shapes. In addition, we calculated the Chamfer distance between the ground truth and the reconstructed meshes for the test set. A subset of 30,000 data points was selected, and the computed average Chamfer distance A.4 was 0.003. The number might be quite decent in the context of Chamfer distances for natural 3D objects as reported in studies (Mescheder et al. 2019] Park et al., 2019b) Sitzmann et al. 2020a; Liu et al., (023) related to SDF reconstruction. These findings indicate that ProteinINR effectively acquires generalizable INRs that can accurately depict the uneven surfaces of proteins.
Downstream evaluation We compare the performance of the structure encoder initialized with weights from a pre-trained ProteinINR model and without such initialization across various down- stream tasks related to protein function. Intriguingly, we can see that ESM-GearNettINR-MC and CSMM-GearNet-INR outperform the previous ofateoof-the-art model. CSMM-Gearet-MCC. when taking the summation of all scores. This demonstrates our main contribution clearly, emphasizing that incorporating surface-related features, which have not been explored by previous models, into protein pre-training representation learning enables comprehensive representation learning for pro- teins. Additionally, we observe a rapid decrease in pre-training loss as depicted in Figure 4 which provides an additional evidence.
8

---

Published as a conference paper at ICLR 2024
Table 3: Results EC task depending pre-training order.
Method
Pre-training
GO-BP
GO-MF
G0-CC
Sum
AUPR
AUPR
AUPR
AUPR
ISMI-GeurNet INR-MC
sequences + surfazes 3D strccuurs
$9.6
90.3
51.5
33.2
68.3
55.0
50.4
35.7
50.5
528.1
GeurNet-INR-NN GearNer-MC-INE
suriaces 3D stuctures
$6.9
58.9
49.5
26.0
65.4
56,1
47.7
25.6
51.1
498.5
JD simettres surfaces
$4.1
86.0
46.9
25,9
62.1
54.5
44.8
272
47.6
478.9
Protein function primarily occurs on the surface and is closely associated with surface features. The observed enhancement in protein function tasks indicates that acquiring surface understanding using INR is advantageous. In contrast, a noticeable enhancement in performance is not observed in the FC task. Since surface features imply higher representations of the outer part of protein structure, these features may not contribute substantially to classifying the overall fold structure. Otherwise, as the process pre-training progresses, the loss gap between models diminishes, We attribute this trend to the nature of our encoder, which focuses solely on alpha carbons. While surface information derived from full-atom information, the encoder only learns from alpha carbon structures. So, we hypothesize that the 2nd stage mutual information maximization during pre- pre-training training structure data biases the model toward alpha carbon structure information after st stage surface pre-training.
Nonetheless, under these limited conditions, the models including the protein surface modality show performance gains.
Experiment on order Considering the original results (Table 2 it is evident that ESM has the most dominant impact on the downstream tasks, revealing the supportive role of structure and surface in enhancing performance on downstream tasks. Table 3 enables us to compare the significance of structure and surface pre-training while excluding the dominant influence of sequence. We can see that the structure encoder (GearNet-INR-MC -NR-MC), which learned structure information last, had superior performance compared to GearNet-MC-INR, which was pre-trained the opposite manner. Based on the results of GearNet-INR (466.1) I) and GearNet-MC (498.3) shown in Table it seems that in the absence of pre-training on sequences, structure pre-training has a greater influence on downstream tasks compared to surfaces. We conjecture that this observation supports the findings shown in Table g
Ablation study on 3D latent embedding In ProteinINR, we incorporate a 3D convolution layer introduce spatial inductive bias to the latent space. To evaluate the effect of the approach, conduct an analysis of the learning curve of INR when incorporating or excluding spatial inductive bias. As depicted in Figure in Appendix, the existence of spatial inductive bias clearly enhances the learning of INR.
CONCLUSION
We propose a pre-training strategy for learning from sequences, structures, and surfaces of proteins to achieve better protein representation. For the first time, we use INR to pre-train the protein sur- face, introducing method we call ProteinINR. We confirm that ProteinINR effectively reconstructs the protein surfaces. Moreover, the results on the downstream tasks demonstrate that learning the protein surface can lead to better protein representation.
Our work represents an important step towards incorporating protein surfaces, which play crucial role in protein functions. There are several interesting avenues for further research: generating new proteins from the latent representation of surfaces pre-train; applying our approach other types of molecules, such as mmlll-molecule drugs; and identifying a better strategy for integrating all three modalities, particularly the effective integration of surface and structure pre-training.
Meanwhile, our approach has limitation dependency on protein structure, so the use of predicted structures may worsen the performance of method for proteins without known structures.

---

Published as a conference paper at ICLR 2024
7 REPRODUCIBILITY
The architectural design of ProteinINR is influenced by TransINR, and we utilize the decoder model introduced by Lee al (Lee al. 2023). The DSPoint and KPConv, and GearNet are imple- mented from their official codes. The traiing dataset used in pre-training structures is prepared from the AF2 prediction dataset, similar to in GearNet. The procedure for generating SDF data is implemented in accordance with the approach described in the DeepSDF framework. To assess the performance of downstream tasks, we use the well-published TorchDrug framework (Zhu et al.) 2022). We describe detailed information regarding the training and evaluation in Section 4, Section 5, and Appendix.
8 ETHICAL STATEMENT
In this work, we focus advancing the topic of protein representation learning by incorporating sur- face information alongside sequence and 3D structure-basse based representations. We acknowledge the importance of ethical considerations in scientific research and we aim to provide further clarification on the following ethical aspects.
We provide transparent and comprchensive details about our methodology, experiments, and results. We list any limitations or potential biases in our research.
Our research aims to elucidate the significance of surfaces in protein representation learning, hence potentially influencing drug discovery and enzyme development. This would have the potential impact wide range of applications, including the identification of innovative therapeutic targets, the development more promising drugs, improvements in agricultural productivity, and ultimately, improvements in human health.
Our purpose of this research is to make constructive and positive contribution to the domain of protein science while upholding the ethical conduct of our research.
REFERENCES
Minkyung Baek, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong, Lisa N Kinch, R Dustin Schaeffer, al. Accurate prediction of protein structures and interactions using three-track neural network. Science, 373(6557):871- 876. 2021.
Timur Bagautdinov, Chenglei Wu, Jason Saragih, Pascal Fua, and Yaser Sheikh. Modeling facial geometry using compositional vaes. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3877-3886, 2018.
Matthias Bauer, Emilien Dupont, Andy Brock, Dan Rosenbaum, Jonathan Schwarz, and Hyunjik Kim. Spatial functa: Scaling functa to imagenet classification and generation. arXiv preprint arXiv:2302.031300 2023.
Helen M Berman, John Westbrook, Zukang Feng, Gary Gilliland, Talapady N Bhat, Helge Weissig, Ilya N Shindyalov, and Philip E Bourne. The protein data bank. Nucleic acids research, 28(1): 235-242, 2000.
Brigitte Boeckmann, Amos Bairoch, Rolf Apweiler, Marie-Claude Blatter, Anne Estreicher, Elis- abeth Gasteiger, Maria J Martin, Karine Michoud, Claire Donovan, Isabelle Phan, al. The swiss-prot protein knowledgebase and its supplement trembl in 2003. Nucleic acids research, 31 (1):365-370, 2003.
Can Chen, Jingbo Zhou, Fan Wang, Xue Liu, and Dejing Dou. Structure-aware protein self- supervised learning. Bioinformatics, btad 2023.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
10

---

Published as a conference paper at ICLR 2024
Yinbo Chen and Xiaolong Wang. Transformers meta-learners for implicit neural representations. In European Conference on Computer Vision, pp. 170-187. Springer, 2022.
Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8628-8638, 2021.
Evgeni Chernyaev. Marching cubes 33: Construction of topologically correct isosurfaces. Technical report, 1995.
Michael L Connolly, Solvent-accessible surfaces of proteins and nucleic acids. Science, 221(4612): 709-713, 1983.
Brian Curless and Marc Levoy. A volumetric method for building complex models from range images. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pp. 303-312, 1996.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint (0X.vv881000480)) 2018.
Emilien Dupont, Hyunjik Kim, SM Eslami, Danilo Rezende, and Dan Rosenbaum. From data to functa: Your data point is function and you can treat it like one. arXiv preprint arXiv:2201. 12204, 2022a.
Emilien Dupont, Hrushikesh Loya, Milad Alizadeh, Adam Golinski, Yee Whye Teh, and Arnaud Doucet. Coin++: Data agnostic neural compression. arXiv preprint arXiv:2201.12900 12904, 1(2)4, 2022b.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rihawi, Yu Wang. Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. Prottrans: Towards crack- ing the language of life's code through sulf-supervised deep learning and high performance com- puting. arxiv 2020. arXiv preprint arXiv:2007.0622 2007.
Haoqiang Fan, Hao Su, and Leonidas J Guibas. A point set generation network for 3d object recon- struction from a single image. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 605-613, 2017.
Pablo Gainza, Freyr Sverrisson, Frederico Monti, Emanuele Rodola, D Boscaini, MM Bronstein, and BE Correia. Deciphering interaction fingerprints from protein molecular surfaces using geo- metric deep learning. Nature Methods, 17(2): 192-192, 2020.
Vladimir Gligorijević, P Douglas Renfrew, Tomasz Kosciolek, Julia Koehler Leman, Daniel Beren- berg, Tommi Vatanen, Chris Chandler, Bryn C Taylor, Ian M Fisk, Hera Vlamakis, et al. Structure- based protein function prediction using graph convolutional networks. Nature communications, 12(1):3168, 2021.
Daniele Grattarola and Pierre Vandergheynst. Generalised implicit neural representations. Advances in Neural Information Processing Systems, 35:30446-30458, 2022.
Pedro Hermosilla, Marco Schäfer, Matëj Lang, Gloria Fackelmann, Pere Pau Vázquez, Barbora
Kozlíková, Michael Krone, Tobias Ritschel, and Timo Ropinski. Intrinsic-extrinsie convolution and pooling for learning on 3d protein structures. arXiv preprint arXiv:2007.06252. 2020.
Jie Hou, Badri Adhikari, and Jianlin Cheng. Deepsf: deep convolutional neural network for mapping protein sequences to folds. Bioinformatics, 3495):1255-130: 2018.
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. Advances neural information processing systems, 32, 2019.
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael Townshend, and Ron Dror. Learning 2020.
from protein structure with geometric vector perceptrons. arXiv preprint
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvumakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-555 2021.

---

Published as a conference paper at ICLR 2024
Heewoo Jun and Alex Nichol. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463, 2023.
Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. Continual pre- training of language models. In The Eleventh International Conference on Learning Representa- tions, 2022.
Doyup Lec, Chiheon Kim, Minsu Cho, and Wook-Shin Han. Locality- generalizable implicit neural representation. In Advances in Neural Information Processing Systems, 2023.
Zeming Lin, Halil Akin, Roshan Rao. Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Allan dos Santos Costa, Maryam Buzel-Zarandee Tom Sercu, Sal Candido, et al. Language models of protein sequences the scale of evolution enable accurate structure prediction. bioRxiv, 2022.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science,
(11166637::123-1113 2023.
Weixiao Liu, Yuwei Wu, Sipu Ruan, and Gregory S Chirikjian Marching-primitives: Shape abstrac- tion from signed distance function. In Proceedings of the IEEEJCVF Conference on Computer Vision and Pattern Recognition, pp. 8771-8780, 2023.
primitives:
Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alex Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. Advances in Neural Information Processing Systems, 34:29287-29303, 2021.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc- the IEEE/CVF conference on computer vision and pattern recognition, pp. 4460-4470, 2019.
cupancy networks: Learning 3d reconstruction in function space. In Proceedings the
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Pascal Notin, Mafalda Dias, Jonathan Frazer, Javier Marchena Hurtado, Aidan N Gomez, Debora Marks, and Yarin Gal. Tranception: protein fitness prediction with autoregressive transform- ers and inference-time retrieval. In International Conference on Machine Learning, pp. 16990- 17017. PMLR, 2022.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition (CVPR), June 2019a.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 165-174, 2019b.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 652-660, 2017a
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet+++: Deep hierarchical fea- ture learning point sets in metric space. Advances in neural information processing systems, 30, 2017b.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training. 2018.
Mehdi SM Sajjadi, Henning Meyer, Etienne Pot, UIs Bergmann, Klaus Greff, Noha Radwan, Suhani Vora, Mario Lučić, Daniel Duckworth, Alexey Dosovitskiy, et al. Scene representation trans- former: Geometry-free novel view synthesis through set-latent scene representations. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6229-6238. 2022.
12

---

Published as a conference paper at ICLR 2024
Michel F Sanner, Arthur J Olson, and
Spehner. Reduced surface: efficient way to compute molecular surfaces. Biopolymers, 38(3):305- 320, 1996.
Ayan Sinha, Jing Bai, and Karthik Ramani. Deep learning 3d shape surfaces using geometry images. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, Oc- tober 11-14, 2016, Proceedings, Part VI 14, pp. 223-240. Springer, 2016.
Vincent Sitzmann, Eric Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta-leaming signed distance functions. Advances in Neural Information Processing Systems, 33:10136-1011 10147, 2020a.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in neural information processing systems, 33:7462-7473, 2020b.
Vignesh Ram Somnath, Charlotte Bunne, and Andreas Krause. Multi-scale representation learning on proteins. Advances in Neural Information Processing Systems, 34:25244-252555 2021.
Baris E Suzek, Hongzhan Huang, Peter McGarvey, Raja Mazumder, and Cathy H Wu. Uniref: comprehensive and non-radundant uniprot reference clusters. Bioinformatics, (2110))1222-1222 2007.
Freyr Sverrisson, Jean Feydy, Bruno E Correia, and Michael M Bronstein. Fast end-to-end learning on protein surfaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15272-15281, 2021.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Process- ing Systems, 33:7537-7547, 2020.
Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P Srinivasan, Jonathan T Barron, and Ren Ng. Learned initializations for optimizing coordinate-based neural representa- tions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2846-2855, 2021.
Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox. Octree generating networks: Efficient convolutional architectures for high-resolution 3d outputs. In Proceedings of the IEEE interna- tional conference on computer vision, pp. 2088-2096, 2017.
Hugues Thomas, Charles R Qi,
Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6411-6420, 2019.
Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, et al. Alphafold protein structure database: massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic acids research, 50(D1):D439 D4449--44444,,, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Nitika Verma, Edmond Boyer, and Jakob Verbeek. Feastnet: Feature-steered graph convolutions for 3d shape analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2598-2606, 2018
Yiqun Wang, Yuming Shen, Shi Chen, Lihao Wang, Fei Ye, and Hao Zhou. Learning harmonic molecular representations on riemannian manifold. arXiv preprint arXiv:2303.15522 2023.
Zichen Wang, Steven A Combs, Ryan Brand, Miguel Romero Calvo, Panpan Xu, George Price, Nataliya Golovach, Emmanuel O Salawu, Colby J Wise. Sri Priya Ponnapalli, al. Lm-gyp: an extensible sequence and structure informed deep learning framework for protein property predic- tion. Scientific reports, 12(1 ):632,, 2022.
13

---

Published as a conference paper at ICLR 2024
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Minghao Xu, Xinyu Yuan, Santiago Miret, and Jian Tang. Protst: Multi-modality learning of protein sequences and biomedical texts. In International Conference on Machine Learning, 2023.
Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao, and Thomas Funkhouser. 3dmatch: Learning local geometric descriptors from rgb-d reconstructions. In Pro- ceedings of the IEEE conference computer vision and pattern recognition, pp. 1802-1811, 2017.
Renrui Zhang. Ziyao Zeng, Ziyu Guo, Xinben Gao, Kexue Fu, and Jianbo Shi. Dspoint: Dual-seale point cloud recognition with high-frequency fusion. arXiv preprint arXiv:2111.10332 2021.
Zuobai Zhang, Minghao Xu, Arian Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Protein representation learning by geometric structure pretraining. arXiv preprint arXiv:2203..06125. 2022.
Zuobai Zhang, Minghao Xu, Vijil Chenthamarakshan, Aurélie Lozano, Payel Das, and Jian Tang. Enhancing protein language models with structure-base based encoder and pre-training. arXiv preprint arXiv:2303300225. 2023.
Zhaocheng Zhu, Chence Shi, Zuobai Zhang, Shengchao Liu, Minghao Xu, Xinyu Yuan, Yangtian Zhang, Junkun Chen, Huiyu Cai, Jiarui Lu, Chang Ma, Runcheng Liu, Louis-Pascal Xhonneux, Meng Qu, and Jian Tang. Torchdrug: A powerful and flexible machine learning platform for drug discovery. arxiv preprint arXIv:2202.08320, 2022.
14

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 DETAILED INFORMATION ON PRE-TRAINING
A.1.1 GRARNETT-EGGF-IEEOM FOR STRUCTURE ENCODER
To encode the structural information of proteins, we use the GearNee--dgee-ECCom architec- ture (Zhang et al. 2022). In GearNet architecture, protein is represented by a graph 9 EV,E, R) where represents residues of proteins, denotes edges between residues, and R represents edge types. Edges in GearNet are of three types: sequential edges, which are edges between two residues located within a certain distance on the protein sequence; radius edges, which are edges between two residues that have Euclidean distance of less than a specific value in 3D coordinates; and k-nearest neighbors edges, which are edges between a specific node and its k-nearest neighbors in terms of Eu- clidean distance in 3D coordinates. GearNet-Edge-IECC CECoo has two additional elements compared to GearNet: i) the Edge element, which transforms edges into nodes resulting in g' = (V",', RR) and facilitates message passing between edges, and ii) the IEConv element, which applies learnable kernel function to the edge, inspired by the previous work (Hermosilla et al. 20200).
A.1.2 PRE-TRAINING OF PROTEIN STRUCTURES
To pre-train GearNet- Edge--ECo,, we employ the multi-view contrastive learning approach, which shows the highest performance in the previous work (Zhang et al., 2022). Multi-view contrastive learning aims to make the embeddings of related substructures similar, while rendering the em- beddings of unrelated substructures distinct, akin to SimCLR (Chen et al. (020). Substructures are extracted either by taking a subsequence from the sequence or by extracting a subspace in 3D space. Table 4 presents the hyperparameters used in pre-training of structural data for ESM- GearNet-IECony and GearNet-IEConw, We save the model checkpoints every 5 epochs, and among the saved checkpoints, we use the checkpoint with the lowest loss for downstream tasks, We use 64 NVIDIA A100 80GB gpus for pre-training.
Table 4: GearNet-EEde-IECooo and ASSM-GearNet--ddge-EECCC hyperparameters GearNet-Edge--ECCom
ESM--GearNet-
Subsequence maximum length Subspace minimum neighbors
50
50
15
Not used ESM-Ib
Sequence model
Not used
Batch size
48
48
Optimizer
Adam
Adam
Learning rate
1.0e-3
2, 2.0e-4
# epochs
50
50
A.2 DATASETS USED IN STRUCTURE PRE-TRAINING
We utilize protein structure prediction data for 20 species and Swiss-Prot (Boeckmann et al. 2003)
from AlphaFold Protein Structure Database version 2 Varradi et al. 2022). After processing to convert pdb files to protein graphs and remove the proteins with errors, the statistics of the graphs we finally used are presented in Table
A.3 DETAILED INFORMATION ON DOWNSTREAM TASK
For downstream task, we basically use the framework provided by Ceearet(ZZaagg et al. (022) and all experiments shown in Table are performed under same conditions. The number of datasets used in each downstream task is described in Table 6 We finetune the models with batch size as 16 per step. All other settings are used as provided by the framework. The models demonstrating the highest performance validation set chosen report the results in Table 2
15

---

Published as a conference paper at ICLR 2024
Table 5: The number of protein structures used per species
Proteome ID
Taxonomy
# structures
UP000006548
Arabidopsis thaliana
27393
UP000001940
Cacnorhabditis elegans
19658
UP000000559
Candida albicans
5956
UP000000437
Danio rerio
24595
UP000002195
Dictyostelium discoideum
12592
UP000000803
Drosophila melanogaster
13424
UP000000625
Escherichia coli
4363
UP000008827
Glycine max
55747
UP000005640
Homo sapiens
23280
SP000008153
Leishmania infantum
7903
UP000000805
Methanocaldococcus jannaschii
1772
UP000000589
Mus musculus
21571
UP00000155$
Mycobacterium tuberculosis
3988
UP000059680
Oryza sativa subsp. japonica
43623
UP000001450
Plasmodium falciparum
5162
UP000002494
Rattus norvegicus
21209
UP0000023
Saccharomyces cerevisiae
6026
UP000002485
Schizosaccharomyces pombe
5123
UP000008816
Staphylococcus aureus
2885
UP000002296
Trypanosoma cruzi
18992
UP000007305
Zea mays
39258
Swiss-Prot
541938
Total
906458
Table 6: The number of datasets for downstream tasks.
Train Fold Classification
Dataset
Validation
# Test
Enzyme Commission 15,170
1,686
1,860
Gene Ontology 28,305
3,139
3.148
12,312
736
718
A.4 CHAMFER DISTANCE
The Chamfer distance is commonly used metric for shape evaluation, favored for its simplicity Fan et al. (2017). When comparing two point sets, S1 and S2, this metric the summation of nearest- neighbor distances between each point and its closest corresponding point in the other set as follows:
dChamfer distance (2)
Σ
min vES
v12
Σ VESa
min | -v11 sES;
(5)
16

---

Published as a conference paper at ICLR 2024
Loss per model over epoch
1.2
GearNet-MC
GearNeet-NNRMMC
ESM-GearNNet-CC
ESM-GearNe--INR-MC
1.0
0.8
Loss
0.6
0.4
10
20
30
40
50
Epoch
Figure 4: The loss of multi-view contrastive learning of GearNet-MC, GearNet-INR-MC, ESM- McarNet-MC. and ESM-GearNet-INR-MA Compared to GearNet-MC and ESM-GearNet-MC, observe that the GearNet-INR-Me and ESM-GeerNee-INRMMC which learn surface information, exhibit a faster decrease loss initially.
Training loss per model over epoch
ProteinINR
31
ProteinINR latent
PSNR
28
27
5
10
15
20 Epoch
25
30
35
40
Figure 5: Training curves of ProteinINR with regard the utilization of 3D latent bias.
17


Reviews:
[{"review_id": "XTArshwNDG", "reviewer": "Reviewer_1KWf", "summary": "Proteins can be represented in many different ways (sequence, structure and surfaces). While the sequence and structure based representations have been explored a lot in the literature, surface based representations have not been explored. This paper proposes a pretraining strategy that incorporates all three modalities for protein representation learning. They propose Implicit Neural Representations to learn the surface characteristics of proteins.", "questions": "1. Why do you think is the performance so much dependent on the pretraining order (Table 3). Please clarify this in the paper.\n2. Does seq->surface->3d structure outperform seq->structure->surface for all tasks? Table 3 only shows EC.\n3. It would be interesting to see if we can avoid the structure encoder altogether by just relying on sequence and surface features.\n4. In Table 1, it would be nice to have the performance on just ESM.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is novel and is the first method to use surface based pretraining for proteins. This could open up further avenues of study.\n- Incorporates several advances from computer vision (DeepSDF, DSPoint, KPConv, decoder by Lee et al., 2023 etc) into the protein domain.\n- The evaluation is fair and is done against state of the art methods.\n- Pretraining is done on a large number of structures\n- The paper is well written and easy to follow.", "weaknesses": "- Performance of the method is more or less in line with existing work (ESM-Gearnet-MC), with any improvements being marginal for some tasks.\n- Some more abalations/baselines would clarify the contribution of the surface based features  (see questions section).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mfuaBBOBtN", "reviewer": "Reviewer_AyLE", "summary": "This paper introduced a protein multimodal representation learning by leveraging information from protein sequence, structure, and surfaces. Specifically, the authors proposed a model which uses INR to encode protein surface information by using point cloud representation and fused it into the structure embedding. Empirically the authors have shown the performance boost comparing to the original baseline methods which only use structure information.", "questions": "1. Why does pre-training order impact the EC tasks? Does this observation apply to other downstream tasks as well?\n2. How does batchsize in finetuning affect the overall downstream performance?\n3. In section 4.1.4, how is the clamp value decided?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper has shown that fusing protein surface data with protein structure data generates better protein representation that benefits various downstream tasks.\n2. The proposed ProteinINR architecture is modularized and the encoders can be easily replaced by other implementations.\n3. Various downstream tasks experiments are performed. \n4. Clear description of training procedures and descriptions of the training data usage.", "weaknesses": "1. Lack of in-depth analysis on the effect of pre-training order. In Table 3, it is shown that GearNet-INR-MC performs better than GearNet-MC-INR for both $F_{max}$ and AUPR, however, no further analysis is provided.\n2. INR seems only work well when both structure and sequence information are present. In Table 2, both GearNet-INR-MC and GearNet-INR have similar performance as their counterparts. Only ESM-GearNet-INR-MC shows relatively bigger improvements comparing to ESM-GearNet-MC.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hq4WrXEjpF", "reviewer": "Reviewer_VAn9", "summary": "While there are numerous works on pre-training protein representations, most studies have focused on sequences-only, structures-only, or a hybrid of sequences and structures. This work proposes to further incorporate protein surface information for the pre-training. It employed a series-fusion approach where a model is pre-trained in the order of sequences, surfaces, and structures. Another essential part of the work is in the surface encoder which learns Implicit Neural Representations. The surface encoder is based on a proposed ProteinINR framework which consists of a transformer encoder on top of a point encoder, a structure encoder, and spatial latent representations. The experiments demonstrate that ProteinINR can reconstruct protein surfaces pretty well and the pre-trained representations outperform previous SOTA on several downstream tasks.", "questions": "- [Reproducibility] Do you have plans for making pre-trained representations and both training/evaluation codes publicly available?\n- [Ablations] Can you show how INR is important in the pre-training framework compared to previous methods to represent 3D surfaces?\n- [Experiments] Have the authors checked the possible inclusion of data from downstream tasks within the pre-training data? \n- [Experiments] Can the pre-trained representations be helpful for downstream tasks where only sequences are available and structures are not?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "According to the authors’ claim, this is the first work to propose a pre-training scheme for protein representation that incorporates sequence, structures, and especially surfaces. The authors adopted several ideas from previous works to develop ProteinINR which can effectively learn surface characteristics of proteins. I think they mostly explained the proposed method clearly with appropriate background information and without overstating their contributions. They demonstrate the proposed method is effective in learning better protein representations. The significance of the proposed method could vary, but I believe it can help a broad range of researchers with proper further actions.", "weaknesses": "- [Reproducibility] Although the authors provided implementation details, they didn’t have any statements regarding public availability. Since the point of pretrained representations is in using them for various downstream tasks, I think that their significance becomes quite small without making ways to obtain the pretrained representations publicly available.\n- [Ablations] Would it be possible to develop a surface encoder without implicit neural representations? While the authors discussed the advantage of INR compared to previous methods to represent surfaces, current experiments do not show how important INR is compared to them.\n- [Experiments] One of the clear weaknesses of the proposed approach is that it eventually relies on pretraining the structure encoder, GearNet. In other words, it's only applicable for downstream tasks where protein structures are known, which might be the most important in a real world. I'm curious whether authors have thought about downstream tasks where only sequences are available and structures are not. It would be interesting to see how the performance of pre-trained models changes with the usage of predicted protein structures (maybe from AlphaFold) instead of the true ones.\n\nMinor comments\n- [Introduction] It would be nice to have more explanation on protein surface characteristics, particularly regarding the relationship between structural features and molecular surfaces. Unfamiliar readers might think that surface information is already well-contained within 3D structures, such that cannot clearly understand the need for incorporating surface information for the pre-training.\n- [Figure 2] xyz seems to indicate coordinates, but it's inconsistent with notations from the texts.\n- [Sec 4.1.1.] Does the protein encoder mean point encoder? Due to the prevalent terminologies (point encoder, structure encoder, protein encoder, etc.), there seems to be a little confusing use of terminologies. The authors might want to take a look into the issue.\n- [Experiments] Can you further provide the performance of ESM-GearNet-INR? It would help to show the effectiveness of INR with sequence pretraining but without structure pretraining.\n- [Sec 5.3] It would be better to explain the Chamfer distance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ar0jgfXEwT", "reviewer": "Reviewer_rs2T", "summary": "This paper proposes to jointly pretrain protein surface and structure based on the pretrained ESM-1b. Consequently, the final protein representation includes three-kind information, which are protein sequence, surface, and structure. The pretrained protein representations are finetuned on three downstream tasks, which are enzyme classification, gene ontology term prediction and fold classification. The propose model performs best in most cases among all the ablation models.", "questions": "I also have the following questions:\n\n1. In 4.1.1, N= 16384. What does N refers to? Residue number? R refers to this. Number of vertices on surface? But in 5.1, the author mentioned they sampled 500,000 points for each protein. Besides, Does 500,000 sample points refer to the point cloud after downsampling? According to my opinion, using MSMS, there would be 100-120 vertices around each residue, which means 500,000 points at least represents a protein with length 5,000. That is a very long sequence. I don't believe the average minimum length of the pretraining proteins is 5,000.\n\n2. In Figure1, there is surface encoder and sequence encoder. In title 4.1.1, there is protein encoder. In title 4.1.3, there is Transformer encoder. It's kind of confusing. The author may need to unify the name and give an overall description.\n\n3. Why the author used ESM-1b instead of ESM-2. ESM-2 performs better.\n\n4. In Figure 3, is the reconstructed mesh obtained through the donwsampled vertices? And the original one is the one generated by MSMS?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "As far as I know, this paper is the first one to involve surface information into protein pretraining, and then applied the pretrained protein representations for downstream protein understanding tasks.", "weaknesses": "1. **Many of the components used in the proposed method are not new.** For example, for surface preparation, the paper just followed the apporoach proposed in [1]. For surface dowmsampling, the paper use DSPoint method.  For latent representation learning, the paper follow Spatial Functa. It seems the proposed method combine several existing modules to get a new pipeline.\n\n[1] Fast end-to-end learning on protein surfaces. CVPR 2021.\n\n2. **The paper lacks some important baselines.** In the main results, the paper only compares to the ablation models, which seems not enough. At least, the paper should compare to [1], [2]. Besides, the paper seems lacks the baseline of directly finetuning ESM-1b (ESM-2). I don't know how much benefits  that adding the additional surface and structure pretraining will bring to directly finetuning ESM series.\n\n[2] LEARNING HARMONIC MOLECULAR REPRESENTATIONS ON RIEMANNIAN MANIFOLD. ICLR 2023.\n\n3. **Some claims are not right.** In the first several lines of page 6, the paper mentioned \"Incorporating these characteristics into the point cloud encoder leads to the formation of embeddings that encompass both the surface’s geometric structure and chemical attributes.\" On the surface, the author only used atom category and distance, which are chemical features. For geometric features, the author may need to calculate Gaussian curvature, mean curvature, heat kernel signature and something like that, which are geometric features. Besides, I think the calim \"pre-training sequence, structure, and surface features\" might be not true because there is no sequence pretraining stage involved. Instead, this paper used pretrained ESM-1b, and continued pretraining based on this model, which is continual learning.\n\n4. **The paper missed some important details.**  For example, in SDF, we need the atom type for the current vertex and its neighbors. However, the author didn't mention what atoms they used. The four atoms on backbone? Or the full 44-kind atoms as in AlphaFold2?\n\n5. **Function evaluation:** It's just a suggestion. One of the most important motivation of this paper is: surface encodes the function of proteins, so involving surface can get better protein representation. Therefore, it's very natural to apply the learned protein representations to prediction the protein functions. However, the paper didn't do such kind of experiments. I suggest the author add some protein function evaluation, such as protein fitness landscape.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)