ProtGO: Function-Guided Protein Modeling for Unified Representation Learning
Authors: Bozhen Hu
Cheng Tan
Yongjie Xu
Zhangyang Gao
Jun Xia
Lirong Wu
Stan Z. Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Teacher-student
Knowledge Distillation
Protein Function
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: Protein representation learning is indispensable for various downstream applications of artificial intelligence for bio-medicine research, such as drug design and function prediction. However, achieving effective representation learning for proteins poses challenges due to the diversity of data modalities involved, including sequence, structure, and function annotations. Despite the impressive capabilities of large language models in biomedical text modelling, there remains a pressing need for a framework that seamlessly integrates these diverse modalities, particularly focusing on the three critical aspects of protein information: sequence, structure, and function. Moreover, addressing the inherent data scale differences among these modalities is essential. To tackle these challenges, we introduce ProtGO, a unified model that harnesses a teacher network equipped with a customized graph neural network (GNN) and a Gene Ontology (GO) encoder to learn hybrid embeddings. Notably, our approach eliminates the need for additional functions as input for the student network, which shares the same GNN module. Importantly, we utilize a domain adaptation method to facilitate distribution approximation for guiding the training of the teacher-student framework. This approach leverages distributions learned from latent representations to avoid the alignment of individual samples. Benchmark experiments highlight that ProtGO significantly outperforms state-of-the-art baselines, clearly demonstrating the advantages of the proposed unified framework.

ProtGO: Function-Guided Protein Modeling for Unified Representation Learning
Bozhen Hul 1,2, Cheng Tan**, Yongjie Xu², Zhangyang Gao?, Jun Xia?,
Lirong Wu", Stan Z. Li
'Zhejiang University Weettakee University
(hubozhen, tancheng, stan 2q. 1i)owesstsls edu.cn
Abstract
Protein representation learning is indispensable for various downstream applica- tions of artificial intelligence for bio-medicine research, such as drug design and function prediction. However, achieving effective representation learning for pro- teins poses challenges due to the diversity of data modalities involved, including sequence, structure, and function annotations. Despite the impressive capabilities of large language models in biomedical text modelling, there remains a pressing need for framework that seamlessly integrates these diverse modalities, particularly focusing on the three critical aspects of protein information: sequence, structure, and function. Moreover, addressing the inherent data scale differences among these modalities is essential. To tackle these challenges, we introduce ProtGO, unified model that harnesses teacher network equipped with eustomized graph neural network (GNN) and Gene Ontology (GO) encoder to learn hybrid embeddings. Notably, our approach eliminates the need for additional functions as input for the student network, which shares the same GNN module. Importantly, we utilize a domain adaptation method to facilitate distribution approximation for guiding the training of the teacher-student framework. This approach leverages distributions learned from latent representations to avoid the alignment of individual samples. Benchmark experiments highlight that ProtGO significantly outperforms state-of- the-art baselines, clearly demonstrating the advantages of the proposed unified framework.
Introduction
Proteins constitute the fundamental structural and functional components within cells and organisms and serve as indispensable biomolecules thereof. These biomolecules are composed of línear sequences of amino acids, linked together by peptide bonds, intricately folding into complex three- dimensional (3D) structures [ Recent groundbreaking advancements, exemplified by AlphaFold models (2] 3] have revolutionized protein structure prediction, leveraging artificial intelligence techniques with unprecedented accuracy. Consequently, a significant scientific challenge emerges: unravelling the intricate relationships among protein's 's sequence, structure, and function an endeavour crucial for understanding disease mechanisms 4 Protein science primarily encompasses three core types of information: sequence, structure, and function, as illustrated Figure []] There, Gene Ontology (GO) annotations offer standardized framework for delineating gene and protein functions, covering molecular functions, cellular components, and biological processes. These annotations furnish a comprehensive grasp of protein functionality across multiple dimensions and
*Equal contribution. :Correspondence: {stan. 2e...q..lllee 15)taaeslaeeeedd
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

are the purpose of a protein study. Designed to be ppecies-agnosiee the GO vocabulary spans a diverse array of biological functions, rendering applicable various biological contexts and organismsf
Protein representation learning emerges as a dynamic research domain, aiming to uncover underlying patterns within raw protein data, providing invaluable insights applicable across diverse downstream tasks [5]. Recently, protein language models (LMs) have emerged as powerful tools for processing protein sequences, showcasing their capability to learn the certain "grammar of life' from vast collections of protein sequences 6 Models such as ProtTrans 771, the ESM series 6 g+0 and
XTrimoPGLM [ leverage transformer architectures and attention mechanisms to autonomously uncover intrinsic patterns, undergoing self-supervised pre-training on extensive datasets. In contrast to sequences, protein structures entail continuous 3D coordinate data [12), necessitating distinct
modelling strategies. Graph neural network (GNN)-based models [13] 14 have been developed and adapted to represent protein 3D structures [5 16 For instance, GearNet [17] encodes both
sequential and spatial features by facilitating message passing between nodes and edges across multiple types of protein graphs.
While protein LMs and GNNs have demonstrated im- pressive performance across various protein-related tasks, such as predicting protein mutational stability and Enzyme Commission (EC) numbers [18) most of these methods overlook the utilization of func- tional information 6 77. However, integrating func- tional annotations is crucial for enhancing model ca- pabilities and unveiling the intrinsic relationships be- tween protein sequences and functions [18] [9]. Re- cent research explores token-level protein knowledge through pre-training on biomedical texts, which con-
Sequence
Structure
Function
GO- GO-BP-0006228 Iicondaatic Osiherodattas uxtivle
GO-MF-0016681
GO-CC:0031974
montrannetrioort hrmenn
Figure 1: Protein sequence, structure, and
tain sequential and biological information [9] 20 Nonetheless, the disparity between the vast number of protein sequences and the limited availability of structures and annotations presents a significant challenge (21). For instance, UniPare contains over 500 million (M) sequences [22], whereas the Protein Data Bank (PDB) houses only about 190 thousand (K) structures (23), with approximately 5M triplets in ProteinKG25 (24), comprising around 600K proteins and 50K attribute terms. This discrepancy in scale hampers the translation of the success achieved in sequence modelling into structure and function modelling. Consequently, enhancing the integration and fusion of information from these three modalities poses a crucial and formidable challenge in protein science.
function.
Considering the disparity in data categories and scales between protein sequences, structures, and functions, we introduce ProtGO, comprehensive multimodal framework for protein representation learning. ProtGO leverages a teacher model to glean insights from triplets comprising sequences, structures, and functions, distilling this knowledge to guide the training of the student model. However, given that functional annotations are lacking for the vast majority of sequenced proteins [25 26) such information may not always be available for the teacher model in downstream tasks. Consequently, opt to train a sequence-structurg student model, which can be readily applied to various downstream tasks, with the teacher model serving solely provide functional knowledge to the student model. To facilitate knowledge transfer from teacher to student, we employ domain adaptation techniques to align the distributions of latent spaces between the teacher and student models. Specifically, we minimize the Kullback-Leible (KL) divergence to mitigate distribution discrepancies between the teacher and student domains.
The contributions of this paper can be summarized: 1) We propose ProtGO to integrate multimodal information about proteins, encompassing sequence, structure, and functions. The teacher-student framework enables the learning of unified representations suitable for diverse downstream tasks where protein model is needed. 2) We pioneer the adaptation of knowledge distillation methods to connect protein teacher-student networks, infusing functional information into student representations through distribution approximation and domain adaptation. 3) We validate ProtGO by outperforming prevailing protein representation methods across a range of tasks, including protein fold prediction, enzyme reaction classification, GO term prediction, and EC number prediction.

---

2 Related Work
2.1 Representation Learning for Protein
Unimodal Protein Representation Learning. There are two primary categories of methods in unimodal protein representation learning: sequence-based and structure-based approaches. Sequence- based methods aim to derive representations directly from amino acid sequences [27]. with notable efforts focused on enhancing model sizes or scaling datasets 7 TO 37 For instance, Chen it al.J proposed xTrimoPGLM, a unified model capable of learning from protein sequences to address both protein understanding and generation tasks concurrently, boasting staggering 100 billion (B) parameters and 1 trillion (T) training tokens. In addition to sequence-basee approaches, structure- based encoders have emerged to leverage the 3D structural information of proteins. For example, IEConv [3] accommodates the inherent inductive bias in protein structure modelling by introducing graph convolution layer that integrates intrinsic and extrinsic distances between nodes. ProNet [3 provides geometric representations across multiple levels of structure granularity.
Multimodal Protein Representation Learning. In the realm of protein multimodality learning, methodologies such as GearNet [17 have been devised to concurrently exploit both sequence and structural information. GearNet represents sequential and geometric features as graph node and edge features, employing a mnessage-passing mechanism to encode them 33]. CDConv [12] introduces continuous-disee convolution to model sequential and geometric features. Acknowledging the -qquiaaqui properties in protein structures, equivariant and invariant features are tailored as model inputs B3 34] Furthermore, the integration of factual biological knowledge has demonstrated enhancements in pre-trained LMs protein sequences (24. KeAP [19] and ProtST [20] train biomedical LMs using masked language modelling (35] as the pretext task. Particularly noteworthy,
MASSA [8] initially derives sequence-structure embeddings from existing pre-trained models
10
33
subsequently aligning them globally with GO embeddings using five pre-training objectives.
2.2
Knowledge Distillation
Knowledge distillation involves the process of transferring knowledge from larger teacher model to a smaller student model [36 37]. Significant advancements have been made in graph-based knowledge distillation, leading the development of various methodologies 38 39). RDD (40]
mandates the student model to faithfully replicate the complete node embeddings of the teacher, ensuring the transfer of more informative knowledge. Another notable approach, GraphAKD [4]], employs adversarial learning to distill node representations from the teacher to the student. This method effectively distills knowledge from both local and global perspectives, exhibiting superior performance compared to earlier graph distillation techniques [42
2.3 Domain Adaptation
Domain adaptation aims to develop a model from labelled data in a source domain that can be effectively applied to target domain by minimizing dissimilarities between their distributions [43] 45). Methods for distribution alignment focus on reducing disparities in both marginal and conditional representation distributions between the source and target domains [46 47). Adversarial learning techniques have demonstrated remarkable effectiveness mitigating the discrepancy between source and target domains [48)5 Semi-supervised domain adaptation strategies aim to minimize the source-target gap with limited labelled target data (5]55) In our work, we apply domain adaptation methods to align the distributions of representations learned by teacher and student networks to avoid dependence on individual samples.
Method
3.1 Preliminaries
Here, we provide definitions and relevant notations for the problem, and background knowledge of the local coordinate system (LCS).
3

---

Function
Teacher (source domain)
Enceder
GO-BP: 8006228
Clasification
GO-ME:
La
GO-CC: 0031974
Sequeece
Pilza Gg.A)
Structure
Poeling
Domain Adaptation
GNN
Sequeace
(Pa)
Strueture
Pooling
G2
Message passing
Message passing
Student (target domain)
Classification Heal
The overall framework of ProtGO consists of two branches: teacher model in the source domain and student model in the target domain, connected by knowledge distillation loss.
Figure 2:
Problem Statement. Mathematically, we represent a protein graph as G = EV,,, X, E), where {mi))iml,....>> and & = {Eeg (éij);.j=1,.... denote the vertex and edge of residues, respectively. We denote the position of residue by the coordinate of Ca. the collection by the position matrix is denoted as P = |- where P. E g3x1, The node and edge feature matrices are X = [ and E leigle.=z1..... the feature vectors of node and edge are E Rd and ei E I2 d1 and d, are the initial feature dimensions. The set of & GO annotations for proteins denoted as A {A: k==..... where A, {0, 1} the indicator for annotation The goal of protein graph representation learning is to find Now-dimensiona. embedding for each protein.
There is source domain S for the teacher model with the data distribution ps Gs, A) in the latent space, and there is also a target domain for the student model with the data distribution Pr (2T (G) in the latent space. 2g, 27 latent embeddings from the teacher and student networks for protein graphs Gs and Gr.
Local Coordinate System. In order avoid the usage of complicated -E(3)-equivariant models, the invariant and locally informative features are developed from the LCS [55), which is defined as:
O bb; n
b
n4.
(1)
P.-P.-1 b; where u, 1|P-PP ill I 11ll
u: 4.41
u. M
by is the negative bisector of the angle
|| |
between the rays (P._1 P.) and (P.+1 P.).
P. O -O))
eij = Concat (I P31. 1711,07 Of [P-
(2)
Note that the edge feature vector Cij is the concatenation the geometric features for protein 3D structures, including distance, direction, and orientation, where ||-|| denotes the 1²-norm.
3.2 Overall Framework
The overall framework of ProtGO is illustrated in Figure 2 It consists of two branches that train teacher model and student model via iterative knowledge distillation. Compared to the student, the teacher has an additional annotation encoder module comprised of several fully connected layers. This transforms GO annotations into functional embeddings, combined with sequence-struce struccure embeddings from the GNNs to form the final kmowledge-enhaneee embeddings 23. Previous works have successfully utilized label-augmented techniques to enhance model training [56. 57). This technique involves encoding labels and combining them with node attributes through concatenation or summation. By doing so, improves feature representation and enables the model to effectively utilize valuable information from labels. Importantly, instead of directly minimizing the distances

---

between sample-dependent embeddings, denoted as and 2r, we introduce a sample-independent method. This is uccomplished by aligning the latent space of the student with that of the teacher, achieved through the approximation of distributions of embeddings from both networks. This distribution alignment approach avoids dependence on individual sample inputs. It is noteworthy that our primary objective is to derive comprehensive embeddings for the student model, with less emphasis on the training specifics of the teacher model. Consequently, the teacher model can be trained on multiple datasets, whereas the student does not need to have access to the same datasets when data modalities are not available.
Protein Graph Message Passing. protein sequence consists of n residues, which are deemed as graph nodes. We concatenate the one-hot encoding of residue types with the physicochemical proper- ties of each residue, namely, a steric parameter, hydrophobieity, volume, polarizability, isoelectric point, helix probability, and sheet probability [58 [5], which are used as the graph node features æ;, These node features capture meaningful biochemical characteristics, enabling the model to learn which residues tend to be buried, exposed, tightly packed, etc. We define the sequential distance, ly = ||i jIl, and spatial distance di = ||P Pyll. where P, is the coordinate of the C2 atom of the i-th residue. An edge Egj exists if:
by < and dig <r.
where L,y r, are predefined radius thresholds, ex consists of geometric features of the protein structure, defined in Eq. 2 We convolve node and edge features from sequence and structure simultancously and formulate the message passing mechanism as:
h (0) BN (FC (x:)),
(1) o (BN( E Weihh (1-1)
"
(4)
(.E)((
h h + Dropont(FC((u"))
This mechanism (as shown in Eq. can fuse and update the node and edge features, which includes aggregation and update functions, where FC(.), (N(-), Dropout(-) represent fully connected, batch normalization, and dropout layers, (-) is the activation function LeakyReLU and W is the learnable convolutional kernel. (u;) refers the neighbors of node u; and h the representation of node v in the 1-th message passing layer. The node and edge features are processed together in Eq. [] After message passing operations, sequence pooling layer is applied reduce the sequence length, providing a simple but effective way to aggregate key patterns. After average pooling, the residue number halved; we expand the radius rs 2r, to update the edge conditions and perform the message passing and pooling operations again. These operations can make the GNNs cover more distant nodes gradually. The teacher and student models share the same GNN architecture to process protein sequences and structures. Finally, global pooling layer is applied to obtain the graph-level protein embeddings, denoted as hgg and 27 for the teacher and student. Detailed model descriptions presented in Appendix E.
Protein Domain Adaption. As shown in Figure 2 the teacher model consists of GNNs and an auxiliary annotation encoder, which a multi-layer perceptron (MLP) that provides imcctio-frrenndl protein representations. The annotations associated with Gs serve as the input for the annotation encoder, resulting in the extraction of feature vector hA Therefore, we can combine h and the graph-level protein embeddings hy learned from Gs together:
h MLP((A)
(5)
z5 hA + odi
where a is a hyper-parameter, controlling the balance between the contribution of the annotation embeddings æA and the protein embeddings hg in the combined representations.
As depicted Figure 2 the generated protein embeddings 2ş contain sequence, structure, and func-
tion information, guiding the training of the student model. Since Knowledge-enhaneed embeddings zs are intended be aligned with 27, we obtain 25 from the entire protein and GO term datasets to avoid dependence individuals. Then, we calculate the distributions of 2g and zr to better capture
5

---

the inherent uncertainty in the teacher's 's and student's 's latent spaces, in which the real distributions are sample-independent. The minibatch is adopted to approximate the quantities ps(2s) and pr (2r):
ps(=s)
Eps(Gs (p(=s Gs. A)]
Bg
ps(Es|
A
Bs
(6)
(T(2T)
(Gr) (2T
(2r(2TGTT GT)]
Bs pr (2T )
Bs
where Bg is the batch size. Gaussian distribution θ is assumed for protein embeddings, which exhibit smoothness and symmetry properties that reasonably mimic the expected continuity and unimodality of the embeddings aggregated over many residues. We employ the reparameterization trick [60) sample the protein embeddings.
ps(=s) = 03) pT(2T) = ((T,,o))
where μss o3 and pe,, of are the mean and variance values of the embeddings for the teacher and student models, providing summary of the distribution using first- and second-order statistics.
shows that the conditional misalignment in the representation space bounded by the conditional misalignment in the input space. We have:
Proposition 2 in Appendix [
M V2
Sundent S Lreacher +
(8)
C V KT. (ps(2) || Pr())] + Bps(C) [KT (ps((G) pr(])))]
(9)
where Suudet is the ideal target domain loss, and Lreacher is the teacher's supervised loss, M is bound, see Appendix F (p) [KL [ps G) || pT(y]G)]] often small and fixed (not dependent on the representation 2, and y is the function label). To reduce the generalization bound, we can focus (3:
on optimizing the marginal misalignment with a hyper-paramee
Lreacher + (KL (ps(2) || (=)])
(10)
Eq. can be used in an unsupervised way for the student to predict functions, which near the ideal target domain loss. For the proposed framework, ProtGO (shown in Figure 2). train the teacher model. When the student model has task labels, we adopt a hybrid loss L to train the student model, where the Led KL (2) [p))2] is optimize the marginal misalignment between teacher and student models. Therefore, the final loss L. with a hyper-paramedt ,3 for the student model is formulated as:
the Leacher to
L Lstdeent + 3Ldd
(11)
The objective function of the teacher model Preacher is the cross entropy loss for protein graph classification. The hybrid loss for the student model has a cross entropy loss Sauuennt for classification and regularization loss Laa for domain-adapted knowledge distillation.
4 Experiments
4.1 Training Details
The proposed multimodal knowledge distillation framework, ProtGO, undergoes an easy training process. A dataset comprising approximately 30,000 proteins, each associated with 2,752 GO annotations from the GO dataset, is utilized without further categorization into biological process (BP), molecular function (MF), and cellular component (CC) classes [67]. These classes serve as input to the annotation encoder the teacher model, yielding an overall Faxx 0.489 for the teacher model. Subsequently, the student model is trained. The optimization is performed using the Adam optimizer through the PyTorch library, and the performance metrics are computed as mean values over three initializations. Further details regarding experimental settings available in Appendix E.2

---

Table 1: Accuracy (%) of fold classification and enzyme reaction classification. The best results shown in bold.
Fold Classification
Enzyme
Modality (Input)
Method
Fold SuperFamily Family
Reaction
CNN 61
11.3
13.4
53.4
51.7
ResNet [27
10.1
7.21
23.5
24.1
Sequence
LSTM 27
6.41
4.33
18.1
11.0
Transformer 27
9.22
8.81
40.4
26.6
GCN 662
16.8
21.3
82.8
67.3
GAT 663
12.4
16.5
72.7
55.6
Structure
3DCNN_MQA 64
31.6
45.4
92.5
72.2
GraphQA
65
23.7
32.5
84.4
60.8
GVP [66]
16.0
22.5
83.8
63.5
ProNet-Amino Acid
32
51.5
69.9
99.0
86.0
ProNet-Backbome
32
52.7
70.3
99.3
86.4
ProNet-All-Atom
32
52.1
69.0
99.0
85.6
CRL [6
47.6
70.2
99.2
87.2
Sequence-Structure
GearNet 17
28.4
42.6
95.3
79.4
GearNet-IEConv
17
42.3
64.1
99.1
83.7
GearNet-Edge 17
44.0
66.7
99.1
86.6
GearNet- Eddeeet-ddg--ECoon
17
48.3
70.3
99.5
85.3
CDConv 12
56.7
77.7
99.6
88.5
ProtGO (Student)
60.5
79.4
99.8
89.4
4.2 Tasks and Baselines
Following the tasks in ΙEсоv 31 and CDConv 12). evaluate ProtGO on four protein tasks: protein fold classification, enzyme reaction classification, GO term prediction, and EC number prediction. Detailed task descriptions are presented in Appendix B Dataset statisties shown in Table 4 in the appendix.
Baselines. The proposed method is compared with existing protein representation learning methods, which are classified into three categories based on their inputs, which could be a se-
quence, 3D structure, both sequence and structure. 1) Sequence-based encoders, including
CNN [61] ResNet (27), LSTM [27] and Transformer (27). 2) Structure-based methods (GCN [62
GAT 663 3DCNN_MO
64
Sequence-structure based models, e.g., GVP (66), CRL [16]
ProNet 33] GearNet [17. CDConv 12), etc. GearNet-IEECon and GearNetEEdge-EEComm
[7 add
the IEConv layer on GearNet.
4.3 Results of Fold and Enzyme Reaction Classification
Table presents a performance comparison for protein fold and enzyme reaction prediction across various methods, with results reported average values. As depicted in the table, the proposed ProtGO consistently achieves the highest performance across all four test sets for both fold and reaction prediction tasks. The superiority of eqquence-structur--bbese structure-ong approaches is evident, underscoring the advantages of jointly modelling sequence and structure information. ProtGO achieves a remarkable improvement in accuracy of over 6.7% compared to prior techniques on the Fold test set, highlighting its efficacy in learning mappings between sequence, structure, and function. Furthermore, despite both CDConv and ProtGO employing sequence-structur convolution architectures, ProtGO demonstrates superior performance over the CDConv model. This observation suggests that the teacher-student training paradigm adopted in ProtGO facilitates the acquisition of enhanced protein embeddings by the student model.
methods over sequence- or

---

Table 2: Fax of GO term prediction and EC number prediction. The best results shown in bold.
Modality (Input)
Method
GO-BP GO-MF GO-CC EC
CNN 61
0.244
0.354
0.287
0.545
ResNet 2
0,280
0.405
0.304
0.605
Sequence
LSTM 27
0.225
0.321
0.283
0,425
Transformer 127
0.264
0.211
0,405
0.238
GCN 62
0.252
0.195
0.329
0.320
Structure
GAT 63
0.284
0.317
0.385
0.368
3DCNN_MON 64
0.240
0.147
0.077 0.305
GraphQA 65
0.308
0.329
0.413
0.509
GVP 66
0.326
0.426
0.420
0.489
CRL 16
0.421
0.624
0.431
GearNet 17
0.356
0.503
0.414
0.730
Sequence-Structure
GearNet-IEConv 17
0.381
0.563
0.422
0.800
GearNet-Fdge
0.403
0.580
0.450
0.810
Edge-Fon-dg
17
0.400
0.581
0.430
0.810
CDConv
[1]
0,453
0.654
0.479
0.820
ProtGO (Student)
0.464
0.667
0.492
0.857
4.4 Results of GO Term and ECC Number Prediction
Following the protocol in GearNet 17 the test sets for GO term and EC number prediction only contain PDB chains with less than 95% sequence identity the training set, ensuring rigorous evaluation. The student model conducts the experiments, and the teacher model's annotations are not classified into these classes, avoiding data Icakage. Table (Jswos comparative results between different protein modeling methods these tasks, with performance measured by F Faxx which balances precision and recall, working well even if positive and negative classes are imbalanced. The mean values of three independent all test sets for both GO and EC prediction, outperforming other approaches. This demonstrates ProtGO's strong capabilities for predicting protein functions and activities. Overall, the consistent improvements verify the benefits of injecting functional information into sequence-strucnt models, done in ProtGO's teacher-student framework. The results cement ProtGO's effectiveness using knowledge distillation techniques.
are reported. ProtGO achieves the highest F max across
(a) KL training loss of fold (b) Accuracy of fold clas-(c) KL training loss of EC (d) F Fmax of EC number classification
sification
number prediction
prediction
Figure 3: The KL training loss curves (a), (c) and test performance (b), (d) on the tasks of fold classification and EC number prediction. The red curve denotes that Lua conducts its function, while the green curve denotes calculated the value of Lia, but it is not involved in the process of the gradient backpropagation (BP).
4.5 Ablation Study
Table
presents ablation studies of the proposed ProtGO model on the four downstream tasks. We examine the impact of removing the teacher model, which means removing the Lea- We also remove the annotation encoder the teacher, which means that we incorporate function information into the loss function for the teacher models. As shown in Table 3 removing the teacher model altogether

---

Table 3: Ablation experiments of proposed method. w/o AE-T denotes without the annotation encoder in the teacher model. w/o teacher means without the teacher model and directly using the student model, which also means without Lkd-
Fold Classification
Enzyme
GO
Method
EC
Fold
Superfamily
Family
Reaction
BP
MF
CC
ProtGO
60.5
79.4
99.8
89.4
0.464
0.667
0.492
0.857
w/o AE-T 60.4
79.1
99.7
88.9
0.454
0.664
0.490
0.854
w/o Teacher
57.8
78.7
99.6
88.6
0.458
0.660
0.484
0.851
(w/o Teacher) leads to substantial performance drops across all tasks compared to the full ProtGO. This shows the teacher's knowledge distillation provides useful signals for the student model. Besides, removing the annotation encoder in the teacher (w/o AE-T) also degrades performance, though less severely. Despite being a label-augmented strategy, the annotation encoder exhibits minimal influence, indicating low sensitivity and limited impact on test performance. Our student model is specifically designed to process protein sequences and structures as inputs, enabling function independently without the need for guidance from the teacher model.
Figure 3 illustrates the comparisons of the knowl- edge distillation loss Lkd with and without involve- ment in backpropagation during training. When LEd is excluded from the gradient backpropagation process, exhibits decrease alongside the clas- sification loss Ltudent- However, its value remains substantially higher compared to when Lkd is in- cluded in the training process. Similar observations are presented for the accuracy and F max on the fold classification and EC number prediction. The no- table disparity observed between the distillation loss and the test performance with and without involve- ment in backpropagation suggests that the KI. loss does indeed play significant role in guiding the student model's learning process. Its presence influ- ences the model's performance and convergence.
G0-8P
GO
60-0C
Figure 4: Performance comparisons of the teacher and student of ProtGO GO term prediction.
We compare the performance of the teacher and the student on the tasks of GO term prediction. From the provided Figure it is evident that incorporating functional information as the input the annotation encoder signihcantly enhances performance. particularly for MF and CC term prediction. These two classes have fewer categories and are more accessible, resulting higher scores.
Conclusion
In this paper, we proposed ProtGO, a multimodal protein representation learning framework inte- grating information from protein sequences, structures, and function annotations. While the teacher network requires extra information about function annotations as input, such is not always avail- able for the student model. The student model amends the problem by mimicking the behavior or predictions of the teacher model. Our main focus is to obtain comprehensive embeddings for our primary concern.
the student model, whereas the complete training of the teacher model We estimate the latent embedding distributions for the teacher-studen model and learn annotation- enriched student representations by distribution approximation. Compared to the mainstream protein representation learning techniques. ProtGO achieves superior performances in predicting protein families, reactions, GO terms, and EC numbers. These consistent improvements across benchmarks highlight the advantages of this approach for informative protein representation learning. A limitation that this base model is not pre-trained on large-scale datasets. One way improvement is to integrate ProtGO with other pre-training strategies (refer to Appendix G). This would need to manage the computational resources required for training and inference and ensure compatibility between different model architectures and training objectives.

---

Acknowledgements
This work was supported by the National Science and Technology Major Project (No. 15101), the National Natural Science Foundation of China Project (No. U21A20427 and No. 623B2086), Project (No. WU2022A009) from the Center of Synthetic Biology and Inte- grated Bioengineering of Westlake University and Project (No. WU2023C019) from the Westlake University Industries of the Future Research Funding.
References
[1] Bozhen Hu, Jun Xia, Jiangbin Zheng, Cheng Tan, Yufei Huang, Yongjie Xu, and Stan Z Li. Protein language models and structure prediction: Connection and progression. arXiv preprint arXiv:2211.16742. 2022.
[2] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool. Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873) 2021.
[3] Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, pages 1-3, 2024.
[4] Onur Serçinoglu and Pemra Ozbek. Sequence-sttncture-fntiiiin relationships in class mhc: A local frustration perspective. PloS one, 15(5)=00232849, 2020.
[5] Serbulent Unsal, Heval Atas, Muammer Albayrak, Kemal Turhan, Aybar C Acar, and Tunca Dogan. Learning functional properties of proteins with language models. Nature Machine Intelligence, 4(3):227-245, 2022.
[6] Zeming Lin et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. BioRxiv, 2022:500902, 2022.
[7] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang. Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, al. Prottrans: Toward understanding the language of life through self-supervised learning. IEEE transactions pattern analysis and machine intelligence, 44(10)7112-7127, 2021.
[8] Alexander Rives al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15)ee01623311 18, 2021.
[9] Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. In International Conference on Machine Learning, pages 8844-8856. PMLR, 2021.
[10] Roshan M Rao, Joshua Meier, Tom Sercu, Sergey Ovchinnikov, and Alexander Rives. Trans- former protein language models are unsupervised structure learners. bioRxiv, 2020. doi: 1..1100/2200..1111 15.15.227.. URL URLhtttps://wet httpp://www.biooxivv http://wwww..booooooooooootttttt 12.15. 422761vv
[11] Bo Chen, Xingyi Cheng, Pan Li, Yangli-ao Geng, Jing Gong, Shen Li, Zhilei Bei, Xu Tan, Boyan Wang, Xin Zeng, et al. xtrimopglm: unified 100b-scale pre-trained transformer for deciphering the language of protein. arXiv preprint mrXiv:2401.01/1 2024.
[12] Hehe Fan, Zhangyang Wang, Yi Yang, and Mohan Kankanhalli. Contimuous-discrete convolu- tion for geometry-sequence modeling in proteins. In The Eleventh International Conference on Learning Representations, 2023.
[13] Yue Liu, Ke Liang, Jun Xia, Sihang Zhou, Xihong Yang, Xinwang Liu, and Stan Li. Dink-net: Neural clustering large graphs. In Proc. of ICML, 2023.
10

---

[14] Lirong Wu, Haitao Lin, Bozhen Hu, Cheng Tan. Zhangyang Gao, Zicheng Liu, and Stan Z Li. Beyond homophily and homogeneity assumption: Relation-based frequency adaptive graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[15] Federico Baldassarre, David Menéndez Hurtado, Arn Elofsson, and Hossein Azizpour. Graphqa: protein model quality assessment using graph convolutional networks. Bioin- formatics, page 360-366, Apr 2021. doi: 10. 1001999/ooiifommaii///baa71... URL http 7/dx dolxxdoi.oo orgi..org/10 100g/10..0 1ww./000iinneiee htt./////aaoooomtatata/////aaa/a
[16] Pedro Hermosilla and Timo Ropinski. Contrastive representation learning for 3d protein structures, arXiv preprint arXiv:2205 2022.
[17] Zuobai Zhang, Minghao Xu, Arian Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Protein representation learning by geometric structure pretraining. In International Conference Learning Representations, 2023.
[18] Fan Hu, Yishen Hu, Weihong Zhang, Huazhen Huang, Yi Pan, and Peng Yin. A multimodal protein representation framework for quantifying transfcrability across biochemical downstream tasks. Advanced Science, page 2301223,, 2023.
[19] Hong- Yu Zhou, Yunxiang Fu, Zhicheng Zhang, Cheng Bian, and Yizhou Yu. Protein rep- resentation learning via knowledge enhanced primary structure modeling. arXiv preprint arXiv:2301.13335 13154, 2023.
[20] Minghao Xu, Xinyu Yuan, Santiago Miret, and Jian Tang. Protst: protein sequences and biomedical texts. In International Conference on Machine Learning, pages 183749-87766 PMLR, 2023.
learning of
[21] Michael Ashburner, Catherine A. Ball, Judith A. Blake, David Botstein, Heather Butler, J. Michael Cherry, Allan P. Davis, Kara Dolinski, Selina S. Dwight, Janan T. Eppig. Mi- dori A. Harris, David P. Hill, Laurie Issel-Tarver, Andrew Kasarskis, Suzanna Lewis, John C. Matese, Joel E. Richardson, Martin Ringwald. Gerald M. Rubin, and Gavin Sherlock. Gene ontology: tool for the unification of biology. Nature Genetics, page 25-29, May 2000. doi: 10.1038/75556. http://:://dx doi. org/10. 1038/75556
[22] UniProt Consortium. Update on activities at the universal protein resource (uniprot) in 2013. Nucleic acids research, 41(D1))D43-D47, 2012.
[23] Helen M. Berman, John D. Westbrook, Zukang Feng, Gary L. Gilliland, Talapady N. Bhat, Helge Weissig, Ilya N. Shindyalov, and Philip E. Bourne. The protein data bank. Nucleic Acids Research, 2000.
[24] Ningyu Zhang, Zhen Bi, Xiaozhuan Liang, Siyuan Cheng, Haosen Hong, Shumin Deng, Jiazhang Lian, Qiang Zhang, and Huajun Chen. Ontoprotein: Protein pretraining with gene ontology embedding. arXiv preprint arXiv:2201.111477 2022.
[25] Mateo Torres, Haixuan Yang, Alfonso E Romero, and Alberto Paccanaro. Protein function prediction for newly sequenced organisms. Nature Machine Intelligence, 3(12):0050-106, 2021.
[26] Nabil Ibtehaz, Yuki Kagaya, and Daisuke Kihara. Domain- allows protein function prediction using function-awand domain embedding representations. Communications Biology, 6(1):1103 2023.
[27] Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer learning with tape. Advances in neural information processing systems, 32, 2019.
[28] Erik Nijkamp, Jeffrey Ruffolo, Eli N Weinstein, Nikhil Naik, and Ali Madani. Progen2: exploring the boundaries of protein language models. Cell systems, 14(11):968-8 -7788897,, 2023.
[29] Noelia Ferruz, Steffen Schmidt, and Birte Höcker. Protgpt2 is a deep unsupervised language model for protein design. Nature Communications, Jul 2022. doi: 10. 10.10088/444447-0222200001
URL utLhttt:: http:hhp/// //dx doi. org/ 10. 00008///114477-223220000

---

[30] Michael Heinzinger. Maria Littmann, lan Sillitoe, Nicola Bordin, Christine Orengo, and Burkhard Rost. Contrastive learning on protein embeddings enlightens midnight zone, NAR genomics and bioinformaticx, 4(2):lqac043, 2022.
[31] Pedro Hermosilla, Marco Schäfer, Matëj Lang, Gloria Fackelmann. Pere Pau Vázquez, Barbora Kozlíková, Michael Krone, Tobias Ritschel, and Timo Ropinski. Intrinsic-extrinsic convolution and pooling for learning on 3d protein structures. International Conference on Learning Representations, 2021.
[32] Limei Wang. Haoran Liu, Yi Liu, Jerry Kurtin, and Shuiwang Learning hierarchical protein representations via complete 3d graph networks. In The Eleventh International Conference on Learning Representations, 2023.
[33] Bowen Jing, Stephan Fismann, Patricia Suriana, Raphael JL. Townshend, and Ron Dror. Learn- ing from protein structure with geometric vector perceptrons. arXiv preprint 1rXiv:2009.01411, 2020.
[34] Yuzhi Guo, Jiaxiang Wu, Hehuan Ma, and Junzhou Huang. Self-supervised pre-training for protein embeddings using tertiary structures. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6801-6809, 2022.
[35] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04.05 2018.
[36] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint
2015.
[37] Ke Liang, Lingyuan Meng, Meng Liu, Yue Liu, Wenxuan Tu, Siwei Wang, Sihang Zhou, Xinwang Liu, Fuchun Sun, and Kunlun He. A survey of knowledge graph reasoning on graph types: Static, dynamic, and multi-modal. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[38] Jing Liu, Tongya Zheng, Guanzheng Zhang, and Qinfen Hao. Graph-based knowledge distilla- tion: A survey and experimental evaluation. arXiv preprint arXiv:2302.14644 2023.
[39] Yijun Tian, Chuxu Zhang, Zhichun Guo, Xiangliang Zhang, and Nitesh V Chawla. Nosmog: Learning noise-robust and structure-aware mips on graphs. arXiv preprint arXiv:2208. 10010, 2022.
[40] Wentao Zhang, Xupeng Miao, Yingxia Shao, Jiawei Jiang, Lei Chen, Olivier Ruas, and Bin Cui. Reliable data distillation on graph convolutional network. In Proceedings of the 2020 ACM SIGMOD International Conference Management of Data, pages 1399-1414, 2020.
[41] Huarui He, Jie Wang, Zhanqiu Zhang, and Feng Wu. Compressing deep graph neural networks via adversarial knowledge distillation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 534-544, 2022.
[42] Ruifei He, Shuyang Sun, Jihan Yang, Song Bai, and Xiaojuan Qi. Knowledge distillation as efficient pre-training: Faster convergence, higher data-efficiency, and better transferability. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 9161-9171, 2022.
[43] Abolfazl Farahani, Sahar Voghoei, Khaled Rasheed, and Hamid R Arabnia. A brief review of domain adaptation. Advances in data science and information engineering: proceedings from ICDATA 2020 and IKE 2020, pages 877-894, 2021.
[44] Garrett Wilson and Diane Cook. survey of unsupervised deep domain adaptation. ACM Transactions on Intelligent Systems and Technology (TIST), 11(5):1-46, 2020.
[45] Mei Wang and Weihong Deng. Deep visual domain adaptation: survey. Neurocomputing, 312:135-155 2018
12

---

[46] A Tuan Nguyen, Toan Tran, Yarin Gal, Philip HS Torr, and Atlim Günes Baydin. Kl guided domain adaptation. arXiv preprint arXiv:2106.07788 2021.
[47] Mingsheng Long, Yue Cao, Jianmin Wang. and Michael Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pages 90-105. PMLR, 2015.
[48] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pages 1180-1189. PMLR, 2015.
[49] Mingsheng Long, Zhangjie Cao, Jianmin Wang. and Michael I Jordan. Conditional adversarial domain adaptation. Advances in neural information processing systems, 31, 2018.
[50] Zhongyi Pei, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Multi-adversarial domain adaptation. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[51] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, Trevor Darrell, and Kate Saenko. Semi-supervised domain adaptation via minimax entropy. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8050-8058, 2019.
[52] Taekyung Kim and Changick Kim. Attract, perturb, and explore: Learning feature alignment network for semi-supervised domain adaptation. Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 591-607. Springer, 2020.
[53] Pin Jiang, Aming Wu, Yahong Han, Yunfeng Shao, Meiyu Qi, and Bingshuai Li. Bidirectional adversarial training for semi-supervised domain adaptation. In IJCAI, pages 934-940, 2020.
[54] Can Qin, Lichen Wang, Qianqian Ma, Yu Yin, Huan Wang, and Yun Fu, Contradictory structure learning for sepi-supervised domain adaptation. In Proceedings of the 2021 SIAM International Conference Data Mining (SDM), pages 576-584. SIAM, 2021.
[55] John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph-based protein design. Advances in neural information processing systems, 32, 2019.
[56) Samy Bengio, Jason Weston, and David Grangier. Label embedding trees for large multi-class tasks. Advances in neural information processing systems, 23, 2010.
[57] Xu Sun, Bingzhen Wei, Xuancheng Ren, and Shuming Ma. Label embedding network: Learning label representation for soft training of deep networks. arXiv preprint arXiv: /710. 10393, 2017.
[58] Gang Xu, Qinghua Wang. and Jianpeng Ma. Opus-rota4: gradient- based protein side-chain modeling framework assisted by deep learning-based predictors. Briefings in Bioinformatics, 23( 23(1):bbab529, 2022.
[59] Jack Hanson, Kuldip Paliwal, Thomas Litfin, Yuedong Yang, and Yaogi Zhou. Improving predic- tion of protein secondary structure, backbone angles, solvent accessibility and contact numbers by using predicted contact maps and an ensemble of recurrent and residual convolutional neural networks. Bioinformatics, 35(14):2403-2410 2019.
[60] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 1rX1v:1312.6114, 2013.
[61] Amir Shanehsazzadeh, David Belanger, and David Dohan. Is transfer learning necessary for protein landscape prediction? arXiv preprini arXiv:2011.0344 2020.
[62] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. ICLR, 2017.
[63] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio, al. Graph attention networks. In ICLR, 2018.
[64] Georgy Derevyanko, Sergei Grudinin, Yoshua Bengio, and Guillaume Lamoureux. Deep convolutional networks for quality assessment of protein folds. Bioinformatics, 34(23):4046 4053, 2018.
13

---

[65] Federico Baldassarre, David Menéndez Hurtado, Ame Elofsson, and Hossein Azizpour. Graphqa: protein model quality assessment using graph convolutional networks. Bioinfor- maticy, 2020.
[66] Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael JL Townshend, and Ron Dror. Learn- ing from protein structure with geometric vector perceptrons. arXiv preprint arXiv:2009.01411, 2020.
[67] Vladimir Gligorijević, Douglas Renfrew, Tomasz Kosciolek, Julia Koehler Leman, Daniel Berenberg, Tommi Vatanen, Chris Chandler, Bryn C Taylor, Ian M Fisk, Hera Vlamakis, al. Streeture-based protein function prediction using graph convolutional networks. Nature communications, 126))3368, 2021.
[68] Jie Hou, Badri Adhikari, and Jianlin Cheng. Deepsf: deep convolutional neural network for mapping protein sequences folds. Bioinformatics, 34(8): -90121235-133 2018.
[69] Edwin Webb al. Enzyme nomenclature 1992. Recommendations of the Nomenclature Com- mittee of the International Union of Biochemistry and Molecular Biology the Nomenclature and Classification of Enzymes. Number Ed. 6. Academic Press, 1992.
[70] Marina Omelchenko, Michael Y Galperin, Yuri Wolf, and Eugene Koonin. Non- homologous isofunctional enzymes: systematic analysis of alternative solutions in enzyme evolution. Biology direct, 5:1-20, 2010.
[71] Helen M Berman, John Westbrook, Zukang Feng, Gary Gilliland, Talapady N Bhat, Helge Weissig, Ilya N Shindyalov, and Philip Bourne. The protein data bank. Nucleic acids research, 28(1):235-242, 2000.
[72] UniProt Consortium. Uniprot: a worldwide hub of protein knowledge. Nucleic acids research, 47(D1):D506-D515 2019.
[73] Bozhen Hu, Cheng Tan, Jun Xia, Jiangbin Zheng, Yufei Huang, Lirong Wu, Yue Liu, Yongjie Xu, and Stan Z Li. Learning complete protein representation by deep coupling of sequence and structure. bioRxiv, pages 2023-07, 2023.
[74] Microsoft. Neural Network Intelligence, 2021. URL httptp:://ggthuu..co///.ccc/// nni
[75] Maxat Kulmanov al. Protein function prediction as approximate semantic entailment. Nature Machine Intelligence, 6(2):220-228, 2024.
(76] Youhan Lee, Hasun Yu, Jaemyung Lee, and Jaehoon Kim. Pre-training sequence, structure, and surface features for comprehensive protein representation learning. In ICLR, 2023.
[77] Z Zhang, C Wang, M Xu, V Chenthamarakshan, AC Lozano, Das, and J Tang. A systematic study of joint representation learning protein sequences and structures. Preprint at htpp:/arxxiv 0rg/abs/23003.0225,, 2023.
14

---

A Social Impact
Our proposed framework, ProtGO, enable advanced protein analyses and provide effective and comprehensive representations that incorporate the information from protein sequences, structures, and functions. However, there may exist broader impacts and harmful activities. In detail, the representations are potentially misused, e.g., designing harmful molecules or proteins based on these representations. Wet lab experiments may be needed for the newly found mechanisms or functions of proteins based on the learned representations.
B Task
Fold Classification. In order understand how protein structure and evolution interact, it is crucial to be able to predict fold classes [68]. This dataset contains 16,712 total proteins across 1,195 fold classes. Three test
provided. Fold: proteins from the same superfamily are excluded during training; SuperFamily: proteins from the same family are not used for training; and Family: the training set includes proteins from the same family.
Enzyme Reaction Classification. Enzyme reaction classification can be viewed as a protein function prediction task based the enzyme-catalyzed reactions defined by the four levels of enzyme commission numbers 669 70]. We use the dataset [31] 71 containing 29,215 training proteins, 2,562 validation proteins, and 5,651 test proteins, spanning 384 four-level EC classes.
GO Term Prediction. The aim of Gene Ontology (GO) term prediction is to predict whether given protein should be annotated with particular GO term. As we have stated before, proteins are categorized into three hierarchical ontologies: molecular function (MF), biological process (BP), and cellular component (CC). Specifically, MF denotes molecular activities of protein, BP refers to larger biological processes it is involved in, and CC describes subcellular locations and extracellular components 772 Accurately assigning GO terms is crucial for understanding protein function and assessing computational methods.
EC Number Prediction. This task aims to predict the 538 Enzyme Commission (EC) numbers the third and fourth level hierarchies for different proteins [67]. which provide precise informa- tion about protein's enzymatic function, based on the protein's 's features. The large number of classes at the third and fourth EC levels makes this a challenging multi-class prediction problem in bioinformatics.
C Evaluation Metric
max provides an overall metric that combines both accuracy and coverage of the predictions. calculated by first determining the precision and recall for each protein, then averaging these results 2e is the prediction probability for the j-th class of the i-th protein, given
over all proteins 17 67
the decision threshold E [0, 1}, the precision and call are given as:
Σ, I[( 1 2 1) n6)
E, 5) 2 no))
precision, (t)
recall,
Σ, (p(
where E {0, 13 is the corresponding binary class label, and I {0, 1} an indicator function. If there are N proteins in total, these protein-leve precision and recall values are averaged over all proteins obtain the overall precision and recall for the dataset, then the average precision and recall are defined as:
ΣN precision, (t)
recall(t) Σ recall, (t)
precision(t)
ΣN 21
N
Finally, F Paax is defined as the maximum value of the F-score over thresholds.
precision(t) recall(t)
F max = max {
(12)
precision(() + recall(t)
15

---

D Complexity Analysis
Our main focus on generating comprehensive embeddings for the student model, with less emphasis placed on the training specifics of the teacher model. Regarding the student model, the computational complexity of one message passing layer this framework is ((nd,), where d, represents the average node degree, typically much smaller than n. The time complexity is directly related to the computational complexity of the message passing layer; as graph convolution is performed on nodes and edges simultaneously, the time complexity remains (nd,), linear with the number of residues n. Denoting the size of the batch as Ba, the overall computational complexity is merely (B,
E
Experiment Setup
E.1
Model Details
The radius rs threshold increases from to 16, and 11. We set two message passing layers with one average sequence pooling per GNN. After the pooling layer, the number of residues is halved, and we update the edge conditions before performing another round of message passing and pooling operations, as illustrated in the model Figure The final GNNs include eight message-massing and four pooling layers, which are sufficient for achieving satisfactory results. The number of initial feature channels is 256, increased to 2048. The annotation encoder has 2 FC layers changing feature channels from 2752 to 2048, The classification head is a liner layer for predicting classes. For the teacher model, we use 2 to get the predicted annotations by the classification head and calculate the loss by Lteacher. The final loss C is used for the training of the student model.
We know spatially adjacent residues can still exist even when the sequence distance is large 73 We perform sequence average pooling and change edge conditions after once pooling. These operations enable the protein graph to cover more distant nodes.
E.2
Training Details
Dataset statisties [17 of the four downstream tasks are summarized in Table 4 The proposed framework conducted experiments on NVIDIA-SMI A100 GPUs and NVIDIA Tesla V100 GPUs, implemented with PyTorch 1. 1.13ccu117 and PyTorch Geometric 2.3.1 with CUDA 11.2.
Table 4: Dataset statisties. #X means the number of X.
Dataset
#Train
# Validation
#Test
Enzyme Commission
15, 550
1,929
1,919
Gene Ontology
898
3,322
3,415
Fold Classification Fold
12,312
736
718
Fold Classification Superfamily
12, 312
736
1,254
Fold Classification Family
12,
736
1, 1.2
Reaction Classification
29, 215
2,562
5, 651
In biology, a linear combination of original data with Gaussian noise way to augment the protein data:
34
is simple but effective
(P,, (P,, az) + O O (HR: o
(13)
where μi and d are selected as the random noise's 's mean (expectation) and standard deviation.
parrmtepasseseers related to the networks are set the same across different datasets: Adam optimizer with learning rate 4 1e - 3, weight decay decay 5e 4, epochs T' 300, Gaussian noise 14 = 0, 0 = 0.1, it indicates trivial perturbation introduced to the protein native structures.
The other dataset-specific hyper-pane parameters are determined by an AutoML toolkit NNI (74) with the search spaces. The loss weight hyper-parameter is related to the value of the task-specific loss 3= (1, 0, 0.01, 0.001, 0), and [10, 1, 0.1, 0.01, 0.001, 0}. As for the batch size and training epochs, etc., which influence the convergence speed of deep learning models, we set and 500 respectively.
16

---

F KL Guided Domain Adaption
Assuming source and target domains have the support and share the representation mapping p(z|G), this means these two domains have the same datasets of protein graphs and functions. Given the representation 2, learn classifier predict the label y through the predictive distribution p(ylz) that is an approximation of the ground truth. During training, the representation network p(=[G) and the classifier p(y) 2)) are trained jointly the source domain and we hope that they can generalize to the target domain, meaning that both p(=|G) and p(y|z) are kept unchanged between training and testing.
We define the predictive distribution of y given G as
ply G) Eptela) [[(ا|=)
(14)
We have single from the source model G)² for each protein. The training objective of the source domain is
Leeacher BG..--s(GGa)).e-ap..
log ptyl z)] (p log p(y|=)]
(15)
We consider the two assumptions of the representation = on the source domain:
Assumption 1. Is(2. Is(G, y), where Is(,,)) is the mutual information term, calculated on the source domain. particular:
(s(2, y) Eps(..v) log Ps(z. 3)
1s(G, y) y Eps(GG)) log Ps(G.y)
(16)
ps(s)ps(s))
ps (s(())ps())
The mutual information quantifies the amount of information shared between the variables and (or G and y) in the source domain. It measures the dependence or correlation between these variables in the context of the source domain data. This often referred to as the "sufficiency assumption' since it indicates that the representation z has the same information about the label the original input protein graph G, and is sufficient for this prediction task the source domain. Note that the data processing inequality indicates that Is(z,y) [G v). here assume that contains maximum information about y.
Assumption 2. ps [G) = Ep(=IG) [>s[]]=)]
When this assumption holds, the predictive distribution p(y|G) will approximate ps(w)G), as long as p(yl2) approximates (s)))z).
The above assumptions ensure that the teacher network has good performance in the source domain. Now, we continue to consider the test loss and how we can reduce it. The loss of the target domain is:
S (pr()
|- log p(v|G)] = (p [- log ((y))
< E (G.)) [Epp)aa |- - log [[y|=)]]
(17)
(p log p[y[=)]
Since we do not know the target domain and the target data distribution, there is no way to guarantee the invariance (both marginally and conditionally) of the representation 2. Therefore, We introduce the following proposition that ensures generalization bound of the target domain loss based on the source domain loss and the KL divergence:
Proposition 1. If the loss
log ((y]=) is bounded by M, have:
M
Limddent
Liencher
KI. (ps(
pr (T(
2
(18)
M
Cienber
KL [ps(z)
Pr())) + E Eps(s) [KL (ps(y(z) || Pr(v[=)]]
Proposition 2. If Assumption and hold, and if Ps(G.y) (G(
< oo (i.e., there exists N', which can be
arbitrarily large, such that (s(G.)) N'), we have pr(G.w)
E Epy(C) [KL (ps(vlz) 2) 2]] < Bps(C) KL [ps(([G) || pr(}]G)]
(19)
17

---

Table 5: The comparison results with pre-training methods (Fmax) on GO term and EC number prediction. The best results are shown in bold.
Category
Method
GO-BP
GO-MF
GO-CC
EC
ESM-1b 8
0,452
0.659
0,477
0.869
Sequence
ESM-2
0.472
0.662
0.472
0.874
ProtST-ESM-IIE lb 20
0.480
0.661
0.488
0.878
Sequence- Funntiinn
ProIST-ESM-2 [20]
0,482
0,668
0.487
0,878
DeepGO-SE 775
0.438
0.564
0.427
0,810
ESM-GearNet 777
0,516
0.684
0,506
0.890
GearNet-ESM-INR-N
76
0.518
0.683
0.504
0,896
Seguence-Structurr
ProtGO-ESM (Student)
0.520
0.693
0.536
0.887
This shows that the conditional misalignment in the representation space is bounded by the conditional misalignment in the input space. then follows that:
M
Leencher
KT. [T.((s(=)
|| pr())] (G)))) [KT. [ps((/G) pr (y))
(20)
22
We know can represent the underlying functional label for the student model. Although the student model may have these functional labels, but can assume that they exist for theoretical reasons. The derived misalignment Eq. 20 and the derived loss Eq. 8 are based on the assumption that the source and target domains have the same support set. Thus, the loss of Eq. 8 can be used in an unsupervised way for the student to predict functions. However, the student model is applied to different downstream tasks, like classification, which has classification classes. Thus, we add the supervised student loss Lstudent and the knowledge distillation loss the Lid as the final hybrid loss for the student to improve its performance on classification tasks.
G Comparisons with Pre-training Methods
To ensure fair comparisons with pre-trained models, we integrate ESM-2 (650M) (6) into ProtGO, resulting the creation of ProIGO-ESM. In this approach, ESM embeddings are utilized to enhance the graph node features. Our assessment evaluates ProtGO-EST -EEGGOM-MMMMM- against various pre-training tech- niques in the tasks of protein function and EC number prediction. This evaluation includes range of methods, such sequence- eequence-based approaches like ESM-1b [8] and ESM-2 [6]; cequence-functtin models including ProtST (20] and DeepGO-SE [75 and sequence-structure methodologies such as GearNet-ESM [17 and GeearNet-ESM-INR..
[76). Detailed comparative results are presented in Table 5 Notably, our proposed model, ProtGO-ESM, outperforms other methods across these sequence-based, sequence-structure
and requence-functionn- pre-training techniques.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have introduced these.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section S
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
19

---

Justification:
Guidelines:
See Appendix
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Dataset, models and settings are introduced in this manuscript.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
Justification:
Codes are in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//uuuii wwwiicccguie//eeeeeeeeeis CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
These are introduced in the main texts and appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Mean values are reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
21

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix H.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: h/tppr//////rrss/e//// htt.://....cpp......ttt.c.st....
Answer: [Yes]
Justification: we preserve the anonymity.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Social impact is discussed in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
22

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper does not have such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited original papers.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee wwworeeswthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
23

---

Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/coddemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not have this content.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "zRfZy0mMCM", "reviewer": "Reviewer_kUCf", "summary": "In the paper entitled \"ProtGO: Function-Guided Protein Modeling for Unified Representation Learning\", the authors proposed a KD-based framework to incorporate GO knowledge to learn an unified, multi-modal embedding for a given protein. The cross-domain knowledge make the embeddings performs good in various downstream tasks.", "questions": "1. In table 1 and table 2, the authors may consider add ESM-1b/ESM-2/ESM-3 as sequence modality baselines. \n2. Also table 1 and 2, the authors may consider add ESM-IF1 as structural modality baselines.\n3. Also table 1 and 2, the authors may consider add GO/sequence+GO modality as input, add DeepGO-SE as baselines. (https://www.nature.com/articles/s42256-024-00795-w)\n4. I am interested in the downstream task performance differences for proteins with GOs (where the teacher can teach the student well) and proteins without GOs (where the teacher model may fail). Moreover, GO terms consist of 3 parts, molecular functions, cellular components, and biological processes, I wonder whether the authors could explore the effectiveness of each part through comprehensive downstream experiments.", "limitations": "The authors have addressed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy-to-follow.\n2. The proposed unified framework is novel and interest, it may lay as foundations for futural research on protein embedding.\n3. The introduction of KD for transferring GO knowledges could potentially overcome the data insufficient problem.", "weaknesses": "Lack of sufficient baseline method, as I believe in all modalities (sequence/structure/GO), there exists other powerful methods that the authors may ignore. \n\nMinor:\nIn line 4, the authors wrote \"including sequence, structure, domains, motifs, and...\", generally domains and motifs are structural annotations, they should not be listed together with sequence and structure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer kUCf,\n\nWe are grateful for your thorough review. Your comments are highly valued, and we would like to express our heartfelt gratitude. We do our utmost to address the questions you have raised:\n\n**Q1** In table 1 and table 2, the authors may consider add ESM-1b/ESM-2/ESM-3 as sequence modality baselines.\n\n**A1** Thank you for your valuable feedback! In actuality, ProtGO does not function as a pre-training model, making it unjust to draw comparisons with pre-training methodologies. Consequently, we integrate ESM-2 (650M) [1] with ProtGO, denoted as ProtGO-ESM, where the ESM embeddings serve as a component of graph node features. Our evaluation juxtaposes ProtGO-ESM against pre-training techniques in tasks like protein function prediction and EC number prediction. This includes a spectrum of methods: sequence-based approaches such as ESM-1b [2] and ESM-2; sequence-function based models like ProtST [3]; and sequence-structure based methodologies like GearNet-ESM [4], SaProt [5], and GearNet-ESM-INR-MC [6]. Notably, the outcomes of ESM-3 [7] are pending due to resource constraints. The comparative findings are detailed in Table 1 of the one-page rebuttal pdf. Notably, our proposed model, ProtGO-ESM, emerges as the top performer across sequence-based, sequence-structure based, and sequence-function based pre-training strategies. \n\n**Q2** Also table 1 and 2, the authors may consider add ESM-IF1 as structural modality baselines.\n\n**A2** Thank you for your informative reviews! Protein design involves the computational creation of amino acid sequences that can fold into a specific protein structure. Methods like ESM-IF1 [8], PiFold [9], and VFN-IF [10] are dedicated to protein design, distinct from protein representation learning for function prediction. By focusing on protein inverse folding, we apply our approach to CATH 4.2 dataset, as detailed in Table 2 of the one-page rebuttal pdf. Our method demonstrates exceptional performance in this context, nearly outperforming other approaches in this task. \n\n**Q3** Also table 1 and 2, the authors may consider add GO/sequence+GO modality as input, add DeepGO-SE as baselines.\n\n**A3** Thank you for your reviews! We have compared our model, ProtGO-ESM, with pre-training methods with sequence, sequence-structure and sequence function as inputs; the results are shown in Table 1 in the one-page rebuttal pdf. \n\n\n**Q4** I am interested in the downstream task performance differences for proteins with GOs (where the teacher can teach the student well) and proteins without GOs (where the teacher model may fail). Moreover, GO terms consist of 3 parts, molecular functions, cellular components, and biological processes, I wonder whether the authors could explore the effectiveness of each part through comprehensive downstream experiments.\n\n**A4** Thank you for your reviews! (1) The effectiveness of the teacher model in instructing the student is notably higher for GO terms with a higher frequency. Conversely, our experiments reveal that when the frequency of a GO term falls below 50, the teacher model may struggle. For instance, in the case of the GO term GO:0030027, denoted as lamellipodium, the performance of the teacher model is suboptimal.\n(2) Through experimental analysis, we segmented GO terms into three categories: molecular functions (MF), cellular components (CC), and biological processes (BP). By utilizing only one category of GO terms as input for the GO encoder of the teacher model, we aimed to assess the efficacy of each category. Our findings indicate that focusing on a single category, such as MF, primarily enhances predictions related to MF in subsequent tasks. While specialization in a specific category can improve accuracy within that domain, it may not directly translate to improved predictions in the other two categories (BP and CC). This divergence stems from the distinct biological aspects represented by each category, each with its unique characteristics and interrelations. \n\n\n**Q5** Minor: In line 4, the authors wrote \"including sequence, structure, domains, motifs, and...\", generally domains and motifs are structural annotations, they should not be listed together with sequence and structure.\n\n**A5** Thank you for your suggestion! We will rectify this in the revised version.\n\n*Thank you again for all the efforts that helped us improve our manuscript! We have tried our best to address your concerns as it is important for my graduation; we respectfully thank you for supporting the acceptance of our work. Also, please let us know if you have any further questions. Look forward to further discussions!*\n\n[1] Lin, Z., et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.\n\n[2] Rives, A., et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. PNAS, 2021.\n\n[3] Xu, M., et al. Protst: Multi-modality learning of protein sequences and biomedical texts. ICML, 2023.\n\n[4] Zhang, Z., et al. Protein representation learning by geometric structure pretraining. ICML, 2022b.\n\n[5] Su et al. SaProt: Protein Language Modeling with Structure-aware Vocabulary. ICLR, 2024.\n\n[6] Lee, Y., et al. Pre-training sequence, structure, and surface features for comprehensive protein representation learning. 2023.\n\n[7] Hayes, T., et al. Simulating 500 million years of evolution with a language model. bioRxiv, 2024.\n\n[8] Hsu, C., et al. Learning inverse folding from millions of predicted structures. ICML, 2022.\n\n[9] Gao, Z., et al. PiFold: Toward effective and efficient protein inverse folding. ICLR, 2022a.\n\n[10] Mao, W., et al. De novo protein design using geometric vector field networks. arXiv, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "koOUR2ifwL", "reviewer": "Reviewer_oeK7", "summary": "This paper proposes to learn hybrid embeddings for the protein and the GO terms. By further applying the teacher-student training schedule, during inference, the additional input of GO terms is not necessary. Experimentally, the authors demonstrates that the model achieves better results on severls tasks, e.g., folding classification, reaction classification and GO term classification.", "questions": "I wonder how the authors avoid data leakage. What's the sequence and structure similarity (which can use AlphaFold DB to measure) between the functional annotation dataset and the other datasets.", "limitations": "n/a", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "[+] ProtGO proposes to improve the protein representation with extra side information, the function descriptions. \n[+] With teacher-student approach, it is unnecessary to have additional functions as input for the student network, simplifying the inference process.\n[+] Benchmark experiments demonstrate that ProtGO significantly outperforms state-of-the-art baselines.", "weaknesses": "[-] The proposed method only demonstrate results on several small benchmarks. More results on larger scale benchmarks can be useful, e.g., residue prediction, mutation effect prediction, etc. Functions are related with the folding / reaction / GO term / EC numbers, I wonder whether it is helpful to demonstrate the performance on other problems, to demonstrate the generalization of the proposed method.\n[-] The proposed method should learn better functions, therefore, binding affinity prediction may be a better downstream or zero-shot task to demonstrate the model performance. \n[-] The experiment results can introduce more about the error bars.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer oeK7,\n\nWe are grateful for your thorough review. Your comments are highly valued, and we would like to express our heartfelt gratitude. We do our utmost to address the questions you have raised:\n\n**Q1** The proposed method only demonstrate results on several small benchmarks. More results on larger scale benchmarks can be useful, e.g., residue prediction, mutation effect prediction, etc. Functions are related with the folding / reaction / GO term / EC numbers, I wonder whether it is helpful to demonstrate the performance on other problems, to demonstrate the generalization of the proposed method.\n\n**A1** Thank you for your valuable feedback! Protein design involves the computational creation of amino acid sequences that can fold into a specific protein structure. Methods like ESM-IF1 [1], PiFold [2], and VFN-IF [3] are dedicated to protein design, distinct from protein representation learning for function prediction. By focusing on protein inverse folding, we apply our approach to CATH 4.2 dataset, as detailed in Table 2 of the one-page rebuttal pdf. Our method demonstrates exceptional performance in this context, which demonstrates the generalization of the proposed method.\n\n\n**Q2** The proposed method should learn better functions, therefore, binding affinity prediction may be a better downstream or zero-shot task to demonstrate the model performance.\n\n**A2** Thank you for your informative reviews! We use the binding affinity prediction dataset used in [4] and [5]. DNA Binding Site Prediction Result trained on DNA-573 Train, tested on DNA-129 Test. RNA Binding Site Prediction Result trained on RNA-495 Train, tested on RNA-117 Test. We use the AUC for evaluation. Our proposed achieves best results on the binding affinity prediction, illustrating its generalization ability.\n\n|Method | DNA (AUC) | RNA (AUC) |\n| :--- | :---: | :---: |\n|SVMNUC [6] | 0.812 | 0.729 |\n| Coach-D [7] | 0.761 | 0.663 | \n| NucBind [6] | 0.797 | 0.715 |\n| GraphBind [8] | 0.927 | 0.854 |\n| VABS-Net [5] | 0.912 | 0.834 |\n|ProtGO (Ours) | **0.941** | **0.878** | \n\n**Q3** The experiment results can introduce more about the error bars.\n\n**A3** Thank you for your reviews! The mean values are reported, we have calculated the ariane of results on protein function prediction. Our results have low error bars.\n\n| Method | GO-BP | GO-MF | GO-CC | EC |\n| :--- | :---: | :---: | :---: | :---: |\n| ProtGO(Student) | 0.464 (0.005) | 0.667(0.002) | 0.492(0.006) | 0.857(0.008) |\n\n**Q4** I wonder how the authors avoid data leakage. What's the sequence and structure similarity (which can use AlphaFold DB to measure) between the functional annotation dataset and the other datasets.\n\n**A4** Thank you for your questions! (1) Employing the teacher-student framework, the student model learns from the teacher model's latent embeddings through knowledge distillation loss. To ensure data integrity, the test sets utilized in downstream tasks are excluded, addressing concerns regarding data leakage. (2) For the functional annotation dataset used in the teacher model and downstream datasets used in the student model, the sequence similarity between it and EC number prediction dataset is only 25\\%, the structural similarity is 19\\%; the sequence similarity between it and GO term prediction dataset is 75\\%. (3) Adhering to the GearNet [9] protocol, the test sets for GO term and EC number prediction exclusively comprise PDB chains with less than 95\\% sequence identity to the training set, allowing for the generation of varied cutoff splits.  \n\n*Thank you again for all the efforts that helped us improve our manuscript! We have tried our best to address your concerns as it is important for my graduation; we respectfully thank you for supporting the acceptance of our work. Also, please let us know if you have any further questions. Look forward to further discussions!*\n\n[1] Hsu, C., et al. Learning inverse folding from millions of predicted structures. ICML, 2022.\n\n[2] Gao, Z., et al. PiFold: Toward effective and efficient protein inverse folding. ICLR, 2022a.\n\n[3] Mao, W., et al. De novo protein design using geometric vector field networks. arXiv, 2023.\n\n[4] Zhang, C., et al. Us-align: universal structure alignments of proteins, nucleic acids, and macromolecular complexes. Nature methods, 2022.\n\n[5] Zhuang, W., et al. Pre-Training Protein Bi-level Representation Through Span Mask Strategy On 3D Protein Chains. ICML. 2024. \n\n[6] Su, H., et al. Improving the prediction of protein–nucleic acids binding residues via multiple sequence profiles and the consensus of complementary methods. Bioinformatics, 2019.\n\n[7] Wu, Q., et al. Coach-d: improved protein–ligand binding sites prediction with re- fined ligand-binding poses through molecular docking. Nucleic acids research, 2018.\n\n[8] Xia, Y., et al. Graphbind: protein structural context embedded rules learned by hierarchical graph neural networks for recognizing nucleic- acid-binding residues. Nucleic acids research, 2021.\n\n[9] Zhang, Z., et al. Protein representation learning by geometric structure pretraining. ICML, 2022b.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XqmRy5yBxg", "reviewer": "Reviewer_qMp2", "summary": "The authors distill the knowledge from function annotation. ProtGO acquires performance improvement through knowledge distillation. Compared with other methods only rely on structure and sequence, ProtGO outperforms these baselines.", "questions": "See Weaknesses.", "limitations": "None.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. ProtGO introduces a novel method that can utilize function information, indicating improved performance.", "weaknesses": "1. Protst[1] also utilizes function information of proteins, should also be listed as a baseline. Protst also does not need function information and only use protein sequence after alignment. \n\n2. It's confusing to find, in ablation study, ProtGO still outperforms all the baselines without teacher. The authors should explain the difference between backbone GNN model and other baselines. Furthermore, the improvement of teacher-student module is trivial compared with the improvement of the backbone GNN. This needs to be clarified.\n\n3. A question is that if it's necessary to introduce domain adaptation. The ablation study about this is missing.\n\n[1]. Protst: Multi-modality learning of protein sequences and biomedical texts", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer qMp2,\n\nWe are grateful for your thorough review. Your comments are highly valued, and we would like to express our heartfelt gratitude. We do our utmost to address the questions you have raised:\n\n**Q1** Protst[1] also utilizes function information of proteins, should also be listed as a baseline. \n\n**A1** Thank you for your valuable feedback! In actuality, ProtGO does not function as a pre-training model, making it unjust to draw comparisons with pre-training methodologies. Consequently, we integrate ESM-2 (650M) [1] with ProtGO, denoted as ProtGO-ESM, where the ESM embeddings serve as a component of graph node features. Our evaluation juxtaposes ProtGO-ESM against pre-training techniques in tasks like protein function prediction and EC number prediction. This includes a spectrum of methods: sequence-based approaches such as ESM-1b [2] and ESM-2; sequence-function based models like ProtST [3]; and sequence-structure based methodologies like GearNet-ESM [4], SaProt [5], and GearNet-ESM-INR-MC [6]. The comparative findings are detailed in Table 1 of the one-page rebuttal pdf. Notably, our proposed model, ProtGO-ESM, emerges as the top performer across sequence-based, sequence-structure based, and sequence-function based pre-training strategies. \n\n**Q2** It's confusing to find, in ablation study, ProtGO still outperforms all the baselines without teacher. The authors should explain the difference between backbone GNN model and other baselines. Furthermore, the improvement of teacher-student module is trivial compared with the improvement of the backbone GNN. This needs to be clarified.\n\n**A2** Thank you for your informative reviews! (1) The student model is meticulously crafted as a Graph Neural Network (GNN) model capable of encoding protein sequences and structures simultaneously, demonstrated in Eq.4 of the manuscript. This robust protein encoder adeptly integrates protein sequences and structures, outperforming alternative sequence-structure methodologies. A sequence pooling layer is utilized to condense sequence length effectively, facilitating the aggregation of crucial patterns. In this GNN architecture, each pair of message-passing layers is succeeded by an average sequence pooling layer, totaling eight message-passing layers in the model. (2) The sequence average pooling functions execute tailored average pooling operations on input tensors based on calculated indices, dividing the sequence length by two and flooring the result. These functions aggregate and synthesize information from input tensors through scatter operations to generate output tensors. Following each average pooling layer, the number of residues is halved, expanding the radius threshold $r_s$ to $2r_s$ post-pooling. This extension enables center nodes' neighbors to encompass progressively distant and infrequent nodes, concurrently reducing computational complexity. These operations enable the model to capture both local and global features effectively. (3)\nThe ablation study's Table 5 showcases the student model's autonomous performance without teacher guidance, demonstrating its ability to independently model protein sequences and structures. (4) Additionally, Figure 3 in the manuscript illustrates the enhancements resulting from incorporating functional information, affirming the advantages of this augmentation. \n\n**Q3** A question is that if it's necessary to introduce domain adaptation. The ablation study about this is missing.\n\n**A3** Thank you for your reviews! The theory of domain adaptation forms the foundation for deriving Eq.10 and Eq.11, as detailed in Appendix F of the manuscript. In the realm of domain adaptation, employing a supervised loss is preferred when the student model operates with distinct task labels [2].\n\n*Thank you again for all the efforts that helped us improve our manuscript! We have tried our best to address your concerns as it is important for my graduation; we respectfully thank you for supporting the acceptance of our work. Also, please let us know if you have any further questions. Look forward to further discussions!*\n\n[1] Xu, M., et al. Protst: Multi-modality learning of protein sequences and biomedical texts. In International Conference on Machine Learning. 2023.\n\n[2] Berthelot, D., et al. Adamatch: A unified approach to semi-supervised learning and domain adaptation. arXiv preprint arXiv:2106.04732.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
First and foremost, we would like to express our sincere gratitude for the insightful and constructive feedback provided by the reviewers on our manuscript. We greatly appreciate their positive reception of ProtGO's potential and its timely relevance in the field of protein research.

We are particularly thankful for Reviewers **kUCf** and **qMp2**'s recognition of the novelty of our study, which may lay as foundations for futural research on protein embedding; We also appreciate their acknowledgment of the introduction of knowledge distillation for transferring GO knowledges, which could potentially overcome the data insufficient problem (Reviewer **kUCf**, **oeK7**). The reviewers have acknowledged the promising quality and significance of our work, achieving better results on tasks, e.g., folding classification, reaction classification and GO term classification (Reviewer **kUCf**, **oeK7**). Additionally, Reviewer **kUCf** has expressed the acknowledgment of our wirting, which is well-organized and easy-to-follow.

**Comparisons with pre-training methods:**
We appreciate the feedback received, particularly addressing the absence of comparisons with pre-training methods. It is essential to note that ProtGO does not operate as a pre-training model, making direct comparisons with pre-training methodologies inappropriate. To address this, we integrate ESM-2 (650M) [1] with ProtGO, forming ProtGO-ESM, where ESM embeddings enhance graph node features. Our assessment contrasts ProtGO-ESM with pre-training techniques in protein function and EC number prediction tasks. This evaluation encompasses diverse methods, including ESM-1b [2], ESM-2 for sequence-based approaches, ProtST [3] for sequence-function models, and GearNet-ESM [4], SaProt [5], and GearNet-ESM-INR-MC [6] for sequence-structure methodologies. Detailed comparative results are outlined in Table 1 of the one-page rebuttal PDF. Notably, our proposed model, ProtGO-ESM, emerges as the top performer across various pre-training strategies.

**Generalization Ability:**
In the one-page rebuttal PDF, we present results on protein design, showcased in Table 2, underscoring the generalization capacity of our method. Additionally, we conducted binding affinity prediction experiments on DNA [7] and RNA [8] datasets. As depicted in the table below, our method excels in binding affinity prediction, emphasizing its robust generalization capabilities. 

|Method | DNA (AUC) | RNA (AUC) |
| :--- | :---: | :---: |
|SVMNUC [9] | 0.812 | 0.729 |
| Coach-D [10] | 0.761 | 0.663 | 
| NucBind [9] | 0.797 | 0.715 |
| GraphBind [11] | 0.927 | 0.854 |
| VABS-Net [8] | 0.912 | 0.834 |
|ProtGO (Ours) | **0.941** | **0.878** | 

*Once again, we sincerely appreciate the reviewers' feedback and remain committed to continuously improving our research and manuscript based on their valuable insights. Thank you again!*


[1] Lin, Z., et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.

[2] Rives, A., et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. PNAS, 2021.

[3] Xu, M., et al. Protst: Multi-modality learning of protein sequences and biomedical texts. ICML, 2023.

[4] Zhang, Z., et al. Protein representation learning by geometric structure pretraining. ICML, 2022b.

[5] Su et al. SaProt: Protein Language Modeling with Structure-aware Vocabulary. ICLR, 2024.

[6] Lee, Y., et al. Pre-training sequence, structure, and surface features for comprehensive protein representation learning. 2023.

[7] Zhang, C., et al. Us-align: universal structure alignments of proteins, nucleic acids, and macromolecular complexes. Nature methods, 2022.

[8] Zhuang, W., et al. Pre-Training Protein Bi-level Representation Through Span Mask Strategy On 3D Protein Chains. ICML. 2024. 

[9] Su, H., et al. Improving the prediction of protein–nucleic acids binding residues via multiple sequence profiles and the consensus of complementary methods. Bioinformatics, 2019.

[10] Wu, Q., et al. Coach-d: improved protein–ligand binding sites prediction with re- fined ligand-binding poses through molecular docking. Nucleic acids research, 2018.

[11] Xia, Y., et al. Graphbind: protein structural context embedded rules learned by hierarchical graph neural networks for recognizing nucleic- acid-binding residues. Nucleic acids research, 2021.

Meta-review: This paper proposes a protein representation learning framework based on a knowledge distillation scheme, where a GNN teacher with trimodal knowledge of the protein (sequence, structure, and functionality) distills the knowledge into a GNN student with only sequence-structure information. The idea is that the functional annotations of the proteins are important but may be missing in many downstream tasks. The proposed method achieves superior performance over existing methods on multiple benncharks.

The reviewers found the proposed work as well-motivated, the framework as novel and effective based on the strong benchmark performance. However, there were concerns regarding missing large-scale experiments, rationale on why the proposed framework performs well even without KD, and discussion of a similar work. However, those concerns were addressed during the rebuttal period and there was a consensus to accept the paper.
Decision: Accept (poster)