Learning Complete Protein Representation by Dynamically Coupling of Sequence and Structure
Authors: Bozhen Hu
Cheng Tan
Jun Xia
Yue Liu
Lirong Wu
Jiangbin Zheng
Yongjie Xu
Yufei Huang
Stan Z. Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Graph Neural Network
Protein Sequence
Structure
Primary area: machine_learning_for_other_sciences_and_fields

Abstract: Learning effective representations is imperative for comprehending proteins and deciphering their biological functions. Recent strides in language models and graph neural networks have empowered protein models to harness primary or tertiary structure information for representation learning. Nevertheless, the absence of practical methodologies to appropriately model intricate inter-dependencies between protein sequences and structures has resulted in embeddings that exhibit low performance on tasks such as protein function prediction. In this study, we introduce CoupleNet, a novel framework designed to interlink protein sequences and structures to derive informative protein representations. CoupleNet integrates multiple levels and scales of features in proteins, encompassing residue identities and positions for sequences, as well as geometric representations for tertiary structures from both local and global perspectives. A two-type dynamic graph is constructed to capture adjacent and distant sequential features and structural geometries, achieving completeness at the amino acid and backbone levels. Additionally, convolutions are executed on nodes and edges simultaneously to generate comprehensive protein embeddings. Experimental results on benchmark datasets showcase that CoupleNet outperforms state-of-the-art methods, exhibiting particularly superior performance in low-sequence similarities scenarios,  adeptly identifying infrequently encountered functions and effectively capturing remote homology relationships in proteins.

Learning Complete Protein Representation by Dynamically Coupling of Sequence and Structure
Bozhen Hu Hu,, Cheng Tan Jun Xia², Yue Liu Lirong Wu?,
Jiangbin Zheng?, Yongjie Xu², Yufei Huang?, Stan Z. Lift 'Zhejiang University WWestlake University 3National University of Singapore (hubozhen, tancheng, stan. wtt..qqq..ieeee.. www.iietttlakee... att edu. cn
Abstract
Learning effective representations imperative for comprehending proteins and deciphering their biological functions. Recent strides in language models and graph neural networks have empowered protein models to harness primary or tertiary structure information for representation learning. Nevertheless, the absence of practical methodologies to appropriately model intricate inter-dependencies be- tween protein sequences and structures has resulted in embeddings that exhibit low performance on tasks such as protein function prediction. In this study, we intro- duce CoupleNet, novel framework designed to interlink protein sequences and structures to derive informative protein representations. CoupleNet integrates mul- tiple levels and scales of features in proteins, encompassing residue identities and positions for sequences, as well as geometric representations for tertiary structures from both local and global perspectives. A two-type dynamic graph is constructed to capture adjacent and distant sequential features and structural geometries, achiev- ing completeness the amino acid and backbone levels. Additionally, convolutions are executed on nodes and edges simultaneously to generate comprehensive protein embeddings. Experimental results on benchmark datasets showcase that CoupleNet outperforms taate-of-thh-asd methods, exhibiting particularly superior performance in sequenee similarities scenarios, adeptly identifying infrequently encountered functions and effectively capturing remote homology relationships in proteins.
1 Introduction
Proteins, the fundamental building blocks of life, serve crucial roles across diverse array of applica- tions, ranging from therapeutics to materials. Comprising 20 distinct amino acids linked by peptide bonds, proteins form intricate sequences that dictate their one-dimensional (ID) structure, ultimately determining their biochemical functions 11. Due to recent progress protein sequencing (2). massive numbers of protein sequences are now available. For example, the UniProt database, housing over 200 million protein sequences, has become a valuable resource for research [3. Notably, the development of large-scale language models (LMs) in natural language processing has substantially benefited protein research owing to similarities between human languages and protein sequences (4.6). For instance, models like ProtTrans 7 and ESM-series [8 9 have proven the successful utility of protein LMs to process protein sequences.
Thanks to the recent significant progress made by AlphaFold2 [10] in three-dimensional (3D) structure prediction, large number of protein structures from their sequence data are now made available. The latest release of AlphaFold protein structure database [1] provides broad coverage of UniProt (3] Recently proposed structure-based protein encoders become utilize geometric features 12
*Equal contribution. =Correspondence: (stan, zq. (tt...g......ssslleee weetaaae edu.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

e.g.. ProNet [15 learns representations of proteins with 3D structures at different levels, like the amino acid, backbone or all- atom levels. Concurrently, methods employing graph neural networks (GNNs) and LMs (LSTMs or attention models) [14 16
17. such as GearNet [1], have been
developed to process both sequence and structure information.
The ID sequence and 3D structure of a protein provide different types of information, the discrete sequential orders, residue types, and coordinates, as shown in Figure [ Figure Appendix [ Models can learn coevolutionary and geometric information from sequences and structures, for example, whether residues contact or not. Although protein's sequence determines its structure. various works have demonstrated the effectiveness of learning from either sequences or structures [9] [8] However, directly fusing representations from sequence encoder and structure encoder cannot explore their relationships, and current protein GNN methods have drawbacks in integrating such sequential and structural information. In detail, the information propagation is difficult for long-range dependencies in large protein graphs, and messages attenuate over many rounds of passing in GNNs, although there are several works aiming to tackle such a problem [19] 20 Besides, message passing typically assumes localized neighborhood relationships, but amino acid interactions can be complex and long-range. We need to consider the structural and chemical properties of a residue that are highly dependent surrounding neighbors, and capturing different conformers requires modeling the entire protein structure holistically, as the conformation of an amino acid is constrained by steric and hydrogen bonding with nearby residues, and the conformers correspond to the same protein, adopting slightly different 3D structures. Therefore, a proper protein sequence-structure structure modeling method is necessary and important to recognize these challenges and factors to obtain comprchensive and effective representations.
In this work, we model the relative positions of residues in the sequence and the spatial arrangement of atoms in Euclidean space simultancously. We propose CoupleNet to construct two categories of graphs dynamically to cover the multiple scales of sequential features and structural geometries, which achieve completeness at the residue and backbone levels. Such global completeness theoretically guaranteed to incorporate 3D information completely without information loss, while the local view would miss the long-range effects of subtle conformational changes happening distantly. For instance, the open and closed conformers of an enzyme may have similar local binding pockets but differ in global clamshell arrangement [2T]. In order to better capture the local and global relationships and relieve the problems that exist in deep GNNs, we dynamically build new protein graphs in different conditions based on the depth of the network. For feature fusing, we take advantage of graph convolutions, performing node and edge convolutions simultaneously rather than passing messages separately on nodes and edges. Thus, the contributions of this paper are threefold: (I) A novel woo-graph-based approach for modeling sequential and 3D geometric features, ensuring global completeness in protein representation. (2) The proposal of CoupleNet, which performs concurrent convolutions on nodes and edges, effectively integrating protein sequence and structure. The dynamically changed graphs can better model the node-edge relationships and utilize the intrinsic associations between sequences and structures. (3) Empirical validation showcases the superior performance of the proposed model compared to current mainstream protein representation learning methods across diverse tasks, including protein fold classification, enzyme reaction classification, Gene Ontology (GO) term prediction, and Enzyme Commission (EC) number prediction. Our experiments reveal that this method excels in predicting functions that rarely appear, effectively captures protein remote homology relationships.
2 Related Work
Protein Representation Learning. Protein representation learning has emerged dynamic and promising field within biology, playing a crucial role in diverse downstream applications in protein science. Given the multifaceted nature of protein structures, current methodologies predominantly fall into three categories: protein LMs tailored for sequences, structure models emphasizing geometry, and hybrid approaches seamlessly integrating both aspects. Considering proteins as sequences of amino acids, akin to the structure of human languages, TAPE [22) establishes a benchmark for a variety of protein models, including ID CNN, LSTM, and Transformer architectures. Einaggar et al. have successfully trained six transformer variants, such as ProtBert and ProtT',, on extensive amino acid sequences. Similarly, the ESM-series R 9 2 adopt a transformer architecture and a masked language modeling strategy, achieving robust representations through training on large-scale
2

---

databases. Besides, several methods aim to encode the spatial information of protein structures
using techniques such as convolutional neural networks (CNNs) 24, or GNNs [77 25
26
For
instance, SPROF [ employs distance maps to predict protein sequence profiles, while IEConv [22
introduces a convolution operator to capture relevant structural levels. GVP-GNN 26 designs
the geometric vector perceptrons (GVP) to learn both scalar and vector features in an equivariant and invariant manner. ProNet [1]] learns hierarchical protein representations at multiple tertiary structure levels of granularity. Additionally, CDConv (28) introduces continuous-discred convolution, utilizing irregular and regular approaches to model both geometry and sequence structures. protein clustering method [29] is proposed by applying an iterative clustering strategy to group the nodes into clusters based on their ID and 3D positions and assigned scores to obtain hierarchical protein representations. Moreover, some protein learning methods concurrently model multiple levels of structures [1 28 30,, and PromtProtein [31 adopts a prompt-guided multi-task learning strategy for incorporating various protein structures.
Complete Message Passing, While SphereNet [32 introduces a spherical message passing scheme for precise 3D molecular learning, ensuring completeness within the edge-based 1-hop local neigh- borhood, this completeness does not extend to the entire 3D graph. Building upon this limitation, ComENet 3 innovatively incorporates rotation angles and spherical coordinates to achieve global completeness in 3D information on molecular graphs. By integrating these meticulously designed geo- metric representations into the established message passing scheme (34), the complete representation for whole 3D graph is ultimately achieved [15].
Unlike these methods, we couple sequence and structure via dynamically changed graphs and different geometric representations to attain complete representations throughout the entire protein 3D graph.
3 Methodology
3.1 Preliminaries
Notations. A 3D graph represented as G = (V,E,)), where {us) and & {Eyj),J=....., denote the vertex and edge sets with n nodes, respectively, and p = (P.)i=i...... is the set of position matrices, where P: E xx represents the position matrix for node v; We treat each amino acid as graph node for protein, then k; depends on the number of atoms in the i-th amino acid. The node feature matrix is X = [a:]i=m....... where H; E Rds is the feature vector of node U1 The edge feature matrix F. [e.gle.s=1..... where ei E Rdde the feature vector of edge Sig. dy is the dimension of feature vector æ;, and d, denotes the dimension of eij-
Invariance and Equivariance. We consider affine transformations that preserve the distance between any two points, i.e., the isometric group SF(3) (refer Appendix BB) in the Fuclidean space. This is called the symmetry group, and it turns out that SE(3) is the special Euclidean group that includes 3D translations and the 3D rotation group SO(3) [35 36).
Given the function f Rm Rm', assuming the given symmetry group G acts IR and Rm',. then f is G-equivariant if,
= Sff((x) Vo E R" G
where T' and S. are the transformations. For the SE(3) group, when m' 1, the output of f is scalar, we have
[(T = [(x), Vo E Rm G
thus f is SE(3) invariatn.
Complete Geometric Representations. A geometric transformation F(.) ((:))s is complete if for two 3D graphs G (V,E, P1) and G = (V,E, p2) there exists T, SE(3) such that the representations
F(G') = - (P,) for l....n
The operation T3 would not change the 3D conformation of a graph IS 62) And F(G) P. positions can generate geometric representations, which can also be recovered from them.
3

---

1.33A
Ra
peptide
H41
plane
1.45A
backbone
V
4
123"
1.52A
22
Ωg
C
121
1.23A
118
120
116
H2
Figure 1:
The polypeptide chain depicting the characteristic backbone bond lengths, angles, and torsion angles ( 2, S2,). The planar peptide groups are denoted as shaded gray regions, indicating that the peptide plane differs from the geometric plane calculated from 3D positions.
Message Passing Paradigm. Message passing mechanism is mainly applied in graph convolutional networks (GCNs) (37), which follows an iterative scheme of updating node representations based on the feature aggregation from nearby nodes.
h (0) BN (FC (ar:),
u S (hl) (h4--110
Joy (Ne)),
(4)
h (1) SUpdate (hl (n"-",," " (
where FC(.) and BN)()) mean the linear transformation and batch normalization respectively. ((:)) denotes the neighbours of node Us- f and {() Update aggregation and transformation functions at the /-th layer, which are permutation invariant and equivariant of node representations.
3.2 Sequence-Structuue Graph Construction
Specifically, we represent each amino acid as a node, considering the residue types and their positions i 1,. in the sequence, we define the sequential graph primarily on the sequence, if || -jll < I, the edge Eij exists, where a hyper-parameter, and ||-I| denotes the 12-norm. Besides, predefine radius r, and build the radius graph. There exists radius edge between node v and ey if ||des, ca < where dij.Ca = Ca,ca Ca
Ca. and Pa,ca denotes the 3D position of Ca in the i-th residue. We dynamically change the predefined thresholds with the depth the network cover nodes from the local to the global.
Firstly, we design base approach at the amino acid level (aa) called CoupleNetas that only uses the Ca positions the structures. Inspired by IIngaaham et al.) we construct a local coordinate system (LCS) for each residue (Figure 7 (a) in the appendix).
bb;
T2i b
(5)
X denotes the vector outer product. Then, can get the geometric representations at the amino acid level of protein 3D graph,
where u. = PLoo P.Co-Pa-1.Ca b; u. n. X
- co
x
F(G) = (Ildg,call of dij.Ca
Q Q2))
(6)
[Tdi
where is the matrix multiplication. This implementation is SE())-eqiivarian and obtains complete representations at this level; if have Q, the LCS Q, can be easily obtained from F(G) sj.o-
For a node Dr the node features Fi at the amino acid level the concatenation of the one-hot embeddings of amino acid types and the physicochemical properties of each residue, namely, steric parameter, hydrophobicity, volume, polarizability, isoelectric point, helix probability and sheet
probability (39 40), which provide quantitative insights into the biochemical nature of residues.
Secondly, as illustrated in Figure T. CoupleNet considers backbone atoms Co,, C, N, (as depicted in Figure 2) In detail, the peptide bond displays partial double-bond character due to resonance 41

---

indicating that the three non non-hy non-hydrogen atoms comprising the bond are coplanar, with limited free rotation about the bond due to this coplanar property. The N; c. and C.i C bonds constitute the two bonds in the basic repeating unit of the polypeptide backbone. These single bonds allow unrestricted rotation until sterically restricted by side chains [42 43 Since the coordinates of C., can be obtained as we have the complete representations at the amino acid level, the coordinates of other backbone atoms based on these rigid bond lengths and angles are able to be determined with the remaining degree of the backbone torsion angles 4 V Sz The omega torsion angle around the N peptide bond typically restricted to nearly 180 (trans) but can approach (cis) in rare instances. Other than the bond lengths and angles presented in Figure all the H bond lengths measure approximately 1 Å.
For the sequential graph, we compute the sine and cosine values of 2, V,, S2 for each amino acid i, and also use them as node features for node eq.
2; = ai ((sin cos) ( (2))
where || denotes concatenation. There is no isolated node for the designed graph, which means the backbone atoms can be determined one by one along the polypeptide chain based on the po- sitions of Ca and these three backbone dihedral angles. Therefore, the existing presentations (F and [a:]e=1.....n are complete the backbone level for the sequential graph.
For the radius graph, we want to get the positions of backbone atoms in any two amino acids i and j. Inspired by trRosetta [44], the relative rotations and distances are computed, including the distance Cdij.c)) three dihedral angles (wiy 0:j, Oji) and two planar angles (yii, P.j;), as shown in Figure (?)b in the appendix. These interresidue geometries define the relative locations of the backbone atoms of two residues in their details [44]. Therefore, these six geometries are complete for amino acids at the backbone (bb) level for the radius graph. The graph edges contain the relative spatial information between any two neighboring amino acids Eij = =F())ij,au ([GG))j,llFF(().... [FG(),
F(G) = (dig,Ca (sin cos) (iguiyy00.... PPt))
(8)
Protein structures that are SE(3) equivalents have the same 3D conformation, differing in orienta- tion/positioning. Graph representations must encode these structures equivalently since the underlying molecular properties are identical. Constructing the relationships between sequence and structure can help the model learn more comprehensive protein representations, which ensures the model focuses on meaningful aspects of protein structures.
3.3 Sequence-Structure Graph Convolution
We employ graph convolution to embed sequences and structures simultaneously, exploring their rela- tionships to generate effective embeddings. Different from previous works [1 28), we innovatively construct two categories of graphs for sequence and structure and design comprehensive sequential and structural representations to achieve completeness at the amino acid and backbone levels. We then convolve node and edge features aided by the message passing mechanism.
Dynamic graph
Dynamic graph
Global attentoon
HSHGLFKKL... KVESRDGT
Residue properties
Node
Node
Pooling
Node
Pcoling
Nade
Boftiax
Complete geometrees
Edge
Edge
Sriquence-
structure no distanca relationship
Message pessing
Meesage passing
Output
An illustration of CoupleNet. This framework processes protein sequences and structures to get complete geometries and properties used as graph node and edge features, where the sequential and structural graph is dynamically changed depending on their distance relationships and the network depth. Convolutions happen on the nodes and edges simultaneously to capture the relationships from the local to the global.
Figure 2:
5

---

In order to implement convolution on nodes and edges simultaneously between sequence and structure, we set Eij to exist if the following conditions are satisfied:
| jll <1 and ||de,cool|< <r
The existing node and edge feature matrices (X, F)) are complete representations of protein 3D graph to reconstruct its backbone atom positions. When the thresholds r, are small, Eq. [defines the local environment [45] of an amino acid, and the structural and chemical properties of residue are highly dependent on surrounding residues.
Compared with the equation Eq. 4 the proposed CoupleNet first applies FC(() layer and a BN(-) layer to the node features to obtain the initial encoded representation. Then the aggregation function
A is applied to gather neighboring features of nodes and edges by convolution, where a(-) is the activation function, LeakyReLU. W is the learnable convolutional kernel matrix whose learnable parameters have concern with the number of nodes edges. We use the dropout and FC(.) layer and add a residual connection from the previous layer for update function f( JUpdate"
(0) = BN (FC (ax)), h
(1) = o (BN( E
Weih ((-1))))),
4
(10)
h (1)
h (1) + Dropout(FC(u!"))
By incorporating complete geometric representations to the commonly used message passing frame- work IO). complete message passing scheme can be achieved [15] 32 (3) which can capture small changes due to such rigid transformations in coordinate positions. Complete representations allow for powerful equivariance and invariance properties to be encoded, which makes the learned models robust. By incorporating complete geometries, the convolution and pooling operations on irregular and non-Euclidean data like graphs are defined and conducted, enabling more expressive modeling for protein data.
3.4
Model Architecture
The overall framework of CoupleNet is shown in Figure 2 The inputs to the graph are the calculated sequential and structural representations (X, E). We employ complete message passing and sequence pooling layers to obtain the deeply encoded graph-level representations. After one average pooling layer, the number of residues reduces by half. Thus, we expand the radius r to after once pooling, which makes neighbors of center nodes gradually cover more distant and rare nodes, also reducing the computational complexity.
Differences with Existing Protein Modeling Methods. The proposed approach representing the sequence and the 3D geometric structure of a protein differs from several existing protein models [12] [4]. Specifically, GearNet [14] has 21 + types of edges; there are only two different types of graphs in the proposed CoupleNet radius graph and sequential graph. Importantly, the threshold in the radius graph in GearNet is set to be constant, but we change the threshold of radius dynamically to learn different distance relationships. The message passing mechanism only executes on nodes in CoupleNet instead of on nodes and edges alternately used in GearNet. Moreover, CoupleNet performs convolutions on nodes and edges simultancously with several pooling layers to reduce the sequence length, which also largely different from ComENet (33) and GearNet.
Complexity Analysis. Considering the computational complexity of one message passing layer in this framework, it is O(nd,), where d, is the average node degree, and d,, & n. The time complexity is related to the computational complexity of the message passing layer; as we conduct the graph convolution on nodes and edges simultaneously, the time complexity also ((d,)). Assuming there are me edges in the graph, di and d2 mean the feature dimensions of nodes and edges, the space complexity is (nd, + mzdz) for every message passing layer. Using B, to denote the size of the batch, the final computational complexity is only ((B,ndn).

---

Table I: Accuracy (%) fold classification and enzyme reaction classification. [*] denotes the results are taken from 28 The best and suboptimal results are shown in bold and underline.
Fold Classification
Enzyme
Input
Method
Fold
SuperFamily
Family
Reaction
ResNet 22
10.1
7.21
23.5
24.1
Sequence
Transformer 122
9.22
8.81
40.4
26.6
3DCNN_MON 24
31.6
45.4
92.5
72.2
Structure
]EConv (atom level) [2]
45.0
69.7
98.9
87.2
GraphQA [25)
23.7
32.5
84.4
60.8
GVP (26
16.0
22.5
83.8
65.5
ProNet-Amino Acid 15
51.5
69.9
99.0
86.0
ProNet-Backbone 15
52.7
70.3
99.3
86.4
IEConv (residue level)
12
47.6
70.2
99.2
87.2
Sequence-Struee Structure
GearNet [14]
28.4
42.6
95.3
79.4
GearNet-IEConv [4]
42.3
64.1
99.1
83.7
GearNet-Edge
44.0
66.7
99.1
86.6
MoaNee-EEdg-IICCAA
14
48.3
70.3
99.5
85.3
CDConv 28
56.7
77.7
99.6
88.5
CoupleNet (Proposed)
60.6
82.1
99.7
89.0
Experiments
4.1 Datasets, Settings and Baselines
Following the tasks in IEconv 12 and Gear- Net [14], we evaluate CoupleNet on four protein tasks: protein fold classification, enzyme reaction classification, GO term prediction, and EC number prediction. For the task of fold and reaction clas- sification, the performance is measured by mean accuracy. For GO Term and EC Prediction, F max is used as the evaluation metric. The performance measured with mean values of five different increase the
initializations. As stated before, predefined radius r to 2r after one pooling layer, from 4 to 16, and / set to be constant number 11, and the number feature channels is also dou- bled. In this condition, when the number of nodes decreases, is constant, r increases and neighbors of center nodes gradually cover distant nodes. We design the sequential and radius graph instead of the k-nearest neighbor graph because constant k makes some neighboring nodes far away from the center node. Distances of group of neighbor nodes are larger than 20 Á, which cannot be seen as contacts
Figure 3:
The violin plot of the relationships distances between sequence and structure on the GO term prediction dataset, the sequential distance i| - jll is from to n-1, and the x-axis means sequential distance subtract one, the y- axis means dij.ca- The dashed red line connects the median values.
46
We present the dataset statistics Table 4 and conduct experiments to analyze these datasets. Figure shows the distance relationships between sequence and structure on the GO term training dataset with 29898 proteins. We adjacent. According to the trend of the medians, when the sequence long, atoms may need more space arrange in the 3D space. The violin plot distance relationships the other three datasets presented in Figure
that when the sequential distance is large, there still exist nodes spatially
We compare our proposed method with existing protein representation learning methods, which are classified into three categories based on their inputs: sequence (amino acid sequence), 3D structure, both sequence and structure. Sequence-based encoders, including CNN [47).

---

Table 2: mmxx GO term and EC number prediction. [*] means the results are taken from best and suboptimal results are shown in bold and underline.
28 The
Category
Method
GO-BP
GO-MF
GO-CC
EC
ResNet
0.280
0.405
0.304
0.605
Sequence
Transforma
22
0.264
0.211
0.405
0.238
GCN 37
0.252
0.195
0.329
0.320
Structure
GAT 48
0.284
0.317
0.385
0.368
3DCNN_MS MQA 24
0.240
0.147
0.305
0.077
GraphQA 25
0.308
0.329
0.413
0.509
GVP [26]
0.326
0.426
0.420
0.489
]EConv (residue level) 12
0.421
0.624
0.431
GearNet 14]
0.356
0.503
0.414
0.730
Sequence- Structure
GearNet-IECCnn 14
0.381
0.563
0.422
0.800
GearNet-Fdge 14
0.403
0.580
0.450
0.810
GearNet-Fddge
14
0.400
0.581
0.430
0.810
CDConv (28]
0.453
0.654
0.479
0,820
CoupleNet (Proposed)
0.467
0.669
0.494
0.866
ResNet [22]. LSTM 122 and Transformer 22
2) Structure-baser methods (GCN 37 GAT 48
3DCNN MQA 24, IEConv (atom level) 12 Sequence-struee based models, e.g.. GVP 26
ProNet 15 GearNet [4] CDConv 28 etc.
4.2 Results of Fold and Reaction Classification, EC and GO term Prediction.
Table I provides the comparisons on the fold and enzyme reaction classification. From this table, we can see that the proposed model CoupleNet achieves the best performance across all four test sets on the fold and enzyme reaction classification compared with recent state-of-the-rr methods. Especially on the Fold and SuperFamily test sets, CoupleNet improves the results by about 4%, showing that CoupleNet is proficient at learning the mappings between protein sequences, structures, and functions. Moreover, CDConv [28] ranks second among these methods. Both CDConv and our method are implemented by sequence- structure convolution. This phenomenon illustrates that coupling sequences and structures of proteins conducive learning better protein embeddings. Our proposed model utilizes complete geometric representations and specially designed dynamically changed graphs, achieving state-of-the-art results.
srruuncc-tttuuuree
We follow the split method in 14 to guarantee that the test set only comprises PDB chains with sequence identity no higher than 95% to the training set for GO term and EC number prediction. Proteins are organized into three ontologies: molecular function (MF), biological process (BP), and cellular component (CC) for the task of GO term prediction. Table compares different protein modeling methods on GO term prediction and EC number prediction. The proposed model, CoupleNet yields the highest F Fmax across these four test sets of two tasks, outperforming other prevalent models. This indicates that CoupleNet can effectively predict the functions, locations, and enzymatic activities of proteins.
We learn protein representations terms of protein sequences and structures, which is essential as building hierarchical dependencies gets universal representations when there is low similarity between the training and test sets. We compare the protein graph methods, GearNet, and the proposed CoupleNet by different cutoff splits. Proteins in the test are categorized into four groups based on their similarity to the training set (30%, 40%, 50%, 70%), not by the default split rate (95%). The results shown in Figure 9 in the appendix, which indicate that even when there is low similarity between the training and test sets, our model also has the highest scores, which demonstrates the superiority and robustness of the proposed model.
4.3 Analysis of Experiments on Protein Length

---

Table Ablation of CoupleNet, we compare with the base model, CoupleNetas: and the models
removing the sequence (w/o sequence). structure, or related geometries.
Fold Classification
Enzyme
GO
Method
EC
Fold
SuperFamily
Family
Reaction
BP
MF
CC
CoupleNet
60.6
82.1
99.7
89.0
0.467
0.669
0.494
0.866
CoupleNeta.
57.8
78.7
99.6
88.6
0.458
0.660
0.484
0.851
w/o sequence
60.0
81.6
99.6
88.4
0.441
0.650
0.456
0.700
w/o structure
26.1
36.4
92.9
81.3
0.406
0.586
0.427
0.625
w/o $, V, Ω 60.3
81.3
99.6
88.7
0.463
0.666
0.490
0.862
w/o d, w, 0, 60.4
81.5
99.7
88.9
0,461
0.666
0,488
0.864
To assess the model's proficiency in handling proteins of varying lengths, we conducted a categorization based on the sequence length of proteins using the mean length of the unseen test set as a threshold. This allowed us to dis- tinguish between small and large proteins. For instance, proteins were grouped into two cate- gories based on the mean length: 149.4 for Fold, 186.7 for SuperFamily, and 162.4 for Family, 299.8 for the GO dataset.
2008
606
0.43
705
605
505
405
206
20R
108
UN
Foid
SuperFamlly
Family
GO
60-MF
60-0C
The protein Fold classification task involves identifying remote homology relationships (i.e., determining proteins belong to the same fam- ily or superfamily), CoupleNet demonstrates its capability to capture such relationships. No- tably, higher accuracies are observed for relatively large proteins with sequence lengths surpassing the mean length, as indicated in Figure On the other hand, the GO term prediction task exhibits less correlation with protein sequence length, which relies more on the local environment of residues [49].
Laagge a Smalll
Figure 4: Percentage accumulation chart of results on large and small proteins. The vertical axis shows the percentage, where the red line indicates 50%.
4.4 Ablation Study
We examined the impact of removing the sequence information, which means removing the encoding of amino acid types for each node; removing the structure information, which means removing fea- tures related to protein geometry (F(G)ao 4 V, S2, dc, w, 0, s, subscripts are omitted for brevity); removing the backbone torsion angles (w/o Φ V,, (2) and removing the interresidue geometric struc- ture representations (w/o dc, w, 0, sp). shown Table 3 removing either sequence or structure causes performance drop all tasks, demonstrating that both types of information are critical for the proposed method. When removing the structure, the performance decreases more significantly, suggesting that structural information provides more important and comprchensive clues compared with sequence information alone. From these tables, we can also find that these complete geometries (Φ) V,, S2 and dc,,,, W. 6, (p) provide similar information, with one of their removals leading minor performance drops for the reason that they both provide complete geometries, but from different perspectives. Compared with CoupleNel,a. CoupleNet achieves significant improvements on the four tasks, demonstrating the importance of complete representations at the backbone level in learning protein embeddings.
5 Conclusion and Limitation
In this work, we propose CoupleNet, novel protein representation learning method that dynamically fuses protein sequences and multi-level structures by conducting convolution on graph nodes and edges. We design the sequential and radius graphs, achieving completeness on them at different protein structure levels. Our approach achieves state-of-the-er results on the protein tasks, demon- strating the superiority of our proposed method. limitation that this framework needs protein sequences and structures to be available, which may not be suitable for the sequence-only data as the input. future direction to develop a large-scale model based on the multi-modal protein data types to enhance performance.

---

Acknowledgements
This work was supported by the National Science and Technology Major Project (No. 15101), the National Natural Science Foundation of China Project (No. U21A20427 and No. 623B2086), Project (No. WU2022A009) from the Center of Synthetic Biology and Inte- grated Bioengineering of Westlake University and Project (No. WU2023C019) from the Westlake University Industries of the Future Research Funding.
References
[1] Andrew W Senior, Richard Evans, John Jumper. James Kirkpatrick, Laurent Sifre. Tim Green, Chongli Qin, Augustin Zídek, Alexander WR Nelson, Alex Bridgland, et al. Improved protein structure prediction using potentials from deep learning. Nature, 577(7792):706777 2020.
[2] Bin Ma. Novor: real-time peptide de novo sequencing software. Journal of the American Society for Mass Spectrometry, 26(1 11:1111:18188888 1894, 2015.
[3] UniProt Consortium. Uniprot: a worldwide hub of protein knowledge. Nucleic acids research, 47(D1):D506 D515, 2019.
[4] Noelia Ferruz and Birte Höcker. Controllable protein design with language models. Nature Machine Intelligence, 4(6))521-532, 2022.
[5] Bozhen Hu, Jun Xia, Jiangbin Zheng, Cheng Tan, Yufei Huang, Yongjie Xu, and Stan Z Li. Protein language models and structure prediction: Connection and progression. arXiv preprint arX1v:2211.16742, 2022.
[6] Yue Liu, Xiaoxin He, Miao Xiong, Jinlan Fu, Shumin Deng, and Bryan Hooi. Flipattack: Jailbreak lims via flipping. arXiv preprint arXiv:2410.02832,, 2024.
[7] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, al. Prottrans: Toward understanding the language of life through self-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 44(10):1112-122) 2021.
[8] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 1 (18(15):e201623) 18, 2021.
[9] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. BioRxiv, 2022: 500902, 2022.
[10] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool Russ Bates, Augustin Židek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-588) 2021.
[11] Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, al. Alphafold protein structure database: massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic acids research, 50(D1):D439 (441)) 2022.
[12] Pedro Hermosilla, Marco Schäfer, Matēj Lang, Gloria Fackelmann, Pere Pau Vázquez, Barbora Kozliková, Michael Krone, Tobias Ritschel, and Timo Ropinski. Intrinsic-extrinsic convolution and pooling for learning on 3d protein structures. arXiv preprint arXiv:2007.06255 2020.
[13] Pedro Hermosilla and Timo Ropinski. Contrastive representation learning for 3d protein structures. arXiv preprint
2022.
10

---

[14] Zuobai Zhang, Minghao Xu, Arian Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Protein representation learning by geometric structure pretraining. In The Eleventh International Conference on Learning Representations, 2023.
[15] Limei Wang, Haoran Liu, Yi Liu, Jerry Kurtin, and Shuiwang Ji. Learning hierarchical protein representations via complete 3d graph networks. In The Eleventh International Conference on Learning Representations, 2023.
[16] Fang Wu, Dragomir Radev, and Jinbo Xu. When geometric deep learning meets pretrained protein language models. bioRxiv, pages 2023-01, 01, 2023.
[17] Vladimir Gligorijević, Douglas Renfrew, Tomasz Kosciolek, Julia Koehler Leman, Daniel Berenberg, Tommi Vatanen, Chris Chandler, Bryn C Tayylor Ian M Fisk, Hera Vlamakis, et al. Structure-based protein function prediction using graph convolutional networks. Nature communications, 12(1):3168, 2021.
[18] Cheng Tan, Zhangyang Gao, Jun Xia, Bozhen Hu, and Stan Li. Generative de novo protein design with global context. arXiv preprint v7Xi:2204.10673. 2022.
[19] Meng Liu, Zhengyang Wang, and Shuiwang Ji. Non-local graph neural networks. IEEE transactions on pattern analysis and machine intelligence, 44(12):102701000 10276, 2021.
[20] Kaixiong Zhou, Xiao Huang, Yuening Li, Daochen Zha, Rui Chen, and Xia Hu. Towards deeper graph neural networks with differentiable group normalization. Advances in neural information processing systems, 33:4917-49255 2020.
[21] Dušan Petrović, Valeria A Risso, Shina Caroline Lynn Kamerlin, and Jose M Sanchez-Ruiz Conformational dynamics and enzyme evolution. Journal of the Royal Society Interface, 15 (144):20180330, 2018
[22] Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. Evaluating protein transfer learning with tape. Advances in neural information processing systems, 32, 2019.
[23] Roshan M Rao, Jason Liu, Robert Verkuil, Joshua Meier, John Canny, Pieter Abbeel, Tom Sercu, and Alexander Rives. Msa transformer. In International Conference on Machine Learning. pages 8844-8856. PMLR, 2021.
[24] Georgy Derevyanko, Sergei Grudinin, Yoshua Bengio, and Guillaume Lamoureux. Deep convolutional networks for quality assessment of protein folds. Bioinformatics, 34(23):4046 4053, 2018.
[25] Federico Baldassarre, David Menéndez Hurtado, Arne Elofsson, and Hossein Azizpour. Graphqa: protein model quality assessment using graph convolutional networks. Bioinfor- matics, 37(3):360-366, 2021.
[26] Bowen Jing. Stephan Eismann, Patricia Suriana, Raphael JL Townshend, and Ron Dror. Learn- ing from protein structure with geometric vector perceptrons. arXiv preprint arXiv:2009.01411, 2020.
[27] Sheng Chen, Zhe Sun, Lihua Lin, Zifeng Liu, Xun Liu, Yutian Chong, Yutong Lu, Huiying Zhao, and Yuedong Yang. To improve protein sequence profile prediction through image captioning on pairwise residue distance map. Journal of chemical information and modeling, 60(1):3913399, 2019.
[28] Hehe Fan, Zhangyang Wang, Yi Yang, and Mohan Kankanhalli. Continoous-discrete convolu- tion for geometry-sequence modeling in proteins. In The Eleventh International Conference on Learning Representations, 2023.
[29] Ruijie Quan, Wenguan Wang, Fan Ma, Hehe Fan, and Yi Yang. Clustering for protein represen- tation learning. arXiv preprint arXiv:2404. 002544 2024.
[30] Can Chen, Jingbo Zhou, Fan Wang, Xue Liu, and Dejing Dou. Structure-aware protein self- supervised learning. Bioinformatics, btdd): 2023.

---

[31] Zeyuan Wang, Qiang Zhang, Hu Shuangwei, Haoran Yu, Xurui Jin, Zhichen Gong, and Huajun Chen. Multi-level protein structure pre-training via prompt learning. In The Eleventh International Conference on Learning Representations, 2023.
[32] Yi Liu, Limei Wang, Meng Liu, Yuchao Lin, Xuan Zhang, Bora Oztkkin, and Shuiwang Ji. Spherical message passing for 3d molecular graphs. In International Conference on Learning Representations, 2021.
[33] Limei Wang, Yi Liu, Yuchao Lin, Haoran Liu, and Shuiwang Ji. Comenet: Towards complete and efficient message passing for 3d molecular graphs. arXiv preprint arXiv:2206.088 08515, 2022.
[34] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. International conference on machine learning. pages 1263-1272. PMLR, 2017.
[35] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se (3)-transformers: 3d roto-translation equivariant attention networks, Advances in Neural Information Processing Systems, 33:1970-1981, 2020.
[36] Weitao Du, He Zhang, Yuanqi Du, Qi Meng, Wei Chen, Nanning Zheng, Bin Shao, and Yan Liu. Se (3) equivariant graph neural networks with complete local frames. In International Conference Machine Learning, pages 5583-5608. PMLR, 2022.
[37] Thomas N Kipf and Max Welling. Sumi-supervised classification with graph convolutional 2016.
networks. arXiv preprint
[38] John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph-based protein design. Advances in neurul information procexsing systems, 32, 2019.
[39] Gang Xu, Qinghua Wang, and Jianpeng Ma. Opus-rota4: gradient-based protein side-chain modeling framework assisted by deep bearning-based predictors. Briefings in Bioinformatics, 23( (3(1):bber bbbbb552,,, 2022.
[40] Jack Hanson, Kuldip Paliwal, Thomas Littin, Yuedong Yang, and Yaoqi Zhou. Improving predic- tion of protein secondary structure, backbone angles, solvent accessibility and contact numbers by using predicted contact maps and an ensemble of recurrent and residual convolutional neural networks. Bioinformatics, 35114))2403-2 2410, 2019.
[41] Manfred S Weiss, Andreas Jabs, and Rolf Hilgenfeld. Peptide bonds revisited. Nature structural biology, 5(8):676-676, 1998.
[42] David L Nelson, Albert L Lehninger, and Michael M Cox. Lehninger principles of biochemistry. Macmillan, 2008
[43] K Peter C Vollhardt and Neil E Schore. Organic chemistry: structure and function. Macmillan, 2003.
[44] Jianyi Yang, Ivan Anishchenko, Hahnbeom Park. Zhenling Peng, Sergey Ovchinnikov, and David Baker. Improved protein structure prediction using predicted interresidue orientations. Proceedings of the National Academy of Sciences, 117(3):1496 1503, 2020.
[45] Bin Huang, Tingwen Fan, Kaiyue Wang, Haicang Zhang, Chungong Yu, Shuyu Nie, Yangshuo Qi, Wei-Mou Zheng, Jian Han, Zheng Fan, Shiwei Sun, Sheng Ye, Huaiyi Yang, and Dongbo Bu. Accurate and efficient protein sequence design through learning concise local environment of residues. Bioinformatics, 39(3): 03 2023.
[46] G Marius Clore and Angela M Gronenborn. Nmr structure determination of proteins and protein complexes larger than 20 kda. Current opinion in chemical biology. 2(5):564-570, 1998.
[47] Amir Shanchsazzadeh, David Belanger, and David Dohan. Is transfer learning necessary for protein landscape prediction? arXiv preprint (XXXv:2011.034433 2020.
[48] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.10003, 2017.
12

---

[49] Ilya Levental and Ed Lyman. Regulation of membrane protein structure and function by their lipid nano-environment. Nature Reviews Molecular Cell Biology, 24(2):10--222, 2023.
[50] Jie Hou, Badri Adhikari, and Jianlin Cheng. Deepsf: deep convolutional neural network for mapping protein sequences to folds. Bioinformatics, 34(8): 1295-13033 2018.
[51] Edwin C Webb al. Enzyme nomenclature 1992. Recommendations of the Nomenclature Com- mittee of the International Union of Biochemistry and Molecular Biology on the Nomenclature and Classification of Enzymes. Number Ed. 6. Academic Press, 1992.
[52] Marina Omelchenko, Michael Y Galperin, Yuri Wolf, and Eugene Koonin. Non- homologous isofunctional enzymes: systematic analysis of alternative solutions in enzyme evolution. Biology direct, 5:1-20, 2010.
[53] Helen M Berman, John Westbrook, Zukang Feng. Gary Gilliland, Talapady Bhat, Helge Weissig, Ilya N Shindyalov, and Philip E. Bourne. The protein data bank, Nucleic acids research, 28(1):235-2444 242, 2000.
[54] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
13

---

A Protein Sequence and Structure
There exist relationships between sequence and structure, shown in Figure the contact maps be inferred from sequences to obtain 3D coordinates, while the distant contacts provide constraints for multiple sequence alignment (MSA), Besides, protein sequence and structure provide different types of information (Figure 6), like residue types and 3D arrangements.
MSA ADI
Residue contact
Structure
ADRLI
A
II G
ADKLYCT H D E ADTLY MIM HOK A TDILY Y T DE Q AD L T R
Statistical inference
ADRLYMI M H H
G
Constraint
T ESRRY
G
O
ADRL
G
G
ADRL
Coevolution
Cuevolved positions
Figure 5: Relationship between protein sequences and the structure of one protein in the alignment. The positions that coevolved are highlighted red and light blue. Residues within these positions where changes occurred shown in blue. Given such MSA, found between two residues that these sequence positions are spatially adjacent, i.e., they are contacts [5]. The protein tertiary structure be inferred from such contacts.
infer correlations statistically
Primary structure
Tertiary structure
GLU
Amino acids: AISGLPFKKL.KVESRIDDD
Positions: 1,2,.../f
Figure 6: Illustration of protein sequence and structure. 1) The primary structure comprises amino acids. 2) The tertiary structure with atom arrangement in Euclidean space is presented, where each atom has specific 3D coordinate. Amino acids have fixed backbone atoms (Ca, C, O) and side-chain atoms that vary depending on the residue types.
B SE(3)
The collection of 4 real matrices of the SE(3) is shown as:
F11
T12
T13
[
R
F21
T22
T23
12
(1I)
F31
T32
T33
13
0
0
1
where R E SO(3) and E R3, SO(3) is the 3D rotation group. R satisfying Rd R = and (Rl 1.
C
Geometric Features
Firstly, we present the local coordinate system, which is shown in Figure a(a), b, is the negative bisector of the angle between the rays (P:-1,c, Pr.c.) and (P++1.c. PP.c...
Secondly, we introduce the relative rotations and distances in trRosetta 44 including the distance (dig, C2), ) three dihedral angles (wij. Ozg, 0zx)) and two planar angles (piy, (pt)), shown in Figure 7
14

---

Cj
Cşs
C
Wij
Pet1.a
AnL
dajes
residue
N₂
tü++1
Pa..a
Cas
APs residue
b;
0ij
Ui
Cat
b: 14
P--
N
C;
(a)
(b)
Figure 7: Protein geometries. (a) The local coordinate system. Pac., is the coordinate of C, in residue i. (b) Interresidue geometries including angles and distances, including the distance (dij,cp). three dihedral angles (wigg Oig. Ofe) and two planar angles (Pij, Pj))
D Social Impact
The development of CoupleNet marks a significant advancement in the realm of protein research with profound potential for societal impact. By effectively bridging the gap between protein sequences and structures, CoupleNet offers unprecedented insights into the intricate world of proteins and their biological functions. This breakthrough has far-reaching implications for various fields, including medicine and biotechnology. With the ability to accurately predict protein functions, CoupleNet can expedite drug discovery processes, enabling the development of novel therapeutics for combating diseases more efficiently. Furthermore, its capability to identify infrequently encountered functions and capture remote homology relationships promises revolutionize our understanding of protein evolution and diversity, paving the way for innovative solutions in fields such as personalized medicine and bioprocessing. Ultimately, the deployment of CoupleNet has the potential drive transformative advancements in healthcare and biotechnology. ultimately leading to improved human health, enhanced crop yields, and sustainable industrial practices.
E Tasks
Fold Classification. Protein fold is predict the fold class label given protein, which is crucial for understanding how protein structure and protein evolution interact [50]. In total, this dataset contains 16, 712 proteins with 1, 195 fold classes. There three test sets available. Fold: Training excludes proteins from the same superfamily. Superfamily: Training does not include proteins from the same family. Family: Proteins from the same family are included in the training.
Enzyme Reaction Classification. Reaction categorization aims to predict a protein's class of enzyme-catalyzed reactions, according to all four levels of the EC number [51 52). Following the setting in [12]. this dataset has 37, 248 proteins from 384 four-level EC numbers 53
GO Term Prediction. The goal of GO term prediction is to foretell whether protein related to a certain GO term. Following [17), The three ontologies, MF, BP, and CC, hierarchically connected, functional classes. MF describes activities that occur at the molecular level, BP represents the larger processes, and CC describes the parts of cell or its extracellular environment (3).
EC Number Prediction. This task seeks to predict the 538 ECC numbers from the third level and fourth levels of different proteins [17], which describe their catalysis of biochemical reactions.
15

---

Details of Datasets and Training Setup
For all datasets, data augmentation strategy by adding noise to the training set to increase the variability data. For example, we update the position of Cau.
P.C.a Pi... NNNN, x
(12)
where ,.,, N are the mean (expectation) and variance of the normal distribution, which are set to and 0.1 in experiments, Dataset statistics of our four downstream tasks are summarized in Table
Table 4: Dataset statistics. #X means the number of X.
Dataset
#Train
# #aliddtion
#Test
EC
15, 550
1, 7..29
1, 1,919
GO
29, 898
3, 3,322
3,415
Fold
12,312
736
718
Superfamily
12,312
736
1,254
Family
12,312 Reaction Classification
736
1,272
29, 215
2, 2,2
5, 6,65
Settings. The proposed models are trained with the Adam optimizer [54] which are conducted on single NVIDIA-SMI A100 GPU through PyTorch 1.13+cul and PyTorch Geometric 2.3.1 with CUDA 11.2. The number of the initial feature channels is 256. The learning rate is set to 0.001. The radius threshold increases from 4 to 16, and is set to 11. For the number of pooling layers that related to the depth of the network, hyperparameters for them. There are four pooling layers that are sufficient to achieve satisfactory results; every two message-passinng passing layers are followed by average pooling layer. Thus, there eight message- message-passing layers. In detail, the sequence average pooling functions perform customized average pooling operations
employ the grid search trick to search suitable
the input tensors based on the calculated indices (dividing the length of the sequence by 2 and flooring the result). It aggregates and summarizes information from the input tensors using scatter operations produce the output tensors. As for the batch size and training epochs, etc., which influence the convergence speed of deep learning models, we also adopt the grid search method to get group of values; the details shown in Table
Table 5: More details of training setup
Fold Enzyme Reaction GO EC
Batch size Epoch
24 64
400
400
500 500
G
Evaluation Metric
F Fiaax is calculated by first determining the precision and recall for each protein, then averaging these v is the prediction probability for the j-th class of the i-th protein, given the decision threshold [0, 1], the precision and call give as:
results over all proteins [14 17 28
E, 5 2 mo
precision,
E, 2t)1
(13)
E, II ( n9!)
recall,
16

---

200
123
50
250
500
750
1000
1250
sequence
(a) Fold
30
1250
2500 2730 1250
(b) Enzyme Reaction
200
spataal spataa
100
250
500
750
Sequeiice distacce
(c)
Figure 8: The violin plot of the relationships of distances between sequence and structure on the fold and enzyme reaction classification and EC number prediction datasets.
17

---

where b/ E {0,1} is the corresponding binary class label, and E {0, 1) is an indicator function. If there are N proteins total, then the average precision and recall are defined as:
Σ precision, Σ [E, p1 (11
precision(t)
(1)))) 21)
(14)
recall(t) Σ. recall.(1)
N
Finally, Finxx is defined the maximum value of F-score over all thresholds,
precision(() recall precision(() + recall(t)
F max max
(15)
H Completeness Analysis
(V,E,P), capture the geometric representations based on the atoms' 3D positions and use sequential and structural representations as the node and edge features. For 3D structure, based on the definition of completeness and the rigorously demonstrated method to show the calculated geometries can achieve completeness for structures [[5], we guarantee the completeness of the selected geometric representations the amino acid and backbone levels of structures.
Given a protein 3D graph
The geometric representations SE(3) invariant (distances, angles) and SB(3) equivariant (directions, orientations). Therefore, it is natural for Eq. to hold from right to left. To demonstrate Eq. 14433holding from left to right, need to show F(G) T2 (P), where Tg does not change the 3D conformation of 3D graph. Thus, we need to show positions can be determined by F(G).
The base approach CoupleNetaa only considers the C. coordinates and constructs LCS for each residue. F(G) provides complete representations. First, when n 1, it holds. Assume the case holds such that F(G).a am is complete. Then, need prove the case = k + still holds. This is obvious because if fy is the (l: + 1) 1)-th node connected to node v; among the existing & nodes, the LCS 2j can be easily obtained from Q; and F(G)
When considering the backbone atoms C., C, O, F(G) is complete. shown in Figure I the remaining degree of freedom at the backhone level is the rotation angles Φ;, V,, Ω based on the rigid bond lengths and angles. Such backbone torsion angles are calculated and concatenated with a, into x,, Besides, for any residues and j. the calculated six inter-residue geometries fully define the relative locations of backbone atoms. Therefore, there are no other remaining degrees of freedom. Consequently, the obtained geometric representations at the backbone level are complete.
I Analysis of Imbalanced Class Experiments
The classes in the protein fold classification task tend to be imbalanced. This dataset contains 16,712 proteins with 1,195 fold classes, including three categories: Fold, SuperFamily, and Family. We provide the top 10 most frequent classes and the mean prediction accuracies on Fold, SuperFamily, and Family test sets. The results are presented in Table where #Class means the total number of samples of this class. From the table, we can find that the proposed model, CoupleNet, is not heavily affected by the category unbalance problem. For example, there are 1.254 proteins with 1,187 classes in the SuperFamily dataset. A large number of classes have only one protein sample, but we got mean accuracy of 82. percent on SuperFamily.
18

---

0.700
0200
0.535
0.511
0.600
0.300
0,419
6428
0467
0,500
0.444
0.333
0.478
8-57
0,59
0.350
6313
0.35
0.5
6.000
0,552
0.20
0.200
0.200
0.100
0.100
0.000
307
70%
95%
30%
40%
50%
98%
(a) GO-ME
(b) GO-BP
6.000
04344
1.00
8.666
0:501
0,423
6-4F7
1,50
8:553
0,401
4.40
0.31
0:451
8,4T0
8:00
9.500
0.533
0,5N0
0.645
0,599
0,352
0.15
A.700
0:300
9:500
8.100
0.00
30%
40%5
50%
70%
95%
30%
40%
50%
70%
90%
GeerNet
GCearNee
CDCler
CCopleeNet
Gearrha-lidge
CDCCser
CoupleNet
(c) GO-CC
(d) EC
Figure 9: max on GO term and EC number prediction under different cutoffs.
Table 6: Accuracy (%) on fold classification, the top 10 most frequent classes, and the mean prediction accuracies are listed.
Fold
SuperFamily
Family
Class
# Class
Accuracy
Class
# Claas
Accuracy
Class
# Class
Accuracy
661
30
86.7
36
1.0
284
181
1.0
482
28
64.3
661
34
94.1
491
34
1.0
458
28
96.4
458
29
93.1
458
29
1.0
1068
20
85.0
23
78.3
459
23
1.0
284
20
80.0
284
20
1.0
1107
20
1.0
364
20
15.0
328
19
1.0
661
19
1.0
1
16
81.3
546
19
1.0
619
19
1.0
400
73.3
519
18
3
1.0
1107
14
78.6
509
16
1.0
501
16
1.0
23
71.4
547
16
1.0
321
1.0
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction mentioned protein representation leaning, design, results and findings.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Limitations have been mentioned these in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
We provide complete protein representations, which is demonstrated in Sec-
tion[H]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Dataset, models and settings are introduced in this paper.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [Yes]
Justification: Codes are in the supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctpppp.ccppulli//guiiie wwwiicccguie///eeeeeeeeis CoodeSubeiiieeee ssi wswinnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Experimental settings are introduced in Appendix Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Mean values of five different initializations are reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
22

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Computer resources are reported in Appendix F
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: htttps://eeut http::/:////ips/.eeeeeeee http://.....cooccocttccccsceeaaa
Answer:
[Yes]
Justification: make sure to preserve anonymity.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Potential Social impact is presented in Appendix D
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
23

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: the paper does not have such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The rightful owners of the assets utilized in this paper, including code, data, and models, are duly acknowledged and credited in accordance with ethical and academic standards.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, pwpprswi www.wwtthth.....c com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve such content.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "Ic23bZXxtp", "reviewer": "Reviewer_fLEX", "summary": "The authors tackle the limitation of modeling inter-dependencies between protein sequences and structures. To solve this limitation, this work proposes CoupleNet which dynamically couples protein sequences and structures. Specifically, the authors propose to construct two-type dynamic graphs (sequential graph and radius graph) and execute convolutions on nodes and edges to encode proteins.", "questions": "* Which pooling strategy did you adopt? Please elaborate on the pooling process.\n* Could you explain the advantages of this work from the difference with existing methods in Section 3.4?", "limitations": "The authors discuss the limitation in Section 5.", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 3, "strengths": "* The proposed method is memory-efficient as it adopts hierarchical pooling.\n* Considering different types of features such as torsion angles, dihedral angles, and planar angles, and conducting the ablation study on them is interesting.", "weaknesses": "* The tackling problem has already been solved by recent works [1, 2] which weakens the contribution of the proposed work. ESM-GearNet [1] successfully fuses the sequential and structural representations, and SaProt [2] develops a structure-aware vocabulary allowing to fusing the sequential and structural properties in one model. Also, the pre-training strategy of GearNet [3] could solve the problem as it aims to learn the similarity between the subsequence graph and the structural (radius) graph. I suggest the authors concisely compare those works and elaborate on the advantages of the proposed method.\n* For the experimental results, the related works [1, 2] that model the sequences and structures simultaneously should be compared.\n* The ablation study is not related to the core techniques of this paper. Even though the core techniques of this paper are dynamic pooling and constructing sequential and radius graphs, the authors do not consider them for the ablation study, hindering the readers from knowing what component largely contributes to the performance.\n* As dynamic pooling is one of the core ideas of this paper, in the main body, the authors should explain what kind of pooling strategy is leveraged, what the effect of the proposed dynamic pooling is, and why it is important to model the two graphs.\n\n[1] Zhang et al., \"Enhancing Protein Language Models with Structure-based Encoder and Pre-training\", ArXiv, 2023.\n\n[2] Su et al., \"SaProt: Protein Language Modeling with Structure-aware Vocabulary\", ICLR, 2024.\n\n[3] Zhang et al., \"Protein Representation Learning by Geometric Structure Pretraining\", ICLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer fLEX,\n\nWe are grateful for your thorough review. Your comments are highly valued, and we would like to express our heartfelt gratitude. We do our utmost to address the questions you have raised:\n\n**Q1** I suggest the authors concisely compare those works and elaborate on the advantages of the proposed method.\n\n**A1** Thank you for your valuable feedback! (1) As shown in Lines 221-229 in the manuscript, there are only two different types of graphs in the proposed CoupleNet: radius graph and sequential graph; we did not use the k-nearest graph, as some nodes have distant neighbors when having $k$ neighbors. (2) The threshold in the radius graph in GearNet is set to be constant, but we change the threshold of radius dynamically to learn different distance relationships. (3) The message passing mechanism only executes on nodes in CoupleNet instead of on nodes and edges alternately used in GearNet. CoupleNet performs convolutions on nodes and edges simultaneously with several pooling layers. (4) ESM-GearNet combines ESM embeddings with GearNet embeddings; SaProt also uses ESM embeddings; it only uses the $C_\\alpha$ coordinate at the structure level, but CoupleNet models the coordinates of all backbone atoms completely.\n\n**Q2** Comparisons with the related works.\n\n**A2** Thank you for your informative reviews! Factually, CoupleNet is not a pre-training model; it is not fair to compare it with pre-training methods. Thus, we combine ESM-2 (650M) [1] with CoupleNet, named ESM-CoupleNet, using generated ESM embeddings as one part of graph node features. We compare ESM-CoupleNet with pre-training methods on protein function prediction and EC number prediction, including sequence-based methods, ESM-1b [2], ESM-2; sequence-function based methods, ProtST [3]; sequence-structure based methods ESM-S [4], GearNet-ESM [5], SaProt [6]. The comparison results are provided in Table 1 of the one-page rebuttal pdf. From this table, we can see that our proposed model, ESM-CoupleNet, achieves the best results among sequence-based, sequence-structure based, and sequence-function based pre-training methods.\n\n**Q3** The ablation study is not related to the core techniques of this paper. \n\n**A3** Thank you for your reviews! (1) In this paper, a novel two-graph-based approach for modeling sequential and 3D geometric features is proposed, ensuring global completeness in protein representation, which has been demonstrated in the appendix. We have done the ablations on the sequential and radius graphs by removing different input features, as shown in Table 3 in the paper. (2) CoupleNet performs concurrent convolutions and sequence poolings on nodes and edges. Convolution and pooling operations consist of the network. We remove the pooling operations in the network, and the results are shown in the following table. We can see without pooling operations the model's performance degrades significantly.\n\n| Method | GO-BP | GO-MF | GO-CC | EC |\n| :--- | :---: | :---: | :---: | :---: |\n| CoupleNet | 0.467 | 0.669| 0.494 | 0.866 |\n| w/o pooling | 0.362 | 0.535 | 0.420 | 0.748 |\n\nBesides, in Figure 4 of the manuscript, CoupleNet demonstrates its capability to capture long-range relationships; higher accuracies are observed for relatively large proteins with sequence lengths surpassing the mean length. This also illustrates the effectiveness of the dynamic pooling operations.\n\n**Q4** Questions about pooling strategy.\n\n**A4** Thank you for your valuable reviews! (1) As shown in Section 3.4, Appendix F, and Figure 2, we employ complete message passing and sequence pooling layers to obtain the deeply encoded graph-level representations. Every two message-passing layers are followed by an average pooling layer. There are eight message-passing layers in the model. The sequence average pooling functions perform customized average pooling operations on the input tensors based on the calculated indices (dividing the length of the sequence by two and flooring the result). It aggregates and summarizes information from the input tensors using scatter operations to produce the output tensors (torch_scatter.scatter_mean). (2) After one average pooling layer, the number of residues reduces by half, and we expand the radius threshold $r$ to $2r$ after once pooling, which makes neighbors of center nodes gradually cover more distant and rare nodes, also reducing the computational complexity. These operations make the model capture the local to the global features. \n\n**Q5** Could you explain the advantages of this work from the difference with existing methods in Section 3.4?\n\n**A5** Thank you! We have stated the differences in Section 3.4. We will rectify it to explain extra advantanages in the revised version. For example, our model are more memory-efficient as we adopt dynamic (hierarchical) pooling to reduce the seuqnce length, the dynamically changed graphs can better model the node-edge relationships. We model the sequential and 3D geometric features, ensuring global completeness in protein representations.\n\nThank you again! In case our answers have justifiably addressed your concerns, we respectfully thank you that support the acceptance of our work. Also, please let us know if you have any further questions. Look forward to further discussions!\n\n[1] Lin, Z., et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.\n\n[2] Rives, A., et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. PNAS, 2021.\n\n[3] Xu, M., et al. Protst: Multi-modality learning of protein sequences and biomedical texts. ICML. 2023.\n\n[4] Zhang, et al. Structure-informed protein language model. arXiv, 2024.\n\n[5]  Zhang, et al. A systematic study of joint representation learning on protein sequences and structures. arXiv, 2023.\n\n[6] Su et al. SaProt: Protein Language Modeling with Structure-aware Vocabulary. ICLR, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KsU8NfRWKA", "reviewer": "Reviewer_ANTD", "summary": "This work proposes the CoupleNet, a novel framework designed to interlink protein sequences and structures to derive informative protein representations.\n It integrates multiple levels and scales of features, constructing a dynamic graph to capture both local and global structural geometries. \nExperimental results demonstrate that CoupleNet outperforms state-of-the-art methods in multiple different tasks, e.g., folding / reaction classification, GO term / EC number prediction.", "questions": "1. For the structure module, is the model robust to perturbations? Or, the model can aware of small changes in the structures? NMR PDB files which includes multiple models can be a test set.\n\n2. The structure files are not available for real problems. I wonder what's the difference (in the model output space) between Alphafold structures or other folded structures and the real structures? Especially, on multiers or interface data.", "limitations": "n/a", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "[+] The proposed method is well-introduced and easy to follow.\n[+] The improved performance in protein function prediction makes CoupleNet highly valuable for practical applications in biology and medicine.", "weaknesses": "1. The experiment results do not include error bars and related analysis about multiple trials.\n\n2. The proposed method looks quite general. Applying the architecture to more real-world problems, e.g., residue design / engineering may be helpful. Or, including some discussions about these topics may be useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ANTD,\n\nWe are grateful for your thorough review. Your comments are highly valued, and we would like to express our heartfelt gratitude. We do our utmost to address the questions you have raised:\n\n**Q1** The experiment results do not include error bars and related analysis about multiple trials.\n\n**A1** Thank you for your valuable feedback! We have done multiple trials. The performance is measured with mean values for five different initializations. We report the mean (variance) for the proposed CoupleNet:\n\n| Method | Fold | SuperFamily | Family | Enzyme Reaction | GO-BP | GO-MF | GO-CC | EC |\n| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| CoupleNet | 60.6(0.36) | 82.1(0.63) | 99.7(0.04) | 89.0(0.17) | 0.467(0.005) | 0.669(0.002) | 0.494(0.005) | 0.866(0.008) |\n\n**Q2** The proposed method looks quite general. Applying the architecture to more real-world problems, e.g., residue design / engineering may be helpful. Or, including some discussions about these topics may be useful.\n\n**A2** Thank you for your informative reviews! Protein design is the computational approach to designing amino acid sequences that fold into a predefined protein structure. ESM-IF [1], PiFold [2], and VFN-IF [3] are methods aiming to do protein design, where the task differs from protein representation learning of function prediction. Removing the sequence information, we do protein inverse folding by our method on CATH 4.2, the results are shown in Table 2 in the one-page rebuttal pdf. Our method also achieves almost the best results on this task. \n\n**Q3** For the structure module, is the model robust to perturbations? Or, the model can aware of small changes in the structures? NMR PDB files which includes multiple models can be a test set.\n\n**A3** Thank you for your reviews! We have conducted the noise analysis experiments, the results are shown in Section 4.2. Proteins in the test set are categorized into four groups based on their similarity to the training set, 30\\%, 40\\%, 50\\%, 70\\%, not by the default split rate of 95\\%. The results indicate that even when there is a low similarity between the training and test sets, our model also has the highest scores, which demonstrates the robustness of the proposed model. The model can be aware of small changes in the structures, as we have constructed the complete structural representations to ensure global completeness; this has been demonstrated in Appendix H. Such global completeness is theoretically guaranteed to incorporate 3D information completely without information loss, while the local view would miss the long-range effects of subtle conformational changes happening distantly.\n\n\n**Q4** The structure files are not available for real problems. I wonder what's the difference (in the model output space) between Alphafold structures or other folded structures and the real structures? Especially, on multiers or interface data.\n\n**A4** Thank you for your valuable reviews! The proposed model is a basic framework to model protein sequences and strucutres, if the real structures are not available, we can use Alphafold predicted structures. We have not consider the difference between predicted structures and the real structures, as in experiments, the strucutres we used are all from PDB files. The proposed model can also model protein complex.\n\n\nThank you again for all the efforts that helped us improve our manuscript! In case our answers have justifiably addressed your concerns, we respectfully thank you for supporting the acceptance of our work. As you know, your support holds great significance for us. Also, please let us know if you have any further questions. Look forward to further discussions!\n\n[1] Hsu, C., et al. Learning inverse folding from millions of predicted structures. ICML, 2022.\n\n[2] Gao, Z., et al. PiFold: Toward effective and efficient protein inverse folding. ICLR, 2022.\n\n[3] Mao, W., et al. De novo protein design using geometric vector field networks. arXiv, 2023.\n\n[4] Zhang et al. Protein Representation Learning by Geometric Structure Pretraining. ICLR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v2fa0Z843B", "reviewer": "Reviewer_JwZm", "summary": "This paper considers the problem of learning numerical protein representations using both sequential and structural data, and presents CoupleNet, a method that uses two graph types, one based on the amino acid sequence and the other based on the protein's tertiary structure, to extract protein representations via graph convolutions. Compared with several baselines, it is demonstrated that CoupleNet improves the performance of the learned representations across multiple downstream tasks.", "questions": "- On line 133, it is mentioned that the positions can be recovered from the geometric representations. I believe that is not entirely correct, because the recovered positions may not be unique, but they can be uniquely recovered modulo the transformation $T_g$. Please verify that this is correct.\n\n- Could you please explain how average pooling is done and why the number of residues is reduced by half after every pooling layer? Does this statement mean that every pair of nodes will be combined into one node? How are the pairs determined? How are the edges determined for the combined nodes (since every node now corresponds to two residues with two different positions, instead of a single residue in the original input graph)?\n\n- On lines 227-228 in Section 3.4, it is mentioned that convolutions are performed on edges as well. However, based on the message-passing operation in Eq. (10), it seems that only node-level convolutions are performed, and there are no edge-level convolutions (even though the edge features are used for aggregating features at each layer).", "limitations": "The main limitation of this work is the lack of sufficient structural data as compared to sequential data for proteins, which the authors also allude to in Section 5. It would be helpful to comment on possible extensions to the case where, for some proteins, only sequential data is available, and what would happen if, for those proteins, only the sequence graph is considered, whereas for the rest of the proteins, both sequence and structure graphs are used in the CoupleNet pipeline.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Combining sequence and structure data in proteomics is an important and timely problem. The performance gains of the proposed approach against baseline methods seem significant to some extent.", "weaknesses": "- My main concern with the paper is the lack of comparisons (both in terms of the performance and the novelty of the method) with state-of-the-art protein language models and their structure-aware versions. It would be beneficial to compare the performance of CoupleNet with methods such as ESM-2 [A], ESM-S [B], S-PLM [C], ESM-GearNet [D], or SaProt [E].\n\n  [A] Lin, Zeming, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin et al. \"Evolutionary-scale prediction of atomic-level protein structure with a language model.\" Science 379, no. 6637 (2023): 1123-1130.\n\n  [B] Zhang, Zuobai, Jiarui Lu, Vijil Chenthamarakshan, Aurélie Lozano, Payel Das, and Jian Tang. \"Structure-informed protein language model.\" arXiv preprint arXiv:2402.05856 (2024).\n\n  [C] Wang, Duolin, Mahdi Pourmirzaei, Usman L. Abbas, Shuai Zeng, Negin Manshour, Farzaneh Esmaili, Biplab Poudel et al. \"S-PLM: Structure-aware Protein Language Model via Contrastive Learning between Sequence and Structure.\" bioRxiv (2023): 2023-08.\n\n  [D] Zhang, Zuobai, Chuanrui Wang, Minghao Xu, Vijil Chenthamarakshan, Aurélie Lozano, Payel Das, and Jian Tang. \"A systematic study of joint representation learning on protein sequences and structures.\" arXiv preprint arXiv:2303.06275 (2023).\n\n  [E] Su, Jin, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan. \"Saprot: Protein language modeling with structure-aware vocabulary.\" bioRxiv (2023): 2023-10.\n\n- The writing quality of the paper could be substantially improved, in my opinion. Some examples:\n    - It seems that the graph $G$ on lines 114-115 is fully connected based on the definition of $\\mathcal{E}$, whereas it is not necessarily the case based on the following sections.\n    - The statement \"$T_g$ and $S_g$ are the transformations\" on line 127 needs further elaboration.\n    - What is the domain and range of $\\mathcal{F}$ on line 130?\n    - The description of a message-passing layer in Eq. (4) is unnecessarily restrictive, since not all message-passing mechanisms use batch normalization or a fully-connected layer. It also does not take edge weights/features into account.\n    - What is the high-level intuition behind Eqs. (5) and (6)? What does the $\\times$ notation mean? Is it elementwise multiplication?\n    - What are the input/output dimensions of the FC layer in Eq. (10)? Are the edge features $e_{ij}$ multiplied elementwise by the neighboring node features?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer JwZm,\n\nWe are grateful for your thorough review. Your comments are highly valued, and we would like to express our heartfelt gratitude. We do our utmost to address the questions you have raised:\n\n**Q1** Comparisons with state-of-the-art protein language models and their structure-aware versions.\n\n**A1** Thank you for your valuable feedback! Factually, CoupleNet is not a pre-training model; it is not fair to compare it with pre-training methods. Thus, we combine ESM-2 (650M) [1] with CoupleNet, named ESM-CoupleNet, using generated ESM embeddings as one part of graph node features. We compare ESM-CoupleNet with pre-training methods on protein function prediction and EC number prediction, including sequence-based methods, ESM-1b [2], ESM-2; sequence-function based methods, ProtST [3]; sequence-structure based methods ESM-S [4], GearNet-ESM [5], SaProt [6]. The comparison results are provided in Table 1 of the one-page rebuttal pdf. From this table, we can see that our proposed model, ESM-CoupleNet, achieves the best results among sequence-based, sequence-structure based, and sequence-function based pre-training methods.\n\n**Q2** Writing quality.\n\n**A2** Thank you for your suggestions! （1）In the following sections, we have definitions about the edge $\\varepsilon_{ij}$, it is important to ensure the preciseness of the graph definition. (2) As shown in Section 3.1, in the definition of Invariance and Equivariance, the transformations $T_g$ and $S_g$ represent the actions of the symmetry group (like rotations and translations). (3) The range of $\\mathcal{F}(\\cdot)$ includes the geometric representations of the input 3D graphs, which are as listed in the paper, like $\\Phi, \\Psi, \\Omega, \\omega, \\theta, \\varphi$. (4) We present Eq. 4 as we update the message-passing mechanism in Eq. 10, and we will rectify our descriptions. (5) Eq. 5 is the definition of the local coordinate system as first presented in [7], which is shown in Figure 7(a) in the appendix. The symbol $\\times$ represents the cross product operation. (6) As shown in Appendix F, the input/output dimensions of the FC layer in Eq.10 are 256. We do not update the edge features $\\boldsymbol{e}_{ij}$ by the neighboring node features, which is different from GearNet [8].\n\n\n**Q3** On line 133, it is mentioned that the positions can be recovered from the geometric representations.\n\n**A3** Thank you for your reviews! The definitions of complete geometric representations are preliminaries proposed by [9]; this is correct as the geometric representations $\\mathcal{F}(G)$ are complete, the structures can be calculated by these features if we use complete geometries.\n\n**Q4** Questions about average pooling.\n\n**A4** Thank you for your valuable reviews! As shown in Section 3.4, Appendix F and Figure 2, we employ complete message passing and sequence pooling layers to obtain the deeply encoded graph-level representations. Every two message-passing layers are followed by an average pooling layer. There are eight message-passing layers in the model. The sequence average pooling functions perform customized average pooling operations on the input tensors based on the calculated indices (dividing the length of the sequence by 2 and flooring the result). It aggregates and summarizes information from the input tensors using scatter operations to produce the output tensors (torch_scatter.scatter_mean). (2) After one average pooling layer, the number of residues reduces by half; for example, only one of the two consecutive graph nodes ($v_i, v_{i+1}$) remains. We reconstruct the graphs by Eq. 9 by expanding the radius threshold $r$ to $2r$ after once pooling, which makes neighbors of center nodes gradually cover more distant and rare nodes, also reducing the computational complexity. \n\n**Q5** On lines 227-228 in Section 3.4...\n\n**A5** Thank you for your comments! In Lines 227-228, we stated that the message passing mechanism only executes on nodes in CoupleNet instead of on nodes and edges alternately used in GearNet. CoupleNet is our proposed model.\n\n**Q5** The lack of sufficient structural data as compared to sequential data for proteins...\n\n**A5** Thank you! For some proteins, only sequential data is available, we can consider using the AlphaFold predicted structures. This model is only developed for modeling protein sequences and structures concurrently. In the era of AlphaFold, it is a trend to modeling protein sequences and structures, using the structural information to enahce the model's performance.\n\nThank you again! In case our answers have justifiably addressed your concerns, we respectfully thank you that support the acceptance of our work. Also, please let us know if you have any further questions. Look forward to further discussions!\n\n[1] Lin, Z., et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.\n\n[2] Rives, A., et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. PNAS, 2021.\n\n[3] Xu, M., et al. Protst: Multi-modality learning of protein sequences and biomedical texts. ICML. 2023.\n\n[4] Zhang, et al. Structure-informed protein language model. arXiv, 2024.\n\n[5]  Zhang, et al. A systematic study of joint representation learning on protein sequences and structures. arXiv, 2023.\n\n[6] Su et al. SaProt: Protein Language Modeling with Structure-aware Vocabulary. ICLR, 2024.\n\n[7] John, et al. Generative models for graph-based protein design. NeurIPS, 2019.\n\n[8] Zhang, et al., Protein Representation Learning by Geometric Structure Pretraining. ICLR. 2023.\n\n[9] Liu, et al. Spherical message passing for 3d molecular graphs. ICLR, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
First and foremost, we would like to express our sincere gratitude for the insightful and constructive feedback provided by the reviewers on our manuscript. 

We are particularly thankful for the Reviewers' recognition of the method of our study; it is important to combine sequences and structures (Reviewer **fLEX**, **JwZm**). We also appreciate their acknowledgment of the experiments we have conducted, which is interesting and better than other methods, making it highly valuable for practical applications in biology and medicine (Reviewer **fLEX, JwZm, ANTD**). Reviewer **fLEX** acknowledged that the proposed method is memory-efficient as it adopts hierarchical pooling, and Reviewer **ANTD** presented that our method is well-introduced and easy to follow. 

We are grateful for the feedback received, particularly regarding the lack of comparisons with the pre-training methods; indeed, our method is a basic network that models protein sequences and structures, which is not a pre-training model, in order to compare with pre-training models fairly, we use ESM-2 [2] embeddings as the graph node features and develop ESM-CoupleNet. We compare ESM-CoupleNet with pre-training methods on protein function prediction and EC number prediction, including sequence-based methods, ESM-1b [2], ESM-2; sequence-function based methods, ProtST [3]; sequence-structure based methods ESM-S [4], GearNet-ESM [5], SaProt [6]. The comparison results are provided in Table 1 of the one-page rebuttal pdf. From this table, we can see that our proposed model, ESM-CoupleNet, achieves the best results among sequence-based, sequence-structure based, and sequence-function based pre-training methods.

In the one-page rebuttal pdf, we also provide the results on protein design, which is to design amino acid sequences that fold into a predefined protein structure, different from protein representation learning of function prediction. Removing the sequence information, we do protein inverse folding by our method on CATH 4.2, the results are shown in Table 2 in the one-page rebuttal pdf. These results illstrates the generalization of our method. 

Once again, we sincerely appreciate the reviewers' feedback and remain committed to continuously improving our research and manuscript based on their valuable insights. Thank you again!

[1] Lin, Z., et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022.

[2] Rives, A., et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. PNAS, 2021.

[3] Xu, M., et al. Protst: Multi-modality learning of protein sequences and biomedical texts. ICML. 2023.

[4] Zhang, et al. Structure-informed protein language model. arXiv, 2024.

[5]  Zhang, et al. A systematic study of joint representation learning on protein sequences and structures. arXiv, 2023.

[6] Su et al. SaProt: Protein Language Modeling with Structure-aware Vocabulary. ICLR, 2024.

Meta-review: The paper proposes CoupleNet, a novel framework to learn protein representations by dynamically coupling protein sequences and structures. It presents an innovative approach to integrating multiple levels and scales of protein features, and demonstrates superior performance compared to state-of-the-art methods, particularly in scenarios with low sequence similarities. 

The reviewers pointed out that the core techniques of dynamic pooling and graph construction were not thoroughly explored in the ablation study, and the paper could benefit from a more concise comparison with related works and a clearer explanation of the advantages of the proposed method. Additionally, the writing quality and clarity of the paper could be improved.    

Despite some areas for improvement, the paper presents a novel and effective method for protein representation learning, which has the potential to impact the field. The authors have also actively addressed the reviewers' concerns during the rebuttal phase, further strengthening the paper's contributions.
Decision: Accept (poster)