Title: Learning Complete Protein Representation by Deep Coupling of Sequence and Structure
Venue: NeurIPS 2023
Accepted: no
Keywords: Bioinformatics

Protein Representation Learning
Abstract: Learning effective representations is crucial for understanding proteins and their biological functions. Recent advancements in language models and graph neural networks have enabled protein models to leverage primary or tertiary structure information to learn representations. However, the lack of practical methods to deeply co-model the relationships between protein sequences and structures has led to suboptimal embeddings. In this work, we propose CoupleNet, a network that couples protein sequence and structure to obtain informative protein representations. CoupleNet incorporates multiple levels of features in proteins, including the residue identities and positions for sequences, as well as geometric representations for tertiary structures. We construct two types of graphs to model the extracted sequential features and structural geometries, achieving completeness on these graphs, respectively, and perform convolution on nodes and edges simultaneously to obtain superior embeddings. Experimental results on a range of tasks, such as protein fold classification and function prediction, demonstrate that our proposed model outperforms the state-of-the-art methods by large margins. 

Abstract

Learning effective representations is crucial for understanding proteins and their biological functions. Recent advancements in language models and graph neural networks have enabled protein models to leverage primary or tertiary structure information to learn representations. However, the lack of practical methods to deeply co-model the relationships between protein sequences and structures has led to suboptimal embeddings. In this work, we propose CoupleNet, a network that couples protein sequence and structure to obtain informative protein representations. CoupleNet incorporates multiple levels of features in proteins, including the residue identities and positions for sequences, as well as geometric representations for tertiary structures. We construct two types of graphs to model the extracted sequential features and structural geometries, achieving completeness on these graphs, respectively, and perform convolution on nodes and edges simultaneously to obtain superior embeddings. Experimental results on a range of tasks, such as protein fold classification and function prediction, demonstrate that our proposed model outperforms the state-of-the-art methods by large margins.

1 Introduction

Proteins are the fundamental building blocks of life and play essential roles in a diversity of applications, from therapeutics to materials. They are composed of 20 different basic amino acids, which are lined by peptide bonds and form a sequence. The one-dimensional (1D) sequence of a protein determines its structure, which in turn determines its biochemical function [40]. Due to recent progress in protein sequencing [34], massive numbers of protein sequences are now available. For example, the UniProt [3] database contains over 200 million protein sequences with annotations, e.g. , gene ontology (GO) terms, similar proteins, family and domains. Notably, the development of large-scale language models (LMs) in natural language processing has substantially benefited protein research owing to similarities between human language and protein sequences [16, 27]. For instance, models like ProtTrans [14] and ESM-series [39, 33] in learning protein representations have proven successful utility of pre-training protein LMs with self-supervision to process protein sequences.

Thanks to the recent significant progress made by AlphaFold2 [30] in three-dimensional (3D) structure prediction, a large number of protein structures from their sequence data are now made available. The

latest release of AlphaFold protein structure database [43] provides broad coverage of UniProt [3]. 30

Recently proposed structure-based protein encoders become to utilize geometric features [25, 24, 31

53], e.g. , ProNet [47] learns representations of proteins with 3D structures at different levels, like the 32

amino acid, backbone or all-atom levels. There also exists a group of methods that build graph neural 33

networks and LMs (LSTMs or attention models) to process sequence and structure [53, 50, 19], for 34

example, GearNet [53] encodes sequential and spatial features by alternating node and edge message 35

passing on protein residue graphs. 36

Amino acids: HSHGLFKKL...KVESRDGT

Primary structure

Tertiary structure

GLU

Positions: 1,2,...,

n

H

S

L

F

H

G

K

K

L

The 1D sequence and 3D structure of a protein provide different types of information, in detail, as 37 shown in Figure 1, compared with the 1D sequential order and amino acids in peptide chains, the 38 tertiary structure provides 3D coordinates of each atom in protein residues, which allow them to 39 perform precise functions. Although a protein's sequence determines its structure, various works 40 have demonstrated the effectiveness of learning from either sequence or structure [33, 25]. However, 41 rich constraints between the sequence and structure of a protein, which may be critical for protein 42 tasks [4], have yet to be fully explored. Most protein sequence-structure modeling methods cannot 43 deeply integrate the information behind sequence and structure for the reason that they tend to fuse 44 representations together, extracted from sequence and structure encoders, respectively, by message 45 passing mechanism [8] or by simple concatenation operations. 46

47

48

49

50

51

52

53

54

In this work, we aim to learn protein representations by deeply coupling the protein sequences and structures. Considering the relative positions of residues in the sequence and the spatial arrangement of atoms in the Euclidean space, the proposed CoupleNet constructs two categories of graphs for them, respectively. The complete representations are obtained at the amino acid and backbone levels on the two graphs, which are used as node and edge features to learn the final graph-level representations. Rather than concatenating sequence and structure representations, we take advantage of graph convolutions, performing node and edge convolutions simultaneously. The contributions of this paper are threefold:

55

56

57

58

59

60

61

62

63

64

We propose a novel two-graph-based approach for representing the sequence and the 3D geometric structure of a protein, which is an effective way to guarantee completeness.

We propose CoupleNet, a model that performs convolutions on nodes and edges of graphs to effectively integrate protein sequence and structure. This can better model the node-edge relationships and utilize the intrinsic associations between sequences and structures.

Practically, the proposed model is verified by obtaining new state-of-the-art experimental results compared with current mainstream protein representation learning methods on a range of tasks, including protein fold classification, enzyme reaction classification, GO term prediction, domain prediction, and enzyme commission number prediction.

2 Related Work

Protein Representation Learning Protein representation learning has become an active and promis65 ing direction in biology, which is essential to various downstream tasks in protein science. Because 66 of the different levels of protein structures, existing methods mainly fall into three categories: protein 67 LMs for sequences, structure models for geometry, and hybrid methods for both of them. As proteins 68 are sequences of amino acids, considering their similarities with human languages, UniRep [1], 69 UDSMProt [42] and SeqVec [23] use LSTM or its variants to learn sequence representations and 70 long-range dependencies. TAPE [37] benchmarks a group of protein models, e.g. , 1D CNN, LSTM, 71 and Transformer by various tasks. Elnaggar et al. [14] have trained six successful transformer variants 72

on billions of amino acid sequences, like ProtBert, and ProtT5. Similarly, ESM-series [39, 38, 33] 73 employs a transformer architecture and a masked language modeling strategy to train robust represen74 tations based on large-scale databases. Besides the protein sequence, as we have stated before, the 75 3D geometric structure is vital to enhance protein representations. Most methods commonly seek to 76 encode the spatial information of protein structures by convolutional neural networks (CNNs) [11], 77 or graph neural networks [19, 2, 29]. For instance, SPROF [7] employs distance maps to predict 78 protein sequence profiles, and IEConv [25] introduces a convolution operator to capture all relevant 79 structural levels of a protein. GVP-GNN [29] designs the geometric vector perceptrons (GVP) for 80 learning both scalar and vector features in an equivariant and invariant manner, Guo et al. [21] 81 adopt SE(3)-invariant features as the model inputs and reconstruct gradients over 3D coordinates to 82 avoid the usage of complicated SE(3)-equivariant models. ProNet [47] learns hierarchical protein 83 representations at multiple tertiary structure levels of granularity. Moreover, CDConv [15] proposes 84 continuous-discrete convolution using irregular and regular approaches to model the geometry and 85 sequence structures. Some protein learning methods model the multi-level of structures at the same 86 time [53, 6, 15], except for the primary structure and the tertiary structure, the second refers to the 87 3D form of local segments of proteins (e.g., α -helix, β -strand), the quaternary is a protein multimer 88 comprising multiple polypeptides, for example, PromtProtein [48] adopts a prompt-guided multi-task 89 learning strategy for different protein structures with specific pre-training tasks. While previous 90 works have attempted to combine protein sequence and structure, we focus on profoundly integrat91 ing them by specifically designing two types of graphs respectively and conducting convolutions 92 simultaneously to learn protein representations. 93

94

95

96

97

98

99

Complete Message Passing Mechanism ComENet [46] proposes rotation angles and spherical coordinates to fulfil the global completeness of 3D information on molecular graphs. By incorporating these designed geometric representations into the message passing scheme [18], the complete representation for a whole 3D graph is eventually yielded [47]. Unlike these methods, we couple sequence and structure via corresponding graphs and different geometric representations to obtain completeness representations.

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

thus f is SE(3)-invariant. 118

3 Method

3.1 Preliminaries

Notations We represent a 3D graph as G = ( V , E , P ) , where V = { v i } i =1 ,...,n and E = { ε ij } i,j =1 ,...,n denote the vertex and edge sets with n nodes in total, respectively, and P = { P i } i =1 ,...,n is the set of position matrices, where P i ∈ R k i × 3 represents the position matrix for node v i . We treat each amino acid as a graph node for a protein, then k i depends on the number of atoms in the i -th amino acid. The node feature matrix is X = [ x i ] i =1 ,...,n , where x i ∈ R d v is the feature vector of node v i . The edge feature matrix is E = [ e ij ] i,j =1 ,...,n , where e ij ∈ R d ε is the feature vector of edge ε ij . d v and d ε denote the dimensions of feature vectors x i and e ij .

Invariance and Equivariance We consider affine transformations that preserve the distance between any two points, i.e. , the isometric group SE(3) in the Euclidean space. This is called the symmetry group, and it turns out that SE(3) is the special Euclidean group that includes 3D translations and the 3D rotation group SO(3) [17, 12]. The matrix form of SE(3) is provided in Appendix A.1.

Given the function f : R m → R m ′ , assuming the given symmetry group G acts on R m and R m ′ , then f is G-equivariant if,

where T g and S g are the transformations. For the SE(3) group, when m ′ = 1 , the output of f is a scalar, we have

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

Complete Geometric Representations A geometric transformation F ( · ) is complete if two 3D graphs G 1 = ( V , E , P 1 ) and G 2 = ( V , E , P 2 ) , there exists T g ∈ SE(3) such that the representations

The operation T g would not change the 3D conformation of a 3D graph [46]. Positions can generate geometric representations, which can also be recovered from them.

Message Passing Paradigm Message passing mechanism is mainly applied in graph convolutional networks (GCNs) [32], which follows an iterative scheme of updating node representations based on the feature aggregation from nearby nodes.

where FC( · ) and BN( · ) mean the linear transformation and batch normalization respectively. N ( v i ) denotes the neighbours of node v i . f ( l ) Agg and f ( l ) Update are aggregation and transformation functions at the l -th layer, which are permutation invariant and equivariant of node representations.

3.2 Sequence-Structure Graph Construction

Specifically, we represent each amino acid as a node, considering the residue types and their positions i = 1 , 2 , · · · , n (See Figure 1) in the sequence, we define the sequential graph primarily on the sequence, if ∥ i -j ∥ < l , the edge ε ij exists, where l is a hyperparameter. Besides the sequential graph, we predefine a radius r , and build the radius graph, and there exists an edge between node v i and v j if ∥ P i, C α -P j, C α ∥ < r , where P i, C α denotes the 3D position of C α in the i -th residue.

Firstly, we design a base approach called CoupleNet aa that only uses the C α positions of the structures. Inspired by Ingraham et al. [28], we construct a local coordinate system (LCS) for each residue, as shown in Figure 2.

where u i = P i, C α -P i -1 , C α ∥ P i, C α -P i -1 , C α ∥ , b i = u i -u i +1 ∥ u i -u i +1 ∥ , n i = u i × u i +1 ∥ u i × u i +1 ∥ . Then we can get the geometric representations at the amino acid level of a protein 3D graph,

where · is the matrix multiplication, this implementation is SE(3)-equivariant and obtains complete representations at the amino acid level; as if we have Q i , the LCS Q j can be easily obtained by F ( G ) ij,aa .

For a node v i , the node features x i,aa in the baseline approach is the concatenation of the one-hot embeddings of the amino acid types and the physicochemical properties of each residue, namely, a steric parameter, hydrophobicity, volume, polarizability, isoelectric point, helix probability and sheet probability [51, 22], which provide quantitative insights into the biochemical nature of each amino acid. And F ( G ) ij,aa is set as edge features for CoupleNet aa .

Secondly, we consider all backbone atoms C α , C , N , O in CoupleNet. In detail, the peptide bond exhibits partial double-bond character due to resonance [20], indicating that the three non-hydrogen atoms comprising the bond (the carbonyl oxygen, carbonyl carbon, and amide nitrogen) are coplanar,

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

peptide plane

1.33Å

1.45Å

1.52Å

1.23Å

backbone

Ri1

N+1

亚i

02

123°

Ci

122°

2i

当

Cgit1

Ci-1)121°

N:

118

120°116

Hi2

as shown in Figure 3. There is some rotation about the connection. The N i -C αi and C αi -C i bonds, are the two bonds in the basic repeating unit of the polypeptide backbone. These single bonds allow unrestricted rotation until sterically restricted by side chains [35, 45]. Since the coordinates of C α can be obtained as we have the complete representations at the amino acid level, the coordinates of other backbone atoms based on these rigid bond lengths and angles are able to be determined with the remaining degree of the backbone torsion angles Φ i , Ψ i , Ω i . The omega torsion angle around the C -N peptide bond is typically restricted to nearly 180 ◦ (trans) but can approach 0 ◦ (cis) in rare instances. Other than the bond lengths and angles presented in Figure 3, all the H bond lengths measure approximately 1 Å.

For the sequential graph, we compute the sine and cosine values of Φ i , Ψ i , Ω i for each amino acid i , and use them as another part of nodes features for node v i .

where ∥ denotes concatenation. There is no isolated node for the designed graph, which means the backbone atoms can be determined one by one along the polypeptide chain based on the positions of C α and these three backbone dihedral angles. Therefore, the existing presentations [ F ( G ) ij,aa ] i,j =1 ,...,n and [ x i ] i =1 ,...,n are complete at the backbone level for the sequential graph.

For the radius graph, we want to get the positions of backbone atoms in any two amino acids i and j . Inspired by trRosetta [52], the relative rotation and distance are computed including the distance ( d ij, C β ), three dihedral angles ( ω ij , θ ij , θ ji ) and two planar angles ( φ ij , φ ji ), as shown in Figure 4, where d ij, C β = d ji, C β , ω ij = ω ji , but θ and φ values depend on the order of residues. These interresidue geometries define the relative locations of the backbone atoms of two residues in all their details [52], because the torsion angles of N i -C αi and C αi -C i do not influence their positions. Therefore, these six geometries are complete for amino acids at the backbone level for the radius graph. The graph edges contain the relative spatial information between any two neighboring amino acids e ij = F ( G ) ij,aa ∥F ( G ) ij,bb , where

residue

residue

Cj

CBi

dij,cβ

Nj

CBi

Pij

0ij

Coi

Ni

Ci

The designed node and edge features, x i and e ij , for the sequential and radius graphs, provide a 189 new perspective to represent protein sequences and structures. Such integration can bring better 190 performance for the following graph-based learning tasks. 191

192

193

194

195

196

197

198

199

200

201

202

212

3.4 Model Architecture

Building upon the sequence-structure graph convolution, we build the CoupleNet, as shown in 213 Figure 5. The inputs to the graph are the calculated sequential and structural representations ( X,E 214 Following the existing protein graph models [15, 25, 47], our CoupleNet employs graph pooling 215 layers to obtain deeply encoded, graph-level representations. After pooling, due to the decrease 216 in nodes, we increase the predefined radius r to include more neighbors. The message passing 217 mechanism only executes on nodes for the consideration of reducing model complexity. Another 218 reason is that representations of sequences and structures have already been coupled by equation 219 Eq. 4. A detailed description of the model architecture is provided in Appendix A.2. 220

Residue

Geometric

representatons

Edge

Node

Node

Pool-

ing

Edge

Node

Node

Pool-

ing

Node

...

...

...

Message passing

Message passing

FC

Soft-

Output

3.3 Secqunce-Structure Graph Convolution

Inspired by the message passing paradigm and continuous-discrete convolution [15], sequences and structures are encoded successfully together by convolutions. To deeply couple sequences and structures of proteins and encode them jointly, we employ convolution to embed them simultaneously, exploring their relationships to generate comprehensive and effective embeddings. Different from previous works, we innovatively construct two categories of graphs for sequence and structure and design various sequential and structural representations to achieve completeness on them at the amino acid and backbone levels. We then convolve node and edge features with the help of the message passing mechanism.

In order to implement convolution on nodes and edges simultaneously between sequence and structure, we set ε ij to exist if the following conditions are satisfied

The existing node and edge feature matrices ( X,E ) are complete representations of a protein 3D 203 graph to reconstruct its backbone atom positions. Compared with the equation Eq. 4, the proposed 204 CoupleNet first apply a FC( · ) layer and a BN( · ) layer to the node features to obtain the initial 205 encoded representation. Then the f ( l ) Agg is applied to gather neighboring features of nodes and 206 edges by convolution, where σ ( · ) is the activation function. We use the dropout and add a residual 207 connection from the previous layer as f ( l ) Update . For the consideration that the spatial arrangement and 208 tight positioning of specific amino acids, which may be spaced widely apart on the linear polypeptide, 209 are necessary for proteins to operate as intended [10], l is set to be a relatively large number, see 210 Appendix A.2 for details. 211

).

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

4 Experiments

4.1 Datasets and Settings

The models are trained with the Adam optimizer [31] using the PyTorch and PyTorch Geometric libraries. Detailed descriptions of the datasets and experimental settings are provided in Appendix A.3. Following the tasks in IEconv [25], GearNet [53] and CDConv [15], here, we evaluate the CoupleNet on four protein tasks: protein fold classification, enzyme reaction classification, GO term prediction and enzyme commission (EC) number prediction.

Fold Classification Protein fold is to predict the fold class label given a protein, which is crucial for understanding how protein structure and protein evolution interact [26]. In total, this dataset contains 16 , 712 proteins with 1 , 195 fold classes. There are three test sets available, Fold: Training excludes proteins from the same superfamily. Superfamily: Training does not include proteins from the same family. Family: Proteins from the same family are included in the training.

Enzyme Reaction Classification Reaction categorization aims to predict a protein's class of enzyme-catalyzed reactions, according to all four levels of the EC number [49, 36]. Following the setting in [25], this dataset has 37 , 248 proteins from 384 four-level EC numbers [5].

GO Term Prediction The goal of GO term prediction is to foretell whether a protein is related to a certain GO term. Following [19], these proteins are organized into three ontologies: molecular function (MF), biological process (BP), and cellular component (CC), which are hierarchically connected, functional classes. MF describes activities that occur at the molecular level, BP represents the larger processes, and CC describes the parts of a cell or its extracellular environment [3].

EC Number Prediction This task seeks to predict the 538 EC numbers from the third level and fourth levels of different proteins [19], which describe their catalysis of biochemical reactions.

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

4.2 Baselines

We compare our proposed method with existing protein representation learning methods, which are classified into three categories based on their inputs, which could be a sequence (amino acid sequence), 3D structure or both sequence and structure. 1) Sequence-based encoders, including CNN [41], ResNet [37], LSTM [37] and Transformer [37]. 2) Structure-based methods (GCN [32], GAT [44], 3DCNN_MQA [11], IEConv (atom level) [25]). 3) Sequence-structure based models, e.g. , GVP [29], ProNet [47], GearNet [53], CDConv [15], etc . GearNet-IEConv and GearNetEdge-IEConv [53] add the IEConv layer based on GearNet, which is found important in fold classification.

4.3 Resluts of Fold and Reaction Classification.

Table 1 provides the comparisons on the fold and enzyme reaction classification. The results are reported in terms of accuracy ( % ) for these two tasks. From this table, we can see that the proposed model CoupleNet achieves the best performance across all four test sets on the fold and enzyme reaction classification compared with recent state-of-the-art methods. Especially on the Fold and SuperFamily test sets, CoupleNet improves the results by about 4% , showing that CoupleNet is proficient at learning the mapping between protein sequences, structures and functions. Moreover, CDConv [15] ranks second among these methods, both CDConv and our method are implemented by sequence-structure convolution. This phenomenon illustrates that deeply coupling sequences and structures of proteins is conducive to learning better protein embeddings. And our proposed CoupleNet model utilizes complete geometric representations and the specially designed message passing mechanism, achieving new state-of-the-art results.

4.4 Results of GO Term and EC Prediction

We follow the split method in [19, 53] to guarantee that the test set only comprises PDB chains with 264 sequence identity no higher than 95% to the training set for GO term and EC number prediction. 265 Table 2 compares different protein modeling methods on GO term prediction and EC number 266 prediction. The results are reported in terms of F max , which considers both precision and recall for 267 evaluation, the equation of F max is provided in Appendix A.4. The proposed model, CoupleNet 268 yields the highest F max across these four test sets of two tasks, outperforming other state-of-the-art 269 models. This indicates CoupleNet can effectively predict the functions, locations, and enzymatic 270 activities of proteins. These results once again illustrate the importance of deeply coupled sequences 271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

and structures. The improvements of CoupleNet over the suboptimal CDConv [15] model indicate the advanced modeling power of CoupleNet.

We employ different cutoff splits following [19, 15], which means that the proteins in the test set are divided into groups that have, respectively, 30 % , 40 % , 50 % , 70 % , and 95 % similarity to the training set for GO term and EC number prediction, as shown in Figure 6 and Appendix A.5. From Figure 6, we can see that our proposed model CoupleNet achieves the highest F max scores across all cutoffs, especially when the cutoffs are at 30 % to 50 % . There is a larger margin compared with GearNet, GearNet-Edge [53] and CDConv [15]. This demonstrates that CoupleNet, which utilizes complete geometric representations, is more robust, especially when there is a low similarity between the training and test sets.

4.5 Ablation Study

Table 3 presents an ablation study of the proposed CoupleNet model on the four protein tasks. We examined the impact of removing the backbone torsion angles (w/o Φ , Ψ , Ω ) and removing the interresidue geometric structure representations (w/o d, ω, θ, φ ). The former is designed for the sequential graph, and the latter is for the radius graph to achieve completeness at the protein backbone level. However, we combine the two types of graphs together to enhance the relationships between sequence and structure. From Table 3, we can also find that these complete geometries provide complementary information to amino acid position features, with one of their removals leading to minor performance drops for the reason that they both provide complete geometries from different perspectives. Removing Φ , Ψ , Ω causes larger performance degradation compared with removing d, ω, θ, φ . Such comparisons indicate that the backbone dihedral angles may have more effects on learning protein representations in these experimental settings. Compared with CoupleNet aa , CoupleNet achieves significant improvements on the four tasks, demonstrating the importance of complete structural representations at the backbone level in learning protein embeddings.

5 Conclusions and Limitations

In this work, we propose CoupleNet, a novel protein representation learning method that deeply fuses protein sequences and multi-level structures by conducting convolution on graph nodes and edges simultaneously. We design the sequential graph and the radius graph, achieving completeness on them at different protein structure levels. Our approach achieves new state-of-the-art results on the protein tasks, which demonstrates the superiority our the proposed method. A limitation is that the detailed inter-relationships between sequence and structures remain to be explored and uncovered. We leave such research for future work.

While our model can enable advanced protein analyses and provide effective representations, there may exist broader impacts and harmful activities. The representations could potentially be misused, e.g. , for designing harmful molecules or proteins.

EC

1.000

0.866

0.900

0.800

0.700

0.600

0.557

0.625

0.634

0.723

0.570

0.646

0.659

0.752

0.615

0.694

0.702

0.790

0.693

0.7570.768

0.834

0.730

0.810

0.820

0.500

0.400

0.300

0.200

0.100

0.000

30%

40%

50%

70%

%S6

GearNet

GearNet-Edge

CDConv

CouNet

315

References

Ethan C. Alley et al. 'Unified rational protein engineering with sequence-based deep represen316 tation learning'. In: Nature Methods (2019). 317

Federico Baldassarre et al. 'GraphQA: protein model quality assessment using graph convolu318 tional networks.' In: Bioinformatics (2020). 319

Alex Bateman. 'UniProt: A worldwide hub of protein knowledge'. In: Nucleic Acids Research 320 (2019). 321

Tristan Bepler and Bonnie Berger. 'Learning the protein language: Evolution, structure, and 322 function'. In: Cell systems 12.6 (2021), pp. 654-669. 323

Helen M Berman et al. 'The protein data bank'. In: Nucleic acids research 28.1 (2000), 324 pp. 235-242. 325

Can Chen et al. 'Structure-aware protein self-supervised learning'. In: Bioinformatics 39.4 326 (2023), btad189. 327

Sheng Chen et al. 'To Improve Protein Sequence Profile Prediction through Image Captioning 328 on Pairwise Residue Distance Map'. In: Journal of Chemical Information and Modeling 329 (2020). 330

Yihong Chen et al. 'Refactor gnns: Revisiting factorisation-based models from a message331 passing perspective'. In: Advances in Neural Information Processing Systems 35 (2022), 332 pp. 16138-16150. 333

334

335

336

G Marius Clore and Angela M Gronenborn. 'NMR structure determination of proteins and protein complexes larger than 20 kDa'. In: Current opinion in chemical biology 2.5 (1998), pp. 564-570.

337

338

339

340

Srinivasan Damodaran. 'Amino acids, peptides and proteins'. In: Fennema's food chemistry 4 (2008), pp. 425-439.

Georgy Derevyanko et al. 'Deep convolutional networks for quality assessment of protein folds'. In: Bioinformatics 34.23 (2018), pp. 4046-4053.

Weitao Du et al. 'SE (3) Equivariant Graph Neural Networks with Complete Local Frames'. 341 In: International Conference on Machine Learning . PMLR. 2022, pp. 5583-5608. 342

343

344

345

Arun Kumar Dubey and Vanita Jain. 'Comparative study of convolution neural network's relu and leaky-relu activation functions'. In: Applications of Computing, Automation and Wireless Systems in Electrical Engineering: Proceedings of MARC 2018 . Springer. 2019, pp. 873-880.

346

347

348

349

350

Ahmed Elnaggar et al. 'ProtTrans: Towards Cracking the Language of Lifes Code Through Self-Supervised Deep Learning and High Performance Computing'. In: IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).

Hehe Fan et al. 'Continuous-Discrete Convolution for Geometry-Sequence Modeling in Proteins'. In: The Eleventh International Conference on Learning Representations . 2023.

Noelia Ferruz and Birte Höcker. 'Controllable protein design with language models'. In: 351 Nature Machine Intelligence (2022), pp. 1-12. 352

Fabian Fuchs et al. 'Se (3)-transformers: 3d roto-translation equivariant attention networks'. 353 In: Advances in Neural Information Processing Systems 33 (2020), pp. 1970-1981. 354

Justin Gilmer et al. 'Neural message passing for quantum chemistry'. In: International 355 conference on machine learning . PMLR. 2017, pp. 1263-1272. 356

Vladimir Gligorijevi´ c et al. 'Structure-based protein function prediction using graph convolu357 tional networks'. In: Nature communications 12.1 (2021), p. 3168. 358

ER HARD GROSS and JOHANNES MEIENHOFER. 'The Peptide Bond'. In: Major Methods 359 of Peptide Bond Formation: The Peptides Analysis, Synthesis, Biology, Vol. 1 1 (2014), p. 1. 360

Yuzhi Guo et al. 'Self-supervised pre-training for protein embeddings using tertiary structures'. 361 In: Proceedings of the AAAI Conference on Artificial Intelligence . Vol. 36. 6. 2022, pp. 6801362 6809. 363

Jack Hanson et al. 'Improving prediction of protein secondary structure, backbone angles, 364 solvent accessibility and contact numbers by using predicted contact maps and an ensemble 365 of recurrent and residual convolutional neural networks'. In: Bioinformatics 35.14 (2019), 366 pp. 2403-2410. 367

Michael Heinzinger et al. 'Modeling the language of life - Deep Learning Protein Sequences'. 368 In: bioRxiv (2019). 369

Pedro Hermosilla and Timo Ropinski. 'Contrastive representation learning for 3d protein 370 structures'. In: arXiv preprint arXiv:2205.15675 (2022). 371

Pedro Hermosilla et al. 'Intrinsic-Extrinsic Convolution and Pooling for Learning on 3D 372 Protein Structures'. In: International Conference on Learning Representations (2021). 373

Jie Hou, Badri Adhikari, and Jianlin Cheng. 'DeepSF: deep convolutional neural network for 374 mapping protein sequences to folds'. In: Bioinformatics 34.8 (2018), pp. 1295-1303. 375

Bozhen Hu et al. 'Protein Language Models and Structure Prediction: Connection and Pro376 gression'. In: arXiv preprint arXiv:2211.16742 (2022). 377

John Ingraham et al. 'Generative models for graph-based protein design'. In: Advances in 378 neural information processing systems 32 (2019). 379

Bowen Jing et al. 'Learning from Protein Structure with Geometric Vector Perceptrons'. In: 380 Learning (2020). 381

John Jumper et al. 'Highly accurate protein structure prediction with AlphaFold'. In: Nature 382 596.7873 (2021), pp. 583-589. 383

Diederik P Kingma and Jimmy Ba. 'Adam: A method for stochastic optimization'. In: arXiv 384 preprint arXiv:1412.6980 (2014). 385

Thomas N Kipf and Max Welling. 'Semi-supervised classification with graph convolutional 386 networks'. In: arXiv preprint arXiv:1609.02907 (2016). 387

Zeming Lin et al. 'Language models of protein sequences at the scale of evolution enable 388 accurate structure prediction'. In: BioRxiv (2022). 389

Bin Ma. 'Novor: real-time peptide de novo sequencing software.' In: Journal of the American 390 Society for Mass Spectrometry (2015). 391

David L Nelson, Albert L Lehninger, and Michael M Cox. Lehninger principles of biochemistry . 392 Macmillan, 2008. 393

Marina V Omelchenko et al. 'Non-homologous isofunctional enzymes: a systematic analysis 394 of alternative solutions in enzyme evolution'. In: Biology direct 5 (2010), pp. 1-20. 395

Roshan Rao et al. 'Evaluating protein transfer learning with TAPE'. In: Advances in neural 396 information processing systems 32 (2019). 397

Roshan M Rao et al. 'MSA transformer'. In: International Conference on Machine Learning . 398 PMLR. 2021, pp. 8844-8856. 399

Alexander Rives et al. 'Biological Structure and Function Emerge from Scaling Unsupervised 400 Learning to 250 Million Protein Sequences'. In: Proceedings of the National Academy of 401 Sciences of the United States of America (2019). 402

Andrew W Senior et al. 'Improved protein structure prediction using potentials from deep 403 learning'. In: Nature 577.7792 (2020), pp. 706-710. 404

405

406

Amir Shanehsazzadeh, David Belanger, and David Dohan. 'Is transfer learning necessary for protein landscape prediction?' In: arXiv preprint arXiv:2011.03443 (2020).

407

408

409

Nils Strodthoff et al. 'UDSMProt: universal deep sequence models for protein classification'. In: Bioinformatics 36.8 (Jan. 2020), pp. 2401-2409. ISSN: 1367-4803. DOI: 10.1093/ bioinformatics/btaa003 .

Mihaly Varadi et al. 'AlphaFold Protein Structure Database: massively expanding the structural 410 coverage of protein-sequence space with high-accuracy models'. In: Nucleic acids research 411 50.D1 (2022), pp. D439-D444. 412

Petar Velickovic et al. 'Graph attention networks'. In: stat 1050.20 (2017), pp. 10-48550. 413

K Peter C Vollhardt and Neil E Schore. Organic chemistry: structure and function . Macmillan, 414 2003. 415

Limei Wang et al. 'ComENet: Towards Complete and Efficient Message Passing for 3D 416 Molecular Graphs'. In: arXiv preprint arXiv:2206.08515 (2022). 417

Limei Wang et al. 'Learning Hierarchical Protein Representations via Complete 3D Graph 418 Networks'. In: The Eleventh International Conference on Learning Representations . 2023. 419

Zeyuan Wang et al. 'Multi-level Protein Structure Pre-training via Prompt Learning'. In: The 420 Eleventh International Conference on Learning Representations . 421

Edwin C Webb et al. Enzyme nomenclature 1992. Recommendations of the Nomenclature Com422 mittee of the International Union of Biochemistry and Molecular Biology on the Nomenclature 423 and Classification of Enzymes. Ed. 6. Academic Press, 1992. 424

Fang Wu, Dragomir Radev, and Jinbo Xu. 'When Geometric Deep Learning Meets Pretrained 425 Protein Language Models'. In: bioRxiv (2023), pp. 2023-01. 426

Gang Xu, Qinghua Wang, and Jianpeng Ma. 'OPUS-Rota4: a gradient-based protein side-chain 427 modeling framework assisted by deep learning-based predictors'. In: Briefings in Bioinformat428 ics 23.1 (2022), bbab529. 429

Jianyi Yang et al. 'Improved protein structure prediction using predicted interresidue ori430 entations'. In: Proceedings of the National Academy of Sciences 117.3 (2020), pp. 1496431 1503. 432

Zuobai Zhang et al. 'Protein representation learning by geometric structure pretraining'. In: 433 International Conference on Learning Representations . 2023. 434

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenge of integrating protein sequence and structure information to improve protein representations. The authors propose CoupleNet, a network that utilizes graph convolutions to model the relationships between protein sequences and structures. The approach involves constructing two types of graphs to model sequential features and structural geometries and performing convolutions on nodes and edges simultaneously. The proposed approach outperforms state-of-the-art methods on various protein-related tasks.\n\nSTRENGTHS:\n1) Comprehensive Protein Features: CoupleNet incorporates multiple levels of features in proteins, including residue identities, positions, and geometric representations. By considering both primary (sequence) and tertiary (structure) information, the model captures different aspects of proteins, leading to more informative and rich representations. The authors have demonstrated good work in engineering the features of their model. By carefully considering both primary (sequence) and tertiary (structure) information, the model has succeeded in capturing diverse aspects of proteins, leading to more informative and robust representations. \n\n2) Experimental Results: The paper presents experimental results on a range of protein-related tasks, including protein fold classification, function prediction, and domain prediction. The results demonstrate that CoupleNet outperforms state-of-the-art methods by large margins. The comprehensive evaluation and superior performance validate the effectiveness of the proposed model.\n\nWEAKNESSES:\n1) Limited Novelty: While the proposed approach in CoupleNet is interesting and effective, it does not offer a significant departure from existing methods. Similar approaches, such as GearNet, have previously explored this concept by integrating the radius and sequence information as different edge types within a single graph. Therefore, the idea of constructing two separate graphs in CoupleNet, while slightly different in implementation, does not present a significant departure from existing methods.\n\n2) A weakness of the paper is the lack of a detailed explanation regarding why the proposed model, CoupleNet, performs better than the state-of-the-art methods. While the experimental results demonstrate superior performance, the authors do not provide a thorough analysis or insights into the specific aspects of the model architecture or design choices that contribute to its improved performance. Without a clear explanation of the underlying factors that make the model more effective, it becomes challenging for readers to fully understand and interpret the advantages of CoupleNet over existing approaches.\n\n3) Lack of Code Release: One notable weakness of the paper is the absence of code release. The lack of code availability hinders the reproducibility and transparency of the research.\n\n\n[1] Zhang, Zuobai, et al. \"Protein representation learning by geometric structure pretraining.\n\nQUESTIONS:\n1) Can you explain why the use of two separate graphs in CoupleNet is preferable to integrating sequence and structure information as different edge types within a single graph, as demonstrated in previous works like GearNet?\n\n2) Could you clarify the computational complexity of the proposed model? How does the performance of CoupleNet scale with increasing data volume, particularly as protein databases continue to grow in size?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a representation learning framework using GNN for protein datasets. A key contribution is combining sequence and structure information using the proposed GNN. The learned representations achieve better performance on downstream tasks of protein fold classification and function prediction.\n\n\nThe sequence+structure methods have been previously investigated in several works, as included in the baseline. I don’t see the motivation behind using a “sequence-structure graph” that differentiates the novelty of CoupleNet. I suggest updating the abstract and introduction by mentioning the limitations of existing “ sequence+structure” approaches and then specifying how CoupleNet addresses that. Could discuss the drawbacks of existing “feature fusion” methods.\n\nI think the primary area of this paper should be applications. The construction of a joint sequence-structure graph is specific to modelling proteins, and the rest of the GNN operations are standard. The structure graph is based on Ingraham et al., and the sequence graph is based on trRosetta et al.. The message-passing scheme is based on ComENet. I don’t see any methodological novelty.\nMoreover, empirical results are reported without proper discussion on application to protein problems. The empirical results of baselines are reported based on the article [15]. Without error bars or cross-validation (commonly done for fold prediction and function prediction), it is hard to make any conclusions. At this point, I lean towards rejection.\n\n########\nPost Rebuttal\n########\n\nI have read the author's rebuttal. Overall, the authors did a great job in responding to the comments, and I have accordingly adjusted my score, recommending accepting the paper.\n\nSTRENGTHS:\nLearning protein representations that combine sequence and structure information is relevant to the community. Using GNNs is a good demonstration of DNNs' applications to biological problems like protein function prediction.\n\nA new joint sequence-structure 3D graph that can model complex interdependencies for learning representations that achieve good improvement over existing approaches.\n\nWEAKNESSES:\nThe introduction doesn't provide sufficient motivation behind the proposed method. It is unclear what the paper means by \"...methods cannot deeply integrate the information….\" What is \"deep integration\"? The proposed method itself uses a type of message-passing mechanism, so what is the issue with message-passing cited in the article [8]? \n\nThe paper should first identify the issues in existing \"feature combining\" methods. Then motivate how the proposed CoupleNet addresses that. For instance, could take an example of a specific protein where existing methods perform poorly and then discuss the complex interdependence between structure and sequence for that particular example that needs appropriate modelling for improving the performance of downstream tasks such as fold prediction. \n\nI would suggest adding a paragraph that discusses the importance of sequence-level information and another on structure-level information for tasks such as fold classification or function prediction. Discuss the importance of local/non-local residue contact information. Then establish how encoding for such a multitude of information can be helpful for downstream applications. I would suggest taking a look at non-homologous proteins. Could discuss the availability of large-scale high-resolution structure data and how deep GNNs can leverage that.\n\n\nLine 94-99: No clear explanation of the benefits of complete message passing. Why is it essential to consider \"global\" completeness? What is the downside of \"local\" completeness? Why not cite SphereNet [a] here? Which notion of \"completeness\" does this paper build on?\nIt is implicit from the equation, but that simply is restating from the ComENet paper. The paper should be well explained, avoiding room for any such ambiguity.\n\nThis paper needs to explain better the need for global \"completeness\", which can be done by discussing the importance of conformers like structure. Discuss the equivalence of 3D graphs under SE(3) transformations for protein structures and why that's important to be considered for representation learning.\n\nIn my understanding, the classes in the fold classification task tend to be unbalanced. I suggest authors report per class accuracy.\n\n[a] Liu, Yi, et al. \"Spherical message passing for 3d graph networks.\" arXiv preprint arXiv:2102.05013 (2021).\n\nThere is no comparison of time-space complexity.\n\nQUESTIONS:\nLine 60: I do not understand “new state-of-the-art experimental results….”. Do the predicted functions for specific proteins are verified by experiments in the lab? I don’t see any such evidence. Perhaps authors should consider being careful in their wording.\n\nFor classification problems, I suggest reporting error bars (+/- std) or k-fold cross-validation. Please update accordingly in Tables 1 and 2.\n\nFrom Table 3, it appears backbone torsion angles and inter-residue geometric representations do not lead to any noticeable improvement. Is there an example of a protein where this information is helpful? If so, perhaps discuss that protein structure to understand this complementary information better.\n\n\nFigure 1 right side is too small. Perhaps use PyMol and report an enhanced Figure that’s readable.\n\nLine 6: “..to suboptimal embeddings”. It doesn’t seem informative here. Suboptimal can only be in the context of the problem they are used for. Perhaps say something like, “...led to embeddings that achieve low performance on tasks such as protein function prediction”.\nLikewise, “...superior embeddings” in LIne 13. Also,  the wording “...deeply co-model…” doesn’t seem appropriate. Could say something like “...model complex interdependencies between protein sequences…”. \n\n\nLine 15: Perhaps add a sentence that shows the improvement percentage. \n\nLine 119: “if two 2D” -> “if for two 2D”\n\nPlease be precise in technical descriptions.\n\nLine 158: “There is some rotation about the connection”. It is unclear what is “connection” here and how one specifies “some rotation”. \n\nIn Equation (9) after Line 2, please specify which norm ||.|| refers to. In Equation 10, what is “W”? Is it a matrix or a scalar? If W is a matrix, why is it not in bold as done for matrices in the earlier part of the paper? Likewise, what is “\\sigma” in the Equation? For Dropout, what is the dropout probability? \n\nLine 192: Correct the typo “Secquence”-> “Sequence”\n\nLine 251: “Resluts” -> “Results”\n\nThe x and y-axis on the Figures should be labelled. Figure 6, including the ones in an appendix, does not label the axes.\n\nCaptions of Figures should be self-explanatory. Likewise, Tables should have a sentence to convey the findings or interpretation of scores. I suggest updating the caption of Figures 4, 5, 6, 7 and 8—especially 5 as it doesn’t convey the working of CoupleNet—Similarly, Table 3.\n\nLIMITATIONS:\nAlthough a sentence is added on limitation, ”. A limitation is that the detailed inter-relationships between sequence and structures remain to be explored and uncovered”. This sentence is just vague and provides no meaningful information. The paper consistently discusses “deeply co-model sequence and structures together”. So what could not be “deeply modelled” and needs more exploration? \n\nThe sequence and structure of information have been combined in several existing works. The main issue is that the paper doesn’t establish clear motivation for combining the two using a combined graph. Moreover, the need to consider “completeness.”", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposed CoupleNet to co-model the protein sequences and structures. CoupleNet separately builds a sequence-based graph and a radius graph for message passing. It achieved state-of-the-art performance on several datasets compared with recent baselines.\n\nSTRENGTHS:\n1. The proposed CoupleNet utilized much biological knowledge to incorporate useful geometric information like backbone torsion angles and inter-residual torsion angles. This information may help the model better capture the structural information.\n\n2. The experiments demonstrated the SOTA performance over a wide range of baselines. Ablation studies demonstrated the effectiveness of \"coupling\" the sequence and structural information.\n\n3. The protein figures clearly and concisely defined the geometric features used in the paper.\n\nWEAKNESSES:\n1. The construction of two separate graphs (one for the sequence and one for the structure) is not very innovative. In fact, GearNet (already cited by the authors) already used this formulation and the description of the two graph construction in this paper is very similar to GearNet.\n\n2. Though the backbone structure can be completely determined by the descriptor, the residue information is lost in the proposed model. Failure to capture this information may affect the performance on downstream tasks.\n\n3. There is some confusion regarding the graph construction that can be clarified. See the following questions.\n\nQUESTIONS:\n1. Please elaborate on the major differences from GearNet in terms of graph construction. It seems that apart from the feature generation process, the graphs built will be identical.\n\n2. In Eq.10, when updating $\\\\mathbf{u}\\_i$, the authors used $W\\\\mathbf{e}\\_{ij}\\\\mathbf{h}\\_j$. There will be a size mismatch if matrix multiplication is assumed. Please clarify the operation here. Also, the authors mentioned $f_{Agg},f_{Update}$ in the text, but none appears in Eq.10.\n\n3. The use of enlarging cut-off distances needs justification. Theoretically, stacking GNN layers alone can also capture the information. The author may do some additional ablation studies on different $r$ and/or $l$.\n\n4. How will the loss of residue information affect the final prediction results?\n\n5. Section 4.3 title is mistyped.\n\nLIMITATIONS:\nLimitations and potential negative societal impacts were properly addressed in the manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a network called CoupleNet that combines protein sequence and structure information to generate informative protein representations. The network utilizes multiple levels of features, including residue identities and positions for sequences, as well as geometric representations for tertiary structures. It constructs two types of graphs to model sequential features and structural geometries, and performs convolution on nodes and edges simultaneously to obtain superior embeddings. Experimental results demonstrate that the proposed model outperforms state-of-the-art methods on various protein-related tasks. The paper highlights the significance of complete structural representations in learning protein embeddings and suggests further exploration of the inter-relationships between sequence and structures.\n\nSTRENGTHS:\n* The paper is clearly written and easy to follow. Related work is thoroughly discussed, situating the context and contributions of the proposed method relative to prior studies.\n\n* The experimental results show substantial gains over previous state-of-the-art baselines on benchmark datasets for protein structure prediction. The proposed method achieves these results with a relatively small training dataset, whereas some baselines utilize much larger resources such as the AlphaFold database. This indicates the model may be more data-efficient and able to elicit more from limited information.\n\nWEAKNESSES:\n* The relationships between protein sequence and structure, especially as they relate to function, are not deeply explored or discussed. For tasks like protein function prediction that aim to determine the utility or effects of a protein, understanding the connection between its sequence, structure, and biological role is critical.\n\n* The proposed method achieves promising results for these functional tasks through an end-to-end modeling approach, but additional analysis interpreting what the model has learned about sequence-structure-function relationships would strengthen scientific validity. Has the model captured complex, nuanced relationships, or is performance driven more by statistical associations in the training data? Discussion of these relationships and how the model may be representing them would address concerns about the depth of knowledge actually obtained.\n\n* For full reproducibility and scientific validity, additional details on the experimental setup, hyperparameter selections, and sensitivity analyses are needed. e.g. The variance or confidence intervals of reported results should be provided to determine their reliability and sensitivity to stochastic effects. Point estimates alone do not indicate the variability across trials or uncertainty in conclusions.The effects of different random seeds on performance should be analyzed to confirm results do not depend highly on a single seed selection. Sensitivity to initialization is an important consideration, especially for complex neural networks. Hyperparameter choices require further explanation and analysis of the effects of varying key values such as layer sizes, attention heads, learning rates, loss trade-offs, etc. The initial values selected may bias conclusions if performance is highly sensitive to these hyperparameters. Exploring this sensitivity would reinforce the results do not depend entirely on the specific choices made.\n\nQUESTIONS:\nQ1. Are additional datasets available to evaluate the generalizability of the proposed method to other structure prediction or molecular modeling tasks? Applying the model to other datasets or tasks would reinforce that the capabilities demonstrated generalize beyond the current benchmarks used\n\nQ2. The ablation study shows removing sequence features results in little performance loss for fold classification and function prediction. Does this indicate sequence information is not critical for these specific tasks? Or is there another explanation for why eliminating these features does not significantly impact results? Additional discussion is needed to interpret this finding.\n\nQ3. Are the knowledge and modeling capabilities required for strong performance on the structure prediction and classification tasks fundamentally different from those required to design new protein or molecular structures from scratch? Success on one set of tasks may not imply abilities will directly translate to generation, even if a shared model architecture is used. Discussion of this concern may be useful for readers.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}