Continuous-Discrete Convolution for Geometry-Sequence Modeling in Proteins
Authors: Hehe Fan
Zhangyang Wang
Yi Yang
Mohan Kankanhalli
Venue: ICLR 2023 poster (2023)
Keywords: Protein representation learning
3D geometry modeling
1D sequence modeling
continuous convolution
discrete convolution.

Abstract: The structure of proteins involves 3D geometry of amino acid coordinates and 1D sequence of peptide chains. The 3D structure exhibits irregularity because amino acids are distributed unevenly in Euclidean space and their coordinates are continuous variables. In contrast, the 1D structure is regular because amino acids are arranged uniformly in the chains and their sequential positions (orders) are discrete variables. Moreover, geometric coordinates and sequential orders are in two types of spaces and their units of length are incompatible. These inconsistencies make it challenging to capture the 3D and 1D structures while avoiding the impact of sequence and geometry modeling on each other. This paper proposes a Continuous-Discrete Convolution (CDConv) that uses irregular and regular approaches to model the geometry and sequence structures, respectively. Specifically, CDConv employs independent learnable weights for different regular sequential displacements but directly encodes geometric displacements due to their irregularity. In this way, CDConv significantly improves protein modeling by reducing the impact of geometric irregularity on sequence modeling. Extensive experiments on a range of tasks, including protein fold classification, enzyme reaction classification, gene ontology term prediction and enzyme commission number prediction, demonstrate the effectiveness of the proposed CDConv. 

Published as a conference paper at ICLR 2023
CONTINUOUS-DISCRET
CONVOLUTION HOMETRY-SEQUENCE MODELING IN PROTEINS
FOR
Hehe Fan 1.3. Zhangyang Wang', Yi Yang' & Mohan Kankanhalli
'Zhejiang University
"The University of Texas Austin
National University of Singapore
ABSTRACT
The structure of proteins involves 3D geometry of amino acid coordinates and ID sequence of peptide chains. The 3D structure exhibits irregularity because amino acids are distributed unevenly in Euclidean space and their coordinates are continuous variables. In contrast, the 1D structure is regular because amino acids are arranged uniformly in the chains and their sequential positions (orders) are discrete variables. Moreover, geometric coordinates and sequential orders are two types of spaces and their units of length are incompatible. These inconsistencies make it challenging to capture the 3D and ID structures while avoiding the impact of sequence and geometry modeling on each other. This paper proposes Continuous-Disette regular approaches to model the geometry and sequence structures, respectively. Specifically, CDConv employs independent learnable weights for different regular sequential displacements but directly encodes geometric displacements due to their irregularity. In this way, CDConv significantly improves protein modeling by reducing the impact of geometric irregularity on sequence modeling. Extensive experiments on range of tasks, including protein fold classification, enzyme reaction classification, gene ontology term prediction and enzyme commission number prediction, demonstrate the effectiveness the proposed CDConv.
Convolution (CDConv) that uses irregular and
INTRODUCTION
Proteins are large biomolecules and are essential for life. Understanding their function is significant for life sciences. However, usually requires enormous experimental efforts (Wüthrich, 2001; Jaskolski al., 2014; Bai et al., 2015; Thompson al., 2020) to find out their function. Recently, with the development of deep learning, emerging computational and data-driven approaches are particularly useful for efficient protein understanding (Derevyanko et al., 2018; Ingraham al., 2019; Strokach et al., 2020; Cao et al., 2021; Jing et al., 2021; Jumper et al., 2021; Shanchsazzadeh et al., 2020), including protein design, structure classification, model quality assessment, function prediction, etc. Because the function of proteins is based on their structure, accurately modeling protein structures can facilitate a mechanistic understanding of their function to life.
Proteins made up of different amino acids. There are 20 types of amino acids (residues) commonly found in plants and animals and a typical protein is made up of 300 or more amino acids. Because these amino acids are linked by peptide bonds and form chain (shown in Fig. 1). proteins exhibit a 1D sequence structure. Moreover, because amino acids are arranged uniformly the chains and their orders discrete, the sequence structure is regular. In this case, 1D Convolutional Neural Network (CNN) (Kulmanov et al., 2018; Hou et al., 2018: Kulmanov & Hoehndorf, 2021) and Long Short-Term Memory (LSTM) (Bepler & Berger, 2019; Rao al., 2019; Alley et al., 2019; Strodthoff et al., 2020) are widely used to model the regular ID sequence structure of proteins.
In addition to the ID sequential order in peptide chains, each amino acid is with 3D coordinate that specifies its spatial position in a protein. As shown in Fig. 1, those 3D coordinates describe a geometry structure, which is also crucial for protein recognition. As mentioned by Alexander et al. (2009),
Part of this work was done when Hehe Fan a Research Fellow National University of Singapore.

---

Published as a conference paper at ICLR 2023
[3+1JD position
feature
regular structure
discrete 1D order
continuous
residue
3D coordinate
type
(18.09, 5.83, 18.78)
P
25, 9.24, 18.87)
(-18.37, 8,50, -6.16)
irregular 3D structure
protein (3+1)D structure
protein data structure
Illustration of the geometry sequence structure of a protein. The dot color indicates amino acid (residue) types. 1) Amino acids are linked by peptide bonds and form a chain, which exhibits regular ID sequence structure because they are arranged uniformly and their sequential orders are discrete. In addition, amino acids are with 3D coordinates that determine geometry structure, which exhibits irregularity because they distributed unevenly in Euclidean space and their coordinates are continuous variables.
Figure 1:
proteins with similar peptide chains may fold into very different 3D geometry structures. Conversely, proteins with similar 3D geometry structures may have entirely different amino acid chains (Agrawal Kishan, 2001). Therefore, is necessary to consider both the and 3D structures in protein modeling. However, different from the sequence structure, the geometry structure is irregular because amino acids are distributed unevenly in Euclidean space and their coordinates are continuous variables. This increases the challenge for neural networks understand proteins.
To model the ID and 3D structures in proteins, Gligorijević et al. (2021) employed an LSTM and a Graph Convolutional Network (GCN) for sequence and geometry modeling. respectively. Because the two structures are processed separately, it may not properly understand proteins' local geometry- sequence structure. In contrast, few unified networks try model the types of structures together (Baldassarre et al., 2021; Jing et al., 2021; Zhang et al., 2022: Hermosilla & Ropinski, 2022). However, those methods process geometric and sequential displacements together or model the sequence structure in similar way to geometry modeling, thus neglecting the regularity of the 1D structure. Moreover, because the length units of the sequential and geometric spaces in proteins are not compatible, treating their distances similarly may mislead deep neural networks.
In this paper, we first propose and formulate new class of convolution operation, named Continuous- Discrete Convolution (CDConv), make the most of the dual discrete and continuous nature of the data to avoid the impact regular and irregular modeling on each other. Specifically, CDConv employs independent learnable weights reflect different regular and discrete displacements but directly encodes continuous displacements due to their irregularity. Then, we implement a (3+1)D I)D CDConv and use this operation construct hierarchical 13+1)D CNN for geometry-sequenee mod- eling in proteins. We apply our network to protein fold classification, enzyme reaction classification, gene ontology term prediction and enzyme commission number prediction. Experimental results demonstrate the effectiveness of the proposed method. The contributions of this paper are fourfold:
We propose new class of convolution, i.e., CDConv, which unifies continuous and discrete convolutions and makes the most of the regularity and irregularity in data, respectively.
We implement (3+1)D CDConv for geometry-sequent modeling in proteins. Based on the convolution, construct deep neural networks for protein representation learning.
We conduct extensive experiments on four tasks and the proposed method surpasses the previous methods by large margins, resulting in the new state-of-te -hthe-of-the-art accuracy.
We find that amino acids in central regions may more important than those in surface areas. This may need to be verified via biochemical experiments in the future.
RELATED WORK
Protein Representation Learning. Protein representation learning attracts increasing attention the fields of protein modeling and structural bioinformatics and is critical in biology. Because proteins are sequences amino acids, ID CNN, LSTM and Transformer are employed for sequence- based protein representation learning (Amidi et al., 2018; Kulmanov et al., 2018; Hou et al., 2018; Rao et al.,
2

---

Published as a conference paper at ICLR 2023
2019; Bepler & Berger, 2019; Alley et al., 2019; Strodthoff al., 2020; Shanehsazzadeh et al., 2020; Kulmanov & Hochndorf, 2021). Besides the sequence structure, the 3D geometry structure is also used to enhance protein representations, based on amino acid or atom 3D coordinates (Derev et al., 2018; Ingraham et al., 2019; Gligorijević et al., 2021; Baldassarre et al., 2021; Jing et al., 2021; Wang et al., 2021; Hermosilla et al., 2021; Zhang et al., 2022; Chen et al., 2022; Hermosilla & Ropinski, 2022) or protein surfaces (Gainza et al., 2020; Sverrisson et al., 2021; Dallago et al., 2021). Our method is based on ID sequential orders and 3D geometric coordinates of amino acids. Different from those existing methods, we propose to employ discrete and continuous manners to model regular orders and irregular coordinates, respectively.
Discrete and Continuous Convolution. Convolution is one of the most significant operations in the deep learning field and has made impressive achievements in many areas, including but not limited to computer vision and natural language processing. Convolution can be defined as functions on a discrete or continuous space. For example, images can be seen in a 2D discrete space of pixels and thus number of 2D discrete CNNs are used for image processing (Krizhevsky et al., 2012; Simonyan & Zisserman, 2015; Szegedy et al., 2016; He et al., 2016). Similarly, languages can be seen in ID discrete space of words or characters and therefore can be modeled by ID discrete CNNs (Kim, 2014; Yin et al., 2017; Bai et al., 2018). Continuous convolutions are usually used process irregular data (Schütt al., 2017; Simonovsky & Komodakis, 2017; Wang al., 2018; Romero et al., 2022), especially to 3D point clouds (Thomas et al., 2018; Wu et al., 2019; Hu et al., 2020; Fuchs al., 2020; Fan al., 2021: Shi al., 2021), or sometimes used to improve flexibility for discrete convolutions (Shocher et al., 2020). When implementing an instance of the proposed (3+1)D CDConv for protein modeling, the 3D processing part is inspired by those point cloud methods.
3 PROPOSED CONTINUOUS-DISCRETT CONVOLUTION
In this section, we first briefly review the discrete and continuous convolutions for regular and irregular structure modeling, respectively. Then, we present the proposed Dontinuous-Discrete Convolution (CDConv) in detail. Third, we introduce the (3+1)D CDConv for geouetry-sequence modeling in proteins. Fourth, we present an implementation for the proposed (3+1)D I)D CDConv. Last, we incorporate our (3+1)D CDConv into a hierarchical architecture for protein representation learning.
3.1 PRELIMINARY: CONVOLUTION
The power of convolutional neural networks (CNNs) comes from local structure modeling by convolutions and global representation learning via hierarchical or pyramid architectures. Convolution is defined as the integral of the product of two functions after one is reversed and shifted. The convolution of f and g is written as follows,
Convolution: (f) = / g(T)f(t+
TEQ
where Q is neighborhood in a space. In deep learning, f(t) usually is the feature at position t, i.e., f() fi e xxx1 where is the number of input feature channels, and g(T) E C'C'' is usually implemented as parametric kernel function, where C' is the number of output feature channels.
Traditionally, convolutions are mainly used for image processing (Krizhevsky et al., 2012) and natural language processing (Kim, 2014). Because positions of pixels and words or characters are regular, discrete convolutions are used to model their structures as follows,
Discrete Convolution: f.
Σ ((Os) fi-s. AED
(2)
where matrix multiplication, and D is a regular and discrete neighborhood and can be seen as the kernel size of convolution. The key to discrete convolution is to employ independent group of learnable weights for every displacement A. As shown in Fig. 2(a), for a 2D discrete convolution with kernel size 3 x 3, there are nine independent groups of weights for the convolution: (0,-1.-1): 111)) 1.1. (o. O (0,0)) 0 f...), (1,-1)) 00,,)) 0 11,1)) }. For discrete convolution, usually an identity function.
3

---

Published as a conference paper at ICLR 2023
0,
0 (-1,-1)
(-111)
3x3
0n
(a)
(L--))
0 (1.1)
0_1
(b)
(d)
Figure 2: Illustration of discrete and continuous convolutions and examples of continuous-discrete spaces. (a) 2D discrete convolution. Pixels distributed evenly in the 2D space and their coordinates are discrete. The convolution capturing the structure of x 3 local area with nine independent groups of learnable weights. (b) 2D continuous convolution. For continuous space, even in a small area, there can be countless types of displacements. is parameterized by the same weights A but directly takes the displacement continuous-disered space. Points are distributed regularly on the 2D xy plane but irregularly along the ID axis. Four groups of weights continuous-diserett space. Points are located irregularly on the 2D plane but regularly along the ID axis. Three groups of weights used reflect the three discrete ID positions.
this case, the convolution
input. (c) (1+2)D
used to reflect the 2 discrete 2D positions.
(d) (2+1)D
Continuous convolutions mainly used handle irregular data, especially point clouds. For irregular and continuous space, even in an extremely small neighborhood C, there can be countless displacements. Therefore, it is impossible assign an independent group of learnable weights for every displacement ó. In this case, the kernel function is parameterized by the same weights 0 for displacements but takes the displacement an input,
Continuous Convolution: f:
Σ SEC
g(ō; 0) f148-
(3)
We illustrate a continuous convolution in Fig. 2(b).
3.2 CONTINUOUS-DISCHRR DISCRETE CONVOLUTION
Besides exclusively regular or irregular, some data may simultaneously exhibit both regularity and irregularity. In this case, the neighborhood can be hybrid, i.e., S C x D, and each displacement in such a space consists discrete part A and a continuous part 8. To model regular and irregular structures, propose continuous-discred convolution (CDConv),
Continuous - Discrete Convolution: f
g(ō; 0 fi-A+6-
(4)
(6.A)ECXD
The crucial design of CDConv is employ independent weights {0s}aED for different regular displacements A but directly encode continuous displacements due to their irregularity. In this way, CDConv facilitates regular-irregular modeling and reduces the impact of the irregularity of continuous displacements on the regular and discrete structure modeling.
The proposed CDConv is a new class of convolution and we can derive different variants based on different combinations of continuous and discrete spaces. In this paper, we define (i+j)D as hybrid space with an iD continuous structure and a jD discrete structure. The (1+2)D and (2+1)D continuous-discret spaces are illustrated in Fig. 2(c) and Fig. 2(d), respectively.
3.3 (3+1)D CDCONV FOR PROTEIN MODELING
In protein, the amino acids are linked by peptide bonds and form a chain. Let (fi.... fx) denote the features of the amino acids in protein, where f. Cxx1 denotes the feature of the 1-th amino
4

---

Published as a conference paper at ICLR 2023
t-1 ++1
COMi () fre
((0:0)) f
fr-2
4+4
++3
g(Prss p:;0.) -aa
(b)
Figure 3: Illustration of the proposed (3+1)D CDConv for protein modeling. (a) The convolution is capturing the structure
neighborhood centred at the t-the amino acid. (b) Based on the ID sequential kernel size I 11 and the 3D geometric radius r, the (1 - and (4 + 4) amino acids are selected as neighbors and others are exc excl acluddd because their sequential or geometric distances are too far away. (c) Based on the geometric displacement (p++a pt) and the sequential displacement A, the kernel function ((t++ Pr: 0aa) generates the kernel weights for each selected neighbor. The neighbor features are transformed by the generated kernel weights with matrix multiplication, which are then aggregated as the new feature of the t-th amino acid. In this way, the local area structure of the t-th amino acid captured.
24-2)-th 2)hhh (1 101-1pp 1)-th, 1-th, (1 1)-th
acid and N is the number of amino acids. In addition to the 1D order in the sequence, each amino acid has 3D coordinate that specifies its spatial position, i.e., Pt R3x1, The values of 1D sequential orders are regular and discrete while those of 3D coordinates are irregular and continuous non-integers. In this case, amino acids can be seen in a (3+1)D continuous-discrete space and can employ the proposed convolution to capture the structure of geometry- neighborhood. resulting in the (3+1)D CDConv.
(3+1)D CDConv: f.
Σ
g(p++ - P:; P:)
(5)
I -
where the neighborhood is defined with a 1D sequential kernel size I and 3D geometric radius r. We illustrate how (3+1)D CDConv captures the geometry-sequent structure of a local area in Fig. 3.
Discussion
Comparison with existing methods. The proposed (3+1)D CDConv is substantially different from existing methods that usually process geometric and sequential displacement together or in similar irregular manners. For example, GraphQA (Baldassarre et al., 2021) concatenates the 3D distance with a radial basis function and the ID distance with the one-hot representation. The edge in GVP (Jing et al., 2021) embeds the unit vector in the 3D direction, the 3D distance with radial basis function and the sinusoidal encoding of the ID displacement. IECony (Hermosilla & Ropinski, 2022) encodes normalized ID displacements. GearNet (Zhang al., 2022) directly concatenates 3D and 1D distances. Those methods can be summarised as ((5, A]) or (+))). A)), where [+ demotes concatenation and 7 and 7) are encoding functions. In contrast, (3+1)D CDConv does not encode or embed ID displacements but employs independent weights for different discrete displacements. In this way, CDConv makes the most of the regularity of sequential orders. Moreover, because the units of length of the sequential and geometric spaces are not compatible, capturing the two structures in different ways improves the proposed CDConv for protein understanding.
Comparison with two-stream 3D- DD-CNNNIDDCCN architectures. To make the most of the regularity of sequential orders and the irregularity of geometric coordinates, we can first employ an independent 3D continuous CNN branch and an independent 1D discrete CNN branch to extract the global geometric and sequential features, respectively, and then fuse the two features as the final protein representation (illustrated in Appendix D). However, because the two types of structures in proteins are modeled separately, the local (3+1)D structure is not properly captured. In contrast, CNNs based on our (3+1)D CDConv simultaneously model the local sequenry structure and thus extract more discriminative local features than ID-CNN.. As shown in Sec. 4.5.2, our (3+1)D CDConv-based CNN significantly outperforms the corresponding 3D-CNN+ 1D-N+I approach.
5

---

Published as a conference paper at ICLR 2023
3.4 AN IMPLEMENTATION FOR (3+1)D CDCONV
There can be many ways to implement the kernel function y for the proposed (3+1)D CDConv. In this paper, inspired by PointConv (Wu al., 2019) and PSTNet (Fan et al., 2021), implement y as follows,
g(P+A p:: 02) 0 (pi+a -P))-
However, the above implementation not rotationally invariant. Inspired by Hermosilla & Ropinski (2022), we replace the 3D displacement encoding with relative spatial encoding (Ingraham et al., 2019),
g(P++A - Pr, O,, 014400s) = 02 (lpp++ -
P++A Pt
Of O(+D);
(7)
PL+A
Pell
where O: (b,, nu. b, x n²) and u,
P.- b PIP:-P
1 -u nhe
I l
Xtu411
and
x7 OA RC Finally, (3+1)D CDConv is implemented as follows,
fi
Σ
g(p++ - Pt, O, OctAiOs) fit
(8)
P PRHA-PPElSS rr -1/31
The matrix multiplication result is then reshaped from RC'xC to RC
3.5 CONTINOUS-DISCRE CONVOLUTIONAL NEURAL NETWORK
Based on the proposed CDConv, hierarchical deep neural networks can be built for global protein representation learning. To this end, we stack multiple CDConv layers and downsample amino acids as the number of layers increases.
To construct hierarchical networks, late layers should have larger receptive fields than early layers. For the 1D sequential kernel size I, we fix it because convolutions at late layers will capture relatively longer sequential regions due amino acid downsampling. However, for the geometry modeling, need to progressively entarge the radius of each layer so that late layers have larger receptive fields.
Following Hermosilla et al. (2021) and Hermosilla & Ropinski (2022), build ResNet (He et al., 2016) with multiple (h) CDConv layers. After every two CDConv layers, a sequence sum pooling with downsampling rate 2 is performed to halve the protein resolution and build pyramid architecture. For the i-th CDConv layer, the radius is set to [ + 1]r, and the output dimension is set to 1 Co. where and are the initial radius and the initial number of feature channels, respectively. In this way, the geometric receptive field and feature dimension progressively increase.
EXPERIMENTS
4.1 EVALUATION TASKS AND DATASETS
Following Hermosilla et al. (2021); Hermosilla & Ropinski (2022); Zhang et al. (2022), we evaluate the proposed method on four tasks; protein fold classification, enzyme reaction classification, gene ontology (GO) term prediction and enzyme commission (EC) number prediction. Protein fold classification includes three evaluation scenarios: fold, superfamily and family. GO term prediction includes three sub-tasks: biological process (BP), molecular function (MF) and cellular component (CC) ontology term prediction. More details about these tasks and datasets are shown Appendix
Protein fold and enzyme reaction classification are single-label classification tasks. Mean accuracy is used as the evaluation metric. GO term and EC number prediction are multi-label classification tasks. The Fmax accuracy is used as the evaluation metric (details are described in Appendix C).
4.2 TRAINING SETUP
We implement our method with the PyTorch-Gend Geometric (PyG) library. The number CDConv layers h is set to 8. The sequential kernel size is set to 11 for fold classification, 25 for reaction

---

Published as a conference paper at ICLR 2023
Table 1: Accuracy (%) of protein fold classification and enzyme catalytic reaction classification.
"Results are from (Hermosilla et al., 2021). 'Results are from (Zhang et al., 2022).
Fold Classification
Enzyme
Input
Method
Fold
Superfamily
Family
Reaction
CNN (Shanehsazzadeh et al., 2020)!
11.3
13.4
53.4
51.7
ResNet (Rao et al., 2019)*
10.1
7.21
23.5
24.1
ID
LSTM (Rao al.. 2019)
6,41
4.33
18.1
11.0
Transformer (Rao et al., 2019)
9.22
8.81
40.4
26.6
GCN (Kipf Welling, 2017)*
16.8
21.3
82.8
67.3
GAT (Velickovic et al., 2018)
12.4
16.5
72.7
55.6
3D CNN (Derevyanko al., 2018)*
31.6
45.4
92.5
72.2
3D+Topo
|EConv (atom level) (Hermosilla al., 2021)
45.0
69.7
98.9
87.2
GraphQA (Baldassarre et al., 2021)*
23.7
32.5
84.4
60.8
GVP (Jing et al., 2021)*
16.0
22.5
83.8
65.5
IEConv (residue level) (Hermosilla & Ropinski, 2022)
47.6
70.2
99.2
87.2
(3+1)D
GearNet (Zhang al., 2022)
28.4
42.6
95.3
79.4
GearNet-IEConv (Zhang et al., 2022)
42.3
64.1
99.1
83.7
GearNet-Edge (Zhang et al., 2022)
44.0
66.7
99.1
86.6
GearNeet-Edge-IBCCom (Zhang et al., 2022)
48.3
70.3
99.5
85.3
CDConv (ours)
56.7
77.7
99.6
88.5
classification and 15 for GO term and EC number prediction. The initial radius T, is set to 4. More details of implementation and training setup are provided in Appendix E
4.3 COMPARISON WITH STATE-OF-THE. OF-THE-ART
compare our method with existing ID-only, 3D-only and (3+1)D methods. Results are shown in Table I and Table 2. The IEConv method includes an atom-level variant (Hermosilla et al., 2021) and an amino-acid- acid-level variant (Hermosilla & Ropinski, 2022). Because the atom-level IEConv is based on the 3D coordinates of atoms and the topology structure of bonds between atoms, we denote the input of atom-level IEConv as "3D+Topoo".
Note that none of the existing methods achieved state-of-the-art accuracy all of the four tasks. In contrast, our method significantly outperforms all the existing methods. For example, on the superfamily fold classification, SoarNee--Eggelln
CDConv outperforms the previous state-of-the-a method, i.e.,
by 6.9%. On the BP ontology term prediction, our CDConv outperforms the previous best method, i.e., the residue-level IEConv, by 3.2%. This demonstrates the effectiveness of the proposed CDConv that employs discrete and continuous approaches for regular sequence and irregular geometry modeling, respectively.
4.4 How DOES CONTINUOUS-DDEEE DISSCREE CNN CAPTURE PROTEIN STRUCTURE?
To investigate the behavior of our CNN, visualize the outputs of CDConvs. We treat the norm || of the output features as the activation strength of amino acids and visualize the activation strength in Fig. 4. Visualizations of proteins of different sizes for all CDConv layers illustrated in Appendix T. Recall that, to build a pyramid architecture, the CNN downsamples amino acids after every two layers. The early layers (e.g., CDConv and CDCony tend to focus on the central region of the protein. Then, the activation spreads to the entire protein so that the late layers (e.g., CDConv and CDConv 8) can capture the whole structure. This indicates that the central structure may provide the most informative clues in early recognition and may be more important than other areas.
To verify the explanation for the above phenomenon, we mask the same number (20%) of amino acids in the central and surface areas, respectively. As shown in Fig. 5(a), masking the central area leads to more accuracy drops, indicating that amino acids in central regions may be more important than those in surface areas. More details and results are shown in Appendix J.
hhttps://o thub. htt.:ccccccceaeceeeeeeeeeee Cont inousssD3 wwwpte-ceeeeeeee uution

---

Published as a conference paper at ICLR 2023
Table 2: F max of gene ontology term prediction and enzyme commission number prediction. 'Results from (Wang al., 2021). Resullt from (Zhang al., 2022).
Gene Ontology
Enzyme
Input
Method
BP
MF
CC
Commission
CNN (Shanehsazzadeh et al., 2020)
0.244
0.354
0.287
0.545
ResNet (Rao et al., 2019)
0,280
0.405
0.304
0.605
ID
LSTM (Rao et al., 2019)
0.225
0.321
0.283
0.425
Transformer (Rao et al., 2019)²
0.264
0.211
0.405
0.238
GCN (Kipf & Welling, 2017)!
0.252
0.195
0.329
0.320
3D
GAT (Velickovic al., 2018)
0.284
0.317'
0.385
0.368!
3D CNN (Derevyanko et al., 2018)!
0.240
0.147
0.305
0.077
GraphQA (Baldassarre et al., 2021)!
0.308
0.329
0.413
0.509
GVP (Jing al., 2021)
0.326"
0.426
0.420
0.4891
IFConv (residue level) (Hermosilla & Ropinski, 2022)
0.421
0.624
0.431
GearNet (Zhang et al., 2022)
0.356
0.503
0.414
0.730
(3+1)D
GearNet-IECCon (Zhang et al., 2022)
0.381
0.563
0.422
0,800
GearNet-Edee Edge (Zhang et al., 2022)
0,403
0.580
0.450
0.810
(Zhang et al., 2022)
0.400
0.581
0.430
0,810
CDConv (ours)
0.453
0.654
0.479
0.820
input
CDConv
CDConv
CDConv
CDConv 8
Figure 4: Visualization of how the proposed network captures protein structure. The color of input dots indicates amino acid types and that of CDConv outputs denotes the strength of the activation. The early layers tend to focus the central region and the late layers try to capture the whole structure. This may indicate that the central region is more important than other areas to proteins.
4.5
ABLATION STUDY
4.5.1 IMPACT OF CONTINUOUS GEOMETRY AND DISCRETE SEQUENCE MODELING
In this paper, we propose use continuous and discrete approaches to model the geometry and sequence structures, respectively. To verify the effectiveness, we conduct tross-experiments. To model the regular sequence structure in continuous manner, directly encode the sequential displacement A. To capture the irregular geometry structure discrete manner, 3D coordinates are voxelized with size 0.5. Results are shown in Table 3.
First, when replacing discrete modeling with directly encoding on the sequence structure capture, the accuracy drops. This verifies our motivation that regular structure should be modeled in regular manner. Second, when using the discrete method to modeling the geometry structure, the accuracy drops significantly. This is because quantization errors are inevitable during voxelization and voxelized coordinates cannot precisely describe the geometry structure.
4.5.2 COMPARISON WITH TWO-STREAM 3D-CNN+ II-NN
To avoid the impact of irregular and regular modeling on each other, one can employ independent continuous 3D-CNN and an independent discrete ID-CNN to build a two-stream network, resulting in 3D-CNN+ID-CNN We compare CDConv-based CNN with the corresponding 3D-CNN+ID. CNN. The 3D-CNN is implemented by PointConv with relative spatial encoding. Each of the 3D-CNN stream and the ID-CNN stream consist of the same number of layers CNN's. As shown in Table 3, our CDConv-based CNN surpasses the 3D-CNNIID-CNY by large margin. Moreover, compared the 3D geometry-only continuous CNN, the two-stream method
CDConv-based
8

---

Published as a conference paper at ICLR 2023
Table 3: Influence of continuous and discrete modeling 3D and 1D structure capture, respectively, and comparison with two-stream 3D-CNN+ID-CNA "RI" indicates rotational invariance.
3D Geometry
Sequence
Fold Classification
Enzyme
Gene Ontology
Enzyme
RI
Structure
Structure
Fold Superfamily Family
Reaction
BP
MF
CC
Commission
Discrete Continuous
X
31.5
42.6
85.9
72.6
0.336 0.526
0.355
0.664
V
41.9
65.5
99.0
86.5
0.426
0.605
0.435
0.737
X
Discrete
13.9
20.2
86.6
71.1
0.371
0.548
0.436
0.561
X
Continuous
9.1
12.8
71.1
64.0
0.348
0.488
0.417
0,482
Discrete Continuous
Discrete
x
38.9
48.2
90.6
77.3
0.378
0.560
0.410
0.701
Continuous
51.7
74.2
99.2
87.1
0.432
0.618
0.446
0.782
Discrete
Continuous
x
34.2
43.8
87.0
74.1
0.340
0.544
0.369
0.681
-wo-Stream 3D CNN+ID-CNN /
43.5
65.4
98.4
86.1
0,412
0.601
0,414
0.758
Continuous
Discrete
56.7
77.7
99.6
88.5
0.453
0.654
0.479
0.820
1.0
Unmasked
Fold
1.0
Fold
Surface-masked
0.885
Super
Super
Central-masked
0.9
Family
0.9
Family
0.777
Reaction
Reaction
0.726
0.8
GO-BP
0.8
GO-BP
0.7
GO-CC
GO-CC
0.6
0.567
0.542
0.7
EC
0.6
0.5
0.6
0.5
0.488
0.333
0.5
0,260
Fold
Superfamily
Reaction
(a) impact of surface/central mask
(b) geometric radius
(c) sequential kernel size I
Figure 5: (a) To investigate which area is more important, we mask the same number of amino acids in central and surface regions and evaluate the impact on protein recognition. (b) Impact of initial geometric radius. (c) Impact of sequential kernel size on protein modeling.
does not significantly improve the accuracy or even leads to worse accuracy. These indicate that the 3D 3D-CNN+ID CNN strategy is not effective to capture the local (3+1)D structure of proteins and the global representations of the two structures may not benefit each other.
4.5.3 IMPACT OF GEOMETRIC RADIUS AND SEQUENTIAL KERNEL SIZE
The geometric radius and sequential kernel size control the range of the spatial and chain structure to be modeled, respectively. As shown in Fig. 5(b) and Fig. 5(c), using too small radius and kernel size cannot capture sufficient structure information and using too large radius and kernel size will decrease the discriminativeness of spatial local structure for modeling. thus leading to inferior accuracy. The impact of the number of channels and CDConv layers are shown in Appendix F and in Appendix G.
5 CONCLUSION
We first propose new class of convolutions, i.e., CDConv, which uses continuous and discrete approaches to model the irregular and regular structures in data, respectively. Then, we implement (3+1)D CDConv to model the sequence and geometry structures in proteins. We apply CDCony to range of protein modeling tasks and achieve the new state-of-the -fe--o--tthtrttt accuracy. Moreover, by investigating the behavior of CDCony, we find that amino acids in central regions may be more important than those in surface areas to proteins, which can be further verified via experimental efforts. As new class convolutions, the applications of CDConv are not limited to protein modeling and can be other scenarios where data exhibits both regularity and irregularity. The proposed (3+1)D CDConv can also be used to recognize non-coding RNAs and may be potentially improved with different implementations. However, geometric radius and sequential kernel size to capture local structure, which is somewhat rigid. This can be alleviated in future studies by integrating self-attention to adaptively capture the local structure.
convolution, CDConv requires predefined and fixed

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This research is supported by the Agency for Science, Technology and Research (A STAR) under its AME Programmatic Funding Scheme (#A (#A18A200046).
REFERENCES
Vishal Agrawal and Radha KV Kishan. Functional evolution two subtly different (similar) folds. BMC structural biology, 1(1):1-6, 2001.
Patrick A Alexander, Yanan He, Yihong Chen, John Orban, and Philip N Bryan. A minimal sequence code for switching protein structure and function. Proceedings of the National Academy of Sciences, (06650):21149-21155 2009.
Ethan C Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M Church. Unified rational protein engineering with sequence-based deep representation learning, Nature methods, 16(12)1315-1322 2019.
Afshine Amidi, Shervine Amidi, Dimitrios Vlachakis, Vasileios Megalooikonomou, Nikos Paragios, and Evangelia I Zacharaki. Enzynet: enzyme classification using 3d convolutional neural networks on spatial representation. PeerJ, 6xe4750, 2018.
Minkyung Baek and David Baker. Deep learning and protein structure modeling. Nature methods, 19 (1):13-14, 2022.
Minkyung Back, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov. Gyu Ric Lee, Jue Wang. Qian Cong, Lisa N Kinch, Dustin Schaeffer, al. Accurate prediction protein structures and interactions using three-track neural network. Science, 373(6557):871-88) 1-776, 2021.
Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv, 1803.01271, 2018.
Xiao-Ches Bai, Greg McMullan, and Sjors HW Scheres. How cryo-em is revolutionizing structural biology. Trends in biochemical sciences, 40(1):49-57, 2015.
Federico Baldassarre, David Menéndez Hurtado, Ame Elofsson, and Hossein Azizpour. Graphqa: protein model quality assessment using graph convolutional networks. Bioinform. -663)360--366 2021.
Tristan Bepler and Bonnie Berger. Learning protein sequence embeddings using information from structure. In ICLR, 2019.
Yue Cao, Payel Das, Vijil Chenthamarakshan, Pin- Yu Chen, Igor Melnyk, and Yang Shen. Fold2seq: A joint (o) embedding-basse generative model for protein design. In ICML, 2021.
Can Chen, Jingbo Zhou, Fan Wang, Xue Liu, and Dejing Dou. Structure-aware protein self-supervised learning. arXiv, 2204.04213, 2022.
Christian Dallago, Jody Mou, Kadina E. Johnston, Bruce J. Wittmann, Nicholas Bhattacharya, Samuel Goldman, Ali Madani, and Kevin Yang. FLIP: benchmark tasks fitness landscape inference for proteins. In NeurIPS Track on Datasets and Benchmarks, 2021.
Georgy Derevyanko, Sergei Grudinin, Yoshua Bengio, and Guillaume Lamoureux. Deep convolu- tional networks for quality assessment of protein folds. B 34(23)4046-4053, 2018.
Carl Doersch and Andrew Zisserman. Multi-task self-supervised visual learning. In ICCV, pp. 2070-2078 2017.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Wang Yu, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. Prottrans: Towards cracking the language of lifes code through self sulf-super deep learning and high performance computing. IEEE Transactions on Pattern Analysis and Machine Intelligence, pp. 1-1, 2021. doi: 10.1 109/TPAM..2011 200933331111...
10

---

Published as a conference paper at ICLR 2023
Hehe Fan, Xin Yu, Yuhang Ding, Yi Yang, and Mohan S. Kankanhalli. Pstnet: Point spatio-temporal convolution on point cloud sequences. ICLR, 2021.
Hehe Fan, Ping Liu, Mingliang Xu, and Yi Yang. Unsupervised visual representation learning via
dual-level progressive similar instance selection. IEEE Trans. Cybern., 52(9):8851-888) 8861, 2022. doi: (0.9//CB..2011100000
Fabian Fuchs, Daniel E. Worrall, Volker Fischer, and Max Welling. transfrrrosssrmer:: 3d roto-
translation equivariant attention networks. In NeurIPS, 2020.
Pablo Gainza, Freyr Sverrisson, Frederico Monti, Emanuele Rodola, D Boscaini, MM Bronstein, and BE Correia. Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nature Methods, 17(2):184-192, 2020.
Vladimir Gligorijević, P Douglas Renfrew, Tomasz Kosciolek, Julia Koehler Leman, Daniel Beren- berg, Tommi Vatanen, Chris Chandler, Bryn C Taylor, Ian M Fisk, Hera Vlamakis, et al. Structure- based protein function prediction using graph convolutional networks. Nature communications, 12 (1):1-14, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Pedro Hermosilla and Timo Ropinski. Contrastive representation learning for 3d protein structures. arXiv, 2205. 2022.
Pedro Hermosilla, Mareo Schäfer, Matej Lang. Gloria Fackelmann, Pere-Pau Vázquez. Barbora Kozlíková, Michael Krone, Tobias Ritschel, and Timo Ropinski. Intrinsic-extrinsic convolution and pooling for learning on 3d protein structures. ICLR, 2021.
Jie Hou, Badri Adhikari, and Jianlin Cheng. Deepsf: Deep convolutional neural network for mapping protein sequences to folds. In ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics, 2018.
Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic segmentation of large-scale point clouds. In CVPR, pp. 11104-11114, 2020.
Ian R Humphreys, Jimin Pei, Minkyung Back, Aditya Krishnakumar, Ivan Anishchenko, Sergey Ovchinnikov, Jing Zhang, Travis Ness, Sudeep Banjade, Saket R Bagde, et al. Computed structures of core eukaryotic protein complexes. Science, 374(6573) 2021.
John Ingraham, Vikas K. Garg, Regina Barzilay, and Tommi S. Jaakkola. Generative models for graph-based protein design. In NeurlPS, 2019.
Mariusz Jaskolski, Zbigniew Dauter, and Alexander Wlodawer. A brief history of macromolecular crystallography, illustrated by a family tree and its n obel fruits. The FEBS journal, 281 3985-4009, 2014.
Bowen Jing. Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron O. Dror. Learning from protein structure with geometric vector perceptrons. ICLR, 2021.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-50) 2021.
Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP. 2014.
Thomas N. Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. In ICLR, 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolu- tional neural networks. In NeurIPS, 2012.

---

Published as a conference paper at ICLR 2023
Maxat Kulmanov and Robert Hoehndorf. Deepgoplus: improved protein function prediction from sequence. Bioinform., 37(8):1187, 2021.
Maxat Kulmanov, Mohammad Asif Khan, and Robert Hoehndorf. Deepgo: predicting protein functions from sequence and interactions using deep ontology-aware classifier. Bioinform. 34 (4):660-668, 2018.
Alexey Murzin, Steven E Brenner, Tim Hubbard, and Cyrus Chothia. Scop: a structural classifica- tion of proteins database for the investigation of sequences and structures. Journal of molecular biology, 247(4):536-540, 1995.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John F. Canny, Pieter Abbeel, and Yun S. Song. Evaluating protein transfer learning with TAPE. In NeurlPS, 2019.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proc. Natl. Acad. Sci. USA, L18(15)c2016223113 18, 2021. doi: 200073/pmas.201622999
David W. Romero, Anna Kuzina, Erik J. Bekkers, Jakub Mikolaj Tomezak, and Mark Hoogendoorn. Ckconv: Continuous kernel convolution for sequential data. In ICLR, 2022.
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. Schnet: A continuous-files convolutional neural network for modeling quantum interactions. NeurIPS, 2017.
Amir Shanehsazzadeh, David Belanger, and David Dohan. Is transfer learning necessary for protein landscape prediction? arXiv, 201 1.03443, 2020.
Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. IEEE Trans. Pattern Anal. Mach. Intell., 43(8):2647-666 2021. doi: 1011099PAAM1
Assaf Shocher, Ben Feinstein, Niv Haim, and Michal Irani. From discrete continuous convolution layers. arXiv, 2006. 1 1100, 2020.
Martin Simonovsky and Nikos Komodakis. Dynamic edge-conditioned filters in convolutional neural networks on graphs. In CVPR, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. Yoshua Bengio and Yann LeCun (eds.), ICLR, 2015.
Nils Strodthoff, Patrick Wagner, Markus Wenzel, and Wojciech Samek. Udsmprot: universal deep sequence models for protein classification. Bioinform.. 36(8):2401-2409 2020.
Alexey Strokach, David Becerra, Carles Corbi- Verge, Albert Perez-Riba, and Philip M Kim. Fast and flexible protein design using deep graph neural networks. Cell systems, 11(4):402-411, 2020.
Freyr Sverrisson, Jean Feydy, Bruno E. Correia, and Michael M. Bronstein. Fast end-to-end learning protein surfaces. In CVPR, 2021.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In CVPR, 2016.
Nathaniel Thomas, Tess E. Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and translation-equivariaat neural networks for 3d point clouds. arXiv, 1802.08219, 2018.
Michael C Thompson. Todd O Yeates, and Jose A Rodriguez. Advances in methods for atomic resolution macromolccular structure determination. F1000Research, 9, 2020.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In ICLR, 2018.
12

---

Published as a conference paper at ICLR 2023
Shenlong Wang. Simon Suo, Wei-Chiu Ma, Andrei Pokrovsky, and Raquel Urtasun. Deep parametric continuous convolutional neural networks. In CVPR, 2018.
Zichen Wang, Steven A Combs, Ryan Brand, Miguel Romero Calvo, Panpan Xu, George Price, Nataliya Golovach, Emannuel O Salawu, Colby J Wise, Sri Priya Ponnapalli, et al. Lm-gvp: A generalizable deep learning framework for protein property prediction from sequence and structure. bioRxiv, 2021.
Edwin C Webb. Enzyme nomenclature 1992. Number Ed. 6. Academic Press, 1992.
Wenxuan Wu, Zhongang Qi. and Fuxin Li. Pointconv: Deep convolutional networks on 3d point clouds. In CVPR, 2019.
Kurt Wüthrich. The way structures of proteins. Nature structural biology, 8(1)):923995,, 2001.
Wenpeng Yin, Katharina Kann, Mo Yu, and Hinrich Schütze. Comparative study of CNN and RNN natural language processing. arXiv, 1702.01923, 2017.
Zuobai Zhang, Minghao Xu, Arian R. Jamasb, Vijil Chenthamarakshan, Aurélic C. Lozano, Payel Das, and Jian Tang. Protein representation learning by geometric structure pretraining. arXiv, 2203,06125, 2022.
13

---

Published as a conference paper at ICLR 2023
A PROBLEM FORMULATION
To evaluate the proposed CDConv, we apply it to the protein classification problem. Classification is fundamental task for pattern recognition and representation learning. Specifically, given protein, which consists of a list of amino acid coordinates P E R3xN and a list of amino acid types F RRIx,, where N is the number of amino acids, the goal of protein classification is to map P and F to prediction vector E R where k is the number of protein classes or categories,
y ( F)).
The O is a deep neural network and θ is its parameters.
Then, for single-label classification, the softmax function usually used convert y to prediction probabilities, i.e., softmax (y). The class with the highest probability will be considered as the final prediction.
For multi-label classification, the sigmoid function is usually used to convert y to binary prediction probabilities, ie., a (3). Those classes whose probabilities are higher than predefined threshold, e.g., 0.5, will be considered as the final predictions.
B DETAILS OF EVALUATION TASKS AND DATASETS
Protein Fold Classification, Protein fold classification is important in the study of the relationship between protein structure and protein evolution. The fold classes indicate protein secondary structure compositions, orientations and connection orders. We follow Hermosilla et al. (2021) to conduct protein fold classification on the training/validationness splits of the SCOPe 1.75 data set of Hou et al. (2018), which in total contains 1,,712 proteins with 1,195 fold classes. The 3D coordinates of the proteins were collected from the SCOPe 1.75 database (Murzin et al., 1995). The data set provides three different evaluation scenarios. 1) Fold, in which proteins from the same superfamily are not used during training. 2) Superfamily, in which proteins from the same family are not provided during training. 3) Family, in which proteins of the same family are available during training. Mean accuracy is used as the evaluation metric.
Enzyme Reaction Classification. Enzyme reaction classification can be seen as a protein function classification task, which is based on the enzyme-catalyzed reaction according to all four levels of the Enzyme Commission (EC) number (Webb, 1992). We use the dataset collected by Her- mosilla et al. (2021), which includes 384 four-level EC classes and 29,215/2,562/5.00 proteins for viininn//alidlationnesss, respectively. Mean accuracy used the evaluation metric.
Gene Ontology Term Prediction. This task aims to predict the functions of a protein via multiple Gene Ontology (GO) terms, which can be seen as multi-label classification task. Following Glig- orijević et al. (2021), we classify proteins into hierarchically related functional classes organized into three ontologies: biological process (BP) with 1,943 classes, molecular function (MF) with 489 classes and cellular component (CC) with 320 classes The dataset contains 29, /98/33322/3,44 proteins for rraiiin//alldatiiontesss respectively. The Fmax accuracy is used as the evaluation metric.
Enzyme Commission Number Prediction. Different from enzyme reaction classification, this task aims to predict the three-level and four-level 538 EC numbers. We use the training/validationn// splits of Gligorijević et al. (2021), which contains 15.550/1,729/1,99 prediction is also a multi-label classification task. We use F max as the evaluation metric.
proteins in total. EC number
For GO term and EC number prediction, we follow the multi-cutoff splits in (Gligorijević et al., 202 to ensure that the test set only contains PDB chains with sequence identity no more than 95% to the training set, which is also used in (Wang al., 2021; Zhang et al., 2022). Accuracy with all cutoffs is shown in Appendix K.
C EVALUATION METRIC F max
Multi-label classification can be seen as consisting of a series of binary classification tasks. The protein-centria maximum F-Score, i.e., F Fmaxx defined in Gligorijević al. (2021), can be used to measure the accuracy of multi-label classification. Suppose X [0, 1] is decision threshold, y
14

---

Published as a conference paper at ICLR 2023
the prediction probability for the j-th class of the i-th protein, b/ {0, 1 is the corresponding binary class label and / is the number classes. F-Score is based on the precision and recall the predictions for each protein,
Σj 2(1 Er
En Σ
precision,(A)
rccall,
and the average precision and recall over proteins are defined as follows,
Ex precision, (A)
recall(A) Σn recall:
precision(A)
Σ ( A)) 21
where N is the number of proteins. Then, Fmax the maximum F-Score among the different thresholds tested in the range [0, I].
precision(A) recall(A) ) precision( A) + recall(A)
F max
max AE
D FRAMEWORK ILLUSTRATION
The power of CNNS comes from local structure modeling by convolutions and global representation learning via hierarchical architectures. We illustrate our CDConv-based CNN in Fig. 6(a). Our CDConv captures the (3+1)D structure in each local area. To build the pyramid framework, amino acids are downsampled along CDConv layers. The final (3+1)D feature is used for classification.
To avoid the impact of the regularity of sequential orders and the irregularity geometric coordinates on each other, we can employ two-stream 3D-CNN+I ID-CNN framework, as illustrated in Fig. 6(b). First, an independent 3D continuous CNN branch and independent 1D discrete CNN branch are used to extract the global geometric and sequential features, respectively. Then, the two features are fused as the final protein representation. However, because the two types of structures proteins are modeled separately, the local (3+1)D structure is not properly captured. In contrast, our (3+1)D CDConv- CNNs can simultaneously model the local geometry-sequee extract more discriminative local features than 3D-CNN+ID-CND architectures.
structure and thus
E
MORE DETAILS OF IMPLEMENTATION AND TRAINING SETUP
We provide more details of implementation and training setup in Table 4. Our model is imple- mented based on Py ToToh--Geommane 2.0.4 and PyTorch 1.10.0 with CUDA 11.3.1 and cuDNN 8.2.0. Experiments are conducted on a single Nvidia Quadro RTX A5000 GPU.
Table 4: More details of implementation and training setup.
Hyper-parameter
Fold
Enzyme Reaction
GO-BP
GO-MF
GO-CC
EC
Batch size
8
8
24
24
64
24
Channels (c.)
256
256
256
128
128
256
In data pre-processing, amino acid coordinates are normalized as follows,
p
Pis E=
p. = -p.
where p; is the coordinate of i-th amino acid and N is the number of amino acids. During training, we add the simple isotropic additive gaussian noise (Jitter) to amino acid coordinates for data augmentation.
In practice, we find that directly neglecting the neighboring amino acids whose sequential distances greater than [1/2] but geometric distances less than not conducive optimization. In the
15

---

Published as a conference paper at ICLR 2023
(3+1)D feature
LG
10coo drae
88.00 383, 38751
classification
(-:3)) 49) -616
CDConv
CDConv (a) (3+1)D CDConv-bass CNN
CDConv
3D feature
cen nanas
128.0 28:751
3D Conv
3D Conv
3D Conv
classification
feature
OFEE!
1D Conv
1D Conv
1D Conv
(b) Two-stream 3D-CCN+1D-CNN
Figure Illustration of (3+1)D CDConv-based CNN and two-stream 3D-CCN+ID-CNN.
implementation, we include those geometric but non-sequential neighbors as follows,
f(
Σ
g(P+A Pt. Or, O..A0:))) fft
II++A Pillsr. - -1123/5ASU121
Σ
g(P+A Pt, Pt, O,, 014::0))) fft
|pt+s Prllsr, A<-11/2
Σ
g(t+A P:. O. 0.44; .
PILA Sr. A>
where 0_ and A_ are two additional groups of learnable weights for those "special" neighbors. In this way, networks not significantly sensitive to the sequential kernel size hyper-parameter I, as shown in Fig. 5(c).
F IMPACT OF THE NUMBER OF FEATURE CHANNELS
Similar to most CNNs, we use multiple CDCony layers to build hierarchical or pyramid frameworks for protein representation learning. For the i-th CDConv layer, the number of channels is set to [ [c. where c, the initial number of channels. this way, the feature dimension progressively increases when CNNs become deep.
16

---

Published as a conference paper at ICLR 2023
1.0
Fold
1.0
Fold
Super
Super
0.9
Family
0.9
Family
Reaction
Reaction
GO-MF
GO-MF
0.8
GO-BP
0.8
GO-BP
GO-CC
GD-CC
0.7
0.7
0.6
0.6
0.5
0.5
0.4
0.4
16
32
64
128
256
512
2
10
(a) number of channels co
(b) number of layers (network depth) h
Figure 7: Impact of the number of channels Co and the number of CDConv layers (h) on protein recognition. Usually, 2 128 and 2 are sufficient to achieve satisfactory accuracy.
As shown in Fig. 7(a), using too few channels cannot carry enough information and therefore lead to inferior accuracy. However, when employing too many channels, the accuracy does not significantly increase. This means that the number of weights or parameters is enough. To further improve the accuracy, we may need to develop more effective CDConv implementations or network architectures.
G IMPACT OF THE NUMBER OF LAYERS (NETWORK DEPTH)
By default, we use eight CDConv layers (h = 8) to build our CNN for protein representation learning. In this section, investigate the influence of the number of layers. As shown in Fig. 7(b), when too small, networks fail to effectively capture the protein structure. However, when networks are deep, the accuracy even slightly decreases. This is because, with depth increasing, the receptive fields of late layers are too large, which decreases the discriminativenes: of local structures. The other reason is that too many layers may lead overfitting.
Note that for protein fold classification and GO-CC term prediction, using six CDConv layers (h = 6) already achieves satisfactory accuracy. However, for the consistency in visualization, still use eight CDConv layers.
H IMPACT OF GEOMETRY AND SEQUENCE MODELING ON PROTEIN UNDERSTANDING
The proposed CDConv employs the 3D geometry structure and the ID sequence structure in proteins for recognition. In this section, we analyze the influence of each structure.
As shown in Table 3, both the two structures contribute to protein understanding. For example, without the sequence structure modeling, the fold classification accuracy decreases from 56.7% to 41.9%. Without the geometry structure modeling, the GO-BP term prediction F max decreases from 0.453 to 0.371.
For the two types of modeling, geometry structure contributes more than sequence structure. For example, the 3D-only CDCony achieves 65.5% superfamily fold classification while the ID-only CDConv achieves 20.8%. This indicates that geometry structure is more discriminative than sequence structure for protein recognition.
17

---

Published as a conference paper at ICLR 2023
I IMPACT OF RELATIVE SPATIAL ENCODING
Table 5: Comparison of relative spatial encoding and direct 3D displacement encoding for protein
fold classification (%), enzyme reaction classification (%), gene ontology prediction (F max) and
enzyme commission prediction (F max).
Fold Classification
Enzyme
Gene Ontology
Enzyme
Method
Fold
Superfamily
Family
Reaction
BP
MF
CC
Commission
3D displacement encoding
43.3
57.6
95.6
86.0
0.349
0.594
0.405
0.757
Relative spatial encoding
56.7
77.7
99.6
88.5
0.453
0.654
0.479
0.820
To make method rotationally invariant, replace the 3D displacement encoding in Point- Conv (Wu et al., 2019) with relative spatial encoding (Ingraham et al., 2019). The details are as follows,
3D displacement encoding:
0 (pi+A Pt).
as (1pt+ - Pell, O Pe+A Pt O Oc+a).
relative spatial encoding:
P p.
In this section, we conduct experiments to evaluate the effectiveness of relative spatial encoding. As shown in Table 5, relative spatial encoding significantly improves the accuracy, demonstrating the effectiveness of rotational invariance for protein modeling.
MORE DETAILS AND RESULTS OF THE STUDY ON THE IMPORTANCE OF SURFACE AND CENTRAL AMINO ACIDS
99.6
100
96.1
Unmasked Surface-masker Central-masked
90
88.4
accracy Faa (%)
82.0
80
76.3
70
65.4
66.2
57.3
48.6
47.9
45.3
37.2
39.4
30.0
30
27.3
Fold-Family
GO-BP Term
GO-MF Term
GO-CC Term
EC Number
Figure 8: Impact surface and central mask on protein recognition. Masking central area leads to more drops of accuracy. This may indicate that amino acids in central regions may be more important than those surface areas.
To investigate which of the surface and central regions is more important, we mask the same number (20%) of amino acids in the surface and central areas, respectively. Specifically, we first calculate the center of protein as follows,
P
1=1
where N is the number of amino acids and p; denotes the 3D coordinate of the i-th amino acid. For surface region masking, we remove the 20% farthest amino acids from the center based on the geometry distance. For central region masking, remove the 20% closest amino acids to the center.
The results family fold classification, GO term prediction and EC number prediction are shown in Fig. 8. Note that amino acids in central areas usually denser than those in surface regions. This
18

---

Published as a conference paper at ICLR 2023
means that, when masking the same number of amino acids, actually mask more surface regions. Even so, masking central area leads to more drops of accuracy. This indicates that amino acids in central regions may be more important than those in surface areas. However, future experimental efforts are required to further verify this phenomenon.
K GO TERM AND EC NUMBER PREDICTION UNDER DIFFERENT SEQUENCE CUTOFFS
Table 6: F Fiax GO term and EC number prediction under different cutoffs. Our method is compared with CNN (Shanehsazzadeh
al., 2020). ResNet (Rao et al., 2019), LSTM (Rao et al., 2019),
Transformer (Rao et al., 2019), GCN (Kipf & Welling, 2017), GearNet and GearNet-Edge (Zhang et al., 2022). Results are from (Zhang et al., 2022).
Cutoff
30%
40%
50%
70%
95%
30%
40%
50%
70%
95%
Method
GO-BP
GO-MF
CNN*
0.197
0.195
0.197
0.211
0.244
0.238
0.243
0.256
0.292
0.354
ResNeet
0.230
0.230
0.234
0.249
0.280
0.282
0.288
0.308
0.347
0,405
LSTM"
0.194
0.192
0.195
0.205
0.225
0,223
0.229
0.245
0.276
0.321
Transformer'
0.267
0.265
0.262
0.262
0.264
0.184
0.187
0.195
0.204
0.211
GCN*
0,251
0.250
0.248
0.248
0.252
0.180
0.183
0.187
0.194
0.195
GearNet
0.309
0.309
0.315
0.336
0.356
0.382
0.397
0.425
0,474
0.503
GcarNet-Edge
0.345
0.347
0.354
0.378
0,403
0.444
0.461
0.490
0.537
0,580
CDConv (ours)
0.381
0.390
0.401
0.428
0.453
0.533
0.553
0.577
0.621
0.654
Method
GO-CC
EC
CNN'
0.258
0.257
0.260
0.263
0.387
0.366
0.361
0.372
0.429
0.545
ResNet'
0.277
0.273
0,280
0.278
0.304
0.409
0.412
0.450
0.526
0.605
LSTM"
0.263
0,264
0.269
0.270
0.283
0.247
0.249
0.270
0.333
0.425
Transformer'
0.378
0.382
0.388
0.395
0.405
0.167
0.173
0.175
0.197
0.238
GCN"
0.318
0.318
0.320
0.323
0.329
0.245
0.246
0.246
0.280
0.320
GearNet
0.381
0.385
0.393
0.398
0.414
0.557
0,570
0.615
0.693
0,730
GearNet-Edge
0.394
0.394
0.401
0.408
0,450
0.625
0.646
0.694
0.757
0.810
CDConv (ours)
0.428
0.435
0.440
0.451
0.479
0.634
0.659
0.702
0.768
0.820
For GO term and EC number prediction, we can employ different cutoff splits (Gligorijević et al., 2021): the proteins in the test set are split to have < 30%, < 40%, < 50%. < 70 % and 95 % similarity to the training set, respectively. The results are shown Table 6. Our CDConv significantly outperforms the existing methods in most experimental scenarios. For example, for GO-MF term prediction under cutoff 40%, our CDConv outperforms GearNet-Edge by 9.2%. For GO-CC term prediction under cutoff 70%, CDConv outperforms GearNet-Edge by 4.3%. This demonstrates the effectiveness of the proposed method.
L COMPARISON WITH PRETRAINING METHODS
Recently, pretraining or self-supervised learning on 3D protein structures has gained a lot of attention in the fields of protein modeling and structural bioinformatics. The goal of those methods to exploit unlabeled protein data to boost accuracy by designing self-supervised or contrastive objective functions. For example, Hermosilla & Ropinski (2022) proposed to train networks by maximizing the similarity between representations from sub-structures of the same protein, and minimizing the similarity between sub-structures from different proteins. Zhang et al. (2022) proposed to maximize the similarity between representations of different augmented views of the same protein while minimizing the agreement between views of different proteins, and to predict amino acid types, the Euclidean distance between nodes connected in the protein graph, the angle between nodes and the dihedral angles between edges. However, as mentioned by Fan et al. (2022), is unclear whether few self-supervision tasks sufficient for learning effective representations and which
19

---

Published as a conference paper at ICLR 2023
Table 7: Accuracy (%) of protein fold classification. RResults are from (Hermosilla & Ropinski, 2022). Ressults from (Zhang al., 2022).
Fold Classification
Method
Pretraining Dataset
(Size)
Fold
Superfamily
Family
DeepFRI
Pfam (10M)
15.3
20.6
73.2
ESM-Ib
UniRef50 (24M)
26.8
60.1
97.8
ProtBERT-BEDD
BFD (2.1B)
26.6
55.8
97.6
IEConv (amino level)
PDB (476K)
50.3
80.6
99.7
GearNet-Edge-IE IEConv with Multiview Contrast
AlphaFoldDB (805K)
54.1
80.5
99.9
BearNet-EdgeeIEOO IEConv with Residue Type Prediction
AlphaFoldDB (805K)
48.8
71.0
99.4
Feda-NetFdge--FCCom with Distance Prediction
AlphaFoldDB (805K)
50.9
73.5
99.4
Geadreet-Fdge-IRCoom with Angle Prediction CearNeet-FdgeIFFCooe with Dihedral Prediction
AlphaFoldDB (805K)
56.5
76.3
99.6
AlphaFoldDB (805K)
51.8
77.8
99.6
IECoиv (residue level)
47.6
70.2
99.2
GearNet-Edge-IECon
48.3
70.3
99.5
CDConv (ours)
56.7
77.7
99.6
Table 8: Accuracy (%) of enzyme catalytic reaction classification and F Fmax of gene ontology term prediction and enzyme commission number prediction. *Results are from (Wang et al., 2021).
RRssults are from (Hermosilla & Ropinski, 2022). RRssults are from (Zhang et al., 2022).
Enzyme
Gene Ontology
Enzyme
Method
Pretraining Dataset (Size)
Reaction
BP
MF
CC
Commission
DeepFRI
Pfam (10M)
63.3'
0,399t
0.465t
0.460*
0.631
ESM-1b
UniRef50 (24M)
83.1
0.470
0.657
0.488
0.864
ProdBERT-BED
BFD (2.1B)
72.2'
0,279*
0.456*
0.408*
0.838'
LM-GVP
UniRef100 (216M)
0.417
0.545
0.527
0.664
IEConv (amino level)
PDB (476K)
88.1
0.468
0.661
0.516
GearNet-Edge with Multiview Contrast
AlphaFoldDB (805K)
87.5
0.490
0.654
0.488
0.874
GearNet- Edge with Residue Type Prediction
AlphaFoldDB (805K)
86.6
0.430
0.604
0.465
0.843
GearNet-EEdg with Distance Prediction
AlphaFoldDB (805K)
87.5
0.448
0.616
0.464
0.839
GearNet- EegrNet-EEde with Angle Prediction GearNet-Edge with Dihedral Prediction
AlphaFoldDB (805K)
86.8
0.458
0.625
0.473
0.853
AlphaFoldDB (80SK)
87.0
0.458
0.626
0.465
0.859
IEConv (residue level)
87.2
0.624
0.431
GearNet-Edge
86.6
0.403
0.580
0.450
0.810
CDConv (ours)
88.5
0.453
0.654
0.479
0.820
task would be considered more. Doersch Zisserman (2017) even attempted to combine several self-supervised tasks to obtain better visual representations.
Different from those pretraining or self-supervised learning works, our method focuses on designing an effective fundamental operation to capture the protein structure, instead of task-related objective or loss functions. An effective fundamental operation general and independent of specific training strategies and may improve the baseline for the pretraining self-supervised learning works.
In this section, to show the effectiveness, we compare our CDConv to pretraining or self-supervised learning methods: DeepFRI (Gligorijević et al., 2021), DeepFRI (Gligorijević et al., 2021). ESM- lb (Rives et al., 2021), ProtBERT-BFI BFD (Elnaggar et al., 2021), LM-GVP (Wang et al., 2021), aeino-acid-level IEConv (Hermosilla & Ropinski, 2022) and GearNet-based methods (Zhang et al., 2022).
As shown in Table 7 and Table 8, without any pretraining or self-supervised learning, CDConv achieves comparable accuracy with those methods and even outperforms them on fold and enzyme reaction classification.
20

---

the relative Published as a conference paper at ICLR 2023
Table 9: Accuracy (%) of distance-only-baase ccompletely-ootationallyy---- (3+1)D CDConv on protein fold classification. 'Results are from (Hermosilla et al., 2021).
'Results are from (Zhang
al., 2022).
Method
Fold
Superfamily
Family
GraphQA (Baldassarre et al., 2021)
23.7
32.5
84.4
GVP (Jing et al., 2021)²
83.8
IEConv (residue level) (Hermosilla & Ropinski, 2022)
47.6
70.2
99.2
GearNet (Zhang et al., 2022)
28.4
42.6
95.3
GearNet-IEConv (Zhang al., 2022)
42.3
64.1
99.1
GcarNet-Edge (Zhang al., 2022)
44.0
66.7
99.1
EeanNet-Ege--ECCom (Zhang et al., 2022)
48.3
70.3
99.5
Distance-only-base CDConv
distance normalization
73.2
99.3
Distance-only-badd CDConv w/ distance normalization
48.6
73.5
99.4
Relative- wateesspppaaaaloiiie based CDConv
56.7
77.7
99.6
M ROTATION EQUIVARIANCE FOR OTHER SCENARIOS
In this paper; we propose new class of convolution, i.e., CDConv, to make the most of the dual discrete and continuous nature of the data to avoid the impact of regular and irregular modeling on each other. The key design of CDConv is to employ independent learnable weights for different regular displacements but directly encode irregular displacements. As prototype, CDConv can be instantiated to various instances. In this paper, we implement a (3+1)D CDConv for protein representation learning. When implementing the 3D geometry modeling part, spatial encoding technique.
The relative spatial encoding is independent of our CDConv and can be replaced with other techniques. When for other scenarios, we can integrate the corresponding rotation equivariance technique into CDConv. Moreover, there be low-bound solution to completely avoid the rotational variance problem: only use the distance information and do not model direction,
g(P++A Pr: 0s) 0s (llp++a - Pell/r),
where the distance is normalized with the neighbor searching radius Finally, the new (3+1)D CDConv implemented as follows,
Σ
g(P+4 PP; St
-.l.- - 0/2
Then, we evaluate this distance-only-basee "'omppeeeyyoootttooalllin-iaaann (3+1)D CDConv on protein fold classification. As shown Table 9, the new (3+1)D CDCony still achieves comparable accuracy with the previous state-of-fe suute-of-the-as -he-tt-- methods.
N COMPUTATIONAL EFFICIENCY AND SCALABILITY
To investigate the performance our method on relatively large proteins, we collect the proteins that contain more than 500 amino acids from the enzyme reaction classification dataset, leading to subset with large proteins, as shown in Table 10.
However, because the training data significantly reduces (from 29,215 5 2,498 proteins), the subset of large proteins may lead to inferior accuracy. Therefore, for a fair comparison, we randomly sample normal subset that contains the same number of proteins that of large proteins.
Then, we conduct enzyme reaction classification on the two subsets with a single Nvidia Quadro RTX A5000 GPU and Intel(R) Xeon(R) Silver 4214R CPU @ 2.40GHz. We also report the mean running time per prediction. As shown Table 11, our CDConv can still effectively and efficiently recognize enzyme reactions.
21

---

Published as a conference paper at ICLR 2023
Table 10:
Details of the original dataset, the subset of large proteins and the normal subset of
randomly sampled proteins for enzyme reaction classification.
Training
Valiadation
Test
Dataset
Largest Smallest Average
Size
|aaggstt
Smallest Average
Size
|Largest
Smallest Average
Size
Original
3,725
18
296.2
29,215]
1,314
24
346.0
2,562
1,119
20
293.1
5,651
Large
3,725
500
722.3
2,498
1,314
500
711.6
319
1,119
500
611.0
450
Normal
2,650
23
293.6
2,498
1,314
31
363.1
319
1,119
25
299.3
450
Table 11: Performance comparison of CDConv enzyme reaction classification between large and normal proteins.
Subset
Accuracy
Running Time
Normal Protcins
67.7%
10.2
Large Proteins
65.3%
13.2
DISCUSSION ON ATTENTION BASED METHODS
Recently, attention-bre based methods (Jumper et al., 2021: Back al., 2021: Humphreys al., 2021; Fuchs et al., 2020) are also widely used for protein-related problems, such as protein structure prediction generation. As discussed in Sec. 5, attention-based techniques may potentially improve our method.
Using deep learning model kind of data can be divided into two parts: searching related elements (e.g., amino acids) and 2) encoding the local structure of the related elements. Attention- based methods can adaptively reflect the correlation degree of two elements and therefore are good at searching related elements. Our CDConv focuses encoding the region of the related elements by making the most the dual discrete and continuous nature of the data. The kinds of methods may facilitate each other, which can be studied in the future.
P INFLUENCE OF MISSING COORDINATES
When some residues' coordinates are not available, we can set their coordinates to a pre-defined value, e.g., (0, 0, 0). In this way, our CDConv can still be used for protein modeling.
To evaluate the effectiveness of this strategy, we randomly select a certain percentage of amino acids and set their coordinates (0, 0, 0) in training and test and conduct protein fold classification. As shown in Table 12, even with 20% missing coordinates, our CDConv still achieves promising results.
Q INFLUENCE OF VOXEL SIZE FOR DISCRETIZING 3D COORDINATES
In our experiments, when using discrete convolutions for 3D geometry structure modeling, we follow the practice in most discrete CNNs and employ kernel size 3, which means there 3 3 27 group of learnable weights for each convolution.
Because the computation resource limited, we fix the kernel size to 3 x 3. this case, there is a trade-off, which is also limitation, to use discrete convolutions for 3D geometry modeling:
To increase the receptive field size, networks have to reduce the resolution of proteins by increasing voxel size.
When using high- reghuuesolution proteins, convolutions can only capture small region each time.
In summary, due the limited computation resource, have fix the kernel size and set it to small range. In this way. we want to increase the receptive field, we can accordingly reduce the resolution. If we want higher resolution, the receptive field has be relatively "shrunk".
22

---

Published as a conference paper at ICLR 2023
Table 12: Accuracy (%) of CDConv with missing coordinates protein fold classification.
Percentage of Residues with Missing Coordinates
Fold
Superfamily
Family
0%
56.7
77.7
99.6
5%
54.6
75.8
99.5
10%
51,6
74.0
99.5
20%
49.8
73.2
99.3
30%
44.0
69.1
99.1
40%
42.8
65.0
Table 13: Influence of voxel size for discrete convolution on protein fold classification.
Voxel Size
Fold
Superfamily
Family
0.5
31.5
42.6
85.9
1.0
30.6
32.3
73.4
27.8
66.4
4.0
19.5
22.9
51.5
8.0
15.0
13.2
35.0
In experiments, we found setting the voxel size for discretizing 3D coordinates to 0.5 achieves the best accuracy. The results for other voxel sizes shown in Table 13. When increasing the voxel size, although the 3 x 3 can capture a relatively large receptive field, the accuracy does not increase. This is because increasing voxel size results in low resolution, which cannot properly describe protein structures.
R EMPLOYING PROTEIN STRUCTURE PREDICTION MODELS FOR PROTEIN CLASSIFICATION
recent years, deep-learning-aases methods have made tremendous achievements in protein structure prediction, such AlphaFold (Jumper al., 2021) and RoseTTAFos (Back et al., 2021). In this section, we investigate using protein structure prediction methods for protein classification.
There is difference in output between protein structure prediction and classification.
Protein structure prediction. The goal of structure prediction is to make residue-wise predictions. The output is N. or XCXX where is the number amino acids and C' is the number of prediction channels. For structure prediction, C 3.
Protein classification. Classification aims to make global predictions, in which global protein representation vector is learned, i.e., RC, In our method, C 1024 or C 2048. Because CNNs usually hierarchical and pyramid, which captures structures from local to global, it is natural for CNNs to learn global representations.
Another difference that protein structure prediction usually exploits external knowledge, e.g., Multiple Sequence Alignments (MSAs) and protein templates. For classification, existing methods are usually based on the given limited training data.
In this section, we convert TTAFoll (Back et al., 2021) l) for protein classification. Specifically, use the outputs of the feature extractor in RoseTTAFole MSA feature E R x384 pair representation 288 and xyz E IR Xx3x3 To generate a global representation vector based these outputs, we first perform sum on the second dimension of the pair representation and get TR N x28s and flatten xyz R Then, concatenate MSA feature, processed pair representation and xyz: N ((3844288++)
R 2888
R Nx681 Third, append fully-connected layer with ReLU to increase the dimension of the concatenation from 681 to 1024, i.e N 1024, Last, we sum those features and generate global representation IR 1024 In this way, classification can be performed.
We use the modified When MSAs not exploited, we follow Baek & Baker (2022) to only use the peptide chain itself.
for protein fold classification. The results are shown in Table 14.
23

---

Published as a conference paper at ICLR 2023
Table 14: Accuracy (%) of the modified RoseTTAF by us on protein fold classification. "Results from (Hermosilla al., 2021). 'Results are from (Zhang al., 2022).
Method
Fold
Supertamily
Family
GraphQA (Baldassarre et al., 2021)*
23.7
32.5
84.4
GVP (Jing et al., 2021)1
16.0
22.5
83.8
IEConv (residue level) (Hermosilla & Ropinski, 2022)
47.6
70.2
99.2
GearNet (Zhang et al., 2022)
28.4
42.6
95.3
GearNet-IECCnn (Zhang et al., 2022)
42.3
64.1
99.1
GearNet- Edge (Zhang et al., 2022)
44.0
66.7
99.1
(Zhang et al., 2022)
48.3
70.3
99.5
CDConv (ours)
56.7
77.7
99.6
Fold
22.7
28.2
85.5
RoseTTAFold (with MSA)
24.2
31.4
87.0
RoseTTAFold (with Template)
31.8
39.5
91.1
RoseTTAFold (with MSA and Template)
33.4
40.4
92.3
Table 15: Accuracy (%) of generalized CDConv with missing coordinates on protein fold classifica- tion.
Percentage of Residues with Missing Coordinates
Fold
Superfamily
Family
0%
56.7
77.7
99.6
5%
54.8
76.2
99.6
10%
52.9
75.7
99.5
20%
50.8
74.5
99.5
30%
47.1
73.1
99.2
40%
46.0
69.2
99.0
When protein templates are not exploited, we use the input protein geometry structure as the only template. Probably because the architecture of RoseTTAFold is not suitable for global representation learning, it achieves limited accuracy. Converting RoseT to pyramid architecture may boost the accuracy for protein classification, which can be studied in the future.
We notice that MSAs and protein templates help classification. They may play a data augmentation role in protein classification. Moreover, protein templates help more than MSAs. This may indi- cate that the 3D geometry structure is more important than the ID sequence structure for protein recognition.
GENERALIZED CDCONV FOR EXPLICITLY DEALING WITH MISSING 3D COORDINATES
To explicitly deal with missing 3D coordinates, we generalize our CDConv as follows,
ft
Σ
Xe Pr. O1) +1 missing (
where Lexsiting indicates that the neighbor's coordinate exists and 1missing indicates that the coordinate missing. When coordinate is available, CDConv models the gcometry-sqquent structure with the function (: (=) When coordinate is not available, CDCony only models the sequence structure with the function (²
When all coordinates are missing, the generalized CDConv becomes 1DConv. Therefore, 1DCony be seen special case of the generalized CDConv.
We employ the generalized CDConv for protein fold classification. As shown in Table 15, the extended CDConv can effectively alleviate the coordinate-missing problem.
24

---

Published as a conference paper at ICLR 2023
T VISUALIZATION OF CDCONV OUTPUTS
We visualize the outputs downsamples amino acids after every layers. No matter for large small proteins, the early layers mainly focus on the central regions of the proteins. Then, the activation spreads to the entire proteins to capture the whole structure. This indicates that the central region of proteins may be more important than other areas. However, experimental efforts are required to further verify this phenomenon.
the CDConv layers in our CNNs in Fig. 9. Recall that the CNN
U INVESTIGATE WHETHER THE CENTRAL REGION OF PROTEIN IS REALLY MORE IMPORTANT
In this paper, we find that amino acids in central regions may be more important than those in surface areas via those visualizations of CDConv outputs. To further verify this phenomenon, we mask the same number of amino acids in the central and surface areas, respectively. Then, we find that masking the central area leads to more accuracy drops, indicating that amino acids in central regions may be more important than those in surface areas.
Even so, this phenomenon may be an artefact of the proposed method. To verify the phenomenon, biochemical experiments are required. However, currently do not have such resources to con- duct biochemical experiments. We hope that the protein engineering community would verify the phenomenon in the future.
25

---

Published as a conference paper at ICLR 2023
CDConv
Figure 9: Visualization of how the proposed network captures protein structure. The color of input dots indicates amino acid type and the color of CDConv outputs denotes the strength of the activation. The early layers tend to focus on the central region and the late layers tend to capture the whole structure, This may indicate that the central region more important than other areas for protein recognition.
COConV
CDConv
CDConv
CDCOnV
CDConv3
CDConv
CDConV 1
Imput
26


Reviews:
[{"review_id": "vJ3yl8UQ4CD", "reviewer": "Reviewer_JFLd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors propose continuous-discrete convolution (CDConv) for joint modeling of geometric and sequential protein structures. This is motivated by the observation of regular sequence structure and irregular geometry structure of proteins. The proposed instantiation of CDConv is rotational invariant, which does not require additional data augmentation and is efficient to optimize. Empirical evaluation on several benchmark datasets demonstrates the effectiveness of the proposed method.", "summary_of_the_review": "The motivation of joint modeling of regular sequence structure and irregular geometry structure of proteins is reasonable. The proposed CDConv achieves this goal by decomposing convolutional kernels into independent weights for different sequential displacements and continuous relative spatial encoding, which is rotational invariant and efficient to optimize. However, some comparison with alternative convolutional mechanisms is not well conducted, as the difference in rotation invariance and necessity for data augmentation may also affect these results. Additional discussion may be needed.", "strength_and_weaknesses": "Pros:\n1. The observation of regular sequence structure and irregular geometry structure is valuable, which leads to the main motivation of this paper. Previous works indeed fail to perfectly cover both aspects of protein structures.\n2. The proposed relative spatial encoding is rotational invariant, which is critical for efficiency model optimization. In addition, such rotational-invariant implementation is more computationally efficient than spherical harmonics based ones, e.g., SE(3)-Transformer.\n\nCons:\n1. I am not sure whether the proposed method can properly deal with protein structure where some residues have missing 3D coordinates, possibly due to the difficulty in experimental determination. According to Equation (7), the construction of relative spatial encoding relies on 3D coordinates of adjacent residues’ C-alpha atoms.\n2. As described in Section 4.2, the sequential kernel size is set to different values in different tasks (fold classification: 11; reaction classification: 25; Go term and EC number prediction: 15). Does this indicate that the proposed model is somehow sensitive to the choice of sequential kernel size? How should this be alleviated?\n3. In Table 3, discrete convolution for 3D geometry structures is evaluated. For discretized 3D coordinates, it would be hard to preserve the rotational equivariance/invariance, thus explicit data augmentation should be adopted for more sufficient training. Could you please provide additional details on how data augmentation is adopted here?\n4. In Table 3, it would be nice to explicitly highlight which methods are rotational invariant (thus does not require data augmentation) and which are not, so as for easier comparison.\n5. In Section 4.5.1, the voxel size for discretizing 3D coordinates is set to 0.5A. Since the geometry radius is 4A, this means that the discrete convolutional kernel is of size 8 x 8 x 8, which may have the risk of over-parameterization and under-fitting. Will this affect the performance of such baseline methods?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dKS56qk9Tz", "reviewer": "Reviewer_hLLm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Discrete convolutions have been widely used on 2D images and 1D protein sequences.  And continuous convolutions have been widely used on 3D data such as point clouds and to some extent on 3D protein structures.  The paper proposes to apply both at once and shows one can obtain improved empirical results with such an approach on protein classification tasks.\n\nHowever, attention based mechanisms e.g. as in AlphaFold and RosettaFold have been largely replacing convolutions over the past 2 years.  The authors should comment on these methods as an alternative.  E.g. why not repurpose the AlphaFold (or a relevant subset of its architecture) to combine the 1D and 3D information?", "summary_of_the_review": "I recommend accepting the paper because it clearly describes a simple and effective new architecture for protein classification problems.\n", "strength_and_weaknesses": "Strengths:\n* The idea is a simple and practically effective melding of two widely used ideas in protein modeling with a large practical and empirical upside.\n* The empirical performance of the method is a strength.  It has state of the art performance on all classification tasks benchmarked.  Can the authors comment on situations in which they expect the approach might not perform as well?\n\n\n\nWeaknesses:\n* The authors should comment on attention-based alternative methods.  In particular the approach of constructing an NxN dimensional pair representation influenced by both 1D and 3D information, and using it to computed biases on attention weights is at this point very well recognized to be extremely effective (at least in the context of protein structure prediction).  Presumably this same mechanism /architecture could be applied to these classification tasks.  Is there a reason why the authors do not compare to this?\n* I find the terminology “(3+1)D” to be very off-putting and distracting since my mind collapses (3+1) into (4), which I suppose is not intended.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "H5E9JOo7Vr", "reviewer": "Reviewer_nEe2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposed a new genre of convolution, namely continuous-discrete convolution (CDConv) for protein representation learning, which is folded from 1D discrete amino acid chains into a 3D continuous space. Rather than separately aggregating 1D and 3D representation, the proposed (3+1)D CDConv unifies continuous and discrete convolutions in the 3D and 1D spaces. The proposed method was validated on a variety of protein properties prediction (classification) tasks and achieves SOTA performance on them.", "summary_of_the_review": "The paper has proposed an interesting convolution model for protein representation learning which is of good quality.", "strength_and_weaknesses": "**Strengths:**\n\n1. The motivation is clearly explained. \n\n2. The proposed framework is original and powerful (in experiments).\n\n3. The framework suits the application of protein representation learning well, and it can potentially inspire the community for representing other complex entities. \n \n**Weaknesses:**\n\n1. It seems the embedding method can only be applied to specific tasks. As stated in Section 3.4, the convolution design satisfies rotation invariant, which is sufficient for properties prediction. However, rotation equivariance is usually required for broader scenarios of protein representation. It is questionable whether the proposed scheme can still fit in.\n\n2. The computational efficiency and the scalability of the proposed method should have been discussed. Considering the protein sequence can go more than thousands of tokens, and the current convolution scheme stack multiple layers to progressively enlarge its receptive field, it is necessary to investigate its performance (and speed) on relatively large proteins. \n\n3. As the experimental results are very impressive, revealing the program really helps back up the reliability of the reported scores.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-vCgBz2hl3z", "reviewer": "Reviewer_qyBv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a Continuous-Discrete Convolution (CDConv) for Geometry-Sequence modeling in proteins. The goal is to increase the accuracy of four tasks, protein fold classification, enzyme reaction classification, gene ontology term prediction, and enzyme commission number prediction. The paper also explained how the CDConv captures the features of protein structure.", "summary_of_the_review": "The paper is to unify continuous convolution and discrete convolution for modeling protein structure. They claim that the results of their method are state-of-the-art accuracy in four tasks. In addition, they explained how the CDConv capture the features of protein structure, i.e., the early layers tend to focus on the central region of the protein and the late layers can capture the whole structure.\n\nHowever, there are some issues in this paper:\na) The author did not give a formal problem formulation. \nb) The description of the experiment is not clear. In section 4.4, they did not give the details of how to mask the central area. The mask method may change the feature of the continuous features. Thus, more accuracy may be caused by the mask method rather than the importance of central amino acids.\n", "strength_and_weaknesses": "Strength:\n1. The paper address a very important problem. \n2. The paper did a good job of integrating 1D features of the peptide chain and 3D features of amino acid coordinates.\n3. The rotation invariance is guaranteed by the convolution kernel design.\n\nWeakness:\n1. There is no formal problem formulation.\n2. The description of the experiment is not clear. For example, the author did not provide details about how to mask the central amino acids in 4.4.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster