Pre-training Protein Structure Encoder via Siamese Diffusion Trajectory Prediction
Authors: Zuobai Zhang
Minghao Xu
Aurelie Lozano
Vijil Chenthamarakshan
Payel Das
Jian Tang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Protein representation learning
diffusion models
self-supervised learning

Abstract: Due to the determining role of protein structures on diverse protein functions, pre-training representations of proteins on massive unlabeled protein structures has attracted rising research interests. Among recent efforts on this direction, mutual information (MI) maximization based methods have gained the superiority on various downstream benchmark tasks. The core of these methods is to design correlated views that share common information about a protein. Previous view designs focus on capturing structural motif co-occurrence on the same protein structure, while they cannot capture detailed atom/residue interactions. To address this limitation, we propose the Siamese Diffusion Trajectory Prediction (SiamDiff) method. SiamDiff builds a view as the trajectory that gradually approaches protein native structure from scratch, which facilitates the modeling of atom/residue interactions underlying the protein structural dynamics. Specifically, we employ the multimodal diffusion process as a faithful simulation of the structure-sequence co-diffusion trajectory, where rich patterns of protein structural changes are embedded. On such basis, we design a principled theoretical framework to maximize the MI between correlated multimodal diffusion trajectories. We study the effectiveness of SiamDiff on both residue-level and atom-level structures. On the EC and ATOM3D benchmarks, we extensively compare our method with previous protein structure pre-training approaches. The experimental results verify the consistently superior or competitive performance of SiamDiff on all benchmark tasks compared to existing baselines. The source code will be made public upon acceptance.

Under review as a conference paper at ICLR 2023
PRE-TRAINING PROTEIN STRUCTURE ENCODER VIA SIAMESE DIFFUSION TRAJECTORY PREDICTION
Anonymous authors Paper under double-blind review
ABSTRACT
Inspired by the governing role of protein structures on protein functions, structure- based protein representation pre-training is recently gaining interest but remains largely unexplored. Along this direction, pre-training by maximizing the mutual information (MI) between different descriptions of the same protein (i.e., corre- lated views) has shown some preliminary promise, while more in-depth studies are required to design more informative correlated views. Previous view designs focus on capturing structural motif co-occurrence in the same protein structure, while they cannot capture detailed atom- and residue-level interactions and also the statistical dependencies of residue types along protein sequences. To address these limitations, we propose the Siamese Diffusion Trajectory Prediction (SiamDiff) method. SiamDiff employs the multimodal diffusion process as a faithful simula- tion of the structure-sequence co-diffusion trajectory that gradually and smoothly approaches the folded structure and the corresponding sequence of a protein from scratch. Upon native protein and its correlated counterpart obtained with random structure perturbation, we build two multimodal (structure and sequence) diffusion trajectories and regard them as two correlated views. A principled theoretical framework is designed to maximize the MI between such paired views, such that the model can acquire the atom- and residue-level interactions underlying protein structural changes and also the residue type dependencies. We study the effective- ness of SifmDiff on both residue-leve and atom-level structural representations. Experimental results EC and ATOM3D benchmarks show that the performance Sifm is consistently competitive on all benchmark tasks, compared with existing baselines. The source code will be made public upon acceptance.
INTRODUCTION
In the past year, thanks to the rise of highly accurate and efficient protein structure predictors based on deep learning (Jumper et al. 2021 Back et al. 202T), the gap between the number of reported protein sequences and corresponding (computationally) solved structures is greatly narrowed. These advances open the opportunity of self-supervised protein representation learning based on protein structures, i.e., learning informative protein representations from massive protein structures without using any annotation. Compared to extensively studied sequence- based protein self-supervised learning like protein language models (EInaggar et al. 2021 Rives al. 202I), structure- based methods could learn more effective representations to indicate protein functions, since a protein sequence determines its structure, and the structure the determinant of its diverse functions (Harms & Thornton (010))
To attain this goal, some recent works have explored different self-supervised learning strategies on protein structures, including contrastive learning (Zhang al.) [202] Hermosilla & Ropinski [2022]
self-prediction (Zhang et al.) [022 Chen et al., 2022) and denoising score matching (Guo et al. 2022
Wu al. 2022a). Among these works, mutual information (MI) maximization based methods (Zhang et al. 2022; Hermosilla & Ropinskij (022) achieve superior performance on protein function and structural class prediction. At the core of these methods, different structural descriptions of the same protein (i.e., correlated views) are built to capture the co-occurrence of structural motifs. However, such view construction scheme fails to capture detailed atom- and residue- interactions and also the statistical dependencies of residue types along protein sequences. Therefore, the MI maximization based on these views may not produce effective representations for the tasks that require to model detailed local structures (e.g., the Residue Identity task from ATOM3D (Towwshend et al. 2020))

---

Under review as a conference paper at ICLR 2023
minor differences on structures and sequences due to point mutations (e.g., the Mutation Stability Prediction task from ATOM3D), as demonstrated by the experimental results in Tables and 2
To tackle such limitation, in this work, propose the Siamese Diffusion Trajectory Prediction (SiamDitf)) method to jointly model fine-grained atom- and residue-level interactions and residue type dependencies. Specifically, given native protein, first derive its correlated counterpart by random structure perturbation. We further extend the original protein and the generated counterpart respectively by the multimodal diffusion process, in which we transform both the protein structure and the protein sequence towards random distribution by gradually and smoothly adding noises. Such a diffusion process is verified as a faithful simulation of the reeuunneeeee co-diffusion trajectory by recent studies (Anand & Achim 2022 Luo et al.] 2022). We regard the diffusion trajectories of the original protein and the generated counterpart as two correlated views, and a principled theoretical framework is designed to maximize the MI between such paired views. Under such a learning framework, the model can acquire the atom- and residue-level interactions underlying protein structural changes and also the residue type dependencies. The learned protein representations are expected to boost diverse types of downstream tasks.
SiamDiff can be flexibly applied to both residue-level and atom-level structures for effective rep- resentation learning. We employ different selervuue algorithms to pre-train residue-level and atom-level structure encoders, and the praitrained models are extensively evaluated on Enzyme Commission number prediction (Gligorijević et al.] 202T) and ATOM3D (Townshend et al. 2020) benchmarks. Experimental results verify that SifmDiff consistently achieve competitive perfor- mance on all benchmark tasks and on both structure levels, in contrast to existing baselines.
2 RELATED WORK
Protein Structure Representation Learning. The community witnessed a surge of research interests in learning informative protein structure representations using structure- encoders and training algorithms. The encoders are designed to capture protein structural information on different granularity, including residue-level structures (Gligorijević et al., 202T, Wang et al., 2002b Zhang
eet al. 2022), atom-level structures (Hermosilla et al.) 202 Jing al.) 202 12 Wang et al. 2022a)
and protein surfaces (Gainza et al. 2020 Sverrisson et al. 2021 Somnath et al. (021))
Recent works study pre-training massive unlabeled protein structures for generalizable representa- tions, covering contrastive learning (Zhang et al. 2022 Hermosilla & Ropinski] 2022). self- pelf-prediction of geometric quantities (Zhang et | 2022 Chen et al.) (222)) and denoising score matching (Guo)
ct al. 2022 Wu et al. 2022a). All these methods only employ native proteins for pre-training. By
comparison, the proposed SifmDiff uses the information from multimodal diffusion trajectories to better acquire atom- and residue-level interactions and residue type dependencies.
Diff'usion Probabilistic Models (DPMs). DPM was first proposed in |/oSo--ickssteinn et (2015) and has been recently rekindled for its strong performance on image and waveform generation (Ho
al. 2020 Chen al. 2020a). Recent works Nicchol & Dhariwal 202T Song et al. 2020; 2021)
have improved training and sampling for DPMs. Besides the DPMs for continuous data, some works
study discrete DPMs and achieve impressive results on generating texts (Austin al. 2021 Li al.,
2022 images (Austin et al.) (02T) and image segmentation data (Hoogeboom et al. 2021
Inspired by these progresses, DPMs have been recently adopted to solve the problems in chemistry and biology domain, including molecule generation (Xu et al. 2022, Hoogeboom et al. [0022) Wu
ct al. 2022b), molecular representation learning (Liu et al. 20022, protein design (Anand & Achim
2022
Luo et al. 2022) and motif- scaffo-iaafol (Trippe et al., (2022). In this work, we novelly study
how
DPMs can help protein representation learning, which aligns with a recent effort (Abstreiter
et al.
2021) diffusion-based image representation learning.
PRELIMINARIES
3.1 PROBLEM DEFINITION
Notations. A protein with n,, residues (amino acids) and n. atoms can be represented as a sequence- structure tuple P = (S, R). We use S = [s,, $2, ... sn,] to denote its sequence with s; as the type of the i-th residue, while R [r1, r2..., rn..] IR.. x3 denotes its structure with r; the Cartesian
2

---

Under review as a conference paper at ICLR 2023
coordinates of the i-th atom. In the paper, we construct graph for each protein with edges connecting atoms with the Euclidean distance lower than threshold. Besides, we also consider residue- protein graphs, concise version of atom graphs that enable efficient message passing between nodes and edges. As in Zhang et al. (002), we only keep the alpha carbon atom of each residue and add sequential, radius and -nnaerstt neighbor edges different types of edges.
Equivariance. Equivariance is ubiquitous in machine learning for modeling the symmetry in physical systems (Thomas al., 2018 Weiler al., 2018 Batzner et al. 2022) and is shown be critical for successful design and better generalization of 3D neural networks (Köhler et al. (020). Formally, function J. : X + y is equivariant w.n.f a group G if F px (z) = Py F(r), where px and Py are transformations corresponding to an element E G acting on X' and V respectively. The function is invariant w.r/ G if the transformations Py is identity. In this paper, we consider the SE(3) group, iee, rotations and translations in 3D space.
Problem Definition. Given a set of unlabeled proteins P (2,,P2, ......, our goal is to train a protein encoder (SS, R) to extract representations that are SE(3)-invariant w.t. protein structures R.
3.2 DIFFUSION MODELS ON PROTEINS
Diffusion models are class of deep generative models with latent variables encoded by a forward diffusion process and decoded by reverse generative process (Sohl-Dickstein et al. (013). There have been recent efforts on applying denoising difffusion models for protein generation (Luo et al. 2022 Anand & Achim 2022). We use po to denote the ground-truth protein and p for / I,....
to be the latent variables over T diffusion steps. Modeling the protein as an evolving thermodynamic system, the forward process gradually injects small noise the data pd until reaching random noise distribution at time T. The reverse process learns to denoise the latent variable towards the data distribution. Both processes are defined Markov chains:
(P1:77 (P)) T (P'(pt Po(P::T-1 = IT Po(P--11(t))
(1)
where q (PP()pt-1) defines the forward process at step and po (P' p(t) with learnable parameters f defines the reverse process at step t.
The generation of protein relies on the joint diffusion process on structures and sequences. Follow- ing[ Luo et al. (2022), we assume 1) the separate definition of the forward process on structures and the conditional independence of sequences and structures in the reverse process: u(P'(pt-))
sequences and 2)
q( (R ( q(S*)((* (()))s(=)), pa pa(pt11pp (p') pe ( po(("-1(p'') (2)
Next, we discuss how to define the diffusion models on protein structures and sequences, respectively.
Diffusion models on 3D structures. Since the coordinates of atoms are continuous variables in the 3D space, the forward process can be defined by adding random Gaussian noise. Then, the reverse process can be parameterized as Gaussian with learnable mean and user-defined variance. That is, = (R''' (o o), (3)
(R'' 1 (R': v where B1, Br series of fixed variances and o, can be any user-defined variance. Since R' is available as an input, following Ho et al. (2020), we reparameterize the mean ua(P! as:
3,R'=1 1,,),, (o
R' (R'-_ 3r cP((P
(PP.)))) _
Ho (P),,)
VI- aa
where aa 1 Br, a. - a and the neural network eØ(-) learns to decorrupt the data and should be cranslationiivaarit and rotation-equivariant w.r.t. the protein structure R².
Diffusion models on sequences. The key of diffusion models discrete attributes is to define transition matrices for the Markov chain. Austin et al. (202I) adds an absorbing state [MASK] for the Markov chain and then each residue either stays the same transits to [MASK] with some probability at each step. For the reverse process, a neural network Po is defined to predict the probability of S and then parameterize the diffusion trajectory with the probability q(S (-,5'( (5)). That is,
((S((8' = Cat (S': p = s s p x Σso (St 5 ()) (5)
where s² E R, x21 denotes the one-hot feature for the sequence St, Q' E 221 denotes the corresponding transition matrix at step f, (at(-) is the categorical distribution and S enumerates all possible residue types.
3

---

Under review as a conference paper at ICLR 2023
T. VIYCGERGEE... ...
VVVCERRFF....
Identity
P1
P1
P
Pi
P (8,
P2
P2
P2
Structure Perturbation
..YYLNGGERGFF...
Figure I: High-level illustration of SiamDiff. Given protein P (S, R), we first obtain two correlated starting states P1 and P2 via identity and structure perturbation functions, respectively. Then, we apply multimodal diffusion process the starting states to construct two siamese trajectories, where we use to denote masked residues. To maximize the mutual information between these trajectories, we predict the noises one trajectory with representations from the other.
4 SIAMDIFF: SIAMESE DIFFUSION TRAJECTORY PREDICTION
4.1 MOTIVATION AND OVERVIEW
Motivation. Recent supervisd learning approaches learn informative representations by maxi- mizing mutual information (MI) between representations of multiple views of the same object with shared information, whose effectiveness has been proved on images (Chen et al., (020b). natural languages (Hjelm et al.] 2018). graphs (Hassani & Khasahmadi] 2020). small molecules (Liu et al.) 202I) and proteins (Zhang et al. 2022 For MI maximization based protein representation learning, previous works (Zhang al. [022 Hermosilla & Ropinski 2022) only focus on capturing protein substructure co-occurrences, while they fail to capture two important properties of proteins: (1) the detailed atom- and residue-level interactions, and (2) the statistical dependencies of residue types along protein sequences.
In this work, introduce the multimodal diffusion trajectory (i.e., the diffusion processes of pair of corresponding sequence and structure) to incorporate these two properties, where (1) rich atom and residue interactions along protein formation can be learned from the structure diffusion process, and (2) residue type dependencies can be learned from the sequence diffusion process. By maximizing the MI between the multimodal diffusion trajectories of pair of correlated proteins, more informative protein representations can be learned. The graphical illustration of this idea is shown in Fig. I We next present the overview of two key components of our learning framework.
Correlated protein diff'usio trajectories construction. Our framework is based on the contrast of correlated protein trajectories, a.k.a., siamese trajectories. These trajectories are generated by a multimodal diffusion process, which is verified to be effective for protein structure- serreeceeeeeeeeeeee co- design (Anand& & Achim 2022 Luo et al. (0022) and thus can reflect physical and chemical principles underlying protein formation (Anfinsen
1972 Dill & MacCallum 2012).
Specifically, we first construct two correlated states from one protein as the starting points of the trajectories. Given the original protein p = (S, R), deem it as the native state P1 = (S1, R1) and build a correlated state Pz (Sg, R2) by randomly perturbing the protein structure, i.e., S2 S,, R2 R1 where _ is the noise drawn from normal distribution.
Then, for states P1 and Pz. we sample their diffusion trajectories PP:T and P with the mutlimodal diffusion process defined in Sec. 3.2 Take P1 (S1, R1) for example. We start the diffusion process from the state pd- P,. The diffusion process is defined by the joint diffusion on structures and sequences, i.e. {(P) P (PP) q(R ((R--TRR)p( ( We utilize the conditional Gaussian
4

---

Under review as a conference paper at ICLR 2023
distributions in (3) derive trajectories structures R1 and the discrete Markov chain in to derive the sequence diffusion processes SI.T. In this way, we define the trajectory P {(S), Ri)) for Pi and can derive the stamese trajectory P {(S), Ry) similarly.
Maximizing mutual information (MI) between representations from siamese trajectories. The main focus of approach is maximize the MI between representations of siamese trajectories constructed as above (see App. A for related works about MI maximization). For notations, we use the bold symbol denote the representation of object and use p and po. denote the corresponding random variables of representations of the siamese trajectories P and P.-T. Because directly optimizing MI is intractable, we instead maximize an approximate lower bound of MI described in the following proposition (see proof in App. B.I..
Proposition 1 With some approximations, the mutual information between representations of two siamese trajectories is lower bounded by:
T(POTT P P2T)) -2 + (((-2)) + +C,
where C is a constant independent of our encoder and the term from trajectory PO- to pa: is defined as
((b-aa)
= -Epger p00.T
Ei=1 DEL (p(PL-LPPd Ph) PoPP p.
(6)
with a being either 2 - or 2.
The terms share the similar formula as the ELBO loss in diffusion models (Ho et al. 2020). Take [(2++) for example. Here q (PI PD) is posterior analytically tractable with our detinition of each diffusion step ((P [P in (3 and 5). The reverse process is learnt to generate a less noisy state p|-1 given the current state Pf and representations of the siamese trajectory p0: which are extracted by the protein encoder to pre-trained.
Essentially, we perform mutual prediction between two siamese trajectories, which is similar to the idea of mutual representation reconstruction Grill et al. (2020); Chen He (202T). In practice, though adding structural noises, Pi and P2 share information about the same protein and thus the whole trajectory of P2 may give too many clues for the denoising towards PI-1, which is harmful for pre-training. To address this issue, we parameterize P (..TT) with po (P)- pp).).) For diffusion sequences, further guarantee that the same of residues are masked in S and St to avoid the leakage of ground-truth residue types across views.
Below we describe the parameterization derive the pre-training objective from (6) in Sec. 4.3] and discuss advantages of our method and its connection with existing works in Sec.
the generation process pe (PI-1IPt Pt)) in Sec. 4.2
4.2 MODELING PROTEIN GENERATION PROCESS
Different from the generation process in traditional diffusion models, the parameterization of Po(P(-L(pt Pb) should inject information from Pj. Therefore, we apply an iE(3)-invarian encoder Φ(-) to Pt. and the extracted atom and residue representations (denoted as at and h$) are employed for this denoising step. Given the conditional independence in (2), this generation process can be decomposed into that protein structures and sequences shown below.
Generation process on protein structures. As in (4) modeling the generation process of protein structures is to model the noise on R4 and gradually decorrupt the noisy structure. This can be parameterized with noise prediction network eø (Pí, P/ t) that is translation manslatioonnvvriannt and rotation- cquivariant w.r.t. Ri Besides, the noise applied on Ri should not change with transformations on R2,, ca should be SE(3)-invariant w.t.f. R2-
To achieve these goals, we define our noise prediction network with atom representations at (which w.r.t. R2) and atom coordinates rf (which is SEuivvaauu w.E.t. Ri). We draw inspirations from recent works (Satorras et al.) (02I) to build an equivariant output based on normalized directional vectors between adjacent atom pairs. Each edge (i, is encoded by its length
5

---

Under review as a conference paper at ICLR 2023
|rí; - rişll2 and the representations of two end nodes a agj, and the encoded score may will be further used for aggregating directional vectors. Specifically, we have
ris Micrri Irff--i, with ma.j MLP(a a2j. MLP([Iri, -rfjll)));
(P) P2, EsENTO maj"
where (1) denotes the neighbors of the atom in the corresponding graph of Pi. Note that ( 2. 1) achieves the equivariance requirement, majj -inaa-ina w.E.t R4 and Ri while rí: rí is -rassation-invariant invariant and rotation-egge renttion-qqivariintt w.E.f. Rí.
Generation process on protein sequences. As in (5) the generation process on sequences aims to predict masked residue types in St with predictor pg. In our setting of mutual prediction, we define the predictor based on representations of the same residues in Sž which are also masked. Hence, for each masked residue i in S2. we feed its representation h2g to an MLP and predict the type of the corresponding residue type oi, in Sp:
Pa ( (S P II, Po 11, (ML
where the softmax function is applied all residue types.
4.3 PRE-TRAINING OBJECTIVE
Given the defined forward and reverse process on two trajectories, we now derive the pre-training
objective based on the mutual diffusion loss in (6) We take the ( for example and its counterpart can be derived in the same way. It can be proved that with the independence assumptions in 2). the objective can be decomposed into a structure loss (R.22+1) and a sequence loss ((221) (see proof in App. B.2
C ( =E ET DKL (R, (R P()
(7)
((S =E DT DKL St) 1 P())]
(8)
Structure loss (R.2-+1) It has been shown in Ho et al. (202)) that the loss function can be simplified under parameterization by calculating KL divergence between Gaussians weighted L2 distances between means Eg and e (see details App. B.3))
( Σ == 7t F.E.OO [[le- - co 1)]
where the coefficients In are determined by the variances $1.-.., Sr. In practice we follow the suggestion in Ho et al. (2020) to set all weights 7 1 for the simplified loss simple 2(R
Since co designed be rotation- equivvr-iuuvaaiiant w.r.t Rf. to make the loss function invariant w.r.r. Ri, the supervision c is also supposed to achieve such equivariance. So we adopt the chain-rule approach proposed [nXu et al. (2022), which decomposes the noise on pairwise distances to obtain the modified noise vector as supervision. We refer readers to Xu et al. (2022) for more details.
Sequence loss (6.2-11) Since we parameterize po P P() with Pn(SPPP Pt)) and S 5o) as in (), it can be proved that the t-th KL divergence term in (8 reaches zero when go (S (P|, P() puts all mass the ground truth Si (see proof in App. B.4 Therefore, for the purpose of pre-training, we can simplify the KL. divergence to the cross entropy loss between the correct residue type sit and the prediction:
C(S.2-1))
EL CE (sf (s)) Pi. Pb)).
(10)
Final pre-training objective. To summarize, the ultimate training objective for our method is
Liotal ( (C( +
(18.2+1) (R.1++2)
((S.1-2)
simple
simple
+ Simple
(11)
where ((..b=a) is the loss term defined by predicting trajectory po:T from trajectory P!:

---

Under review as a conference paper at ICLR 2023
4.4 DISCUSSION
Now we discuss the advantages of our method over previous works. The benefits of these critical designs will be empirically demonstrated by experiments in Sec. [.3
Advantages of multimodal denoising. Compared with previous works focusing on either protein sequences (Yang et al. 2022) or structures (Guo et al.) 2022), in this work, we perform joint denoising on both modalities. Note that given sequence and a structure that exist in the nature with high probability, the sequence-structure tuple P (S, R) may not be valid state of this protein. Consequently, instead of modeling the marginal distribution, we are supposed to model the joint distribution of protein sequences and structures, which can be achieved by multimodal denoising.
Connection with diffusion models. Diffusion models have achieved outstanding performance on image and text generation tasks (Dhariwal & Nichol, |021 Li et al. 2022) and recently been
applied on unsupervised representation learning (Abstreiter et al. 2021 The key to its success is the
denoising objective at different noise levels, which has also been used for self-supervised learning in earlier works of scheduled denoising autoencoders (Geras & Sutton 2014 Chandra & Sharma 2014). Our method differs from these existing works by incorporating the idea of mutual prediction of two siamese diffusion trajectories so as to maximize the mutual information between them.
5 EXPERIMENTS
We conduct experiments on both residue and atom levels to prove the effectiveness of our method.
5.1
EXPERIMENTAL SETUPS
Pre-training datasets. Following Zhang et al. (2022), we pre-train our models with the AlphaFold protein structure database (Jumper et al., 2021; Varadi et al., 2021), including both 365K proteome-
wide predicted structures and 440K Swiss- Prot (Consortium) 2021 predicted structures.
Downstream benchmark tasks. For downstream evaluation, we adopt the EC protein function prediction task (Gligorijević et al. 2021 and four ATOM3D tasks (Townshend et al. 2020).
Enzyme Commission (EC) number prediction task aims to predict ECC numbers of proteins which describe their catalysis behavior in biochemical reactions. This task is formalized as 538
binary classification problems. We adopt the dataset splits from Gligorjevvi et al. (222)) and
use the test split with 95% sequence identity cutoff' following Zhang et al. (2022).
Protein Structure Ranking (PSR) task predicts global distance test scores of protein structure predictions submitted to the Critical Assessment of Structure Prediction (CASP) (Kryshtafovych et al. 2019) competition. This benchmark dataset is split according to the competition year.
Mutation Stability Prediction (MSP) task seeks to predict whether a mutation will increase the stability of protein complex not (binary classification). The benchmark dataset is split upon a 30% sequence identity cutoff among different splits.
4. Residue Identity (RES) task studies the structural role of an amino acid under its local environment. model predicts the type of the center amino acid based on its surrounding atomic structure. The environments in different splits are with different protein topology classes.
Baseline methods. We evaluate our method on both residue- and atom-level structures. GearNet- Edge (Zhang et al.) 2022) and GVP (Jing et al. 2021a) are employed as backbone models for residue and atom levels, respectively. GearNet-Edge models protein structures with different types of edges and edge-type-speciff convolutions, which is further enhanced by message passing between edges. GVP constructs atom graphs and adds vector channels for modeling equivariant features. Based on these two encoders, compare the proposed method with previous protein structural pre-training algorithms including multiview contrastive learning (Zhang et al.) 2022). denoising score matching (Guo et al. 2022) and four self- self-prediction methods (Zhang et al. 2022 iee. residue type, distance, angle and dihedral prediction. Details can be found in App.
Training and evaluation. For fair comparison, we pre-train our model for 50 epochs the AlphaFold protein structure database, following Zhang et al. (2022). For downstream evaluation, we fine-tune the pre-trained models for 200 epochs EC and 50 epochs Atom3D tasks except RES.
7

---

Under review as a conference paper at ICLR 2023
Table Residue-level results on EC and Atom3D tasks. We use 8, and $ to denote the first, second and third best performance among all models.
Method
EC
PSR
MSP
Mean
Rank
Frux
AUPR
Global
Meun p
AUROC
AUPRC
GearNet-hddee Hddge
0.810
0.835
0.739
0.374
0.586
0.192
Denoising Score Matching
0.842
0.862
0.823
0.472
0.559
0.183
8.0
Residue Type Prediction
0.843
0.870
0.839
0.497
0.642
0.229
4,3
w/ pre-training
Distance Prediction
0.839
0.780
0.437
9.3
Angle Prediction
0.853
0.880
0.794
0.449
0.632
0.191
6.3
Dihedral Prediction
0.8597
0
0.735
0,366
0.617
0.210
6.3
Multiview Contrast
0.874*
0.892
0.736
0,362
0.656
0.220
4.7
SiamDiff
0.8641
0,882
0.829!
0.506
0.695
0.331
1,5*
SiamDiff (wh sequusttuct diff.)
0.802
0.787
0.726
0.338
0.626
0,200
9.7
SiamDiff (w/ seq. diff.)
0.858
0.876
0.789
0.474
0.638
0.202
5.2
SlamDiff (w/ struct. diff.)
0.856
0.873
0.818
0.481
0.660
0.246
4.07
SiamDiff (wo MI max.)
0.855
0.875
0.826
0,462
0.616
0.180
6.7
Table Atom-level results on Atom3D tasks. Accuracy is abbreviated as Acc. We use * and to denote the first, second and third best performance among all models.
PSR
MSP
RES
Mean Rank
Method
Global p
Mcan p
AUROC
AUPRC
Acc.
GVP
0.809
0.486
0,652
0,228
0.550
Denoising Score Matching
0.849
0.535
0.625
0.209
0.558
7.2
Residue Type Prediction
0.845
0.527
0.664
0.254
0.558
5.6
w/ pre-training
Distance Prediction
0.825
0.513
0.621
0,250
0.504
8.8
Angle Prediction
0.872*
0,5451
0.659
0.244
0,581
3,2
Dihedral Prediction
0.852
0.538
0.6921
0,260
0.585
2,4*
Multiview Contrast
0.848
0.518
0.640
0.187
0.335
9.0
SiamDiff
0.854
0.548*
0.694
0.263*
0.563
1.6*
SiamDiff (w/o seq &sstruct diff.)
0.796
0.487
0.640
0.216
0.516
9.2
Sifm (w/ seq. diff.)
0.850
0.670
0.255!
0.472
5.2
SiamDiff (w/o struct. diff.)
0.823
0.519
0.642
0,222
0.560
7.0
SiamDiff (w/ MI max.)
0.850
0.540
0.631
0,217
0.556
6.2
Due to the large size of the RES dataset, we set the time limit as 24 hours and thus only fine-tine each model for 12 epochs. More training hyperparameters are stated App. D
For EC prediction, we employ Fmux and AUPR evaluation metrics, following the original bench- mark (Gligorijević et al. 202I).. For PSR prediction, we utilize the global and mean Spearman's 's to assess the ranking performance. The MSP task measures the binary classification performance with AUROC and AUPRC, and the micro-averaged accuracy serves as the evaluation metric of RES. Detailed definitions of F maxx global Spearman's 's p and mean Spearman's 's p are stated in Appendix
5.2 EXPERIMENTAL RESULTS
Tables and report the results of GearNet-Edge and GVP on residue- and atom-level graphs, respectively. We not include the results of GVP on EC and GearNet-Edge on RES tasks due to their poor performance. We analyze and discuss these results below.
Overall, Sifm gains consistently superior or competitive performance on all benchmarks across residue- and atom-level structures. On all residue- and atom-level benchmark tasks, the performance of Sifm is among the top three, which not attained by previous protein structure pre-training methods. Although dihedral prediction performs well atom-level downstream tasks, its performance on the residue-level PSR and MSP tasks is not satisfactory. The extensive experimental results demonstrate the strength of Difm boosting diverse types of cructure-based protein understanding tasks, including function prediction (EC), protein model quality assessment (PSR), mutation effect prediction (MSP) and structural role understanding (RES).
On EC, SifmDiff outperforms all baselines except Multiview Contrast. Note that this function prediction task aims predict their catalyzed biochemical reactions, which are mainly determined
8

---

Under review as a conference paper at ICLR 2023
by the structures of active/binding sites on protein surfaces. The superior performance of Multiview Contrast can be attributed to the design of its correlated views, which aims to capture the statistical relevance of these protein substructures that reflect protein functions and thus aligns representations of proteins with similar functional sites. In contrast, SiamDiff maximizes MI between siamese trajectories for modeling atom- and residue-level interactions underlying protein structural changes.
On PSR, SifmDiff is the best in terms of mean p and ranks the second in terms of global p. Among all baselines, only Residue Type Prediction can achieve consistently good performance on both structure levels. It is worth noticing that both SiamDiff and residue type prediction can acquire the compatibility of protein sequence sequence and structure. Specifically, Residue Type Prediction models the conditional distribution of sequences given structures, while SiamDiff models the joint distribution of sequences and structures. Such cross-modality modeling mechanisms benefit their performance on this task where the validity of generated structures is assessed given a protein sequence.
On MSP, Sifm achieves the best performance. This task classifies a set of similar mutant structures into two groups according to their ability to stabilize protein-protein interactions. Such classification is mostly determined by the local structure change around the mutation site, which can be well modeled by the structure diffusion process in SiamDiff. Essentially, structure diffusion guides the model to recover the original/stable structure from the perturbed/desttablizeed one, which is highly correlated to the objective of MSP. This can also explain why only applying the structure diffusion objective (SiamDiff w/o seq. diff.) still achieves the third best performance at the atom level.
On RES, SifmDiff ranks the third place, lagging behind Angle and Dihedral Prediction. In this task, the types of target residues are predicted given coordinates of its three backbone atoms and atoms from surrounding residues. is known that dihedral angles the protein backbone can determine the distribution of plausible side chain structures (Shapovalov & Dunbrack Jr. 2011) and further imply residue types. Therefore, by capturing this information explicitly or implicitly, Dihedral Prediction and Angle Prediction achieve very good results. By comparison, though SifmDiff is not specifically designed for dihedral modeling, it can still achieve competitive performance on this task.
5.3 ABLATION STUDY
We study different components of our method the last blocks of Tables [
Effect of multimodal diffusion. We study three degenerated settings of multimodal diffusion, ie.. w/o sequence and structure diffusion, w/o sequence diffusion and w/o structure diffusion. For the setting without sequence and structure diffusion, we simply perform contrastive learning between two correlated starting states from a protein. Compared to the original method, all these three settings lead to performance decay on all tasks. These results prove the necessity of both structure and sequence diffusion for SiamDiff to learn structure- and sequence-awar aware protein representations.
Effect of MI maximization. We evaluate our method under the setting without MI maximization. Specifically, we only use the original protein to derive the multimodal diffusion trajectory, and multimodal denoising is performed
this single trajectory. This setting is also clearly inferior to the original SiamDiff, since it cannot capture shared information between correlated views. Such modeling capability is important on many tasks, e.g., general protein function prediction EC (Fmax score before and after removing this component: 0,864 v.s. 0.855), and it is thus an essential component of Diff More ablation studies are provided in App. C.1
CONCLUSIONS AND FUTURE WORK
In this work, we propose the Siamese Diffusion Trajectory Prediction (SiamDiff) for protein structure pre-training. SifmDiff deems the whole diffusion trajectory of a protein a view and seeks to maximize the mutual information between correlated trajectories. multimodal diffusion process in which both the protein folded structure and the protein sequence gradually corrupted towards chaos, so as to capture the joint distribution of structures and sequences. Extensive experiments diverse types of tasks and both residue- and atom-level structures verify the consistently superior or competitive performance of Sifm against previous baselines.
particular, we consider the
In future works, we will enhance SifmDiff with a dedicated multimodal encoder that can better model the protein structure and sequence in a joint fashion, and we will also explore how to incorporate the large-scale protein sequence corpus into the pre-training process.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
For the sake of reproducibility, in App. D we provide detailed data processing schemes, graph con- struction schemes, model configurations, pre-training configurations and fine-tuning configurations. All source code of this work will be released to public upon acceptance.
REFERENCES
Korbinian Abstreiter, Stefan Bauer, Bernhard Schölkopf. and Arash Mehrjou. Diffusion-basee representation learning. arXiv preprint 1405577705., 2021.
Namrata Anand and Tudor Achim. Protein structure and sequence generation with equivariant denoising diffusion probabilistic models. arXiv preprint (iti::2205.50019 2022.
Christian B Anfinsen. The formation and stabilization of protein structure. Biochemical Journal, 128 (4):737, 1972.
Jacob Austin, Daniel Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:1 7981-11993, 2021.
Minkyung Baek, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang. Qian Cong, Lisa Kinch, R Dustin Schaeffer, et al. Accurate prediction of protein structures and interactions using three-track neural network. Science, 373(6557):811-776 2021.
Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Korblluhh Nicola Molinari, Tess Smidt, and Boris Kozinsky. E (3)eqqiivriiant graph neural networks for data-efficient and accurate interatomic potentials. Nature communications, 13(1):1-11, 2022.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. Mutual information neural estimation. In International conference on machine learning, pp. 531-540. PMLR, 2018.
B Chandra and Rajesh Kumar Sharma. Adaptive noise schedule for denoising autoencoder. In International conference on neural information processing, pp. 535-542. Springer, 2014.
Can Chen, Jingbo Zhou, Fan Wang, Xue Liu, and Dejing Dou. Structure-aware protein self-supervised learning. arXiv preprint arXiv:2204.044 042113. 2022.
Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. arXiv preprint XXXx:20099007133 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp.
1597-1607. PMLR, 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 35750-15755 2021.
The UniProt Consortium. Uniprot: the universal protein knowledgebase in 2021. Nucleic Acids Research, 40(D1):4400-4489 2021.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
Ken A Dill and Justin L MacCallum. The prodein-folding problem, 50 years science, 338(6110): 1042-1046, 2012.
Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time. iv. Communications on Pure and Applied Mathematics, 36(2):183-25 1983.
10

---

Under review as a conference paper at ICLR 2023
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Wang Yu, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. Prottrans: Towards cracking the language of lifes code through self- self-supervised deep learning and high performance computing. IEEE Transactions on Puttern Analysis and Machine Intelligence, pp. 1-1, 2021. doi: 10.1 109/TPAMI.2021 200353831.....
Bent Fuglede and Flemming Topsoe. Jensen-shannon divergence and hilbert space embedding. In International Symposium onInformation Theory, 2004. ISIT 2004. Proceedings. pp. 31. IEFE, 2004.
Pablo Gainza, Freyr Sverrisson, Frederico Monti, Emanuele Rodola, D Boscaini, MM Bronstein, and BE Correia. Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nature Methods, 17(2):14-192, 2020.
Krzysztof Geras and Charles Sutton. Scheduled denoising autoencoders. arXiv preprint arXiv: 1406:334063269 2014.
Vladimir Gligorijević, P Douglas Renfrew, Tomasz Kosciolek, Julia Koehler Leman, Daniel Beren- berg, Tommi Vatanen, Chris Chandler, Bryn C Taylor, Ian M Fisk, Hera Vlamakis, et al. Structure- based protein function prediction using graph convolutional networks. Nature communications, 12 (1):1-14, 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21223 2020.
Yuzhi Guo, Jiaxiang Wu, Hehuan Ma, and Junzhou Huang. Self-supervised pre-training for protein embeddings using tertiary structures. In AAAI, 2022.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
Michael U Gutmann and Aapo Hyvärinen. Noise-contrastive estimation of unnormalized statistical models, with applications natural image statistics. Journal of machine learning research, 13(2), 2012.
Michael Harms and Joseph W Thomton. Analyzing protein structure and function using ancestral gene reconstruction. Current opinion in structural biology, 20(3):360-3333 2010.
Kaveh Hassani and Amir Hosein Khasahmadi. Contrastive multi-view representation learning on graphs. International Conference on Machine Learning, pp. 4116-4126 PMLR, 2020.
Pedro Hermosilla and Timo Ropinski. Contrastive representation learning for 3d protein structures. arXiv preprint arXiv:2255. 2022.
Pedro Hermosilla, Marco Schäfer, Matěj Lang. Gloria Fackelmann, Pere Pau Vázquez. Barbora Kozliková, Michael Krone, Tobias Ritschel, and Timo Ropinski. Intrinsic-extrinsic convolu- tion and pooling for learning on 3d protein structures. International Conference on Learning Representations, 2021.
R Devon Hjelm, Alex Fedorov, Samuel
Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint /ttv://88.00660 2018.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling, Argmax flows and multinomial diffusion: Learning categorical distributions, Advances in Neural Information 2021.
Processing Systems,

---

Under review as a conference paper at ICLR 2023
Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning, pp. 8867-8887, 2022.
Bowen Jing, Stephan Eismann, Pratham N. Soni, and Ron O. Dror. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations, 2021a.
Bowen Jing. Stephan Eismann, Pratham N Soni, and Ron O Dror. Equivariant graph neural networks for 3d macromolecular structure. arXiv preprint (.X.x:2106.3338 2021b.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger,
Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-558
2021.
Jonas Köhler, Leon Klein, and Frank Noé. Equivariant flows: exact likelihood generative learning for symmetric densities. In International conference machine learning, pp. 5361-5370. PMLR, 2020.
Andriy Kryshtafovych, Torsten Schwede, Maya Topf. Krzysztof Fidelis, and John Moult. Critical assessment of methods of protein structure prediction (casp)-round xiii. Proteins: Structure, Function, and Bioinformatics, 17(12):0011-122 2019.
Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. Diffusion- Im improves controllable text generation. arXiv preprint arXiv:2205. 14217, 2022.
Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. Pre- training molecular graph representation with 3d geometry. In International Conference on Learning Representations, 2021.
Shengchao Liu, Hongyu Guo, and Jian Tang. Molecular geometry pretraining with se (3)-invariant denoising distance matching. arXiv preprint arXiv:2206. 13602, 2022.
Shitong Luo, Yufeng Su, Xingang Peng. Sheng Wang, Jian Peng, and Jianzhu Ma. Antigen-specine antibody design and optimization with diffusion- generative models. bioRxiv, 2022.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171, 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1rXiv:1807.03744 2018
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15), 2021.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. In International conference machine learning, pp.
PMLR, 2021.
Maxim V Shapovalov and Roland L Dunbrack Jr. smoothed backbone-dependent rotamer library for proteins derived from adaptive kernel density estimates and regressions. Structure, 19(6): 844-858, 2011.
Jascha Sohl-Dickstenn Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine Learning. 2015.
pp.
Vignesh Ram Somnath, Charlotte Bunne, and Andreas Krause. Multi-scale representation learning proteins. Advances in Neural Information Processing Systems, 34, 2021.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 1. 134466, 2020.
12

---

Under review as a conference paper at ICLR 2023
Yang Song, Conor Durkan, lain Murray, and Stefano Ermon. Maximum likelihood training of score-based diffusion models. Advances in Neural Information Processing Systems, 34:1415-1428, 2021.
Freyr Sverrisson, Jean Feydy, Bruno E Correia, and Michael M Bronstein. Fast end-to-end learning on protein surfaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 15272-15281, 2021.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equivarianl neural networks for 3d point clouds. arXiv preprint arXiv: 1002.v:80200822 (9, 2018.
Raphael JL Townshend, Martin Vögele, Patricia Suriana, Alexander Derry, Alexander Powers, Yianni Laloudakis, Sidhika Balachandar, Bowen Jing, Brandon Anderson, Stephan Eismann, et al. Atom3d: Tasks on molecules in three dimensions. arXiv preprint arXiv:2012.04035 2020.
Brian L Trippe, Jason Yim, Doug Tischer, Tamara Broderick, David Baker, Regina Barzilay, and Tommi Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motif- scaffolding problem. arXiv preprint arXiv:2206.0411 2022.
Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, et al. Alphafold protein structure database: massively expanding the structural coverage of protein-sequence space with high-aceuracy models. Nucleic acids research, 2021.
Limei Wang, Haoran Liu, Yi Liu, Jerry Kurtin, and Shuiwang Ji. Learning protein representations via complete 3d graph networks. arXiv preprint arXiv:2207.1266 2022a.
Zichen Wang, Steven A Combs, Ryan Brand, Miguel Romero Calvo, Panpan Xu, George Price, Nataliya Golovach, Emmanuel O Salawu, Colby J Wise, Sri Priya Ponnapalli, et al. I.--yp: an extensible sequence and structure informed deep learning framework for protein property prediction. Scientific reports, 12(1):1-12, 2022b.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. Advances in Neural Information Processing Systems, 31, 2018.
Fang Wu, Qiang Zhang. Dragomir Radev, Yuyang Wang, Xurui Jin, Yinghui Jiang. Zhangming Niu, and Stan 7. Li. Pre-training of deep protein models with molecular dynamics simulations for drug binding. arXiv preprint arXiv:2204.00) 2064. 2022a.
Lemeng Wu, Chengyue Gong, Xingchao Liu, Mao Ye, and Qiang Liu. Diffusion based molecule generation with informative prior bridges. arXiv preprint arXiv:2209.0088: 2022b.
Minghao Xu, Hang Wang, Bingbing Ni, Hongyu Guo, and Jian Tang. Self-supervised graph-level representation learning with local and global structure. In International Conference on Machine Learning, pp. 11548-111 11558. PMLR, 2021.
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv preprint arXiv:2203.022 029233 2022.
Kevin K Yang, Niccolò Zanichelli, and Hugh Yeh. Masked inverse folding with sequence transfer for protein representation learning. bioRxiv, 2022.
Zuobai Zhang, Minghao Xu, Arian Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Protein representation learning by geometric structure pretraining. arXiv preprint arXiv:2203.06122 2022.
Zhaocheng Zhu, Chence Shi, Zuobai Zhang, Shengehao Liu, Minghao Xu, Xinyu Yuan, Yangtian Zhang, Junkun Chen, Huiyu Cai, Jiarui Lu, et al. Torchdrug: powerful and flexible machine learning platform for drug discovery. arXiv preprint arXiv:2202.08320, 2022.
13

---

Under review as a conference paper at ICLR 2023
A MORE RELATED WORKS
Mutual Information (MI) Estimation and Maximization. MI can measure both the linear and non-linear dependency between random variables. Some previous works (Belghazi et al. 2018 Hjelm et al. 2018) try to use neural networks to estimate the lower bound of MI, including Donsker- Varadhan
representation (Donsker & Varadhan 1983), Jensen-Shannon Shannon divergence (Fuglede & Topsoe 2004)
and Woise-Contrassie with InfoNCE loss (Oord et al., 2018) maximizes lower bound of MI and is broadly shown to be
Estimation (NCE) (Gutmann Hyvärinen 2010: 2012). The optimization
superior representation learning strategy (Chen et al. 2020b Hassani & K Khasahmadij 2020) Xu et al.
2021 Liu et al. 2021 Zhang al., 2022). In this work, adopt the MI lower bound proposed by Liu et al. (2022) with two conditional log-like hig-likelihoods, and we formulate the learning objective by
mutually denoising the multimodal diffusion processes of two correlated proteins.
B
PROOFS
B.1 PROOF OF PROPOSITION 1
Proof. First, the mutual information between representations of two trajectories defined as:
I(PU-T PO:T) Epp pPu upl PO-T pO.T) [lg p(P PP.T) p(PP "p(PP" (TPP)
(12)
where the joint distribution is defined as p(P P..T) p(P?. P2)q(P!!. PO (PP (P)).. Next, we can derive lower bound with this definition:
I(POT P
P(PLT O:T : PO.T)
log
p(P (PP PP)p((P)) (P (P!)
P(PITT P.TTT (P(P))p ( (PI: 1:T' PP)) 1:7 (P9)
>E
log
PPP:T P
DO
log
2
p(P (P
((P
P
(P
1:1
pP 1:1
2
1
p(
PPO::T PO.TT
P(PO:T POTTT ((P))(PP) 1.7 P (PD)
log
+ log
2
P(P 0
2) 1
P
1.7
1
P Pi)g(Pi) 1:7 (PD)
1
p(P( 0:
P.T
0:
p(P@ 0:T (P.TT
E
log
log
((P 1:T (P9)
(
1:T 1P0)
2
However, since the distribution of representations are intractable to sample for optimization, we in- stead sample the trajectories ppoIT and po:T from our defined diffusion process, ie.,, p(PP: :PO:T) p(PP, PP) (PP)) ( ( Besides, instead of predicting representations, we use the repre- sentations from one trajectory to recover the other trajectory, which reflects more information than its representation. With these approximations, the lower bound above can be further written as:
EE log p(PoTT P + log
P(PA- (P.TT (P (2))
PPP Jog
P(P (PT)) q(P) (P))
Eppr par
(PP)" + log
c(P) PP)
We now show the first term on the right hand side can be written as the loss defined in O The derivation is very similar with the proof of Proposition 3 in Xu et al. (2022). We include it here for
14

---

Under review as a conference paper at ICLR 2023
completeness:
1PP 0:2 log
Bpp7 ppg
o (PI1T
P(P P?:T) (PP(PP
=Bpp pP-- T log
(P2) ((P\(PD)
p(P) (P/, p2t)
a(P) (PP)
} (PP()))))
=Bppor .
log
log
9(P) (PI Pt)
(PP P!-))
=Bpar p
log (PF (PT) + log log POiT) + Σ
log
1PPi'1P,,70 PP)
1=
= Ppp pg.
Σ DEL
+ c(2--1)
=- c(2-1) + ((2-1))
where we merge the term p p(P P into the sum of KL divergences for brevity and use ((2-1) to denote the constant independent of encoder. Note that the counterpart be derived in the same way. Adding these two terms together finishes the proof of Proposition
B.2 PROOF OF PRE-TRAINING Loss DECOMPOSITION
We restate the proposition of pre-training loss decomposition rigorously as below.
Proposition 2 Given the assumptions 1) the separation of the diffusion process protein structures and sequences
q(Pi)Pt-L) = q(R'IRR)-)) ((S)) 1
(13)
and 2) the conditional independence of the generation process
Pe(P!-1(Pl Pb)) po (R', (P. Pt)) 1 Pt))
(14)
can be proved that
((b+ta) (( + ((S,b+a))
(15)
where the three loss terms are defined as
(b =E 1-1
[ DKL (g(P)- pt Pa) po (Pb P())]
((R,baa) =E ET 1= DEL (g(R) a
T
- RR', R2) ( P:: P())
((S,bma) =E [T DEL (g(St 1S 12 1ppt P
with + a referring to the term from trajectory PO. to P.:T.
Proof. Let c.: to denote the t-th KL divergence term in LO), Then, we have
c.0-10) DxI. (g(Pt- Pd 1Po (P!- p!:T)
= DKL (La(Rt- R², R (5
- lp, p.: 1pp pb:T) DKL (q(R(-11p)) 1oo (P. )p: + DEL (q(S%- ISt
S S. (R',
(p) (Pos--1p p!.T)
C (R,bb+a))
L (S,b-ra)
where the assumptions [33 and [14) in the second equality. The third equality is due to the additive property of the KL divergence for independent distributions, Adding T KL divergence terms together will lead to (15).
15

---

Under review as a conference paper at ICLR 2023
B.3 PROOF OF SIMPLIFIED STRUCTURE LOSS
For completeness, we show how to derive the simplified structure loss, The proof is directly adapted from (Xu al. 2022).
Proposition 3 Given the definition of the forward process
B,R(-1,B11). ,B+l);
(16)
and the reverse process
Po ( P() - -N(R'-' :oo t) o2),
(17)
1 R². Bt co Pi
Po Pbit) (P
(18)
Vau
v1 az
the structure loss function
((R Ei= DKL (g(R)- ( P())
(19)
can simplified to
((R = =En= (.lerN((o.)) [lle- - co t) 1)]
(20)
where 7t 2a:(1-a;-1) with at = -3 Br αe = II 0gs and b -> is either 2 or -
First, we prove ( R.) = NN varRo (1 - a,) Let e be the standard Gaussian random variable at time step Then, we have
Proof.
R¹4 Va + Viee
Var-iarR V + VBc
-VaR! + V0.01-1-.0021111 + +... VoI-BB-----1 + Vic.
which suggests that the mean of R2 is Va,Ro and the variance matrix is (ayat-1...a20 +... a + 3,) = (1 a.
Next, we derive the posterior distribution as:
1 RR,,R (R', R (R 1 R2,)
(R2,
N(R:; Ro B.T) (RRR-1 : Va-iRR (1 ()-
NR / Ro, (1 (xt))
=M(R--1 vā,- Rd varl 1 5 1)
1 0t-1 B.T).
Rb
āt
1 at
Let B, 1-4;
3t, then the f-th KL divergence term can be written as:
1-a,
DKL.
(q(Rt-1 R. -Ipp Pb))
V Rd + vot(1 - āx-1))
Be
(pt a: Pb: |
R²
23,
1 1- ãt
v 0t
VI
at
Vo
Rt - -1-out
+ Vanl - a(-1)
Be
<p(Pt Pb,)) |
E.
Rt.
Rt.
2,9
1- 01
Vor
1 - i
Voh
vI a.
83
Ee ||c
(o(P, P P)
2,,
ar(1
=m,,
|||e -o- P 1)12)
which completes the proof.
0
16

---

Under review as a conference paper at ICLR 2023
B.4 PROOF OF SIMPLIFIED SEQUENCE LOSS
Now we show the equivalence of optimizing sequence loss ((S,b+a)) and the masked residue type prediction problem Sg.
Proposition 4 Given the definition of reverse process on protein sequences
Po IP' P() Esy - Sa) po (Pa. P)),
(21)
the sequence loss ((S.b+a) reaches zero when pa (S Pb)) puts all mass the ground truth S
Proof.
The loss function can be written ((S
T t=1 DEL (9(S) S S ( Pb))]
Esg 9(5 1
Sa) (5 (S
Pl. Pa)
Σ
E
DKL t=1
Z
where Z is the normalization constant. Hence, when pe (5 Pd Pb) puts all the ground truth S the distribution - Pt) will be identical with q( (SS S2), which makes the KL divergence become zero.
C MORE EXPERIMENTAL RESULTS
C.1 ABLATION STUDY
Ablation study of on
Ablation study EC
Fraxx AUPR
Frax AUPR
0.85
0.85
0.24
0.01
0.3
10
50
100
1000
(a)
(b)
Figure 2: Ablation study of (a) structure perturbation scale and (b) time step number T on EC.
Effect of structure perturbation scale In Fig. 2(a). we present the performance of SiamDiff on EC under different structure perturbation scales. We can observe the superiority of 0.3 over other settings both evaluation metrics. This result illustrates that moderately large structure perturbations favor Sif'' effectiveness. Under such condition, the correlated views can distinguish from each other while share sufficient common information, leading the moderate difficulty of the SiamDiff pre-training task.
Effect of time step number T. Fig. 2)b shows SiamDiff's performance EC by using different time step numbers. can be observed that too few time steps (i.e., or 10 time steps) and too many time steps (i.e., 1000 steps) both lead to the inferior performance. In these situations, the task of each denoising step is either too hard (with too few time steps) or too simple (with too many time steps). Therefore, is suggested SiamDiff pre-training method.
moderately many time steps (e.g., 50 100 time steps) in the
D EXPERIMENTAL DETAILS
In this section, we introduce the details of our experiments. All these methods are developed based on PyTorch and TorchDrug (7hu al. 2022).
17

---

Under review as a conference paper at ICLR 2023
Graph construction. For atom graphs. connect atoms with Euclidean distance lower than distance threshold. For PSR and MSP tasks, we remove all hydrogen atoms following Unng et al. (2021b). For residue graphs, discard all non-alpha-carbon atoms and add three different types of directed edges: sequential edges, radius edges and K-nearest neighbor edges. For sequential edges, two atoms connected if their sequential distance is below threshold and these edges are divided into different types according to these distances. For two kinds of spatial edges, we connect atoms according to Euclidean distance and k-nearest neighbors. We further apply long range interaction filter that removes edges with low sequential distances.
Backbone models. We directly borrow the best hyperparameters reported in the original paper of GearNet-Edge (Zhang al. 2022). We adopt the same configuration of relational graph construction, i.e., the sequential distance threshold dseq
the radius dradius 10.0A, the number of neighbors k= == 10 and the long range interaction cutoff chong = 5. We use one-hot features of residue types as node features and concatenate (I) one-hot features of end nodes, (2) one-hot features of edge types. (3) sequential distance, (4) spatial distance as edge features. Then we use 6 message passing layers with 512 hidden dimensions and ReI.U as the activation function. For edge message passing, the edge types on the line graph are determined by the discretized angles. The hidden representations in each layer of GearNet will be concatenated for the final prediction. Since only alpha carbon atoms are kept in the graph, their representations are used for both atom and residue representations.
For GVP, the original design only includes atom types as node features, which makes pre-training tasks with residue type prediction very difficult to learn. To address this issue, slightly modify its architecture to add add the embedding of atom and corresponding residue types as atom features. Then, the default configurations in Jing et al. (2021b) are adopted. We construct for each protein by drawing edges between atoms closer than 4.5A. Each edge is featured with 16-dimensional Gaussian RBF encoding of its Euclidean distance. We use five GVP layers and hidden representations with 16 vector and 100 scalar channels and use ReLU and identity for scalar and vector activation functions, respectively. The dropout rate is set as 0.1. The final atom representations are followed by a mean pooling layer obtain residue and protein representations.
atom graph
Pre-training methods. Here we briefly introduce the considered baselines. Multiview Contrast aims to maximize the mutual information between correlated views, which are extracted by randomly chosen augmentation functions to capture protein sub-structures. Residue type, distance, angle and dihedral prediction masks single residues, single edges, edge pairs and edge triplets, respectively, and then predict the corresponding properties. Denoising score matching performs denoising on noised pairwise distance matrices based on the learnt representations.
For all baselines in (Zhang et al. 2022), we adopt the original configurations, For Multiview Contrast, we use subsequence cropping that randomly extracts protein subsequences with no more than 50 residues and space cropping that takes all residues within a 1 15A Euclidean ball with a random center residue. Then, either an identity function or a random edge masking function with mask rate equal to 0.15 is applied for constructing views. The temperature T in the InfoNCE loss function is set as 0.07. We set the number of sampled items in each protein as 256 for Distance Prediction and as 512 for Angle and Dihedral Prediction. The mask rate for Residue Type Prediction is set as 0.15. When masking a residue on atom graphs, we discard all non-backbone atoms and set the residue features as zero. Since the backbone models and tasks in our paper are quite different with those in Guo jet al. (2022), we re-implement the method on our codebase. We consider 50 different noise levels log- linearly ranging from 0.01 to 10.0.
For our method, we set the variance of structure perturbation noises e as 0.3 when constructing the other view. For structure diffusion, we use sigmoid schedule for variances 3r with the lowest variance 31 = 1e and the highest variance 3r = 0.1. For sequence diffusion, we simply set the cumulative transition probability to [MASK] over time steps as a linear interpolation between minimum mask rate 0.15 and maximum mask rate 1.0. The number of diffusion steps is set as 1000.
All other optimization configurations for these pre-training methods are reported in Table 3 All methods are pre-trained on four Tesla A 100 GPUs and Table repoort the batch sizes on each GPU.
Fine-tuning on downstream tasks. For all models on all downstream tasks, we apply the a three- layer MI.P head for prediction, the hidden dimension of which is set the dimension of model
18

---

Under review as a conference paper at ICLR 2023
Table 3: Optimization configurations for pre-training methods. Here max length denotes the maximum number of residues kept in each protein and Ir stands for learning rate.
Max length
Batch size
Method
Optimizer
Ir
residue
atom
residue
atom
Residue Type Prediction
100
100
96
64
Adam
le-3
Distance Prediction
100
100
128
64
Adam
le-3
Angle Prediction
100
100
96
64
Adam
le-3
Dihedral Prediction
100
100
96
64
Adam
le-3
Multiview Contrast
64
Adam
le-3
Denoising Score Matching
200
200
12
12
Adam
le-4
SiamDiff
150
100
32
32
Adam
le-4
outputs. The batch sizes for each model are chosen according the memory limit. For all residue-level tasks, we fine-tune the GearNet-Edge on 4 GPUs (A100 for EC/GO and V100 for PSR/MSP) with the batch size a single one 2. For all atom-level tasks except RES, set the batch size 2 and fine-tune the model on a single V100 GPU. For RES, we fine-tune the pre-trained GVP on A100 GPUs with the batch size single one 32.
Evaluation metrics. We clarify the definitions of Fmax (used in EC), global Spearman's 's p (used in PSR) and mean Spearman's 's p (used in PSR) as below:
max denotes the protein-centric maximum F-score. first computes the precision and recall for each protein at decision threshold E [0, 1]:
precision, (t) = Σ, 1 E P.(1)TT)) Σ, E
recall, E, 1 E P.(
(22)
Aff
where f' denotes functional term in the ontology, Ti is the set of experimentally determined functions for protein i, (t) is the set of predicted functions for protein whose scores are greater equal to t, and [-]] represents the indicator function. After that, the precision and recall are averaged over all proteins:
precision(t) (t) Σ precision, (t), recall
Σ rtc.
(23)
where N denotes the total number of proteins, and (1) denotes the number of proteins which contain at least one prediction above the threshold t, i.e., [P) > 0.
Based on these two metrics, the F Fmax score is defined as the maximum value of F-measure over all thresholds:
2 precision(t) recall(t) precision (t) + recall())
F Fmax max
(24)
Global Spearman's p for PSR measures the correlation between the predicted global distance test (GDT_ [GDT_TS)ae score and the ground truth. It computes the Spearman's 's p between the prediction and the ground truth over all test proteins without considering the different biopolymers that these proteins lie in.
Mean Spearman's P for PSR also measures the correlation between GDT predictions and the ground truth. However, it first splits all test proteins into multiple groups based on their corresponding biopolymers, then computes the Spearman's 's p within each group, and finally reports the mean Spearman's 's p over all groups.
19


Reviews:
[{"review_id": "vtB3-iVQEzl", "reviewer": "Reviewer_kp2m", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper applied diffusion model to learn the protein structure from protein sequence. Compared with previous method, they introduced contrast learning and utilized mutual information. The experiment revealed that the proposed method showed marginal performance increase. ", "summary_of_the_review": "The paper focus on the residual level prediction accuracy. It utilize diffusion model with added mutual information training. However, there lack some clear linkage between diffusion model and synthetic mutual information, which makes the method not intuitive. Moreover, the added noise is quite straightforward, which is not optimal to reflect conformation change. ", "strength_and_weaknesses": "Strength:\nThe proposed method tried to achieve better protein residual presentation by introducing contrast learning and maximize mutual information.\nThe proposed method showed marginal performance increase compared with existing methods.\n\nWeakness:\nAs far as I regard, the proposed method is not quite intuitive. Diffusion model is already a method that apply graduate noise. Such that, introducing another level of random noise for contrast learning is not intuitive to me. The marginal performance increase also suggest some conflict between diffusion model and the contrast learning. I was wondering is there some theoretical analysis between diffusion model and the introduced random noise?\nI also do not quite agree with the claim that added random noise could reflect the conformational fluctuation. Simply add the noise on the structure is not equivalent to random permutation. Sometimes, a random permutation could drastically change the protein structure. Simply add random noise, could never reflect such conformational change. \n\n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "tZNO-VQCnr", "reviewer": "Reviewer_yEUV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors consider the task of unsupervised learning of protein structures. They do so by maximizing the mutual information between two diffusion trajectories that start with a protein and a perturbed copy of the protein. The authors prove a lower bound of the mutual information which resembles an ELBO and specifically depends on the likelihood of one trajectory given the other.\nAs such one trajectory is used to compute the noises in the other trajectory.\nBy doing so, the procedure encodes both the structure and sequences jointly.", "summary_of_the_review": "- I am giving this paper a weak reject due to the minor concerns noted in the above two sections, specifically about reporting on the experimental results and making the writing more accessible to the general audience at ICLR. I am willing to upgrade my rating based on my interaction with the authors during the discussion phase.", "strength_and_weaknesses": "### Strengths\n- The paper comes up with a lower bound for the mutual information between two diffusion sequences and uses it to lean an encoder for protein structure. This is an interesting case of theory guiding practice and makes the work interesting.\n- The ablation study shows that all the components of the procedure contribute to improving the model.\n\n### Weaknesses\n- What are correlated views? These show up in the abstract and introduction without a simplified explanation and throw off a reader who is (even very slightly) outside the field.\n- Table 1 and 2 are not color blind friendly, the community seems to use bold and underline to denote first and second, One can also use markers like $\\dagger$ and $\\circ$. Standard deviations are missing. Provide context for $F_{\\text{max}}$, and $\\rho$ in the caption for ease of skimming. In fact I was unable to find a definition of $F_{\\text{max}}$.\n- The abstract talks about comparing with motif based methods, this comparison doesn't show up in the experiments. \n- What do you mean by `We do not include the results of GVP on EC and GearNet-Edge on RES tasks due to their poor performance.`?\n- [Personal nitpick] Page 2 first sentence - shortening Tables to Tabs is non-standard.\n- The writing was pretty dense and I required multiple reads to understand the text. However the figures helped and the fact that this is not my primary field of research is not the authors' fault.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kUbpWWPAhyy", "reviewer": "Reviewer_UBXw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed to use diffusion model as a pretrain model for several downstream tasks.\n\nIt decomposes the structure and uses a DDPM like diffusion for continuous variable and a multimodel diffusion for categorical variable. Experiment shows that the diffusion model serves as a good pretrain model.", "summary_of_the_review": "See above.\n\nMy evaluation is a educated guess.", "strength_and_weaknesses": "Strength:\nUsing diffusion model as a pretrain model seems to be a new application.\n\nWeakness:\n1. I am not an expert of the application area this paper considers. But simply examing the experiment, it seems the improvement is not large. There is no repeated experiment and it's unknown whether the performance is significant.\n\n2. The overall technical novelty is low. It follows a standard design of diffusion model and the contribution seems restricted to diffusion as a pretrain model?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XC9E6VCah3", "reviewer": "Reviewer_jQVb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose SiamDiff, a method that employs a multimodal diffusion process to simulate the structure-sequence co-diffusion trajectory. SiamDiff maximises the Mutual Information (MI) between paired correlated views of proteins, where one is the native protein and the other is obtained with a structure perturbation. Under this learning framework, the authors argue that the model can acquire the atom- and residue-level interactions underlying protein structural changes, as well as the residue type dependencies. Extensive experiments on diverse and standardised downstream tasks are shown to demonstrate the value of the learned protein representations.", "summary_of_the_review": "My recommendation is an accept. I believe that this is a good paper introducing some novel ideas that are very relevant to protein representation learning. The evaluation of the learned representations in downstream tasks was made following well-established benchmarks. Although, the performance achieved is not the highest for every task, the general value of these representations is evident from the empirical results. \nI do expect the authors to address the issues raised in this review, mainly about clarity.\n", "strength_and_weaknesses": "The authors introduce some interesting and novel ideas in this paper. They show that their Siamese multimodal diffusion process can obtain reasonable protein representations that are useful for several different downstream tasks. Modelling the joint distribution of protein sequences and structures through multimodal denoising allows them to capture SiamDiff to capture the atom and residue-level interaction. Another interesting point is their idea of maximising the MI between two Siamese diffusion trajectories, one based on the native view of the protein, and the other view being generated by a structure perturbation.\n\nThe mathematical framework presented is sound. The authors also present the proofs to every proposition they make. They also mention that the full code will be available for reproducibility. \n\nThe experiments shown in the paper are very thorough, where all the tasks have standardised experimental settings in well-established benchmarks. The empirical results are strong enough to prove the effectiveness of the protein representations obtained by SiamDiff.\n\nI feel that the paper is lacking clarity. While I understand that there are a lot of concepts to fit in the manuscript, I believe that this paper could benefit from high-level intuition behind the main ideas of the paper. The notation feels overloaded and it is hard to follow.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject