SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zuobai Zhang $^{1,2*}$ , Minghao Xu $^{1,2*}$ , Aurélie Lozano $^{3}$
Vijil Chenthamarakshan<sup>3</sup>, Payel Das<sup>3†</sup>, Jian Tang<sup>1,4,5†</sup>
* equal contribution † corresponding author
<sup>1</sup>Mila - Québec AI Institute, <sup>2</sup>Université de Montréal, <sup>3</sup>IBM Research,
$^{4}$ HEC Montréal,  $^{5}$ CIFAR AI Chair
{zuobai.zhang,minghao.xu}@mila.quebec,
{ecvijil,aclozano,daspa}@us.ibm.com, jian.tang@hec.ca
Self-supervised pre-training methods on proteins have recently gained attention, with most approaches focusing on either protein sequences or structures, neglecting the exploration of their joint distribution, which is crucial for a comprehensive understanding of protein functions by integrating co-evolutionary information and structural characteristics. In this work, inspired by the success of denoising diffusion models in generative tasks, we propose the DiffPreT approach to pre-train a protein encoder by sequence-structure joint diffusion modeling. DiffPreT guides the encoder to recover the native protein sequences and structures from the perturbed ones along the joint diffusion trajectory, which acquires the joint distribution of sequences and structures. Considering the essential protein conformational variations, we enhance DiffPreT by a method called Siamese Diffusion Trajectory Prediction (SiamDiff) to capture the correlation between different conformers of a protein. SiamDiff attains this goal by maximizing the mutual information between representations of diffusion trajectories of structurally-correlated conformers. We study the effectiveness of DiffPreT and SiamDiff on both atom- and residue-level structure-based protein understanding tasks. Experimental results show that the performance of DiffPreT is consistently competitive on all tasks, and SiamDiff achieves new state-of-the-art performance, considering the mean ranks on all tasks. Code will be released upon acceptance.
Machine learning-based methods have made remarkable strides in predicting protein structures [44, 5, 50] and functionality [55, 24]. Among them, self-supervised (unsupervised) pre-training approaches [20, 61, 87] have been successful in learning effective protein representations from available protein sequences or from their experimental/predicted structures. These pre-training approaches are based on the rationale that modeling the input distribution of proteins provides favorable initialization of model parameters and serves as effective regularization for downstream tasks [25]. Previous methods have primarily focused on modeling the marginal distribution of either protein sequences to acquire co-evolutionary information [20, 61], or protein structures to capture essential characteristics for tasks such as function prediction and fold classification [87, 33]. Nevertheless, both these forms of information hold significance in revealing the underlying functions of proteins and offer complementary perspectives that are still not extensively explored. To address this gap, a more promising approach for pre-training could involve modeling the joint distribution of protein sequences and structures, surpassing the limitations of unimodal pre-training methods.
To model this joint distribution, denoising diffusion models [37, 65] have recently emerged as one of the most effective methods due to their simple training objective and high sampling quality and diversity [2, 54, 40, 74]. However, the application of diffusion models has predominantly been explored in the context of generative tasks, rather than within pre-training and fine-tuning frameworks that aim to learn effective representations for downstream tasks. In this work, we present a novel approach that adapts denoising diffusion models to pre-train structure-informed protein encoders<sup>1</sup>. Our proposed approach, called DiffPreT, gradually adds noise to both protein sequence and structure to transform them towards random distribution, and then denoises the corrupted protein structure and sequence using a noise prediction network parameterized with the output of the protein encoder. This approach enables the encoder to learn informative representations that capture the inter-atomic interactions within the protein structure, the residue type dependencies along the protein sequence, and the joint effect of sequence and structure variations.
In spite of these advantages, DiffPreT ignores the fact that any protein structure exists as a population of interconverting conformers, and elucidating this conformational heterogeneity is essential for predicting protein function and ligand binding [27]. In both DiffPreT and previous studies, no explicit constraints are added to acquire the structural correlation between different conformers of a specific protein or between structural homologs, which prohibits capturing the conformational energy landscape of a protein [56]. Therefore, to consider the physics underlying the conformational change, we propose Siamese Diffusion Trajectory Prediction (SiamDiff) to augment the DiffPreT by maximizing the mutual information between representations of diffusion trajectories of structurally-correlated conformers (i.e., siamese diffusion trajectories). We first adopt a torsional perturbation scheme on the side chain to generate randomly simulated conformers [36]. Then, for each protein, we generate diffusion trajectories for a pair of its conformers. We theoretically prove that the problem can be transformed to the mutual prediction of the trajectories using representations from their counterparts. In this way, the model can keep the advantages of DiffPreT and inject conformer-related information into representations as well.
Both DiffPreT and SiamDiff can be flexibly applied to atom-level and residue-level structures to pre-train protein representations. To thoroughly assess their capabilities, we conduct extensive evaluations of the pre-trained models on a wide range of downstream protein understanding tasks. These tasks encompass protein function annotation, protein-protein interaction prediction, mutational effect prediction, residue structural contributions, and protein structure ranking. In comparison to existing pre-training methods that typically excel in only a subset of the considered tasks, DiffPreT consistently delivers competitive performance across all tasks and at both the atomic and residue-level resolutions. Moreover, SiamDiff further enhances model performance, surpassing previous state-of-the-art results in terms of mean ranks across all evaluated tasks.
2 DiffPreT: Diffusion Models for Pre-Training
Recently, there have been promising progress on applying denoising diffusion models for protein structure-sequence co-design [54, 2]. The effectiveness of the joint diffusion model on modeling the distribution of proteins suggests that the process may reflect physical and chemical principles underlying protein formation [3, 18], which could be beneficial for learning informative representations. Based on this intuition, in this section, we explore the application of joint diffusion models on pre-training protein encoders in a pre-training and fine-tuning framework.
Notation. A protein with  $n_r$  residues (amino acids) and  $n_a$  atoms can be represented as a sequence-structure tuple  $\mathcal{P} = (\mathcal{S},\mathcal{R})$ . We use  $\mathcal{S} = [s_1,s_2,\dots ,s_{n_r}]$  to denote its sequence with  $s_i$  as the type of the  $i$ -th residue, while  $\mathcal{R} = [\pmb {r}_1,\pmb {r}_2\dots ,\pmb {r}_{n_a}]\in \mathbb{R}^{n_a\times 3}$  denotes its structure with  $\pmb{r}_i$  as the Cartesian coordinates of the  $i$ -th atom. To model the structure, we construct a graph for each protein with edges connecting atoms whose Euclidean distance below a certain threshold.
Equivariance. Equivariance is ubiquitous in machine learning for modeling the symmetry in physical systems [67, 75] and is shown to be critical for successful design and better generalization of 3D
<sup>1</sup>It is important to note that protein structure encoders in this context take both sequences and structures as input, distinguishing them from protein sequence encoders as established in previous works.
Figure 1: A pre-training and fine-tuning framework for DiffPreT. During pre-training, diffusion processes are applied to both protein structures and sequences, with * indicating masked residues. Noise prediction networks, parameterized with an encoder  $\phi(S, \mathcal{R})$ , are employed to restore the original states. The learned encoder  $\phi(S, \mathcal{R})$  is subsequently fine-tuned on downstream tasks.
networks [46]. Formally, a function  $\mathcal{F}:\mathcal{X}\to \mathcal{Y}$  is equivariant w.r.t. a group  $G$  if  $\mathcal{F}\circ \rho_{\mathcal{X}}(x) = \rho_{\mathcal{Y}}\circ \mathcal{F}(x)$ , where  $\rho_{\mathcal{X}}$  and  $\rho_{\mathcal{Y}}$  are transformations corresponding to an element  $g\in G$  acting on the space  $\mathcal{X}$  and  $\mathcal{Y}$ , respectively. The function is invariant w.r.t  $G$  if the transformations  $\rho_{\mathcal{Y}}$  is identity. In this paper, we consider SE(3) group, i.e., rotations and translations in 3D space.
Problem Definition. Given a set of unlabeled proteins  $\mathcal{D} = \{\mathcal{P}_1, \mathcal{P}_2, \ldots\}$ , our goal is to train a protein encoder  $\phi(S, \mathcal{R})$  to extract informative  $d$ -dimensional residue representations  $\pmb{h} \in \mathbb{R}^{n_r \times d}$  and atom representations  $\pmb{a} \in \mathbb{R}^{n_a \times d}$  that are SE(3)-invariant w.r.t. protein structures  $\mathcal{R}$ .
2.2 Diffusion Models on Proteins
Diffusion models are a class of deep generative models with latent variables encoded by a forward diffusion process and decoded by a reverse generative process [63]. We use  $\mathcal{P}^0$  to denote the ground-truth protein and  $\mathcal{P}^t$  for  $t = 1,\dots ,T$  to be the latent variables over  $T$  diffusion steps. Modeling the protein as an evolving thermodynamic system, the forward process gradually injects small noise to the data  $\mathcal{P}^0$  until reaching a random noise distribution at time  $T$ . The reverse process learns to denoise the latent variable towards the data distribution. Both processes are defined as Markov chains:
$$
q \left(\mathcal {P} ^ {1: T} \mid \mathcal {P} ^ {0}\right) = \prod_ {t = 1} ^ {T} q \left(\mathcal {P} ^ {t} \mid \mathcal {P} ^ {t - 1}\right), \quad p _ {\theta} \left(\mathcal {P} ^ {0: T - 1} \mid \mathcal {P} ^ {T}\right) = \prod_ {t = 1} ^ {T} p _ {\theta} \left(\mathcal {P} ^ {t - 1} \mid \mathcal {P} ^ {t}\right), \tag {1}
$$
where  $q(\mathcal{P}^t |\mathcal{P}^{t - 1})$  defines the forward process at step  $t$  and  $p_{\theta}(\mathcal{P}^{t - 1}|\mathcal{P}^t)$  with learnable parameters  $\theta$  defines the reverse process at step  $t$ . We decompose the forward process into diffusion on protein structures and sequences, respectively. The decomposition is represented as follows:
$$
q \left(\mathcal {P} ^ {t} \mid \mathcal {P} ^ {t - 1}\right) = q \left(\mathcal {R} ^ {t} \mid \mathcal {R} ^ {t - 1}\right) \cdot q \left(\mathcal {S} ^ {t} \mid \mathcal {S} ^ {t - 1}\right), \quad p _ {\theta} \left(\mathcal {P} ^ {t - 1} \mid \mathcal {P} ^ {t}\right) = p _ {\theta} \left(\mathcal {R} ^ {t - 1} \mid \mathcal {P} ^ {t}\right) \cdot p _ {\theta} \left(\mathcal {S} ^ {t - 1} \mid \mathcal {P} ^ {t}\right), \tag {2}
$$
where the reverse processes use representations  $\pmb{a}^t$  and  $\pmb{h}^t$  learned by the protein encoder  $\phi_{\theta}(\mathcal{S}^t, \mathcal{R}^t)$ .
Forward diffusion process  $q(\mathcal{P}^t |\mathcal{P}^{t - 1})$ . For diffusion on protein structures, we introduce random Gaussian noises to the 3D coordinates of the structure. For diffusion on sequences, we utilize a Markov chain approach with an absorbing state [MASK], where each residue either remains the same or transitions to [MASK] with a certain probability at each time step [4]. Specifically, we have:
$$
q \left(\mathcal {R} ^ {t} \mid \mathcal {R} ^ {t - 1}\right) = \mathcal {N} \left(\mathcal {R} ^ {t}; \sqrt {1 - \beta_ {t}} \mathcal {R} ^ {t - 1}, \beta_ {t} I\right), q \left(\mathcal {S} ^ {t} \mid \mathcal {S} ^ {t - 1}\right) = \text {r a n d o m} _ {\text {m a s k}} \left(\mathcal {S} ^ {t - 1}, \rho_ {t}\right), \tag {3}
$$
Here,  $\beta_{1},\ldots ,\beta_{T}$  and  $\rho_{1},\ldots ,\rho_{T}$  are a series of fixed variances and masking ratios, respectively. random_mask  $(S^{t - 1},\rho_t)$  denotes the random masking operation, where each residue in  $S^{t - 1}$  is masked with a probability of  $\rho_{t}$  at time step  $t$
Reverse process on structures  $p_{\theta}(\mathcal{R}^{t - 1}|\mathcal{P}^t)$ . The reverse process on structures is parameterized as a Gaussian with a learnable mean  $\mu_{\theta}(\mathcal{P}^t,t)$  and user-defined variance  $\sigma_t$ . Given the availability of
$\mathcal{R}^t$  as an input, we reparameterize the mean  $\mu_{\theta}(\mathcal{P}^t,t)$  following Ho et al. [37]:
$$
p _ {\theta} (\mathcal {R} ^ {t - 1} | \mathcal {P} ^ {t}) = \mathcal {N} (\mathcal {R} ^ {t - 1}; \mu_ {\theta} (\mathcal {P} ^ {t}, t), \sigma_ {t} ^ {2} I), \mu_ {\theta} (\mathcal {P} ^ {t}, t) = \frac {1}{\sqrt {\alpha_ {t}}} \left(\mathcal {R} ^ {t} - \frac {\beta_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} \epsilon_ {\theta} (\mathcal {P} ^ {t}, t)\right), (4)
$$
where  $\alpha_{t} = 1 - \beta_{t}$ ,  $\bar{\alpha}_{t} = \prod_{s=1}^{t} \alpha_{s}$  and the network  $\epsilon_{\theta}(\cdot)$  learns to decorrupt the data and should be translation-invariant and rotation-equivariant w.r.t. the structure  $\mathcal{R}^t$ .
To define our noise prediction network, we utilize the atom representations  $\pmb{a}^t$  (which is guaranteed to be SE(3)-invariant w.r.t.  $\mathcal{R}^t$  by the encoder) and atom coordinates  $\pmb{r}^t$  (which is SE(3)-equivariant w.r.t.  $\mathcal{R}^t$ ). We build an equivariant output based on normalized directional vectors between adjacent atom pairs. Each edge  $(i,j)$  is encoded by its length  $\| \pmb{r}_i^t - \pmb{r}_j^t \|_2$  and the representations of two end nodes  $\pmb{a}_i^t, \pmb{a}_j^t$ , and the encoded score  $m_{i,j}$  will be used for aggregating directional vectors. Specifically,
$$
\left[ \epsilon_ {\theta} \left(\mathcal {P} ^ {t}, t\right) \right] _ {i} = \sum_ {j \in \mathcal {N} ^ {t} (i)} m _ {i, j} \cdot \frac {\boldsymbol {r} _ {i} ^ {t} - \boldsymbol {r} _ {j} ^ {t}}{\left\| \boldsymbol {r} _ {i} ^ {t} - \boldsymbol {r} _ {j} ^ {t} \right\| _ {2}}, \text {w i t h} m _ {i, j} = \mathrm {M L P} \left(\boldsymbol {a} _ {i} ^ {t}, \boldsymbol {a} _ {j} ^ {t}, \mathrm {M L P} \left(\left\| \boldsymbol {r} _ {i} ^ {t} - \boldsymbol {r} _ {j} ^ {t} \right\| _ {2}\right)\right), \tag {5}
$$
where  $\mathcal{N}^t (i)$  denotes the neighbors of the atom  $i$  in the corresponding graph of  $\mathcal{P}^t$ . Note that  $\epsilon_{\theta}(\mathcal{P}^t,t)$  achieves the equivariance requirement, as  $m_{i,j}$  is SE(3)-invariant w.r.t.  $\mathcal{R}^t$  while  $\boldsymbol {r}_i^t -\boldsymbol {r}_j^t$  is translation-invariant and rotation-equivariant w.r.t.  $\mathcal{R}^t$ .
Reverse process on sequences  $p_{\theta}(\mathcal{S}^{t - 1}|\mathcal{P}^t)$ . For the reverse process  $p_{\theta}(\mathcal{S}^{t - 1}|\mathcal{P}^t)$ , we adopt the parameterization proposed in [4]. The diffusion trajectory is characterized by the probability  $q(\mathcal{S}^{t - 1}|\mathcal{S}^t,\tilde{\mathcal{S}}^0)$ , and we employ a network  $\tilde{p}\theta$  to predict the probability of  $S^0$ :
$$
p _ {\theta} \left(\mathcal {S} ^ {t - 1} \mid \mathcal {P} ^ {t}\right) \propto \sum_ {\tilde {\mathcal {S}} ^ {0}} q \left(\mathcal {S} ^ {t - 1} \mid \mathcal {S} ^ {t}, \tilde {\mathcal {S}} ^ {0}\right) \cdot \tilde {p} _ {\theta} \left(\tilde {\mathcal {S}} ^ {0} \mid \mathcal {P} ^ {t}\right), \tag {6}
$$
We define the predictor  $\tilde{p}_{\theta}$  with residue representations  $\boldsymbol{h}^t$ . For each masked residue  $i$  in  $\mathcal{S}^t$ , we feed its representation  $\boldsymbol{h}_i^t$  to an MLP and predict the type of the corresponding residue type  $s_i^0$  in  $\mathcal{S}^0$ :
$$
\tilde {p} _ {\theta} \left(\tilde {S} ^ {0} \mid \mathcal {P} ^ {t}\right) = \prod_ {i} \tilde {p} _ {\theta} \left(\tilde {s} _ {i} ^ {0} \mid \mathcal {P} ^ {t}\right) = \prod_ {i} \operatorname {S o f t m a x} \left(\tilde {s} _ {i} ^ {0} \mid \mathrm {M L P} \left(\boldsymbol {h} _ {i} ^ {t}\right)\right), \tag {7}
$$
where the softmax function is applied over all residue types.
2.3 Pre-Training Objective
Now we derive the pre-training objective of DiffPreT by optimizing the diffusion model above with the ELBO loss [37]:
$$
\mathcal {L} := \mathbb {E} \left[ \sum_ {t = 1} ^ {T} D _ {\mathrm {K L}} \left(q \left(\mathcal {P} ^ {t - 1} \mid \mathcal {P} ^ {t}, \mathcal {P} ^ {0}\right) | | p _ {\theta} \left(\mathcal {P} ^ {t - 1} \mid \mathcal {P} ^ {t}\right)\right) \right]. \tag {8}
$$
Under the assumptions in (2), it can be shown that the objective can be decomposed into a structure loss  $\mathcal{L}^{(\mathcal{R})}$  and a sequence loss  $\mathcal{L}^{(\mathcal{S})}$  (see proof in App. C.2):
$$
\mathcal {L} ^ {(\mathcal {R})} := \mathbb {E} \left[ \sum_ {t = 1} ^ {T} D _ {\mathrm {K L}} \left(q \left(\mathcal {R} ^ {t - 1} | \mathcal {R} ^ {t}, \mathcal {R} ^ {0}\right) | | p _ {\theta} \left(\mathcal {R} ^ {t - 1} | \mathcal {P} ^ {t}\right)\right) \right],
$$
$$
\mathcal {L} ^ {(S)} := \mathbb {E} \left[ \sum_ {t = 1} ^ {T} D _ {\mathrm {K L}} \left(q \left(\mathcal {S} ^ {t - 1} \mid \mathcal {S} ^ {t}, \mathcal {S} ^ {0}\right) \| p _ {\theta} \left(\mathcal {S} ^ {t - 1} \mid \mathcal {P} ^ {t}\right)\right) \right]. \tag {9}
$$
Both loss functions can be simplified as follows.
Structure loss  $\mathcal{L}^{(\mathcal{R})}$ . It has been shown in Ho et al. [37] that the loss function can be simplified under our parameterization by calculating KL divergence between Gaussians as weighted L2 distances between means  $\epsilon_{\theta}$  and  $\epsilon$  (see details in App. C.3):
$$
\mathcal {L} ^ {(\mathcal {R})} = \sum_ {t = 1} ^ {T} \gamma_ {t} \mathbb {E} _ {\epsilon \sim \mathcal {N} (0, I)} \left[ \| \epsilon - \epsilon_ {\theta} (\mathcal {P} ^ {t}, t) \| _ {2} ^ {2} \right], \tag {10}
$$
where the coefficients  $\gamma_{t}$  are determined by the variances  $\beta_{1},\ldots ,\beta_{t}$ . In practice, we follow Ho et al. [37] to set all weights  $\gamma_{t} = 1$  for the simplified loss  $\mathcal{L}_{\mathrm{simple}}^{(\mathcal{R})}$ .
Since  $\epsilon_{\theta}$  is designed to be rotation-equivariant w.r.t.  $\mathcal{R}^t$ , to make the loss function invariant w.r.t.  $\mathcal{R}^t$ , the supervision  $\epsilon$  is also supposed to achieve such equivariance. Therefore, we adopt the chain-rule approach proposed in Xu et al. [82], which decomposes the noise on pairwise distances to obtain the modified noise vector  $\hat{\epsilon}$  as supervision. We refer readers to Xu et al. [82] for more details.
Sequence loss  $\mathcal{L}^{(S)}$ . Since we parameterize  $p_{\theta}(\mathcal{S}^{t - 1}|\mathcal{P}^t)$  with  $\tilde{p}_{\theta}(\tilde{\mathcal{S}}^0 |\mathcal{P}^t)$  and  $q(\mathcal{S}^{t - 1}|\mathcal{S}^t,\tilde{\mathcal{S}}^0)$  as in (6), it can be proven that the  $t$ -th KL divergence term in  $\mathcal{L}^{(S)}$  reaches zero when  $\tilde{p}_{\theta}(\tilde{\mathcal{S}}^0 |\mathcal{P}^t)$  assigns
all mass on the ground truth  $S^0$  (see proof in App. C.4). Therefore, for pre-training, we can simplify the KL divergence to the cross-entropy between the correct residue type  $s_i^0$  and the prediction:
$$
\mathcal {L} _ {\text {s i m p l e}} ^ {(S)} = \sum_ {t = 1} ^ {T} \sum_ {i} \mathrm {C E} \left(s _ {i} ^ {0}, \tilde {p} _ {\theta} \left(s _ {i} ^ {0} \mid \mathcal {P} ^ {t}\right)\right), \tag {11}
$$
where  $\mathrm{CE}(\cdot ,\cdot)$  denotes the cross-entropy loss.
The ultimate training objective is the sum of simplified structure and sequence diffusion losses:
$$
\mathcal {L} _ {\text {s i m p l e}} = \mathcal {L} _ {\text {s i m p l e}} ^ {(\mathcal {R})} + \mathcal {L} _ {\text {s i m p l e}} ^ {(\mathcal {S})}. \tag {12}
$$
2.4 Two-Stage Noise Scheduling
Previous studies on scheduled denoising autoencoders on images [23] have shown that large noise levels encourage the learning of coarse-grained features, while small noise levels require the model to learn fine-grained features. We observe a similar phenomenon in structure diffusion, as depicted in Fig. 2. The diffusion loss with large noise at a fixed scale (orange) is smaller than that with small noise at a fixed scale (green), indicating the higher difficulty of structure diffusion with small noises. Interestingly, the opposite is observed in sequence diffusion, where the denoising accuracy with small noise (green) is higher than that with large noise (orange). This can be attributed to the joint diffusion effects on protein sequences and structures. The addition of large noise during joint diffusion significantly disrupts protein structures, making it more challenging to infer the correct protein sequences. For validation, we com
pare the loss of sequence-only diffusion pre-training (blue) with joint diffusion pre-training (red) in Fig. 2. Sequence diffusion achieves higher denoising accuracy than joint diffusion when using uncorrupted structures, supporting our hypothesis.
Figure 2: Structure diffusion loss and sequence denoising accuracy of pre-training with different noise scheduling strategies.
Unlike recent denoising pre-training methods that rely on a fixed noise scale as a hyperparameter [85], DiffPreT incorporates a denoising objective at various noise levels to capture both coarse- and fine-grained features and consider the joint diffusion effect explained earlier. Both granularities of features are crucial for downstream tasks, as they capture both small modifications for assessing protein structure quality and large changes leading to structural instability. In our implementation, we perform diffusion pre-training using  $T = 100$  noise levels (time steps). Following the intuition of learning coarse-grained features before fine-grained ones in curriculum learning [8], we divide the pre-training process into two stages: the first stage focuses on large noise levels ( $t = 10, \dots, 100$ ), while the second stage targets small noise levels ( $t = 1, \dots, 9$ ). As observed in Fig. 2, structure diffusion becomes more challenging and sequence diffusion becomes easier during the second stage, as we discussed earlier. However, even with this two-stage diffusion strategy, there remains a significant gap between the accuracy of sequence diffusion and joint diffusion. We hypothesize that employing protein encoders with larger capacities could help narrow this gap, which is left as our future works.
3 SiamDiff: Siamese Diffusion Trajectory Prediction
Through diffusion models on both protein sequences and structures, the pre-training approach proposed in Sec. 2 tries to make representations capture (1) the atom- and residue-level spatial interactions and (2) the statistical dependencies of residue types within a single protein. Nevertheless, no constraints or supervision have been added for modeling relations between different protein structures, especially different conformers of the same protein. Generated under different environmental factors, these conformers typically share the same protein sequence but different structures due to side chain rotation driving conformational variations. These conformers' properties are highly correlated [58], and their representations should reflect this correlation.
In this section, we introduce Siamese Diffusion Trajectory Prediction (SiamDiff), which incorporates conformer-related information into DiffPreT by maximizing mutual information (MI) between diffusion trajectory representations of correlated conformers. We propose a scheme to generate simulated conformers (Sec. 3.1), generate joint diffusion trajectories (Sec. 3.2), and transform MI maximization into mutual denoising between trajectories (Sec. 3.3), sharing similar loss with DiffPreT.
3.1 Conformer Simulation Scheme