SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. First, we prove  $q(\mathcal{R}_a^t |\mathcal{R}_a^0) = \mathcal{N}(\mathcal{R}_a^t;\sqrt{\bar{\alpha}_t}\mathcal{R}_a^0,(1 - \bar{\alpha}_t)I)$ . Let  $\epsilon_{i}$  be the standard Gaussian random variable at time step  $i$ . Then, we have
$$
\begin{array}{l} \mathcal {R} _ {a} ^ {t} = \sqrt {\alpha_ {t}} \mathcal {R} _ {a} ^ {t - 1} + \sqrt {\beta_ {t}} \epsilon_ {t} \\ = \sqrt {\alpha_ {t - 1} \alpha_ {t}} \mathcal {R} _ {a} ^ {t - 2} + \sqrt {\alpha_ {t - 1} \beta_ {t - 1}} \epsilon_ {t - 1} + \sqrt {\beta_ {t}} \epsilon_ {t} \\ = \dots \\ = \sqrt {\bar {\alpha} _ {t}} \mathcal {R} _ {a} ^ {0} + \sqrt {\alpha_ {t} \alpha_ {t - 1} . . . \alpha_ {2} \beta_ {1}} \epsilon_ {1} + \dots + \sqrt {\alpha_ {t - 1} \beta_ {t - 1}} \epsilon_ {t - 1} + \sqrt {\beta_ {t}} \epsilon_ {t}, \\ \end{array}
$$
which suggests that the mean of  $\mathcal{R}_a^t$  is  $\sqrt{\bar{\alpha}_t}\mathcal{R}_a^0$  and the variance matrix is  $(\alpha_{t}\alpha_{t - 1}\dots \alpha_{2}\beta_{1} + \dots +$ $\alpha_{t - 1}\beta_{t - 1} + \beta_t)I = (1 - \bar{\alpha})I.$
Next, we derive the posterior distribution as:
$$
\begin{array}{l} q (\mathcal {R} _ {a} ^ {t - 1} | \mathcal {R} _ {a} ^ {t}, \mathcal {R} _ {a} ^ {0}) = \frac {q (\mathcal {R} _ {a} ^ {t} | \mathcal {R} _ {a} ^ {t - 1}) q (\mathcal {R} _ {a} ^ {t - 1} | \mathcal {R} _ {a} ^ {0})}{q (\mathcal {R} _ {a} ^ {t} | \mathcal {R} _ {a} ^ {0})} \\ = \frac {\mathcal {N} (\mathcal {R} _ {a} ^ {t} ; \sqrt {\alpha_ {t}} \mathcal {R} _ {a} ^ {t - 1} , \beta_ {t} I) \cdot \mathcal {N} (\mathcal {R} _ {a} ^ {t - 1} ; \sqrt {\bar {\alpha} _ {t - 1}} \mathcal {R} _ {a} ^ {0} , (1 - \bar {\alpha} _ {t - 1}) I)}{\mathcal {N} (\mathcal {R} _ {a} ^ {t} ; \sqrt {\bar {\alpha} _ {t}} \mathcal {R} _ {a} ^ {0} , (1 - \bar {\alpha} _ {t}) I)} \\ = \mathcal {N} (\mathcal {R} _ {a} ^ {t - 1}; \frac {\sqrt {\bar {\alpha} _ {t - 1}} \beta_ {t}}{1 - \bar {\alpha} _ {t}} \mathcal {R} _ {a} ^ {0} + \frac {\sqrt {\alpha_ {t}} (1 - \bar {\alpha} _ {t - 1})}{1 - \bar {\alpha} _ {t}} \mathcal {R} _ {a} ^ {t}, \frac {1 - \bar {\alpha} _ {t - 1}}{1 - \bar {\alpha} _ {t}} \beta_ {t} I). \\ \end{array}
$$
Let  $\tilde{\beta}_t = \frac{1 - \bar{\alpha}_{t-1}}{1 - \bar{\alpha}_t} \beta_t$ , then the  $t$ -th KL divergence term can be written as:
$$
\begin{array}{l} D _ {\mathrm {K L}} \left(q \left(\mathcal {R} _ {a} ^ {t - 1} \mid \mathcal {R} _ {a} ^ {t}, \mathcal {R} _ {a} ^ {0}\right) | | p _ {\theta} \left(\mathcal {R} _ {a} ^ {t - 1} \mid \mathcal {P} _ {a} ^ {t}, \boldsymbol {\mathcal {P}} _ {b} ^ {t}\right)\right) \\ = \frac {1}{2 \tilde {\beta} _ {t}} \left\| \frac {\sqrt {\bar {\alpha} _ {t - 1}} \beta_ {t}}{1 - \bar {\alpha} _ {t}} \mathcal {R} _ {a} ^ {0} + \frac {\sqrt {\alpha_ {t}} (1 - \bar {\alpha} _ {t - 1})}{1 - \bar {\alpha} _ {t}} \mathcal {R} _ {a} ^ {t} - \frac {1}{\sqrt {\alpha_ {t}}} \left(\mathcal {R} _ {a} ^ {t} - \frac {\beta_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} \epsilon_ {\theta} (\mathcal {P} _ {a} ^ {t}, \mathcal {P} _ {b} ^ {t}, t)\right) \right\| ^ {2} \\ = \frac {1}{2 \tilde {\beta} _ {t}} \mathbb {E} _ {\epsilon} \left\| \frac {\sqrt {\bar {\alpha} _ {t - 1}} \beta_ {t}}{1 - \bar {\alpha} _ {t}} \cdot \frac {\mathcal {R} _ {a} ^ {t} - \sqrt {1 - \bar {\alpha} _ {t}} \epsilon}{\sqrt {\bar {\alpha} _ {t}}} + \frac {\sqrt {\alpha_ {t}} (1 - \bar {\alpha} _ {t - 1})}{1 - \bar {\alpha} _ {t}} \mathcal {R} _ {a} ^ {t} - \frac {1}{\sqrt {\alpha_ {t}}} \left(\mathcal {R} _ {a} ^ {t} - \frac {\beta_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} \epsilon_ {\theta} (\mathcal {P} _ {a} ^ {t}, \mathcal {P} _ {b} ^ {t}, t)\right) \right\| ^ {2} \\ = \frac {1}{2 \tilde {\beta} _ {t}} \cdot \frac {\beta_ {t} ^ {2}}{\alpha_ {t} (1 - \bar {\alpha} _ {t})} \mathbb {E} _ {\epsilon} \left\| \epsilon - \epsilon_ {\theta} \left(\mathcal {P} _ {a} ^ {t}, \mathcal {P} _ {b} ^ {t}, t\right) \right\| \\ = \gamma_ {t} \mathbb {E} _ {\epsilon} \left[ \| \epsilon - \epsilon_ {\theta} \left(\mathcal {P} _ {a} ^ {t}, \mathcal {P} _ {b} ^ {t}, t\right) \| _ {2} ^ {2} \right], \\ \end{array}
$$
C.4 Proof of Simplified Sequence Loss
Now we show the equivalence of optimizing sequence loss  $\mathcal{L}^{(\mathcal{S},b\to a)}$  and the masked residue type prediction problem on  $S_{a}^{0}$ .
Proposition 4 Given the definition of reverse process on protein sequences
$$
p _ {\theta} \left(\mathcal {S} _ {a} ^ {t - 1} | \mathcal {P} _ {a} ^ {t}, \mathcal {P} _ {b} ^ {t}\right) \propto \sum_ {\tilde {\mathcal {S}} _ {a} ^ {0}} q \left(\mathcal {S} _ {a} ^ {t - 1} | \mathcal {S} _ {a} ^ {t}, \tilde {\mathcal {S}} _ {a} ^ {0}\right) \cdot \tilde {p} _ {\theta} \left(\tilde {\mathcal {S}} _ {a} ^ {0} | \mathcal {P} _ {a} ^ {t}, \mathcal {P} _ {b} ^ {t}\right), \tag {28}
$$
the sequence loss  $\mathcal{L}^{(\mathcal{S},b\to a)}$  reaches zero when  $\tilde{p}_{\theta}(\tilde{S}_a^0 |\mathcal{P}_a^t,\mathcal{P}_b^t)$  puts all mass on the ground truth  $S_{a}^{0}$ .
Proof. The loss function can be written as:
$$
\begin{array}{l} \mathcal {L} ^ {\left(\mathcal {S}, b \rightarrow a\right)} := \mathbb {E} \left[ \sum_ {t = 1} ^ {T} D _ {\mathrm {K L}} \left(q \left(\mathcal {S} _ {a} ^ {t - 1} | \mathcal {S} _ {a} ^ {t}, \mathcal {S} _ {a} ^ {0}\right) | | p _ {\theta} \left(\mathcal {S} _ {a} ^ {t - 1} | \mathcal {P} _ {a} ^ {t}, \mathcal {P} _ {b} ^ {t}\right)\right)\right] \\ = \mathbb {E} \left[ \sum_ {t = 1} ^ {T} D _ {\mathrm {K L}} \left(q \left(\mathcal {S} _ {a} ^ {t - 1} \mid \mathcal {S} _ {a} ^ {t}, \mathcal {S} _ {a} ^ {0}\right) \Bigg | \Bigg | \frac {\sum \bar {\mathcal {S}} _ {a} ^ {0} q \left(\mathcal {S} _ {a} ^ {t - 1} \mid \mathcal {S} _ {a} ^ {t} , \tilde {\mathcal {S}} _ {a} ^ {0}\right) \cdot \tilde {p} _ {\theta} \left(\tilde {\mathcal {S}} _ {a} ^ {0} \mid \mathcal {P} _ {1} ^ {t} , \mathcal {P} _ {2} ^ {t}\right)}{Z}\right) \right], \\ \end{array}
$$
where  $Z$  is the normalization constant. Hence, when  $\tilde{p}_{\theta}(\tilde{\mathcal{S}}_a^0|\mathcal{P}_a^t,\mathcal{P}_b^t)$  puts all mass on the ground truth  $S_{a}^{0}$ , the distribution  $p_{\theta}(S_{a}^{t - 1}|\mathcal{P}_{a}^{t},\mathcal{P}_{b}^{t})$  will be identical with  $q(S_{a}^{t - 1}|S_{a}^{t},S_{a}^{0})$ , which makes the KL divergence become zero.
In this section, we introduce the details of our experiments. All these methods are developed based on PyTorch and TorchDrug [88].
Downstream benchmark tasks. For downstream evaluation, we adopt the EC prediction task [24] and four ATOM3D tasks [68].
1. Enzyme Commission (EC) number prediction task aims to predict EC numbers of proteins which describe their catalysis behavior in biochemical reactions. This task is formalized as 538 binary classification problems. We adopt the dataset splits from Gligorijevic et al. [24] and use the test split with  $95\%$  sequence identity cutoff following Zhang et al. [87].
2. Protein Interface Prediction (PIP) requires the model to predict whether two amino acids from two proteins come into contact when the proteins bind (binary classification). The protein complexes of this benchmark are split with  $30\%$  sequence identity cutoff.
3. Mutation Stability Prediction (MSP) task seeks to predict whether a mutation will increase the stability of a protein complex or not (binary classification). The benchmark dataset is split upon a  $30\%$  sequence identity cutoff among different splits.
4. Residue Identity (RES) task studies the structural role of an amino acid under its local environment. A model predicts the type of the center amino acid based on its surrounding atomic structure. The environments in different splits are with different protein topology classes.
5. Protein Structure Ranking (PSR) predicts global distance test scores of structure predictions submitted to the Critical Assessment of Structure Prediction (CASP) [47] competition. This dataset is split according to the competition year.
Graph construction. For atom graphs, we connect atoms with Euclidean distance lower than a distance threshold. For PSR and MSP tasks, we remove all hydrogen atoms following Jing et al. [42]. For residue graphs, we discard all non-alpha-carbon atoms and add three different types of directed edges: sequential edges, radius edges and K-nearest neighbor edges. For sequential edges, two atoms are connected if their sequential distance is below a threshold and these edges are divided into different types according to these distances. For two kinds of spatial edges, we connect atoms according to Euclidean distance and k-nearest neighbors. We further apply a long range interaction filter that removes edges with low sequential distances. We refer readers to Zhang et al. [87] for more details.
Atom-level backbone models. To adapt GearNet-Edge to atom-level structures with moderate computational cost, we construct the atom graph by using only the spatial edge with the radius
$d_{\mathrm{radius}} = 4.5\mathring{\mathrm{A}}$ . We concatenate one-hot features of atom types and residue types as node features and concatenate (1) one-hot features of residue types of end nodes, (2) one-hot features of edge types, (3) one-hot features of sequential distance, (4) spatial distance as edge features. The whole model is composed of 6 message passing layers with 128 hidden dimensions and ReLU activation function. For edge message passing, we employ the discretized angles to determine the edge types on the line graph. The final prediction is performed upon the hidden representation concatenated across all layers.
Residue-level backbone models. We directly borrow the best hyperparameters reported in the original paper of GearNet-Edge [87]. We adopt the same configuration of relational graph construction, i.e., the sequential distance threshold  $d_{\mathrm{seq}} = 3$ , the radius  $d_{\mathrm{radius}} = 10.0\AA$ , the number of neighbors  $k = 10$  and the long range interaction cutoff  $d_{\mathrm{long}} = 5$ . We use one-hot features of residue types as node features and concatenate (1) features of end nodes, (2) one-hot features of edge types, (3) one-hot features of sequential distance, (4) spatial distance as edge features. Then we use 6 message passing layers with 512 hidden dimensions and ReLU as the activation function. For edge message passing, the edge types on the line graph are determined by the discretized angles. The hidden representations in each layer of GearNet will be concatenated for the final prediction.
Baseline pre-training methods. Here we briefly introduce the considered baselines. Multiview Contrast aims to maximize the mutual information between correlated views, which are extracted by randomly chosen augmentation functions to capture protein sub-structures. Residue type, distance, angle and dihedral prediction masks single residues, single edges, edge pairs and edge triplets, respectively, and then predict the corresponding properties. Denoising score matching performs denoising on noised pairwise distance matrices based on the learnt representations.
For all baselines in [87], we adopt the original configurations. For Multiview Contrast, we use subsequence cropping that randomly extracts protein subsequences with no more than 50 residues and space cropping that takes all residues within a  $15\AA$  Euclidean ball with a random center residue. Then, either an identity function or a random edge masking function with mask rate equal to 0.15 is applied for constructing views. The temperature  $\tau$  in the InfoNCE loss function is set as 0.07. We set the number of sampled items in each protein as 256 for Distance Prediction and as 512 for Angle and Dihedral Prediction. The mask rate for Residue Type Prediction is set as 0.15. When masking a residue on atom graphs, we discard all non-backbone atoms and set the residue features as zero. Since the backbone models and tasks in our paper are quite different with those in Guo et al. [29], we re-implement the method on our codebase. We consider 50 different noise levels log-linearly ranging from 0.01 to 10.0.
In DiffPreT, for structure diffusion, we use a sigmoid schedule for variances  $\beta_{t}$  with the lowest variance  $\beta_{1} = 1e - 4$  and the highest variance  $\beta_{T} = 0.1$ . For sequence diffusion, we simply set the cumulative transition probability to [MASK] over time steps as a linear interpolation between minimum mask rate 0.15 and maximum mask rate 1.0. The number of diffusion steps is set as 100. In SiamDiff, we adopt the same hyperparameters for multimodal diffusion models. We set the variance of torsional perturbation noises as  $0.1\pi$  on the atom level and that of Gaussian perturbation noises as 0.3 on the residue level when constructing the correlated conformer.
All other optimization configurations for these pre-training methods are reported in Table 5. All methods are pre-trained on 4 Tesla A100 GPUs and Table 5 reports the batch sizes on each GPU.
Fine-tuning on downstream tasks. For all models on all downstream tasks, we apply a three-layer MLP head for prediction, the hidden dimension of which is set to the dimension of model outputs. The number of used gpus and batch sizes for each model are chosen according the memory limit. All residue-level tasks are run on 4 V100 GPUs while all atom-level tasks are run on A100 GPUs.
Evaluation metrics. We clarify the definitions of  $\mathrm{F}_{\mathrm{max}}$  (used in EC), global Spearman's  $\rho$  (used in PSR) and mean Spearman's  $\rho$  (used in PSR) as below:
-  $\mathbf{F}_{\mathrm{max}}$  denotes the protein-centric maximum F-score. It first computes the precision and recall for each protein at a decision threshold  $t \in [0,1]$ :
$$
\operatorname {p r e c i s i o n} _ {i} (t) = \frac {\sum_ {f} \mathbb {1} [ f \in P _ {i} (t) \cap T _ {i} ]}{\sum_ {f} \mathbb {1} [ f \in P _ {i} (t) ]}, \quad \operatorname {r e c a l l} _ {i} (t) = \frac {\sum_ {f} \mathbb {1} [ f \in P _ {i} (t) \cap T _ {i} ]}{\sum_ {f} \mathbb {1} [ f \in T _ {i} ]}, \tag {29}
$$
Table 5: Optimization configurations for pre-training methods. Here max length denotes the maximum number of residues kept in each protein and lr stands for learning rate.
Method	Max length	Batch size	Optimizer	lr
residue	atom	residue	atom
Residue Type Prediction	100	100	96	64	Adam	1e-3
Distance Prediction	100	100	128	64	Adam	1e-3
Angle Prediction	100	100	96	64	Adam	1e-3
Dihedral Prediction	100	100	96	64	Adam	1e-3
Multiview Contrast	-	-	96	64	Adam	1e-3
Denoising Score Matching	200	200	12	12	Adam	1e-4
DiffPreT	150	100	16	64	Adam	1e-4
SiamDiff	150	100	16	32	Adam	1e-4
Table 6: Optimization configurations for downstream evaluations. Here max length denotes the maximum number of residues kept in each protein and lr stands for learning rate.
Task	# GPUS	Batch size	Optimizer	lr
residue	atom	residue	atom
EC	4	N/A	2	N/A	Adam	1e-4
PIP	N/A	1	N/A	8	Adam	1e-4
MSP	4	1	1	8	Adam	1e-4
RES	N/A	4	N/A	64	Adam	1e-4
PSR	4	1	8	8	Adam	1e-4
where  $f$  denotes a functional term in the ontology,  $T_{i}$  is the set of experimentally determined functions for protein  $i$ ,  $P_{i}(t)$  is the set of predicted functions for protein  $i$  whose scores are greater or equal to  $t$ , and  $\mathbb{1}[\cdot]$  represents the indicator function. After that, the precision and recall are averaged over all proteins:
$$
\operatorname {p r e c i s i o n} (t) = \frac {1}{M (t)} \sum_ {i} \operatorname {p r e c i s i o n} _ {i} (t), \quad \operatorname {r e c a l l} (t) = \frac {1}{N} \sum_ {i} \operatorname {r e c a l l} _ {i} (t), \tag {30}
$$
where  $N$  denotes the total number of proteins, and  $M(t)$  denotes the number of proteins which contain at least one prediction above the threshold  $t$ , i.e.,  $|P_{i}(t)| > 0$ .
Based on these two metrics, the  $\mathrm{F}_{\mathrm{max}}$  score is defined as the maximum value of F-measure over all thresholds:
$$
F _ {\max } = \max  _ {t} \left\{\frac {2 \cdot \operatorname {p r e c i s i o n} (t) \cdot \operatorname {r e c a l l} (t)}{\operatorname {p r e c i s i o n} (t) + \operatorname {r e c a l l} (t)} \right\}. \tag {31}
$$
- Global Spearman's  $\rho$  for PSR measures the correlation between the predicted global distance test (GDT_TS) score and the ground truth. It computes the Spearman's  $\rho$  between the prediction and the ground truth over all test proteins without considering the different biopolymers that these proteins lie in.
- Mean Spearman's  $\rho$  for PSR also measures the correlation between GDT_TS predictions and the ground truth. However, it first splits all test proteins into multiple groups based on their corresponding biopolymers, then computes the Spearman's  $\rho$  within each group, and finally reports the mean Spearman's  $\rho$  over all groups.
E Results of Pre-Training on Different Sizes of Datasets
In the main paper, we followed the setting in Zhang et al. [87] and used AlphaFold Database v1 as our pre-training dataset for fair comparison. Here, we investigate the impact of pre-training on different dataset sizes. Since previous work by Zhang et al. [87] showed minimal differences between using experimental or predicted structures, we conduct experiments on the AlphaFold Database in this section and do not use PDB as our pre-training dataset. We utilize the preprocessed clustered
Table 7: Atom-level results of pre-training on different sizes of datasets on Atom3D tasks.
Pre-training Dataset	Size	PSR	MSP	PIP	RES
Global ρ	Mean ρ	AUROC	AUROC	Acc.
AlphaFold DB v1	365K	0.829	0.546	0.698	0.884	0.460
Clustered AlphaFold DB	10K	0.816	0.501	0.586	0.880	0.444
Clustered AlphaFold DB	50K	0.797	0.498	0.648	0.879	0.450
Clustered AlphaFold DB	100K	0.805	0.540	0.685	0.882	0.454
Clustered AlphaFold DB	500K	0.848	0.537	0.599	0.880	0.459
Clustered AlphaFold DB	2.2M	0.840	0.560	0.700	0.882	0.460
AlphaFold Database provided in [6], which includes 2.2M non-singleton clusters with an average of 13.2 proteins per cluster and an average pLDDT of 71.59. For each cluster, we use the representative structure from [6]. To explore the effects of dataset size, we pre-train our encoder using 10K, 50K, 100K, 500K, and 2.2M clusters in the database. The results, shown in Table 7, reveal a general trend of increased performance with larger datasets. However, for certain tasks like MSP, the performance does not consistently improve, possibly due to the limited size and variability of the downstream datasets. Overall, scaling the model to the entire AlphaFold Database holds promise for achieving performance gains.
F Results of Multimodal Pre-Training Baselines
Table 8: Atom-level results of multimodal pre-training baselines on Atom3D tasks.
Method	PSR	MSP	PIP	RES
Global ρ	Mean ρ	AUROC	AUROC	Acc.
GearNet-Edge	0.782	0.488	0.633	0.868	0.441
w/ pre-training	Residue Type Prediction	0.826	0.518	0.620	0.879	0.449
Residue Type & Distance Prediction	0.817	0.518	0.665	0.873	0.402
Residue Type & Angle Prediction	0.837	0.524	0.642	0.878	0.415
Residue Type & Dihedral Prediction	0.823	0.494	0.597	0.871	0.414
DiffPreT	0.821	0.533	0.680	0.880	0.452
SiamDiff	0.829	0.546	0.698	0.884	0.460
In Sec. 5, we include all pre-training baselines from existing works, which solely focus on unimodal pre-training objectives and overlook the joint distribution of sequences and structures. In this section, we introduce three additional pre-training baselines that leverage both sequence and structure information. These baselines combine residue type prediction with distance/angle/dihedral prediction objectives. Following established settings, we pre-train the encoder and present the results in Table 8. While the Residue Type & Angle Prediction method achieves higher performance on PSR, there are still substantial gaps compared to our DiffPreT and SiamDiff across other tasks. Notably, the introduction of geometric property prediction tasks leads to a drop in performance on RES, indicating that a simple combination of pre-training objectives diminishes the benefits of each individual objective. Once again, these findings underscore the effectiveness of our methods in modeling the joint distribution of protein sequences and structures.
G Results of Different Diffusion Models for Pre-Training
In Sec. 2, we consider joint diffusion models on protein sequences and structures for pre-training. In this section, we explore the performance of different diffusion models when applied for pre-training. The results are shown in Table 9.
First, we simply run diffusion models on protein sequences and structures for pre-training, both of which achieve improvement compared with the baseline GearNet-Edge. By combining two diffusion models, DiffPreT can achieve better performance on PIP and PSR. This advantage will be further
Table 9: Results of different diffusion models for pre-training on atom-level Atom3D tasks.
Method	PIP	RES	PSR
AUROC	Accuracy	Global ρ	Mean ρ
GearNet-Edge	0.868	0.441	0.782	0.488
w/ pre-training	DiffPreT	0.880	0.452	0.821	0.533
SiamDiff	0.884	0.460	0.829	0.546
Sequence Diffusion	0.879	0.456	0.802	0.508
Structure Diffusion	0.877	0.448	0.813	0.518
Torsional Diffusion	0.877	0.442	0.819	0.505
increased after using siamese diffusion trajectory prediction as shown in Table 3. It can be observed that sequence diffusion achieves better performance than DiffPreT due to the consistency of objectives between pre-training and RES tasks.
Besides, we also consider diffusion models on torsional angles of protein side chains for pre-training. This method has shown its potential in the protein-ligand docking task [15]. For each residue, we randomly select one side-chain torsional angle and add some noises drawn from wrapped Gaussian distribution during the diffusion process. Then, we predict the added noises with the extracted atom representations corresponding to the torsional angle. In Table 9, we can see clear improvements on PIP and PSR tasks compared with GearNet-Edge. This suggests that it would be a promising direction to explore more different diffusion models for pre-training, e.g., diffusion models on backbone dihedral angles.
H Results of Pre-Training GVP
Table 10: Atom-level results of GVP on Atom3D tasks. Accuracy is abbreviated as Acc.
Method	PSR	MSP	PIP	RES	Mean Rank
Global ρ	Mean ρ	AUROC	AUROC	Acc.
GVP	0.809	0.486	0.610	0.846	0.481	8.6
w/ pre-training	Denoising Score Matching	0.849	0.535	0.625	0.851	0.529	5.4
Residue Type Prediction	0.845	0.527	0.652	0.847	0.518	5.8
Distance Prediction	0.825	0.513	0.632	0.836	0.483	7.6
Angle Prediction	0.872	0.545	0.637	0.881	0.557	2.0
Dihedral Prediction	0.852	0.538	0.677	0.881	0.555	2.2
Multiview Contrast	0.848	0.518	0.656	0.833	0.490	6.4
DiffPreT	0.850	0.540	0.631	0.851	0.542	4.4
SiamDiff	0.854	0.548	0.673	0.863	0.554	2.2
To study the effect of our proposed pre-training methods on different backbone models, we show the pre-training results on GVP [41, 42] in this section.
Setup. GVP constructs atom graphs and adds vector channels for modeling equivariant features. The original design only includes atom types as node features, which makes pre-training tasks with residue type prediction very difficult to learn. To address this issue, we slightly modify its architecture to add the embedding of atom and corresponding residue types as atom features. Then, the default configurations in Jing et al. [42] are adopted. We construct an atom graph for each protein by drawing edges between atoms closer than  $4.5\mathring{\mathrm{A}}$ . Each edge is featured with a 16-dimensional Gaussian RBF encoding of its Euclidean distance. We use five GVP layers and hidden representations with 16 vector and 100 scalar channels and use ReLU and identity for scalar and vector activation functions, respectively. The dropout rate is set as 0.1. The final atom representations are followed by two mean pooling layers to obtain residue and protein representations, respectively. All other hyperparameters for pre-training and downstream tasks are the same as those in App. D.
Table 11: Spearman correlation on GB1 with the 2-vs-rest split.
Method	Spearmanr ρ
CNN	0.320
ESM-1b	0.550
GearNet-Edge	0.651
w/ Multiview Contrast	0.714
w/ Residue Type Prediction	0.697
w/ Distance Prediction	0.685
w/ Angle Prediction	0.686
w/ Dihedral Prediction	0.661
w/ SiamDiff	0.747