SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 12: Comparison between SiamDiff with random torsional perturbation and sampling from a rotamer library.
Method	PIP	MSP	RES	PSR
AUROC	AUROC	Accuracy	Global ρ	Mean ρ
SiamDiff	0.884	0.698	0.460	0.829	0.546
w/ rotamer library	0.877	0.631	0.449	0.834	0.521
Experimental results. The results are shown in Table 10. Among all pre-training methods, SiamDiff, angle prediction and dihedral prediction are the top three. This is different from what we observe in Table 1, where residue type and distance prediction are more competitive baselines. We hypothesize that this is because GearNet-Edge includes angle information in the encoder while GVP does not. Therefore, GVP will benefit more from pre-training methods with supervision on angles. Nevertheless, we find that SiamDiff is the only method that performs well on different backbone models.
I Results on Protein Engineering Task
To further prove the effectiveness of our proposed pre-training methods, we add experiments on the GB1 dataset from FLIP [16]. As this is a protein engineering task with mutated sequences, we assume that the backbone structure remains unchanged after mutation, to save costs in generating mutant structures. We only keep CA atoms in the wild type protein structure as the input to the encoder. We benchmark residue-level methods in Table 11 in the attached file, alongside CNN and ESM-1b baselines from the FLIP paper.
According to Table 11, we observe that modeling structural information is beneficial compared with using only sequential information, even under the assumption that all mutants share the same backbone structure. Among all pre-training methods, SiamDiff demonstrates the most significant improvements over the baseline, once again validating the effectiveness of our method.
J Effects of Approximate Conformer Generation
In SiamDiff, we introduce approximate conformer generation mechanism by randomly rotating torsional angles. We hypothesize that highly realistic conformers are not vital for better representations. To confirm, an extra rebuttal experiment is performed in Table 12. Instead of random perturbation, we sample from a rotamer library [62] based on residue types and backbone angles. Table 12 shows random torsional perturbation still outperforms sampling from a rotamer library in most tasks, confirming our hypothesis. This can be attributed to the fact that the objective of pre-training is to learn common information between diverse views through mutual prediction, as SimCLR and SimSiam. Considering this perspective, introducing random torsional noise allows us to generate more diverse conformers compared to solely relying on realistic conformer distributions.
(A) GearNet-Edge representations with random initialization
(B) GearNet-Edge representations pre-trained with first-stage SiamDiff
(C) GearNet-Edge representations pre-trained with two-stage SiamDiff
Figure 4: Visualization of GearNet-Edge representations on 4 randomly selected proteins from AlphaFold Database with sequence and structure noises at different levels. Three models are employed for extracting representations: (A) without pre-training, (B) with first-stage SiamDiff, and (C) with two-stage SiamDiff pre-training. For each protein, we consider  $t = 1,\dots,100$  noise levels and randomly generate 32 noisy samples for each noise level using the scheme in SiamDiff. Samples with small noise levels are highlighted in blue, while those with large noise levels are marked in red.
K Visualization of SiamDiff Embeddings upon Noises
To explore pre-training insights, we visualize UMAP representations of 4 random AlphaFold DB proteins in Fig. 4. Several interesting phenomena can be observed:
- Randomly initiated representations in Fig. 4A form a clear, continuously color-changing trajectory (blue to red). This confirms that the forward diffusion process gradually adds noise to proteins, leading to smooth changes in their representations, as expected for diffusion models.
- After pre-training with large noise scales, the encoder maintains the color smoothness of the trajectory, which is desired for effective denoising during the backward diffusion process. Intriguingly, pre-training narrows the trajectory compared to the broader trajectory without pre-training, particularly at the two ends. This suggests that first-stage pre-training clusters proteins with similar levels of added noise, even for large and diverse noises. This clustering property proves useful for detecting large perturbations in downstream tasks, such as mutation stability prediction in MSP, as opposed to the diverse representation distributions in Fig. 4A.
- Continuing with small noise scale pre-training, the trajectory becomes much narrower in the middle and even breaks for some proteins. This indicates that by focusing on only slightly perturbed samples during pre-training, our model becomes capable of discerning proteins with small and large noises, making it more effective for fine-grained downstream tasks like PSR and
PIP. However, the red end of the trajectory is thicker than that in Fig. 4B, which may imply some forgetting behavior in the second-stage pre-training.