SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Particularly for multi-point mutations, DiffAffinity notably outperforms all other methods, suggesting that our model effectively captures the crucial relationship between mutational effects and side-chain conformations, regardless of the number of mutations. DiffAffinity surpasses the performance of the RDE-Network and ESM2*, indicating that the latent representations derived from our diffusion probabilistic model, SidechainDiff, are more beneficial than those from flow-based generative models and other self-supervised pre-training representations with the same network architecture for the same task. More analysis of the three pre-training representations can be found in App C.2.
Furthermore, DiffAffinity-Linear shows that even a straightforward linear projection of latent representations can achieve performance comparable to FoldX and ESM-IF and outperform ESM-1v. The reduced performance observed with DiffAffinity*, when the SidechainDiff encoder is absent, further emphasizes the effectiveness of the latent representations extracted from SidechainDiff.
In subsequent sections (Section 4.2 and 4.3), we employ the three models trained on different splits of the SKEMPI2 dataset without any fine-tuning on downstream tasks related to predicting binding affinity. This approach allows us to assess our model's performance in diverse tasks.
Table 1: Evaluation of prediction on the SKEMPI2 dataset
Method	Mutations	Overall	Per-Structure
Pearson	Spearman	RMSE	MAE	AUROC	AUPRC	Pearson	Spearman
FoldX	all	0.319	0.416	1.959	1.357	0.671	0.839	0.376	0.375
single	0.315	0.361	1.651	1.146	0.657	0.839	0.382	0.360
multiple	0.256	0.418	2.608	1.926	0.704	0.841	0.333	0.340
Rosetta	all	0.311	0.346	1.617	1.131	0.656	0.810	0.328	0.298
single	0.325	0.367	1.183	0.987	0.674	0.834	0.351	0.418
multiple	0.199	0.230	2.658	2.024	0.621	0.798	0.191	0.083
flex ddG	all	0.402	0.427	1.587	1.102	0.675	0.866	0.414	0.386
single	0.425	0.431	1.457	0.997	0.677	0.874	0.433	0.435
multiple	0.398	0.419	1.765	1.326	0.669	0.854	0.401	0.363
ESM-1v	all	0.192	0.157	1.961	1.368	0.541	0.735	0.007	-0.012
single	0.191	0.157	1.723	1.192	0.549	0.770	0.042	0.027
multiple	0.192	0.175	2.759	2.119	0.542	0.678	-0.060	-0.128
ESM-IF	all	0.319	0.281	1.886	1.286	0.590	0.768	0.224	0.202
single	0.296	0.287	1.673	1.137	0.605	0.776	0.391	0.364
multiple	0.326	0.335	2.645	1.956	0.637	0.754	0.202	0.149
ESM2	all	0.133	0.138	2.048	1.460	0.547	0.738	0.044	0.039
single	0.100	0.120	1.730	1.210	0.541	0.734	0.019	0.036
multiple	0.170	0.163	2.658	2.021	0.566	0.746	0.010	0.010
ESM2*	all	0.623	0.498	1.615	1.179	0.721	0.887	0.362	0.316
single	0.625	0.468	1.357	0.986	0.707	0.879	0.391	0.342
multiple	0.603	0.529	2.15	1.67	0.758	0.909	0.333	0.304
DDGPred	all	0.630	0.400	1.313	0.995	0.696	0.892	0.356	0.321
single	0.652	0.359	1.309	0.936	0.656	0.884	0.351	0.318
multiple	0.591	0.503	2.181	1.670	0.759	0.913	0.373	0.385
RDE-Net	all	0.632	0.527	1.601	1.142	0.731	0.887	0.415	0.376
single	0.637	0.491	1.341	0.961	0.720	0.885	0.413	0.385
multiple	0.601	0.567	2.157	1.631	0.768	0.898	0.390	0.360
DA-Linear1	all	0.326	0.305	1.954	1.399	0.642	0.857	0.222	0.222
single	0.318	0.293	1.649	1.175	0.651	0.854	0.209	0.202
multiple	0.277	0.288	2.593	1.961	0.629	0.867	0.193	0.195
DiffAffinity*single	all	0.646	0.538	1.578	1.113	0.742	0.741	0.415	0.392
single	0.657	0.523	1.312	0.931	0.742	0.741	0.417	0.396
multiple	0.613	0.542	2.133	1.606	0.750	0.750	0.407	0.379
DiffAffinity	all	0.669	0.556	1.535	1.093	0.744	0.896	0.422	0.397
single	0.672	0.523	1.288	0.923	0.733	0.887	0.429	0.409
multiple	0.650	0.602	2.051	1.540	0.784	0.921	0.414	0.387
<sup>1</sup> DA-Linear is short of DiffAffinity-Linear just to save space in the table.
4.2 Prediction of mutational effects on binding affinity of SARS-CoV-2 RBD
SARS-CoV-2 initiates infection by binding to ACE2 protein on host cells via the viral spike protein. The receptor binding domain (RBD) of the spike proteins exhibits high-affinity binding to ACE2.
The technique of deep mutational scanning has been employed to experimentally quantify the effects of all single-point mutations on the binding affinity of the ancestral Wuhan-Hu-1 RBD (PDB ID: 6M0J) [Starr et al., 2022]. These measurements have guided the survey of SARS-CoV-2 evolution and identified several significant mutation sites on RBD that substantially contribute to the binding affinity. A total of 15 significant mutation sites like NE501<sup>3</sup> have been identified (see App. A.5). We predicted all 285 possible single-point mutations for these sites and calculated the Pearson correlation coefficient between the experimental and predicted  $\Delta \Delta G$ .
Our results show that DiffAffinity outperforms all baseline methods (Table 2), highlighting its potential to facilitate biologists in understanding the evolution of SARS-CoV-2. Furthermore, we note that DiffAffinity exhibits a substantial improvement over DiffAffinity*, indicating the effectiveness of the learned representations from SidechainDiff.
4.3 Optimization of human antibodies against SARS-CoV-2
The receptor-binding domain (RBD) of the SARS-CoV-2 virus spike protein (PDB ID: 7FAE) plays a pivotal role in the binding process with the host's ACE2 protein. It serves as a prime target for neutralizing antibodies against SARS-CoV-2 [Shan et al., 2022]. In our experiment, we predict all 494 possible mutations at 26 sites within the CDR region of the antibody heavy chain. We expect that predicting mutational effects on binding affinity can facilitate in identifying top favorable mutations that can enhance the neutralization efficacy of antibodies.
These mutations are ranked in ascending order according to their  $\Delta \Delta G$  values, with the most favorable (lowest  $\Delta \Delta G$ ) mutations positioned at the top. To enhance neutralization efficacy, we aim to accurately identify and rank the five most favorable mutations. DiffAffinity successfully identifies four of these mutations within the top  $20\%$  of the ranking and two within the top  $10\%$  (Table 3). DiffAffinity consistently outperforms all the baseline methods, indicating its superior ability to predict the effects of mutations. This highlights DiffAffinity's potential as a robust tool for optimizing human antibodies.
Table 2: Pearson correlation coefficient in SARS-COV-2 binding affinity
Method	Pearson
FoldX	0.385
RDE-Net	0.438
DiffAffinity*	0.295
DiffAffinity	0.466
Table 3: Rankings of the five favorable mutations on the human antibody against SARS-CoV-2 receptor-binding domain (RBD) by various competitive methods
Method	TH31W	AH53F	NH57L	RH103M	LH104F
FoldX	4.25%	14.57%	2.43%	27.13%	63.77%
RDE-Net	5.06%	12.15%	55.47%	50.61%	9.51%
DiffAffinity*	7.29%	0.81%	19.03%	84.21%	28.54%
DiffAffinity	7.28%	3.64%	18.82%	81.78%	10.93%
4.4 Prediction of side-chain conformations
To assess the generative capacity of SidechainDiff straightforwardly, we employ it for predicting side-chain conformations of specific amino acids given their structural context. Our approach involves sampling rotamers from the distribution modeled by SidechainDiff and selecting the rotamer with the highest likelihood from a pool of 100 samples. For this task, we draw upon the test dataset from the PDB-REDO test split.
For a rigorous evaluation, we compare SidechainDiff with both energy-based methods, including Rosetta [Leman et al., 2020] and SCWRL4 [Krivov et al., 2009], as well as the deep learning-based methods, including RDE, AttnPacker [McPartlon and Xu, 2023], and DLPacker [Misiura et al., 2022]. All methods are run under the same settings to ensure a fair comparison. We evaluate the performance
$^3$ NE501Y denotes the substitution of asparagine (N) at position 501 on chain E with tyrosine (Y), where N and Y represent the single-letter codes for the amino acids asparagine and tyrosine, respectively.
using the Mean Absolute Error (MAE) and steric clash number of the predicted rotamers [McPartlon and Xu, 2023].
In terms of MAE for rotamer prediction, SidechainDiff outperforms energy-based methods and achieves comparable results to deep learning-based methods such as RDE and AttnPacker (Table 4 and Table S1). Notably, our method surpasses all the other methods in terms of the steric clash number. It's noteworthy that while methods like AttnPacker focus on predicting the torsion angles of static side-chain conformations, SidechainDiff emphasizes the distribution of side-chain conformations under various physical constraints. We measure the physical constraints of mutation sites using the contact number, i.e., the number of neighboring amino acids within an  $8\mathring{A}$  radius. Generally, a higher contact number indicates a more constrained structural context. We observe that for mutation sites with higher constraints, SidechainDiff tends to produce more accurate side-chain conformations (Table 5). We further illustrate cases showing the trade-off between accuracy and diversity achieved by SidechainDiff under various physical constraints (Appendix D and Figure S5a-S5d).
Table 4: Evaluation of predicted side-chain conformations
Method	\( {\chi }^{\left( 1\right) } \)	\( {\chi }^{\left( 2\right) } \)	\( {\chi }^{\left( 3\right) } \)	\( {\chi }^{\left( 4\right) } \)	Average	Clash number
Proportion	100%	82.6%	28.2%	12.6%	-
SCWRL4	24.33°	32.84°	47.42°	56.15°	28.21°	118.44
Rosetta	23.98°	32.14°	48.49°	58.78°	28.09°	113.31
RDE	17.13°	28.47°	43.54°	58.62°	21.99°	34.41
DLPacker	21.40°	29.27°	50.77°	74.64°	26.42°	62.45
AttnPacker	19.65°	25.17°	47.61°	55.08°	22.35°	57.31
DiffAffinity	18.00°	27.41°	43.97°	56.65°	22.06°	27.70
Table 5: Accuracy of side-chain conformations stratified based on the contact number
Contact number	Average x
1 ~ 7	28.70
8 ~ 10	22.39
11 ~ 19	15.43
We present SidechainDiff, a Riemannian diffusion-based generative model for protein side-chains. It excels in both generating side-chain conformations and facilitating representation learning. Utilizing the learned representations, our method achieves state-of-the-art performance in predicting  $\Delta \Delta G$  across various test datasets, including the SKEMPI2 dataset and the latest SARS-CoV-2 dataset. Moreover, we demonstrate its effectiveness in antibody screening and its superior side-chain packing accuracy.
In future research, we can consider integrating the side-chain clash loss [Jumper et al., 2021] to refine our generative process. Additionally, we can explore building an end-to-end diffusion-based generative model for protein full-atom structures by integrating our model with existing backbone structure generation methods [Watson et al., 2023, Yim et al., 2023]. The main limitation of our model is the lack of consideration of backbone structure changes induced by mutations. Addressing the limitation could lead to enhanced performance.
We would like to thank the National Key Research and Development Program of China (2020YFA0907000), and the National Natural Science Foundation of China (32370657, 32271297, 82130055, 62072435), and the Project of Youth Innovation Promotion Association CAS to H.Z. for providing financial support for this study and publication charges. The numerical calculations in this study were supported by ICT Computer-X center.
Rebecca F Alford, Andrew Leaver-Fay, Jeliazko R Jeliazkov, et al. The Rosetta All-Atom Energy Function for Macromolecular Modeling and Design. Journal of Chemical Theory and Computation, 13(6):3031-3048, 2017.