SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Previous research by Jumper et al. [32] addressed this concern by offering an alternative angle prediction,  $\chi + \pi$ , and minimizing the minimum distance between the ground-truth and both predictions. In our diffusion-based framework, we employ a distinct method. Specifically, the  $\pi$ -rotation-symmetry results in the equivalence  $(\chi_i) \sim (\chi_i + k\pi)$  in torsion space, differing from the normal equivalence relationship in torsion space by a factor of  $2k$ . Consequently, we can still define the forwarding diffusion process in torsion space, albeit with a modification to Equation 4:
Figure 4: Distribution of  $\pi$ -rotation-symmetry torsional angles (Blue) and  $2\pi$ -rotation-symmetry (Red).
$$
p _ {t \mid 0} \left(\chi^ {\prime} \mid \chi\right) \propto \sum_ {\boldsymbol {d} \in \mathbb {Z} ^ {m}} \exp \left(- \frac {\| \boldsymbol {\chi} - \boldsymbol {\chi} ^ {\prime} + c \pi \boldsymbol {d} \| ^ {2}}{2 \sigma^ {2} (t)}\right), \quad \boldsymbol {c} \in \{1, 2 \} ^ {m} \tag {10}
$$
where  $c_{i} = 1$  for  $\pi$ -rotation-symmetric rigid groups, and  $c_{i} = 2$  otherwise.
Protein side-chain packing. Conventional approaches for protein side-chain packing (PSCP) involve minimizing the energy function over a pre-defined rotamer library [27, 73, 37, 3, 1, 35, 71, 7]. The choice of rotamer library, energy function, and energy minimization procedure varies among these methods. These methods rely on search heuristics and discrete sampling, limiting their accuracy. Currently, efficient methods like OSCAR-star [37], FASPR [27], SCWRL4 [35] do not incorporate deep learning and depend on rotamer libraries.
Several ML methods exist for side-chain prediction [47, 46, 69, 70, 73, 39, 71], including SIDE-Pro [47], which trains 156 feedforward networks to learn an additive energy function over pairwise atomic distances; DLPacker [46], which uses a deep U-net-style neural network to predict atom positions and selects the closest matching rotamer; OPUS-Rota4 [70], which employs multiple deep networks and utilizes MSA as input to predict side-chain coordinates and obtain a final structure;
and AttnPacker [45], which builds transformer layers and triangle updates based on components in Tensor Field Network [59] and SE(3)-Transformer [17] and achieves the state-of-the-art performance. In contrast, our method focuses solely on torsion space degrees of freedom and leverages an autoregressive diffusion model to accurately and efficiently model rotamer energy.
Diffusion models on molecules and proteins. The Diffusion Probabilistic Model (DPM), which was introduced in [55], has recently gained attention for its exceptional performance in generating images and waveforms [23, 10]. DPMs have been employed in a variety of problems in chemistry and biology, including molecule generation [72, 26, 68, 30], molecular representation learning [40], protein structure prediction [67], protein-ligand binding [12], protein design [2, 44, 28, 66, 38, 74], motif-scaffolding [60], and protein representation learning [76]. In this work, we investigate diffusion models in a new setting, protein side-chain packing, and propose a novel autoregressive diffusion model. Note that our definition of autoregressive diffusion model differs from existing works [25].
Dataset. We use BC40 for training and validation, which BC40 is a subset of PDB database selected by  $40\%$  sequence identity [63]. Following the dataset split in [45], there are 37266 protein chains for training and 1500 protein chains for validation. We evaluate our models on CASP13 and CASP14. Training set is curated so that no structure share sequence similarity with test set by  $\geq 40\%$ .
Baselines. We compare DiffPack with deep learning methods, like AttnPacker [45], DLPacker [46] and traditional methods including SCWRL4 [35], FASPR [27] and RosettaPacker [8]. Details can be found in Appendix H.1
Metrics. We evaluate the quality of generated side-chain conformations using three metrics: (1) Angle MAE measures the mean absolute error of predicted torsional angles. (2) Angle Accuracy measures the proportion of correct predictions, considering a torsional angle correct if the deviation is within  $20^{\circ}$ . (3) Atom RMSD measures the average RMSD of side-chain atoms for each residue.
Since predicting surface side-chain conformations is considered more challenging, some results are divided into "Core" and "Surface" categories. Core residues are defined as those with at least  $20C_{\beta}$  atoms within a  $10\AA$  radius, while surface residues have at most  $15C_{\beta}$  atoms in the same region.
5.2 Side-Chain Packing
Table 1 summarizes the experimental result in CASP13, our model outperforms all other methods in predicting torsional angles, achieving the lowest mean absolute errors across all four Angle MAE categories  $(\chi_{1},\chi_{2},\chi_{3}$  , and  $\chi_4$  ). Additionally, DiffPack shows the highest Angle Accuracy for all residues  $(69.5\%)$  , core residues  $(82.7\%)$  , and surface residues  $(57.3\%)$  , where for surface residue the accuracy is increased by  $20.4\%$  compared with previous state-of-the-art model AttnPacker. These results demonstrate that our model is better at capturing the distribution of the side chain torsion angles in both protein surfaces and cores. As for the atom-level side chain conformation prediction, DiffPack clearly outperforms other models in Atom RMSD. Moreover, the intrinsic design of DiffPack ensures that the generated structures have legal bond lengths and bond angles, while previous models in atom-coordinate space (e.g. AttnPacker) can easily generate side chains with illegal bond constraints without post-processing (as illustrated in Figure 6A).
Similarly, DiffPack outperforms other methods in all Angle MAE categories on the CASP14 dataset (Table 2). It achieves the highest Angle Accuracy for all residues  $(57.5\%)$ , core residues  $(77.8\%)$ , and surface residues  $(43.5\%)$ . Furthermore, DiffPack reports the best Atom RMSD for all residues  $(0.793\mathring{\mathrm{A}})$ , core residues  $(0.356\mathring{\mathrm{A}})$ , and surface residues  $(0.956\mathring{\mathrm{A}})$ .
Despite its superior performance on both test sets, our model, DiffPack, has a significantly smaller number of total parameters (3,043,363) compared to the previous state-of-the-art model, AttnPacker (208,098,163), which relies on multiple layers of complex triangle attention. This substantial reduction  $(68.4\times)$  in model size highlights the efficiency of diffusion-based approaches like DiffPack, making them more computationally feasible and scalable solutions for predicting side-chain conformations.
Method	ANGLE MAE °↓	ANGLE ACCURACY % ↑	ATOM RMSD Å ↓
χ1	χ2	χ3	χ4	All	Core	Surface	All	Core	Surface
SCWRL	27.64	28.97	49.75	61.54	56.2%	71.3%	43.4%	0.934	0.495	1.027
FASPR	27.04	28.41	50.30	60.89	56.4%	70.3%	43.6%	0.910	0.502	1.002
RosettaPacker	25.88	28.25	48.13	59.82	58.6%	75.3%	35.7%	0.872	0.422	1.001
DLPacker	22.18	27.00	51.22	70.04	58.8%	73.9%	45.4%	0.772	0.402	0.876
AttnPacker	18.92	23.17	44.89	58.98	62.1%	73.7%	47.6%	0.669	0.366	0.775
DiffPack	15.35	19.19	37.30	50.19	69.5%	82.7%	57.3%	0.579	0.298	0.696
Table 1: Comparative evaluation of DiffPack and prior methods on CASP13.
Method	ANGLE MAE °↓	ANGLE ACCURACY % ↑	ATOM RMSD Å ↓
χ1	χ2	χ3	χ4	All	Core	Surface	All	Core	Surface
SCWRL	33.50	33.05	51.61	55.28	45.4%	62.5%	33.2%	1.062	0.567	1.216
FASPR	33.04	32.49	50.15	54.82	46.3%	62.4%	34.0%	1.048	0.594	1.205
RosettaPacker	31.79	28.25	50.54	56.16	47.5%	67.2%	33.5%	1.006	0.501	1.183
DLPacker	29.01	33.00	53.98	72.88	48.0%	66.9%	33.9%	0.929	0.476	1.107
AttnPacker	25.34	28.19	48.77	51.92	50.9%	66.2%	36.3%	0.823	0.438	1.001
DiffPack	21.91	25.54	44.27	55.03	57.5%	77.8%	43.5%	0.770	0.356	0.956
Table 2: Comparative evaluation of DiffPack and prior methods on CASP14.
5.3 Side-Chain Packing on Non-Native Backbone
In addition to native backbones, another interesting and challenging problem is whether the side chain packing algorithm can be applied to non-native backbone (e.g., backbones generated by protein folding algorithms). In this regard, we extend DiffPack to accommodate non-native backbones generated from AlphaFold2 [32]. Table 3 gives the quantitative result of different algorithms including AlphaFold2's side-chain prediction on the CASP13-FM test set. All metrics are calculated after aligning the non-native backbone of each residue to the native backbone. As observed, DiffPack achieves state-of-the-art on most metrics. Notably, DiffPack is the only model that consistently outperforms AlphaFold2 in all metrics, showcasing its potential to refine AlphaFold2's predictions.
To analyze the contribution of different parts in our proposed method, we perform ablation studies on CASP13 and CASP14 benchmarks. The results are shown in Table 4.
Autoregressive diffusion modeling. We evaluate our autoregressive diffusion modeling approach against two baselines: joint diffusion  $(\chi_{1,2,3,4})$  and random diffusion  $(\chi_i)$ . Joint diffusion models the joint distribution of the four torsional angles of each residue and performed diffusion and denoising on all angles simultaneously, while random diffusion perturbs one random torsional angle per residue and generates all angles simultaneously during inference. Our approach outperforms both baselines (Table 4). Training loss curves (Figure 5) show that random diffusion has an easier time optimizing its loss than joint diffusion, but struggles with denoising all angles at once due to a mismatch between training and inference objectives. Our autoregressive scheme strikes a balance between ease of training and quality of generation, achieving good performance.
We plot the training curves of the conditional distributions for the four torsional angles in DiffPack, finding that  $\chi_{1}$  and  $\chi_{2}$  are easier to optimize and perform better than random diffusion, likely due to discarding atoms from subsequent angles to overcome cumulative perturbation effects. However, training on the smaller set of valid residues with  $\chi_{3}$  and  $\chi_{4}$  is inefficient. Future work should address the challenge of training these angles more efficiently and mitigating cumulative errors in autoregressive models.
Inference. Three techniques are proposed in Section 3.5 to improve the inference of DiffPack. To evaluate the effectiveness of these techniques, we compare them with three baselines that do not
Figure 5: Training loss curves for different diffusion models.
Method	ANGLE MAE ° ↓	ANGLE ACCURACY % ↑	ATOM RMSD Å ↓
χ1	χ2	χ3	χ4	All	Core	Surface	All	Core	Surface
AlphaFold2*	35.20	31.10	51.38	56.95	51.3%	71.5%	38.7%	1.058	0.521	1.118
SCWRL	34.94	30.84	50.45	56.75	51.3%	69.5%	39.0%	1.079	0.550	1.148
FASPR	34.83	30.85	50.60	56.74	50.8%	67.9%	39.8%	1.073	0.573	1.114
RosettaPacker	35.43	31.63	51.33	56.18	50.9%	70.6%	38.5%	1.070	0.526	1.139
DLPacker	34.38	31.57	55.84	67.02	49.9%	69.1%	37.0%	1.032	0.543	1.090
AttnPacker	33.23	31.97	50.53	58.20	51.0%	68.4%	39.1%	0.981	0.512	1.027
DiffPack	31.25	30.17	48.32	56.82	55.5%	74.3%	41.9%	0.978	0.490	1.056
Table 3: Comparative evaluation on CASP13-FM non-native backbones generated by AlphaFold2.
Method	CASP13 ANGLE MAE °↓	CASP14 ANGLE MAE °↓
χ1	χ2	χ3	χ4	χ1	χ2	χ3	χ4
DiffPack	15.35	19.19	37.30	50.19	21.91	25.54	44.27	55.03
- w/ joint diffusion (χ1,2,3,4)	17.14	23.72	35.96	45.30	26.80	34.51	52.77	63.41
- w/ random diffusion (χi)	17.11	27.27	44.75	56.42	23.26	32.69	49.21	51.92
- w/o multi-round sampling	16.26	23.13	40.60	52.67	23.86	30.71	47.54	55.80
- w/o annealed temperature	15.55	22.56	39.32	51.37	22.82	29.61	45.86	55.22
- w/o confidence models	16.12	22.92	39.92	50.63	22.86	29.30	45.80	54.08
Table 4: Ablation study on CASP13 and CASP14.
use these techniques. For the baselines without multi-round sampling and annealed temperature, we simply resume the sampling procedure in the vanilla diffusion models. For the baseline without confidence models, we only draw one sample from our model instead of using confidence models to ensemble multiple samples. As shown in Table 4, the mean absolute error of the four torsional angles increases for the three baselines, demonstrating the effectiveness of our proposed techniques.
5.5 Case Study
DiffPack accurately predict the side-chain conformation with chemical validity. As shown in Figure 6A and Figure 6B. DiffPack accurately predict the side-chain conformation with a substantially lower RMSD (0.196Å and 0.241Å) compared with other deep learning methods. Furthermore, DiffPack consistently ensures the validity of generated structures, while AttnPacker without post-processing sometimes violates the chemical validity due to its operation on atom coordinates.
DiffPack correctly identifies the  $\pi$  stacking interaction. Accurate reconstruction of  $\pi$  stacking interaction between side-chain has traditionally been challenging. Traditional method usually requires a specific energy term for modeling this interaction. Interestingly, DiffPack has shown the ability to implicitly model this interaction without the need of additional prior knowledge (Figure 6C).
(A) CASP ID: T0951
(B) CASP ID: T0971
(C) CASP ID: T0949
Figure 6: Case studies on DiffPack. Predictions from different methods are distinguished by color. (A) DiffPack accurately predicts the side-chain conformation. AttnPacker-NoPP produces an invalid glutamic acid structure since  $\mathrm{O}^{\delta 1}$  is too close to  $\mathrm{O}^{\delta 2}$ . (B) DiffPack accurately predicts the  $\chi_{1}$  of leucine. (C) DiffPack correctly identifies  $\pi-\pi$  stacking interactions, indicated by dashed lines.
In this paper, we present DiffPack, a novel approach that models protein side-chain packing using a diffusion process in torsion space. Unlike vanilla joint diffusion processes, DiffPack incorporates an autoregressive diffusion process, addressing certain limitations. Our empirical results demonstrate the superiority of our proposed method in predicting protein side-chain conformations compared to existing approaches. Future directions include exploring diffusion processes for sequence and side-chain conformation co-generation, optimizing computational efficiency in side-chain packing, and considering backbone flexibility in the diffusion process.
We would like to thank Ramanarayan Mohanty from Intel for his support in optimizing the speed of DiffPack. We also extend our appreciation to Zhaocheng Zhu, Sophie Xhonneux and Chuanrui Wang for their constructive feedback on the manuscript. This project is supported by Twitter, Intel, the Natural Sciences and Engineering Research Council (NSERC) Discovery Grant, the Canada CIFAR AI Chair Program, Samsung Electronics Co., Ltd., Amazon Faculty Research Award, Tencent AI Lab Rhino-Bird Gift Fund, a NRC Collaborative R&D Project (AI4D-CORE-06) as well as the IVADO Fundamental Research Project grant PRF-2019-3583139727.
[1] Rebecca F. Alford, Andrew Leaver-Fay, Jeliazko R. Jeliazkov, Matthew J. O'Meara, Frank DiMaio, Hahnbeom Park, Maxim V. Shapovalov, P. Douglas Renfrew, Vikram Khipple Mulligan, Kalli Kappel, Jason W. Labonte, Michael S. Pacella, Richard Bonneau, Philip Bradley, Roland L. Dunbrack, Rhiju Das, David Baker, Brian Kuhlman, Tanja Kortemme, and Jeffrey J. Gray. The rosetta all-atom energy function for macromolecular modeling and design. bioRxiv, 2017.
[2] Namrata Anand and Tudor Achim. Protein structure and sequence generation with equivariant denoising diffusion probabilistic models. arXiv preprint arXiv:2205.15019, 2022.
[3] Aleksandra E. Badaczewska-Dawid, Andrzej Kolinski, and Sebastian Kmiecik. Computational reconstruction of atomistic protein structures from coarse-grained models. Computational and Structural Biotechnology Journal, 18:162 - 176, 2019.
[4] Minkyung Baek, Frank DiMaio, Ivan V. Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong, Lisa N. Kinch, R. Dustin Schaeffer, Claudia Millán, Hahnbeom Park, Carson Adams, Caleb R. Glassman, Andy M. DeGiovanni, Jose H. Pereira, Andria V. Rodrigues, Alberdina A. A. van Dijk, Ana C. Ebrecht, Diederik Johannes Opperman, Theo Sagmeister, Christoph Buhlheller, Tea Pavkov-Keller, Manoj K. Rathinaswamy, Udit Dalwadi, Calvin K. Yip, John E. Burke, K. Christopher Garcia, Nick V. Grishin, Paul D. Adams, Randy J. Read, and David Baker. Accurate prediction of protein structures and interactions using a 3-track neural network. Science (New York, N.Y.), 373:871 - 876, 2021.
[5] Federico Baldassarre, David Ménendez Hurtado, Arne Elofsson, and Hossein Azizpour. Graphq: protein model quality assessment using graph convolutional networks. Bioinformatics, 37:360 - 366, 2020.
[6] Md Shariful Islam Bhuyan and Xin Gao. A protein-dependent side-chain rotamer library. In BMC bioinformatics, volume 12, pages 1-12. Springer, 2011.
[7] Yang Cao, Lin Song, Zhichao Miao, Yun Hu, Liqing Tian, and Taijiao Jiang. Improved sidechain modeling by coupling clash-detection guided iterative search with rotamer relaxation. Bioinformatics, 27 6:785-90, 2011.
[8] Sidhartha Chaudhury, Sergey Lyskov, and Jeffrey J Gray. Pyrosetta: a script-based interface for implementing molecular modeling algorithms using rosetta. Bioinformatics, 26(5):689-691, 2010.
[76] Zuobai Zhang, Minghao Xu, Aurélie C. Lozano, Vijil Chenthamarakshan, Payel Das, and Jian Tang. Physics-inspired protein encoder pre-training via siamese sequence-structure diffusion trajectory prediction. ArXiv, abs/2301.12068, 2023.
A More Related Work
Geometric deep learning on biomolecules. Learning protein representations based on 3D geometric information is crucial for various protein tasks. Recent advancements have led researchers to develop architectures that preserve properties such as invariance and equivariance for essential transformations like rotation and translation. These approaches have utilized various techniques such as node/atom message passing[19, 51, 52, 49], edge/bond message passing [31, 9], and directional information [33, 41, 34] to encode molecular graphs in 2D or 3D. Notably, recent models have been generalized to protein 3D structures [20, 5, 29, 22, 21] and protein surfaces [18, 58, 13, 56], demonstrating remarkable performance on diverse tasks. In this work, we utilize a state-of-the-art protein structure encoder, GearNet-Edge [75] to obtain SE(3)-invariant representations for denoising in the torsion space.
The impact of our work extends well beyond the realm of protein structure prediction, with implications across various fields of biological science and medical research. The ability to accurately predict protein side-chain conformations is of vital importance in understanding the intricate functioning of proteins and their interactions with other molecules, including substrates, inhibitors, and drugs.
Drug Design and Discovery: Accurate prediction of side-chain conformations could considerably advance the field of drug discovery and design, where the interaction between proteins and small molecule drugs or inhibitors is often determined by the precise conformation of side chains. Improved accuracy in side-chain packing can facilitate more precise predictions of protein-ligand binding sites and affinities, thereby expediting the development of new therapeutics.
Protein Engineering: Effective application of our method could also revolutionize protein engineering, where side-chain packing plays a pivotal role in protein stability, function, and interaction. Enhancing our understanding of side-chain packing will help engineer proteins with desired properties, including altered substrate specificity, increased stability, or novel functionality.
Disease Understanding: Many diseases, such as Alzheimer's, Parkinson's, and various cancers, are tied to misfolded proteins or mutations that affect protein structure. Accurate side-chain prediction can thus provide key insights into the structural consequences of such mutations, supporting the development of targeted treatments.
Bioinformatics and Computational Biology: On a technical note, our work provides a novel perspective for side-chain packing problem by formulating it as a diffusion process in the torsional space, which could inspire further innovations in bioinformatics and computational biology. Moreover, given the efficiency of our approach, it is anticipated to facilitate the rapid and scalable analysis of large-scale protein datasets, aiding in the exploration of the proteome.
A.2 Limitation
In this study, we focus exclusively on protein side-chain prediction under the assumption of a fixed and highly accurate backbone. Nevertheless, in real-world scenarios, protein backbones may be generated using existing structure prediction methods, which may not provide sufficient accuracy. It is essential to address how our methods can be adapted to accommodate such settings, and this aspect remains an important future direction for our research. Additionally, we intend to explore the application of our methods in various downstream tasks, such as protein-protein interactions and protein engineering tasks, as part of our future research directions.
B.1 Steric Clash
In order to quantify whether the generated structure has severe steric clash, we add an additional metric Steric Clash Count measures the mean number of clash pairs appearing in generated structures. Following previous work [45, 36], the distance threshold between different types of atoms is initially defined by van der Waals radii and further adjusted to account for factors such as H-bond and disulfide
bridges. A protein atom pair is identified to have a steric clash if the inter-atomic distance is within  $\mathrm{X}\%$  of the threshold. Here, X is chosen as  $100\%$ ,  $90\%$ , and  $80\%$ , representing varying levels of stringency and providing a more comprehensive assessment of steric clashes.
Method	CASP13 CLASH COUNT ↓	CASP14 CLASH COUNT ↓
100%	90%	80%	100%	90%	80%
SCWRL	115.3	20.6	4.6	124.0	24.6	6.5
FASPR	112.8	23.3	5.6	130.5	29.5	8.7
RosettaPacker	73.8	7.9	2.6	100.6	10.1	3.4
DLPacker	64.3	7.3	2.0	74.1	10.5	3.0
AttnPacker-noPP	40.1	5.7	1.5	49.5	10.2	3.5
DiffPack	37.5	4.6	0.9	46.5	6.0	1.1
Table 5: Mean Clash Pair Number of Generation Result
From Table 5, it is clear that the DiffPack method outperforms the other techniques across all levels of stringency for both CASP13 and CASP14 datasets. This superiority suggests that the DiffPack method produces protein structures with fewer steric clashes, indicating a more realistic and physically plausible model of protein side chain packing.
At the  $100\%$  distance threshold, DiffPack generates  $67.5\%$  fewer clashes than SCWRL,  $66.7\%$  fewer than FASPR,  $49.1\%$  fewer than RosettaPacker, and  $41.7\%$  fewer than DLPacker for the CASP13 dataset. Similarly, for the CASP14 dataset, it generates  $62.5\%$  fewer clashes than SCWRL,  $64.3\%$  fewer than FASPR,  $53.8\%$  fewer than RosettaPacker, and  $37.2\%$  fewer than DLPacker.
At the  $90\%$  and  $80\%$  distance thresholds, the comparative reduction in clashes by DiffPack is also evident. This trend suggests that DiffPack consistently generates structures with fewer steric clashes, which is crucial for generating biologically feasible protein structures.
Interestingly, the AttnPacker-noPP method also shows a considerable reduction in clashes compared to other methods, especially SCWRL and FASPR, but it is still surpassed by the performance of DiffPack. This suggests that the autoregressive diffusion model used in DiffPack is more capable of managing steric clashes in protein structure generation, demonstrating the effectiveness of this approach.
B.2 Visualization of Sampling Results
Figure 7: Visualization of sampling results. DiffPack (green), AttnPacker (purple), DLPacker (red), RosettaPacker (orange) are evaluated on T1000 (top), T0954 (middle), T1020 (bottom) cases.
C Details of Autoregressive Diffusion Models
C.1 Atom groups for each residue type
For completeness, we provide the definition of torsional angles and corresponding atom groups for each residue in Table 6. These definitions align with those presented in the AlphaFold2 paper [32].