DiffPack: A Torsional Diffusion Model for Autoregressive Protein Side-Chain Packing
Authors: Yangtian Zhang
Zuobai Zhang
Bozitao Zhong
Sanchit Misra
Jian Tang
Venue: NeurIPS 2023 poster (2023)
Keywords: protein side-chain packing
diffusion models
autoregressive models
geometric deep learning
TL;DR: Accurate protein side-chain packing via an autoregressive diffusion model in torsion space

Abstract: Proteins play a critical role in carrying out biological functions, and their 3D structures are essential in determining their functions. 
Accurately predicting the conformation of protein side-chains given their backbones is important for applications in protein structure prediction, design and protein-protein interactions. Traditional methods are computationally intensive and have limited accuracy, while existing machine learning methods treat the problem as a regression task and overlook the restrictions imposed by the constant covalent bond lengths and angles. In this work, we present DiffPack, a torsional diffusion model that learns the joint distribution of side-chain torsional angles, the only degrees of freedom in side-chain packing, by diffusing and denoising on the torsional space. To avoid issues arising from simultaneous perturbation of all four torsional angles, we propose autoregressively generating the four torsional angles from $\chi_1$ to $\chi_4$ and training diffusion models for each torsional angle. We evaluate the method on several benchmarks for protein side-chain packing and show that our method achieves improvements of 11.9% and 13.5% in angle accuracy on CASP13 and CASP14, respectively, with a significantly smaller model size ($60\times$ fewer parameters). Additionally, we show the effectiveness of our method  in enhancing side-chain predictions in the AlphaFold2 model. Code is available at https://github.com/DeepGraphLearning/DiffPack.

DiffPack: A Torsional Diffusion Model for Autoregressive Protein Side-Chain Packing
Yangtian Zhang 1.2
Zuobai Zhang 1.+
Bozitao Zhong!.
Sanchit Misra' Jiin Tang 4A5t
"qual contribution
fcorresponding author
Mila Québec Al Institute
Universitéé de Montréal Intel
*HEC Montréal
SCIAAR AI Research Chair
contacts: <yangtian.zhang zhang, zuobai.zhang @ @mila.quebee, jian.tang hcc.ca
Abstract
Proteins play critical role in carrying out biological functions, and their 3D structures are essential in determining their functions. Accurately predicting the conformation of protein side-chains given their backbones is important for ap- plications in protein structure prediction, design and protein-protein interactions. Traditional methods are computationally intensive and have limited accuracy, while existing machine learning methods treat the problem as a regression task and over- look the restrictions imposed by the constant covalent bond lengths and angles. In this work, we present DiftPack, a torsional diffusion model that learns the joint dis- tribution of side-chain torsional angles, the only degrees of freedom in side-chain packing, by diffusing and denoising on the torsional space. To avoid issues arising from simultaneous perturbation of all four torsional angles, we propose autoregres- sively generating the four torsional angles from to and training diffusion models for each torsional angle. We evaluate the method on several benchmarks for protein side-chain packing and show that our method achieves improvements of 11.9% and 13.5% in angle accuracy on CASP13 and CASP14, respectively, with a significantly smaller model size (60x fewer parameters). Additionally, we show the effectiveness of our method in enhancing side-chain predictions in the AlphaFold2 model. Code is available at
Introduction
Proteins crucial for performing diverse range of biological functions, such as catalysis, signaling, and structural support. Their three-dimensional structures, determined by amino acid arrangement, are crucial for their function. Specifically, amino acid side-chains play critical role in the stability and specificity of protein structures by forming hydrogen bonds, hydrophobic interactions, and other non- covalent interactions with other side-chains the protein backbone. Therefore, accurately predicting protein side-chain conformation is an essential problem in protein structure prediction [16, 15, 11], design [48, 54, 14, 65] and protein-protedd interactions [64, 24].
Despite recent advancements in deep learning models inspired by AlphaFold2 for predicting the positions of protein backbone atoms [32, 4], predicting the conformation of protein side-chains remains a challenging problem due to the complex interactions between side chains. In this work, we focus the problem of predicting side-chain conformation with fixed backbone structure, a. a.k.a., protein side-chain packing. Traditional methods for side-chain prediction rely on techniques such as rotamer libraries, energy functions, and Monte Carlo sampling [27, 73, 37, 1, 35, 71, 7]. However, these methods are computationally intensive and struggle to accurately capture the complex energy landscape of protein side chains due to their reliance search heuristics and discrete sampling.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Several machine learning methods have been proposed for side-chain prediction, including DL- Packer [46], AttnPacker [45], and others [47, 69, 70, 73, 39, 71J. DLPacker, the first deep learning- based model, employs 3D convolution network to learn the density map of side-chain atoms, but it lacks the ability to capture rotation equivariance due to its 3D-CNN structure. contrast, AttnPacker, the current state-of-the-ar model, directly predicts side-chain atom coordinates using Tensor Field Network and SE(3)-Trang Transformer, ensuring rotation equivariance in side chain packing. However, does not consider the restrictions imposed by covalent bond lengths and angles, leading to inefficient training and unnatural bond lengths during generation. Furthermore, previous methods that treat side-chain packing
regression problem assume single ground-truth side-chain structure and overlook the fact that proteins can fold into diverse structures under different environmental factors, resulting in distribution of side-chain conformations.
In this study, we depart from the standard practice of focusing on atom-level coordinates in Cartesian space as in prior research [45, 46]. Instead, we introduce DiffPack, a torsional diffusion model that models the exact degree of freedom in side-chain packing, the joint distribution of four torsional angles. By perturbing and denoising in the torsional space, we use learn the gradient field for the joint distribution of torsional angles. This result in a much smaller conformation space of side-chain, thereby capturing the intricate energy landscape of protein side chains. Despite its effectiveness, a direct joint diffusion process on the four torsion angles could result in steric clashes and accumulative coordinate displacement, which complicates the denoising process. To address this, we propose an autoregressive diffusion process and train separate diffusion models to generate the four torsion angles from X1 X4 in autoregressive manner. During training, each diffusion model only requires perturbation on its corresponding torsional angle using a feacher-forcing strategy, preserving the protein structure and avoiding the aforementioned issues. To improve the capacity of our model, we further introduce three schemes in sampling for consistently improving the inference results: multi-round sampling, annealed temperature sampling, and confidence models.
--ivaraan network to
We evaluate our method on several benchmarks for protein side-chain packing and compare it with existing state-of-the-an methods. Our results demonstrate that DiffPack outperforms existing state- of-the-art approaches, achieving remarkable improvements of 11.9% and 13.5% in angle accuracy on CASP13 and CASP14, respectively. Remarkably, these performance gains are achieved with significantly smaller model size, approximately 60 times fewer parameters, highlighting the potential of autoregressive diffusion models in protein structure prediction. Furthermore, showcase the effectiveness of our method in enhancing the accuracy of side-chain predictions in the AlphaFold2 model, indicating its complementary nature existing approaches.
2 Background
Protein. Proteins are composed of a sequence of residues (amino acids), each containing an alpha carbon (C,) atom bounded an amino group (-NH2), a carboxyl group (-COOH) and side-chain (-R) that identifies the residue type. Peptide bonds link consecutive residues through dehydration synthesis process. The backbone of a protein consists of the Ca atom and the connected nitrogen, carbon, and oxygen atoms. We use S = [s1, $2...., .... s,,] to denote the sequence of a protein with n residues, where 8; E (o.,.. 19} denotes the type of the i-th residue.
Protein Conformation. Physical interactions between residues make a protein fold into its native 3D structure, a.k.a., conformation, which determines its biologically functional activity. We use X' = [21, 22, ..., _m]] to denote the structure of a protein, where x; denotes the set of atom coordinates belonging the i-th residue. The backbone structure (bb) of the i-th residue is a subset consisting of backbone atoms, i.e., Ca, N, C, O, while the side-chain consists of the remaining atoms 22(*))
J; \\a .,.(bb] The backbone conformation (bb) and side-chain conformation (se) of the protein are defined as the set of backbones and side-chains of all residues, respectively.
Protein Side-Chain Packing. Protein side-chain packing (PSCP) problem aims predict the 3D coordinates (()) of side-chain atoms given backbone conformations (hb) and protein sequence S.
That is, we aim to model the conditional distribution ((x( (x00)

---

Forward Process
(A)
213
277
Torsion Distribution
Torsion Distribution
Anum Message Paaning
Mesrqge
Torus Force Field
(B)
p(x)
pbx lx)
p(x1 Xx)
X2.
Figure I: Overview of DifiPack. Given a protein sequence and backbone structure, we aim model the conditional distribution of side-chain conformation. (A) Distribution of side-chain conformation modeled through diffusion process in torsion space T SE(3)-invariant network is used to learn the torus force field (torsion score). (B) Four torsion angles are generated autoregressively across all residues.
3 Methods
In this paper, we introduce an autoregressive diffusion model DiffPack predict the side-chain conformation distribution in torsion space. We address the issue of overparameterization by in- troducing torsion-based formulation of side-chain conformation in Section 3.1. Then we give formulation of the torsional diffusion model in Section 3.2. However, directly applying the diffusion model encounters challenges in learning the joint distribution, which we address by introducing an autoregressive-styl model in Section 3.3. We then provide details of the model architecture in Section 3.4, followed by an explanation of the inference procedure in Section 3.5.
3.1 Modeling Side-Chain Conformations with Torsional Angles
Previous methods [45, 46] model the side chain conformation as a series of three-dimensional coordinates in the Cartesian space. However, this approach does not take into account the restrictions imposed by constant covalent bond lengths and angles the side chain's degree of freedom, resulting in inefficient training and unnatural bond lengths during generation.
To overcome this overparameterization issue, we propose modeling side chain conformation torsion space. As illustrated in Figure 2, torsional angles directly dictate protein side-chain conformation by determining the twist between two neighboring planes. Table lists the atom groups to define corresponding the neighboring plane for each residue type. The number of torsional angles varies across different residues, with a maximum of four (X1. X2+ Xa. X4). Modeling side chains in torsion space reduces the number of variables by approximately one third and restricts degrees of freedom [6]. Formally, we transform the problem of side-chain packing to modeling the distribution of torsional angles:
Figure 2: Illustra- tion of four tor- sional angles.
(x) (1()) S) + p X1, X2. X3. Xa (1M))

---

where Xx E [0, 2 use m, denote the number of residues with X;. The space of possible side chain conformation is, therefore, reduced to an m-dimension sub-manifold M C TR with m mm + m2 + mg m4.
th is a vector of the i-th torsional angles of all residues in the protein and we
3.2 Diffusion Models on Torsional Space
Denoising diffusion probabilistic models are generative models that learn the data distribution via forward diffusion process and reverse generation process [23]. We follow [30] to define diffusion models on the torsional space with the continuous score-based framework in [57]. For simplicity, we omit the condition and aggregate torsional angles on all residues as torsional vector X E [0, )
Starting with the data distribution as Po(X), the forward diffusion process is modeled by stochastic differential equation (SDE):
dx f(x,)) dt + g(t) dun
where is the Wiener process on the torsion space and f(x. 1), g(() the drift coefficient and diffusion coefficient, respectively. Here we adopt Variance-Exploding SDE where f(x, t) = 0 and g(f) is exponentially decayed with With sufficiently large T, the distribution pr approaches uniform distribution over the torsion space. The reverse generation process samples from the prior and generates samples from the data distribution po(x) via approximately solving the reverse SDE:
dx () - g2 V log)((x:)] di + g(l) dw,
where a neural network is learned to fit the score Vx x log (Xx) of the diffused data [23,57] Inspired by [30], convert the torsional angles into 3D atom coordinates and define the score network on Euclidean space, enabling it to explicitly learn the interatomic interactions.
The training process involves sampling from the perturbation kernel of the forward diffusion and computing its score to train the score network. Given the equivalence (XI: .m)) ~ (X1 + 2TT, ..., Xm)... (X1,..,, Xm + 2 in torsion space, the perturbation kernel is wrapped Gaussian distribution on R This means that any X- x' E [0, 2T) the perturbation kernel is proportional to the sum of exponential terms, which depends on the distance between X and X":
(x)(x(lx) Edezm exp X x x'+2dd11
(r)
In order to sample from the perturbation kernel, we sample from the corresponding unwrapped isotropic normal and take eiement-wise mod 2n. The kernel's scores are pre-computed using a numerical approximation. During training, uniformly sampled time step t and the denoising score matching loss is minimized:
(f) = E2
[lls(x..) t) V, Pogg
(5)
where the weight factors x(t) 1/Ex~p o(10) V [og)))(((00] are also pre-computed.
3.3 Autoregressive Diffusion Models
The direct approach to torsional diffusion models introduces noise on all torsional angles simultane- ously and poses two significant challenges:
Adding noises to torsional angles is equivalent with rotating side-chain atoms in and beyond the corresponding atom groups. For instance, if X is rotated, it affects the coordinates of atoms in the X2, X3, and X4 groups. This cumulative effect complicates the denoising process of the latter three angles. Similar issues arise after rotating X2 and Xa- The effect of rotating X1 is illustrated in Figure 3.
1. Cumulative coordinate displacement:
2. Excessive steric clash: Adding noises to all torsional angles may damage protein structures and complicate the denoising process in our score network. In Appendix C, Figure compares the number of atom clashes observed when using noise schemes in vanilla diffusion models and autoregressive--ttyle ones to show the straightforward application of diffusion models results in significantly more steric clashes
Figure 3: Effects of rotating XI-
To address the aforementioned issues, we propose an autoregressive diffusion model over the torsion space. We factorize the joint distribution of the four torsional angles into separate conditional

---

distributions. Specifically, we have
p(X,, X2, X3, X4) p(xi) p(xalxi) p(x)/x1.2) p(x4(x1,22).
(6)
This allows us to model the side-chain packing problem as a step-by-step generation of torsional angles: first, we predict the first torsional angles Xi for all residues given the protein backbone structure; next, based on the backbone and the generated X1. we predict the second torsional angles X2; and so on for Xs and X4-
We train a separate score network for each distribution on the right-hand side using the torsional diffusion model from Section 3.2. To train the autoregressive model, we use a teacher-forcing strategy. Specifically, when modeling these angles fixed. We then rotate Xi by sampling noises from the perturbation kernel in (4) and discard all atoms belonging to the Xi+I,...4 groups to remove the dependency on following torsional angles. This approach eliminates the cumulative effects of diffusion process Xi and preserves the overall structure of the molecule, thus overcoming the aforementioned challenges. The generation process is illustrated in Figure I and the training procedure is described in Algorithm 1.
(I.....--1) we assume that Xi...,i-1 are ground truth and keep
3.4 Model Architecture
To model p(x: (1)....-1),. we utilize score network constructed using the 3D coordinates ((x, t) of backbone atoms and atoms in the X1.....-1 groups. The score network's 's output represents the noise on torsional angles, which should be SE(3)-invariant w.t. the conformation X. Unlike previous methods [45] operating in Cartesian coordinates that require an SE(3) SE(3)-equivariant score network, our method only requires SE(3)-invariance, providing greater flexibility in designing the model architecture. To ensure this invariance, we employ GearNet-Edge [75], a saute-of-the-as protein structure encoder. This involves constructing a multi-relational graph with atoms as nodes, where node features consist of one-hot encoding for atom types, corresponding residue types, and time step embeddings. Edges are added based on chemical bond and 3D spatial information, determining their type. To learn representations for each node, we perform relational message passing between them [50]. We denote the edge between nodes i and j with type r as (i.jrr r) and set of relations as R.
We use h to denote the hidden representation of node at layer I. Then, message passing can be written as:
(1)
h = h. (1-1) to (BatchNorm (EeER W, ESEN, (h5-") Linear (m.))))))
where W, is the learnable weight matrix for relation type NN-() is the neighbor set of j with relation type r, and o-) is the activation function. Edge representations m (i, are obtained through edge message passing. We use e as the abbreviation the edge r... Two edges e and e2 connected if they share common end node, with the connection type determined by the discretized angle between them. The edge message passing layer can be written as:
m (1)
= (aachhNorm EEER W WeEeNeAt))) mea ")²"
(8)
where R' is the set of relation types between edges and (ei) is the neighbor set of e with relation type r. After obtaining the hidden representations of all atoms at layer L, we compute the residue representations by taking the mean of the representations of its constituent atoms. The residue representations are then fed into an MLP for score prediction. The details of architectural components are summarized in Appendix D.
3.5 Inference
After completing the training, we adopt the common practice of autoregressive and diffusion models for inference, as described in Algorithm We generate four torsional angles step by step as described in Section 3.3. When sampling X based on the predicted torsional angles X1....i-1, we begin by sampling a random angle from the uniform prior and then discretize and solve the reverse diffusion. At each time step, generate atoms in the X1..... group and use learned score network for denoising. We also discover several simple techniques that significantly improve our performance, including multi-round sampling, annealed temperature sampling and confidence models.
Predictor-corrector sampling. After discretizing the reverse diffusion SDE, we perform multiple Langevin steps subsequent to each denoising step. This hybrid method, which mixes the denoising
5

---

process with Langevin dynamics, is called ppedictor-correcte sampling, as suggested by [57]. This approach can be seen introducing an equilibration process stabilize Pt, and is demonstrated to be effective in diffusion process.
Annealed temperature sampling. When designing generative model, two critical aspects to consider quality and diversity. The diffusion model often suffers from overdispersion, which prioritizes diversity over sampling quality. However, in the context of side chain packing, quality is more important than diversity. Directly using the standard reverse sampling process may lead to undesirable structures. Following [28], we utilize an annealed temperature sampling scheme to mitigate this issue. Specifically, We modify the reverse SDE by adding annealed weight d to the score function (details in Appendix E):
dx = - do²
(t) dw, where d To (T (t)
Vx (og dt
max
(9)
dt
dt
The above modification results in a reverse process approximating the low temperature sampling process, where ideally decreasing tempeture T lead to sampling process biased towards quality.
Confidence model. As per common practice protein structure prediction [32], we train confidence model to select the best prediction among multiple conformations sampled from our model. The model architecture we use the same as that used in diffusion models, which takes the entire protein conformation as input and outputs representations for each residue. We train the model on the same dataset, using the residue representations to predict the negative residue-level RMSD of our sampled conformations. When testing, rank all conformations based on confidence model.
3.6 Handling Symmetric Issues
Torsional angles generally exhibit periodicity of 2x. However, certain rigid side-chain structures possess a n-rrtation-symmeerr, meaning that rotating the torsional angle by does not yield distinet physical structures, as demonstrated in Figure 4. For instance, in the case of tyrosine residue, the phenolic portion of its side chain ensures that a ı rotation of its X2 torsion angles only alters the internal atom name without affecting the actual structure.
0+
330
300
60
Previous research by Jumper et al. [32] addressed this concern by offering an alternative angle prediction, X and minimizing the minimum distance between the ground and both predictions. In our diffusion- based framework, employ distinct method. Specifically, the crootation-ymmmette results the equivalence (xs) (Xi + kr) in torsion space, differing from the normal equivalence relationship in torsion space by factor of 2k. Con- sequently, can still define the forwarding diffusion process in torsion space, albeit with a modification to Equation 4:
240
120*
210*
150*
180
Figure 4: Distribution of rotation-symmetry torsional an- gles (Blue) and rottoiition- symmetry (Red).
( x Ea exp
|lx x tendll'
e E (1,2)m
(10)
(t)
where c; = for r-rotation r-rotation-yymmetrii symmetric rigid groups, and c; = 2 otherwise.
4
Related Work
Protein side-chain packing. Conventional approaches for protein side-chain packing (PSCP) involve minimizing the energy function over a pre-defined rotamer library [27, 73, 37, 3, 1, 35, 71, 7]. The choice of rotamer library, energy function, and energy minimization procedure varies among these methods. These methods rely on search heuristics and discrete sampling, limiting their accuracy. Currently, efficient methods like OSCAR-star [37], FASPR [27], SCWRL4 [35] do not incorporate deep learning and depend on rotamer libraries.
Several ML methods exist for side-chain prediction [47, 46, 69, 70, 73, 39, 71], including SIDE- Pro [47]. which trains 156 feedforward networks to learn an additive energy function over pairwise atomic distances; PaPkak [46], which uses deep U-net-style neural network to predict atom positions and selects the closest matching rotamer; OPUS-Rota4 [70], which employs multiple deep networks and utilizes MSA as input to predict side-chain coordinates and obtain a final structure;

---

and AttnPacker [45], which builds transformer layers and triangle updates based on components in Tensor Field Network [59] and 3E(3)-Transfone [17] and achieves the state-of- of-the-ar art perfor- mance. In contrast, our method focuses solely on torsion space degrees of freedom and leverages an autoregressive diffusion model to accurately and efficiently model rotamer energy.
Diffusion models on molecules and proteins. The Diffusion Probabilistic Model (DPM), which was introduced in [55], has recently gained attention for its exceptional performance in generating images and waveforms [23, 10]. DPMs have been employed in a variety of problems in chemistry and biology, including molecule generation [72, 26, 68, 30], molecular representation learning [40], protein structure prediction [67], pootein-ligane binding [12], protein design [2, 44, 28, 66, 38, 74], motif- moaff-gccaffol [60]. and protein representation learning [76]. In this work, we investigate diffusion models in a new setting, protein side-chain packing, and propose novel autoregressive diffusion model. Note that our definition of autoregressive diffusion model differs from existing works [25].
5 Experiments
5.1 Experimental Setup
Dataset. We use BC40 for training and validation, which BC40 is a subset of PDB database selected by 40% sequence identity [63]. Following the dataset split in [45], there 37266 protein chains for training and 1500 protein chains for validation. We evaluate our models on CASPI3 33 and CASP14. Training set curated so that no structure share sequence similarity with test set by 2 40%.
Baselines. We compare DiffPack with deep learning methods, like AttnPacker [45], DLPacker [46] and traditional methods including SCWRL4 [35], FASPR [27] and RosettaPacker [8]. Details can be found in Appendix H.1
Metrics. We evaluate the quality of generated side-chain conformations using three metries: (1) Angle MAE measures the mean absolute error of predicted torsional angles. (2) Angle Accuracy measures the proportion of correct predictions, considering a torsional angle correct if the deviation is within 20" (3) Atom RMSD measures the average RMSD of side-chain atoms for each residue.
Since predicting surface side-chain conformations is considered more challenging, some results are divided into "Core" and "Surface" categories. Core residues are defined those with at least 20 C3 atoms within a 10Å radius, while surface residues have at most 15 C3 atoms in the same region.
5.2 Side-Chain Packing
Table 1 summarizes the experimental result in CASPI3, our model outperforms all other methods in predicting torsional angles, achieving the lowest mean absolute errors across all four Angle MAE categories (X1, X2, X3, and X4). Additionally, DiffPack shows the highest Angle Accuracy for all residues (69.5%), core residues (82.7%). and surface residues (57.3%), where for surface residue the accuracy is increased by 20.4% compared with previous state-of-the-an model AttnPacker. These results demonstrate that our model is better at capturing the distribution of the side chain torsion angles in both protein surfaces and cores, As for the atom-level side chain conformation prediction, DiftPack clearly outperforms other models in Atom RMSD. Moreover, the intrinsic design of DiffPack ensures that the generated structures have legal bond lengths and bond angles, while previous models in utom-coordinate space (e.g. AttnPacker) can easily generate side chains with illegal bond constraints without prot-processing (as illustrated in Figure 6A).
Similarly, DiffPack outperforms other methods in all Angle MAR categories on the CASP14 dataset (Table 2). It achieves the highest Angle Accuracy for all residues (57.5%), core residues (77.8%), and surface residues (43.5%). Furthermore, DiffPack reports the best Atom RMSD for all residues (0.793 Å), core residues (0.356 Å), and surface residues (0.956 Å).
Despite its superior performance on both test sets, our model, DiffPack, has significantly smaller number of total parameters (3,043.363) compared to the previous state-of-the-art model, AttnPacker (208,098,166) 163), which relies on multiple layers of complex triangle attention. This substantial reduction (68.4x) in model size highlights the efficiency of diffusion-basee approaches like DifiPack, making them more computationally feasible and scalable solutions for predicting side-chain conformations.
7

---

ANGLE MAE %
ANGLE ACCURACY 1
ATOM RMSD Å
Method
X1
X2
X3
X4
All
Core
Surface
All
Core
Surface
SCWRL
27.64
28.97
49.75
61.54
56.2%
71.3%
43.4%
0,934
0,495
1.027
FASPR
27.04
28.41
50.30
60.89
56.4%
70.3%
43.6%
0,910
0.502
1.002
RosettaPacker
25.88
28.25
48.13
59.82
58.6%
75.3%
35.7%
0.872
0,422
1.001
DLPacker
22.18
27.00
51.22
70.04
58.8%
73.9%
45.4%
0.772
0.402
0.876
AttnPacker
18.92
23.17
44.89
58.98
62.1% %
73.7%
47.6%
0.669
0.366
0.775
DiflPack
15.35
19.19
37.30
50.19
69.5%
82.7 %
57.3%
0.579
0.298
0.696
Table I: Comparative evaluation of DiflPack and prior methods on CASP13.
ANGLE MAE 4
ANGLE ACCURACY %
ATOM RMSD Á 4
Method
X1
X2
X3
X4
Core
Surface
All
Core
Surface
SCWRL
33.50
33.05
51.61
55.28
45.4%
62.5%
33.2%
1.062
0.567
1.216
FASPR
33.04
32.49
50.15
54.82
46.3%
62.4%
34.0%
1.048
0.594
1.205
RosettaPacker
31.79
28.25
50.54
56.16
47.5%
67.2%
33.5%
1.006
0.501
1.183
DLPacker
29.01
33.00
53.98
72.88
48.0%
66.9%
33.9%
0,929
0.476
1.107
AttnPacker
25.34
28.19
48.77
51.92
50.9%
66.2%
36.3%
0.823
0.438
1.001
DiffPuck
21.91
25.54
44.27
55.03
57.5% %
77.8%
43.5%
0.770
0.356
0.956
Table 2: Comparative evaluation of DiffPack and prior methods on CASP14.
5.3
Side-Chain Packing Non-Native Backhone
In addition to native backhones, another interesting and challenging problem whether the side chain packing algorithm can be applied to non-native backbone (e.g. backbones generated by protein folding algorithms). In this regard, we extend DiftPack to accommodate non-native backbones generated from AlphaFold2 [32]. Table 3 gives the quantitative result of different algorithms including AlphaFold22s side-chain prediction on the CASP13-FM. test set. All metrics are calculated after aligning the non-native backbone of each residue to the native backbone. As observed, DiffPack achieves state-of sitie--f--h-attt ttthher-t art on most metrics. Notably, DiffPack is the only model that consistently outperforms AlphaFold2 in all metries, showcasing its potential refine AlphaFold2's ' predictions.
5.4 Ablation Study
To analyze the contribution of different parts in our proposed method, we perform ablation studies on CASP13 and CASP14 benchmarks. The results are shown Table
Autoregressive diffusion modeling, We evaluate our autoregressive diffusion modeling approach against two baselines: joint diffusion (11223,1) and random diffusion (xi). Joint diffusion models the joint distribution of the four torsional angles of each residue and performed diffusion and denoising on all angles simultaneously, while random diffusion perturbs one random torsional angle per residue and generates all angles simultaneously during inference. Our approach outperforms both baselines (Table 4). Training loss curves (Figure 5) show that random diffusion has easier time optimizing its loss than joint diffusion, but struggles with denoising all angles at once due to mismatch between training and inference objectives. Our autoregressive scheme strikes a balance between ease of training and quality of generation, achieving good performance. We plot the training curves of the conditional distributions for the four torsional angles in DiftPack, finding that X1 and X2 are easier to optimize and perform better than random diffusion, likely due to discarding atoms from subsequent angles to overcome cumulative perturbation effects. However, training the smaller set of valid residues with X's and Xa is inefficient. Future work should address the challenge of training these angles more efficiently and mitigating cumulative errors in autoregressive models.
Joint (p.2...l
Random
Autoregressive (aval
Autoregressivee ig.)
Dilusion Lokn
Autoregressive (z))
Autorogressive ix))
Autoregressive ixx)
Epach
Figure 5: Training loss curves for different diffusion models.
Inference. Three techniques are proposed in Section 3.5 improve the inference of DiffPack. To evaluate the effectiveness of these techniques, we compare them with three baselines that do not
8

---

ANGLE MAE
ANGLE ACCURACY 1
ATOM RMSD Å
Method
X1
X2
X3
X4
All
Core
Surface
All
Corc
Surface
AlphaFold2
35.20
31.10
51.38
56.95
51.3%
71.5%
38.7%
1.058
0.521
1.118
SCWRL
34.94
30.84
50.45
56.75
51.3%
69.5%
39.0%
1.079
0.550
1.148
FASPR
34.83
30.85
50.60
56.74
50.8%
67.9%
39.8%
1.073
0.573
1.114
RosettaPacker
35.43
31.63
51.33
56.18
50.9%
70.6%
38.5%
1.070
0.526
1.139
DLPacker
34.38
31.57
55.84
67.02
49.9%
69.1%
37.0%
1.032
0.543
1.090
AttaPacker
33.23
31.97
50.53
58.20
51.0%
68.4%
39.1%
0.981
0.512
1.027
DiffPack
31.25
30.17
48.32
56.82
55.5 %
74.3%
41.9%
0.978
0,490
1.056
Table 3: Comparative evaluation CASP13-FM non-native backbones generated by AlphaFold2.
CASP13 ANGLE MAE +
CASP14 ANGLE MAE
Method
X1
X2
X3
X4
X1
X2
X3
X4
DiftPack
15,35
19.19
37.30
50.19
21.91
25.54
44.27
55,03
w/ joint diffusion (X1.2.3.4)
17.14
23.72
35.96
45.30
26.80
34.51
52.77
63.41
w/ random diffusion (x:)
17.11
27.27
44.75
56.42
23.26
32.69
49.21
51.92
w/o multi-round sampling
16.26
23.13
40.60
52.67
23.86
30.71
47.54
55.80
annealed temperature
15.55
22.56
39.32
51.37
22.82
29.61
45.86
55.22
w/o confidence models
16.12
22.92
39.92
50.63
22.86
29.30
45.80
54.08
Table 4: Ablation study on CASP13 and CASP14.
these techniques. For the baselines without multi-round sampling and annealed temperature, we simply resume the sampling procedure in the vanilla diffusion models. For the baseline without confidence models, we only draw one sample from our model instead using confidence models to ensemble multiple samples. As shown in Table 4, the mean absolute error of the four torsional angles increases for the three baselines, demonstrating the effectiveness of our proposed techniques.
5.5 Case Study
DiffPack accurately predict the side-chain conformation with chemical validity. As shown in Figure 6A and Figure 6B. DiffPack accurately predict the side-chain conformation with substantially lower RMSD (0. 1966 and 0.241A) compared with other deep learning methods. Furthermore, DiffPack consistently ensures the validity generated structures, while AttnPacker without post- processing sometimes violates the chemical validity due to its operation atom coordinates.
DiffPack correctly identifies the æ stacking interaction. Accurate reconstruction π stacking interaction between side-chain has traditionally been challenging. Traditional method usually requires a specific energy term for modeling this interaction. Interestingly, DiffPack has shown the ability to implicitly model this interaction without the need of additional prior knowledge (Figure 6C).
RUID o
INAD Roe Srunh
RNSD ground
cip 1.44
Aitepake 2.08
K 2.00
2086
(A) CASP ID: T0951
(B) CASP TO971
(C) CASP ID:
T0949
Figure 6: Case studies on DiffPack. Predictions from different methods are distinguished by color. (A) DiffPack accurately predicts the side-chain conformation. AttPPaker-NNoP produces an invalid glutamic acid structure since O too close to 82 (B) DiflPack accurately predicts the Xi of leucine. (C) DifiPack correctly identifies म- stacking interactions, indicated by dashed lines.

---

Conclusions
In this paper, we present DiffPack, a novel approach that models protein side-chain packing using a diffusion process in torsion space. Unlike vanilla joint diffusion processes, DiffPack incorporates an autoregressive diffusion process, addressing certain limitations. Our empirical results demonstrate the superiority of our proposed method in predicting protein side-chain conformations compared to existing approaches. Future directions include exploring diffusion processes for sequence and side-chain conformation co-generation, optimizing computational efficiency in side-chain packing, and considering backbone flexibility in the diffusion process.
Acknowledgement
We would like to thank Ramanarayan Mohanty from Intel for his support in optimizing the speed of DiffPack. We also extend our appreciation to Zhaocheng Zhu, Sophie Xhonneux and Chuanrui Wang for their constructive feedback on the manuscript. This project supported by Twitter, Intel, the Natural Sciences and Engineering Research Council (NSERC) Discovery Grant, the Canada CIFAR AI Chair Program, Samsung Electromics Co., Ltd., Amazon Faculty Research Award, Tencent AI Lab Rhino-Bine Bird Gift Fund, a NRC Collaborative R&D Project (AI4D-CORE-0)) as well as the IVADO Fundamental Research Project grant PRF-2019-35831139727
References
[1] Rebecca F. Alford, Andrew Leaver-Fay, Jeliazko R. Jeliazkov, Matthew J. O'Meara, Frank DiMaio, Hahnbeom Park, Maxim Shapovalov, P. Douglas Renfrew, Vikram Khipple Mulligan, Kalli Kappel. Jason W. Labonte, Michael S. Pacella, Richard Bonneau, Philip Bradley, Roland L. Dunbrack, Rhiju Das, David Baker, Brian Kuhlman, Tanja Kortemme, and Jeffrey Gray. The rosetta all-atom energy function for macromolecular modeling and design. bioRxiv, 2017.
[2] Namrata Anand and Tudor Achim. Protein structure and sequence generation with equivariant denoising diffusion probabilistic models. arXiv preprint arXiv:2205. /500 2022.
[3] Aleksandra E. Badaczewska-Dart Andrzej Kolinski, and Sebastian Kmiecik. Computational reconstruction of atomistic protein structures from coarse- -eaiined models. Computational and Structural Biotechnology Journal, 18:162 176, 2019.
[4] Minkyung Baek, Frank DiMaio, Ivan V. Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong, Lisa N. Kinch, R. Dustin Schaeffer, Claudia Millán, Hahnbeom Park, Carson Adams, Caleb R. Glassman, Andy M. DeGiovanni, Jose H. Pereira, Andria V. Rodrigues, Alberdina A. A. van Dijk, Ana Ebrecht, Diederik Johannes Opperman, Theo Sagmeister, Christoph Buhlheller, Tea Pavkov-Keller, Manoj K. Rathinaswamy, Udit Dalwadi, Calvin K. Yip, John E. Burke, K. Christopher Garcia, Nick V. Grishin, Paul D. Adams, Randy J. Read, and David Baker. Accurate prediction of protein structures and interactions using a 3-track neural network. Science (New York, N. N.Y.), 373:871 876, 2021.
[5] Federico Baldassarre, David Ménendez Hurtado, Ame Elofsson, and Hossein Azizpour. Graphqa: protein model quality assessment using graph convolutional networks. Bioinfor- matics, 37:360 366, 2020.
[6] Md Shariful Islam Bhuyan and Xin Gao. A protein-dependen side-chain rotamer library. In BMC bioinformatics, volume 12, pages 1-12. Springer, 2011 1.
[7] Yang Cao, Lin Song, Zhichao Miao, Yun Hu, Liging Tian, and Taijiao Jiang. Improved side- chain modeling by coupling clash-detection guided iterative search with rotamer relaxation. Bioinformatics, 27 6:785-90, 2011.
[8] Sidhartha Chaudhury, Sergey Lyskov, and Jeffrey J Gray. Pyrosetta: script-based interface for implementing molecular modeling algorithms using rosetta. Bioinformatics, 26(5)6689-691 2010.
10

---

[9] Chi Chen, Weike Ye, Yunxing Zuo, Chen Zheng, and Shyue Ping Ong. Graph networks as universal machine learning framework for molecules and crystals. Chemistry of Materials, 31 (9)33564-3572, 2019.
[10] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. Wavegrad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.00715 2020.
[11] Glay Chinea, Gabriel Padrón, Rob W. Hooft, Chris Sander, and Gert Vriend. The use of position-speciff rotamers in model building by homology. Proteins: Structure, 23, 1995.
[12] Gabriele Corso, Hannes Stärk, Bowen Jing, Regina Barzilay, and Tommi Jaakkola. Diffdock: Diffusion steps, twists, and turns for molecular docking. International Conference on Learning Representations (ICLR), 2023.
[13] Bowen Dai and Chris Bailey- Kellogg. Protein interaction interface region prediction by geo- metric deep learning. Bioinformatics, 2021.
[14] John R. Desjarlais and Tracy M. Handel. Side-chain and backbone flexibility in protein core design. Journal of molecular biology, 290 1:305-18, 1999.
[15] Samanch Farokhirad, Ryan P Bradley, Arijit Sarkar, Andrew Shih, Shannon E. Telesco, Yingting Liu, Ravindra Venkatramani, David M. Eckmann, Portonovo S. Ayyaswamy, and Ravi Radhakrishnan. 3.13 computational methods related molecular structure and reaction chemistry of biomaterials. 2017.
[16] Guilhem Faure, Aurélie Bornot, and Alexandre G. de Brevern. Protein contacts, inter-residue interactions and side-chain modelling. Biochimie, 90 1:626-39, 2008
[17] Fabian B. Fuchs, Daniel E. Worrall, Volker Fischer, and Max Welling. transforrnss 3d roto-translation equivariant attention networks. ArXiv, abs/2006. 10503, 2020.
[18] Pablo Gainza, Freyr Sverrisson, Frederico Monti, Emanuele Rodola, D Boscaini, MM Bronstein, and BE Correia. Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nature Methods, 17(2):184-192 2020.
[19] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. International conference on machine learning. pages 1263-1272. PMLR, 2017.
[20] Vladimir Gligorijević, Douglas Renfrew, Tomasz Kosciólek, Julia Koehler Leman, Daniel Berenberg. Tommi Vatanen, Chris Chandler, Bryn C. Taylor, Ian Fisk, Hera Vlamakis, Ramnik J, Xavier, Rob Knight, Kyunghyun Cho, and Richard Bonneau. Structure-basd based protein function prediction using graph convolutional networks. Nature Communications, 12, 2021.
[21] Pedro Hermosilla and Timo Ropinski. Contrastive representation learning for 3d protein structures. ArXiv, 0as/2205.15677. 2022.
[22] Pedro Hermosilla, Marco Schäfer, Matëj Lang, Gloria Fackelmann, Pere Pau Vázquez, Barbora Kozlíková, Michael Krone, Tobias Ritschel, and Timo Ropinski. Intrinsic-extrinss convolution and pooling for learning on 3d protein structures. International Conference Learning Representations, 2021.
[23] Jonathan Ho, Ajay Jain, and Pieter Abbecl. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[24] Hervé Hogues, Francis Gaudreault, Christopher R. Corbeil, Christophe Deprez, Traian Sulea, and Enrico O. Purisima. Propose: Direct exhaustive protein-proted docking with side chain flexibility. Journal of chemical theory and computation, 14 9:4938 4947, 2018
[25] Emiel Hoogeboom, Alexey A. Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans. Autoregressive diffusion models. International Conference on Learning Representations, 2022. URL https: https: //tp://///////eee./ee//mm wwwpeeerreeeeeeeeeeeeeeeeeeeeeee wtt..eeee...ooorr///ns//333//./

---

[26] Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning. pages 8867-8887, 2022.
[27] Xiaoqiang Huang, Robin Pearce, and Yang Zhang. Faspr: open-source tool for fast and accurate protein side-chain packing. Bioinformatics, (6(12))3758-766 2020.
[28] John Ingraham, Max Baranov, Zak Costello, Vincent Frappier, Ahmed Ismail, Shan Tie, Wujie Wang. Vincent Xue, Fritz Obermeyer. Andrew Beam, et al. Illmmiaating protein space with a programmable generative model. bioRxiv, pages 2022-12, 2022.
[29] Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations, 2021. URL https://openrevia wttp:eeiwwiwwvwwffreeeeeeee net/forum? id=1YLJDvSx6J4.
[30] Bowen Jing, Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi S. Jaakkola. Torsional diffusion for molecular conformer generation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https / /ppp//vieeeeeeeeee/eeee wwt.:nee/efoooeeoee jf _H.
[31] Peter Bjørn Jørgensen, Karsten Wedel Jacobsen, and Mikkel N Schmidt. Neural message passing with edge updates for predicting properties of molecules and materials. arXiv preprint arXiv:
146, 2018.
[32] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger. Kathryn Tunyasuvunakool, Russ Bates, Augustin Zídek. Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-58)) 2021.
[33] Johannes Klicpera, Janek Groß, and Stephan Günnemann. Directional message passing for molecular graphs. In International Conference on Learning Representations (ICIR), 2020.
[34] Johannes Klicpera, Florian Becker, and Stephan Günnemann. Gemnet: Universal directional graph neural networks for molecules. arXiv preprint arXiv:2106.08900 2021.
[35] Georgii G Krivov, Maxim V Shapovalov, and Roland Dunbrack Improved prediction of protein side-chain conformations with scwrl4. Proteins: Structure, Function, and Bioinformatics, 77(4):778-795, 2009.
[36] Ai-Jun Li and Ruth Nussinov. A set of van der waals and coulombic radii of protein atoms for molecular and solvent-accessibbee surface calculation, packing evaluation, and docking. Proteins: Structure, Function, and Bioinformatics, (0(1):11-122) 1998.
[37] Shide Liang, Dandan Zheng, Chi Zhang, and Daron M. Standley. Fast and accurate prediction of protein side-chain conformations. Bioinformatics, 27:2913 -2114,,,11,, 2011.
[38] Yeqing Lin and Mohammed AlQuraishi. Generating novel, designable, and diverse protein structures by equivariantly diffusing oriented residue clouds. ArXiv, abs/2301. 2023,
[39] Ke Liu, Xiangyan Sun, Jun Ma, Zhenyu Zhou, Qilin Dong, Shengwen Peng, Junqiu Wu, Suocheng Tan, Günter Blobel, and Jie Fan. Prediction of amino acid side chain conformation using a deep neural network. ArXiv, abss/707.0883
2017.
[40] Shengehao Liu, Hongyu Guo, and Jian Tang. Molecular geometry pretraining with SE(3)- invariant denoising distance matching. International Conference on Learning Representations, 2023. URL httph:tt////ttt //tp///epeeeeeeeeee//eeeeeeeeeee net ftrmmoormmmm/ fot.miiddd=TWoooiii
[41] Yi Liu, Limei Wang, Meng Liu, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d graph networks. arXiv preprint arXiv:2102.050/1 2021.
[42] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
12

---

[43] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- solver+:: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.00095 2022.
[44] Shitong Luo, Yufeng Su, Xingang Peng, Sheng Wang, Jian Peng. and Jianzhu Ma. Antigen- specific antibody design and optimization with diffusion- generative models for protein structures. 2022. URL http:::tttpppntiieeeee http////pppreeeeee////e//e wttprreewweeeeeeeeoeeneeeaee www.://oorrrddd-soorreeeeee
[45] Matthew McPartlon and Jinbo Xu. Attmpacker: An end-to-end deep learning method for rotamer-free protein side-chain packing. bioRxiv, pages 8022-03, 2022.
[46] Mikita Misiura, Raghay Shroff, Ross Thyer, and Anatoly B Kolomeisky. Dipacker: deep learning for prediction of amino acid side chain conformations in proteins. Proteins: Structure, Function, and Bioinformatics,
2022.
[47] Ken Nagata, Arlo Z. Randall, and Pierre Baldi. Sidepro: A novel machine learning approach for the fast and accurate prediction of side-chain conformations. Proteins: Structure, 80, 2012.
[48] Noah Ollikainen, René M. de Jong, and Tanja Kortemme. Coupling protein side-chain and backbone flexibility improves the re-design of pootein-ligane specificity. PLoS Computational Biology, 11, 2015.
[49] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling, B(n) equivariant graph neural networks. In International Conference Machine Learning, 2021.
[50] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, Proceedings 15, pages 593-607. Springer, 2018.
[51] Kristof T Schütt, Farhad Arbabzadah, Stefan Chmiela, Klaus R Müller, and Alexandre Tkatchenko. Quantum-chemmcal insights from deep tensor neural networks. Nature munications, 8(1):1-8, 2017.
[52] Kristof T Schütt, Pieter-Jan Kindermans, Huziel E Sauceda, Stefan Chmicla, Alexandre Tkatchenko, and Klaus-Rober Müller. Schnet: A continuous-filte convolutional neural network for modeling quantum interactions. arXiv preprint OrXiv:1706, 170.000666,0, 2017.
[53] Maxim V Shapovalov and Roland L Dunbrack. A smoothed buckbone-dependen rotamer library for proteins derived from adaptive kernel density estimates and regressions. Structure, 19(6)844-888 858, 2011.
[54] David Simoncini, Kam Y. J. Zhang, T. Schiex, and Sophie Barbe. A structural homology approach for computational protein design with flexible backbone. Bioinformatics, 2018
[55] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2266622 2265, 2015.
[56] Vignesh Ram Somnath, Charlotte Bunne, and Andreas Krause. Multi-scale representation learning proteins. In Neural Information Processing Systems, 2022.
[57] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. URL https: https:://ppntee /ttp///eenneeeeeeeeeeee/ee
[58] Freyr Sverrisson, Jean Feydy, Bruno E Corrcia, and Michael M Bronstein. Fast end-to-end learning on protein surfaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15272- 15281, 2021.
[59] Nathaniel Thomas, Tess F. Smidt, Steven M. Kearnes, Lusann Yang, Li Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and transsation-equiivarann neural networks for 3d point clouds. ArXiv, abs/1802.08219, 2018.
13

---

[60] Brian L. Trippe. Jason Yim, Doug Tischer, David Baker, Tamara Broderick, Regina Barzilay, and Tommi Jaakkola. Diffusion probabilistic modeling of protein backbones 3d for the motif- scaffolding problem. In The Fleventh International Conference on Learning Representations, 2023. URL htt.sttps://opeeee http:/::////p////////ee/ petpeerrviee.ne//forrrreeeeeeeee seet/foorum?iteeSxaBxxlEEEEEEE
[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[62] Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang. Deep graph library: A graph-centric, highly-performant package for graph neural networks. arXiv preprint 1rXiv:1909.01333 2019.
[63] Sheng Wang, Jian Peng, Jianzhu Ma, and Jinbo Xu. Protein secondary structure prediction using deep convolutional neural fields. Scientific reports, 6(1):1-11, 2016.
[64] Andrew M. Watkins, Richard Bonneau, and Paramjit Arora. Side-chain conformational preferences govern protein-protein interactions. Journal of the American Chemical Society, 138 33:10386-9, 2016.
[65] Andrew M. Watkins, Timothy W. Craven, P. Douglas Renfrew, Paramjit S. Arora, and Richard Bonneau. Rotamer libraries for the high-resolution design of 3-amino acid foldamers. bioRxiv, 2016.
[66] Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew Borst, Robert Ragotte, Lukas F Milles, et al. Broadly applicable and accurate protein design by integrating structure prediction networks and diffusion generative models. bioRxiv, 2022.
[67] Kevin E Wu, Kevin K Yang, Rianne van den Berg, James Y Zou, Alex X Lu, and Ava P Amini Protein structure generation via folding diffusion. arXiv preprint arXiv:2209.16611 2022.
[68] Lemeng Wu, Chengyue Gong, Xingchao Liu, Mao Ye, and Qiang Liu. Diffusion- molecule generation with informative prior bridges. arXiv preprint Ntt:2200900863. 2022.
[69] Gang Xu, Qinghua Wang, and Jianpeng Ma. Opus-rota3: Improving protein side-chain modeling by deep neural networks and ensemble methods. Journal of chemical information and modeling. 2020.
[70] Gang Xu, Qinghua Wang, and Jianpeng Ma. Opus-rota4: gradient-based protein side-chain modeling framework assisted by deep learning-based predictors. Briefings in Bioinformatics, 23, 2021.
[71] Jinbo Xu and Bonnie Berger. Fast and accurate algorithms for protein side-chain packing, J. ACM, 53:533-557, 2006.
[72] Minkai Xu, Lantao Yu, Yang Song, Chence Shi. Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. In International Con- ference on Learning Representations, 2022. URL http:tppp://pppeieeeeeeee net/ formmidd- PzcvxEMzvQC.
[73] Chen Yanover, Ora Schueler-Furman and Yair Weiss. Minimizing and learning energy functions for side-chain prediction. Journal of computational biology a journal of computational molecular cell biology, 15 7:899-911, 2007.
[74] Jason Yim, Brian I.. Trippe, Valentin De Bortoli, Emile Mathieu, A. Doucet, Regina Barzilay, and T. Jaakkola. Se(3) diffusion model with application protein backbone generation. ArXiv, abs/2302.02277, 2023.
[75] Zuobai Zhang, Minghao Xu, Arian Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Protein representation learning by geometric structure pretraining. In International Conference Learning Representations, 2023.
14

---

[76] Zuobai Zhang, Minghao Xu, Aurélie C. Lozano, Vijil Chenthamarakshan, Payel Das, and Jian Tang. Physics-inspired protein encoder pre-training via siamese sequence-structurr diffusion trajectory prediction. ArXiv, abs/2301. 12068, 2023.
15

---

A More Related Work
Geometric deep learning on biomolecules. Learning protein representations based on 3D geo- metric information is crucial for various protein tasks. Recent advancements have led researchers to develop architectures that preserve properties such as invariance and equivariance for essential transformations like rotation and translation. These approaches have utilized various techniques such as node/atom message passing[19, (1,,,,, 51, 52, 49], edge/bond message passing [31, 9], and directional information [33, 41, 34] to encode molecular graphs in 2D or 3D. Notably, recent models have been generalized to protein 3D structures [20, 5, 29, 22, 21] and protein surfaces [18, 58, 13, 56], demonstrating remarkable performance on diverse tasks, In this work, we utilize state-of-the-ar protein structure encoder, GearNet-Edge [75] to obtain SE(3)-invariant representations for denoising the torsion space.
A.1 Broader Impact
The impact of our work extends well beyond the realm of protein structure prediction, with implica- tions across various fields of biological science and medical research. The ability to accurately predict protein side-chain conformations of vital importance in understanding the intricate functioning of proteins and their interactions with other molecules, including substrates, inhibitors, and drugs.
Drug Design and Discovery: Accurate prediction of side-chain conformations could considerably advance the field of drug discovery and design, where the interaction between proteins and small molecule drugs inhibitors is often determined by the precise conformation of side chains. Improved accuracy in side-chain packing can facilitate more precise predictions of protein-ligand binding sites and affinities, thereby expediting the development of new therapeutics.
Effective application of our method could also revolutionize protein engi- neering, where side-chain packing plays a pivotal role in protein stability, function, and interaction. Enhancing our understanding of side-chain packing will help engineer proteins with desired properties, including altered substrate specificity, increased stability, or novel functionality.
Protein Engineering:
Disease Understanding: Many diseases, such as Alzheimer's, Parkinson's, ss and various cancers, are tied to misfolded proteins or mutations that affect protein structure. Accurate side-chain prediction can thus provide key insights into the structural consequences of such mutations, supporting the development of targeted treatments.
Bioinformatics and Computational Biology: On technical note, our work provides novel perspective for side-chain packing problem by formulating it as diffusion process in the torsional space, which could inspire further innovations in bioinformatics and computational biology. Moreover, given the efficiency of our approach, it is anticipated to facilitate the rapid and scalable analysis of large-scale protein datasets, aiding in the exploration of the proteome.
A.2 Limitation
In this study, we focus exclusively on protein side-chain prediction under the assumption of fixed and highly accurate backbone. Nevertheless, in real-world scenarios, protein backbones may be generated using existing structure prediction methods, which may not provide sufficient accuracy. It is essential to address how our methods can be adapted to accommodate such settings, and this aspect remains an important future direction for our research. Additionally, we intend to explore the application of our methods in various downstream tasks, such as protein-proter interactions and protein engineering tasks, as part of our future research directions.
B
Additional Experiment Results
B.1
Steric Clash
In order to quantify whether the generated structure has severe steric clash, we add an additional metric Steric Clash Count measures the mean number of clash pairs appearing in generated structures. Following previous work [45, 36), the distance threshold between different types of atoms is initially defined by van der Waals radii and further adjusted to account for factors such as H-bond and disulfide
16

---

bridges. A protein atom pair is identified to have steric clash if the inter-atomic distance is within X% of the threshold. Here, X chosen as 100%, 90%, and 80%, representing varying levels of stringency and providing a more comprehensive assessment of steric clashes.
CASP13 CLASH COUNT
CASP14 CLASH COUNT
Method
100%
90%
80%
100%
90%
80%
SCWRL
115.3
20.6
4.6
124.0
24.6
6.5
FASPR
112.8
23.3
5.6
130.5
29.5
8.7
RosettaPacker
73.8
7.9
2.6
100.6
10.1
3,4
DLPacker
64.3
7.3
2.0
74.1
10.5
3.0
AttPPacke--noPP
40.1
1.5
49.5
10.2
DittPack
37.5
4.6
0.9
46.5
6.0
1.1
Table Mean Clash Pair Number of Generation Result
From Table 5, it is clear that DiftPack method outperforms other techniques across levels of stringency for both CASPI3 and CASP14 datasets. This superiority suggests that the DiffPack method produces protein structures with fewer steric clashes, indicating a more realistic and physically plausible model of protein side chain packing.
At the 100% distance threshold, DiffPack generates 67.5% fewer clashes than SCWRL, 66.7% fewer than FASPR, 49.1% fewer than RosettaPacker, and 41.7% fewer than DLPacker for the CASPI3 dataset. Similarly, for the CASP14 dataset, generates 62.5 fewer clashes than SCWRL, 64.3% fewer than FASPR, 53.8% fewer than RosettaPacker, and 37 37.2% fewer than DLPacker.
At the 90% and 80% distance thresholds, the comparative reduction in clashes by DiffPack is also evident. This trend suggests that DiffPack consistently generates structures with fewer steric clashes, which is crucial for generating biologically feasible protein structures.
Interestingly, the AttmPacker-non noPP method also shows a considerable reduction in clashes compared to other methods, especially SCWRL and FASPR, but it is still surpassed by the performance of DiffPack. This suggests that the autoregressive diffusion model used in DiffPack is more capable of managing steric clashes in protein structure generation, demonstrating the effectiveness of this approach.
B.2 Visualization of Sampling Results
(A)
RMGD from iround tium (A)
RMAD from ground sluh (A) AtmFacker 2.20
RMSO from pround mim DiPacker: 2:919
RMSD from ground sium RocetaPadin 2012
DimPali 4G5
DitPacic 8300
AttiFackat: 2421
DiPacken 0.055
RonsttaPadoer 0412
Figure Visualization of sampling results. DiffPack (green), AttnPacker (purple). DLPacker (red), RosettaPacker (orange) are evaluated on T1000 (top), T0954 (middle). T1020 (bottom) cases.
17

---

C Details of Autoregressive Diffusion Models
C.1 Atom groups for each residue type
For completeness, we provide the definition torsional angles and corresponding atom groups for each residue in Table These definitions align with those presented in the AlphaFold2 paper [32].
Residue Type
X1
X2
X3
X4
ALA
ARG
N, C C Cy
C C7, C
Cº. C. N C7, CT, NY, CS
ASN
N, C C CT
C C9,08. C7 O41
ASP
N, C C C
C 09.09,0700
CYS
N. C 00.
GLN
N, C C, Cy
C² C² C
C C?, C o C C7, C7.00.00 O
GLU
N, C C.,C9.C
Oa C*,09, C7,0
GLY
HIS
N, C. C
CO CY, No1
ILE
N, 0 C Cy
Ca C9 C
LEU
N, C* C Cy
Ca C7, C
LYS
N. C* C" CY
Ca C?, C
C Cr,CC.C C C C7,
C7, C,C.
MET
N, C C CT
Ca C7
PHE
N, C. C CY
C7, C61 Co C7, C
PRO
N, C C CY
SER
N, Ca C8,00
THR
N, C C Or1
TRP
N, Ca C
Ca C C7, C91
TYR
N. C C CY
Ca, C,, C7, C61
VAL
C C C71
Table Specification of atom groups defining the torsion angles (X1, X2. X3. X4) for each residue type. Torsion angle exhibiting r-rotation-symmeery is colored as purple. Other side-chain torsion angles exhibit 2 -
C.2 Comparison between numbers of atom clashes of joint and autoregressive diffusion
In Figure 8, we provide comparative illustration of atom clashes in protein structures using different noise schemes in autoregressive diffusion and joint diffusion process.
1
N1
N
11
p(x))
p(x Ixi)
piXs
X1.
pix1 X 13. X3)
P(X: X X))
Figure 8: Comparative illustration of atom clashes in protein structures using noise schemes from autoregressive diffusion process (left) and vanilla joint diffusion process (right). Vanila joint diffusion process add noise to all torsion angles simultaneously, resulting in much more steric clash.
18

---

C.3 Algorithm illustration of DiftPack
Algorithm Training Procedure
Input: training dataset D with protein sequence and structure pairs (S,)), learning rate
Output: trained score networks $6 $82 Sb. for torsional angles X X2,
(1) (2) (3)
1: for epoch 4- to epochmax do
2: for (²) in D do
3: calculate torsional angles X1, X2, Xs. Xa from structure X;; for < 4 do
Train each model separately
5: sample 1 E [0, 1J:
6:
sample Ax. from wrapped normal PHo(-10) with a o of maxi
set X: X: Ax:
8:
generate perturbed structure X' with XI..i-1 and X and discard atoms in X++1.4 groups; predict 8x. = 89 (X, t);
9:
10:
update 8 + 0. eV,o,lox - Vxx1P:o(4xx10)1
11: end for
12: end for
13: end for
Algorithm Inference Procedure
Input: protein sequence and backbone conformation (x)))) number steps N, number rounds R
Output: protein conformation X
I: for i 4- 1 4 do
2: sample X O1, 2T]'":;
Initialize with uniform prior
3: generate structure X with XI..I and discard atoms in X groups;
4: for N to do
let = n/N, (f) = o1-t mma v 22 (Amaax Uminn;
6: for r <- to It do
Multi-round sampling
7:
predict 8x. = 8% (2, I):
8:
draw z from wrapped normal with o² 1/N;
9:
set Ax. = (=()()) (oo(t))/NN(x)) /N) g(t)=;
10:
update Xi Xi + Axi:
Update torsional angles
11:
generate structure X' with XI... and discard atoms in Xi+1.44 groups;
12: end for
13:
end for
14:
end for
return structure generated with X X2+ X3,
D Score Network Architecture
Atom graph construction. We represent protein structure as atom-level relational graph, where atoms serve as nodes and are connected by edges based on three conditions: (1) if they are linked by chemical bond, (2) if one is among the 10-nearest neighbors of the other, and (3) if their Euclidean distance is within 4.5A and their sequential distance is above 2. Each type of edge is treated as different relation, while the node feature is created by concatenating the one-hot features atom and residue types. Following [75], edge features are created by concatenating one-hot features of residue types, relation types, and sequential and Euclidean distances. This graph construction considers both geometric and sequential properties of proteins, providing a comprehensive featurization of proteins.
Embedding layer. the embedding layer, we fuse node features with embeddings for encoding time steps. We use sinusoidal embeddings [61] of time as input for linear layer. Node features are passed through a Multi-Layer Perceptron (MLP) and added with time embeddings to obtain the new node features for the model.
Message passing layer. To learn representations for each node, we perform relational message passing between them [50]. We denote the edge between nodes i and with type as (i.j.r) and set
19

---

of relations as R. We use (0 denote the hidden representation of node at layer 1. Then, message passing can be written as:
1 = 1
-
BN Σ EEEER HR W, W. (n (1)11) Linear
m (Gdir)
(1I)
where W, is the learnable weight matrix for relation type (i) is the neighbor set of j with relation type r, BN denotes batch normalization, and (()) is the ReL.U function. We use m (i, r) to denote the representation of edge (i, j. r), computed through edge message passing. We use abbreviation of the edge (i.j.r). Two edges E1 and are connected if they share a common end node. The type of their connection is determined by the angle between them, which is discretized into 8 bins. The edge message passing layer can be written as:
j, the
= BN Σ
YER W; E EEEENtte)
m-
m (1-1)
(12)
where R' is the set of relation types between edges and (ei) ) is the neighbor set of e with relation type r. After obtaining the hidden representations of all atoms at layer L, we compute the residue representations by taking the mean of the representations of its constituent atoms. Finally, apply an MLP to the residue representations to predict the score.
E Annealed Sampling
As discussed in Section 3.5, vanilla diffusion sampling scheme suffers from over-dispersion problem, which have negative influence in side-chain packing. A promising way to tackle this issue is low- temperature sampling, which involves perturbing the initial distribution p(x) with a temperature
factor T. This results in re-normalized distribution pı (x) 2/2 p(x) + Lowering the temperature value shifts the model's focus towards sampling quality, while increasing it emphasizes diversity. However, strict low temperature sampling is expensive iterative process. Merely up-scaling the score function or down-scaling the noise term in the reverse SDE does not resolve this challenge.
Inspired from [28], proposed modified reverse SDE to approximate the low-temperature sampling. Specifically, the modified reverse SDE is first derived by considering a simplified Gaussian data distribution N' (xo: Mdaat aata ). In this simplified case, show that the modified reverse SDE with the annealed weight dS converges to a low-temperature sampling process that prioritizes quality while maintaining diversity.
Considering the simplified data distribution N (Xo: Maaat 2an ) the dime-dependent marginal density
of noise coist-perturbed distribution by applying VE-SDE forwarding process dx
d
dw is
(x( A (x; AAdma , dana + o² (L)
(13)
The corresponding score function of this noise-perturbed distribution then becomes:
A V lxg
Hddata
(14)
data + o2(1)
Now we consider a tew-trrper sampling scheme, where the original data distribution trans- formed to (x(x) = (x) through a temperature factor T. This results in the variance re-scaled by T, i.e.
px(x) N (x: Mdata To + +o² (t))
(15)
5/ To + oz2(1)
Adata X
(16)
2mm + 02(1)
A where dr = To + o2(
(17)
Strictly solving the re-weight coefficient x, is meaningless and would only apply to the simplified Gaussian distribution. Following [28], we assume the perturbed distribution's 's variance is approx-
imately the maximum variance in the VE-SDE 2 + o² (t) R 2 max- Finally we got re-weight
coefficient:
2
A
max
To2 max
(T 1))
20

---

Confidence Selection
As we formulate the sidechain packing problem in the view generative modeling, we can sample multiple conformation from the learned conformation distribution. Here we train additional confidence module to select the most likely sample. The idea of self-predicting sampling quality is also adopted by AlphaFold [32], where pLDDT estimate the quality of protein structure.
Specifically, we utilize a model bearing the same architecture as that used in score prediction. Following the computation of the residue-level representation, additional MLP head, thereby generating a scalar confidence score for each residue. This confidence score is then trained to correspond with the negative residue-level RMSD. During the inference stage, we sample four conformations for each protein and select the one boasting the highest confidence.
augment the model with an
To show how confidence selection improves performance, we randomly select 10 proteins from test set (each having more than 150 residues) and plot the angle accuracy accordance with an increasing number of samples (Figure 9). Further, we have measured the correlation between predicted confidence score and the negative RMSD, yielding Pearson coefficient 0.664 and Spearman coefficient 0.798, respectively. These statistics demonstrate that the predicted confidence score serves as an effective indicator sample quality, thereby justifying its application in our model.
Oracle Selection
Angle accuracy with MAE < 20.
Confidence Selection
0.70
0.69
0.68
0.67
0.66
10
15
25
30
Number samples
Figure 9: Percentage of accurate angles number of samples increasing. "Oraca Selection" denotes selecting the conformation with lowest RMSD. "Confidence Selection" denotes selecting by trained confidence module.
G Inference Time
G.1 Comparative results of inference time
In this section, we evaluate the inference speed of various methods, categorizing them GPU-based or CPU-based. All GPU-based methods' were evaluated on an NVIDIA RTX A100 40GB GPU, while CPU-based methods were assessed on an AMD EPYC 7513 32-Core Processor 2.60 GHz. For algorithms that facilitate batch processing, the batch size was meticulously chosen and fine-tuned an optimal value, taking into account specific computational requirements.
It's essential to recognize that the inference time for DiftPack is highly influenced by the number of samples in confidence selection and the number of rounds in multi-round sampling. We denote DifiPack-vanill as the model that samples one conformation without confidence selection. As illustrated in Table 7. DiftPack-vannnn vanilla outperforms other GPU-based methods terms of both speed and performance. Although the CPU-based method FASPR achieves superior speed through judicious
'The symbol is used to denote methods that utilize GPU processing.
21

---

70%
Dittack gyntienee
Dresck-uutiouud
68%
66%
(%6
64%
62%
Autrraaker
60%
Roset da
58%
56%
5C
10
10p
10.
10
Inference Speed (5)
Figure 10: Comparision of speed and accuracy between different methods. Inference speed is shown in log-scale
optimization within its tree search algorithm, its restricted angle accuracy limits its applicability in contexts where precision takes precedence over speed especially in sidechain packing.
DiffPack's s flexibility allows for integration with various supplementary techniques as shown in Section 3.5, trading off some speed for enhanced performance. For detailed comparative analysis. we provide plot (shown in Figure 10) that elucidates the interplay between DiffPack's 's speed and corresponding performance. Among the variations of DiffPack, we refer the model amalgamated with confidence selection DiffPack-confdence, and the one further augmented with multi-round sampling as siffPack-multrruuu
G.2 Further acceleration DiffPack
is noteworthy that DiffPack's speed is primarily constrained by the multi-step nature of the denoising process. Recently, significant research efforts [42, 43] have been directed towards accelerating this process. Pursuing integration with these speed-enhancing methods constitutes an exciting avenue for future research and potential further optimization.
Beyond the algorithm itself, DiffPack's s performance is intricately tied to the specific architectural design of EddgNet-Edge, which is utilized to learn the score function within the torsion space. This architecture leverages multi-relational message passing within the protein's 's edge-graph (i.e. line- graph). We have invested some efforts in optimizing the inference speed of GearNet-Edge Edge, based DGL [62]. Our preliminary findings have yielded promising results, reducing the inference time from 1.29s/protein to 0.67s/protein single A100 GPU, and to 3.9s/protein single SPR CPU.
INFERENCE TIME(S) /
ANGLE ACCURACY
SCWRL FASPR
2.71
56.2%
0.10
56,4%
RosettaPacker
217.80
58.6%
DLPacker!
28.50
58.8%
AttnPacker!
6.33
62.1%
DifiPack-vanila'
1.29
67.0%
Table Average Inference Time of Different Methods
H Experimental Setup
H.1 Details of Baselines
SCWRL4. SCWRL4 [35] widely used and well-established method for protein side-chain
conformation prediction. It employs graph-based approach, where side chains are represented as
22

---

nodes in a graph, and edges connect pairs of nodes that have potential steric clashes. The algorithm performs a combinatorial search to find the most probable side-chain conformation with minimal steric clashes and optimal energy, using a bickhone-dependen rotamer library and a statistical potential energy function derived from known protein structures.
FASPR. FASPR (Fast and Accurate Side-chain Prediction using Rotamer libraries) [27] is a prediction method that leverages bockbone-dependers rotamer libraries [53] and a custom-built energy function to efficiently predict side-chain conformations. The method involves using Dead-End Elimination (DEE) and tree decomposition to find the set of rotamers that allows the protein to adopt the Global Minimum Energy Conformation (GMEC). FASPR is designed to achieve balance between computational speed and accuracy, making suitable for large-scale protein modeling applications.
RosettaPacker: RosettaPacker [8] is a component of the Rosetta molecular modeling suite, which is widely recognized for its versatility and accuracy in various applications, including protein structure prediction, protein-protein docking, and protein design. The RosettaPacker algorithm utilizes Monte Carlo simulations combined with a detailed all-atom energy function to sample side- chain conformation and optimize packing interactions. The method is known for its ability to explore broad conformational space, making it particularly effective for challenging prediction tasks that involve significant side-chain rearrangements.
DLPacker. DLPacker [46] is deep learning- based method for side-chain conformation prediction, which employs convolutional neural networks (CNNs) to predict the atom density map of side-chain conformation from the amino acid sequence and backbone conformation. The method incorporates local and non-local context information from the protein structure to make predictions, and it is trained on large dataset of high-resolution protein structures. By leveraging the power of deep learning, DLPacker can capture complex sequence-structure relationships, leading to improved prediction accuracy.
AttnPacker. AttnPacker [45] is a state-of-the-art method for side-chain conformation prediction that utilizes the attention mechanism, powerful technique commonly employed in deep learning architec- tures for tasks involving sequence data. The method incorporates multiple layers of complex triangle attention operations, which enable it to Iearn long-range dependencies and spatial relationships in protein structures, AttnPacker's architecture allows it to model complex protein conformations with high accuracy; however, its large number of parameters makes it computationally demanding compared to some of the other methods.
Results of baselines are directly taken from the AttnPacker paper [45].
H22 Training Details
For torsional diffusion, we utilize the Variance-Exploding SDE (VE-SDE) framework, where 4) 0 and g(() = // The choice of (1) follows the exponential decay defined in previous research [57], given by (t) -1-'a max with min 0.01 0max = n, and E (0, 1). During training, we randomly sample a time step and embed it with sinusoidal embeddings, which then fused with node features. For inference, we use 10 time steps interpolated between min and mmax for generation. At each time step, perform four rounds of sampling. Subsequently, for each target, we randomly select four different predictions from our model and employ our confidence model to choose the best prediction. To encode the protein structure and denoise torsional angles, we employ a 6-layer GearNet-Edge model with a hidden dimension of 128. For edge message passing, the connections between edges are divided into 8 bins based on the angles between them.
All the models are trained using the Adam optimizer with a learning rate of le-4 and batch size of 32. The training process is performed on 4 A 1100 GPUs for 400 epochs.
23


Reviews:
[{"review_id": "z2MfLnrRl3", "reviewer": "Reviewer_cTo4", "summary": "The protein side-chain packing problem consists of predicting the positions of atoms in amino-acid side chains given the backbone structure and residue identities. The paper proposes to do this using a diffusion model that accounts for physical constraints and models side chain structures as joint distributions over torsional angles. Furthermore, they find that generating the torsional angles autoregressively improves the generation quality. DiffPack outperforms existing methods with fewer model parameters and can enhance side chain predictions from AlphaFold2. ", "questions": "- Is there any intuition for choosing the variance exploding SDE instead of variance preserving?\n- How well-calibrated are the confidence predictions?\n- What hardware was used to train the model? How long was it trained for?\n- What downstream biological or engineering applications, if any, can DiffPack do that are not accessible with existing methods?\n- How exactly does the model move between atomic coordinates and score / confidence predictions?\n- When generating, how many different conformations are being generated and passed to the confidence model?\n- What work remains to be done in side-chain packing? Does DiffDock completely solve the problem? \n- How hard would it be to train an order-agnostic autoregressive diffusion here? Would we expect that to produce better samples than a model with a fixed decoding order, as shown here?", "limitations": "- The authors should address where DiffDock still does not solve the side chain packing problem, either in general or in specific cases. ", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper provides a unique solution to the important and well-studied protein side-chain packing problem. The writing is generally clear, and the work is well-contextualized in the literature. The autoregressive diffusion framework is intuitively effective, and the formulation seems correct. This is backed up by strong empirical results when comparing to previous work and in the ablations. Providing confidence scores, reducing the number of parameters compared to previous models, and being able to refine AlphaFold2 predictions will make the method very useful for downstream practitioners. ", "weaknesses": "In general, this is a strong submission with few major weaknesses. \n\n### Major\n\n- Clarity: it's a bit unclear to me how the model moves between atomic coordinates in GearNet and predicting the scores and confidences on each angle. A few more lines of text or a subfigure could be very helpful here. Likewise, it would be nice to have some more details about model size, training hyperparameters, and training hardware. \n- Soundness: The paper talks about confidence scores and shows that they improve generations. It would be better to also have a table or figure showing how well the confidence scores are calibrated. \n- Soundness: are the baselines also given the chance to generate multiple conformations and then to have a confidence model pick the best one? If not, the comparisons are not quite one-to-one. \n- Significance: While I appreciate the case studies shown in 5.5, the paper would be stronger with more context here. What downstream biological or engineering applications, if any, can DiffPack do that are not accessible with existing methods?\n\n### Minor\n\n- There are a few minor points on clarity: The standard in the field seems to be DDPM (denoising diffusion probabilistic models) instead of DPM, as used in line 252. In Figure 4, the legend says blue and yellow, but I see blue and red in the figure. In Figure 5, the colors for the four autoregressive curves are very difficult to tell apart -- it might help to also vary the linestyle. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your time and review! We think that the majority of your detailed concerns about our method are covered in the Appendix. We'll revise the order as you've recommended to prevent any confusion. Here's our response to your concerns.\n___\n**Q1: Clarity about GearNet Score Prediction**\n\nFor atom representations, we build a graph combining bonds and 3D coordinates. GearNet-Edge is used on this graph. Residue representations are obtained by averaging atom embeddings. These feed into an MLP for score function and confidence score output. We've addressed this in Sec. 3.4 and will provide more details.\n___\n**Q2: Calibration of Confidence Scores**\n\nWe've done extra experiments to assess the confidence score calibration in the global rebuttal, as per your suggestions.\n___\n**Q3: Are the baselines also given the chance to generate multiple conformations？ If not, is the comparison fair?**\n\nThanks for noting this! In the benchmark, traditional methods like SCRRL4, FASPR, and RosettaPacker produce multiple conformations, choosing the best based on an energy function. In contrast, deep learning methods except DiffPack are end-to-end regression models, yielding one pose at a time.\n\nIt’s worth noting that DiffPack is the first deep learning method that solves the problem with generative modeling. That’s also one reason that makes our method more effective than others – we capture **the whole conformation distribution** instead of “**the mean conformation**”. As such, predicting side-chain conformation essentially involves sampling from high probability regions of the distribution, while confidence selection helps in identifying these regions.\n\nIn contrast, regression-based models such as AttnPacker and DLPacker directly predict the final pose as a result of their regression-based intrinsic design. So there is no need for these models to generate multiple conformations\n\nTo further address your concern about potentially unfair comparisons between regression-based and generative models, we've conducted two additional experiments for a more direct comparison shown in attachment Tab. 1:\n\n1. **Limiting DiffPack’s generation samples to 1**: We constrain DiffPack to generate only one sample, excluding confidence score selection. From the experiment results below, we can see that our models still outperforms other methods by large margin. This is due to the fact that sampling from a probability distribution is typically biased towards regions of higher probability density, without any additional selection module.\n2. **Allowing other deep learning based models to generate multiple samples**: Despite the intrinsic design of regression-based models not supporting multiple conformation generation directly, varied seed and initialization can lead to different generation results. In this experiment, we enabled these models to generate multiple samples with oracle selection. Even when selecting the best predicted pose based on the ground truth RMSD, the overall results remained largely unchanged, with DiffPack outperforming the competition.\n\nWe hope this detailed explanation and the additional experiments satisfactorily address your concern!\n___\n**Q4: What downstream biological or engineering applications, if any, can DiffPack do that are not accessible with existing methods?**\n\nEssentially, DiffPack does not directly solve a new problem, but instead enhances sidechain packing with significant performance improvement. This allows for applications in areas requiring accurate modeling of sidechain conformation, such as protein-protein docking and protein mutation prediction.  Further details on potential applications are available in App. A.1.\n___\n**Q5: What's the intuition behind choosing the variance exploding SDE instead of variance preserving?**\n\nThe choice of a variance exploding stochastic differential equation (VE-SDE) over variance preserving (VP-SDE) is primarily due to the nature of non-Euclidean torsion space $\\mathbb{T}$, where defining an origin isn't straightforward. In traditional Euclidean spaces, the drift term $f(x,t) = -1/2x$ of VP-SDE can target the origin, but in $\\mathbb{T}$, this isn't easily feasible. With VE-SDE, the drift term is neglected, making it a favored option in these contexts. While other options exist, VE-SDE is one of the simplest and most effective solutions available.\n___\n**Q6: What hardware was used to train the model? How long was it trained for?**\n\nThe training of our model was conducted on 4xA100 GPUs. The training spanned over a total of 400 epochs, with the entire process taking approximately 4 days for each model. Specific training details are included in App. F.2. \n___\n**Q7: How many different conformations are being generated and passed to the confidence model?**\n\nOur approach samples four different conformations from the diffusion models during sampling, which are subsequently evaluated by the confidence model. More details are provided in App. F.2.\n___\n**Q8: What work remains to be done in side-chain packing? Does DiffPack completely solve the problem?**\n\nRegarding side-chain packing, our model is accurate for various applications, but there's room for improving side-chain conformation prediction. For instance, enhancing results on non-native AlphaFold2 backbones is crucial, especially for generating complete atomic conformations in de novo designed sequences.\n___\n**Q9: Is it promising to train an order-agnostic autoregressive diffusion here?**\n\nWhile an order-agnostic autoregressive diffusion model has been proposed in previous work[1], we believe it is not suitable for the domain of sidechain packing. As elucidated in Sec. 3.3, the challenge of cumulative coordinate displacement can be mitigated only by fixing the order from $\\chi_1$ to $\\chi_4$. Without this specific ordering, the underlying issues would persist and likely hinder the training process of the model.\n\n[1]. Hoogeboom et al. \"Autoregressive diffusion models.\" arXiv, 2021."}, {"review_id": "md0XodRmvF", "reviewer": "Reviewer_Npgs", "summary": "This paper focuses on the task of sidechain packing in proteins, where one wishes to predict the positions of the side-chain atoms given the positions of the backbone atoms. To this end, the main contribution of the paper is DiffPack, an extension of recently proposed torsional diffusion models to the side-chain packing task. This is achieved by combining three aspects:\n\ni) autoregressive sampling of the four side-chain torsion angles, with a separate diffusion model trained for each, conditioned on the previous torsion angles\n\nii) multi-round and annealed temperature sampling to improve quality of generated samples\n\niii) Modifications to the transition kernel formulation on the torus for residues where the torsion angles have a periodicity of $\\pi$. \n\nExperimental performance compared to various baselines for side-chain sampling showcases the improved performance offered by DiffPack.", "questions": "See the above section for questions.\n\nEdit: I have read the authors rebuttal, and post the discussion phase, still maintain my assessment about the paper.", "limitations": "The authors have addressed limitations associated with their work", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is very well written, clear and easy to understand. The technical choices made throughout the paper are sound and well motivated, and the application area considered is well suited for the torsional diffusion framework. \n\n2. The authors compare their method to a variety of baselines for side-chain packing and showcase improved experimental performance of their method.", "weaknesses": "1. The technical contributions are largely incremental - the formulations regarding torsional diffusion have already been well explored in previous recent papers. \n\n2. Some questions / clarifications regarding the experimental evaluation:\n\n    * It is unclear to me the benefits offered by using auto-regressive models for each torsion angle as opposed to joint diffusion. From Table 1, 2 and 4, one can see that, for about 4$\\textdegree$ variation in MAE of $\\chi_1$, the RMSD drops by 0.1. From Table 4, the $\\chi_1$ MAE between the autoregressive model and the joint diffusion model is about 2$\\textdegree$, giving a much smaller corresponding drop in RMSD. Using auto-regressive models definitely seems to accelerate training / convergence as noted in Fig 5. However, the errors in discretization during sampling, idealization of bond lengths and angles when reconstructing coordinates could end up reducing the improved training performance of the autoregressive models, and eventually offer similar values in RMSD. \n\n    * Could the authors add a table regarding the run times associated with the evaluations of the different deep learning baselines? How many samples are generated for each protein before selection with the confidence model?\n\n3. An anonymous link for the code submission is not provided, making it harder to verify reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your suggestions! Here is our response to your concern.\n___\n**Q1: Potential Concern in Incremental Technical Contribution**\n\nThanks for your review! Formulation of diffusion models on Riemannian manifold(e.g. torsion space) was first proposed by Valentin De Bortoli[1] and further applied in various domains such as molecule conformation generation[2], antibody optimization[3], molecule docking[4] and protein backbone design[5]. However, to our knowledge, **DiffPack represents the first work of Riemannian diffusion model in the domain of protein side chain packing**, which significantly improves the prediction accuracy, as demonstrated in our experiments.\n\n1. **Compared with other work where the Riemannian diffusion model is applied, protein side-chain packing presents a much higher degree of freedom.** Earlier studies have focused on either small molecules [2][4] (averaging ~50 atoms) or residue-level proteins [5] (averaging ~200 residues). In our case, we had to model an average of 3000 atoms. Even though we constrained the degree of freedom by choosing the torsion space as the model output, this complexity still far exceeds previous scenarios.\n\n   Coupled with the issues discussed in Section 3.3 (Cumulative Coordinate Displacement and Excessive Steric Clash), this complexity posed a significant challenge to directly train and sample from a diffusion model in this space. As shown in Figure 5, directly optimizing a joint diffusion model suffers from the underfitting problem. To address this problem specific challenge, autoregressive modeling is introduced to factorize the joint distribution into a product of conditional distribution. This significantly alleviated the under-fitting problem.\n\n2. **Compared with previous work in protein side-chain packing, DiffPack is the first method modeling the side-chain problem in a generative manner.** Previous regression based methods (AttnPacker, DLPacker) tend to predict the “mean conformation”, while our proposed method could capture the entire distribution of side-chain conformation. Experiments have validated the effectiveness of our generative modeling.\n\n[1]. De Bortoli, Valentin, et al. \"Riemannian score-based generative modelling.\" Advances in Neural Information Processing Systems 35 (2022): 2406-2422.\n\n[2]. Jing, Bowen, et al. \"Torsional diffusion for molecular conformer generation.\" Advances in Neural Information Processing Systems 35 (2022): 24240-24253.\n\n[3]. Luo, Shitong, et al. \"Antigen-specific antibody design and optimization with diffusion-based generative models for protein structures.\" Advances in Neural Information Processing Systems 35 (2022): 9754-9767.\n\n[4]. Corso, Gabriele, et al. \"Diffdock: Diffusion steps, twists, and turns for molecular docking.\" International Conference on Learning Representations (2023).\n\n[5]. Wu, Kevin Eric, et al. \"Protein structure generation via folding diffusion.\" (2022).\n___\n**Q2: Issues regarding the benefit of autoregressive modeling**\n\nWe appreciate your insights and would like to elaborate on the advantages of autoregressive modeling in our context. Protein sidechain packing indeed poses a formidable challenge due to its vast degree of freedom. When training a model on the joint distribution directly, the computational complexity can grow exponentially with the number of variables due to the \"curse of dimensionality\". Besides, the additional challenges(Cumulative Coordinate Displacement and Excessive Steric Clash) also impose negative influence for directly modeling the joint diffusion process. All of the above will finally lead to an underfitting problem. So in our work, autoregressive is introduced to alleviate this problem. By decomposing the joint distribution into conditional distributions, we simplified the denoising process, enabling us to capture the potential spatial dependencies between $\\chi_k$ and $\\chi_{i<k}$.\n\nRegarding ablation study performance, we respectfully disagree with the reviewer about the improvement achieved by our method. It can be clearly observed that the MAEs of all four angles significantly drop after replacing joint diffusion with autoregressive diffusion on CASP14. Additionally, we provide Atom RMSD results in Table A, illustrating a reduction of approximately 0.12 in RMSD, which is already significant in sidechain packing.\nTable A: Ablation Study on CASP14.\n|#Method|$\\chi_1$ MAE|$\\chi_2$ MAE|$\\chi_3$ MAE|$\\chi_4$ MAE|Atom RMSD|\n|:----:|:----:|:----:|:----:|:----:|:----:|\n|**DiffPack**|**21.91**|**25.54**|**44.27**|**55.03**|**0.770**|\n|w/ joint diffusion|26.80|34.51|52.77|63.41|0.893|\n\nFinally, it's worth noting that autoregressive models aren't necessarily \"better\" or \"worse\" than other types of models. They are simply better suited to certain types of problems\n___\n**Q3: Issues regarding running times**\n\nThanks for your valuable suggestions! We have included the running time information in Table 2 of the attachment. In short, DiffPack is the best model for balancing speed and accuracy. You can find more details in the global rebuttal section and attachment.\n\n| Methods | Inference Time (s) ↓ | Angle Accuracy ↑ |\n|---|:---:|---|\n| SCWRL | 2.71 | 56.2% |\n| FASPR | **0.10** | 56.4% |\n| RosettaPacker | 217.80 | 58.6% |\n| DLPacker* | 28.50 | 58.8% |\n| AttnPacker* | 6.33 | 62.1% |\n| DiffPack_vanila* | 1.29 | **67.0%** |\n___\n**Q4: How many samples are generated for each protein before selection with the confidence model?**\nThanks for your questions! We sample 4 conformations for each protein. The value is chosen to balance the speed and accuracy as shown in the attached Figure 1. Additional details of inference process can be found in Appendix F.2\n___\n**Q5: Issue about code and reproducibility**\nWe have attached the anonymous code link(https://anonymous.4open.science/r/DiffPack-DED9/). Thank you for raising the concern!\n"}, {"review_id": "ZT6w7x1ZCB", "reviewer": "Reviewer_KyoN", "summary": "The paper proposes DiffPack a torsional diffusion model to learn side chain placements. In particular, the authors presents a few modification to vanilla torsional diffusion models that improve the empirical results obtaining strong empirical performance.", "questions": "The statement “We focus exclusively on protein side-chain prediction under the assumption of a fixed and highly accurate backbone” seems to contradict with the AlphaFold2 results presented in the main text.\n\n“We extend DiffPack to accommodate non-native backbones generated from AlphaFold2”. How is this extension done? What is the model trained on?\n\nSection B.1 what is the distance threshold?", "limitations": "In some of the tables the (e.g. Table 3) the bolding of the best number is wrong and biased to the proposed method.\n\nI believe that the discussion on the number of parameters is misleading as this is not a very useful measure of computational feasibility. Instead the authors should compare the methods by runtime (and potentially memory cost).", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The authors propose a number of modifications to the vanilla application of torsional diffusion to side chain predictions and these have enables them to obtain state-of-the-art performance on this important task.", "weaknesses": "Although the empirical results are very good, not many parts of the method are significantly novel. Moreover, among the few modifications that the authors made the presentation of them with respect to the rest of the field and the justification should be improved:\n\n1. Annealed temperature sampling: the low temperature sampling procedure derived from the assumption of the Gaussian distribution seems to be very similar to the one presented in Ingraham et a. (2022) [20]. The authors should include the reference of this technique in the relevant section and clarify the difference between the methods. Moreover, I believe the statement “ideally the sampling process converges to the global optimum when T→0” is wrong or at least misleading (does not specify what the ideal situation is).\n\n2. Multi-round sampling: this is related to the mixing Langevin steps that multiple papers have proposed before (e.g. Ingraham et a. [20]). However, unlike in those approaches, the authors do not limit themselves to a Langevin step that preserves the expected distribution but instead keep the “ODE-term” making therefore very unclear what the process is “theoretically” doing in this process. The authors should provide further discussion of these challenges.\n\n3. Autoregressive diffusion: some of the comparisons (excluding the inference performance) used to motivate autoregressive diffusion versus vanilla are misleading: \n - Figure 7: presents the number of steric clashes during the diffusion process not of the generated structures. This is misleading because these steric clashes may not matter or even help the model to detect them to avoid them. As the autoregressive method (that removes following atoms) artificially removes these steric clashes and this may prevent it to reason about them when generating the distributions. The number of steric clashes should be compared based in the resulting generated structures. \n - Figure 5 compares the loss values of different methods however these are not comparable. E.g. one of the reason for the loss of X1 is lower than X4 might just be fact that the entropy of the distributions of X1 (which is correlated to the lower bound of the score matching loss) is lower than that of X4. Similarly this makes the comparisons of the losses of the different methods not comparable (e.g. the entropy of the conditional distribution obtained by the autoregressive is lower than that of the unconditional obtained by the joint loss).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful questions. We'd like to first clarify that our work's main focus is on formulating a new approach to the sidechain packing task, rather than proposing general improvements to the diffusion process. Below are our specific responses to your queries.\n___\n**Q1: Justification about annealed temperature sampling**\n\nThe annealed temperature sampling used in our work indeed has similarities to Ingraham et al. (2022) [28]. While the core concept is borrowed from this previous work, we specifically derived the VE-SDE version of the annealed weight under the same assumption. Although initially referenced in Appendix E, we understand the explanation may have been unclear. Therefore, we'll enhance the relevant section in the main text for better clarity, in line with your valuable suggestion.\n\nRegarding the statement, “ideally the sampling process converges to the global optimum when T→0”, we understand how this can be misconstrued, and we appreciate your attention to the detail.  What we intended to convey is that theoretically, the distribution $p_T(\\mathbf{x})$ would collapse to a Dirac delta function at the global optimum as **temperature $T$ approaches zero**. However, this is only an approximation in practical applications. We will rewrite this section to eliminate any ambiguity.\n\nThanks again for your valuable suggestions. \n___\n**Q2: Justification about multi-round sampling**\n\nThanks for pointing this out! Upon a closer review of the literature[1], we found that the multi-round sampling is indeed closely connected to langevin dynamics.\n\nSince we utilize VE-SDE which neglect the drift term, the reverse VE-SDE differs from mixing langevin dynamic[1] by only a coefficient $\\sqrt{2}$. This coefficient may be interpreted as a temperature scaling factor, and we hypothesize that it contributes to the improvement in the original multi-round sampling.\n\nBut as you mentioned, the proposed multi-round sampling is indeed a special case of the general mixing langevin dynamics. In light of this, we further conducted a new experiment with the more general mixing langevin dynamics, observing improvements in key metrics, such as an increase in Angle Accuracy from 69.5% to 70.1% in CASP13. These new findings will be incorporated into the revised version of our paper.\n\nYour feedback has provided invaluable guidance, enabling us to refine our method and elucidate the underlying mechanisms. We are sincerely grateful for your contributions to the advancement of this work. Thank you again!\n\n[1] Song, Yang, et al. \"Score-based generative modeling through stochastic differential equations.\" ICLR 2020.\n___\n**Q3:  The number of steric clashes during the diffusion process is misleading.**\n\nThank you for raising this concern!  We appreciate your insights and would like to clarify that Figure 7 primarily focuses on the training phase rather than the inference process. While it’s true that the steric clashes might be managed by a well-trained model during inference, it is important to emphasize that they pose significant challenges during the training phase. Autoregressive diffusion is designed to alleviate these training difficulties, which might not be immediately apparent during the inference period. \n\nYour suggestion to compare steric clashes based on the resulting generated structures is great. We evaluated the clash number within 90% distance threshold, finding that joint diffusion did not reduce more clash pairs (Joint: 6.9 vs Autoregressive: 6.0). Furthermore, as shown in Table 4, the substantial improvements in practical applications achieved by autoregressive diffusion over joint diffusion demonstrate that autoregressive diffusion is a better choice in practical application.\n___\n**Q4: Loss values of different methods in Fig. 5 are not comparable.**\n\nThank you for bringing this to our attention! We acknowledge that the loss values comparison in Figure 5 might indeed be misleading due to differing entropies as lower bounds across the methods, rendering the losses not directly comparable. We will certainly revise the figure in line with your valuable suggestions.\n\nIt's also worth emphasizing that the average training loss of autoregressive diffusion is bounded by the entropy of distribution $p\\left(\\boldsymbol{\\chi}_1, \\boldsymbol{\\chi}_2, \\boldsymbol{\\chi}_3, \\boldsymbol{\\chi}_4\\right)$, as is the loss of joint diffusion. While the original comparison might have been imprecise, we believe that a rough comparison can still shed light on the relative optimization or training efficiency of autoregressive modeling versus joint diffusion. Coupled with the experimental results shown in Table 4, these findings affirm the efficacy of autoregressive modeling.\n___\n**Q5: The statement “We focus exclusively on ….” seems to contradict with the AlphaFold2 results presented.**\n\nThank you for raising the concern! The additional experiments with AlphaFold2 aim to showcase potential usages of our method on non-native backbones, which does not change the main focus of the paper. We will tune our statement in the revised version. \n___\n**Q6: How to extend DiffPack on non-native backbones?**\n\nWe simply adapt the DiffPack model trained on native backbones to AF-predicted backbones without any retraining. The results actually showcase the generalization ability of the proposed method. We will clarify this point in the revised version.\n___\n**Q7: In Sec. B, what is the distance threshold?**\n\nFollowing Matthew et al.[45], the distance threshold between different types of atoms is initially defined by van der Waals radii and further adjusted to account for factors such as H-bond and disulfide bridges. We will add more details to related sections.\n___\n**Q8: Issues about computational feasibility**\n\nThanks for your question! We’ve conducted an experiment for the inference speed between different methods. Details can be found in attached materials and global rebuttal block."}, {"review_id": "WBbR6tzfkR", "reviewer": "Reviewer_uGQU", "summary": "The paper proposes DiffPack, a torsional diffusion model that accurately predicts the conformation of protein side-chains given their backbones. DiffPack learns the joint distribution of side-chain torsional angles by diffusing and denoising on the torsional space. To avoid issues arising from simultaneous perturbation of all four torsional angles, the paper proposes autoregressively generating the four torsional angles and training diffusion models for each torsional angle. The method achieves remakrable improvements in angle accuracy on benchmark datasets and enhances side-chain predictions in the AlphaFold2 model. \n\n", "questions": "Please see the above weaknesses.", "limitations": "The authors didn't discuss the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "-The paper is well-written and easy to follow, with a clear description of the proposed DiffPack model. While the idea of using torsional diffusion is not entirely novel, the paper builds on the work of Jing et al. at NeurIPS 2022 on small molecule torsion diffusion and applies it to the problem of protein side-chain packing. \n\n-The paper's technical contributions lie in the development of a torsional diffusion model that considers the restrictions imposed by covalent bond lengths and angles, and the autoregressive generation of torsional angles. \n\n-The paper's results demonstrate the potential of DiffPack in advancing protein structure prediction and design. \n\n", "weaknesses": "-Is there a metric available for assessing the overall conformational plausibility of the generated structures? This question is relevant because in the context of small molecule conformation generation, deep learning methods have been shown to produce conformations with lower overall RMSD, but there are still significant challenges with conformational plausibility (e.g. generated benzene rings are not necessarily planar). t would be beneficial to have a metric that takes into account both the geometric accuracy and conformational plausibility of the generated structures to ensure that they are reliable for downstream applications.\n\n-It would be helpful to include a comparison of model run times between DiffPack and other methods such as Attenpack and force field-based methods. This would enable us to better assess the potential gaps in downstream applications when making proteomic predictions with these methods. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your suggestions! Here is our response to your concern.\n___\n**Q1: Is there a metric available for assessing the overall conformational plausibility of the generated structures?**\n\nThis is an interesting question! Assessing the overall conformational plausibility of generated structures, particularly in complex biological systems like proteins, is a multifaceted task. Researchers usually employ two kinds of  factors to evaluate the plausibility of a conformation.\n\n1. **Steric Clash Assessment**: Steric clash is the most direct way for assessing the plausibility of generated conformation. Conformation which violates chemical constraints will usually impose unnatural overlap between atoms in 3D space. \n\n2. **Physical Energy Assessment**: The total potential energy of a conformation can also be used as a proxy for its plausibility. Conformations that are at or near a local minimum of the energy landscape are typically more plausible. \n\nWe choose steric clash assessment to evaluate the plausibility of a conformation in our work. This measurement does not rely on manually selected energy functions, making it a more straightforward choice. Specifically, the distance threshold between different atoms are first predefined according to van der Waals radii, and further adjusted to account for different chemical interaction(e.g. H-bond and disulfide bridges). We classify the conformation as a steric clash if the pair distance between atoms violates this threshold. Details can be found in Appendix B.\n\nFurthermore, it's essential to highlight that our proposed method, which operates in torsion space, has distinct advantages over models operating in Cartesian space, such as AttnPacker. By treating the functional group as a whole in torsion space, intra-group implausibilities (e.g. non-planar generated benzene rings) are unlikely, leaving only inter-group implausibilities. This inherent characteristic of our approach emphasizes its superiority in ensuring conformational plausibility.\n___\n**Q2: It would be helpful to include a comparison of model run times between DiffPack and other methods such as Attenpack and force field-based methods.**\n\nThanks for your valuable suggestions! We recognize the importance of comparing inference times between our method and other established approaches, and we've included a detailed comparison in global rebuttal block’s attachment. A summary of the experiment results is presented below:\n\n| Methods | Inference Time (s) ↓ | Angle Accuracy ↑ |\n|---|:---:|---|\n| SCWRL | 2.71 | 56.2% |\n| FASPR | **0.10** | 56.4% |\n| RosettaPacker | 217.80 | 58.6% |\n| DLPacker* | 28.50 | 58.8% |\n| AttnPacker* | 6.33 | 62.1% |\n| DiffPack_vanila* | 1.29 | **67.0%** |\n\nAs evident from the table, DiffPack_vanila outperforms other GPU-based deep learning methods (denoted by *) in terms of speed while achieving the highest angle accuracy. Although the CPU-based method FASPR is quicker due to efficient optimizations in the tree search process, its prediction accuracy lags behind. In the sidechain packing task, where prediction accuracy is prioritized over inference time, DiffPack's performance becomes particularly significant. \n\nThis comparison highlights DiffPack’'s balance between computational efficiency and prediction accuracy, showcasing its viability and strength as a choice for this specific application. The comprehensive analysis can be found in the global rebuttal block for further insights. We will also include it in the revised version of the paper.\n"}]

Author rebuttal:
We would like to extend our sincere gratitude for your time and thorough review of our paper. Your insightful suggestions and comments have provided us with valuable perspectives, enabling us to enhance the quality of our work. As there is some common issues raised, we choose to respond to them in the global block.
___
**Common Issue 1: Reproducibility of DiffPack**

Acknowledging the necessity for reproducibility, we have attached the anonymous code (https://anonymous.4open.science/r/DiffPack-DED9/) at this stage. We believe this will address concerns and foster reproducibility within this field.
___
**Common Issue 2: Calibration of Confidence Scores in Confidence Selection**

Several reviewers have inquired about the specifics of confidence selection. To clarify, we conducted additional experiments to demonstrate how confidence selection elevates sampling quality and to assess the calibration of the predicted confidence score.

Specifically, we selected 10 proteins randomly from the test set (each with over 150 residues) and plotted angle accuracy against an increasing number of samples. As depicted in Figure 1, the sampled conformation quality consistently improves with an increased number of samples, notably when the number of samples is <= 5, substantiating the efficacy of our proposed confidence selection.

To further demonstrate how well-calibrated is the predicted confidence score, we have measured the correlation between the predicted confidence score and the negative RMSD, yielding **Pearson coefficient 0.664** and **Spearman coefficient 0.798**, respectively. We plan to present these results in the revised version of paper following your valuable suggestions.
___
**Common Issue 3: Inference Speed and Computation Feasibility in DiffPack**

The computational feasibility of DiffPack has garnered attention from many reviewers.We  performed supplementary experiments to benchmark the inference speed of different methods. Methods are categorized as GPU-based or CPU-based. All GPU-based methods(DLPacker, AttnPacker, DiffPack) were evaluated on an NVIDIA RTX A100 40GB GPU, while CPU-based methods were assessed on an AMD EPYC 7513 32-Core Processor @ 2.60 GHz. For algorithms that facilitate batch processing, the batch size was meticulously chosen and fine-tuned to an optimal value, taking into account specific computational requirements.

It's essential to recognize that the inference time for DiffPack is highly influenced by the number of samples in confidence selection and the number of rounds in multi-round sampling. We denote DiffPack-vanilla as the model that samples one conformation without confidence selection. As illustrated in attached Table 2, DiffPack-vanilla outperforms other GPU-based methods in terms of both speed and performance. Although the CPU-based method FASPR achieves superior speed through judicious optimization within its tree search algorithm, its restricted angle accuracy limits its applicability in contexts where precision takes precedence over speed especially in sidechain packing.

DiffPack's flexibility allows for integration with various supplementary techniques as shown in Section 3.5, trading off some speed for enhanced performance. For a detailed comparative analysis, we provide Figure 2 that elucidates the interplay between DiffPack's speed and corresponding performance. Among the variations of DiffPack, we refer to the model amalgamated with confidence selection as DiffPack-confidence, and the one further augmented with multi-round sampling as DiffPack-multiround.

It is noteworthy that DiffPack's speed is primarily constrained by the multi-step nature of the denoising process. Significant research efforts[1][2] have been directed towards accelerating this process. Pursuing integration with these speed-enhancing methods constitutes an exciting avenue for future research and potential further optimization.

[1]. Song, Jiaming, Chenlin Meng, and Stefano Ermon. "Denoising diffusion implicit models." arXiv preprint arXiv:2010.02502 (2020).
[2]. Lu, Cheng, et al. "Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models." arXiv preprint arXiv:2211.01095 (2022).

___

Once again, we wish to express our profound gratitude for the meticulous review and constructive feedback. Your guidance has been instrumental in refining our work, and we welcome any further questions or comments you may have.


Meta-review: This is a well-executed paper. It studies the known protein design problem of sidechain packing and brings forth a sensical approach based on recent advances in denoising diffusion, namely torsional diffusion, autoregressive generation, and temperature annealing. The experimental results convincingly demonstrate that the proposed approach can place side chains more precisely than previous baselines. A main criticism one could place is that the methodological components that make this work are mostly known and this paper's contribution lies mostly in combining them in light of the problem at hand. Nevertheless, all reviewers agree that the positives outweigh the negatives, urging for acceptance.
Decision: Accept (poster)