SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Residue Type	\( \chi_1 \)	\( \chi_2 \)	\( \chi_3 \)	\( \chi_4 \)
ALA	-	-	-	-
ARG	N, \( C^\alpha, C^\beta, C^\gamma \)	\( C^\alpha, C^\beta, C^\gamma, C^\delta \)	\( C^\beta, C^\gamma, C^\delta, N^\epsilon \)	\( C^\gamma, C^\delta, N^\epsilon, C^\zeta \)
ASN	N, \( C^\alpha, C^\beta, C^\gamma \)	\( C^\alpha, C^\beta, C^\gamma, O^{\delta 1} \)	-	-
ASP	N, \( C^\alpha, C^\beta, C^\gamma \)	\( C^\alpha, C^\beta, C^\gamma, O^{\delta 1} \)	-	-
CYS	N, \( C^\alpha, C^\beta, S^\gamma \)	-	-	-
GLN	N, \( C^\alpha, C^\beta, C^\gamma \)	\( C^\alpha, C^\beta, C^\gamma, C^\delta \)	\( C^\beta, C^\gamma, C^\delta, O^{\epsilon 1} \)	-
GLU	N, \( C^\alpha, C^\beta, C^\gamma \)	\( C^\alpha, C^\beta, C^\gamma, C^\delta \)	\( C^\beta, C^\gamma, C^\delta, O^{\epsilon 1} \)	-
GLY	-	-	-	-
HIS	N, \( C^\alpha, C^\beta, C^\gamma \)	\( C^\alpha, C^\beta, C^\gamma, N^{\delta 1} \)	-	-
ILE	N, \( C^\alpha, C^\beta, C^{\gamma 1} \)	\( C^\alpha, C^\beta, C^{\gamma 1}, C^{\delta 1} \)	-	-
LEU	N, \( C^\alpha, C^\beta, C^\gamma \)	\( C^\alpha, C^\beta, C^\gamma, C^{\delta 1} \)	-	-
LYS	N, \( C^\alpha, C^\beta, C^\gamma \)	\( C^\alpha, C^\beta, C^\gamma, C^\delta \)	\( C^\beta, C^\gamma, C^\delta, C^\epsilon \)	\( C^\gamma, C^\delta, C^\epsilon, N^\zeta \)
MET	N, \( C^\alpha, C^\beta, C^\gamma \)	\( C^\alpha, C^\beta, C^\gamma, S^\delta \)	\( C^\beta, C^\gamma, S^\delta, C^\epsilon \)	-
PHE	N, \( C^\alpha, C^\beta, C^\gamma \)	\( C^\alpha, C^\beta, C^\gamma, C^{\delta 1} \)	-	-
PRO	N, \( C^\alpha, C^\beta, C^\gamma \)	\( C^\alpha, C^\beta, C^\gamma, C^\delta \)	-	-
SER	N, \( C^\alpha, C^\beta, O^\gamma \)	-	-	-
THR	N, \( C^\alpha, C^\beta, O^{\gamma 1} \)	-	-	-
TRP	N, \( C^\alpha, C^\beta, C^\gamma \)	\( C^\alpha, C^\beta, C^\gamma, C^{\delta 1} \)	-	-
TYR	N, \( C^\alpha, C^\beta, C^\gamma \)	\( C^\alpha, C^\beta, C^\gamma, C^{\delta 1} \)	-	-
VAL	N, \( C^\alpha, C^\beta, C^{\gamma 1} \)	-	-	-
Table 6: Specification of atom groups defining the torsion angles  $(\chi_{1},\chi_{2},\chi_{3},\chi_{4})$  for each residue type. Torsion angle exhibiting  $\pi$ -rotation-symmetry is colored as purple. Other side-chain torsion angles exhibit  $2\pi$ -periodicity.
C.2 Comparison between numbers of atom clashes of joint and autoregressive diffusion
In Figure 8, we provide a comparative illustration of atom clashes in protein structures using different noise schemes in autoregressive diffusion and joint diffusion process.
↑↓
$p(\chi_2\mid \chi_1)$
$p(\chi_1,\chi_2,\chi_3,\chi_4)$
Figure 8: Comparative illustration of atom clashes in protein structures using noise schemes from autoregressive diffusion process (left) and vanilla joint diffusion process (right). Vanila joint diffusion process add noise to all torsion angles simultaneously, resulting in much more steric clash.
C.3 Algorithm illustration of DiffPack
Algorithm 1 Training Procedure
Input: training dataset  $\mathcal{D}$  with protein sequence and structure pairs  $(S, \mathcal{X})$ , learning rate  $\alpha$   
Output: trained score networks  $s_{\theta_1}^{(1)}, s_{\theta_2}^{(2)}, s_{\theta_3}^{(3)}, s_{\theta_4}^{(4)}$  for torsional angles  $\chi_1, \chi_2, \chi_3, \chi_4$   
1: for epoch  $\leftarrow 1$  to epochmax do  
2: for  $(S, \mathcal{X})$  in  $\mathcal{D}$  do  
3: calculate torsional angles  $\chi_1, \chi_2, \chi_3, \chi_4$  from structure  $\mathcal{X}$ ;  
4: for  $i \gets 1$  to 4 do  
5: sample  $t \in [0, 1]$ ;  
6: sample  $\Delta \chi_i$  from wrapped normal  $p_{t|0}(\cdot|0)$  with  $\sigma = \sigma_{\min}^{1 - t} \sigma_{\max}^t$ ;  
7: set  $\widetilde{\chi}_i = \chi_i + \Delta \chi_i$ ;  
8: generate perturbed structure  $\tilde{\mathcal{X}}$  with  $\chi_{1..i-1}$  and  $\widetilde{\chi}_i$  and discard atoms in  $\chi_{i+1..4}$  groups;  
9: predict  $\delta \chi_i = s_{\theta_i}^{(i)}(\tilde{\mathcal{X}}, t)$ ;  
10: update  $\theta_i \gets \theta_i - \alpha \nabla_{\theta_i} \| \delta \chi_i - \nabla_{\Delta \chi_i} p_{t|0}(\Delta \chi_i|0)\|$ ;  
11: end for  
12: end for  
13: end for
Algorithm 2 Inference Procedure
Input: protein sequence S and backbone conformation  $\mathcal{X}^{(\mathrm{bb})}$  , number steps  $N$  , number rounds  $R$    
Output: protein conformation X   
1: for  $i\gets 1$  to 4 do   
2: sample  $\chi_{i}\sim U[0,2\pi ]^{m_{i}}$  ; Initialize with uniform prior   
3: generate structure  $\tilde{\mathcal{X}}$  with  $\chi_{1..i}$  and discard atoms in  $\chi_{i + 1..4}$  groups;   
4: for  $n\gets N$  to 1 do   
5: let  $t = n / N,g(t) = \sigma_{\min}^{1 - t}\sigma_{\max}^{t}\sqrt{2\ln(\sigma_{\max} / \sigma_{\min})};$    
6: for  $r\gets 1$  to R do Multi-round sampling   
7: predict  $\delta \pmb {\chi}_i = \pmb{s}_{\theta_i}^{(i)}(\tilde{\pmb{x}},t)$    
8: draw  $\textbf{z}$  from wrapped normal with  $\sigma^2 = 1 / N$  .   
9: set  $\Delta \pmb {\chi}_i = \lambda_t(g(t)^2 /N)\delta \pmb {\chi}_i + g(t)\pmb {z};$    
10: update  $\chi_{i}\leftarrow \chi_{i} + \Delta \chi_{i}$  Update torsional angles   
11: generate structure  $\tilde{\mathcal{X}}$  with  $\chi_{1\dots i}$  and discard atoms in  $\chi_{i + 1\dots 4}$  groups;   
12: end for   
13: end for   
14: end for   
15: return structure  $\mathcal{X}$  generated with  $\chi_1,\chi_2,\chi_3,\chi_4$
D Score Network Architecture
Atom graph construction. We represent protein structure as an atom-level relational graph, where atoms serve as nodes and are connected by edges based on three conditions: (1) if they are linked by a chemical bond, (2) if one is among the 10-nearest neighbors of the other, and (3) if their Euclidean distance is within  $4.5\AA$  and their sequential distance is above 2. Each type of edge is treated as a different relation, while the node feature is created by concatenating the one-hot features of atom and residue types. Following [75], edge features are created by concatenating one-hot features of residue types, relation types, and sequential and Euclidean distances. This graph construction considers both geometric and sequential properties of proteins, providing a comprehensive featurization of proteins.
**Embedding layer.** In the embedding layer, we fuse node features with embeddings for encoding time steps. We use sinusoidal embeddings [61] of time  $t$  as input for a linear layer. Node features are passed through a Multi-Layer Perceptron (MLP) and added with time embeddings to obtain the new node features for the model.
Message passing layer. To learn representations for each node, we perform relational message passing between them [50]. We denote the edge between nodes  $i$  and  $j$  with type  $r$  as  $(i,j,r)$  and set
of relations as  $\mathcal{R}$ . We use  $\pmb{h}_i^{(l)}$  to denote the hidden representation of node  $i$  at layer  $l$ . Then, message passing can be written as:
$$
\boldsymbol {h} _ {i} ^ {(l)} = \boldsymbol {h} _ {i} ^ {(l - 1)} + \sigma \left(\mathrm {B N} \left(\sum_ {r \in \mathcal {R}} \boldsymbol {W} _ {r} \sum_ {j \in \mathcal {N} _ {r} (i)} \left(\boldsymbol {h} _ {j} ^ {(l - 1)} + \operatorname {L i n e a r} \left(\boldsymbol {m} _ {(i, j, r)} ^ {(l)}\right)\right)\right)\right), \tag {11}
$$
where  $\mathbf{W}_r$  is the learnable weight matrix for relation type  $r$ ,  $\mathcal{N}_r(j)$  is the neighbor set of  $j$  with relation type  $r$ ,  $\mathrm{BN}(\cdot)$  denotes batch normalization, and  $\sigma(\cdot)$  is the ReLU function. We use  $\mathbf{m}^{(l)}(i,j,r)$  to denote the representation of edge  $(i,j,r)$ , computed through edge message passing. We use  $e$  as the abbreviation of the edge  $(i,j,r)$ . Two edges  $e_1$  and  $e_2$  are connected if they share a common end node. The type of their connection is determined by the angle between them, which is discretized into 8 bins. The edge message passing layer can be written as:
$$
\boldsymbol {m} _ {e _ {1}} ^ {(l)} = \sigma \left(\mathrm {B N} \left(\sum_ {r \in \mathcal {R} ^ {\prime}} \boldsymbol {W} _ {r} ^ {\prime} \sum_ {e _ {2} \in \mathcal {N} _ {r} ^ {\prime} (e _ {1})} \boldsymbol {m} _ {e _ {2}} ^ {(l - 1)}\right)\right), \tag {12}
$$
where  $\mathcal{R}'$  is the set of relation types between edges and  $\mathcal{N}_r'(e_1)$  is the neighbor set of  $e_1$  with relation type  $r$ . After obtaining the hidden representations of all atoms at layer  $L$ , we compute the residue representations by taking the mean of the representations of its constituent atoms. Finally, we apply an MLP to the residue representations to predict the score.
E Annealed Sampling
As discussed in Section 3.5, vanilla diffusion sampling scheme suffers from over-dispersion problem, which have negative influence in side-chain packing. A promising way to tackle this issue is low-temperature sampling, which involves perturbing the initial distribution  $p(x)$  with a temperature factor  $T$ . This results in a re-normalized distribution  $p_{T}(\mathbf{x}) = \frac{1}{Z} p(\mathbf{x})^{\frac{1}{T}}$ . Lowering the temperature value shifts the model's focus towards sampling quality, while increasing it emphasizes diversity. However, strict low temperature sampling is an expensive iterative process. Merely up-scaling the score function or down-scaling the noise term in the reverse SDE does not resolve this challenge.
Inspired from [28], we proposed a modified reverse SDE to approximate the low-temperature sampling. Specifically, the modified reverse SDE is first derived by considering a simplified Gaussian data distribution  $\mathcal{N}\left(\mathbf{x}_0;\boldsymbol{\mu}_{\mathrm{data}},\sigma_{\mathrm{data}}^2\right)$ . In this simplified case, we show that the modified reverse SDE with the annealed weight  $\lambda_t$  converges to a low-temperature sampling process that prioritizes quality while maintaining diversity.
Considering the simplified data distribution  $\mathcal{N}\left(\mathbf{x}_0;\boldsymbol{\mu}_{\mathrm{data}},\sigma_{\mathrm{data}}^2\right)$ , the time-dependent marginal density of noise-perturbed distribution by applying VE-SDE forwarding process  $d\mathbf{x} = \sqrt{\frac{d\sigma^{2}(t)}{dt}} d\mathbf{w}$  is
$$
p _ {t} (\mathbf {x}) = \mathcal {N} \left(\mathbf {x}; \mu_ {\text {d a t a}}, \sigma_ {\text {d a t a}} ^ {2} + \sigma^ {2} (t)\right) \tag {13}
$$
The corresponding score function of this noise-perturbed distribution then becomes:
$$
\mathbf {s} _ {t} \triangleq \nabla_ {\mathbf {x}} \log p _ {t} (\mathbf {x}) = \frac {\mu_ {\text {d a t a}} - \mathbf {x}}{\sigma_ {\text {d a t a}} ^ {2} + \sigma^ {2} (t)} \tag {14}
$$
Now we consider a low-temperature sampling scheme, where the original data distribution is transformed to  $p_t'(\mathbf{x}) = \frac{1}{Z} p_0(\mathbf{x})^{\frac{1}{T}}$  through a temperature factor  $T$ . This results in the variance re-scaled by  $T$ , i.e.
$$
p _ {t} ^ {\prime} (\mathbf {x}) = \mathcal {N} \left(\mathbf {x}; \mu_ {\text {d a t a}}, T \sigma_ {\text {d a t a}} ^ {2} + \sigma^ {2} (t)\right) \tag {15}
$$
$$
\mathbf {s} _ {t} ^ {\prime} = \frac {\mu_ {\mathrm {d a t a}} - \mathbf {x}}{T \sigma_ {\mathrm {d a t a}} ^ {2} + \sigma^ {2} (t)} \tag {16}
$$
$$
= \lambda_ {t} \mathbf {s} _ {t}, \text {w h e r e} \lambda_ {t} = \frac {\sigma_ {\mathrm {d a t a}} ^ {2} + \sigma^ {2} (t)}{T \sigma_ {\mathrm {d a t a}} ^ {2} + \sigma^ {2} (t)} \tag {17}
$$
Strictly solving the re-weight coefficient  $\lambda_{t}$  is meaningless and would only apply to the simplified Gaussian distribution. Following [28], we assume the perturbed distribution's variance is approximately the maximum variance in the VE-SDE  $\sigma_{\mathrm{data}}^2 +\sigma^2 (t)\approx \sigma_{\mathrm{max}}^2$  . Finally we got a re-weight coefficient:
$$
\lambda_ {t} = \frac {\sigma_ {\mathrm {m a x}} ^ {2}}{T \sigma_ {\mathrm {m a x}} ^ {2} - (T - 1) \sigma^ {2} (t)}
$$
F Confidence Selection
As we formulate the sidechain packing problem in the view of generative modeling, we can sample multiple conformation from the learned conformation distribution. Here we train an additional confidence module to select the most likely sample. The idea of self-predicting sampling quality is also adopted by AlphaFold [32], where pLDDT estimate the quality of protein structure.
Specifically, we utilize a model bearing the same architecture as that used in score prediction. Following the computation of the residue-level representation, we augment the model with an additional MLP head, thereby generating a scalar confidence score for each residue. This confidence score is then trained to correspond with the negative residue-level RMSD. During the inference stage, we sample four conformations for each protein and select the one boasting the highest confidence.
To show how confidence selection improves performance, we randomly select 10 proteins from the test set (each having more than 150 residues) and plot the angle accuracy in accordance with an increasing number of samples (Figure 9). Further, we have measured the correlation between the predicted confidence score and the negative RMSD, yielding Pearson coefficient 0.664 and Spearman coefficient 0.798, respectively. These statistics demonstrate that the predicted confidence score serves as an effective indicator of sample quality, thereby justifying its application in our model.
Figure 9: Percentage of accurate angles as number of samples increasing. "Oraca Selection" denotes selecting the conformation with lowest RMSD. "Confidence Selection" denotes selecting by trained confidence module.
G Inference Time
G.1 Comparative results of inference time
In this section, we evaluate the inference speed of various methods, categorizing them as GPU-based or CPU-based. All GPU-based methods<sup>1</sup> were evaluated on an NVIDIA RTX A100 40GB GPU, while CPU-based methods were assessed on an AMD EPYC 7513 32-Core Processor @ 2.60 GHz. For algorithms that facilitate batch processing, the batch size was meticulously chosen and fine-tuned to an optimal value, taking into account specific computational requirements.
It's essential to recognize that the inference time for DiffPack is highly influenced by the number of samples in confidence selection and the number of rounds in multi-round sampling. We denote DiffPack-vanilla as the model that samples one conformation without confidence selection. As illustrated in Table 7, DiffPack-vanilla outperforms other GPU-based methods in terms of both speed and performance. Although the CPU-based method FASPR achieves superior speed through judicious
<sup>1</sup>The symbol  $\dagger$  is used to denote methods that utilize GPU processing.
Figure 10: Comparison of speed and accuracy between different methods. Inference speed is shown in log-scale
optimization within its tree search algorithm, its restricted angle accuracy limits its applicability in contexts where precision takes precedence over speed especially in sidechain packing.
DiffPack's flexibility allows for integration with various supplementary techniques as shown in Section 3.5, trading off some speed for enhanced performance. For a detailed comparative analysis, we provide a plot (shown in Figure 10) that elucidates the interplay between DiffPack's speed and corresponding performance. Among the variations of DiffPack, we refer to the model amalgamated with confidence selection as DiffPack-confidence, and the one further augmented with multi-round sampling as DiffPack-multiround.
G.2 Further acceleration on DiffPack
It is noteworthy that DiffPack's speed is primarily constrained by the multi-step nature of the denoising process. Recently, significant research efforts [42, 43] have been directed towards accelerating this process. Pursuing integration with these speed-enhancing methods constitutes an exciting avenue for future research and potential further optimization.
Beyond the algorithm itself, DiffPack's performance is intricately tied to the specific architectural design of GearNet-Edge, which is utilized to learn the score function within the torsion space. This architecture leverages multi-relational message passing within the protein's edge-graph (i.e. linegraph). We have invested some efforts in optimizing the inference speed of GearNet-Edge, based on DGL [62]. Our preliminary findings have yielded promising results, reducing the inference time from 1.29s/protein to 0.67s/protein on a single A100 GPU, and to 3.9s/protein on a single SPR CPU.
INFERENCE TIME(S) ↓	ANGLE ACCURACY ↑
SCWRL	2.71	56.2%
FASPR	0.10	56.4%
RosettaPacker	217.80	58.6%
DLPacker†	28.50	58.8%
AttnPacker†	6.33	62.1%
DiffPack-vanila†	1.29	67.0%
Table 7: Average Inference Time of Different Methods
H Experimental Setup
H.1 Details of Baselines
SCWRL4. SCWRL4 [35] is a widely used and well-established method for protein side-chain conformation prediction. It employs a graph-based approach, where side chains are represented as
nodes in a graph, and edges connect pairs of nodes that have potential steric clashes. The algorithm performs a combinatorial search to find the most probable side-chain conformation with minimal steric clashes and optimal energy, using a backbone-dependent rotamer library and a statistical potential energy function derived from known protein structures.
FASPR. FASPR (Fast and Accurate Side-chain Prediction using Rotamer libraries) [27] is a prediction method that leverages backbone-dependent rotamer libraries [53] and a custom-built energy function to efficiently predict side-chain conformations. The method involves using Dead-End Elimination (DEE) and tree decomposition to find the set of rotamers that allows the protein to adopt the Global Minimum Energy Conformation (GMEC). FASPR is designed to achieve a balance between computational speed and accuracy, making it suitable for large-scale protein modeling applications.
RosettaPacker. RosettaPacker [8] is a component of the Rosetta molecular modeling suite, which is widely recognized for its versatility and accuracy in various applications, including protein structure prediction, protein-protein docking, and protein design. The RosettaPacker algorithm utilizes Monte Carlo simulations combined with a detailed all-atom energy function to sample side-chain conformations and optimize packing interactions. The method is known for its ability to explore a broad conformational space, making it particularly effective for challenging prediction tasks that involve significant side-chain rearrangements.
DLPacker. DLPacker [46] is a deep learning-based method for side-chain conformation prediction, which employs convolutional neural networks (CNNs) to predict the atom density map of side-chain conformation from the amino acid sequence and backbone conformation. The method incorporates local and non-local context information from the protein structure to make predictions, and it is trained on a large dataset of high-resolution protein structures. By leveraging the power of deep learning, DLPacker can capture complex sequence-structure relationships, leading to improved prediction accuracy.
AttnPacker. AttnPacker [45] is a state-of-the-art method for side-chain conformation prediction that utilizes the attention mechanism, a powerful technique commonly employed in deep learning architectures for tasks involving sequence data. The method incorporates multiple layers of complex triangle attention operations, which enable it to learn long-range dependencies and spatial relationships in protein structures. AttnPacker's architecture allows it to model complex protein conformations with high accuracy; however, its large number of parameters makes it computationally demanding compared to some of the other methods.
Results of baselines are directly taken from the AttnPacker paper [45].
H.2 Training Details
For torsional diffusion, we utilize the Variance-Exploding SDE (VE-SDE) framework, where  $f(\boldsymbol{\chi}, t) = 0$  and  $g(t) = \sqrt{\frac{d}{dt}\sigma^2(t)}$ . The choice of  $\sigma(t)$  follows the exponential decay defined in previous research [57], given by  $\sigma(t) = \sigma_{\min}^{1 - t}\sigma_{\max}^t$  with  $\sigma_{\min} = 0.01\pi$ ,  $\sigma_{\max} = \pi$ , and  $t \in (0,1)$ . During training, we randomly sample a time step  $t$  and embed it with sinusoidal embeddings, which is then fused with node features. For inference, we use 10 time steps interpolated between  $\sigma_{\min}$  and  $\sigma_{\max}$  for generation. At each time step, we perform four rounds of sampling. Subsequently, for each target, we randomly select four different predictions from our model and employ our confidence model to choose the best prediction. To encode the protein structure and denoise torsional angles, we employ a 6-layer GearNet-Edge model with a hidden dimension of 128. For edge message passing, the connections between edges are divided into 8 bins based on the angles between them.
All the models are trained using the Adam optimizer with a learning rate of 1e-4 and a batch size of 32. The training process is performed on 4 A100 GPUs for 400 epochs.