Full-Atom Protein Pocket Design via Iterative Refinement
Authors: ZAIXI ZHANG
Zepu Lu
Zhongkai Hao
Marinka Zitnik
Qi Liu
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Graph Representation Learning
AI for Science

Abstract: The design of \emph{de novo} functional proteins that bind with specific ligand molecules is crucial in various domains like therapeutics and bio-engineering. One vital yet challenging step is to design the protein pocket, the cavity region of protein where the ligand binds with. Existing methods suffer from inefficient generation, insufficient context modeling (ligand molecule), and incapability of generating sidechain atoms. To overcome the limitations, we propose a \textbf{F}ull-\textbf{A}tom \textbf{I}terative \textbf{R}efinement framework (\textbf{FAIR}) for protein pocket sequence (i.e., residue types) and 3D structure co-design. Generally, FAIR consists of two steps that follow a coarse-to-fine pipeline (backbone atoms to full atoms including sidechain) for full-atom generation. For efficiency, all residue types and structures are updated together in each round (i.e., full-shot refinement). In the first step, the residue types and backbone coordinates are updated with a hierarchical context encoder and two structure refinement modules capturing inter-residue and pocket-ligand interactions. The second step further models the sidechain atoms of pockets and updates residue types to achieve sequence-structure consistency. The structure of the binding ligand is also updated along with the above refinement iterations accounting for its flexibility. Finally, extensive evaluations show
that FAIR outperforms baselines in efficiently designing high-quality pocket sequences and structures. Specifically, the average improvements on AAR and RMSD are over 10$\%$.

Full-Atom Protein Pocket Design via Iterative Refinement
Zaixi Zhang 1.2,4 Zepu Lu Zhongkai Hao?, Marinka Zitnik*, Qi Liu
1 Anhui Province Key Lab of Big Data Analysis and Application,
School of Computer Science and Technology, University of Science and Technology of China
2 State Key Laboratory of Cognitive Intelligence, Hefei, Anhui, China
Dept. of Comp. Sci. and Tech., Institute for AI, THBI Lab, BNRist Center, Tsinghua-Bosch Joint ML Center, Tsinghua, Harvard University [zaixi, zplu} @ maail.usc.ede usw..edu..cccnn hzj21 @ mails. wsinghua.edu..omm
marinka hms.harvard harvard.eddu qiliuql @ ustc.edu.cn
Abstract
The design and optimization of functional proteins that bind specific ligand molecules is paramount in therapeutics and bio-engineering. formidable task in this endeavor is the design of the protein pocket, which is the cavity region of the protein where the ligand binds. Current methods are plagued by inefficient generation, inadequate context modeling of the ligand molecule, and the inability to generate side-chain atoms. Here, we present the Full-Atom Iterative Refinement (FAIR) method, designed address these challenges by facilitating the co-design of protein pocket sequences, specifically residue types, and their corresponding 3D structures. FAIR operates in two steps, proceeding in coarse-to- fine manner (transitioning from protein backbone to atoms, including side chains) for full-atom generation. In each iteration, all residue types and structures are simultaneously updated, process termed full-shot refinement. In the initial stage, the residue types and backbone coordinates are refined using hierarchical context encoder, complemented by two structure refinement modules that capture both inter-residue and pocket-ligand interactions. The subsequent stage delves deeper, modeling the side-chain atoms of the pockets and updating residue types to ensure sequence-structurg -struccrtt congruence. Concurrently, the structure of the binding ligand is refined across iterations to accommodate its inherent flexibility. Comprehensive experiments show that FAIR surpasses existing methods in designing superior pocket sequences and structures, producing average improvement exceeding 10% in AAR and RMSD metrics.
critical yet
Introduction
Proteins are macromolecules that perform fundamental functions in living organisms. An essential and challenging step in functional protein design, such enzymes [52) 27 and biosensors [8). is to design protein pockets that bind specific ligand molecules. Protein pockets usually refer to residues binding with the ligands in the protein-ligand complexes. Nevertheless, such problem is very challenging due to the tremendous search space of both sequence and structure and the small solution space that satisfies the constraints of binding affinity, geometric complementarity. and biochemical stability. Traditional approaches for pocket design use hand-crafted energy functions [42] [T] and
template-matching methods 667 g 48 which can be less accurate and lack efficiency.
"Qi Liu is corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Deep learning approaches, especially deep generative models, have made significant progress in protein design [46 17
20 using experimental and predicted 3D structure information to design 14]. Despite their strong performance, these approaches are not directly applicable to pocket design because the protein pocket structures to designed are apriori unknown [8] To this end, generative models with the capability to co-design both the sequence and structure of protein pockets are needed. In another area of protein design, pioneering research developed techniques to co-design sequence and structure of complementarity determining regions
amino acid sequences 26
30
7
(CDRs) for antibodies (Y-shaped proteins that bind with specifie pathogens) [29] 28 39 33 [6].
Nevertheless, these approaches are tailored specifically for antibodies and are unsuitable for the pocket design of general proteins. For example, prevailing methods generate only protein backbone atoms while neglecting side-chain atoms, which play essential roles in protein pocket-ligang -liand interactions
433. Further, binding targets (antigens) considered by these methods are fixed, whereas binding ligand molecules are more flexible and require additional modeling.
Here, we introduce a full-atom iterative refinement approach (FAIR) for protein pocket sequence- structure co-design, addressing the limitations of prior research. FAIR consists of two primary steps: initially modeling the pocket backbone atoms to generate coarse-grained structures, followed by incorporating side-chain atoms for a complete full-atom pocket design. (1) In the first step, since the pocket residue types and the number of side-chain atoms are largely undetermined, FAIR progressively predicts and updates backbone atom coordinates and residue types via soft assignment over multiple rounds. Sequence and structure information are jointly updated in each round for efficiency, which we refer to full-shot refinement. A hierarchical encoder with residue- and atom-level encoders captures the multi-scale structure of protein-ligand complexes. To update the pocket structure, FAIR leverages pocket residues and between the pocket and the binding ligand in each round. (2) In the second step, FAIR initializes residue types based on hard assignment and side-chain atoms based on the results of the first step. The hierarchical encoder and structure refinement modules are similar to the first step for the full-atom structure update. To model the influence of structure on residue types, FAIR randomly masks a part of the pocket residues and updates the masked residue types based on the neighboring residue types and structures in each round. After several rounds of refinement, the residue type and structure update gradually converge, and the residue sequence-structure becomes consistent. The ligand molecular structure is also updated along with the above refinement processes to account for the flexibility of the ligand.
structure refinement modules to predict the interactions within
Overall, FAIR is an Equivvarrrri generative model that generates. both pocket residue types and the pocket-ligand complex 3D structure. Our study presents the following main contributions:
We preliminarily formulate protein pocket design as a co-generation of both the sequence and full-atom structure of protein pocket (i.e., co-design), conditioned on a binding ligand molecule and the protein backbone (see Sec. [.1]
New task:
Novel method: FAIR is an end-to-end generative framework that generates both the pocket sequence and structure through iterative refinement (see Sec. 33. It addresses the limitations of previous approaches and takes into account side-chain effects, ligand flexibility, and sequence- structure consistency to enhance prediction efficiency.
Strong performance: FAIR outperforms existing methods on various protein pocket design quality metrics, producing average improvements of 15.5% (AAR) and 13.5% (RMSD). FAIR's pocket-generation process is over ten times faster than traditional methods (see Sec. 4
2 Related Work
Protein Design. Computational protein design has brought the capability of inventing proteins with desired structures and functions 22 35 60). Existing studies focus on designing amino acid sequences that fold into target protein structures, i.e., inverse protein folding. Traditional approaches mainly rely on hand-crafted energy functions to iteratively search for low-energy protein sequences with heuristic sampling algorithms,, which are inefficient, less accurate, and less generalizable Recent deep generative approaches have achieved remarkable success in protein design
6
20 3. Some protein design methods also benefit from the latest advances in protein structure prediction techniques, e.g., AlphaFold2 [I.. Nevertheless, they usually assume the target protein structure is given, which is unsuitable for pocket design case. Another
15
26
30
7
14
51
2

---

(a)
(b)
Protein
Full atom structure including sidechains
Pocket sepuece
struchure co-design
Pocket
Ca
N
II
Ligand
Pocket residue type sequence: MTREIALEE
Figure 1: (a) The residue structure, where the backbone atoms are C. .N C,C,O. R represents side chain that determines the residue types. (b) The protein pocket design problem. Pocket (colored in black) consists of a subsequence of residues closest to the binding ligand molecule (formal definition in Sec. 3.1 FAIR aims to co-design the full-atom structures and residue types of the pocket.
line of recent works proposes co-design the sequence and 3D structure of antibodies graphs [29] (28) 39 33 [6]. However, they are designed explicitly for antibody CDR topologies and are primarily unable consider side-chain atoms and the flexibility of binding targets. For more detailed discussions on protein design, interested readers may refer to the comprehensive surveys [22 355.
Pocket Design. Protein pockets are the cavity region of protein where the ligand binds to the protein. In projects to design functional proteins such as biosensors [52 27 and enzymes [5] [8] pocket design is the central task 8 48). Compared with the general protein design/antibody design problems, the pocket design brings unique challenges: (1) protein pocket has a smaller spatial scale than whole proteins/antibodics essential roles in pocket design and require careful modeling [36 [2]; (2) the influence and structural flexibility of target ligand molecules need to be considered to achieve high binding affinity (42]. Traditional methods for pocket design are physics- based methods [42 or template-matee matching methods (67 8 48), which often suffer from the large computational complexity and limited accuracy. So far, the pocket design problem has rarely been tackled with deep learning.
Therefore, the structure and interactions of side-chain atoms play
Structure- Based Drug Design. Structure- drug design is another line of research inquiry that explores 3D ligand molecule design inside protein pockets 49 37 70 69 19]. These studies can be regarded as the dual problem of pocket design studied in our paper. However, the methods primarily focus on generating 3D molecular graphs based on fixed protein pocket structure and may not be easily adapted to the pocket sequence-structure co-design problem examined in our study, We refer interested readers to (7 for comprehensive survey on structure-based drug design.
Full-Atom Protein Pocket Design
Next, introduce FAIR, our method for pocket sequence-strucctue co-design. The problem is first formulated in Sec. 3.1 We then describe the hierarchical encoder that captures residue- and atom-level information Sec. 3.2 Furthermore, we show the iterative refinement algorithms with two main steps in Sec. 3.3 Finally, y, we show the procedures for model training and sampling in Sec. 3.4 and demonstrate the equivvvuivre property of FAIR in Sec. 3.5
3.1 Notation and Problem Statement
We aim to co-design residue types (sequences) and 3D structures of the protein pocket that can fit and bind with the target ligand molecule. As shown in Figure protein--igann binding complex typically consists of protein and a ligand molecule. The ligand molecule can be represented as 3D point cloud M { wx)) 1=1 where Ię denotes the 3D coordinates and dy denotes the atom feature. The protein can be represented length of the sequence. The 3D structure of the protein can be described as point cloud of protein atoms {a.,j (
Nr
sequence of residues A a aN,, where N, the
and æ ((..;) ) E R&3 denote the 3D coordinate of the pocket atoms. n; is the number of atoms in residue determined by the residue types. The first four atoms in any residue correspond its backbone atoms (Ca, N, C, O), and the rest are side-chain atoms. In our work, the protein pocket is defined set of residues in the protein closest to the binding ligand

---

molecule: B = b1 ... .m. The pocket B can thus be represented as an amino acid subsequence of protein: a arm. where {ei..... Cm is the index of the pocket residues in the protein. The index e can be formally given as: {i &
min
| - my 2 S 8), where || l/2
the L2 norm and is the distance threshold. According the distance range of pocket-ligand interactions [43, we set 5 3.5 À in the default setting. Note that the number of residues (i.e., m) varies for pockets due to varying structures of protein-ligand complexes. To summarize this subsection, our preliminary problem statement is formulated below:
Problem Statement: In FAIR, our objective is to learn a conditional generative model for full-atom protein pocket generation conditioned on the protein backbone and the binding ligand. Given the availability existing protein pockets, we may leverage them for reference to construct residue graphs and calculate geometrie features. Our preliminary problem statement formulated considering practical situations such as enzyme optimization.
3.2 Hierarchical Encoder
Leveraging this intrinsic multi-scale protein structure (59], we adopt hierarchical graph transformer
44 (38 (88 to encode the hierarchical context information of protein-ligand complexes for pocket sequence-structtue co-design. The transformer includes both atom- and residue-level encoders, as described below.
3.2.1
Atom-Level Encoder
The atom-level encoder considers all atoms in the protein-ligand complex and constructs a 3D context graph connecting the K, nearest neighboring atoms in A.M. The atomic attributes are firstly mapped to node embeddings h with two MIPs, respectively, for protein and ligand molecule atoms. The edge embeddings eij are obtained by processing pairwise Euclidean distances with Gaussian kernel functions [53]. The 3D graph transformer consists of L Transformer layers [62] with two modules each layer: a multi-head self-attention (MHA) module and a position-wise feed-forward network (FFN). Specifically, in the MHA module of the I-th layer (1 S [ S L), the queries are derived from the current node embeddings h while the keys and values come from the (!)
relational representations: ri [h]] exg (I denotes concatenation) from neighboring nodes:
Weh (1) k: WKT. oi WvT
(I)
where We, W and Wv are learnable transformation matrices. Then, in each attention head {1,2,.... C} (C' is the total number of heads), the scaled dot-product attention mechanism works as follows:
heads-
Σ JEN())
Softmax
q.
k kg
1
where (i) denotes the neighbors of the i-th atom in the constructed graph and d' is the dimension size of embeddings for each head. Finally, the outputs from different heads further concatenated and transformed to obtain the final output of MHA:
MHA, head head." Wo.
where Wo is the output transformation matrix. The output of the atom-level encoder is a set of atom representations (h.}. Appendix includes the details of the atom-level encoder.
3.2.2 Residue-Level Encoder
The residue-level encoder only keeps the C., atoms of residues and coarsened ligand node at the ligand's 's center of mass to supplement binding context information (the coarsened ligand node is appended at the end of the residue sequence as a special residue). Then, a K, nearest neighbor graph constructed at the residue level. We use the original protein pocket backbone atoms for reference to construct the K, nearest neighbor graph and help calculate geometric features. The i-th residue a, can be represented by a feature vector fres describing its geometric and chemical characteristics such volume, polarity, charge, hydropathy, and hydrogen bond interactions according its residue

---

rounds Backbone refinement
I, rounds Full-atom refinement
Hienrehical Encoder
Hicrarchical Encoder
Structure refinemeent
Stracture rofinement
Podite de
Miskcal rosidue
prodiction
type updane
Figure 2: Overview of FAIR with two main steps (T1 rounds of backbone refinement and T2 rounds of full-atom refinement). FAIR co-designs pocket residue types and structures via iterative full-shot refinement. The structure refinement is illustrated with the atoms in the dotted ovals examples.
type a, We concatenate the residue features with the sum of atom-level embeddings h& within each residue to initialize the residue representations; the ligand node representation is initialized by summing up all the ligand atom embeddings:
fres
fac Σ jEres; hg
f!!
Σ
KEM hkk
As for the edge features e, local coordinate frames built for residues, and e are computed according to the distance, direction, and orientation between neighboring residues [26]. Lastly, the encoder takes the node and edge features into the residue-leve graph transformer to compute the residue-level representations. The residue-level graph transformer architecture is similar to that of the atom-level encoder, and details are shown Appendix
In summary, the output of our hierarchical encoder a set of residue representations {fi} and atom representations {h;}, which used in the following subsections for structure refinement and residue type prediction. Since our encoder is based on the atom/residue attributes and pairwise relative distances, the obtained representations are E(3)-invariant
3.3 Iterative Refinement
For efficiency, FAIR predicts pocket residue types and structures via an iterative full-shot refinement scheme:
all the pocket residue types and structures updated together in each round. Since the pocket residue types and the number of side-chain atoms are largely unknown at the beginning rounds, FAIR is designed to have models the backbone atoms of pockets to generate the coarse-graiee coarse-graied structures and then fine-adjusts full-atom residues achieve sequence-structin consistency. Figure shows the overview of FAIR.
main steps that follow coarse-to-fine pipeline: FAIR firstly only
3.3.1 Backbone Refinement
Initialization. Before backbone refinement, the residue types to predict are initialized with uniform distributions over all residue type categories. Since the residue types and the number of side-chain atoms are unknown, we only initialize the 3D coordinates of backbone atoms (Ca, N, C, O) with linear interpolations and extrapolations based the known structures of nearest residues in the
protein. Details are shown the Appendix A.4 Following previous research 8 28 33 we initialize the ligand structures with the reference structures from the dataset the generation process.
Residue Type Prediction. In each round iterative refinement, we first encode the current protein- ligand complex with the hierarchical encoder in Sec. 3.2 We then predict and update the sequences and structures. To predict the probability of each residue type. we apply an MLP and SoftMax function the obtained pocket residue embeddings f. p; Softmax(MLP the predicted distribution over 20 residue categories. The corresponding residue features can then be updated with weighted averaging: fE E, fi.jffee, where Pij and fres the probability and feature vector for the j-th residue category respectively.
where p; R
Structure Refinement. Motivated by the force fields physics 50 and the successful practice in previous works [28 33 we calculate the interactions between atoms for the coordinate update.
PE(3) the group of Euclidean transformations: rotations, reflections, and translations.
5

---

According to previous works [32 12 10]. the internal interactions within the protein and the external interactions between protem and ligand are different. Therefore, FAIR uses two separate modules for interaction predictions. Firstly, the internal interactions are calculated based on residue embeddings:
a<(b.,)) x(a..))
Uij.kj - g(fc, JR. eRAP (( a ((ax,)))))
[ a(axi))
where @RAF (Il - x: a0 I)) is the radial basis distance encoding of pairwise C, distances, the function y is a feed-forward neural network with four output channels for four backbone atoms, =(0.a)-2((z.)
and =<h,, ) (= E IR3 is the normalized vector from the residue atom akj to bi.- For the stability of training and generation, we use original protein backhone coordinates to calculate the RBF encodings. Similarly, for the external interactions between pocket B and ligand M, we have:
gij. g' (h h., GRAP (l(=(bb.s) - -x+1l)) x x((b,,) x.
(ar(b..) x=
which is calculated based the atom representations. o' a feed-forward neural network with one output channel. For the conciseness of expression, here we use hb., and h2 to denote the pocket and ligand atom embeddings, respectively. Finally, the coordinates of pocket atoms (bbs,) and ligand molecule atoms a are updated as follows:
x x(br,) x(br,))
Σ kEN())
gij.ky + Egise
(7)
K,
Ni
3g - x.
Σ i.j
gy isism, 1 S $
(8)
B
where (ii the neighbors of the constructed residue-level KN-NN graph: [B] denotes the number of pocket atoms. Details of the structure refinement modules are shown in the Appendix
3.3.2 Full-Atom Refinement
Initialization and Structure Refinement. After rounds of backbone refinements, the predicted residue type distribution and the backbone coordinates become relatively stable (verified in experi- ments). We then apply the full-atom refinement procedures to determine the side-chain structures. The residue types are initialized by sampling the predicted residue type p; from backbone refinement (hard assignment). The number of side-chain atoms and the atom types can then be determined. The coordinates of the side-chain atoms are initialized with the corresponding a-carbon coordinates. The full-atom structure refinement similar backbone refinement (Sec. (Sec.3..31) with minor adjustments to accommodate the side-chain atoms. We include the details in the Appendix A
Residue Type Update. Since the structures and interactions of side-chain atoms further influence the residue types, proper residue type update algorithms are needed to achieve sequence-structure consistency. Considering the hard assignment of residue types in the full-atom refinement, randomly sample and mask one residue in each protein pocket and update its residue type based on the surrounding environment in each iteration. Specifically, we mask the residue types, remove the sampled residues' side-chain atoms in the input to the encoder, and use the obtained corresponding residue types for residue type prediction. If the predicted result inconsistent with the original residue type, we will update the residue type with the newly predicted one and reinitialize the side- chain structures. As our experiments indicate, the residue types and structures become stable after several refinement iterations, and sequence-structure consistency is achieved.
3.4 Model Training and Sampling
The loss function for FAIR consists of two parts. We apply cross-entropy loss loe for (sampled) pocket residues over all iterations for the sequence prediction. For the structure refinement, we adopt the Huber loss huber for the stability of optimization following previous works 28 56]:
Leeg
ΣΣ
lce (pi-b));
(9)
T
Lstruct
1
Σ
Σ Unuuee((x(bb)) (x(b)),) Σ Unuber (2//, (z,)
(10)
T

---

where T T1 + T2 is the total rounds of refinement. P:. 2b (b,,), and a are the ground-truth residue types, residue coordinates, and ligand coordinates. *(,)) and æ are the predicted ones the 1-th round. In the training process, we aim to minimize the sum of the above two loss functions L Leeg + Estruct similar to previous works [56 After the training of FAIR, we can co-design pocket sequences and structures. Algorithm sampling.
and 2 in Appendix A outline model training and
3.5 Equivariance
FAIR is E(3)-equivarian) which is desirable property of generative protein models 33
64
Tg and the generative process of FAIR as {(b,, x(b,)) =bb))) = -p (ss(A') M,),, where {[b,, x (bbb)) mm indicates the designed pocket sequence and structure. We have {(b,, Tg xxb))) (T B), M)..
Theorem 1. Denote the tr3nsfrrnnoommation
Proof. The main idea is that the ((3)-innvria encoder and E(3) equiivarrr structure refinement lead to an Equiviauiva generative process of FAIR. We give the full proof in Appendix A
4 Experiments
We proceed by describing the experimental setups (Sec. (Sec.4.1) 4.1). We then assess our method for ligand- binding pocket design (Sec. 4.2) on two datasets. We also perform further in-depth analysis of FAIR, including sampling efficiency analysis and ablations in Sec. 4.3
4.1 Experimental Setting
Datasets. We consider two widely used datasets for experimental evaluations. CrossDocked dataset [6] contains 22.5 million protein-molecule pairs generated through cross-docking. We filter out data points with binding pose RMSD greater than 1 Å, leading to a refined subset with around 180k data points. For data splitting,
mmseqs2 [58] to cluster data 30% sequence identity, and randomly draw 100k protein-ligand structure pairs for training and 100 pairs from the remaining clusters for testing and validation, respectively. Binding MOAD dataset (21) contains around 41k experimentally determined protein-ligand complexes. We further filter and split the Binding MOAD) dataset based on the proteins' enzyme commission number 44) resulting 40k protein-ligant pairs for training, 100 pairs for validation, and 100 pairs for testing following previous work [54]. More
data split schemes based on graph clustering [66] 651 may be explored.
43 we redesign all the residues that contain atoms within 3.5 A any binding ligand atoms, leading to an average of around eight residues for each pocket. We sample 100 sequences and structures for each protein- complex in the test set for comprehensive evaluation.
Considering the distance ranges of protein-ligand interactions
Baselines and Implementation. FAIR is compared with five stateoof-the-at representative baseline methods. PocketOptimizer 45) a physics- based method that optimizes energies such as packing and binding-related energies for ligand-binding protein design. DEPACT [ is a
method that follows two-step strategy (67 for pocket design. It first searches the protein-ligand complexes in the database with similar ligand fragments. It then grafts the associated residues into the
protein pocket with PACMatch g. HSRN (28), Diffusion (39], and MEAN (33 are deep-learning-
based models that use auto-regressive refinement, diffusion model, and graph translation method respectively for protein sequence-structure struuccre co-design. They were originally proposed for antibody design, and we adapted them to our pocket design task with proper modifications (see Appendix. for more details).
B
We use AMBER ff14S force field 41 for energy computation and the Dunbrack rotamer library for rotamer sampling in PocketOptimizer. For the template-basee method DEPACT, the template databases are constructed based on the training datasets for fair comparisons. For the deep-learning- based models, including our FAIR, we train them for 50 epochs and select the checkpoint with the lowest loss on the validation set for testing. We use the Adam optimizer with learning rate of 0.0001 for optimization. FAIR, the default setting sets T, and T2 as and 10. Descriptions of baselines and FAIR are provided in the Appendix.
55
B
7

---

Table 1: Evaluation of different approaches on the pocket design task.
CrokeDocked
Binding MOAD
Model
AAR (
RMSD (L)
Vina (
AAR (1)
RMSD (U)
Vina (1)
PocketÜptimizer
27.89-11..9%
1.75=0.08
6,905522.39
28.78=11.3%
1.68:0.12
-7.829-2.41
DEPACT
22.58.18.4899
1.97:10.14
6..70012.13
26.
1.76.0..1
-7.526-2.05
HSRN
11.42+10..4%
2.15=0017
16.555+1.95
33.70=10.1%%
1.83+0.18
-7.349-1.93
Diffusion
13.7%
1.68+0.12
1.8
36.94-+12.9%
1.47400099
-7.724-2.36
MEAN
35.46.28.15%
1.76=20.09
1.8891+1.86
37.16=11.7%%
1.524:0.09
-7.651=1.97
FAIR
40.17.112.6%
1.42.10.07
-7.022.11.75
43.75.11..2%
1.35.10.10
-7.978 1.91
Performance Metrics. To evaluate the generated sequences and structures, we employ Amino Acid Recovery (AAR), defined as the overlapping ratio between the predicted ID sequences and ground truths, and Root Mean Square Deviation (RMSD) regarding the 3D predicted structure of residue atoms. Due to the different residue types and numbers of side-chain atoms in the generated pockets, we calculate RMSD with respect to backbone atoms following [28] 33 To measure the binding affinity between the protein pocket and ligand molecules, we calculate Vina Score with QVina 61
The unit is kcal/mol; lower Vina score indicates better binding affinity. Before feeding to Vina, all the generated protein structures are firstly refined by AMBER force field [41 in OpenMM [33 to optimize the structures. For the baseline methods that predict backbone structures only (Diffusion and MEAN), we use Rosetta I
to do side-chain packing following the instructions in their papers.
4.2
Pocket Design
Comparison with Baselines. Table T shows the results of different methods on the CroseDocked and Binding MOAD dataset. FAIR overperforms previous methods with a clear margin on AAR, RMSD, and Vina scores, which verifies the strong ability of FAIR to co-design pocket sequences and structures with high binding affinity. For example, the average improvements on AAR and RMSD are 15.5% and 13.5% respectively. Compared with the energy-based method PocketOptimizer and the haaplaee-matching-aeaeee method DEPACT, deep-learning- -base methods such as FAIR have better performance due to their stronger ability to model the complex sequence-strrcture dependencies and pocket-ligand interactions. However, these geep-learning-bise based methods still have limitations that restrict their performance improvement. For example, the autoregressive generation manner in HSRN inevitably incurs error accumulation; Diffusion and MEAN fail model the influence of side chain atoms; all these methods ignore the flexibility of ligands. In contrast, FAIR addresses the limitations with properly designed modules, further validated in Sec. 443 Moreover, FAIR performs relatively better on Binding MOAD than the Cross dataset, which may be explained by the higher quality and diversity of Binding MOAD (21] (e.g. Vina score CrossDocked -7.022 vs. -7.978 on Binding MOAD). For example, Binding MOAD has 40k unique protein-ligand complexes, while CrossDocked only contains 18k complexes.
Comparison of Generated and Native Sequences. Figure (a) shows the residue type distributions of the designed pockets and the native ones from two datasets. Generally, we observe that the distributions align very well. Some residue types, such as Tyrosine and Asparagine, have been used more frequently in designed than naturally occurring sequences. In contrast, residue types with obviously reduced usages in the designed sequences include Lysine, Isoleucine, and Leucine.
Convergence of lterative Refinement Process. During the iterations of FAIR, we monitor the ratio of unconverged residues (residue types different from those in the final designed sequences) and AAR, shown in Figure ( (T, = 5 and T₂ 15). We observe that both curves converge quickly during refinement. We find that the AAR becomes stable in just several iterations (typically less than 3 iterations in our experiments) of backbone refinement and then gradually increases with the complete atom refinement, validating the effectiveness of our design. We show the curves of RMSD in Appendix C,3, which also converges rapidly. Therefore, we set Ti = 5 and T2 10 for efficiency as default setting.
Case Studies. In Figure. 3] we visualize two examples of designed pocket sequences and structures from the CrossDocked dataset (PDB: 2pc8) and the Binding MOAD dataset (PDB: 6p7r), respectively. The designed sequences recover most residues, and the generated structures are valid and realistic. Moreover, the designed pockets exhibit comparable or higher binding affinities (lower Vina scores) with the target ligand molecules than the reference pockets in the test set.
8

---

Reference (20c5)
Designed (2pca)
(6p77)
Designed (6p7r)
Packet Seq: EYHNNEYFOW
Pocket Seq: DEHNDYYEQW
Pocket Soc: YYRKVMM
Pocket Seq, FYMKGKM
Vina Scone: -5,88
Vina Score: -7.12
Vina Scone: -7.33
Vina Score: -7.74
Figure 3: Case studies of Pocket design. We show the reference and designed structures/seqquences of two protein pockets from the CrossDocked (PDB: 2pc8) and Binding MOAD datasets (PDB: 6p7r).
4.3 Additional Analysis of FAIR's Performance
Sampling Efficiency Analysis. To evaluate the efficiency of FAIR, we considered the generation time different approaches using single V V100 GPU on the same machine. Average runtimes are shown in Figure First, the generation time of FAIR is more than ten times faster (FAIR: 5.6s vs. DEPACT: 107.25 than traditional PocketOptimizer and DEPACT methods. Traditional methods are time-consuming because they require explicit energy calculations and combinatorial enumerations. Second, FAIR is more efficient than the autoregressive HSRN and diffusion methods. The MEAN sampling time is comparable to FAIR since both full-shot decoding schemes. However, FAIR has the advantage of full-atom prediction over MEAN.
Since RMSD values in Tables 1 and reflect only the global geometry of generated protein pockets, we also consider the RMSD of bond lengths and angles of predicted pockets to evaluate their local geometry (35. We measure the average RMSD of the covalent bond lengths in the pocket backbone (C-N, C=O, and C-C). We consider three conventional dihedral angles in the backbone structure, i.e., Φ s/. 57 and calculate the average RMSD of their cosine values. The results in Table Appendix C.3 show that FAIR achieves better performance regarding the local geometry than baseline methods. The RMSD of bond lengths and angles drop 20% and 8% respectively with FAIR.
Local Geometry of Generated Pockets.
Ablations. In Table 2 we evaluate the effectiveness and necessity of the proposed modules in FAIR. Specifically, we removed the residue-level encoder, the masked residue type update procedure, the full-atom refinement, and the ligand molecule context, denoted as w/o the encoder/consissteeyyffll atom/ligand. The results Table 2 demonstrate that the removal of any these modules leads to decrease in performance. We observe that the performance of FAIR w/o hierarchical encoder drops the most. This is reasonable, as hierarchical encoder captures essential representations for sequence/structtrre prediction. Moreover, the masked residue type update procedure helps ensure sequence-structt structure consistency. Additionally, the full-atom refinement module and the binding ligand essential for modeling side-chain and pocket-ligand interactions, which play important roles in protein- ligand binding.
Further, we replaced the iterative full-shot refinement modules in FAIR with the autoregressive decoding method used in HSRN (2] as a variant of FAIR. However, this variant's performance is also worse than FAIR's. s. As discussed earlier, autoregressive generation can accumulate errors and is more time-consuming than the full-shot refinement in FAIR. For example, AAR drops from 40. 17% to 31 1.0%% on CrossDocked with autoregressive generation.
Table 2: Ablation studies of FAIR.
CroksDocked
Binding MOAD
Model
AAR (f)
RMSD (
Vina ())
AAR (7)
RMSD (L)
Vina (})
w/o hier encoder
25.30++9.866
2.36+0.24
-6.69442.28
19.21
1.82+0.114
-7.235+2.10
w/o consistency
36.7519.219
1.49-0.08
-6.93711.72
37.40=15.0%
1.4911012
-7.752+1.94
w/o full-atom
35.52:1:00.4
1.62.1..13
-6412-12..7
35.91113.5%%
1.54:0.12
-7.544.12.23
w/o ligand
13.16+
1.71=0.11
-6.353+1.91
34.54+14.6%
1.50+0.09
-7,271+1.79
autoregressive
31.04:114.2%
1.56:1.16
1..87411.90
33.65+12.79%
1.46:10.11
-7.76541.84
FAIR
40.17::12.6%
0.02
-7.022:11.75
43.75:15.22%
1.35
-7.978111.1

---

THEM
Lewerege Kmalun Spps
OralDew
Steps
(b)
(c)
Figure 4: (a) The distributions of residue types in the designed pockets and the test sets. (b) The ratio of unconverged residues (residue types different from those in the final designed sequences) and AAR (Amino Acid Recovery) during the iterations of FAIR on the CrossDocked dataset. (c)
Comparisons of average generation time for 100 pockets with baseline methods.
5 Limitations and Future Research
While have made significant strides in addressing the limitations of prior research, several avenues remain for future research. First, we preliminarily formulate the protein pocket co-design problem in this paper. In the future, more realistic problem formulations may be constructed according to the requirements of different scenarios. Second, beyond the design of protein pockets, optimizing properties such the binding affinity of existing pockets is crucial, especially for therapeutic applications. In forthcoming research, we aim to combine reinforcement learning and Bayesian optimization with FAIR refine pocket design. Third, leveraging physics-based and template- matching methodologies can pave the way for the development of hybrid models that are both interpretable and generalizable. For instance, physics-based techniques be employed sample various ligand conformations during ligand initialization. The inherent protein-ligand interaction data within templates can guide pocket design. Lastly, integrating FAIR into protein/enzyme design workflows has the potential to be immensely beneficial for the scientific community. Conducting wet- lab experiments assess the efficacy protein pockets designed using FAIR would be advantageous. Insights gleaned from these experimental outcomes can be instrumental in refining our model, thereby establishing symbiotic relationship between computation and biochemical experiments.
6 Conclusion
We develop a full-atom iterative refinement framework (FAIR) for protein pocket sequence and 3D structure co-design. Generally, FAIR has two refinement steps (backbone refinement and full-atom refinement) and follows coarse-to-fine pipeline. The influence of side-chain atoms, the flexibility of binding ligands, and sequenne-struccture consistency are well considered and addressed. We empirically evaluate our method cross-docked and experimentally determined datasets to show the advantage over existing physics-based, semplate-matching-bsseee efficiently generating high-quality pockets. We hope our work inspire further explorations of pocket design with deep generative models and benefit the broad community.
and deep generative methods in
7
Acknowledgements
This research partially supported by grants from the National Key Research and Development Program China (No. (FF090090)))3) and the University Synergy Innovation Program Anhui Province (GXXT-2021-002) We wish express our sincere appreciation to Dr. Yaoxi Chen and Yifei Hu for their constructive discussions, which greatly enrich this research.
References
[1] Rebecca Alford, Andrew Leaver-Fay, Jeliazko Jeliazkov, Matthew O' O'Maaa, Frank DiMaio, Hahnbeom Park, Maxim V Shapovalov, P Douglas Renfrew, Vikram K Mulligan, Kalli Kappel, al. The rosetta all-atom energy function for macromolecular modeling and design. Journal of chemical theory and computation, -306)33031-3048 2017.
10

---

[2] Amr Alhossary, Stephanus Daniel Handoko, Yuguang Mu, and Chee-Keong Kwoh. Fast, accurate, and reliable molecular docking with quickvina 2. Bioinformatics, 31(13):2214-2216 2015.
[3] Ivan Anishchenko, Samuel J Pellock, Tamuka M Chidyausiku, Theresa A Ramelot, Sergey Ovchinnikov, Jingzhou Hao, Khushboo Bafna, Christoffer Norn, Alex Kang, Asim Bera, et al.
De novo protein design by deep network hallucination. Nature, 600(7889):547-555, 2021.
[4] Amos Bairoch. The enzyme database in 2000. Nucleic acids research, 28(1):304 0003044-0,,00 2000.
[5] Matthew Bick, Per Greisen, Kevin Morey, Mauricio S Antunes, David La, Banumathi Sankaran, Luc Reymond, Kai Johnsson, June Medford, and David Baker. Computational design of environmental sensors for the potent opioid fentanyl. Elife, 6:e28909, 2017.
[6] Edward Boas and Pehr B Harbury. Potential energy functions for protein design. Current opinion in structural biology, 17(2):199-204 2007.
[7] Yue Cao, Payel Das, Vijil Chenthamaraksshan Pin-Yi Yu Chen, Igor Melnyk, and Yang Shen. Fold2seq: A joint sequence (Id)-fold (3d) embedding- based generative model for protein design. In International Conference on Machine Learning, pages 1261-1271. PMLR, 2021.
[8] Yaoxi Chen, Quan Chen, and Haiyan Liu. Depact and pacmatch: workflow of designing de novo protein pockets to bind small molecules. Journal of Chemical Information and Modeling, 62(4):971-985, 2022.
[9] Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep learning-based protein sequence design using proteinmpnn. Science, 378(6615):49-3) 2022.
[10] Christopher M Dobson. Protein folding and misfolding. Nature, 426(6968):884 890, 2003.
[11] Jiayi Dou, Lindsey Doyle, Per Jr Greisen, Alberto Schena, Hahnbeom Park, Kai Johnsson, Barry L. Stoddard, and David Baker. Sampling and energy evaluation challenges in ligand binding protein design. Protein Science, 26(12):2426 2207662237222 2017.
[12] Xing Du, Li, Yuan-Ling Xia, Shi-Meng Ai, Jing Liang, Peng Sang, Xing-Lai Ji, and Shu Shu-Qun Liu. Insights into plodein-ligand interactions: mechanisms, models, and methods. International journal of molecular sciences, 17(2):144, 2016.
[13] Peter Eastman, Jason Swails, John D Chodera, Robert T McGibbon, Yutong Zhao, Kyle A Beauchamp. Lee-Ping Wang, Andrew Simmonett, Matthew P Harrigan, Chaya Stern, et al. Openmm 7: Rapid development of high performance algorithms for molecular dynamics. PLoS computational biology, 13(7)=e 2017.
[14] Noclia Ferruz and Birte Höcker. Controllable protein design with language models. Nature Machine Intelligence, 46)):521-532, 2022.
[15] Noelia Ferruz, Steffen Schmidt, and Birte Höcker. Protgpt2 is a deep unsupervised language model for protein design. Nature communications, 13(1) 13(1):4348, 2022.
[16] Paul G Francoeur, Tomohide Masuda, Jocelyn Sunseri, Andrew Jia, Richard B lovanisci, Ian Snyder, and David Koes. Three-dimensional convolutional neural networks and a cross- docked data set for structure-based drug design. Journal of chemical information and modeling, 60(9):4200-425 4215, 2020.
[17] Wenhao Gao, Sai Pooja Mahajan, Jeremias Sulam, and Jeffrey structural modeling and design. Patterns, 1(9): 100142, 2020.
J Gray. Deep learning in protein
[18] Anum A Glasgow, Yao-Ming Huang, Daniel J Mandell, Michael Thompson, Ryan Ritterson, Amanda L Loshbaugh, Jenna Pellegrino, Cody Krivacic, Roland A Pache, Kyle A Barlow, et al. Computational design of a modular protein sense-response system. Science, 10244448):1024- 1028, 2019.
[19] Jiaqi Guan, Wesley Wei Qian, Xingang Peng, Yufeng Su, Jian Peng, and Jianzhu Ma. 3d equiv- ariant diffusion for target-aware molecule generation and affinity prediction. In International Conference on Learning Representations, 2023.
[20] Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. ICML, 2022.
11

---

[21] Liegi Hu, Mark L Benson, Richard D Smith, Michael G Lemner and Heather A Carlson. Binding moad (mother of all databases). Proteins: Structure, Function, and Bioinformaties. 60(3):333-340, 2005.
[22] Po-Ssu Huang, Scott E Boyken, and David Baker. The coming of age of de novo protein design. Nature, 537(7620):320-3222 2016.
[23] Wenbing Huang, Jiaqi Han, Yu Rong, Tingyang Xu, Fuchun Sun, and Junzhou Huang. Equiv- ariant graph mechanics networks with constraints. arXiv preprint arXiv:2203.066 2244424442, 2022.
[24] Peter Huber. Robust estimation of a location parameter. Breakthroughs in statistics: Method-
ology and distribution, pages 492-518, 1992.
[25] Du Q Huynh. Metrics for 3d rotations: Comparison and analysis. Journal of Mathematical Imaging and Vision, 35(2):155-164: 2009.
[26] John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph-based protein design. Advances in neural information processing systems, 32, 2019.
[27] Lin Jiang, Eric A. Althoff, Fernando R. Clemente, Lindsey Doyle, Daniela Röthlisberger, Alexandre Zanghellini, Jasmine L. Gallaher, Jamie L. Betker, Fujie Tanaka, Carlos F. Barbas, Donald Hilvert, Kendall N. Houk, Barry L.. Stoddard, and David Baker. De novo computational design of retro-aldol enzymes. Science, 319(5868): 1387-1331 2008.
[28] Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Antibody-antigen docking and design via hierarchical structure refinement. In ICML, pages 10217-10227. PMLR, 2022.
[29] Wengong Jin, Jeremy Wohlwend, Regina Barzilay, and Tommi Jaakkola. Iterative refinement graph neural network for antibody sequence-structuue co-design. ICLR, 2022.
[30] Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael JL Townshend, and Ron Dror. Learn- ing from protein structure with geometric vector perceptrons. ICLR, 2021.
[31] John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ron- neberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-588 2021.
[32] Judith Sheeh-SSetharaman, Maki Oikawa, Shaun B Grimshaw, Julia Wirmer, Elke Duchardt, Tadashi Ueda, Taiji Imoto, Lorna J Smith, Christopher M Dobson, and Harald Schwalbe. Long-range interactions within a nonnative protein. Science, 171-6000:111111221 2002.
[33] Xiangzhe Kong, Wenbing Huang, and Yang Liu. Conditional antibody design as 3d equivariant graph translation. ICLR, 2023.
[34] Xiangzhe Kong, Wenbing Huang, and Yang Liu. End-to-end full full-atom antibody design. arXiv preprint arXiv:23322.00203 2023.
[35] Ivan V Korendovych and William F DeGrado. De novo protein design, a retrospective. Quarterly reviews of hiophysics, 53:e3, 2020.
[36] Brian Kuhlman and Philip Bradley. Advances in protein structure prediction and design. Nature Reviews Molecular Cell Biology, 20(1 1):68-6977, 2019.
[37] Meng Liu, Youzhi Luo, Kanji Uchino, Koji Maruhashi, and Shuiwang Ji. Generating 3d molecules for target protein binding. ICMI., 2022.
[38] Shengjie Luo, Tianlang Chen, Yixian Xu, Shuxin Zheng, Tie-Yan Liu, Liwei Wang, and Di He. One transformer can understand both 2d & 3d molecular data. ICLR, 2022.
[39] Shitong Luo, Yufeng Su, Xingang Peng, Sheng Wang, Jian Peng. and Jianzhu Ma. Antigen- specific antibody design and optimization with diffusion-bart generative models. NeurIPS, 2022.
[40] Youzhi Luo and Shuiwang Ji. An autoregressive flow model for 3d molecular geometry generation from scratch. In ICLR, 2021.
[41] James A Maier, Carmenza Martinez, Koushik Kasavajhala, Lauren Wickstrom, Kevin E Hauser, and Carlos Simmerling. ff14sb: improving the accuracy of protein side chain and backbone parameters from ff99sb. Journal of chemical theory and computation, 1168):369633713, 2015.
[42] Christoph Malisi, Marcel Schumann, Nora C. Toussaint, Jorge Kageyama, Oliver Kohlbacher, and Birte Höcker. Binding pocket optimization by computational protein design. PloS one, 7(12): 2012.
12

---

[43] Gilles Marcou and Didier Rognan. Optimizing fragment and scaffold docking by use of molecular interaction fingerprints. Journal of chemical information and modeling, 47(1):195- 207, 2007.
[44] Erxue Min, Runfa Chen, Yatao Bian, Tingyang Xu, Kangfei Zhao, Wenbing Huang, Peilin Zhao, Junzhou Huang, Sophia Ananiadou, and Yu Rong. Transformer for graphs: An overview from architecture perspective. arXiv preprint arXiv:2202.08555. 2022.
[45] Jakob Noske, Josef Paul Kynast, Dominik Lemm, Steffen Schmidt, and Birte Höcker. Pocketop- timizer 2.0: A modular framework for computer-aided findin-binding design. Protein Science, 32(1))e4516, 2023.
[46] Sergey Ovchinnikov and Po-Ssu Huang. Structure-based protein design with deep learning. Current opinion in chemical biology, 13444,444, 2021.
[47] Xingang Peng. Shitong Luo, Jiaqi Guan, Qi Xie, Jian Peng, and Jianzhu Ma. Pocket2mol: Efficient molecular sampling based on 3d protein pockets. ICML, 2022.
[48] Nicholas F Polizzi and William F DeGrado. A defined structural unit enables de novo design of small-molecule-bindin proteins. Science, 1696650) 1227-1233, 2020.
[49] Matthew Ragoza, Tomohide Masuda, and David Ryan Koes. Generating 3d molecules condi- tional receptor binding sites with deep generative models. Chemical science, 13(9):2701- 2713, 2022.
[50] Anthony K Rappé, Carla J Casewit, KS Colwell, William A Goddard III, and W Mason Skiff. Uff. full periodic table force field for molecular mechanics and molecular dynamics simulations. Journal of the American chemical society, 1 11025)::0024-00033
1992.
[51] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, (88115))2006239111 2021.
[52] Daniela Röthlisberger, Olga Khersonsky, Andrew M Wollacott, Lin Jiang, Jason DeChancie, Jamie Betker, Jasmine L Gallaher, Eric A Althoff, Alexandre Zanghellini, Orly Dym, et al. Kemp elimination catalysts by computational enzyme design. Nature, 15307192) 2008.
[53] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In European semantic web conference, pages 593-607. Springer, 2018.
[54] Arne Schneuing, Yuanqi Du Charles Harris, Arian Jamasb, Ili Igashov, Weitao Du Tom Blundell, Pietro Lió, Carla Gomes, Max Welling, Michael Bronstein, and Bruno Correia. Structure-based drug design with equivariant diffusion models. arXiv preprint arXiv:2210. 13699, 2022.
[55] Maxim V Shapovalov and Roland L Dunbrack Jr. A smoothed backbone-dependent rotamer library for proteins derived from adaptive kernel density estimates and regressions. Structure, 19(6):444 858, 2011.
[56] Chence Shi, Chuanrui Wang, Jiarui Lu, Bozitao Zhong, and Jian Tang. Protein sequence and structure co-design with equivariant translation. ICLR, 2023.
[57] Ryan K Spencer, Glenn L Butterfoss, John R Edison, James R Eastwood, Stephen White- lam, Kent Kirshenbaum, and Ronald N Zuckermann. Stereochemistry of polypeptoid chain configurations. Biopolymers, 1106)):23266, 2019.
[58] Martin Steinegger and Johannes Süding. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology. 35(1 (1116))0026-022 2017.
[59] Stephen Stoker. General, organic, and biological chemistry. Cengage Learning, 2015. [60] Arthur G Street and Stephen L Mayo. Computational protein design. Structure, 105-R1099,, 1999.
[61] Oleg Trott and Arthur J Olson. Autodock vina: improving the speed and accuracy of docking with a new scoring function, efficient optimization, and multithreading. Journal of computational chemistry, 31(2):455-45 2010.
13

---

[62] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[63] Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization the transformer architecture.
In International Conference on Machine Learning, pages 10524-10533, PMLR, 2020.
[64] Minkai Xu, Lantao Yu. Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. ICLR, 2022.
[65] Xihong Yang, Jiaqi Jin, Siwei Wang, Ke Liang, Yue Liu, Yi Wen, Suyuan Liu, Sihang Zhou, Xinwang Liu, and En Zhu. Dealmve: Dual contrastive calibration for multi-view clustering. In Proceedings of the 3 3/th ACM International Conference on Multimedia, 2023.
[66] Xihong Yang, Cheng Tan, Yue Liu, Ke Liang, Siwei Wang, Sihang Zhou, Jun Xia. Stan Z Li, Xinwang Liu, and En Zhu. Convert: Contrastive graph clustering with reliable augmentation. In Proceedings of the 31th ACM International Conference on Multimedia, 2023.
[67] Alexandre Zanghellini, Lin Jiang, Andrew Wollacott, Gong Cheng, Jens Meiler, Eric A Althoff, Daniela Röthlisherger, and David Baker. New algorithms and in silico benchmark for computational enzyme design. Protein Science, (112)))277727999 2006.
[68] Zaixi Zhang, Qi Liu, Qingyong Hu, and Chee-Kong Lee. Hierarchical graph transformer with adaptive node sampling. Advances in Neural Information Processing Systems, 2022.
[69] Zaixi Zhang, Qi Liu, Chee-Kong Lee, Chang- Yu Hsich, and Enhong Chen. An equivariant generative framework for molecular graph-structure co-design. Chemical Science, 14(31):8380- 8392, 2023.
[70] Zaixi Zhang, Yaosen Min, Shuxin Zheng, and Qi Liu. Molecule generation for target protein binding with structural motifs. In ICLR, 2023.
[71] Zaixi Zhang, Jiaxian Yan, Qi Liu, Enhong Chen, and Marinka Zitnik. A systematic survey in geometric deep learning for aructure-based drug design. arXiv preprint arXiv:2306. 11768, 2023.
14

---

A Details on FAIR Model
A.1 Training and Sampling Algorithms
Here, we show the poeudo-codes of the training and sampling of FAIR in the Algorithm. and
Algorithm Training of FAIR
Input:
protein sequences A = a1 ... aN, and structures { {a ligand molecule M. Initialize the coordinates of pocket residues {æ<(b))-
Add Gaussian noise to the ligand molecular structure {
Initialize the sequences of pocket residues with uniform distributions over 20 residue categories. Cpred = 0; Petruct = O. # Backbone refinement for 1 1.... Ty:
Obtain the residue embeddings (fr} and atom embeddings (h,} with the hierarchical encoder.
Predict residue types and update coordinates with structure refinement modules.
Csen+ Lee (p) pi). Estruct+ = 2; (nuber (a(b.)) .(b.)) + ΣS Juuber 2z))
end for
Sample residue types from the predicted residue type distributions. Initialize the coordinates of side-chain atoms with the coordinates of corresponding Ca. # Full-atom refinement for t in 1.... T2:
Randomly mask subset of pocket residues. Obtain the residue embeddings {fr] and atom embeddings {h:} with the hierarchical encoder. Predict the residue types of masked residues.
Csey+ = Σ, Loe (P).):).
Reinitialize the masked residue if its residue type conflicts with the predicted result. Update coordinates with structure refinement modules.
Letract+ E; (ar (b,)) + Σ3 |nueer (ar) at)).
end for
Minimize + (Cooq + Latruct).
Algorithm 2 Sampling of FAIR
Input: protein sequences A a1 aN, and structures ( ligand molecule M. Initialize the coordinates of pocket residues (x(b,))
Initialize the sequences of pocket residues with uniform distributions over 20 residue categories.
# Backbone refinement
for 1.... Ty:
Obtain the residue embeddings {fr} and atom embeddings (h:) with the hierarchical encoder.
Predict residue types and update coordinates with structure refinement modules.
end for
Sample residue types from the predicted residue type distributions. Initialize the coordinates of side-chain atoms with the coordinates of corresponding C... Full-atom refinement
for t in 1, ... T2:
Randomly mask subset of pocket residues. Obtain the residue embeddings {ff. and atom embeddings {h,} with the hierarchical encoder. Predict the residue types of masked residues.
Reinitialize the masked residue if its residue type conflicts with the predicted result. Update coordinates with structure refinement modules.
end for
15

---

A.2 Additional Information on Hierarchical Graph Transformer Encoder
Our hierarchical encoder includes the atvel encoder and the residue-level encoder. The powerful 3D graph transformer is used as the model backbone. Here, we first provide additional information on the graph transformer. Residue-level node and edge feature representations follow prior research
126 28
Graph transformer architecture. The atom/residuee-lvvl encoder contains L graph transformer layers. Let h0 be the set of node representations at the (-th layer. In each graph transformer layer, there is a multi-head self-attention (MHA) and feed- forward block (FFN). The layer normalization (LN) is applied before the two blocks [63] [68]. The details of MHA have been shown in See 3.2 and the graph transformer layer is formally characterized as:
(1-11) MHA(LN(( (-")))) + h (1-1)
(11)
h (1) = FFN ((-D)) + (11)) (0
(12)
Residue-level node features. The residue-level encoder only keeps the C., atoms to represent residues and constructs K, nearest neighbor graph at the residue level. We use the original protein pocket backbone atoms for reference to construct the K, nearest neighbor graph and help calculate geometric features, Each residue node is represented by six features: polarity fp E {0. 1). hydropathy fn E [-4.5, 4.5]. volume fo E [60.1, 22..8]. charge fo E {-1, 0, 1}. and whether it is hydrogen bond donor fa E {0, } or acceptor fa E 13. We expand hydropathy and volume features into radial basis with interval sizes 0.1 and 10, respectively. Overall, the dimension of the residue-level feature vector f: is 112.
Residuc-level edge features. For the i-th residue, we let ((...) denote the coordinate of its C,, and define its local coordinate frame O = [Cs; ni; C2 x n; as:
x(a.,1) x(1;-1.1) U: [.<(.,,) æ a.;-,,,))ll
u: 4++1
Ug X Ui+1
n2
(13)
| us+
u; ui+
Based on the local frame, the edge features between residues and j can be computed as:
x(a,,1) - x x(a.1)) [=(a..)) =(a,,)Il (0, O₂))
res C
Concat
Eipos (i- CRBF -a (() O.T
(14)
The edge feature ejj contains four parts. The positional encoding Epos (i j) encodes the relative sequence distance between two residues. The second term eNBr(.) is distance encoding with radial basis functions. The third term is a direction encoding corresponding to the relative direction of
a a(a,,1) in the local frame of the i-th residue. The last term (0, 02) is the orientation encoding of
the quaternion representation q(-) of the spatial rotation matrix O Oj 25 Overall, the dimension of the residue-level edge feature e/ is 39.
A.3
More Details of the Full-atom Structure Refinement
The full-atom structure refinement is similar to the backbone structure refinement in Sec. B.3.I with minor modifications to accommodate side-chain atoms. We consider the internal interactions within the protein and the external interactions between ligand molecules and protein residues. Firstly, the internal interactions within the protein are calculated based on residue embeddings:
x(bs,)) - x(ax)
9ijAj = g(fi, fk, enBr (b...) =<.a,)))))))
(15)
ax (b.(b,)) x(ax)ll
where ERAF (( x x<a,,)))) IIl) is the radial basis distance encoding of pairwise C₂ distances, the function y feed-forward neural network with 14 channels for all side-chain atoms (the maximum number of atoms in a residue is 14). Due to the different numbers of side-chain atoms in different kinds of residues, the normalized direction vector here is =(b.,))- (=(b.,))_((a) E IR3 where z(as) is the average coordinates of residue ag. The external interactions between pocket B and ligand M is the same as the backbone refinement:
x x(b,,) E
gij. = g'ho..y: hgg CRBE (Iaarb - x=ll))
(16)
(x(b,)) - x
16

---

which is calculated based on the atom representations. For the conciseness of expression, here we use hb.,, and h, denote the pocket and ligand atom embeddings, respectively. Finally, the coordinates of pocket atoms x ((b.,)) and ligand molecule atoms w, are updated follows:
x(br,) t x(bs,)) +
Joj.kj
Σ Irj.e
(17)
Kr
Ni
kEN(()
3g
3g
Σ
99.. 1 i m, S m4 S
(18)
B
where (1) indicates the neighbors in the constructed residue-level K,-NN graph: [] denotes the number pocket atoms.
A.4 Structure initialization based on interpolation and extrapolation
Residues with known structure
Linear interpolation
Linear extrapolation
Figure 5: Structure initialization based on interpolation and extrapolation
Figure illlustrates our structure initialization strategy. We initialize the residue coordinates with linear interpolations and extrapolations based on the nearest residues with known structures in the protein. Denote the sequence of residues A 01 a,,. where N₂ is the length of the sequence. Let Z(a,) R3x4 denote the backbone coordinates of the i-th residue. If we want to initialize the coordinates of the i-th residue, we take the following initialization strategy: (I) We use linear interpolation if there are residues with known coordinates at both sides of the i-th residue. Specifically, assume p and q are the indexes of the nearest residues with known coordinates each side of the i-th residue (p i < q), we have: ((a)) - --p1(t
(ag)) + (g i) [ap)) (2) We conduct linear extrapolation if the i-th residue is at the ends of the chain, i.e., no residues with known structures one side of the i-th residue. Specifically, let p and q denote the index of the nearest and the second nearest residue with known coordinates. The position of the i-th residue can be initialized as (a,) = (aa,) + i-p coordinate of their corresponding C.. In practice, we can further apply local geometric constraints to the initialized backbone atoms in each residue.
p-q (Z(pp) - Z(ag)). We initialize the side-chain atoms' coordinates with the
A.5 Equivariance.
FAIR has the desirable property of Eu)iiiuivaree as follow:
Theorem 1. Denote the E(3 transforttto as Tg and the generative process of FAIR as {(b,, Tm (A \ B, M). where {(b,, x(b.)) indicates the designed pocket seqeuce and structure. We huve [(b., Tg xxbb)) (T, (), M).
The E(3)- transfomattionn on the euclidean coordinate I can be represented as: T, x = Oz + t, where O E H3 is the orthogonal transformation matrix, f E 13 is the translation vector. Before we give the proof, we present the following lemmas.
Lemma 1. The hierarchical graph transformer encoder is E(3)-invrriant.
Proof. As shown in Sec. 3.2 the input to the hierarchical graph transformer encoders are scalar node features and pairwise distances, and the output atom and residue sealar features. Therefore, hierarchical encoder is E(3)-invariant by construction.
17

---

Lemma 2. The backbone and full-aton ftll-atom structure refinement are Eg3)-eruivariant.
Proof. Firstly, with the E(3)-invariant encoder, the obtained residue embeddings {fi), atom embed-
dings {, and the output of functions g and g' are -1vaariiaiiant.. Furthermore, VIg E E(3), it easy to have To (xx(b.) x (xxx))) = Oz +t Oz(akk,) - = - Similarly, we have To, (xxb..) x.) = (b,)) xx) and T. (x(b. z(as)) (zbb))))) x ({ax)) ))
Let 2 (bb,)) denote the updated pocket coordinates in Equation. and 17 is then possible to derive
that the updated pocket coordinates are E(3) eqqivvariat as follows:
Tg (..((b.))) + +Tg Kr kEN(()
1
E
Dij.kj + Tg
Ni Dij.
1 Σ
(19)
Ox((,,)) + 1
Σ
Joj.kj +0 Ni
1 Σ gij.s
(20)
Kr kEN(
Olz(ba,)) + Kr kEN())
1
Σ
Dij.kj +
Σ
gij.ol +
(21)
Ni
= Tg a(bs,))'
(22)
Similarly, for the updated ligand coordinates, we have:
Tg - Vg - Tg
gijis
(23)
BB]
fj
Oz, + t 0
gij.*
(24)
B
i
O x.
gis.el
+
(25)
B
Tg x.y
(26)
Here, a', denote the updated ligand coordinates in Equation. and 18 Therefore, the structure refinement modules in backbone and full-atom refinement are E(3) Eq)iciuriarnn
Proof. The predicted residue types are obtained by applying softmax and MLPs on the hidden
representations from the E(3) -(3)-invariant encoder. With the above lemmas, Theorem 1 can be proved since the E(3)-invariant encoder and the Eq3)-equivariant structure refinement constitute the E(3)- equivariant generative process of FAIR. Formally, we have:
[b,,TT (bb))) Po
M))
(27)
which concludes Theorem 1
A.6 Huber Loss
In Equation 10 use Huber loss (24] as the structure refinement loss for the stability of optimization, which is defined as follows:
0.5 (x -) if x y/ C, (lz yl 0.5 c), else.
Uhuber (aober(xx,))) (x.1) v)
(28)
where and y represent the predicted and ground-truth coordinates. The Huber loss has the following property: if the L1 norm of y/ is smaller than c, it is MSE loss, otherwise it is L1 loss. At the beginning of the model training, the deviation between the predicted and ground- -otut coordinates is large, and the LI term makes the loss less sensitive to outliers than MSE loss. The deviation is small when the training is almost complete, and the MSE loss is near In practice, we find that directly using MSE loss sometimes leads to NaN at the beginning of the training, while Huber loss makes the
training procedure more stable. Following the suggestion of previous works 28 33), all our experiments.
we set in
18

---

B Implementation Details
is physic-based computational protein design method that predicts muta- tions in the binding pockets of proteins increase affinity for specific ligand. Its previous version is [42] Generally, there are four main steps in PocketOptimizer: structure preparation, flexibility sampling, energy calculations, and computation of design solutions. Specifically for the energy calculations, both packing-related energies and binding-reladd related energies are considered. As for the output design solutions, we select the top 100 designs identified by PocketOptimizer based on integer linear programming for downstream metric calculations. Following the suggestions in the original paper, we use AMBER ff14S force field 44 for energy computation and the Dunbrack rotamer library [55] for rotamer sampling for PocketOptimizer in our implementation.
PocketOptimizer 45
DEPACT g is template-matchin method that follows the two-step strategy of RosettaMatch
677 for pocket design and has better performance. It first searches the protein-ligand complexes in the database with similar ligand fragments and constructs a cluster model (a set of pocket residues). It then grafts the cluster model into the protein pocket with PACMatch, which places residues in pocket clusters on protein scaffolds by matching key atoms in the cluster model with key atoms of the protein scaffold. The qualities of the generated pockets are evaluated and ranked based on a statistical scoring function. We take the top 100 designed pockets for evaluation. The template databases are constructed in our experiments based on the corresponding training datasets for fair comparisons.
was originally developed for antibody design. It autoregressively predicts the residue sequence and docks the generated structure. To adapt HSRN to our pocket design task, we replace the antigen with the target ligand molecule to provide the context information. We finetune HSRN's hyperparameters based on its performance on the validation set. The hidden size in the message- passing network is 256, the number of layers is 4, and the number of RBF kernels 16.
HSRN 28
Diffusion 39 jointly models both the sequence and structure of an antibody using diffusion probabilistic estimates and equivariant neural network. The diffusion model first aggregates information from the antigen and the antibody during generation. It then iteratively updates each amino acid's s type, position, and orientation. Since the side-chain atoms are not modeled, they reconstructed at the last step with side-chain packing. To adapt Diffusion to our pocket design task, replace the antigen with the target ligand molecule provide the context information. We adjust the key hyperparameters based on the results of the validation sets. The number of layers, hidden dimension, and diffusion steps are 6, 128, and 100, respectively.
MEAN 33 (tackles antibody design as a conditional graph translation problem with the modified GMN 2 as encoders. Specifically, MEAN co-designs antibody sequence and structure via multi- round progressive full-shot scheme, which is more efficient than regressgre or diffusion based approaches. However, it only models protein backbone atoms and neglects side chains, which are important for pottein-pootein and protein- task, we replace the antigen with the target ligand molecule to provide the context information. The hyperparameters are finetuned based on the validation set for hetter performance. We set the hidden size 128, the number of layers as 3, and the number of iterations for decoding as 3.
interactions. To adapt MEAN to our pocket design
FAIR is our full atom pocket sequence-structuse co-design method. The number of layers for the atom and residue-level encoder are and 2, respectively. Ka and K, are set as 24 and 8 respectively. The number of attention heads set as 4; The hidden dimension d is set as 128. The standard deviation of the Gaussian noise added to the ligand coordinates in model training is 0.1.
htupss//jgttbbbcoom/wwwgggggggnd PACMatch PACMac
19

---

Reverepe Tym
1.3
10 Docesion Steps
20
20
Incastion Strps
Berosion Stops
(a)
(h)
(C)
Figure 6: (a) The ratio of unconverged residue types and AAR during the iterations of FAIR on Binding MOAD; (b) The convergence plot of RMSD on CrossDocked; (c) The convergence plot of RMSD on Binding MOAD.
C More Experimental Results
C.1 Pocket Generation without Backbone Reference
In FAIR, the original protein backbone coordinates are leveraged for residue-level KNN graph construction for training and generation stability. However, FAIR also work well without protein backbone coordinates as a reference. our supplementary experiments, FAIR can still achieve comparable performance.
C.2
Convergence of Iterative Refinement
We show the RMSD curves in Table (b) & (c). We observe the RMSD converges rapidly as the protein pocket is refined. The curve has fluctuations near step 5, potentially caused by the initialization of the side-chain atom the beginning of full-atom refinement. The coordinates of side-chain atoms are randomly initialized near the Co and may distort the backbone atoms with the structural refinement. However, the RMSD of FAIR becomes steady quickly after step 10, which again shows the effectiveness of FAIR.
C.3 Local Geometry of the Generated Pockets
Since the RMSD used in Tables | and 2 reflects only the correctness of global geometry in generated protein pockets, we also calculate the RMSD bond lengths and angles in the designed pocket to validate the local geometry following previous work [33]. We measure the average RMSD of the covalent bond lengths in the pocket backbone (C-N, C=O, and C-C). The three conventional dihedral angles in the backbone structure, i.e., Φb 1h, [57] are considered, and the average RMSD of their cosine values are calculated. The results in Table 3 demonstrate that FAIR still achieves much better performance regarding the local geometry than the baseline methods.
Table 3: The average RMSD of bond lengths and dihedral angles of the generated backbone structures on the two datasets.
Models
Bond lengths
Angles (Φ, w)
PocketOptimizer
0.47
0.328
DEPACT
0.68
0.360
HSRN
0.76
0.437
Diffusion
0.49
0,344
MEAN
0.44
0,312
FAIR
0.35
0.287
C.4 Hyperparameter Analysis
Here, we evaluate the influence of hyperparameter choices on FAIR performance. We focus on two hyperparameters, Ti and Th, for iterative refinements. In Figure 7 observe that the performance
20

---

of FAIR is stable when Tj and T2 are larger than 5 and 10, respectively, indicating the refinement processes gradually converge. We set Ti and 22 to 5 and 10 throughout our experiments.
-7.0
AAR0%)
15
-7.5
20
CroaDecoo
T1
(a)
(b)
(c)
1.55
-55
40-
1.50
-70
AAR(%)
1.45
-7.
1.40
ExoaDhaosdd Meakin HOAD
Beiki MDAD
16
14
72
T2
(d)
(e)
(I)
Figure 7: Hyperparameter analysis with respect to Tj (first row) and T2 (second row).
21


Reviews:
[{"review_id": "VYAfDIooDX", "reviewer": "Reviewer_rsaq", "summary": "The Full-Atom Iterative Refinement framework (FAIR) is a novel approach for designing functional proteins that bind with specific ligand molecules. FAIR consists of two steps: full-atom generation and 3D structure co-design. It uses a coarse-to-fine pipeline, updating residue types and structures together in each round. FAIR outperforms baselines in efficiently designing high-quality pocket sequences and structures, with average improvements on AAR and RMSD exceeding 10%.", "questions": "1. Since Transformer with considerable parameters is used, please try to compare the parameters of the proposed method and other deep learning-based methods in the experimental section.\n\n2. As stated in line 196 and 199, the authors use two feed-forward neural networks (g and g’) to encode internal and external interactions, respectively. Apart from the difference in the output channels, are they modeled with the same structure? If it is the same, then try to explain why two identical models (refer to Eqs. (5) and (6)) are used to model different interactions?\n", "limitations": "The method proposed in the paper utilizes a two-stage iterative refinement strategy. However, the internal logic of these two refinement stages is not well correlated. Alternatively, why can't components such as the masking mechanism of the second stage be directly injected into the first stage?", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper investigates protein pocket design, which determines the pocket structure and sequence based on the context of the protein scaffold and the binding ligand molecule.\n2. It provides an end-to-end generative framework called FAIR that uses iterative refinement to co-design the pocket sequence and structure. FAIR solves the drawbacks of earlier research and effectively considers sidechains, ligand flexibility, and consistency of sequence structure for effective prediction.\n3. FAIR outperforms baseline techniques in terms of various pocket design parameters. The gains on AAR and RMSD are often over 10%. FAIR generates data more than ten times quicker than conventional techniques.\n", "weaknesses": "Figure 2 of the paper shows that the proposed FAIR adopts a two-stage learning process, so it is not an end-to-end deep learning framework. Therefore, a necessary concern is the training time of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable questions! We have provided detailed responses to your comments. Please let us know whether we have addressed your concerns. We will be very grateful if you consider increasing scores to support our work.\n\n**Comment 1:** Figure 2 of the paper shows that the proposed FAIR adopts a two-stage learning process, so it is not an end-to-end deep learning framework. Therefore, a necessary concern is the training time of the proposed method.\n\n**Response 1:** Figure 2 shows that there are generally two modules in FAIR, i.e., the backbone refinement and the full-atom refinement modules. However, the two main modules in Figure 2 do not indicate a two-stage learning process. Instead, the training and inference of FAIR are in an end-to-end fashion similar to other deep learning-based baselines. In experiments, it takes around 20 hours to train a FAIR model on a single V100 GPU, which is comparable to MEAN, HSRN, and Diffusion. The results of the generation efficiency comparison in Figure 4(c) further show the advantages of our method.\nWe will illustrate our method more clearly in the final version.\n\n**Comment 2:** Since Transformer with considerable parameters is used, please try to compare the parameters of the proposed method and other deep learning-based methods in the experimental section.\n\n**Response 2:**\nHere we show the number of parameters of FAIR and the other deep learning-based baseline models. We can see that FAIR has comparable or fewer parameters than the baseline methods. Besides the transformer schemes, other factors such as model architecture, hidden dimension size, and the number of layers influence the number of parameters. Generally, FAIR is a light and efficient model.\n\n| Model          | HSRN | Diffusion | MEAN | FAIR |\n|----------------|-------------------|--------------------|---------------------|---------------------|\n| Parameters |8.77M     | 4.00M| 0.70M |0.73M|\n\n\n**Comment 3:** As stated in lines 196 and 199, the authors use two feed-forward neural networks (g and g’) to encode internal and external interactions, respectively. Apart from the difference in the output channels, are they modeled with the same structure? If it is the same, then try to explain why two identical models (refer to Eqs. (5) and (6)) are used to model different interactions?\n\n\n**Response 3:** \nAs discussed in lines 192-194, according to previous works [32, 12, 10], the internal interactions within a protein and the external interactions between a protein and a ligand have different properties. For this reason, FAIR uses two separate modules for interaction predictions. As for Equ. (5), it focuses on interactions within protein residues. The input are the residue embeddings and the distance encodings of pairwise alpha carbon distances. There are four output channels for four backbone atoms. As for Equ. (6), it focuses on the external interactions between the pocket and ligand. The input are atom embeddings and corresponding distance encodings. There is only one output channel. Due to the differences in input, output, and modeling interactions, we cannot use a single network.\nWe will clarify and improve the description of the method in the final version.\n\n\n**Comment 4:** The method proposed in the paper utilizes a two-stage iterative refinement strategy. However, the internal logic of these two refinement stages is not well correlated. Alternatively, why can't components such as the masking mechanism of the second stage be directly injected into the first stage?\n\n**Response 4:** Thanks for the question! FAIR is designed to have two main stages that follow a coarse-to-fine pipeline: FAIR firstly only models the backbone atoms of pockets to generate the coarse-grained structures and then fine-adjusts full-atom residues to achieve sequence-structure consistency. \n\nAt the first stage, since the pocket residue types and the number of sidechain atoms are largely undetermined, we only model the backbone atoms and residue types. \nThe masking mechanism is not appropriate for the first stage since the predicted residue types are less stable and reliable in early iterations.  \n\nAt the second stage, we sample and initialize the residues types and side chain atoms based on the results from the first stage. \nWith the masking mechanism, the residue type and structure update gradually converge and the residue sequence-structure consistency is achieved.\n\nTherefore, the two refinement stages are well correlated.\nWe will add more discussions of our algorithm in the final version.\n"}, {"review_id": "MrMU5NeEg4", "reviewer": "Reviewer_z6bW", "summary": "This paper is the first to introduce a deep learning pipeline for the protein binding pocket re-design task. The architecture consists of a rotation invariant, hierarchical encoder at the residue and all atom level, followed by a hierarchical iterative refinement generative process at the residue and all atom level. The architecture aims to model key inductive biases of the problem at hand, and shows empirical improvements over baselines adopted from similar protein/molecule design tasks.", "questions": "- What is the Vina docking score of the training set? \n    - It would be useful to have this as a baseline to compare all the methods to.\n    - Figure 3 provided Vina score for the reference structures. I'd be keen to see the average Vina score across the train/validation/test sets as an entry in Table 1, too.\n\n- Can you also show Vina docking scores for all the models without re-docking/relaxation?\n    - In the small molecule generation literature, it has been found that many generative models may generate very unrealistic ligand poses and that the Vina score without relaxation is poor.\n\n- Paragraph on line 302 onwards - Why are some residues used more and some lesser by the model? Is this observation related to binding, eg. hydrophobicity properties of residues?\n   - If you are mentioning these observations, it may be worth explaining why...\n\n- How do we know the binding pose of the ligand in advance in real-world design scenarios? Do we always need to start with an existing (crystal) structure of the protein-ligand complex?", "limitations": "The authors have addressed technical limitations but not potential negative societal impact.\n\nOverall, I would encourage the authors to further discuss WHY binding pocket re-design is a meaningful real-world task in more detail that given at present, how it is relevant for designing bio-sensors and/or enzymes (probably I am wrong in my understanding re. enzyme design), and how we can evaluate these tasks in a meaningful manner in-silico. What are some limitations which we cannot address computationally? Perhaps this may not be very relevant if we are judging this paper only from the machine learning perspective, but better contextualizing this project beyond just ML could be useful to readers.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Overall sensible pipeline and inductive biases for all-atom protein pocket redesign.\n   - The FAIR pipeline consists of a residue level, rotation-invariant encoder followed by all atom level encoder to encode the protein in a hierarchical manner, after which new residues are placed and iteratively refined in a hierarchical, rotation-equivariant manner. During iterative refinement, the atom positions of the ligand (small molecule) are also updated to incorporate ligand flexibility.\n   - From a biochemical perspective, considering all-atom/atom level encoding and generation is probably very important for binding, as it is the sidechain orientations that determine ligand binding.\n   - These are all sensible inductive biases -- rotation symmetry, hierarchy, iterative redesign, ligand flexibility -- and have been described clearly.\n   - None of these are particularly novel components, because the technical problems have been solved in other work, but this is an effective application of existing ideas to a novel problem of all-atom pocket redesign.\n\n- Well written paper.\n    - Clear description of each architectural components as well as experimental methodology.\n    - I really loved the paragraph starting at line 294 onwards. This paragraph (and the overall paper) does at good job at showing how each architectural component of FAIR improves over existing papers/frameworks and is adapted to the specific problem at hand.\n    - I do wish that the authors found a way to not make the reader jump to the appendix so many times.\n\n- Comparison to baselines.\n    - Care has been taken to adapt existing papers/methods to the new pocket design task in order to provide baselines.", "weaknesses": "- New task may not require major new technical innovations.\n    - While the binding pocket re-design task is new in the deep learning literature to the best of my knowledge, it seems that adapting existing architectural ideas and putting them together in a smart pipeline (see strengths) works well.\n    - From a machine learning perspective, I could not identify a new technical problem that previous works have not solved. (I may be missing something.) \n\n- Unsure whether pocket re-design is relevant to enzyme design.\n    - I will caveat this weakness by saying I am not a domain expert at all.    \n    - I could immediately see this pipeline be applied for general binding pocket re-design and biosensor applications. One perhaps starts out with a structure of a protein-ligand complex and wants to improve binding affinity. However, perhaps this setup is a bit of a stretch for enzyme design.\n    - For enzymes, we really would ideally like to keep the binding site conserved instead of re-designing it, based on my understanding of the field. In theory, it is of course nice if you can design a new binding pocket to discover new kinds of chemistry. However, as far as I am aware, basic scientists do not currently understand enough about catalytic mechanisms to be able to design a new one from scratch. \n    - Thus, designing a new pocket and **claiming that it catalyzes the same reaction via the same/different mechanism** seems very hard to tackle from a basic science perspective, let alone computationally. Reasons: lack of data and proper annotation of enzymatic mechanisms.\n\n- Hard to evaluate binding pocket re-design.\n    - I felt it was overall challenging to design evaluation metrics and setups for this task, because it is unclear whether recovering the RMSD and/or AA identities of the original ligand binding site is the best outcome for these models.\n    - I have several questions about the Vina docking score evaluations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for their constructive comments!  We hope our detailed response and added experiments better highlight FAIR’s novel contributions. If any remaining questions/concerns make you hesitate to raise the score, we would be grateful if you let us know so we could further improve our work.\n**Due to the limits of the rebuttal, we show additional responses to some comments in the global response.**\n\n**Comment 1:** \nWhile the binding pocket re-design task is new in the deep learning literature to the best of my knowledge, it seems that adapting existing architectural ideas and putting them together in a smart pipeline (see strengths) works well.\n\n**Response 1:** Thanks for the question! In this paper, we study protein pocket design that designs the pocket sequence and structure conditioned on the binding ligand molecule and protein scaffold context. Such a new task brings a series of challenges and it is non-trivial to propose an effective method. For example, most of the previous methods only generate the protein backbone atoms while neglecting the sidechain atoms, which play important roles in protein pocket-ligand interactions. We proposed a novel two-step coarse-to-fine generation procedure that well consider the sidechain atoms. Moreover, the binding ligand molecules are flexible, which is omitted in previous works. FAIR learns to update the molecular coordinates along with the refinement processes to model the flexibility of the ligands.\nOur contributions are summed in lines 63-72 including new tasks, novel methods, and competitive performance. We will state our contributions more clearly in the final version.\n\n**Comment 2:** For enzymes, we really would ideally like to keep the binding site conserved instead of re-designing it, based on my understanding of the field. In theory, it is of course nice if you can design a new binding pocket to discover new kinds of chemistry. However, as far as I am aware, basic scientists do not currently understand enough about catalytic mechanisms to be able to design a new one from scratch.\nThus, designing a new pocket and claiming that it catalyzes the same reaction via the same/different mechanism seems very hard to tackle from a basic science perspective, let alone computationally. Reasons: lack of data and proper annotation of enzymatic mechanisms.\n\n**Response 2:** We agree that the catalytic mechanism of enzymes is complex and is currently not fully understood in biology. The activity of enzymes may also depend on the overall flexibility and electrostatic environment of the protein, making pocket re-designing a challenging task. \nHowever, there are still some successful cases achieving various modification requirements by re-designing the residues in the protein pocket region.\nFor example, Ulrike Scheib et al., conducted pocket transplantation studies based on the homologous polyamine-binding proteins PotF and PotD. Despite having only 35% sequence identity between PotF and PotD, they demonstrated that by transplanting the pocket of PotD into the pocket region of PotF, it is possible to achieve specific substitutions for small molecule binding specificity [r1].\nMoreover, Nicholas F. Polizzi et al., successfully designed six de novo proteins to bind the drug apixaban; two bound with submicromolar affinity with the proposed van der Mer structural units [r2].\n\nIn our work, we propose an end-to-end generative framework FAIR for protein pocket design. We agree that more high-quality data and annotations from domain experts may further improve the performance of FAIR. Depending on the requirements, FAIR can be flexibly adapted to downstream applications. For the cases where we would ideally like to keep the catalysis binding site conserved, we can retain the residues directly related to catalysis and use our method FAIR to design other residues related to binding.\nWe will include these discussions in our final version.\n\n[r1]Scheib U, Shanmugaratnam S, Farías-Rico J A, et al. Change in protein-ligand specificity through binding pocket grafting[J]. Journal of Structural Biology, 2014, 185(2): 186-192.  \n[r2]Polizzi N F, DeGrado W F. A defined structural unit enables de novo design of small-molecule–binding proteins. Science, 2020, 369(6508): 1227-1233.\n\n\n**Comment 3:** I felt it was overall challenging to design evaluation metrics and setups for this task because it is unclear whether recovering the RMSD and/or AA identities of the original ligand binding site is the best outcome for these models.\n\n**Response 3:** We agree that it is challenging to comprehensively evaluate the designed binding pocket. In our paper, we use Amino Acid Recovery (AAR), Root Mean Square Deviation (RMSD), and Vina score to evaluate the designed pockets following previous works on protein/antibody design [3,7,11] and structure-based drug design [37,40,49].  These three metrics are currently established and widely used in the field. In the future, we will explore more evaluation metrics. As mentioned in Appendix D, it is also a good idea to carry out wet-lab experiments to validate the effectiveness of the designed protein pockets in the future.\n\n**Comment 4:** What is the Vina docking score of the training set?\n\n**Response 4:** The Vina docking score of the train/validation/test set are -7.035±2.11/-7.063±1.97/-7.016±2.24 for CrossDocked and -8.216 ±2.09/-8.267±1.97/-8.225±2.02 for Binding MOAD. Therefore, the Vina score of the designed pockets by FAIR is comparable to the data set.\n\n**Comment 5:** Why are some residues used more and some lesser by the model? Is this observation related to binding, eg. hydrophobicity properties of residues?\n\n**Response 5:** The observation may be related to the hydrophobicity properties of residues. While other factors such as the train/test set data distribution and the randomness of residue sampling may influence the generated residues. We will conduct more systematic validations and discussions in the future.\n\n"}, {"review_id": "Z7j57CvjPl", "reviewer": "Reviewer_Luqz", "summary": "The authors study the 3D protein-ligand interaction problem. They introduce a novel method for designing protein biding pockets conditioned on the ligand structure, termed FAIR. Unlike existing methods, FAIR co-designs sequence and structure of the pocket by iteratively modeling both backbone atoms and side chain atoms. The method also models refines ligands coordinates accounting for its flexibility. ", "questions": "Some minor concerns are listed below. \n\nAtom-level decoder:\n\n1.\tThe way section 3.2.1 is written, it seems that the encoder is using all atoms in the protein-ligand complex. The number of atoms in the protein-ligand complex can be very large resulting in a very large KNN graph. This could cause some memory restrictions. Why not focusing only on the pocket residues? \n2.\tHow is the variable number of atoms and atom types present in the ligand (as well as in different side chains) in different training samples handled ? \n3.\tUsing this notation, it’s not clear how the same atoms in different residues have different embeddings. Is this handled in the first MLP layer ?\n\nResidue-level encoder:\n\nIt’s not clear what coarsening procedure is used for different types of ligands.  If it’s treated as a “special” residue, then it’s not clear how a feature vector describing its biochemical properties is formed. How’s the 1-hot encoding containing ligand’s identity and distinguishing it from other ligand molecules formed ? Is there a vocabulary of all the ligands in the training set ? The authors should clarify this. \n", "limitations": "The authors claim that their method can do de novo design of the binding pocket. I wish this was shown in the paper. Most examples that are illustrated in the paper are just dealing with the recovery of the existing binding pockets. \nOne suggestion to illustrate de novo design is by grafting a biding pocket in some existiing protein scaffold in a given location (that was not previously a ligand binding location).  ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The method combines the ideas from many previous ML works dealing with sequence-structure co-design using graph representation. The novelty of this approach is in the iterative refinement that first ensures the stability of the backbone atoms before proceeding to the modeling of the side chain atoms. Thus, the method models the ligand flexibility and the effect of side chains on the residue types. The comparison study includes all the relevant baselines and other methods and the ablation study is extremely useful as it justifies the importance of each block in their architecture.  ", "weaknesses": "Some clarifications in the sections 3.2.1 and 3.2.2 are needed (see questions). Also, the authors should be very careful when using the word \"de novo\". They need to show how their method can perform de novo design or explicitly say what metrics they use to show that their designed sequence are de novo. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their appreciation and detailed suggestions!  If you have any additional questions. Let us know about any other comments we can address. We would be very grateful if you considered increasing the score.\n\n**Comment 1:** The way section 3.2.1 is written, it seems that the encoder is using all atoms in the protein-ligand complex. The number of atoms in the protein-ligand complex can be very large resulting in a very large KNN graph. This could cause some memory restrictions. Why not focusing only on the pocket residues?\n\n**Response 1:** Motivated by the intrinsic hierarchical structure of the protein, we leverage a hierarchical encoder based on 3D graph transformers to encode the hierarchical context information of protein-ligand complexes for pocket sequence-structure co-design. The structure and interactions of protein-ligand atoms play important roles in pocket design [8, 48] so we cannot only focus on residues. To save memory requirements and keep high overall performance, we only consider the protein pocket atoms instead of the whole protein. We will introduce our method more clearly in the final version.\n\n**Comment 2:** How is the variable number of atoms and atom types present in the ligand (as well as in different side chains) in different training samples handled?\n\n**Response 2:** As shown in our code https://anonymous.4open.science/r/FAIR-9691, the computations of the ligand and protein pocket based on torch_geometric are agnostic to the number of atoms and atom types. This is the same for different side chains. With our designed batch operations, protein pocket-ligand complexes with different sizes can be processed in parallel.  \n\n**Comment 3:** Using this notation, it’s not clear how the same atoms in different residues have different embeddings. Is this handled in the first MLP layer ?\n\n**Response 3:** In the atom-level encoder of FAIR, the atomic attributes are mapped to node embeddings with MLPs. If two atoms in different residues have the same atom and residue types, they will have the same initial embedding. The atom embeddings will be further updated based on neighboring atoms and 3D structures with the atom-level encoder. We will describe our model design more clearly in the final version.\n\n\n**Comment 4:** It’s not clear what coarsening procedure is used for different types of ligands. If it’s treated as a “special” residue, then it’s not clear how a feature vector describing its biochemical properties is formed. How’s the 1-hot encoding containing ligand’s identity and distinguishing it from other ligand molecules formed ? Is there a vocabulary of all the ligands in the training set ? The authors should clarify this.\n\n**Response 4:**  The residue-level encoder only keeps the alpha carbon atoms of residues.\nTo supplement binding ligand information, a coarsened ligand node at the \nligand’s center of mass is also considered for the residue-level encoder. The embedding of the coarsened ligand node is initialized by sum pooling the ligand atom embeddings. Therefore, we do not need 1-hot encoding containing ligand’s identity or a vocabulary of all the ligands. In lines 153-154, we stated that the coarsened ligand node is \nappended at the end of the residue sequence as a special residue. We treat the coarsened ligand similar to other residues and construct a Kr nearest neighbor graph. We will make our description clearer in the final version.\n\n**Comment 5:** The authors claim that their method can do de novo design of the binding pocket. I wish this was shown in the paper. Most examples that are illustrated in the paper are just dealing with the recovery of the existing binding pockets. One suggestion to illustrate de novo design is by grafting a binding pocket in some existing protein scaffold in a given location (that was not previously a ligand binding location).\n\n**Response 5:** Thanks for the question and suggestion! In our paper, the protein pocket region is masked and FAIR can co-design pocket residue types and structures. By “de novo”, we mean that our method does not rely on existing reference pockets or templates and can generate the pocket from scratch [r1]. We use recovery rate for evaluation as it is a widely used metric established in the protein design field [r2-r4]. Figure 3 in the main text further shows some cases of pocket design, where the generated pocket has higher binding affinity than the references, demonstrating FAIR's ability for de novo pocket design. We agree that grafting a binding pocket in some existing protein scaffold to a given location is a good task for illustrating the ability of de novo design. \nHowever, the grafting task is beyond the scope of our work and requires more techniques such as pocket detection. Due to the limited time of the rebuttal period, we plan to explore the grafting task in the future. We will also clearly discuss this in the final paper to emphasize the current capabilities of the method vs. future extensions.\n\n[r1]Bennett N R et al., Improving de novo protein binder design with deep learning. Nature Communications, 2023.  \n[r2]Zhangyang Gao et al., PiFold: Toward effective and efficient protein inverse folding, ICLR 2023  \n[r3]Dauparas, Justas, et al., Robust deep learning–based protein sequence design using ProteinMPNN. Science 2022.  \n[r4]John Ingraham, et al., Generative models for graph-based protein design, NeurIPS, 2019  "}, {"review_id": "WKXFpJk2Ga", "reviewer": "Reviewer_4UXg", "summary": "The paper introduces FAIR, the pipeline for protein pocket sequences and 3D structures co-design. It's important in drug design applications, since most of the small molecule drugs (ligands) bind their targets (proteins) inside the pockets. Curranty existing methods have disadvantages (inefficient generation, inability to generate side chains, etc.), that FAIR overcomes. FAIR demonstrates promising results, as proven through thorough and comprehensive experiments.", "questions": "1. Can you please provide the information about FAIR speed (how much time does FAIR need to finish one protein co-design)? Is it possible to use it high-throughput (apply for thousands or millions of proteins)?", "limitations": "The paper provides sophisticated discussion of limitations and future works.", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Originality: The task of protein pocket design is not new and there are several deep learning methods that have already contributed to this field. However, the submission provides new insights and solve yet unsolved issues showing a better performance than previous methods. Related works are cited.\nQuality: The work is complete and technically sound. Authors support the claims and provide a baseline of FAIR performance. Authors compare FAIR with other methods and show the advantages of FAIR.  The paper provides a thorough analysis of strong and weak sides of FAIR. It includes ablations studies and description of how to further improve the approach. \nClarity: The text is written clearly. It contains all the necessary citations. The submission provides a comprehensive explanation and description of all methods and approaches including the technical details of experiments and equations.\nSignificance: It's very important to for drug discovery to be able to design protein pockets since most of the small molecule drugs bind in the pockets. FAIR provides additional contribution by solving the issue with side chain and flexibility design.", "weaknesses": "The paper lack of speed evaluation or discussion of the method (FAIR).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your appreciation and suggestions! We are really grateful for your feedback and acknowledgment of FAIR’s novel contributions and experiments. \n\n**Comment 1:** Can you please provide the information about FAIR speed (how much time does FAIR need to finish one protein co-design)? Is it possible to use it high-throughput (apply for thousands or millions of proteins)?\n\n\n**Response 1:** Thanks for the suggestion! In Figure 4(C), we show the comparisons of average generation time for 100 pockets with baseline methods. We can observe that FAIR is much faster than traditional methods and need less than 1 second to finish one protein pocket co-design on average. \nAs shown in our code https://anonymous.4open.science/r/FAIR-9691, the generation process of FAIR can be parallelized to apply for high-throughput pocket generation. We will add more discussions in our final version."}, {"review_id": "ZnVEoagMsD", "reviewer": "Reviewer_f8hF", "summary": "In this paper, the authors proposed a Full-Atom Iterative Refinement framework (FAIR) for protein pocket sequence and 3D structure co-design. Generally, FAIR has two refinement steps (backbone refinement and full-atom refinement) and follows a coarse-to-fine pipeline. The influence of side-chain atoms, the flexibility of binding ligands, and sequence-structure consistency are well considered and addressed. Extensive experiments on two datasets show the advantage of FAIR in generating high-quality protein pockets.", "questions": "1. Can other protein design methods or structure-based drug design methods be adapted to the studied pocket design problem?\n2. Why there is no hyperparameter weight to balance the two loss functions, Equation 9&10?\n3. Can FAIR be leveraged for pocket optimization tasks?\n", "limitations": "The limitations of FAIR is well discussed in Appendix D. ", "rating": 9, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow. Existing related works are well-discussed. \n2. Figure 1 and 2 clearly illustrate the protein pocket design problem and the proposed method FAIR.\n3. As far as I know, this is the first paper that studies protein pocket sequence-structure co-design with deep learning methods. The importance and background of the problem are well stated. The problem is well formulated in Sec. 3.1. \n4. The coarse-to-fine architecture as well as the full-shot iterative refinement schemes are reasonable and effective. The modeling of side-chain atoms, sequence-structure consistency, and the flexibility of binding ligands are well considered. \n5. Extensive experiments on CrossDocked and Binding MOAD dataset compared with 5 representative  baselines show the effectiveness of FAIR. The code of FAIR is also provided for reproduction. \n", "weaknesses": "1. In the experiments, the authors redesign all the residues that contain atoms within 3.5 Å of any binding ligand atoms. Can FAIR design larger regions of the protein pocket containing more residues?\n2. The authors may explore the influence of initialization strategies on FAIR.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your appreciation and suggestions! Following your suggestions, we added new experiments, clarifications of formulations, and analyses. This revision has considerably improved our initial submission thanks to your constructive comments. We would love to know what you think about our response and if there is anything else we can do to improve the paper. We would greatly appreciate your considering increasing the score. Many thanks!\n\n**Comment 1:** In the experiments, the authors redesign all the residues that contain atoms within 3.5 Å of any binding ligand atoms. Can FAIR design larger regions of the protein pocket containing more residues?\n\n\n**Response 1:** FAIR can design larger regions of protein pocket with more residues. In our experiments, we redesign all the residues that contain atoms within 3.5 Å of any binding ligand atoms in the default setting considering the distance ranges of protein-ligand interactions [r1]. There are an average of 8 residues for each protein pocket.  Here, we perform further experiments to design all the residues that contain atoms within 5.0 Å of any binding ligand atoms, leading to an average of around 22 residues for each pocket. It is more challenging to design pocket with more residues.\n\n\n| Model          | CrossDocked AAR(↑) | CrossDocked RMSD(↓) | CrossDocked Vina(↓) | Binding MOAD AAR(↑) | Binding MOAD RMSD(↓) | Binding MOAD Vina(↓) |\n|----------------|-------------------|--------------------|---------------------|---------------------|----------------------|---------------------|\n| FAIR (3.5 Å)          | 40.17±12.6%  | 1.42±0.07  | -7.022±1.75     | 43.75±15.2%     | 1.35±0.10        | -7.978±1.91     |\n| FAIR (5.0 Å)           | 35.68±11.7%       | 1.63±0.10          | -7.045±1.71         | 39.86±14.0%         | 1.52±0.09            | -7.889±1.84          |\n\nWe observe that FAIR is generally robust to the number of residues to design: the AAR, RMSD, and Vina in 5.0 Å is comparable with 3.5 Å. Therefore, FAIR can design larger regions of a protein pocket. We will conduct more analysis in the appendix of the final paper.\n\n[r1] Gilles Marcou and Didier Rognan. Optimizing fragment and scaffold docking by use of molecular interaction fingerprints. Journal of chemical information and modeling, 47(1):195–207, 2007\n\n**Comment 2:** The authors may explore the influence of initialization strategies on FAIR.\n\n**Response 2:** As shown in Appendix A.4, we initialize the residue coordinates with \nlinear interpolations and extrapolations based on the nearest residues with known structures in the protein. For comparison, we initialize the residue coordinates with their corresponding nearest residues. \n \n| Model          | CrossDocked AAR(↑) | CrossDocked RMSD(↓) | CrossDocked Vina(↓) | Binding MOAD AAR(↑) | Binding MOAD RMSD(↓) | Binding MOAD Vina(↓) |\n|----------------|-------------------|--------------------|---------------------|---------------------|----------------------|---------------------|\n| FAIR (linear interpolation)          | 40.17±12.6%  | 1.42±0.07  | -7.022±1.75     | 43.75±15.2%     | 1.35±0.10        | -7.978±1.91     |\n| FAIR (nearest residue)           | 34.26±12.3%       | 1.83±0.09          | -6.850±1.84         | 36.86±14.0%         | 1.79±0.12            | -7.743±1.70          |\n\nWe can observe that the structure initialization strategies indeed have an influence on the performance. FAIR with the linear interpolation initialization strategy has a better performance than the nearest residue. We will add more discussions of initialization strategies to our final version.\n\n**Comment 3:** Can other protein design methods or structure-based drug design methods be adapted to the studied pocket design problem?\n\n**Response 3:** As discussed in Sec.2 lines 97-101, structure-based drug design can be regarded as the dual problem of pocket design studied in our paper. However, they focus on generating 3D molecular graphs based on the fixed protein pocket structure and can hardly be adapted to our pocket sequence-structure co-design problem. We will organize the related works more clearly in the final version.\n\n**Comment 4:** Why there is no hyperparameter weight to balance the two loss functions, Equation 9&10?\n\n**Response 4:** In experiments, we observe that the two loss functions have roughly the same amplitude, and directly optimizing their sum can work well. Balancing the two loss functions through additional weights (modeled as a hyper-parameter during fine-tuning) may further improve FAIR’s performance. We will discuss the summation of loss functions more in the final version.\n\n**Comment 5:** Can FAIR be leveraged for pocket optimization tasks?\n\n**Response 5:** That is a great suggestion, thank you. Indeed, FAIR can be leveraged for pocket optimization tasks. FAIR is a general architecture and can be combined with popular optimization methods for pocket optimization. For example, we can finetune the generation process of FAIR with reinforcement learning algorithms e.g., PPO [r2] to optimize the properties of the designed pockets. As mentioned in Appendix.D, we will explore pocket optimization tasks in the future.\n\n[r2] Schulman J, Wolski F, Dhariwal P, et al. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.\n"}]

Author rebuttal:
**Global response to all reviewers:**

We thank the reviewers for their appreciation and valuable comments! Generally, the reviewers find our paper a novel approach for designing protein pockets that bind to ligand molecules. In the rebuttal, we have done additional experiments and added more discussions and clarifications. With the constructive suggestions from reviewers, we believe our paper will be improved after the rebuttal period!

In the response, we use [1], [2] ... to refer to the reference papers in the original paper and use [r1], [r2] ... to indicate the additional references in the rebuttal.

**Additional response to Reviewer z6bW:**

**Comment 6:** Can you also show Vina docking scores for all the models without re-docking/relaxation?

**Response 6:** Here we show the directly computed Vina scores for all the models without redocking and relaxation (Vina w/o redocking).  For a fair comparison, we exclude the baseline PocketOptimizer here because it uses re-docking and force field relaxation methods in its generation procedure. Generally, the Vina scores w/o redocking and relaxation are higher (lower binding affinity). Our method FAIR can achieve the lowest Vina score on different settings and datasets. It can be attributed to FAIR’s strong capability of pocket sequence-structure co-design. Meanwhile, the coordinates of ligand molecules are updated during the refinement process. Therefore, FAIR relies less on redocking and relaxation.

| Model          | CrossDocked Vina w/o redocking (↓) | CrossDocked Vina(↓) | Binding MOAD Vina w/o redocking(↓) | Binding MOAD Vina(↓) |
|----------------|-------------------|--------------------|---------------------|---------------------|
| DEPACT |-5.820±2.16   | -6.670±2.13|-6.486±2.23 |-7.526±2.05 |
| HSRN |-5.449±2.01    | -6.565±1.95| -5.870±2.04| -7.349±1.93|
| Diffusion |-5.364±1.88     | -6.725±1.83 |-6.358±2.19|-7.724±2.36 |
| MEAN |-5.672±1.79    | -6.891±1.86  | -6.320±1.88|-7.651±1.97|
| FAIR |**-6.365±1.67**| **-7.022±1.75**|**-7.253±1.72** |**-7.978±1.91**|

**Comment 7:** 
How do we know the binding pose of the ligand in advance in real-world design scenarios? Do we always need to start with an existing (crystal) structure of the protein-ligand complex?

**Response 7:** FAIR does not require the binding pose of the ligand in advance for pocket design because it is able to refine the ligand coordinates along with the pocket generation process. In practice, we can initialize the ligand poses with chemical tools (e.g., rdkit ) or take existing binding ligand structures as a reference if available [8,45].

Meta-review: The paper presents an atomic-level sequence and structure design method tailored to protein biding pockets conditioned on the ligand structure.  
* The reviewers agreed that the approach is novel, well-written, well-motivated, and well-placed in the context of the literature. 
* The proposed method is built from reasonable components found also in other applications of ML to protein design: a coarse-to-fine architecture modeling backbone and residue types first, and side-chains after; as well as the full-shot iterative refinement popular e.g., in antibody design. The work considers also the flexibility of binding ligands, which is a big plus.
* The paper also comes with extensive comparisons and ablations on CrossDocked and Binding MOAD that showcase its benefits over five representative baselines. 

I recommend this work for a spotlight presentation as I think that this paper is likely to influence the field of machine learning for protein design.
Decision: Accept (spotlight)