Protein Sequence and Structure Co-Design with Equivariant Translation
Authors: Chence Shi
Chuanrui Wang
Jiarui Lu
Bozitao Zhong
Jian Tang
Venue: ICLR 2023 poster (2023)
Keywords: protein design
sequence structure co-design
equivariant translation
geometric deep learning

Abstract: Proteins are macromolecules that perform essential functions in all living organisms. Designing novel proteins with specific structures and desired functions has been a long-standing challenge in the field of bioengineering. Existing approaches generate both protein sequence and structure using either autoregressive models or diffusion models, both of which suffer from high inference costs. In this paper, we propose a new approach capable of protein sequence and structure co-design, which iteratively translates both protein sequence and structure into the desired state from random initialization, based on context features given a priori. Our model consists of a trigonometry-aware encoder that reasons geometrical constraints and interactions from context features, and a roto-translation equivariant decoder that translates protein sequence and structure interdependently. Notably, all protein amino acids are updated in one shot in each translation step, which significantly accelerates the inference process. Experimental results across multiple tasks show that our model outperforms previous state-of-the-art baselines by a large margin, and is able to design proteins of high fidelity as regards both sequence and structure, with running time orders of magnitude less than sampling-based methods.

Published as a conference paper at ICLR 2023
PROTEIN SEQUENCE AND STRUCTURE CO-DESIGN WITH EQUIVARIANT TRANSLATION
Chence Sh, Chuanrui Wang Jiarui Lu Bozitao Zhong " Jian Tang
1.2,455
BioGeometry "Mila- Québec AI Institute *Université de Montréal HEE Montréal CCIAR AI Research Chair
chence shi@ (biogeom. com, umont real. ca)
(chuanrui. wang, jiarui 1u, bozitao. zhong) @umontreal.co ca jian. taw.tann@hc....
ABSTRACT
Proteins are macromolecules that perform essential functions in all living organisms. Designing novel proteins with specific structures and desired functions has been Long-standing challenge in the field of bioengineering. Existing approaches generate both protein sequence and structure using either autoregressive models or diffusion models, both of which suffer from high inference costs. In this paper, we propose new approach capable of protein sequence and structure co-design, which iteratively translates both protein sequence and structure into the desired state from random initialization, based context features given priori. Our model consists of a trigonometry-awart encoder that reasons geometrical constraints and interactions from context features, and crottranslation equivariant decoder that translates protein sequence and structure interdependently. Notably, all protein amino acids updated in shot in each translation step, which significantly accelerates the inference process. Experimental results across multiple tasks show that our model outperforms previous state-of-the-art baselines by large margin, and is able to design proteins of high fidelity as regards both sequence and structure, with running time orders of magnitude less than sampling-bass based methods.
INTRODUCTION
Proteins are macromolecules that mediate the fundamental processes of all living organisms. For decades, people are seeking design novel proteins with desired properties (Huang et al., 2016), problem known as de novo protein design. Nevertheless, the problem is very challenging due to the tremendous search space of both sequence and structure, and the most well-established approaches still rely hand-crafted energy functions and heuristic sampling algorithms (Leaver-Fay et al., 2013; Alford et al., 2017), which are prone to arriving at suboptimal solutions and are computationally intensive and time-consuming.
Recently, machine learning approaches have demonstrated impressive performance on different aspects of protein design, and significant progress has been made (Gao al., 2020). Most approaches use deep generative models to design protein sequences based on corresponding structures (Ingraham al., 2019; Jing et al., 2021; Hsu al., 2022). Despite their great potential for protein design, the structures of proteins to be engineered are often unknown (Fischman & Ofran, 2018), which hinders the application of these methods. Therefore, efforts have been made to develop models that co-design the sequence and structure of proteins (Anishchenko et al., 2021; Wang et al., 2021). As pioneer- ing work, Jin et (2021) propose an autoregressive model that co-designs the Complementarity Determining Regions (CDRs) sequence and structure of antibodies based on iterative refinement of protein structures, which spurs a lot follow-up works (Luo et al., 2022: Kong et al., 2022). Nevertheless, these approaches are tailored for antibodies and their effectiveness remains unclear on proteins with arbitrary domain topologies (Anand & Achim, 2022). In addition, they often suffer from high inference costs due autoregressive sampling annealed diffusion sampling (Song & Ermon, 2019; Luo et al., 2022). Very recently, Anand & Achim (2022) propose another diffusion based generative model (Ho et al., 2020) for general protein sequence-structuu co-design, where they

---

Published as a conference paper at ICLR 2023
Antibeal tramssank
GRGRMFSHER... SS annosation
Conditinall Saucure
Sepazoo Cla Dosggg
Proof Phooftaplooo Seqartec D-sign
Antiac
CDR structures
Stractree
DAST ELEMK- NYSSBIT OR SPS
Antign-
CEVIOTLDF
GAST AYSOISLEK.
TSYOPESY CYPTO-TDY
Racibone
Sequemoes
Misked CDR koop
CDR sequeness
Contact map
Sequeriocs
(a)
(b)
(c)
Figure 1: Illustration of three protein design tasks with different context features. (a) Antigen-specific CDR co-design given structure and sequence of antibody framework and the binding antigen. (b) Protein sequence-Stracture co-design conditioned on secondary structure (SS) annotation and binary contact features. (c) Fixed backbone sequence design conditioned on given backbone structures.
adopt three diffusion models to generate structures, sequences, and rotamers of proteins in sequential order. Although applicable to proteins of all topologies. such a sequential generation strategy fails to cross-condition on sequence and structure, which might lead to inconsistent proteins. Besides, the inference process is also expensive due to the use of three separate diffusion processes.
To address the aforementioned issues, in this paper, we propose a new method capable of protein sequence-structure equivariant co-design called PROTSEED. Specifically, we formulate the co-design task translation problem in the joint sequence-structus space based context features. Here the context features represent prior knowledge encoding constraints that biologists want to impose on the protein be designed (Dou al., 2018; Shen et al., 2018). As an illustration, we present three protein design tasks with different given context features in Figure 1. Our PROTSEED consists of trgonometry-awwan aware encoder that infers geometrical constraints and prior knowledge for protein design from context features, and novel roto-translation equivariant decoder that iteratively translates proteins into desired states an end-to-end and equivariant manner. The equivariance property with respect to protein structures during the whole process is guaranteed by predicting structure updates in local frames based on invariant representations, and then transforming them into global frames using change of basis operation. It is worth mentioning that PROTSEED updates sequence and structure of all residues in an one-shot manner, leading to a much more efficient inference process. In contrast to previous method that first generates structure and then generates sequence and rotamers, allow the model to cross-condition on sequence and structure, and encourage the maximal information flow among context features, sequences, and structures, which ensure the fidelity of generated proteins.
We conduct extensive experiments on the Structural Antibody Database (SAbDab) (Dunbar et al., 2014) as well as two protein design benchmark data sets curated from CATH (Orengo al., 1997), and compare PROTSEED against previous state-of-thed the-art methods on multiple tasks, ranging from antigen-specific antibody CDR design to context-conditions protein design. Numerical results show that our method significantly outperforms previous baselines and can generate high fidelity proteins in terms both sequence and structure, while running orders of magnitude faster than sampling- based methods. As proof of concept, we also show by cases that PROTSEED is able to perform de novo protein design with new folds.
protein design and fixed backbone
RELATED WORK
Protein Design. The most well-established approaches on protein design mainly rely on handcrafted energy functions iteratively search low-energy protein sequences and conformations with heuristic sampling algorithms (Leaver-Fay et al., 2013; Alford et al., 2017; Tischer et al., 2020). Nevertheless, these conventional methods are computationally intensive, and are prone to arriving at local optimum due to the complicated energy landscape. Recent advances in deep generative models open the door to data-driven approaches, and a variety of models have been proposed to generate protein sequences (Rives et al., 2021; Shin et al., 2021; Ferruz et al., 2022) or backbone structures (Anand & Huang, 2018; Eguchi et al., 2022: Trippe et al., 2022). To have fine- fine-grain control over designed developed to predict sequences that can fold into given backbone structures (In-
proteins, methods graham et al., 2019; Jing et al., 2021; Anand et al., 2022; Dauparas et al., 2022), a.k.a. fixed backbone design, which achieve promising results but require the desired protein structure be known a priori.
2

---

Published as a conference paper at ICLR 2023
Recently, class of approaches that generate both protein sequence and structure by network halluci- nation have emerged (Anishchenko et al., 2021; Wang et al., 2021), which carry out thousands of gradient descent steps in sequence space to optimize loss functions calculated by pre-trained protein structure prediction models (Yang et al., 2020; Jumper et al., 2021). However, the quality of designed proteins usually relies on the accuracy of structure prediction models, and is sensitive to different random startings. On the other hand, attempts have been made to co-design CDR sequence and struc- ture of antibodies using either autoregressive models (Saka et al., 2021; Jin al., 2021) or diffusion models (Luo et al., 2022). Nevertheless, they are restricted to proteins with specific domain topologies and often suffer from the time-consume conssminn Monte Carlo sampling process. Going beyond antibodies, Anand & Achim (2022) adopt three separate diffusion models to generate sequences, structures, and rotamers of proteins sequentially. Such a method inefficient and fails to cross-condition on both protein sequence and structure. Our model also seeks to co-design protein sequence and structure, but is able to cross-condition on sequence and structure, while being much more efficient.
3D Structure Prediction. Our work is also related to approaches that perform 3D structure prediction by iteratively translating structures in three dimensional space equivariantly (Shi et al., 2021; Luo & Hu, 2021; Hoogeboom et al., 2022; Xu et al., 2022; Zhu et al., 2022). However, these methods represent structures as either molecular graphs or point clouds, and are not applicable to protein structures. On the other hand, protein folding models (Jumper et al., 2021; Baek et al., 2021) that perform protein structure prediction require complete protein sequences as well as their Multiple Sequence Alignments (MSAs) as input, and cannot co-design protein sequence and structure directly.
METHOD
3.1 PRELIMINARIES
Notations. Proteins are macromolecules that can be viewed as chains of amino acids (residues) connected by peptide bonds. In this paper, an amino acid can represented by type 8; E (1, (1....20). C. coordinates æ; E T&3 and the frame orientation O E SO(3), where E (1....N N} and is the number of residues in protein. The æ; and O form a canonical orientation frame with respect to the N. C and C3 atoms, from which the backbone atom positions can be derived. We denote the one-hot encoding of the residue type as s; = onchot(s;) (x;). In the protein sequence and structure co-design task, researchers often provide context features input encourage designed proteins to have desired structural properties. These context features can either be single (residue) features m, IR (e.g., amino acid secondary structure annotations) pair features Zij R (e.g.. binary contact features between residues). With the above notations, protein with N residues can be compactly denoted as {(s), I., 1. The context features known priori can be denoted as (m.) R Nxcm and {zig) ) E NxNe.
Problem Formulation. Given set of context features {m,} E Nx and {zt}) E NXNXG.
the task of protein sequence and structure co-design is the joint generation of residue types and 3D conformations of protein with N residues, i.e., the conditional generation of {(s,, wi, O, ) i based on [m:} and [z:z)- Note that context features vary from setting to setting. For example, in antibody CDR design (Jin et al., 2021: Luo et al., 2022), they are derived from antibody framework and binding antigen structures with CDR region masked, while in full protein design (Anand & Achim, 2022), they can be secondary structure annotations and residue-residue contact features.
Overview. In this paper, we formulate protein sequence and structure co-design as an equivariant translation problem in the joint sequence-strue structure space. Specifically, we develop a trigonometry- aware context encoder to first reason geometrical constraints encoded in context features. Based on the updated context features, protein sequence and structure are jointly generated in an iterative manner by novel roto-translation equivariant decoder, starting from randomly initialized structures and residue types (illustrated in Figure 2). To model interactions between sequence and structure during decoding, we allow information to flow between context features, structures, and residue types in each translation step. In this way, the generated sequence and structure are ensured to be consistent. The pseudo-code of the whole framework can be found in Algorithm 1. The rest of this section is organized as follows: Section 3.2 introduces the trigonometry-awee awwre context encoder. Section 3.3 elaborates the iterative joint sequence-structure decoder and training objectives.
3

---

Published as a conference paper at ICLR 2023
Layers
pt
p++1
Fee dus lypa Dsaiiutien
Residue Type Desttuttor
+0..5
emaatonn
One Shot Sequence Transiation
Oliertation
0f+1
SaqiPA
Cp Postton
C2 Poattoo
Equivariant
Contort Features
Context Features
.+11 313
Encxding Sequence Structure Interple
m
Figure 2: Illustration of the joint sequence-structiue translation process. In each translation layer, the network first captures the interactions of the current protein state and context features via SeqIPA, and then translates the protein sequence and structure into the next state equivariantly.
3.2 TRIGONOMETRY-AWAR CONTEXT ENCODER
Given single features and pair features as input, the goal of the context encoder is to capture the interactions between different context features and infer encoded constraints for the following protein grquence-structurn co-design. We first embed single features and pair features into dimensional space using Multiple Layer Perceptrons (MLPs). We then adopt stack of trigonometry-aware update layer propagate information between single features and pair features.
We represent the updated single features and pair features at (th layer as {m,} and {0 respectively, At each layer, the single features are first updated using variant of multi-head self-attention (denoted as MHA) (Vaswani et al., 2017), with pair features serving as additional input to bias the attention map. Similar to Jumper al. (2021), the pair feature z/y are then updated by the linear projection of outer product of single features m;
I+1 and m) 1+1,
m ++1 MHA (z())),
1+0.5 = Z1, + Linear(m,.* 1+1 s8 sm/ m2+1) ij
(2)
where & is the outer product operation. Notably, enable the information to flow between single features and pair features to better model interactions between context features.
Since pair features (e.g., contact map and distance map) usually related with Euclidean distances and dihedral angles between residues, inspired by AlphaFold 2 (Jumper et al., 2021), we adopt two trigonometry-awinn operations (Eq. 4 and Eq. 5) in each layer to maintain geometric consistency and encourage pair features to satisfy the triangle inequality. Formally, we have:
ăij, bij = o (z1) (-0.5 @ Linear(z!tor (+0.5),
Gijs kig, Vij. bij = Linear(z,) (o00ar((/007)) (40.75)
(3)
1-0.7 = = 140.5 + a O Linear
[ dik b + axs O bRy),
(4)
21j =
141 140.75 + o (2
140.75 c Σ Vasspvu + as5b)u)))
(5)
where o(.) = signoid(Linear(() and 0ijk = softmaxk ( q (ka + kuy) + bys- + bye) is the attention score of a novel trigonometry- attention, the pair feature z; is updated with neighboring features z and Zkj. by enumerating all possible & that form triangle Aijk in terms of residues. Different from Jumper et al. (2021), we tie the attention score Oiikk within each triangle Aijk to reduce the computational burden, while keeping the whole network sensitive triangular interactions among residues. After L rounds of feature propagation, the updated single features {m/ and pair features {2(}) serve as inputs to the decoder for joint protein sequence-
attention. Intuitively, in our trigonometry-ave
sequence-structure structure design.
4

---

Published as a conference paper at ICLR 2023
3.3 JOINT SEQUENCE-STRUCTURE DECODER
In this section, we describe the proposed joint sequence-structure decoder, with the goal of iteratively translating protein sequence and structure into desired states from scratch based on context features. Simply parameterizing the decoder using two neural networks that generate sequence and structure separately is problematic, as we need to ensure the consistency between generated sequence and structure. i.e., the sequence folds to the structure. Mcanwhile, we require our decoder to be roto- translation equivariant (Köhler et al., 2020; Jing et al., 2021) with respect to all protein structures during the decoding. To this end, we develop novel iito-translation equivariant network composed of T consecutive translation layers with weight tying (Dehghani et al., 2018; Bai et al., 2019). In each layer, we update context features, residue structures, and residue types interdependently, by allowing information to propagate among them. It is worth mentioning that residue types and residue structures of all amino acids updated in one shot in each translation step, distinct from previous work that generates them autoregressivell, which significantly accelcrates the decoding procedure.
We represent the updated residue types and residue structures at ht layer as P {(s) x O)) Abusing the notation a little bit, we denote updated context features at 8th layer as {m]) and { Specifically, {m?} and {2() ) are the invariant output of context encoder introduced in Section 3.2. For each amino acid, we initialize the residue type as the uniform distribution over 20 amino acid types, i.e., ge 20 1. C, coordinates as origin in global frame, i.c., x = (0, 0, 0), and frame orientation as identity rotation, i.e., O Iz. We elaborate each translation layer next.
Encoding Sequence-Structure Interplay. In each translation layer, we start by embedding current residue types into dimensional space with feedforward network MLP, 12200 + R s
MLP.(s.). We then adopt variant of Invariant Point Attention (IPA) (Jumper et al., 2021) called SeqIPA to capture the interplay of residue types, residue structures and context features, integrating them all altogether into updated context features. Such a practice is often favored in literature (Anand & Achim, 2022; Luo et al., 2022: Tubiana et al., 2022) as it is aware of the orientation of each residue frame while being translaannlttoon invariant to input and output features. Distinct from vanilla IPA. our SeqIPA takes residue types as the additional input to bias the attention map and steer the representation of the whole protein generated so far:
m m.+1 ii SoqIPA (fmm). {zly), {s} (s/t0.53, (af), (O
Note that SeqIPA is orientation-aware with respect to residue structures and roto-translation invariant to all other representations. We refer readers to Appendix B.2 for more details about the SeqIPA.
Equivariant Structure Translation. Given updated context features, the protein structure is trans- lated towards the next state by updating } and {O:) To update C, positions {z/}, we first predict the change of coordinates (denoted as { within each local residue frame specified by {O:). We then perform change of basis using )O:) to transform (œt) from local frame into global frame (Kofinas et al., 2021; Hsu et al., 2022) to derive equivariant deviation of C. positions (Eq. 7). Intuitively, the deviation of C. positions rotate accordingly when residue frames rotate, which guarantees the equivariance of the C. translation step.
The update for orientation frame O is computed by predicting a unit quaternion vector (Jia, 2008) with a feedforward network, which is then converted to a rotation matrix Ó and ceft-mulippied by O to rotate the residue frame (Eq. 8). We adopt the unit quaternion here because it is more concise representation of a rotation in 3D than a rotation matrix. Since the predicted unit quaternion is an invariant vector, the predicted rotation matrix O is also invariant. Therefore, the translation step of the residue frame is equivariant due the multiplication of rotation matrices. We summarize the whole equivariant structure translation step as follows:
: MLPx (m ++1 .mm?),
x; ++1 =
x 0
Ô convert ( (M...mm." +11 m))),
0 = 0.0%
(8)
where convert is a function that converts a quaternion to a rotation matrix.
One Shot Sequence Translation. The residue type of all amino acids are updated in one shot based on updated context features and current residue types in each translation step. Specifically, we use a feedforward network MLP, predict residue type distributions over 20 amino acid types for the next iteration:
8t+1 softmax (A MLPs(m t+1 ; .m. s s,+0.5)),
(9)
5

---

Published as a conference paper at ICLR 2023
where À is ayper-parameter controlling the temperature of the distribution. See Appendix B.3 for discussions on
and the equivariance property of sequence and structure translation.
Summarizing the above, at (t + 1)th layer, the decoder takes Pt {(s) æ O the imput and computes pi+1 {( _xtt1 " as the output. Based pt, we can efficiently reconstruct full backbone atom positions according to their averaged relative positions with respect to C. recorded in literature (Engh & Huber, 2012; Jumper et al., 2021). It is worth mentioning that distinct from previous works that can only generate backbone atom positions (Jin et al., 2021; Kong al., 2022; Luo al., 2022), our model is capable of full atom position generation by attaching the corresponding sidechain to each residue. In specific, we can enable the decoder to generate four additional torsion angles (X1. X2+ X3: Xx) (McPartlon Xu, 2022) that specify the geometry of sidechain atoms and reconstruct sidechain atom positions together with backbone atom positions.
Training Objective. We denote the reconstructed full backbone atom positions 8th layer as {æty) where j (1.2.3) is the index of backbone atoms (NV, Ca. C.). We use the superscript true to denote the ground truth value for simplicity. The whole network can be jointly optimized, by defining an invariant cross-entropy loss lee over type distributions and an invariant frame align loss (Jumper et al., 2021) over full backbone atom positions at each translation layer:
T
N
Lype 1
Σ
Reo
s
(10)
N
[=1
1=1
T
N
pas N 3N
( - true true
(art)) )l2,
(11)
t=1 k=
where (x(,) (0)))))(00)))) (0 x transforms the backbone coordinates from global coordinate system into local frame of 2.th residue frame at step (specified by æt and O and so does p The loss defined in Eq. 11 essentially calculates the discrepancy of all backbone positions between the prediction and the ground truth, by aligning each residue frume one by one. As all backbone coordinates are transformed into local frames, the loss will stay invariant when two protein structures differ by an arbitrary rotation and an arbitrary translation. Combining Eq. 10 and Eq. 11, the final training objective is L = Ltype Epos-
4 EXPERIMENTS
Following previous works (Jin et al., 2021; Jing et al., 2021; Anand & Achim, 2022), we conduct extensive experiments and evaluate the proposed PROTSEED on the following three tasks: Antibody CDR Co-Design (Section 4.1), Protein Sequence-Structure Co-Design (Section 4.2), and Fixed Backbone Sequence Design (Section 4.3). We also show cases where PROTSEED successfully conducts de novo protein sequence design with new folds in Section 4.4. We describe all experimental setups and results in task- specifci sections.
4.1 ANTIBODY CDR CO-DESIGN
Setup. The first task is to design CDR sequence and structure of antibodies, where the context features are amino acid types as well as inter-residue distances derived from antibody-antigen complexes with CDRs removed. The initial protein structure pD is set as the complex structure except for CDRs, which are randomly initialized. We retrieve untibody-antigen complexes in Apr. 2022 from Structural Antibody Database (SAbDab) (Dunbar et al., 2014), and remove incomplete or redundant complexes, resulting in a subset containing 2,900 complex structures. Following Jin et al. (2021), we focus on the design of heavy chain CDRs and curate three data splits for each type of CDRs (denoted as H1, H2, H3) by clustering corresponding CDR sequences via MMseqs2 (Steinegger & Söding, 2017) with 40% sequence identity. In total, there are 641, 963, and 1646 clusters for CDR H1, H2, and H3. The clusters are then divided into training, validation, and test set with ratio of 8:1:1.
We compare PROTSEED with the following three baselines. RosettaAntiioodyessgg (RAbD) (Adolf- Bryfogle et al., 2018) is physics- based antibody design software. GNN is an autoregressive model that co-designs sequence and structure similar to Jin et al. (2021). We note that we can not directly compare PROTSFED with Jin al. (2021) the setting is different. Diffusion (Luo et al., 2022)

---

Published as a conference paper at ICLR 2023
is diffusion- based method that achieves tfte-of-theeart performance antibody design recently. Following previous works (Jin et al., 2021; Luo et al., 2022), we use three metrics to evaluate the quality of designed CDRs: (1) Perplexity (PPL) measures the inverse likelihhod of native sequences in the predicted sequence distribution, and a lower PPL. stands for the higher likelihood. Note that PPI. should be calculated for each sequence first and then averaged over the test set. For methods that do not define joint distributions over sequences strictly (e.g., DIFFUSION), we use the final sequence distribution to calculate the approximated PPL. (2) RMSD the Root of Mean Squared Deviation of Ca between generated CDRs and ground- truth CDRs with antibody frameworks aligned (Ruffolo et al.. 2022). A lower RMSD stands for smaller discrepancy compared to native CDRs, which indicates a better CDR structure to bind the antigen. (3) Amino acid recovery rate (AAR) is the sequence identity between generated and ground-truth CDRs.
Table 1: PPL, RMSD and AAR of different approaches on the antibody CDR co-design task. (1):
the higher the better. (4): the lower the better.
(F)
RMSD (á. L>
AAR (96, 7) 112
CDR
III
112
113
RABD
3.06 (,33)
2,95 (223)
5.58(.32)
27.36 (1.89)
47.11 (1.74)
22.91 (2.18)
GNN
(.45)
7.64 (153)
13.91 (.39)
2.73 (.06)
(.13)
4.06 (.14)
61.49 (.27)
52.40 (.08)
29,83 (.32)
DIFFUSION
(.09).09)
6.28 (.10)
12:27 (.11)
1.94 (.12)
(.37)
3.95 (.01)
66.38 (:95)
55.61 (.03)
31.22 (.79)
PROTSEED
4.43 (.40)
(.17)
10.88 (388)
1.24 (.03)
(.04)
(.03)
70.22 (.98)
63.53 (.85) 39.27 (.77)
Results. We notice that scripts used to calcuate the above metrics are inconsistent in previous works. For fair comparison, we implement all the baselines and run each model three times with different random seeds. Following previous works (Jin et al., 2021; Luo et al., 2022), the length of the CDR is set be identical to the length of the ground-truth CDR for simplicity, and we sample 100 candidates with the lowest perplexity for each CDR for machine learning-basee based methods. We report the mean and standard deviation of the above metrics on the test set Table 1. Numerical results indicate that PROTSEED consistently outperforms previous state-of-the weff-hhear art baselines by a clear margin on all three metrics for each type of CDRs, which confirms PROT ability to co-design antibodies conditioned on existing binding structures. In particular, as an energy-based method that performs graft antibody design, the performance of RABD is inferior to data-driven approaches. DIFFUSION outperforms GNN as it is equivariant and models the orientation of residues similar to our model, but its performance still falls short of ours. It is worth mentioning that the performance of all models on CDR H3 is worse than that the other two CDR types, as CDR H3 is the most diverse region in an antibody that is critical to antigen binding. We present generated samples on CDR-H3 sequence-stracture co-design and their sidechain interactions with binding antigens in Figure 3(b).
Sampi
Soepple
8.
PAIBD 1.25 SOCCINAL
PIVED EYSDALOR
FRVED SEGOCHOF
KGSPAMOY
(a)
(b)
Figure 3: (a) Average runtime of three approaches on proteins of different sizes. Our PROTSEED runs orders of magnitude faster than the gradient-based method AFDESIGN and the diffusion-basee method DIFFUSION. (b) Examples of CDR- H3 sequence and structure so-designed by our method. The PDB ID of the reference complex is 6FLA, and the antigen is Dengue Virus. The sidechain interactions between the antigen (red) and the antibody (green) are highlighted.
4.2 PROTEIN SEQUENCE AND STRUCTURE CO-DESIGN
Setup. This task evaluates models' capability to design protein sequence and structure conditioned on context features known a priori. The task is brought into attention by Anand & Achim (2022) recently, but there is no rigorous benchmark in the field of machine learning to the best of our knowledge. To this end, we collect 3,877 protein structures from the non-redundant set of CATH S40 (Orengo et al., 1997), calculate amino acid secondary structure by DSSP (Kabsch & Sander, 1983), and split the data set into training, validation, and test set at the topology level with 90/5/5 ratio. We take secondary
7

---

Published as a conference paper at ICLR 2023
structure annotations as single features and binary contact matrices as pair features." ! This setting represents the situation where biologists already know the topology of desired proteins (Dou et al., 2018; Shen et al., 2018), and want the model to design novel proteins without specifying structures.
For this task, we compare PROTSEED with the following two baselines that we believe best describe the current landscape of this task. AFDesign' is hallucination--oe based (Wang al., 2021) method that generates protein sequence and structure by iteratively performing gradient descent in sequence space guided by AlphaFold2. We adopt AFDESIGN as one of our baselines because Alphafold2 is the state-of-the-art protein structure prediction model, and it outperforms other hallucination-baat based methods. Diffusion (Anand & Achim, 2022) another diffusion- based model that generates protein structure, sequence, and rotamers sequentially. and it is not restricted to antibodies. We use the PPL, RMSD, and AAR metrics introduced in Section to evaluate the fidelity of designed proteins.
Results. Since AFDESIGN is incapble of handling discrete contact matrices, we set the loss function that guides the gradient descent as the distogram loss computed by a pre-trained AlphaFold2 (Jumper et al., 2021). For DIFFUSION, we imple- ment by ourselves its source code is not released yet. We run each model for three times with different random seeds and report the mean and standard deviation of the above metrics on the test set. As shown in Table 2. PROTSEED outperforms all the baselines on all three metries significantly, which demonstrates its capability to generate proteins of high fidelity as regards both sequence and structure, and it is not limited to specific domain topologies. The performance of AFDESIGN falls short of other methods as it relies on gradient descent to optimize the sequence, which is prone to getting stuck in the local optimum due to the rough landscape of the loss function. The performance of DIFFUSION is also inferior to ours, as it generates protein using three separate diffusion models and fails to cross-condition on structure and sequence. In contrast, PROTSEED updates sequence and structure interdependently in end-to-end manner.
Table 2: PPL, RMSD and AAR of different approaches on the protein co-design task. (f): the higher the better. (L): the lower the better.
Method
PPL (L)
RMSD (Å, !)
AAR (%, f)
AFDESIGN DIFFUSION
3.47 (.11)
12.05 (.28)
11.63 (.08)
2.33 (.37)
24.97 (.32)
PROTSFED
8.87 (.17)
1.29 (.02)
33.10 (.34)
To demonstrate the efficiency of our method, we test inference stage of different approaches using single V100 GPU card the same machine, and present average runtime of these methods on proteins of different sizes. As indicated by Figure 3(a), our PROTSEED runs orders of magnitude faster than two baseline models all four protein sizes, as both AFDESIGN and DIFFUSION rely on time-consuming Monte Carlo sampling steps.
Table 3: PPL and AAR of different approaches on the fixed backbone sequence design task. (f): the higher the better. ((): the lower the better. Results of baselines are taken from Jing et al. (2021).
PPL. (L)
AAR (%, 1)
Method
Short
Single-chain
All
Short
Single-chain
All
TRA-TRRNNROR
8.94
8.67
6.70
27.3
28.3
36.5
STRUCTURED GNN
8.31
8.88
6.55
28.4
28.1
37.3
GVP-GNN
7.10
7.44
5.29
32.1
32.0
40.2
PROTSEED
7.32
7.38
5.60
34.8
34.1
43.8
4.3 FIXED BACKBONE SEQUENCE DESIGN
Setup. The third task is to design protein sequences that can fold into given backbone structures, which known as fixed backbone design. In this task, context features are dihedral angles and inter-residue distances derived solely from backbone coordinates (Jing et al., 2021), and protein structures are fixed as ground truth in the decoder. We use the CATH 4.2 dataset curated by Ingraham al. (2019), and follow all experimental settings of Jing et al. (2021) rigorously for a fair comparison, i.., using the same data splits and evaluation settings according to their official implementation. We compare PROTSEED with three basclines. Specifically, Structured GNN is an improved version of Ingraham et al. (2019). GVP-GNN (Jing et al., 2021) and Tran-rrosfrrmer (Hsu et al., 2022)
In this work, we say two residues are in contact if the distance between two C. is within 8 Á.
Phtpps: / /i thub. com/ sokrypton/Coo abbes
8

---

Published as a conference paper at ICLR 2023
(a)
(b)
Figure 4: Example of novel proteins designed by PROTSHED. (a) Extending the loop of a native protein (marked in red). (b) Novel | bareel design with different sizes. (c) Transmembrane protein complex design with a custom number of (twelve) a-hhelicss
are state-of-the-ar methods for fixed backbone design built upon Geometric Vector Perceptron (GVP) encoders (Jing et al., 2021). Note that since we not afford training models AlphaFold2- predicted data sets (Hsu et al., 2022) which requires hundreds of GPU days, we focus on the CATH data set for training. We evaluate the performance of all methods using PPL and AAR introduced in Section 1, and drop the RMSD metric as structures are provided in this task.
Results. Following Jing al. (2021), report the evaluation results three test splits including the full CATH 4.2 test set, the short subset (100 or fewer residues) and the single-chain subset. As shown Table 3, PROTSEED achieves competitive results against the state-of-the therth method GVP-GNN in terms of the perplexity, and outperforms all baselines in terms of the amino acid recovery rate. The results indicate that PROTSEED a quite general protein design framework and is also superior in designing protein sequences conditioned on desired backbone structures. Note that the otate-of-the-art method TRSSFORRSRR outperformed by GVP-GNN when trained solely on CATH data set, which is consistent with the results reported in the original paper (Hsu et al., 2022).
4.4
CASE STUDY
So far, we have evaluated PROTSEED's 's capability to design proteins of high fidelity on multiple settings. However, it remains unclear whether the proposed model can go beyond the topologies of existing proteins. To get insight into the proposed method and as a proof of concept, we manually construct set of secondary structure annotations and contact features from scratch, and ask the model trained in the second task to perform de novo protein design based on the context features provided by us. In Figure 4, we show that our PROTSEED succeeds in altering loop lengths of existing proteins, designing novel proteins with idealized topologies, and designing novel protein complexes with custom number of secondary structures. Notably, the designed structures are in close agreement with structures predicted by AlphaFold. taking the designed sequences as input. We further perform protein sequence and structure search against all available databases using FoldSeek (van Kempen et al., 2022) and BLAST (Altschul et al., 1990), and find these synthetic proteins are dissimilar to existing proteins regarding both sequence and structure. The case study serves as the first attempt to apply our model to de novo protein design in more realistic setting, which reveals the possibility of PROTSEED being a powerful tool for protein design in biological research. We refer readers to Appendix C.1 for all details about the case study.
CONCLUSION AND FUTURE WORK
In this paper, we propose novel principle for protein sequence and structure co-design called PROTSEED, which translates proteins in the joint sequence-structure space in an iterative and end- to-end manner. PROTSEED capable of capturing the interplay of sequence, structure, and context features during the translation, and owns a much more efficient inference process thanks to the one-shot translation strategy. Extensive experiments over a wide range of protein design tasks show that PROTSEED outperforms previous state-of-the-art baselines by a large margin, confirming the superiority and generality of our method. Further case studies on de novo protein design demonstrate PROTSEED's ' potential for more practical applications in biological research. Future work includes extending PROTSEEd to the scaffolding task (Trippe et al., 2022) and adopting latent variables (Kingma& & Welling, 2013) enable the context-free protein design.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
For the sake of reproducibility, the pseudo-code of PROTSEED, the parameterization of SoqIPA, as well as hyper-parameters and implementation details are provided in Appendix B. All codes, datasets, and experimental environments will be released upon the acceptance of this work.
ACKNOWLEDGEMENT
We would like to thank all the reviewers for their insightful comments. Jian Tang is supported by Twitter, Intel, the Natural Sciences and Engineering Research Council (NSERC) Discovery Grant, the Canada CIFAR AI Chair Program, Samsung Electronics Co., Ltd., Amazon Faculty Research Award, Tencent Lab Rhino-Bird Gift Fund, an NRC Collaborative R&D Project (A14D-CORE-06) as well as the IVADO Fundamental Research Project grant PRF-2019-3583139722
REFERENCES
Jared Adolf- Brryfllee Oleks Kalyuzhniy, Michael Kubitz, Brian D Weitzner, Xiaozhen Hu, Yumiko Adachi, William R Schief, and Roland L Dunbrack Jr. Rosettaanthodyddsign (rabd): A general framework for computational antibody design. PLoS computational biology, 14(4):e10061 12, 2018.
Rebecca F Alford, Andrew Leaver-Fay, Jeliazko R Jeliazkov, Matthew O'Meara, Frank P DiMaio, Hahnbeom Park, Maxim V Shapovalov, Douglas Renfrew, Vikram Mulligan, Kalli Kappel, et al. The rosetta all-atom energy function for macromolecular modeling and design. Journal of chemical theory and computation, (116)3031-3045 2017.
Stephen Altschul, Warren Gish, Webb Miller, Eugene W Myers, and David J Lipman. Basic local alignment search tool. Journal of molecular biology, 215(3):403-410, 1990.
Namrata Anand and Tudor Achim. Protein structure and sequence generation with equivariant denoising diffusion probabilistic models. arXiv preprint arXiv:2205.15000 2022.
Namrata Anand and Possu Huang. Generative modeling for protein structures. Advances in neural information processing systems, 31, 2018.
Namrata Anand, Raphael Eguchi, Irimpan Mathews, Carla P Perez, Alexander Derry, Russ B Altman, and Po-Ssu Huang. Protein sequence design with a learned potential. Nature communications, 13 (1):1-11, 2022.
Ivan Anishchenko, Samuel Pellock, Tamuka M Chidyausiku, Theresa A Ramelot, Sergey Ovehin- nikov, Jingzhou Hao, Khushboo Bafna, Christoffer Nom, Alex Kang. Asim K Bera, et al. De novo protein design by deep network hallucination. Nature, 600(7889):547-552 2021.
Minkyung Back, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang. Qian Cong, Lisa N Kinch, Dustin Schaeffer, Accurate prediction protein structures and interactions using three-track neural network. Science, 373(6557)8871-777 2021.
Shaojie Bai, Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural Information Processing Systems, 32, 2019.
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob de Haas, Neville Bethel, et al. Robust deep learning-bass protein sequence design using proteinmpnn. Science, pp. eadd2187, 2022.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals. Jakob Uszkoreit, and Eukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.038/7 2018.
Jiayi Dou, Anastassia A Vorobieva, William Sheffler, Lindsey A Doyle, Hahnbeom Park, Matthew J Bick, Binchen Mao, Glenna W Foight, Min Yen Lee, Lauren A Gagnon, et al. De novo design of a fluorescence-activaiing 3-barrel. Nature, (6((772))485-491 2018.
10

---

Published as a conference paper at ICLR 2023
James Dunbar, Konrad Krawczyk, Jinwoo Leem, Terry Baker, Angelika Fuchs, Guy Georges, Jiye Shi, and Charlotte M Deane. Sabdab: the structural antibody database. Nucleic acids research, 42 100:DD141-D11100 2014.
Raphael Eguchi, Christian Choe, and Po-Ssu Huang. Ig-vae: Generative modeling of protein structure by direct 3d coordinate generation. PLoS Computational Biology, 18, 2022.
RA Engh and R Huber. Structure quality and target parameters. 2012.
Noelia Ferruz, Steffen Schmidt, and Birte Höcker. Protgpt2 is deep unsupervised language model for protein design. Nature communications, 13(1):1-10, 2022.
Sharon Fischman and Yanay Ofran. Computational design of antibodies. Current opinion in structural biology, 51:156-162, 2018.
Wenhao Gao, Sai Pooja Mahajan, Jeremias Sulam, and Jeffrey Gray. Deep learning in protein structural modeling and design. Putterns, 1(9): 2020.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851 2020.
Emiel Hoogeboom, Victor Garcia Satorras, Clement Vignac, and Max Welling, Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning, pp. 8867-8887. PMLR, 2022.
Chloe Hsu, Robert Verkuil, Jason Liu, Zeming Lin, Brian Hie, Tom Sercu, Adam Lerer, and Alexander Rives. Learning inverse folding from millions of predicted structures. bioRxiv, 2022.
Po-Ssu Huang. Scott E Boyken, and David Baker. The coming of age of de novo protein design. Nature, 537(7620):320-327 2016.
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. Advances neural information processing systems, 32, 2019.
Yan-Bin Jia. Quaternions and rotations. Com S, 477(577): 2008
Wengong Jin, Jeremy Wohlwend, Regina Barzilay, and Tommi Jaakkola. Iterative refinement graph neural network for antibody sequence-structnn co-design. arXiv preprint 10.04221 2021.
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations, 2021.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvuakkoo,, Russ Bates, Augustin Židek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583
2021.
Wolfgang Kabsch and Christian Sander. Dictionary of protein secondary structure: pattern recognition of hydrogen-bondeed and geometrical features. Biopolymers: Original Research on Biomolecules, 22(12):577-2237, 1983.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
Miltiadis Kofinas, Naveen Nagaraja, and Efstratios Gavves. Roto-translated local coordinate frames for interacting dynamical systems. Advances in Neural Information Processing Systems, 34: 6417-6429, 2021.
Jonas Köhler, Leon Klein, and Frank Noe. Equivariant flows: Exact likelihood generative learning for symmetric densities. In Proceedings of the 37th International Conference Machine Learning. 2020.
Xiangzhe Kong, Wenbing Huang. and Yang Liu. Conditional antibody design as 3d equivariant graph translation. arXiv preprint arXiv:2208. 06073, 2022.

---

Published as a conference paper at ICLR 2023
Andrew Leaver-Fay, Matthew J OeMaaa,, Mike Tyka, Ron Jacak, Yifan Song, Elizabeth H Kellogg, James Thompson, lan W Davis, Roland Pache, Sergey Lyskov, et al. Scientific benchmarks for guiding macromolecular energy function improvement. In Methods in enzymology, volume 523, pp. 109-143. Elsevier, 2013.
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2837-2845, 2021.
Shitong Luo, Yufeng Su, Xingang Peng, Sheng Wang, Jian Peng, and Jianzhu Ma. Antigen-specific antibody design and optimization with diffusion-based generative models. bioRxiv, 2022.
Matt McPartlon and Jinbo Xu. An end-to-end deep learning method for rotamer-free protein side- chain packing. 2022.
Christine A Orengo, Alex D Michie, Susan Jones, David T Jones, Mark B Swindells, and Janet M Thortoon. Cath-a hierarchic classification of protein domain structures. Structure, 5(8: 1997.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, Lawrence Zitnick, Jerry Ma, al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, L18(155cc211223911 2021.
Jeffrey A Ruffolo, Jeremias Sulam, and Jeffrey J Gray. Antibody structure prediction using inter- pretable deep learning. Putterns, 3(2):100406 2022.
Koichiro Saka, Taro Kakuzaki, Shoichi Metsugi, Daiki Kashiwagi. Kenji Yoshida. Manabu Wada, Hiroyuki Tsunoda, and Reiji Teramoto. Antibody design using Istm based deep generative model from phage display library for affinity maturation. Scientific reports, 11(1):1-13, 2021.
Hao Shen, Jorge Fallas, Eric Lynch, William Sheffler, Bradley Parry, Nicholas Jannetty, Justin Decarreau, Michael Wagenbach, Juan Jesus Vicente, Jiajun Chen, et al. De novo design of self-assembling helical protein filaments. Science, 362(6415):705-79) 2018.
Chence Shi, Shitong Luo, Minkai Xu, and Jian Tang. Learning gradient fields for molecular conformation generation. International Conference on Machine Learning, pp. 9558-9568. PMLR, 2021.
Jung-Eun Shin, Adam Riesselman, Aaron W Kollasch, Conor McMahon, Elana Simon, Chris Sander, Aashish Manglik, Andrew C Kruse, and Debora S Marks. Protein design and variant prediction using autoregressive generative models. Nature communications, 12(1):1-11, 2021.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Martin Steinegger and Johannes Söding. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology. 35(11):1026-1028, 2017.
Doug Tischer, Sidney Lisanza, Jue Wang, Runze Dong, Ivan Anishchenko, Lukas F Milles, Sergey Ovchinnikov, and David Baker. Design of proteins presenting discontinuous functional sites using deep learning. Biorxiv, 2020.
Brian Trippe, Jason Yim, Doug Tischer, Tamara Broderick, David Baker, Regina Barzilay, and Tommi Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motif- scaffolding problem. arXiv preprint arXiv:2206.0411 2022.
Jérôme Tubiana. Dina Schneidman- Schneidman-Duhovny, and Haim J Wolfson. Scannet: An interpretable geometric deep learning model for structure- based protein binding site prediction. Nature Methods, pp. 1-10, 2022.
Michel van Kempen, Stephamic Kim, Charlotte Tumescheit, Milot Mirdita, Johannes Söding, and Martin Steinegger. Foldseek: fast and accurate protein structure search. bioRxiv, 2022.
12

---

Published as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems. 30, 2017.
Jue Wang, Sidney Lisanza, David Juergens, Doug Tischer, Ivan Anishchenko, Minkyung Baek, Joseph L Watson, Jung Ho Chun, Lukas F Milles, Justas Dauparas, et al. Deep learning methods for designing proteins scaffolding functional sites. bioRxiv, 2021.
Minkai Xu, Lantao Yu, Yang Song. Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv preprint wxXi:2203.002923, 2022.
Jianyi Yang, Ivan Anishchenko, Hahnbeom Park, Zhenling Peng, Sergey Ovchinnikov, and David Baker. Improved protein structure prediction using predicted interresidue orientations. Proceedings of the National Academy of Sciences, 117(3): 1496-14965333 2020.
Jinhua Zhu, Yingee Xia, Chang Liu, Lijun Wu, Shufang Xie, Tong Wang, Yusong Wang, Wengang Zhou, Tao Qin, Hougiang Li, et al. Direct molecular conformation generation. arXiv preprint 35. 2022.
13

---

Published as a conference paper at ICLR 2023
A EQUIVARIANCE PROPERTY OF SEQUENCE AND STRUCTURE TRANSLATION
We first quickly recap the process of sequence and structure translation in each translation layer. At (t 1)th layer, the decoder takes protein p { æ 0 and context features {m {z/y) as the input. It encodes sequence-structtur interplay and integrates all interactions into updated context features using SeqIPA adapted from Invariant Point Attention (IPA) (Jumper et al., 2021)
in a way that its roto- tro-trannsltioon invariant property is kept. The updates of Ca positions, frame orientations, and type distributions are then predicted based on updated context features. The whole process can be summarized as follows:
+055 MLPo(s)),
(12)
m
[+1
[+1 SeqIPA(fm)), (zfj). (s) (s/t0.6), { (O:1),
(13)
/ (m +1 mm), 2 = A x + O.a
(14)
O convert (m mm)), - 0
(15)
8
[+1 softmax (A MMLP,(m,,) ++1 m. 8, s.+.o5)).
(16)
To derive the equivariance property of each translation step, we use three functions X, O,O.S to denote the network that predicts the C. position translation, orientation translation, and sequence translation described above, respectively. Formally, we have:
Axt = XP')
(17)
0f+1 1 O(P'),
(18)
8; ++1 S(P").
(19)
Note that O,S also take {m²} and {=[y} as input and we omit these context features for simplicty, as they remain invariant to global rigid transformations. X, O, S are not separate networks, and they share the same input and the same SeqIPA, but are equipped with different MLPs. With the above definitions, we can derive the following proposition:
Proposition (Roto-Translate Translation Equivariance). Let TR.r denote any SE(3) transformation (rigid transformation) operating on the protein object P' = {(s), xi. O'),N with a rotation matrix R E SO(3) and translation vector r E RR3 The function X, O, satisfy the following equivariance properties:
o ( = RX(('),
(20)
o T (P') = RO(P'),
(21)
o TR. (Pt) = S(P)),
(22)
where (Pt) {($) at +r, RO
Intuitively, the proposition states that in each translation step, the updates of C. positions and frame orientations are equivariant with respect input protein structures, and the updates of type distributions are invariant.
Proof. We first prove that Eq. 20 holds. Notice that SeqIPA is aware of the orientations of the input structure, and the updated context features invariant (Eq. 13). Therefore, the predicted deviation of C, positions, i.e., $.. is invariant (Eq. 14). Then, we have:
X TOTRR = RO RX(P').
(23)
The Eq, 21 and Eq. 22 can be proved in a similar way.
14

---

Published as a conference paper at ICLR 2023
B
MODEL DETAILS
B.1 PSEUDO CODE
The pseudo code of PROTSEED is provided in Algorithm 1. The proposed PROTSBED consists of a trigonometry-awanl encoder (Algorithm 1, line 2-7) that reasons geometrical constraints and interactions from context features, and roto-translation equivariant decoder (Algorithm 1, line 10- 19) that translates protein sequence and structure interdependently. Starting from the intial single features {m,) E R&xxcm Xcm and pair features ( (z1z} E RR NWXNNexs the whole model iteratively translates both protein sequence and structure into the desired state from random initialization (Algorithm 1, line 9). We note that the whole process does not require MCMC sampling, and runs much faster than autoregressive models and diffusion-baan based models.
The iigonometry-awarr encoder composed of a stack of L encoding layers. Each layer takes single features {m!] } and pair features {2(,) from the last layer as its input, and updates these features with novel attention mechanisms. After L rounds of feature propagation, the updated single features and pair features as inputs to the decoder for joint protein sequence-structurs design (Algorithm 1, line 8).
The roto-translation equivariant decoder iteratively refines the concrete 3D atom coordinates of the protein from random initialization, based on context features calculated by the encoder. The interplay of residue types, residue structures and context features during the decoding process captured by novel orientation- aware attention mechanism (ScqIPA, Appendix B.2). The equivariance property of the structure translation process is guaranteed by mapping invariant predictions local frames to global frames with change of basis (Algorithm 1, line 13-16). It is worth mentioning that PROTSEED updates sequence and structure all residues an one-shot manner, leading to a much more efficient inference process (Algorithm 1, line 17).
Algorithm 1 PROTSEED
Require: Initial single features {m;} E RNN and pair features {ziz} E ERRXNXX,
1: m.. 20 f- Linear(m:),
Lin)mLimeaa Linear(zzj)
m E R., x! E R
2g
2: for I <- to I. do
3: m; <- MHA((m)), (=t
> Eq.
4: Zú
[4u.5 +- z1y Linear(m, @ 1
>> Eq. 2
5: 140.75 t 2g
140.5 + TriangleUpdater (1 1
b Eq. 4
6: 2 +1
140.75 + TriangloUpdaten ( ([2)) (40.75))
b Eq.
7: end for
8: m. 20 Zij
2 f- m
Initialize context features for decoder Initialize protein po
zij 9: po [(( x )
Ji] (() 1, (0,0,, I3)
10: for 1 1
1+0.5 +- -MLP.(s.)
D 8 + E R >> Eq. and Section B.2 b Deviation of C. positions in local frame > Deviation of C, positions in global frame Convert quaternion to rotation matrix
8,
1+1 2 + SeqlPA((m (zl)). (st 1055 (0
m.
13:
a + MLP
m
14:
+l + x + #
15:
O convert (MLP. m?)
16:
0 + 0
17:
8 H1 t softmax (A MLP ++1
mn, s(+0.3))
b Eq. 9
18:
pi+1 +- ((stt1 x 2+1
19: end for
Return: The trajectory of the protein translation (P')
15

---

Published as a conference paper at ICLR 2023
B.2 PARAMETERIZATION OF SEQIPA
SeqIPA is adapted from the Invariant Point Attention (IPA) (Jumper et al., 2021). which takes residue types as the additional input capture the interactions between current decoded sequences, structures, and the context features. We ensure that the additional input does not affect the invariance property of the IPA to make full use of its capacity. Specifically, we propose the following two strategies to parameterize the SeqIPA.
SeqIPA-Additin
Given that {s/+0.5) share the same dimensionality with (mí), J very simple strategy is to just add embeddings of residue types onto single representations. Following the original implementation of IPA, we leave the pair features unchanged in this approach.
m; ; ij
2-1 1+1 = SeqIPA((mm), (z()), {sf (s/+0.5), (. (mf), (01))
(24)
TPA((m) + of st+0.5,, (2(g): (O:)).
(25)
The above equations say that for SeqIPA-Addite we just add the sequence embeddings (s/+0.5}
onto the single representation {m (5q. and feed four inputs to vanilla IPA (Jumper et al., 2021).
DeqlPA-Attention Another more complicated strategy is to construct new set of single representa- tions and pair representations based on the embeddings of the current residue types. Then, we adopt a lightweight encoder similar to the encoder introduced in Section 3.2 to update m and 2/y. which are then fed into the vanilla IPA module. We summarize the computation flow as follows:
m, m; Linear(s,*0.5) Linear(s(+0.. + s,t0.
(26)
mi, žij = Encoder((m), (21z)),
(27)
m:
1+0.5 1+0.5 m + .,; z/3 + Kij,
(28)
m 1+1 ++1 SeqIP(((m (-|;); ); {s. (s,+0.5), (x(),, (O:))
21
(29)
= IPA (005), { (2f+0.3,, (xt). (O,)).
(30)
The above equations say that for SeqIPA-Attention, we leverage another lightweight encoder (Eq.27)
similar to the encoder introduced in Section 3.2 to first update m and z/ (Eq,28), and then feed updated inputs to vanilla IPA (Eq.30).
In practice, we find both strategies work well and their performance is on par with each other. To make the whole model lightweight, we adopt the first strategy across all the experiments in this work. We emphasize that the parameterization of the SeqIPA is quite flexible, as long as it can model interactions between sequences, structures, and context features, and is invariant to the global transformation of input structures. For the concrete computation flows of the IPA module, we refer readers to Algorithm 22 described in the supplementary material of Jumper al. (2021).
B.3 HYPER-PARAMETER AND IMPLEMENTATION DETAILS
PROTSBED is implemented in Pytorch. The iigonometry-awane context encoder is implemented with L= 8 layers, and the sequence-stree struccuue decoder implemented with T 8 layers. The hidden dimension is set as 128 for pair features and 256 for single features across all modules.
For training. we use learning rate of 0.001 with 2000 linear warmup iterations. We empirically find that proper learning rate warmup schedule can lead to faster convergence rate and higher performance. The model is optimized with Adam optimizer on four Tesla V100 GPU cards with distributed data parallel. The estimated time it takes to get converged model is 24 hours.
For inference, the temperature of the sequence distribution, i.e., X, controls the sharpness of the distribution. The larger À will lead to higher (better) AAR and higher (worse) PPL, and vice versa. Since it acts oppositely on AAR and PPL, we simply set it as across the experiments.
All codes, datasets, and experimental environments will be released upon the acceptance of this work.
B.4 FULL ATOM POSITIONS RECONSTRUCTION
The bond lengths and bond angles between backbone atoms are relatively conserved. The C, is connected with N, C, and C3 (except for Glycine which has single hydrogen atom its side
16

---

Published as a conference paper at ICLR 2023
chain) atoms. The C. forms canonical orientation frame with respect N. C, and Cg. Once know the positions of the Ca. and the orientation of the frame, the full backbone atom positions their averaged relative positions with respect to the C. recorded in
can be derived according literature. The positions of all sidechain atoms of the 20 different amino acids can also be compactly specified by four torsion angles (x1, X2- X3, X4) (Jumper al., 2021; McPartlon & Xu, 2022), which also follow amino-acid specific distributions recorded in literature. All these recorded statistics can be found in https::/gtt..cicccceee www.ecorreuuiibbsee....... unibas wttp://eeommeeeeeeeeeeeeeeeeeeee
openstructure/
steree_chemicallee txt.
17

---

Published as a conference paper at ICLR 2023
Figure 5: Superimposition of three generated proteins and their most similar proteins found in the PDB by FoldSeek. left: a novel protein with loop extended. middle: novel B-barrel. right: a novel helical complex.
C EXPERIMENTAL DETAILS
C.1 CASE STUDY
We conduct three case studies to evaluate PROTSEED's capability to perform de novo protein design, including extending the loop of existing proteins, designing novel B-barrels, and designing novel helical complexes. Specifically, we manually curate a set of secondary structure annotations and contact features, and ask the model trained the second task (Section 4.2) to generate novel proteins based on these context features. We elaborate the way we design context features for each setting.
Extending the Loop. this setting, we start by calculating the secondary structure annotations and the contact matrix of an existing protein. We then insert n consecutive "C"" ("C" is the secondary structure annotation for the loop) letters into the original secondary structure annotations at the position where want to extend the loop. Similarly, insert consecutive rows and columns filled with zero into the original contact matrix. For a new-inserted residue indexed by i, we let it to be in contact with i 2, 1, 1, 2.
Designing Novel Beta Barrels. this setting, we grab a simple pattern of B-barrel proteins and then repeat this pattern multiple times to construct the contact features. The secondary structure annotations are also calculated by repeating the annotations of the pattern multiple times.
Designing Novel Helical Complexes. Similar the second case, in this setting, also take simple pattern of helical complexes and construct the contact features by repeating it multiple times. The secondary structure annotations are all to be "H".
In Figure 5, we show the superimposition of three novel proteins designed by PROTSEED against the most similar proteins in the PDB, for each setting, which confirms the novelty of the designed proteins.
18

---

Published as a conference paper at ICLR 2023
Table 4: Ablation study on the antibody CDR co-design task. (t): the higher the better. (1): the lower the better.
PPI. (1)
RMSD (Á, 4)
AAR (%, f)
CDR
H1
H2
H3
H1
H2
H3
H1
H2
H3
PROTSEED
4.43
5.94
10.88
1.24
1.11
3.19
70.22
63.53
39.27
No Sequence Interaction
5.19
7.77
14.78
1.48
1.32
3.24
68. 16
59.14
32.42
Single Iteration Translation
6.03
17.98
26.11
2.17
2.41
5.09
67.49
55.82
29.42
No ScqIPA
13.96
14.57
17.87
18.22
18.92
11.37
9.88
10.14
7.42
C.2 ABLATION STUDY
To gain more insights into the effectiveness of each module in PROTSEED, we conduct additional ablation studies following the setting of the antibody CDR co-design in Section 4.1.
Sequence and Structure. In this ablation study (denoted as No Sequence Interaction), we replace the SeqIPA in Eq.6 with the vanilla IPA (Jumper et al., 2021), and directly predict the distribution of amino acid types for all residues at the last iteration of the decoding process using single features, i.c., (m² ? }. The results shown Table indicate that when the model fails to cross-condition both sequence and structure during the decoding, there is a significant performance drop in all three metrics, especially for PPL and AAR. This confirms the necessity cross-condition sequence and structure during the decoding, and the effectiveness of the proposed SeqIPA.
Effectiveness of Cross-conditioning
Effectiveness of Iterative Translations. In this ablation study (denoted Single Iteration Transla- tion). we replace the T layer decoder with single-layer decoder for protein translation. We note that the vanilla decoder of PROTSEED is composed of T 8 consecutive translation layers with tied weights, and the number of trainable parameters of these two models are the same due to the weight tying. As indicated by Table the single-layer decoder is outperformed by the vanilla decoder by large margin on all metrics. Since the numbers of trainable parameters of these two models are same, this is fair comparison. The results justify the advantages of the iterative translation framework for protein sequence and structure co-design.
Effectiveness of Context Feature Update. this ablation study (denoted No SeqIPA), we freeze all context feature updates in the decoder and remove SeqIPA (Eq.6), and use the outputs of the encoder as the context features during the whole decoding process. As shown by Table 4, the performance degrades dramatically, which demonstrates that the context feature update plays key role PROTSEED.
19

---

Published as a conference paper at ICLR 2023
D DISCUSSION
D.1 CONTEXT FEATURES
In this work, we use the concept of context features (single features {m,} and pair features { } as formulation to unify the inputs of different protein design tasks. We note that (m;) and {zi}] vary from task task, and for most well-defined tasks, they are easy to get. For example, in antibody CDR design tasks (Jin et al., 2021; Luo et al., 2022; Kong et al., 2022). they can be derived from antibody frameworks and structures of binding antigens. the general protein design task proposed by Anand & Achim (2022), they can be derived from second structure annotations and contact maps provided by biologists. In scaffolding tasks, they can be derived from an starting motif (Trippe et al., 2022). The more context features (or constraints) the rescarchers specify, the more control they can have over the designed proteins. We refer readers to Dou et al. (2018); Shen et al. (2018) 8) for two cases of protein design in real world scenarios.
D.2 CONTEXT-FREE PROTEIN DESIGN
Researchers may be interested at generating novel proteins without relying any context features, a.k.a. context-free protein design. We note that PROTSEED a very general framework and can handle this situation with minor modifications. Specifically, we adopt the Variational Autoencoder (VAE) framework (Kingma & Welling, 2013) and approximate the data distribution by learning to map proteins to latent vectors and reconstruct proteins from latents. In this scenario, the context features become the proteins sampled from target data distributions. Since this is out of the scope of this work, we leave it as future work.
20


Reviews:
[{"review_id": "qZQBwDFJ8c", "reviewer": "Reviewer_XDjj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a method for protein sequence and structure codesign. Current methods use either autoregressive or diffusion-based models, which the authors claim are computationally expensive and produce suboptimal solutions. Sequence generative models can produce good designs, but do not model protein structures. The authors address the above issue with their method ProtSeed. ProtSeed uses a triangular-aware encoder to learn geometrical constrain from context features and roto-translation equivariant decoder that iteratively improves protein structure followed by an MLP to decode the amino acid identities from the structure. The author performed experiments on the structural antibody database and 2 protein design benchmarks.", "summary_of_the_review": "A good paper about an interesting method. Hyperparameters and training details need to be discussed. The code and models should be released to facilitate reproduction and future work. Some discrepancies with prior work should also be addressed.\n\nWhat would improve my score: include hyperparameter and training details. Resolve discrepancy with prior work.", "strength_and_weaknesses": "Strengths:\n1. Good introduction and well-presented related work focusing on key aspects.\n2. Good evaluation results compared to previous methods\n3. A novel approach for solving the structure and sequence of protein and study for the de novo protein design\n\nWeaknesses:\n1. Lack of specific hyperparameter and training details\n2. No mention that code and trained models will be released\n3. The application of the antibody CDR design task is clear, but the relevance of the other two tasks is murkier. When would detailed structural specifications be available, but without an existing sequence that forms that fold?\n4. The authors note a difference between their evaluation and the published scores for Diffusion, GNN and RABD models (Table 1). However, the concern remains that the Diffusion model shows significantly better RMSD scores compared to the scores in this paper. What is the source of this discrepancy?\n5. ProtSeed improves over previous methods in terms of PPL, RMSD, and AAR. However, because the task is to design functional proteins, these metrics may not reflect true model performance, because a model with higher RMSD may, in fact, produce (more) functional proteins while also being worse at predicting structure. It would be helpful to provide some additional, functional-oriented, evaluation. For example, by comparing the protein-protein interaction patterns and charge distributions between the designs and the ground truth.\n6. AAR is surprisingly low for all of these methods, including ProtSeed. If only 40% or less of amino acids are correctly recovered, how confident can we be that any of these methods actually produce good designs?\n7. I’m not convinced by AF2 structure prediction as an evaluation. It isn’t surprising that AF2 would predict the designed proteins to fold as intended, because all of these models are trained on the same PDB data and, therefore, probably have the similar pathologies.\n\nOther minor comments:\n1. Figure 4. – To keep a uniform style among all figures, a, b c marks could be moved outside of the figures as in Fig. 1-3\n2. Table 3 – highlight best results as in table 1 and 2\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lHH1tIiZ6jV", "reviewer": "Reviewer_UR24", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "- Protseed introduces a model that can jointly model the sequence and structure information in order to do sequence design.\n- Protseed tests against a suite of state of the art protein design benchmarks, like AlphaDesign, Diffusion, and GVP-GNN, on fixed backbone sequence design and antibody CDR design tasks.\n- The model is able to get state of the art results and beat competitors on most tasks.", "summary_of_the_review": "Because of the strong empirical results and good benchmarking, I would recommend an accept. Comparing to more recent models and on larger datasets would make this paper much stronger.", "strength_and_weaknesses": "Strengths:\n- Paper uses well accepted metrics and implements recent results to make strong comparisons.\n- SeqIPA is an interesting extension to the normal IPA module\n- The results seem to be state of the art, beating out recent design models, at least on join sequence and structure design.\n\nWeaknesses:\n- The comparisons to fixed backbone design is weak, recent models like ESM-IF using larger datasets or prot-MPNN exists, and some effort should be made to compare to those.\n- The work lack ablations, it's unclear which part of the model changes give such a large improvement.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VTX04uae48", "reviewer": "Reviewer_GDvK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a framework for protein sequence and structure generation. The key idea is to involve the structure, sequence, and context together for encoding and decoding. The key modules are trigonometry-aware encoder and roto-translation equivariant decoder in a one-shot way. The experiments are conducted on three different tasks and the results show the ProtSeed is effective. ", "summary_of_the_review": "N/A", "strength_and_weaknesses": "Strength:\n1. The paper works on the protein sequence and structure co-design task, which is important and hot recently. \n2. The method in this work is well-motivated and the modeling way is reasonable and solid. \n3. The results show strong improvements over previous works on different tasks. \n\nWeaknesses:\n1. Most of the paper writing is good. However, in some claims and places, they are not appropriate enough. The authors mainly claim that the specific antibody prediction method, e.g., RefineGNN, is not general and is in a specific domain. However, in my view, their method is general, but they only show the application in antibodies. Besides, the sequence, structure, and rotamers sequentially generation are slow as the authors claimed, but the inconsistent and fanciful proteins may not be the truth. A good sequence can help structure, and a good structure can also help sequence, iteratively update sequence and structure is hard to say a negative point or \"issue\" as the authors mentioned. \n2. From an overview of the technique, the trigonometry-aware encoder and the equivariant decoder are mostly taken from Alphafold2, though some modifications are taken. In terms of this, the novelty is hard to say as big as the authors claimed. Alphafold2 also worked on the protein (though they don't predict the sequences). \n3. There are some places in the method questionable. (a). On the decoder side, within the encoder output, the decoder continuously updates the context feature $m$ and $z$, where the interactions of the context amino acids are involved in the encoding way. However, the update of the amino acids that need to predict seems to be not so clear and reasonable. For example, the $z$ feature is only performed between the amino acids in the context, but no consideration between the predicted amino acids (if I understand correctly of $z$). (b) The description of SeqIPA is not well presented. Though in supplementary (please append this as Appendix after the main text instead of supplementary) the authors try to explain, but the formulation is not shown. When compared with the original IPA, it is still not clear how to calculate with the five different inputs. (c) In eqn. 7 and 8, why the MLP takes m(t+1) and m(0) rather than updated m(t+1)? Is $\\lambda$ necessary? Also more explanations may need for Algorithm 1. (d) In eqn.11, how $x_{ij}$ is calculated since in previous descriptions they are all based on the amino acid level instead of the atom level, this should be noted.  \n4. One disadvantage of this method is that they require the context of $m$ and $z$, amino acid and pair features. This somehow concerns me about the practice in real scenarios. The hard request of these two features may not be friendly for some tasks. For example, how to generate from a general distribution. The authors can provide more discussions. \n5. Though the decoding speed is much improved. I would like to also know more about the training cost, since the method is much more complex than previous ones. For example, small encoders are also involved in the decoder side. Besides, as for the experimental comparisons, since the compared baselines are provided by the authors, the code should be published (also the implemented baseline code). Also, the ablation study is not provided, it is hard to see the contribution of each part. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "I9iBOD7uFdu", "reviewer": "Reviewer_mVZ1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an iterative method for protein sequence-structure co-design. In each generation step, the model first uses IPA network to update the 3D structure ($C_a$ coordinates and the orientation of its frame). It then predicts the amino acid types at all positions. The proposed method is applied to antibody CDR design (co-design) and protein design (fixed backbone and co-design) applications and showed state-of-the-art performance.", "summary_of_the_review": "The paper is a good contribution to ML protein engineering field and I vote for acceptance. The methodology is similar to previous methods, but with useful modifications (e.g., iterative generation of amino acids rather than autoregressive). Moreover, it demonstrated its utility over general proteins (not just antibodies) and therefore has a larger impact.", "strength_and_weaknesses": "Strength:\n1. The proposed method improves upon the previous co-design method (Jin et al 2021) by generating all the amino acids in an iterative refinement process rather than left-to-right autoregressive process.\n2. The inference cost is lower than diffusion models which requires thousands of generation steps.\n\nWeakness:\n1. For protein design tasks, the term \"co-design\" is somewhat misleading. The method does not generate protein structure (and sequence) from scratch. It is conditioned on the contact map + secondary structure, which is a fuzzier representation of the backbone structure but it contains many information of the structure. The contact map needs to be taken from an existing structure, although it can be specified manually for certain types of structures.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster