Conditional Antibody Design as 3D Equivariant Graph Translation
Authors: Xiangzhe Kong
Wenbing Huang
Yang Liu
Venue: ICLR 2023 notable top 5% (2023)
Keywords: conditional antibody generation
equivariant
multi-channel attention

Abstract: Antibody design is valuable for therapeutic usage and biological research. Existing deep-learning-based methods encounter several key issues: 1) incomplete context for Complementarity-Determining Regions (CDRs) generation; 2) incapability of capturing the entire 3D geometry of the input structure; 3) inefficient prediction of the CDR sequences in an autoregressive manner. In this paper, we propose Multi-channel Equivariant Attention Network (MEAN) to co-design 1D sequences and 3D structures of CDRs. To be specific, MEAN formulates antibody design as a conditional graph translation problem by importing extra components including the target antigen and the light chain of the antibody. Then, MEAN resorts to E(3)-equivariant message passing along with a proposed attention mechanism to better capture the geometrical correlation between different components. Finally, it outputs both the 1D sequences and 3D structure via a multi-round progressive full-shot scheme, which enjoys more efficiency and precision against previous autoregressive approaches. Our method significantly surpasses state-of-the-art models in sequence and structure modeling, antigen-binding CDR design, and binding affinity optimization. Specifically, the relative improvement to baselines is about 23\% in antigen-binding CDR design and 34\% for affinity optimization.

Published as a conference paper at ICLR 2023
CONDITIONAL ANTIBODY DESIGN AS 3D EQUIVARI- ANT GRAPH TRANSLATION
Xiangzhe Kong' Wenbing Huang Yang L2u3:
DDept. of Comp. Sci. & Tech., Institute for AI, BNRist Center, Tsinghua University IIstitute for AT Industry Research (AIR), Tsinghua University Beijing Academy of Artificial Intelligence
"Gaoling School of Artificial Intelligence, Renmin University of China
6 Beijing Key Laboratory of Big Data Management and Analysis Methods, Beijing, China Jackie.xXZ@out look. com, hwenbing@1 com, 1iwyang2011ttsinnneeere
ABSTRACT
Antibody design is valuable for therapeutic usage and biological research. Ex- isting deep-learning-base methods encounter several key issues: 1) incomplete context for aieemmentartty-etermminn
Regions (CDRs) generation; 2) inca- pability of capturing the entire 3D geometry of the input structure; 3) inefficient prediction the CDR sequences an autoregressive manner. In this paper, we propose Multi-channel Equivariant Attention Network (MEAN) co-design ID sequences and 3D structures of CDRs. To be specific, MEAN formulates antibody design conditional graph translation problem by importing extra components including the target antigen and the light chain of the antibody. Then, MEAN re- sorts to eqqivariiat message passing along with a proposed attention mecha- nism to better capture the geometrical correlation between different components. Finally, outputs both the ID sequences and 3D structure via multi-round pro- gressive full-shot scheme, which enjoys more efficiency and precision against pre- vious autoregressive approaches. Our method significantly surpasses state-of-the- art models in sequence and structure modeling, untigen-binding CDR design, and binding affinity optimization. Specifically, the relative improvement to baselines is about 23% in antigen-binding CDR design and 34% for affinity optimization.
INTRODUCTION
Antibodies are Y-shaped proteins used by our immune system to capture specific pathogens. They show great potential in therapeutic usage and biological research for their strong specificity: each type of antibody usually binds to a unique kind of protein that is called antigen (Basu et al., 2019). The binding areas are mainly located the so-called gions (CDRs) in antibodies (Kuroda et al., 2012). Therefore, the critical problem of antibody design is to identify CDRs that bind to a given antigen with desirable properties like high affinity and col- loidal stability (Tiller & Tessier, 2015). There have been unremitting efforts made for antibody design by using deep generative models (Saka et al., 2021; Jin et al., 2021). Traditional methods focus on modeling only the ID CDR sequences, while recent work (Jin et al., 2021) proposes to co-design the 1D sequences and 3D structures via Graph Neural Network (GNN).
haaplemmentrrty-Deermmmnnn Re-
Despite the fruitful progress, existing approaches still weak in modeling the spatial interaction between antibodies and antigens. For one thing, the context information is insufficiently considered. The works (Liu et al., 2020; Jin et al., 2021) only characterize the relation between CDRs and the backbone context of the same antibody chain, without the involvement of the target antigen and other antibody chains, which could lack complete clues reflect certain important properties for antibody design, such as binding affinity. For another, they are still incapable of capturing the entire 3D geom- etry of the input structures. One vital property of the 3D Biology is that each structure (molecular, protein, etc) should be independent to the observation view, exhibiting E(3) equivaariva To fulfill
"Corresponding authors: Wenbing Huang, Yang Liu. F.(3) is the group of Fuclidean transformations: rotations, reflections, and translations.

---

Published as a conference paper at ICLR 2023
this constraint, the method by Jin et al. (2021) pre-processes the 3D coordinates certain invariant features before feeding them to the model. However, such pre-procession will lose the message of direction in the feature and hidden spaces, making it less effective in characterizing the spatial prox- imity between different residues in antibodies/antigens,, Further, current generative models (Saka et al., 2021; Jin et al., 2021) predict the amino acids one by one; such autoregressive fashion suffers from low efficiency and accumulated errors during inference.
To address the above issues, this paper formulates antibody design as E(3)-qqivvariant graph trans- lation, which is equipped with the following contributions: 1. New Task. We consider conditional generation, where the input contains not only the heavy-chain context of CDRs but also the infor- mation of the antigen and the light chain. 2. Novel Model. We put forward an end-to-end Multi- channel Equivariant Attention Network (MEAN) that outputs both 1D sequence and 3D structure of CDRs. MEAN operates directly in the space of 3D coordinates with =(3)--quivariaace (other than previously-usee E(3)-invariant models), such that it maintains the full geometry of residues. By alternating between internal context encoder and an external attentive encoder, MEAN lever- ages 3D message passing along with equivariant attention mechanism to capture long-range and ppatially-complicann interactions between different components of the input complex. 3. Efficient Prediction. Upon MEAN, we propose to progressively generate CDRs over multiple rounds, where each round updates both the sequences and structures in full-shot manner. This progressive full- shol decoding strategy is less prone to accumulated errors and more efficient during the inference stage, compared with traditional autoregressive models (Gebauer et al., 2019; Jin et al., 2021). We validate the efficacy of our model on three challenging tasks: sequence and structure modeling, antigen-bindin CDR design and binding affinity optimization. Compared to previous methods that neglect the context of light chain and antigen (Saka et al., 2021; Akbar et al., 2022; Jin et al., 2021), our model achieves significant improvement on modeling the ID/3D joint distribution, and makes a great stride forward on recovering optimizing the CDRs that bind to the target antigen.
2 RELATED WORK
Antibody Design Early approaches optimize antibodies with hand-crafted energy functions (Li et al.. 2014; Lapidoth et al., 2015; and have intrinsic defects because the inter-chain interactions are complicated in nature and cannot be fully captured by simple force fields or statistical functions (Graves et al., 2020). Hence, attention has been paid to applying deep generative models for 1D sequence prediction (Alley et al., 2019; Liu et al.. 2020; Saka et al., 2021; Akbar et al., 2022). Recently, to further involve the 3D structure, Jin et al. (2021) proposes to design the sequences and structures of CDRs simultaneously. It repre- sents antibodies with H(3)-invariant features upon the distance matrix and generates the sequences autoregressively. Its follow-up work (Jin et al., 2022) further involves the epitope, but only consid- ers CDR-H3 without all other components in the antibody. Different from the above learning- based works, our method considers a more complete context by importing the 1D/3D information of the antigen and the light chain. More importantly, we develop an E(3)-equivariant model that is skilled in representing the geometry of and the interactions between 3D structures. Note that antibody de- sign assumes both the CDR sequences and structures are unknown, whereas general protein design predicts sequences based on structures (Ingraham et al., 2019; Karimi et al., 2020; Cao et al., 2021).
et al., 2018), which rely on costly simulations
Equivariant Graph Neural Networks The growing availability of 3D structural data in various fields (Jumper et al., 2021) leads to the emergence of geometrically equivariant graph neural net- works (Klicpera et al., 2020; Liu et al., 2021; Puny et al., 2021; Han al., 2022). In this paper, we exploit the scalarization- bbased E(n)-equivarian GNNs (Satorras et al., 2021) as the building block of our MEAN. Specifically, we adopt the multichannel extension by Huang al. (2022) that natu- rally complies with the multichannel representation of residue. Furthermore, we have developed novel equivariant attention mechanism in MEAN to better capture antibody-antiged interactions.
METHOD
3.1 PRELIMINARIES, NOTATIONS AND TASK FORMULATION
An antibody is a Y-shaped protein (Figure 1) with two symmetric sets of chains, each composed of a heavy chain and a light chain (Kuroda et al., 2012). In each chain, there are some constant
2

---

Published as a conference paper at ICLR 2023
domains, and variable domain (VH/VL) that has three Complementarity Determining Regions (CDRs). Antigen- binding sites occur on the variable domain where the interacting regions are mostly CDRs, especially CDR-H3. The remainder of the variable domain other than CDRs is structurally well conserved and often called framework c
region (Kuroda et al., 2012; Jin et al., 2021). Therefore, previous works usually formalize the antibody design problem as finding CDRs that fit into the given framework region (Shin et al., 2021; Akbar et al., 2022). As suggested by (Fischman & Ofran, 2018; Jin al., 2021), we focus on generating CDRs in heavy chains since they contribute the most to antigen- binding affinity and are the most challenging ones to characterize. Nevertheless, in contrast to previous studies, we additionally incorporate the antigen and the light chain into the context in the form of antibody-antigen complexes, to better control the binding specificity of generated antibodies.
A
B
C
Hocey Chils
Chicchair
Kiligee
Constad Donuse
Figure 1: (A) The structure of residue, where the backbone atoms The structure of an antibody which is symmetric and Y-shaped, and we focus on the three versatile CDRs on the variable domain of the heavy chain. (C) Schematic graph construction for the antigen- antibody complex, with global nodes, internal context edges &in and external interaction edges E
N, C.. C. (B)
We represent each antibody-antigen complex as a graph of three spatially aggregated components, denoted as 9 = (V = [VH, VL. VAJ. & == {Ein, E Here, the components VH. VZ. VA corre- spond to the nodes (i.e. the residues) of the heavy chain, the light chain and the antigen, respectively; En and Eex separately contain internal edges within each component and external edges across com- ponents. To be specific, each node in V. i.e. U, = (h, Z,) is represented as a trainable feature embedding vector h; = Sa, E Rda according to its amino acid type a; and a matrix of coordinates Z; E 3 consisting of m backbone atoms. bone atoms (N, Ca, C, о}. where C. denotes the alpha carbon of the residue and others refer to the atoms composing the peptide bond (Figure 1). We denote the residues in CDRs to be generated as Vc = (Vey Veg: ...y "'nnee)), ). which is a subset of VH. Since the information of each Ve, is unknown in the first place, we initialized its input feature with a mask vector and the coordinates according to the even distribution between the residue right before CDRs (namely, Vo Va,-1) and the one right after CDRs (namely, +1). In our main experiments ($ 4), we select the 48 residues of the antigen closest the antibody in terms of the C. distance as the epitope like Jin et al. (2021). Instead of such hand-crafted residue selection, in Appendix L, we also incorporate the full antigen and let our method determine the epitope automatically, where the efficacy of our method is still exhibited.
our case, we set m = by choosing 4 back-
Edge construction We now detail how to construct the edges. For the internal edges, En is defined as the edges connecting each pair of nodes within the same component if the spatial distance in terms of Ca is below a cutoff distance C1. Note that adjacent residues in a chain are spatially close and we always include the edge between adjacent residues in En- In addition, we assign distinct edge types by setting eij = for those adjacency residues and eij = O for others, to incorporate the 1D position information. For the external edges &ex, they from two different components have distance less than a cutoff 02 (C2 > c1). It is indeed necessary to separate internal and external interactions because their distance scales are very different. The external connections actually represent the interface between different chains, which dominates the binding affinity (Chakrabarti & Janin, 2002), and they are formed mainly through moleclaol forces instead of chemical bonds that form the internal connections within chains (Yan et al., 2008). Note that all edges are constructed without the information of ground-truth CDR positions.
derived if the nodes
Global nodes The shape of CDR loops is closely related to the conformation of the framework region (Baran et al., 2017). Therefore, to make the generated CDRs aware of the entire context of the chain they are in, we additionally insert a global node into each component, by connecting it to all other nodes in the component. Besides, the global nodes of different components are linked to
3

---

Published as a conference paper at ICLR 2023
Jointly predict sequence and structure
External
L-oond Alternate passing
Interaction
Interaction
L.
iinerrnal
AlL
Context
La
Context
Internal Context Message Passing
Message Passing
Heavy Chain
O Light Chain
Antigen
Update graph with weighted residue embedding and structure
The overview of MEAN and the progressive full-shot decoding. In each iteration, we alternate the internal context encoding and external interaction encoding over L layers, and then update the input features and coordinates CDRs for the next iteration with the predicted values.
Figure 2:
each other, and all edges induced by the global nodes are included in &mn- The coordinates of a global node are given by the mean of all coordinates of the variable domain of the corresponding chains.
Task formulation Given the 3D antibody-antigen complex graph 9 (V,E), seek 3D equivariant translator f to generate the amino acid type and 3D conformation for each residue in CDRs Vc. Distinct from ID sequence translation in conventional antibody design, our task requires to output 3D information, and more importantly, we emphasize equivariance to reflect the symmetry of our 3D world- the output of f will tramslatc/rotaterreffeet in the same way as its input. We now present how to design f in what follows.
3.2 MEAN: MULTI-CHANNEL EQUIVARIANT ATTENTION NETWORK
To derive effective translator, it is crucial to capture the 3D interactions of the residues in different chains. The message passing mechanism in -33uiqquiva GNNs (Satorras et al., 2021; Huang al., 2022) will fulfil this purpose. Particularly, we develop the Multi-channel Equivariant Attention Network (MEAN) to characterize the geometry and topology of the input antibody-antigen complex. Each layer of MEAN alternates between the two modules: internal context encoder and external interaction encoder, which is motivated by the biological insight that the external interactions be- tween antibodies and antigens are different from those internal interactions within each heavy/light chain. After several layers of the message passing, the node representations and coordinates are transformed into the predictions by an output module. Notably, all modules are -quiiiauuvari
Internal context encoder Similar to GMN (Huang et al., 2022), we extend EGNN (Satorras et al., 2021) from one single input vector to multichannel coordinates, since each residue is naturally rep- resented by multiple backbone atoms. Suppose in layer I the node features are (h. (1) |i 1,2, ..., n) and the coordinates are (2 1, ..., n). We denote the relative coordinates between node i and j as Z z 2 Then, the information of each node is updated in the following form.
)
mij om (1
(0.. ,n |1 2 Ir
Ciy),
(1)
(1)
h (40.5)
=ha(h!) Σ JEN((HEE)) mij),
(2)
z (40.5)
z [((E)) Σ JEN
1 oz(m))))
(3)
where (i)EE,)) denotes the neighbors of node regarding the internal connections Ens. || I|I returns the Frobenius norm, and Φm. 6z, Φv, Øz are all Multi-Layer Perceptons (MLPs) (Gardner & Dor- ling, 1998). Basically, mij gathers the E(3)-invariant messages from all neighbors; then used to update h; via da and Z, via oz that is additionally left multiplied with (t) keep the direction information. As En also contains the connections between global nodes in different components, the encoder here actually involves inter-component message passing, although in global sense. We use the superscript (I + 0.5) to indicate the features and coordinates that will be further updated by the external attentive encoder in this layer.
4

---

Published as a conference paper at ICLR 2023
External attentive encoder This module exploits the graph attention mechanism (Veličković et al., 2017) to better describe the correlation between the residues of different components, but different from Veličković et al. (2017), design a novel (33-equiivariant graph attention scheme based on the multichannel scalarization in the above internal context encoder. Formally, we have:
explo, ki))
0ig EJEN(HEE)) expla: key))
(4)
((+11) = 11; (((t0.))
+ Σ JEN(EEE) Oij
(5)
z (1+1) z((t,.)) + E EEENTHEL azy Z (14...) (z(Ug),
(6)
where, (i( denotes the neighbors of node defined by the external interactions Eexi qi, kij. and Biy are the query, key, and value vectors, respectively, and agj is the attention weight from
node j to i. Specifically, di = og (1(+t.55) ki Φu 1(2 (1)
(Z (2(110.0)) 2 ((110.6)
hi ((+0.55), and Vig =
1|
()+ all ((3)-invariant, where the functions By. U& O are MLPs. 6 1122 1 1::0.55 hj
(2 ((140.5)
Output module After layers of the alternations between the last two modules, we further con- duct Eq. (1-3) to output the hidden feature h; and coordinates Z,. To predict the probability of each amino acid type, we apply SoftMax on h4 p3 = (oftmax(h:). where pt E IR= the predicted distribution over all amino acid categories.
desirable property of MEAN is that it is -quivvaaurrriant. We summarize it as a formal theorem below, with the proof deferred to Appendix E.
Theorem 1. We denote the translation process by MEAN as ((ps 2.)) [[th then f is equivvarra. In other words, for each transformation g E E(3), we have {
(0) 2,00)
Jiev), )
Z.) Deve f(th 9- 2 Jiev), where the group action instantiated as g Z Oz for orthogonal transformation O E R3x3 and Z = Z + for translation transformation E R3
3.3 PROGRESSIVE FULL-SHOO DECODING
Traditional methods (such as RefineGNN (Jin et al., 2021)) unravel the CDR sequence in an au- toregressive way: generating one amino acid at one time. While such strategy is able to reduce the generation complexity, it inevitably incurs expensive computing and memory overhead, and will hinder the training owing to the vanishing gradient for long CDR sequences. It also acumulates errors during the inference stage. Here, thanks to the rich expressivity of MEAN, we progressively generate the CDRs over T iterations (T' is much smaller than the length of the CDR sequences), and in each iteration, we predict the amino acid type and 3D coordinates of all the nodes in Vc at once. We call our scheme as full-shot decoding distinguish it from previous autoregressive approaches.
To be specific, given the CDRs' amino acid distribution and conformation {x.2, 2 heve from iteration t, first update the embeddings of all nodes: h/ Eド=1p 1 (j)s3, Vi Vc. where
p1 ( (j) returns the probability for the j-th class and s, is the corresponding learnable embedding as defined before. Such weighted strategy leads to less accumulated error during inference compared to the maximum selection counterpart. We then replace the CDRs with the new values (h/ ( Jeve and denote the new graph g(++1), The edges are also constructed dynamically according to the new graph. We update the next iteration as (t+1) (1+1) Jieve = MEAN(
For sequence prediction, we exert supervision for each node at each iteration:
Lzeg
Σ (pl Pp),
T
Vcl
iEVO
where toe denotes the cross entropy between the predicted distribution p and the true one pc- For structure prediction, we only exert supervision on the output iteration. Since there are usually noises in the coordination data, we adopt the Huber loss (Huber, 1992) other than the common MSE
5

---

Published as a conference paper at ICLR 2023
loss to avoid numerical instability (further explanation can be found in Appendix F):
Estruct
1
Σ iEVo
fmuber (2 Z.).
(8)
[Vol
where 2, is the label. One benefit of the structure loss is that it conducts directly in the coordinate space, and it is still ((3))-invrriant as model is -q3)evqauvr This is far more efficient than the loss function used in RefineGNN (Jin et al., 2021). To ensure invariance, RefineGNN should calculate over pairwise distance and angle other than coordinates, which is tedious but necessary since it can only perceive the input of node and edge features after certain invariant transformations. Finally, we balance the above two losses with À to form C Ceeq + ALstruct.
EXPERIMENTS
We assess model on the three challenging tasks: 1. The generative task the Structural Anti- body Database (Dunbar et al., 2014) in 4.1; 2. Antigen-binding CDR-H3 design from a curated benchmark of 60 diverse antibody-antigen Antigen-antibody binding affinity optimization on Structural Kinetic and Energetic database of Mu- tant Protein Interactions (Jankauskaité al., 2019) in 4.3. We also present promising pipeline to apply our model in scenarios where the binding position is unknown in $ 4.4.
complexes (Adolf-Bryfogh) et al., 2018) in 4.2; 3.
Four baselines are selected for comparison. For the first one, we use the LSTM-based approach by Saka et al. (2021); Akbar et al. (2022) to encode the context of the heavy chain and another LSTM to decode the CDRs. We implement the cross attention between the encoder and the decoder, but utilize the sequence information only. Built upon LSTM, we further test C-LSTM to consider the entire context of the antibody-antie token. RefineGNN (Jin et al., 2021) is related to our method as it also considers the 3D geometry for antibody generation, but distinet from our method it is only E(3)-invariant and autoregressively generate the amino acid type of each residue. Since its original version only models the heavy chain, we extend it by accommodating the whole antibody -nttige complex, which is denoted as C-RefineGNN; concretely, each component is identified with special token in the sequence and dummy node in the structure. As denoted before, we term our model as MEAN. We train each model for 20 epochs and select the checkpoint with the lowest loss on the validation set for testing. We use the Adam optimizer with the learning rate 0.001. For MEAN, we run 3 iterations for the progressive full-shot decoding. More details are provided in Appendix D. For LSTM and RefineGNN, we borrow their default settings and source codes for fair comparisons.
complex, where each component separated by a special
4.1 SEQUENCE AND STRUCTURE MODELING
For quantitative evaluation, we employ Amino Acid Recovery (AAR), defined as the overlapping rate between the predicted ID sequences and ground truths, and Root Mean Square Deviation (RMSD) regarding the 3D predicted structure of CDRs. Thanks to its inherent equivariance, our model can directly calculate RMSD of coordinates, unlike other baselines that resort to the Kabsch technique Kabsch (1976) to align the predicted and true coordinates prior to the RMSD computation. Our model requires each input complex to be complete (consisting of heavy chains, light chains, and antigens). Hence, we choose 3,127 complexes from the Structural Antibody Database (Dun- bar al., 2014, SAbDab) and remove other illegal datapoints that lack light chain or antigen. All selected complexes are renumbered under the IMGT scheme (Lefranc et al., 2003), As suggested by Jin al. (2021), we split the dataset into training, validation, and test sets according to the clus- tering of CDRs to maintain the generalization test. In detail, for each type of CDRs, we first cluster the sequences via MMseqs2 (Steinegger & Söding, 2017) that assigns the antibodies with CDR se- quence identity above 40% to the same cluster, where the BL.SUM62 substitution matrix (Henikoff & Henikoff, 1992) is adopted to calculate the sequence identity. The total numbers of clusters for CDR-HI, CDR-H2, and CDR-H3 are 765, 1093, and 1659, respectively. Then we split all clusters into training, validation, and test sets with a ratio of 8:1:1. We conduct 10-fold cross validation to obtain reliable results. Further details provided in Appendix A.
Results Table (Top) demonstrates that our MEAN significantly surpasses all other methods in terms of both the 1D sequence and 3D structure modeling, which verifies the effectiveness of MEAN

---

Published as a conference paper at ICLR 2023
Table 1: Top: 10-fold cross validation mean (standard deviation) for 1D sequence and 3D structure modeling on SAbDab ($4.1), Bottom: evaluations under the setting of Jin et al. (2021), denoted with superscript
CDR-HI
CDR-H2
CDR- H3
Model
AAR
RMSD
AAR
RMSD
AAR
RMSD
LSTM
50.98=5.200%
28.50=+1.555
15.69=0,91%%
C-LSTM
40.93-55.41% %%
1..48
RefineGNN
39.40-55.66%
3.22++029
37.06++3.095
3.64+0.40
21.13+11,595
0.55
C-RefineGNN
33.19=2.99%
3.25
33.53
3.69110.56
18. 18.8811.379%
6.22
MEAN
58.29-7.277% %
0.98+0.16
47.15+33.9%
0.95
36.38 36.38+3.088 +3.08% %
0.21
LSTM
28.02%
24.39%
18.92%
RefineGNN' 30.07%
0.97
27.70%
0.73
27.60%
2.12
MEAN 62.78%
0.94
52.04%
0.89
39.87%
2.20
in modeling the underlying distribution of the complexes. When comparing LSTMMRRfinGGNN with is observed that further taking the light chain and the antigen into account sometimes leads to even inferior performance. This observation suggests it will cause a negative effect if the interdependence between CDRs and the extra input components is not correctly revealed. On the contrary, MEAN is able to deliver consistent improvement when enriching the context, which will be demonstrated later in 5. We also compare MEAN with LSTM and RefineGNN on the same split as RefineGNN (Jin et al., 2021), denoted as MEAN', LSTM', and RefineGNN", respectively. Specifically, both LSTM" and RefineGNN" are trained on the full training set, while our MEAN only accesses its subset composed of complete complexes, which is approximately 52% of the full dataset. All three models are evaluated on the same subset the test set for fair comparisons. Even so, MEAN still outperforms the other two methods remarkably in terms of AAR and achieves comparable results regarding RMSD in Table (Bottom), validating its strong generalization ability.
4.2 ANTIGEN-BINDING CDR-H3 DESIGN
We perform fine-grained validation on designing CDR-H3 that binds to a given antigen. In addition to AAR and RMSD, we further adopt TM-score (Zhang & Skolnick, 2004; Xu & Zhang, 2010), which calculates the global similarity between two protein structures and ranges from to 1, to evaluate how well the CDRs fit into the frameworks. We use the official implementation" to cal- culate TM-score. For autoregressive baselines, we follow Jin et al. (2021) to alleviate the affects of accumulated errors by generating 10,000 CDR-H3s and select the top 100 candidates with the lowest PPL for evaluation. Notably, our MEAN reduces accumulated errors significantly by the progressive full-shot decoding strategy, therefore can directly generate the candidates. We also in- clude RosettaAD for comparison. We benchmark all methods with the 60 diverse complexes carefully selected by Adolf Mlolf-Bryfogle et al. (2018) (RAbD). The training is still conducted on the SAbDab dataset used in $ 4.1, but we eliminate all antibodies from SAbDab whose CDR-F. -BR share the same cluster as those in RAbD to avoid any potential data leakage. We then divide the remainder into the training and validation sets by a ratio of 9:1. The numbers of clusters/antibodies are 1,443/2,638 for training and 160/339 for validation.
et al., 2018). widely used baseline of conventional approach,
Results As shown in Table 2, MEAN outperforms all baselines by a large margin in terms of both AAR and TM-score. Particularly on TM-score, the value by MEAN approaches above 0.99, implying that the designed structure is almost the same as the original one. To better show this, we visualize an example Figure 3, where the generated fragment by MEAN almost overlaps with the ground truth, while the result of RefineGNN exhibits an apparent bias.
4.3 AFFINITY OPTIMIZATION
It is crucial to optimize various properties like binding affinity of antibodies for therapeutic purposes. This can be formulated as search problem over the intrinsic space of generative models. In our case, we jointly optimize the sequence and structure of CDR-H3 to improve the binding affinity of
htttps://haagggneee org wor.ooorg//////o score/

---

Published as a conference paper at ICLR 2023
A
B
COR HB
Cround murt: AMANAGY
MRSHONNTAPPWAA Outinized PAYPENENMMUROO
MEANI ANGOGIY
BetreeRRe COYFE"
Figure 3: (A) The structure of two aniigen-binding CDR-H3s (PDB: lic7) designed by MEAN (left, RMSD=0,49) and RefineGNN (right, RMSD=3.04), The sequences are also provided in the annotations. (B) The antibody-antigen complex structure after affinity optimization (PDB: 2vis, AAG= = -5.13,, with the original and optimized sequences displayed in the annotations.
any given antibody-antigen complex. For evaluation, we employ the geometric network from Shan al. (2022) to predict the change in binding energy (AAG) after optimization. Particularly, we leverage the official checkpoint that is trained on the Structural Kinetic and Energetic database of Mutant Protein Interactions V2.0 (Jankauskaité et al., 2019, SKEMPI V2.0). AAG calibrated under the unit of kcal/mol, and lower AAG indicates better binding affinity. Since all the methods model backbone structures only, we use Rosetta (Alford et al., 2017) to do sidechain packing before affinity prediction. To ensure the expected generalizability, we select total of 53 antibodies from its training set (i.e. SKEMPI V2.0) for affinity optimization. Besides, we split SAbDab pre-processed by the strategy in 4.1) into training and validation sets in ratio of 9:1 for pretraining the model.
Following Jin et al. (2021), we exploit the Iterative Target Augmentation (Yang et al., 2020, ITA) algorithm to tackle the optimization problem. Since the original algorithm is designed for discrete properties, we adapt it for compatibility with our affinity scorer of continuous values. Please refer to Appendix B for detailed description of the adaption. During the process, we discard any unrealistic candidate with PPL above 10 in accordance with Jin et al. (2021). It is observed that our model learns the constraints of net charge, motifs, and repeating amino acids (Jin et al., 2021) implicitly, therefore we do not need to impose them on our model explicitly.
Table 2: Left: Amino acid recovery (AAR), TM-score and RMSD for CDR-H3 design on RAbD benchmark ($ 4.2). Middle: Average affinity change after optimization textsection 4.3). Right: Average CDR lengths and speedups by our full-shot decoding compared to the iterative-refinmment decoding.
Model
AAR
TM-score
RMSD
Model
AAG
length
RusettaAD
22.50%
0.9435
5.52
Random
+1.52
CDR-HI
CDR-H2
7.5
LSTM
22.36%
LSTM
-1.48
CDR-H3
14.1
C-LSTM
22.18%
C-LSTM
-1.83
speodup (train infer))
RefineGNN
29.79%
0.8303
7.55
RefineGNN
-3.98
C-RefineGNN
28.90 98.90%%
0.8317
7.21
C-RefineGNN
-3.79
2 1.5x
CDR-H2
2.1x/ 1.6x
MEAN
36.77%
0,9812
1.81
MEAN
-5.33
CDR-H3
4.1x/18x
Results As shown Table 2 (middle), our MEAN models achieve obvious progress towards dis- covering antibodies with better binding affinity. This further validates the advantage of explicitly modeling the interface with MEAN. Moreover, we provide the predicted AAG of mutating the CDRs random sequences, denoted as Random, for better interpretation of the results. We provide further interpretation in Appendix C. We also provide a visualization example in Figure 3 (B), which indicates our MEAN does produce novel CDR-H3 sequence/structture, with improved affinity.
4.4 CDR-H3 DESIGN WITH DOCKED TEMPLATE
We further provide a possible pipeline to utilize our model in scenarios when the binding complex is unknown. Specifically, for antigens from RAbD (Ayolf-Bryfogle et al., 2018), we aim at generating binding antibodies with high affinity. To this end, we first select an antibody from the database and remove its CDR-H3, then we use HDOCK (Yan et al., 2017) to dock it to the target antigen to obtain a template of antibody-antigen complex. With this template, we employ our model to
httpps:// thub. Hel htt.:/o.eeeeeeieoteeeeeeeeee www.koeeeeeeeeeeeeeooeeeee nng www.ddddd-ccto.
8

---

Published as a conference paper at ICLR 2023
generate antigen-binding CDR-H3s in the same way 4.2. To alleviate the risk of docking inaceuracy, we compose 10 such templates for each antigen and retain the highest scoring one in the subsequent generation. We first refine the generated structure with OpenMM (Eastman et al., 2017) and Rosetta (Alford et al., 2017), and then use the energy functions in Rosetta to measure the binding affinity. The comparison of the affinity distribution between the generated antibodies by our method, those by C-RefineGNN, and the original ones in RAbD is shown in Figure (B). Obviously, the antibodies designed by our MEAN exhibit higher predicted binding affinity. We also present tightly binding example in Figure 4 (A).
0.000
MSSH
0910
Heavy
Light
0.90
CDR-H3: ARRGRGGGWFDY
0.00
Figure 4: (A) CDR-H3 designed by MEAN based a docked template targeting the antigen from PDB 5b8c (Affintt=-59.66) (B) The affinity distribution of the antibodies.
5 ANALYSTS
We test if each proposed technique is necessary MEAN. Table shows that the removal of either the global nodes or the attention mechanism induces performance detriment. This is reasonable since the global nodes transmit information within and between components globally, and the atten- tive module concentrates on the local information around the interface of different components. In practice, the attentive module also provides interpretability over the significance pairwise residue interactions, illustrated in Appendix I. In addition, it is observed that only using the heavy chain weakens the performance apparently, and fails to derive feasible solution for the affinity optimiza- tion task, which empirically supports the necessity of inputting antigens and light chains in MEAN. Moreover, we implement a variant of MEAN by replacing the progressive full-shot decoding with the iterative refinement operation used in RefineGNN, whose performance is worse than MEAN. As discussed before, our full-shot decoding is much more efficient than the iterative refinement process, since the number of iterations in MEAN is 3 which is much smaller than that of the refinement-based variant. As reported in Table 2 (right), our method speeds up approximately 2 to 5 times depend- ing on the lengths of the CDR sequences. We also analyze the complexity. MEAN injected with randomness, and the progressive decoding process in Appendix G, H. and J, respectively.
Table 3: Ablations of MEAN.
SAbDab (CDR-H3)
RAbD TM-score
SKEMPI
Model
AAR
RMSD
AAR
RMSD
AAG
w/o glohal node
33.89=6.81%
2.62+11.03
35.07%
0.9798
1.81
-3.68
w/o attention
35.89-33.44%
2.24++0.31
36.56%
0.9745
1.95
failed
heavy chain only
33.62-16.20%
2.23-10.17
35.82%
0.9728
1.97
failed
iterative refinement
2..85
3.61
35.19%
0.9629
3.20
-5.19
MEAN
36.38
13008%
2.21
36.77%
0.9812
1.81
-5.33
CONCLUSION
In this paper we formulate antibody design as translation from the the entire context of antibody- antigen complex to versatile CDRs. We propose multi-channel equivariant attention network (MEAN) to identify and encode essential local and global information within and between dif- ferent chains. We also propose progressive full-shot decoding strategy for more efficient and precise generation. Our model outperforms baselines by a large margin in terms of three generation task in- cluding distribution learning ID sequences and 3D structures, antigen-binding CDR-H3 design, and affinity optimization. Our work presents insights for modeling antibody- antigen interactions in further research.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work jointly supported by the Vanke Special Fund for Public Health and Health Disci- pline Development of Tsinghua University, the National Natural Science Foundation of China (No. 61925601, No. 62006137), Guoqiang Research Institute General Project Tsinghua University (No. 202IGQG1012) Beijing Academy of Artificial Intelligence, Beijing Outstanding Young Sci- entist Program (No. BJW2YJH002019100000999
REPRODUCIBILITY
The codes for our MEAN available httt:p://ggthubb.co///ttttt/
REFERENCES
Jared Adolf-Bryfolle, Oleks Kalyuzhniy, Michael Kubitz, Brian D Weitzner, Xiaozhen Hu, Yumiko Adachi, William R Schief, and Roland L Dunbrack Jr. Rosettaantibodydesiiu (rabd): A general framework for computational antibody design. PLoS computational biology, 14(4):e10061) 12, 2018.
Rahmad Akbar, Philippe A Robert, Cédric R Weber, Michael Widrich, Robert Frank, Milena Pavlović, Lonneke Scheffer, Maria Chernigovskaya, Igor Snapkov, Andrei Slabodkin, et al. In silico proof of principle of machine learning-basee based antibody design at unconstrained scale. In Mabs, volume 14, pp. 2031482.
Taylor & Francis, 2022.
Rebecca F Alford, Andrew Leaver-Fay, Jeliazko R Jeliazkov, Matthew J O' O'Meara, Frank P DiMaio, Hahnbeom Park, Maxim V Shapovalov, P Douglas Renfrew, Vikram K Mulligan, Kalli Kappel, et al. The rosetta all- alom energy function for macromolecular modeling and design. Journal of chemical theory and computation, 13(6):3031 3000:3031-30444 -048, 2017.
Ethan C Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M Church. Unified rational protein engineering with sequence- -aaeed deep representation learning. Nature methods, 16( (6(12):1315-1)) 1322, 2019.
Dror Baran, M Gabriele Pszolla, Gideon D Lapidoth, Christoffer Norn, Orly Dym, Tamar Unger, Shira Albeck, Michael D Tyka, and Sarel J Fleishman. Principles for computational design of binding antibodies. Proceedings of the National Academy of Sciences, 1 1990410:19900-10055 2017.
Koli Basu, Evan M Green, Yifan Cheng, and Charles S Craik. Why recombinant antibod- ies benefits and applications. Current opinion in biotechnology. 60: 10:153-1583 2019.
Yue Cao, Payel Das, Vijil Chenthamarakshan, Pin- Yu Chen, Igor Melnyk, and Yang Shen. Fold2seq: A joint sequence (1d)-fold (3d) embedding based generative model for protein design. In Inter- national Conference on Machine Learning, pp. 1261-1271. PMLR, 2021.
Pinak Chakrabarti and Joel Janin. Dissecting wrotein-protein recognition sites. Proteins: Structure, Function, and Bioinformatics, 4773):334-343, 2002.
James Dunbar, Konrad Krawezyk, Jinwoo Leem, Terry Baker, Angelika Fuchs, Guy Georges, Jiye Shi, and Charlotte M Deane. Sabdab: the structural antibody database. Nucleic acids research, 42(D1):D1140-DI 146. 2014.
Peter Eastman, Jason Swails, John D Chodera, Robert T McGibbon, Yutong Zhao, Kyle A Beauchamp, Lee-Ping Wang, Andrew C Simmonett, Matthew P Harrigan, Chaya D Stern, et al. Openmm 7: Rapid development of high performance algorithms for molecular dynamics. PLoS computational biology, 13(7)xe 2017.
Sharon Fischman and Yanay Ofran. Computational design of antibodies. Current opinion in struc- fural biology, 51:156-162, 2018.
10

---

Published as a conference paper at ICLR 2023
Matt W Gardner and SR Dorling. Artificial neural networks (the multilayer perceptron) review of applications the atmospheric sciences. Atmospheric environment, 32(14-15):2627-26566 1998.
Niklas Gebauer, Michael Gastegger, and Kristof Schütt. Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules. Advances in Neural Information Processing Systems, 32, 2019.
Jordan Graves, Jacob Byerly, Eduardo Priego, Naren Makkapati, S Vince Parish, Brenda Medellin, and Monica Berrondo. A review of deep learning methods for antibodies. Antibodies, 12, 2020.
Jiaqi Han, Yu Rong, Tingyang Xu, and Wenbing Huang. Geometrically equivariant graph neural networks: A survey, arXiv preprint arXiv:2202.07223. 2022.
Pernille Haste Andersen, Morten Nielsen, and OLE Lund. Prediction of residues in discontinuous b-cell epitopes using protein 3d structures. Protein Science, 15( 2006.
Steven Henikoff and Jorja G Henikoff. Amino acid substitution matrices from protein blocks. Pro- ceedings of the National Academy of Sciences, 89(22): (1122):10915-00113 1992.
Wenbing Huang, Jiaqi Han, Yu Rong, Tingyang Xu, Fuchun Sun, and Junzhou Huang. Equivariant graph mechanics networks with constraints. arXiv preprint 2rXiv:2203.06444,, 2022.
Peter Huber. Robust estimation of a location parameter. In Breakthroughs in statistics, pp. 492- 518. Springer, 1992.
John Ingraham, Vikas Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. Advances in Neural Information Processing Systems, 32, 2019.
Justina Jankauskaité, Brian Jiménez-Gare García, Justas Dapkũnas, Juan Fernández-Resd and lain H Moal. Skempi 2.0: an updated benchmark of changes in protein-protein binding energy, kinetics and thermodynamies upon mutation. Bioinformatics, 35(3):462-48) 469, 2019.
Wengong Jin, Jeremy Wohlwend, Regina Barzilay, and Tommi Jaakkola. Iterative refinement graph neural network for antibody sequence sequence-structure co-design. arXiv preprint arXiv:2110.0462. 2021.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Antibody-antigen docking and design via hierarchical structure refinement. In International Conference on Machine Learning, pp. 10217- 10227. PMLR, 2022.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873);583-588)) 2021.
Wolfgang Kabsch. A solution for the best rotation to relate two sets of vectors. Acta Crystallo- graphica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography, 32 (5):922-99) 223,, 1976.
Mostafa Karimi, Shaowen Zhu, Yue Cao, and Yang Shen. De protein design for novel folds using guided conditional wasserstein generative adversarial networks. Journal of chemical infor- mation and modeling, 60(12):5667-561, 2020.
Johannes Klicpera, Janek GroB, and Stephan Günnemann. Directional message passing for molec- ular graphs. arXiv preprint arXiv:2003.033
2020.
Daisuke Kuroda, Hiroki Shirai, Matthew P Jacobson, and Haruki Nakamura. Computer-aided anti- body design. Protein engineering, design & selection, 25(10)5077552) 2012.
Gideon D Lapidoth, Dror Baran, Gabriele M Pszolla, Christoffer Nor, Assaf Alon. Michael D Tyka, and Sarel J Fleishman. Abdesign: A algorithm for combinatorial backbone design guided by natural conformations and sequences. Proteins: Structure, Function, and Bioinformatics, 83 (8)-1385-1406 2015.
11

---

Published as a conference paper at ICLR 2023
Marie-Paule Lefranc, Christelle Pommié, Manuel Ruiz, Véronique Giudicelli, Elodie Foulquier, Lisa Truong, Valérie Thouvenin- Contee, and Gérard Lefranc. Imgt unique numbering for im- munoglobulin and cell receptor variable domains and ig superfamily v-like domains. Develop- mental & Comparative Immunology, 27(1):55-77, 2003.
Tong Li, Robert J Pantazes, and Costas D Maranas. Optmaven-a new framework for the novo design of antibody variable region models targeting specific antigen epitopes. PloS one, 9(8): e105954, 2014.
Ge Liu, Haoyang Zeng, Jonas Mueller, Brandon Carter, Ziheng Wang, Jonas Schilz, Geraldine Horny, Michael E Birnbaum, Stefan Ewert, and David K Gifford. Antibody complementarity determining region design using high-capacity machine learning. Bioinformatics, 36(7):2126 2133, 2020.
Yi Liu, Limei Wang, Meng Liu, Yuchao Lin, Xuan Zhang, Bora Oztekin, and Shuiwang Ji. Spherical message passing for 3d molecular graphs. International Conference on Learning Representa- fions, 2021.
Omri Puny, Matan Atzmon, Heli Ben-Hamu, Edward Smith, Ishan Misra, Aditya Grover, and Yaron Lipman. Frame averaging for invariant and equivariant network design. arXiv preprint arXiv:2110.03336, 2021.
GN Ramachandran, C Ramakrishnan, and V Sasisekharan. Stereochemistry of polypeptide chain configurations. Journal of Molecular Biology, 7(1):95-99, 1963.
Jeffrey Ruffolo, Jeffrey J Gray, and Jeremias Sulam. Deciphering antibody affinity maturation with language models and weakly supervised learning. arXiv, 2021.
Jeffrey A Ruffolo, Lee-Shin Chu, Sai Pooja Mahajan, and Jeffrey J Gray. Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodies. bioRxiv, 2022.
Koichiro Saka, Taro Kakuzaki, Shoichi Metsugi, Daiki Kashiwagi, Kenji Yoshida, Manabu Wada, Hiroyuki Tsunoda, and Reiji Teramoto. Antibody design using Istm based deep generative model from phage display library for affinity maturation. Scientific reports, 11(1):1-13, 2021.
Vićtor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural net- PMLR, 2021.
works. In International Conference on Machine Learning, pp.
Sisi Shan, Shitong Luo, Ziqing Yang, Junxian Hong. Yufeng Su, Fan Ding, Lili Fu, Chenyu Li, Peng Chen. Jianzhu Ma, et al. Deep learning guided optimization of human antibody against sars-cov-2 variants with broad neutralization. Proceedings of the National Academy of Sciences, 119(11):
e2122954119, 2022.
Jung-Eun Shin, Adam J Riesselman, Aaron W Kollasch, Conor McMahon, Elana Simon, Chris Sander, Aashish Manglik, Andrew Kruse, and Debora Marks. Protein design and variant prediction using autoregressive generative models. Nature communications, 12(1):1-11, 2021.
Martin Steinegger and Johannes Söding. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechmology. 35(1 10:106-1 2017.
Kathryn E Tiller and Peter M Tessier. Advances in antibody design. Annual review of biomedical engineering, 17:191-216, 2015.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint 17 10..0/00..0.0..... 2017.
Jinrui Xu and Yang Zhang. How significant is a protein structure similarity with tm-score= 0.5? Bioinformatics, 26(7):889-895, 2010.
Changhui Yan, Feihong Wu, Robert L Jernigan Drena Dobbs, and Vasant Honavar. Characterization of protein-protedd interfaces. The protein journal, 27(1):59-70, 2008.
Yumeng Yan, Di Zhang, Pei Zhou, Botong Li, and Sheng- You Huang. Hdock: web server for protein pootein and protein- dna/mna docking based on a hybrid strategy. Nucleic acids research, 35651):W365-W33. 2017.
12

---

Published as a conference paper at ICLR 2023
Kevin Yang. Wengong Jin, Kyle Swanson, Regina Barzilay, and Tommi Jaakkola. Improving molec- ular design by stochastic iterative target augmentation. In International Conference on Machine Learning. pp. 10716-10726. PMLR, 2020.
Yang Zhang and Jeffrey Skolnick. Scoring function for automated assessment of protein structure template quality. Proteins: Structure, Function, and Bioinformatics, 57(4):702-710, 2004.
13

---

Published as a conference paper at ICLR 2023
A DETAILS OF SEQUENCE AND STRUCTURE MODELING
10-fold dataset splits We provide the number of clusters and antibodies in each fold of our 10-fold cross validation. When selecting fold for testing. we use fold for validation (for fold as the test set, we use fold 10 for validation) and the union of other folds for training.
Table 4: Statistics of each fold of 10-fold cross validation.
CDR-H H1
CDR-H2
CDR-H3
cluster
antibody
cluster
antibody
cluster
antibody
fold
77
299
110
237
166
317
fold 2
77
260
110
343
166
301
fold 3
77
363
110
288
166
348
fold 4
77
286
109
315
166
271
fold 5
77
195
109
419
166
279
fold
76
241
109
217
166
345
fold 7
76
427
109
266
166
306
fold 8
76
326
109
321
166
320
fold 9
76
210
109
374
166
309
fold
76
520
109
347
165
331
Total 765
3,127
1,093
3,127
1659
3,127
RefineGNN paper setting In Table 1, we also compare LSTM, RefineGNN, and MEAN on the same split used in the paper of RefineGNN (Jin et al., 2021) and the models are denoted LSTM", RefineGNN", and MEAN" Here we provide the sizes of training/validation/ees set as well as their subsets of complete complexes in Table 5. We use the subsets train MEAN", which is only about 52% of the full sets, and all three models are evaluated on the subset of the test set for fair comparison. For RefineGNN*, directly the official checkpoints provided by Jin et al. (2021) for evaluation.
Table 5: Number of antibodies in the training/validatiionness set of Jin et al. (2021) well as their subsets of complete complexes.
CDR-HI
CDR-H2
CDR-H3
full
subset
full
subset
full
subset
training
4,050
2,131
3,876
2,035
3,896
2,051
validation
359
170
483
241
403
227
test
326
184
376
209
437
207
Total
4,735
2,485
4,735
2,485
4.735
3,127
B ITERATIVE TARGET AUGMENTATION ALGORITHM FOR AFFINITY OPTIMIZATION
Notably, the original algorithm is designed for discrete properties, while in $ 4.3 the affinity is continuous. Therefore, we adapt ITA for compatibility with our affinity scorer as follows. The core that we maintain a list of high-quality candidates for each antibody to be optimized during the ITA process. each iteration, we produce candidates for each antibody and sort them together with the candidates in the high-quality list according the scores. Then we retain the top-k candidates while dropping all others. The above process goes through all the candidates before entering the next iteration. is expected that the distribution of the generated antibodies will move towards the higher-affinity landscape. In particular, we run 20 iterations for each pretrained generative model by setting C' 50 and k 4.
the current list
C THE PREDICTOR USED IN AFFINITY OPTIMIZATION
According to Shan et al. (2022), the output the AAG predictor has been calibrated under the unit of kcal/mol, and the correlation between the predicted AAG and the experimental value is 0.65
14

---

Published as a conference paper at ICLR 2023
the test set. It means if the predicted AAG is then the binding affinity is decreased by kcal/mol under a correlation of 0.65.
D EXPERIMENT DETAILS AND HYPERPARAMETERS
For models incorporating antigen, we select 48 residues closest to the antibody in terms of the alpha carbon distance as the antigen information. We conduct experiments CPU cores and 10 GeForce RTX 2080 Ti GPUs. Models using itcrative refinement decoding have intensive GPU memory requirements and are therefore trained with the data-parabl of Pytorch on 6 GPUs. Other models only need GPU to train. We use Adam optimizer with Ir 0.001 and decay the learning rate by 0.95 every epoch. The batch size is set to be 16. All models are trained for 20 epochs and the checkpoint with the lowest loss on the validation set is selected for testing. The training strategy is consistent across different experiments in the paper, and the learning rate of ITT finetuning is also set to 0.001. Furthermore, we provide the hyperparameters for baselines and our MEAN in Table For the RosettaAD (Alford et al., 2017) used in $ 4.2, we adopt the denovo design protocol initializing with random CDRs presented in its manual".
machine with 56
framework
Table 6: Hyperparameters for each model.
hyperparameter
value
description
shared
vocab size
25
There are 21 categories of amino acids in cukaryote, plus the 4 special tokens for heavy chain, light chain, antigen and mask. Dropout rate.
dropout
0.1
(C-)LSTM
hidden size n_layers
256
Size of the hidden states in LSTM. Number of layers of LSTM.
2
Re)feenneNNN
hidden size
256
Size of the hidden states in the message passing network (MPN). Number of neighbors to include in k- k-nearest neighbor (KNN) graph.
k_neighbors
block_size
4
Number of residues in a block.
n.layers
Number layers in MPN.
num
16
Number of RBF kernels to consider.
MEAN
embed.ss size
64
Size of trainable embeddings for each type of amino acids. Size hidden states in MEAN.
hidden_siee
128
layerrs
Number of layers in MEAN.
alpha
0.8
The weight to balance sequence loss and structure loss. Number of iterations for progressive full-shot decoding.
n_iter
For training our MEAN on the split in RefineGNN paper setting, we set alpha 0.6, batch size to be 8, and total training epochs to be 30.
E PROOF OF THEOREM 1
Our MEAN satisfies -ruueqqurae as demonstrated in Theorem 1:
Theorem 1. We denote the translation process by MEAN as ((ps, 2.))ieve S(EN
(0) -2(0) Jiev), then f is E(3)-qquivariant. In other words, for each transformation E E(3), we have f((h) (0)
,9-
Z OZ for orthogonal transformation Ó E 333 and Z = Z + t for translation transformation E R3
( hev) g f((h.)) (0) z Hev)), where the group action is instantiated
In the following, we denote f'(Z) = zT Z Z Z F and the orthogonal group as 0(3) (0 333 0² = 00 Ia}. Prior to the proof, we first present the necessary lemmas below:
'nttp:://wwr commonss. wwtps//oo/o///o///o/e/eeeeeee/ee ant intibody//caettaann ibod besign
cat iaation_ie ceriinnddooee con_dooumentttioonn
15

---

Published as a conference paper at ICLR 2023
Lemma 1. The function f RR3xm - mmxm is invariant 0(3). Namely, Vgg 0(3), have fg Z) = f' (Z), where g = OZ.
Proof. Vg 0(3), have g. oZ. Therefore, we can derive the following equations:
(OZ)
zZOTOZ
56 2) [OZ)
(9)
OZ
z
ZF =f(Z),
(10)
which conclude the invariance of f' on 0(3).
Lemma 2. We denote the internal context encoder of layer as {(h z z(1+0.6)
((1(400.5)
)ev
(i(((( (0) 2 )ivv), then in E3))-qquivariant.
Proof. Vg E(3), have g Z Oz where O 0()) and t E R Therefore have the relative coordinates after transformation as g Z (1) ~ = OZ ij According to Lemma
during the propagations of our internal context encoder, we have E(3) invariint messages:
mij 2m ( .ng
= pm (
(11)
Then it is easy to derive that the hidden states are E(3)-invariant and the coordinates are E(3)- equivariant as follows:
A ((+0.5) =di(hi) E EjENCIEN ma)):
(12)
Oz (+0.5) ( +t= =0|, + [N((iEin) Σ ESENTEE oz +
(13)
= =0Z 11) +t) + [((i)m)) E EyEN(HE.) (OZ. )oz(mss),
(14)
Therefore we have g om ( z (1))
Hevv) (m 99 )EvV)
Lemma 3. We denote the external attentive encoder as
((h(t)) (+11) z ((+1))
Hev
(ea((((((+00))) ((+.5)) then da equivaaaata
Proof. Similar to the proof of Lemma 2, we first derive that the query, key and value vectors -i3)-inrai:ntt:
9: oy (1/ (1(400055)
(15)
), (4005)) Viy (0Z (+.0.5) n (1+0.5)) 00' (2 (141.5)) n (44..5))
ki dz (f (40.5) (1 h (((ta))) -x (1)55))
(16)
(17)
which directly lead to the F(3) -invarinnee of attention weights orj EjENCiA expla, kiy)
exp(g]
Again it is easy to derive that the hidden states are E(3) (iv)aranv and the coordinates are E(3) cquivariant as follows:
h (140.5)
Σ ) OijVij.
(18)
(z +t= =Z (++.5) " aig 2
(++.55)
wiyy) + -t
(19)
= =Oz (0..5)
+t)+ + E)ENT(IE)) aij azjOz (1+0.5) Vij-
(20)
Therefore have g dea ((1.))) 2200)z1101 (((+0.5)
((4.5)
hivv (ea(((1) (1+.5))
.g. 2 (1+0.55) Jiev).
((tt0.))))evv).
n
16

---

Published as a conference paper at ICLR 2023
With the above lemmas, are ready to present the full proof of Theorem follows:
Proof. For each layer in MEAN, according to Lemma 2 and 3, we have:
(1+1) {(h . Z ((+1)))
hiv Jiev
(1hhg
(+0.5) (( . z )((+ hev) de (n (00l(thh).. g-- )ev)), (21)
Since pj is obtained by applying softmax on the hidden representations from the output module, which shares the same formula as Jin. and Z; is the same as the coordination from the output module, it is easy to derive:
(0) (pr.g- Zi)eev = f(th: .9- 2 10) Hivv),
(22)
9. g-((( (0) 0 )iEv) (pi.g hev,,
(23)
Therefore we have:
(0) 99 " hivv) g-. f 7z! hev), f(1hg)
(24)
which concludes Theorem
F
HUBER Loss
We use Huber loss (Huber, 1992) for the modeling of coordinations, which is defined as follows:
1xx v) { 0.5 (a -) iʃ |a
y/ < 6, 8 (1= y/ 0.5 5), else
(25)
reads that if the LI norm of [x - y/ is smaller than d, it is MSE loss, otherwise L1 loss. At the beginning of the training, the deviation of the predicted structure and ground truth is large and the L1 term makes the loss less sensitive to outliers than MSE loss. When the training is almost done, the deviation is small and the MSE loss provides smoothness near O. In practice, we find that directly using MSE loss occasionally causes NaN at the beginning of the training while Huber loss leads to more stable training procedure. We set 5 in our experiments.
G COMPLEXITY ANALYSIS
We provide the complexity analysis here. Suppose the numbers of nodes in the antigen, the light chain, and the heavy chain are NA, Nz, and Nu, respectively. The message passing in Eq. (1-6) for each node involves K neighbors Then, the complexity of our algorithm over T iterations becomes O(2LKT(NA + Nr. + NR)), where the number refers the joint computation of the internal and external encoders. Similarly, the complexity of RefineGNN with only heavy chain is O(LKT" N)), where T" denotes the number of iterations, namely, the length of the CDR region. We would like specify these two points:
maximum, and it is performed over L rounds at each iteration.
1. Even if our algorithm contains more nodes than RefineGNN ((NA + NL NH) vs NH), the extra computation overhead does not remarkably count, since the message passing for each node can be parallelly computed in current deep learning platforms (e.g. Pytorch).
2. We apply the full-shot decoding other than the autoregressive mechanism used in Re- fineGNN, hence (T' = in our experiments) is much smaller than 7" (usually larger than 10), implying more efficiency of our method.
H MODELING WITH RANDOMNESS
The current model is deterministic given the same inputs, but in some scenarios, the diversity of samples is required. To tackle this, we inject randomness into our model by adding standard Gaus- sian noises to the initialized coordinates. We denote this model rand-MEAN and evaluate it on the tasks of sequence-structure modeling and untigen-binding CDR-H3 design. The results Table and Table 8 suggest injecting randomness into MEAN has acceptable impacts the performance.
17

---

Published as a conference paper at ICLR 2023
Table 7: 10-fold cross validation mean (standard deviation) for 1D sequence and 3D structure mod- eling on SAbDab ($4.1) of rand-MEAN.
CDR-H1
CDR-H2
CDR-H3
Model
AAR
RMSD
AAR
RMSD
AAR
RMSD
MEAN
58.29+7.27%% 0.98+0.16 0.9810.12
47.15+3..09% %
0.95++0.05 1.18100.0
36.38-3.08 %
2.21++0.16 2.36:10.19
Band-MEAN
56.50:16.44%
44.011...7%%
2..69%...29%
Table 8: Amino acid recovery (AAR), TM-score and RMSD for CDR-H3 design on RAbD bench- mark of raAN-MEAN.
Model
AAR
TM-score
RMSD
MEAN
36.77% 37.30%
0,9812 0.9794
1.81
rand-MEAN
1.81
I ATTENTION VISUALIZATION
In the external attentive encoder, we apply the attention mechanism to evaluate the weights between residues in different components. It will be interesting visualize what patterns these attentions will discover. For this purpose, we extract the attention weights between the antibody and the antigen from the last layer, and check they can reflect the binding energy calculated by Rosetta (Alford et al., 2017). In detail, for each residue in CDR-H3, we first identify the residue in the antigen that contributes the most to its binding energy. Then we calculate the rank of the identified residue according to the attention weights yielded by MEAN. We obtain the relative rank by normalizing it with the total number of antigen residues in the interface. If the attention weights are meaningful, then the resultant rank distribution will be bias towards small numbers; otherwise, they are dis- tributed evenly between 0 and 1. Excitingly, Figure 5 (B) displays that we arrive at the former case, indicating the close correlation between our attention weights and the binding energy calculated by Rosetta. Figure 5 (A) also visualizes an example of attention weights and the corresponding energy map, which shows that their distributions are similar.
Stronagy amseation
1.99
2.6
T C 5
0.75
0.39
0.5
0.33
4niigan Nusidaus
Antigan Masidacs
Ralatiue Rent
Figure 5: (A) The left is the attention weights from the residues in CDR-H3 to those in antigen (PDB: 4ydk). The right is the relative energy contribution of each pair of residues calculated by Rosetta. (B) The density maps of the relative ranks for the most contributing residue pairs.
J How GRAPH CHANGES DURING PROGRESSIVE DECODING
We depict the variations of the density distribution of PPL and RMSD across different rounds in the progressive full-shot decoding in Figure 6. Between different rounds, the distribution of PPI. remains similar, which is expected since we exert supervision with the ground truth in all the rounds. It is beyond expectation that even if we only supervise the predicted coordination of the last round, the distribution of RMSD shifts rapidly towards the optimal direction.
We additionally calculate the recovery rate of edges in the ground truth graph on the test set, in terms of internal edges within each component and external edges across different components. For the internal edges, the recovery rate is 89% in the beginning and 95% in the end; for the external edges, the recovery rate is 1 11% in the beginning and 84% in the end. The results above indicate that the linear initialization is able recover large part of the internal edges but only very small
18

---

Published as a conference paper at ICLR 2023
0.35
Round
Round
0.30
0.25
0.3
Donetg
0.20
0.2
0.20
0.1
0.55
0.00
Figure 6: The density of PPL and RMSD of different rounds in progressive full-shot decoding. The area under each curve integrates to 1.
percentage of external edges, By our model, we can predict major part of both internal and external edges, suggesting the validity of our design.
K LOCAL GEOMETRY
Since both RMSD and TM-score reflect the correctness of global geometry, further provide RMSD of bond lengths and angles to validate the local geometry for Section 4.2. The bond lengths measured in angstroms and the angles consist of the three conventional dihedral angles of the backbone structure, namely é, 1ú w (Ramachandran et al., 1963). The RMSD of angles is imple- mented as the average of RMSD on their cosine values. The results shown in Table 9 indicate that our model still achieves much better performance regarding the local geometry. We find that Re- fineGNN achieves relatively low performance on modeling local geometry. which might be because that its indirect loss on various invariant features cannot ensure atoms in the backbone are equally supervised. For example, the precision of the coordinates of the carboxy carbon relatively low with high RMSD of 10.8, which results in the high error in bond lengths of local geometry.
Table 9: RMSD of bond lengths and dihedral angles of the backbone structure.
Model
Bond lengths Angles(O), w),w)
RefineGNN 10.56
0.456
C-RefineGNN
9.49
0.423
MEAN
0.37
0.302
L FULL ANTIGEN OR ONLY EPITOPE
In this paper, we use the 48 residues closest to the antibody to represent the antigen information. From the perspective of biology, these residues compose the epitope (i.e. the binding position of antibodies) which is usually located by biological expert or detected via certain computational methods (Haste Andersen al.. 2006) beforehand, and they usually provide enough information for designing the antigen-binding antibodies. Theoretically, only the residues close to the antibody will affect the message passing between the antigen and the antibody because we construct edges based on cutoff of C,, distance, therefore the performance should be similar regardless of using the full antigen or only the epitope. Practically, we also explore the influence of exclusion of other residues in the antigen on our model. Specifically, we incorporate full antigen in the experiment of $ 4.2 and present the results in Table 10. As expected, incorporating full antigen results in similar performance to the onitope-only strategy with slight change in AAR and structure modeling.
Table 10: Results of bindggg CDR-H3 design given different forms of antigen information.
Antigen
AAR
TM-score
RMSD
epitope-only
36.77%
0.9812
1.81
full
37.92%
0.9798
2.00
19

---

Published as a conference paper at ICLR 2023
M LIMITATIONS AND FUTURE WORK
Sidechain generation In this paper, we follow the previous settings Jin et al. (2021) and only model the backbone geometry for fair comparisons. However, sidechains also play essential role in the interactions between antigens and antibodies. A potential method to extend our method to incorporate sidechains by replacing the current node feature with the local full-atom graph of each residue. We leave this for future work.
Data augmentation The size of existing data of antibodies still small since is hard to obtain the 3D structures of antibodies in practice. There several possible ways do data augmentation for further improvement. The first to leverage pretrained models on sequences. Since the 1D sequence data of antibodies are much more abundant than 3D data, it is possible embedding model based on ID sequences and inject the pretrained model into our framework. It also promising to conduct the pretraining on general protein data from PDB and then carry out finetuning on the antibody dataset. Another potential way is to use Alphafold to produce pseudo 3D structures from ID sequences for pretraining. We leave the above considerations for future work.
pretrain an
N EXAMPLES
We display more examples of antigen-binding CDR-H3 designed by our MEAN in Figure 7.
CONOO
P2B-2aB, IMSD-H
Gound ARTROVIL DY
Ground buth: ANDSOWAMOY
Genundtratt: ABScOWaR
0OO
FDS-JHE, SNSD
FOB-3NS, RMSD-O.30
P08-2wet RMSD 44
MExe AVRYNGRON
MEAN AROGONSSAMTIY
MEGC ARGFUV
Dround truth TRHDCTIVER
Ground bruth: ANCDHIIEULL.
Sround muth: ARDL EOGP
Figure More examples from antigen-binding CDR-H -DR-H3 design by our MEAN.
20

---

Published as a conference paper at ICLR 2023
O TOWARDS THE REAL-WORLD QUESTION
Note that the efficacy of our pipeline for affinity optimization influenced by the generalizability of the predictor, hence how choose a desirable predictor is vital. As proof of concept, we currently apply the predictor in Shan et al. (2022) for its easy implementation and fast computation. Since our pipeline is general, idation, which can return real affinity but is time-consuming One can also combine the advantages of the learning- based predictor and wetlab validation to improve the generalizability while keeping efficiency, by. for example, choosing only top-k samples and using the wetlab feedback to rectify the predictor, creating a so-called closed loop between "dry computation" and "wet experiment" akin to the pipeline used in Shan al. (2022). As increasing attention has been paid to this domain, believe more and more robust and efficient predictors will emerge in the future.
possible to replace the predictor with other variants, such as wetlab val-
While the pipeline for affinity optimization only addresses narrow need in the field of antibody discovery, we also discuss the potential pipeline for the 'real-world' question here (i.e. generate a binidng antibody given an arbitrary antigen). The 'real-world' question might be decomposed into several components: epitope identification, antibody structure prediction, docking, CDR design, and affinity prediction. Each component itself is challenging and currently promising topic in the community. In $ 4.4, we actually combined the last three components, where we used HDock for global docking form the initial complex, our MEAN for CDR design, and Rosetta for affinity computation. If we further add the components for epitope identification and antibody structure prediction (such as Igfold (Ruffolo et al., 2021; 2022)), we are able to set up an end2end pipeline for antibody discovery: it can output a desirable antibody (ID sequence and 3D structure) for any given antigen target. However, setting up such an end2end pipeline is challenging, as the accumulated errors from the former components will easily make the latter fail. A potential solution is making all components learnable, and tuning them as a whole.
Lastly, proposed model and the proof-of-concepl experiments we implemented will provide valuable clues for future exploration to derive enhanced techniques. With the efforts of all re- searchers in the field, we have reason to believe that this ultimate problem will be solved, perhaps step by step.
21


Reviews:
[{"review_id": "yWKpA8iz8fN", "reviewer": "Reviewer_JzYy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents a novel method for the co-design of 1D sequence and 3D structures of antibody proteins that can bind to the target antigen. The method proposes to include the target antigen and the light chain of the antibody as additional conditions, and designs a novel E(3)-equivariant network to predict the amino acid types and 3D conformations. Experiments show that the proposed method achieves excellent performance.", "summary_of_the_review": "Overall, this paper proposes a novel method for the antibody design problem and demonstrates the advantages of it through solid experiments. Considering the high-quality contribution of this work, I vote for acceptance.", "strength_and_weaknesses": "Strengths:\n(+) This paper gives a novel formulation of the antibody design problem by including the target antigen and the light chain of the antibody as conditions and designs a novel network architecture for this problem. The novelty contribution is significant and enlightening.\n(+) Comprehensive empirical studies and ablation studies are conducted to demonstrate the advantages of the proposed method over prior methods. The experimental design and results are solid.\n(+) The writing of the paper is very clear and well-organized.\n\nWeaknesses:\n(-) (minor) In the section 3.3, it is better to provide more explanations or details about Huber loss and its advantages.\n(-) (minor) In the end of section 3.1, the antibody design is formulated as a prediction problem, i.e., the output and input are one-to-one. In practice, for a specific antigen, is there only one antibody with a unique stable structure that can bind to it? I will appreciate if the authors can explain why the task is considered a prediction problem rather than a generation problem.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "wWpOKXx0Qt", "reviewer": "Reviewer_HYV9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper formulates antibody design as E(3)-equivariant graph translation. Input to the graph contains information about the antigen along with the CDR information in the heavy and light chains.  An E(3) equivariant attention network is developed. This network can output 1D sequence and 3D structure of CDRs.  This network operated in 3D coordinate space while maintaining E(3) equivariance and the full geometry of residues. The learning alternates between an internal context encoder and an external attentive encoder and aims to capture long-range and spatially-complicated interactions. CDRs are generated in multiple rounds but each round updates sequences and structures as a whole. The model is tested on three tasks and compared against four baseline models with promising results.", "summary_of_the_review": "Although the proposed algorithm uses an existing graph architecture as a backbone, new modifications  seems to  make significant improvements over this existing model. The paper is well-written and theoretically sound with strong experimental evidence. ", "strength_and_weaknesses": "Strengths\n+ Progressive fullshot decoding is more scalable and less prone to accumulated errors compared to autoregressive models that generate amino acids one at a time. \n+ Light chain and antigen information can provide additional context and can generate CDRs with high binding affinity. \n+ A novel equivariant attention mechanism was developed to better capture interactions\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "eM_NNyEJQSQ", "reviewer": "Reviewer_tzFw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel method for joint antibody sequence/structure design. The paper introduces / combines multiple innovations:\n\n* Modeling antibody heavy + light chain, along with antigen\n* Joint sequence / structure design\n* Iterative prediction of outputs\n\nWhile some of these have been introduced before, the combination is very interesting and results in compelling results on the downstream task of antibody design.", "summary_of_the_review": "This is a good paper that is a clear candidate for acceptance at ICLR. Methodology is novel, benchmarks are thorough, and results are good. The chosen problems are highly important and have potential for major impact.", "strength_and_weaknesses": "Strengths:\n* Multiple novel contributions.\n    - Modeling all components of the antibody/antigen complex\n    - joint sequence/structure design\n    - iterative prediction of model outputs.\n* Convincing and strong results on multiple downstream benchmarks. Dataset splits are constructed well and baselines (both neural and non-neural) are compared to appropriately.\n* Ablations of model are included to verify claims as to the utility of the different additions.\n* Well written paper. Model and experimental results are well explained and an appropriate level of detail is provided to understand the data split strategy and baselines. \n\nWeaknesses:\nTo be honest, I see no major weaknesses in the paper that must be addressed before publication. Below, I've listed some questions for future work and other minor comments.\n\nQuestions / Future Work:\n- It is interesting that the graph is constructed without full knowledge of the location of all residues - a linear imputation is used for missing residues. It would be nice to analyze this a bit further - how much do residues move from this initialization? How \"accurate\" is the initial graph structure - are there edges missing/present in the initial graph vs. in the \"true\" graph and how much does the graph change over iterations of the iterative refinement?\n- Related to the point above, it seems this constructions works in part because much of the antibody structure is already known. Would it be possible to apply the method to general sequence design, as opposed to loop re-design (e.g. Dauparas et al. 2022, science.org/doi/10.1126/science.add2187)? This would be a breakthrough result if so.\n- Ultimately, in-silico evaluations for protein design only go so far. It would be great to see the designs verified experimentally.\n- The authors note the current model cannot model sidechains. I do think this is a limitation that future work should explore, as it seems this would be important for the design of binding.\n\nMinor Comments:\n- It was slightly hard for me to associate different tables with experiments / datasets at a glance (e.g. it took me a little bit to figure out what exactly is the difference between Table 1 and Table 2 (left)). Perhaps the experiments section could include a bulleted list of datasets used, or this could be more clearly highlighted in the table captions.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "WJaWW42pKRq", "reviewer": "Reviewer_n2oD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Designing Antibody CDR sequences is of great interest for studying antibody biology and for therapeutics research. This task has been addressed computationally using search based frameworks (eg. using rosetta energies) and more recently, machine learning based generative models. This paper introduces a novel equivariant GNN framework for co-generating CDR sequences and structures. They evaluate the generative capacity of this model using standard benchmarks on sequence and structure generation (i.e Amino Acid Recovery/AAR and Structure RMSD). They demonstrate novel utility of this model in the task of optimizing the binding affinity of a candidate antibody. Lastly, they suggest a pipeline for generating a CDR sequence when the binding complex is unknown using a docking search approach.\n\nA major contribution this paper has made to the field is the capacity to conditionally model the antigen structure in the CDR generation. They implement this via an “external context encoder” attention mechanism which passes information from antigen residues to CDR residues in the hidden layer updates. The architecture for the message passing and attention paradigms are also novel and constructed to be equivariant, which is a required feature for a model operating on coordinate data. As the primary function of antibody CDR regions are to bind to target antigens, it follows that modelling the target structure in the generative model would be important. Thus this is an important contribution. \n\nThis paper compares its method to a previously published generative sequence-structure co-design model (RefineGNN from Jin et al 2021), an unconditional sequence model (from Saka et al 2021) and an Rosetta energy based search method (RAbD, Adolf-Bryfogle 2018). They show state-of-the-art performance on AAR and RMSD metrics, with quite significant improvements. \n\nThe paper also presents a novel sequence generation scheme, Progressive Full-Shot Decoding, which is faster than prior decoding techniques. They show in their ablation study that the decoding scheme hugely contributes to their performance gains.\n\nWhile a prior generative model paper (Jin et al 2021)  developed an algorithm for optimizing the binding affinity of antibodies, that method required an antigen specific oracle trained on experimental data. This paper takes the approach a step further by leveraging a delta-delta-G predictor from Shan et al 2022, which allows the algorithm to be used on any antibody-antigen pair for which a structure is available. The authors show their model outperforms other benchmarks using this pipeline in terms of generated antibody with lowest energy.\n", "summary_of_the_review": "This paper attempts to address the antibody heavy chain CDR sequence-structure co-design problem, though. They extend prior models on this task by incorporating conditioning on the target antigen and the light chain. The importance of incorporating this information is demonstrated by the major performance gains on established benchmarks. They also propose extended utility of this model in the goal of optimizing an antibody’s affinity and also generating a heavy chain CDR when a full binding complex is not present using docking.\n\nThere are some limitations in this paper such as a lack of description of certain modelling decisions and asI wrote above, it does not address the CDR sequence-structure problem in general,  rather only for a specific binding surface which by definition has already had a structure of Ab:Antigen solved.  However, the improvements on benchmarks and new ideas of in silico optimization make this paper a good step in the direction of solving the in silico antibody design problem", "strength_and_weaknesses": "The model architecture the authors develop is very thoughtfully constructed and the gains are demonstrated in the state-of-the-art performance. Their ablation study is an important addition, as it shows which aspect of the method contribute most to its performance gains. The use of Progressive Full-shot decoding was the biggest contributor to performance while also having major run-time gains. This suggests that it could have promise for utilization in other methods as well. \n\nThe authors did not give much explanation as to why a message propagation paradigm was used with the internal encoder and a graph attention scheme used in the external encoder. The ablation study also demonstrated that the attention scheme did not contribute hugely to the performance. Some more explication of the modelling decisions would be beneficial.\n\nThe “Affinity Optimization” experiment is a great section of the paper, and moves towards real-world usage of these methods. The authors show their method is able to generate a CDR with the biggest decrease in delta-delta-G (a proxy for binding affinity) compared to benchmarks. To make the result more biologically interpretable it would have been useful to have an idea of how much a delta-delta-G decrease contributes to a decrease in binding affinity (as measured by Kd for example). This could have been done using the Shan et al model and comparing to test performance on the SKEMPI set. While this correlation would be different for different protein interactions, it could have been a useful discussion point. Using a general delta-delta-G predictor over an oracle model trained on experimental data is a clever extension ofhe Iterative Target Augmentation algorithm for this use-case. However, while the authors do mention that they only benchmark on antibody complexes within the training set of the Shan et al model to ensure generalizability, this points to how the efficacy of this pipeline for any candidate antibody is limited by the generalizability of the predictor. While this is not a fault of their proposed model, if this pipeline is a major point of utility of this paper, it is a point worth discussing.\n\nWhile the authors do cite a major paper in the field for sequence-structure co-design (Jin et al 2021), they do not mention the new paper from that group (Jin et al 2022) which makes similar developments on its prior model as this paper performs (eg. modelling the antigen structure in the CDR generation). Furthermore, in terms of benchmarks, while the authors’ model did show state-of-the-art performance when compared to any cited paper, there are some discrepancies in RefineGNN’s RAbD benchmark performance between the original Jin 2021 paper and this paper’s reproduction of it. Lastly, it seems that the authors criticize RefineGNN for not being invariant in its loss calculation, however this is not the case.\n\nA high level thought : An overall weakness of this work is common to much of the published work in this field; i.e.  the work does not  really address the \"real world\" problem of antibody design; - the 'real-world' question should be given an antigen target, design an antibody that will bind with high affinity and specificity. However good a method is at predicting antibody binding to a specific epitope on a target antigen, and however well the model can generate sequences and structures that bind that epitope - the 'real-world' question should be able to find new epitopes, given an antigen target, design an antibody that will bind with high affinity and specificity.  Nevertheless , these authors do not claim to do this so thats fine - but the ML community should be aware that these evaluations and target problems only address a very narrow need- e.g. I have one epitope  on an antigen of know. structure - can I make a 'better' antibody to that specific target surface? \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%