Iterative Refinement Graph Neural Network for Antibody Sequence-Structure Co-design
Authors: Wengong Jin
Jeremy Wohlwend
Regina Barzilay
Tommi S. Jaakkola
Venue: ICLR 2022 Spotlight (2022)
Keywords: Drug Discovery
Antibody Design
Generative Models
Graph Generation

Abstract: Antibodies are versatile proteins that bind to pathogens like viruses and stimulate the adaptive immune system. The specificity of antibody binding is determined by complementarity-determining regions (CDRs) at the tips of these Y-shaped proteins. In this paper, we propose a generative model to automatically design the CDRs of antibodies with enhanced binding specificity or neutralization capabilities. Previous generative approaches formulate protein design as a structure-conditioned sequence generation task, assuming the desired 3D structure is given a priori. In contrast, we propose to co-design the sequence and 3D structure of CDRs as graphs. Our model unravels a sequence autoregressively while iteratively refining its predicted global structure. The inferred structure in turn guides subsequent residue choices. For efficiency, we model the conditional dependence between residues inside and outside of a CDR in a coarse-grained manner. Our method achieves superior log-likelihood on the test set and outperforms previous baselines in designing antibodies capable of neutralizing the SARS-CoV-2 virus.


Published as a conference paper at ICLR 2022
ITERATIVE REFINEMENT GRAPH NEURAL NETWORK FOR ANTIBODY SEQUENCE-STRUCTURE CO-DESIGN
Wengong Jin', Jeremy Wohlwend", Regina Barzilay', " Tommi Jaakkola"
f Eric and Wendy Schmidt Center, Broad Institute of MIT and Harvard
CSAIL, Massachusetts Institute of Technology {wengong, jwohlwend, zegina, t tommi @csail messsall.iit... mit.edu edu
ABSTRACT
Antibodies are versatile proteins that bind to pathogens like viruses and stimulate the adaptive immune system. The specificity of antibody binding is determined by complementarity -eeeraritt-detreiinngg regions (CDRs) the tips of these Y-shaped proteins. In this paper, we propose a generative model to automatically design the CDRs of antibodies with enhanced binding specificity or neutralization capa- bilities. Previous generative approaches formulate protein design as a structure- conditioned sequence generation task, assuming the desired 3D structure is given a priori. In contrast, we propose to co-design the sequence and 3D structure of CDRs as graphs. Our model unravels sequence autoregressively while itera- tively refining its predicted global structure. The inferred structure in turn guides subsequent residue choices. For efficiency, we model the conditional dependence between residues inside and outside of CDR in a coarse-graines manner. Our method achieves superior log-likelihood on the test set and outperforms previous baselines in designing antibodies capable of neutralizing the SARS-CoVe virus]
INTRODUCTION
Monoclonal antibodies are increasingly adopted as therapeutics targeting a wide range of pathogens such SARS-CoV-2 (Pinto al. 2020). Since the binding specificity of these Y-shaped proteins is largely determined by their compleementarit-ddeeermmmnn tational antibody design to automate the creation of CDR subsequences with desired properties. This problem is particularly challenging due to the combinatorial search space of over 20 possi- ble CDR sequences and the small solution space which satisfies the desired constraints of binding affinity, stability, and synthesizability (Raybould et al. 2019).
regions (CDRs), the main goal of compu-
There are three key modeling questions in CDR generation. The first is how model the relation between sequence and its underlying 3D structure. Generating sequences without the correspond- ing structure (Alley et al. 2019; Shin et al. 2021) can lead to sub-optimal performance (Ingraham) et al. 2019). while generating from a predefined 3D structure (Ingraham al., 2019) is not suit- able for antibodies since the desired structure is rarely known a priori (Fischman Ofran 2018) Therefore, it is crucial to develop models that co-design the sequence and structure. The second question is how to model the conditional distribution of CDRs given the remainder of a sequence (context). Attention-based methods only model the conditional dependence the sequence level, but the structural interaction between the CDR and its context is crucial for generation. The last question relates to the model's ability to optimize for various properties. Traditional physics-based methods (Lapidoth et al. 2015 Adoli-BryTogle et al., 2018) focus on binding energy minimization, but practice, our objective can be much more involved than binding energies (Liu et al. 2020).
In this paper, we represent a sequence-structure pair as a graph and formulate the co-design task as a graph generation problem. The graph representation allows us model the conditional depen- dence between CDR and its context at both the sequence and structure levels. Antibody graph generation poses unique challenges because the global structure expected change when new nodes are inserted. Previous autoregressive models (You et al. 2018 Gebauer al. 2019) cannot
Our code is available at

---

Published as a conference paper at ICLR 2022
modify generated structure because they trained under teacher forcing. Thus errors made in the previous steps can lead to a cascade of errors in subsequent generation steps. To address these problems, propose novel architecture which interleaves the generation of amino acid nodes with the prediction of 3D structures. The structure generation is based on an iterative refinement of global graph rather than sequential expansion of partial graph with teacher foreing. Since the context sequence is long, we further introduce a coarsened graph representation by grouping nodes into blocks. We apply graph convolution coarser level to efficiently propagate the contextual information to the CDR residues. After pretraining our model on antibodies with known structures, we finetune it using a predefined property predictor to generate antibodies with specific properties.
We evaluate our method on three generation tasks, ranging from language modeling to SARS-CoV. neutralization optimization and aatigen-binding antibody design. Our method compared with a standard sequence model (Saka et al. [202 Akbar et al. (021) and a state-of-the-art graph genera- tion method (You et al. (2018) tailored to antibodies. Our method not only achieves lower perplexity on test sequences but also outperforms previous baselines in property-guide antibody design tasks.
2 RELATED WORK
Antibody/protein design. Current methods for computational antibody design roughly fall into two
categories. The first class based on energy function optimization (Pantazes Maranas 2010 Li
et al. 2014 Lapidoth et al.] 2015 Adolf-Bryfogle et al. 2018), which use Monte Carlo simulation
to iteratively modify a sequence and its structure until reaching a local energy minimum. Similar approaches are used in protein design (Leaver-Fay et al. 2011 Tischer et al. (020). Nevertheless,
2019) and our desired objective can be much more complicated than low binding energy (Liu al.
these physics-based methods are computationally expensive (Ingraham et al.
2020
The second class is based on generative models. For antibodies, they are mostly sequence-based (Alley et al. 2019 Shin et al. 2021 Saka et al., 2021 Akbar et al., (221). For proteins, O' O'Connell eet al. (2018;; Ingraham et al. (2019): Strokach et al. (2020); Karimi et al. (2020); Cao et al. (2021) backbone structure or protein fold. Our model also seeks to incorporate 3D structure information for antibody generation. Since the best CDR structures are often unknown for new pathogens, we co-design sequences and structures for specific properties.
further developed models conditioned
Generative models for graphs. Our work is related to autoregressive models for graph generation Yoou et al. 2018 Li al., 2018 Liu et al. 2018 Liao et al.] 2019 Jin et al. (220aa). In particular, Gebauer et al. (2019) developed G-SchNet for molecular graph and conformation co-design. Unlike our method, they generate edges sequentially and cannot modify a previously generated subgraph when new nodes arrive. While Graphite (Grover et al. 2019) also uses iterative refinement to predict the adjacency matrix of a graph, it assumes all the node labels are given and predicts edges only. In contrast, our work combines autoregressive models with iterative refinement to generate full graph with node and edge labels, including node labels and coordinates.
3D structure prediction. Our approach is closely related to protein folding (Ingrabam et al.) 2018 Yang et al.) 2020a Back et al. 2021] Jumper et al. 2021). Inputs to the state-of-the-art models like AlphaFold require a complete protein sequence, its multi-sequence alignment (MSA), and its template features. These models are not directly applicable because we need to predict the structure of an incomplete sequence and the MSA is not specified in advance.
Our iterative refinement model is also related to score matching methods for molecular conformation prediction (Shi et al. 202T) and diffusion- based methods for point clouds (Luo & Hu 2021). These algorithms also iteratively refine a predicted 3D structure, but only for a complete molecule or point cloud. In contrast, our approach learns predict the 3D structure for incomplete graphs and interleaves 3D structure refinement with graph generation.
3 ANTIBODY SEQUENCE AND STRUCTURE CO-DESIGN
Overview, The role of an antibody is to bind to an antigen (e.g. a virus), present it to the immune system, and stimulate immune response. A subset of antibodies known as neutralizing antibodies not only bind to an antigen but can also suppress its activity. An antibody consists of a heavy chain and light chain, each composed of one variable domain (VH/VL.) and some constant domains. The
2

---

Published as a conference paper at ICLR 2022
Antibody sequence
Heavy chain variable region (VH)
CDR-H1
CDR-12
CDR-H3
Antibody structure
Light chain variable region (VL)
CDR-L1
CDR-L2
CDR-L3
Figure 1: Schematic structure of an antibody (figure modified from Wikipedia).
variable domain is further divided into a framework region and three complementarity determining regions (CDRs). The three CDRs on the heavy chain are labeled as CDR-H1, CDR-H2, CDR-H3, each occupying a contiguous subsequence (Figure D] As the most variable part of an antibody, CDRs are the main determinants of binding and neutralization (Abbas et al. 2014).
Following Shin et al. (202T); :; Akbar al. (202T), we formulate antibody design as a CDR generation task, conditioned on the framework region. Specifically, we represent an antibody as graph, which encodes both its sequence and 3D structure. We propose a new graph generation approach called RefineGNN and extend it to handle conditional generation given fixed framework region. Lastly, we describe how to apply RefineGNN to property-guided optimization to design new antibodies with better neutralization properties. For simplicity, we focus on the generation of heavy chain CDRs, though our method can be easily extended to model light chains CDRs.
Notations. An antibody VH domain is represented as sequence of amino acids 8182 8n. Each token 8; in the sequence is called a residue, whose value can be either one of the 20 amino acids special token (MASK), meaning that its amino acid type is unknown and needs to be predicted. The VH sequence folds into a 3D structure and each residue 8, is labeled with three backbone coordinates: Fi for its alpha carbon atom, ii for its carbon atom, and Bi.a for its nitrogen atom.
3.1 GRAPH REPRESENTATION
We represent an antibody (VH) as graph ((8) E) with node features V (V1,-- Un and edge features {e,, hızj- Each node feature v, encodes three dihedral angles (Øs. () related to three backbone coordinates of residue i. For each residue i, we compute an orientation matrix O, representing local coordinate frame (Ingraham et al. 2019) (defined in the appendix). This allows us to compute edge features describing the spatial relationship between two residues i and j:
Bij- = (Fas (i j), RBF - æj,al)), O
Lja0 Biaa
((TTO)))
(1)
|
Dj,aal
The edge feature Gij contains four parts. The positional encoding Epos 16-)) f) encodes the relative distance between two residues in an antibody sequence. The second term RBF(-) is a distance encoding lifted into radial basis. The third term in eiz is direction encoding that corresponds to
the relative direction of æ; in the local frame of residue i. The last term (0, 0,) is the orientation encoding of the quaternion representation q-) of the spatial rotation matrix Og. We only include
edges in the K- K-nearest neighbors graph of ((s) () with K = 8. For notation convenience, we use G shorthand for ((s) when there no ambiguity.
3.2 ITERATIVE REFINEMENT GRAPH NEURAL NETWORK (REFINEGNN)
We propose to generate an antibody graph via an iterative refinement process. Let (()) be the initial guess of the true antibody graph. Each residue initialized special token (MASK) and each edge (i,j) is initialized to be of distance 3/i - j| since the average distance between consecutive residues around three. The direction and orientation features are set to zero. In each generation step 4, the model learns to revise current antibody graph g(t) and predict the label of the next residue + 1. Specifically, it first encodes git) with a message passing network (MPN) with parameter 0
(n (9, (2) (C)),
where (2) a learned representation of residue under the current graph g(t), Our MPN consists of L message passing layers with the following architecture
h (1/+1) LayerNorm
(t.)) (1.)) FFN (h. .h E(s,), e.i)) 0 05155-1, (3) Σ
3

---

Published as a conference paper at ICLR 2022
where h3 (/ v, and h h (, FFN is two-layer feed-forward network (FFN) with ReLU activation function. E(5,) is a learned embedding of amino acid type 8j- Based on the learned residue representations, we predict the amino acid type the next residue + (Figure 2
Pi+1 = softmax W Wahi+
(4)
This prediction gives new graph (t++0.5) with the same edges as ((+) but the node label of t + 1 changed (Figure Next, we need update the structure accommodate the new residue 1. To this end, we encode graph g(++0.5) by another MPN with a different parameter and predict the coordinate of all residues.
{h1 ((+0.5)
h ((t055)3 = MPN (CN
(5)
(++1)
W (6+0.5)
Isisn,ee fa,
(6)
The new coordinates x; 1)++) define new antibody graph G(++1) for the next iteration (Figure 2 We explicitly realize the coordinates of each residue because we need to calculate the spatial edge features for g(++1), The structure prediction (coordinates æ;;) and sequence prediction (amino acid types 11+1) are carried out by two different MPNs, namely the structure network 0 and sequence network 0. This disentanglement allows the two networks to focus two distinct tasks.
Training. During training, we only apply teacher forcing to the discrete amino acid type prediction. Specifically, in each generation step t, residues 1 to are set their ground truth amino acid types $1." St. while all future residues t + 1. ... n are set to a padding token. In contrast, the contin- uous structure prediction is carried out without teacher foreing. In each iteration, the model refines the entire structure predicted in the previous step and constructs new K K-nearest neighbors graph (t+1) |1 S E (a,c,n)) (t--1) of all residues based the predicted coordinates (z((+t)
Loss function. Our model remains rotation and translation invariant because the loss function is computed over pairwise distance and angles rather than coordinates. The loss funetion for antibody structure prediction consists of three parts.
Distance loss: For each residue pair j, we compute pairwise distance between the predicted alpha carbons x ii j.a" We define the distance loss as the Huber loss between the predicted and true pairwise distances
Σ Lhuber (1l00 F.aa || - 1)))
where distance is squared to avoid the square root operation which causes numerical instability.
Dihedral angle loss: For each residue, we calculate its dihedral angle (o!") (1!1) (w(")) based on
the predicted atom coordinates (!) (t) x and x (r)
# x
= Le
(f) (t)) i-1 We define the dihedral
angle loss as the mean square еттог between the predicted and true dihedral angles
C = E, Ca (o.))) a
az)² (sin a sin a;)
C., angle loss: We calculate angles 7; between two vectors a F- v and a!!
iit1aa as
x.
(1)
well as dihedral angles a(1)) between two planes defined by x. i- a (()
(1) (1)
1,1 : aa i+1,a
c
Σ (cos cos 71)2 (cos 18.10 cos 3,)
(9)
In summary, the overall graph generation loss is defined L Leea + Latructs where
Lstruct Σ. cD ((t) c(t)
Lyog Σ, Lee (Pt, 8t).
(10)
The sequence prediction loss Leeg is the cross entropy Loe between predicted and true residue types.
3.3 CONDITIONAL CENERATION GIVEN THE FRA FRAMEWORK REGION
The model architecture described so far is designed for unconditional generation - generates an entire antibody graph without any constraints. In practice, we usually fix the framework region of an
4

---

Published as a conference paper at ICLR 2022
C
amino acid
R
Predict all coordinates
Message
passing
...VT WGGG
Update
..VT WGOG
Update
Vyn VS
sequence
structure
VYYC
Coarsened
Sovdn VT
context sequence
ssS...
Sel
SA
g)(+00.5)
g++h)
Original sequence
DKSS
Context sequence (8)
ACKF
OGKA
TLTA
DKSS
WGOG
TSVT
Figure 2: (A-C) One generation step of RefineGNN. Each circle represents CDR residue and each square represents a residue block in a coarsened context sequence. (D) Sequence coarsening.
antibody and design the CDR sequence only. Therefore, we need to extend the model architecture to learn the conditional distribution P(s'| 8:1,, 8yr), where 8et = 81 ....8... 81-1 and 8>r 8,41 8n are residues outside of the CDR .p...
Conditioning via attention. simple extension of RefineGNN is to encode the non-CDR se- quence using recurrent neural network and propagate information to the CDR through an at- tention layer. To be specific, we first concatenate 8<1 and 85r into a context sequence ă = 8c1 (MASK) ... (MASK) @ 8>r, where @ means string concatenation and (MASK) is repeated times. We then encode this context sequence by a Gated Recurrent Unit (GRU) (Cho et al. 2014 and modify the structure and sequence prediction step (Equation Innlatiiooaanne as
{er, Cn. Ci.n GRU($)
(11)
Pt+1 = softmax Wah 441 + U. attention(CI.. h (t+1) W ((+0.5) ((+00)
(12)
+ U attention(Cim. h (1+0.5))
(13)
Multi-resolution modeling. The attention based approach alone is not sufficient because it does not model the structure of the context sequence, thus ignoring how its residues structurally interact with the CDR's. While this information is not available for new antibodies at test time, we can learn to predict this interaction using antibodies the training set with known structures.
A naive solution is to iteratively refine the entire antibody structure (more than 100 residues) while generating CDR residues. This approach computationally expensive because we need to recom- cannot predict the
pute the MPN encoding for all residues in each generation step. Importantly, context residue coordinates at the outset and fix them because they need to be adjusted accordingly when the coordinates of CDR residues updated in each generation step.
For computational efficiency, we propose a coarse- coarse-grained model that reduces the context sequence length by clustering it into residue blocks. Specifically, bi,+(s) by clustering every b context residues into a block (Figure (D) The new sequence (s) defines coarsened graph (b)) over the residue blocks, whose edges are defined based block coordinates. The coordinate of each block Øb... is defined as the mean coordinate of residues within the block. The embedding of each block E(b,) is the mean of its residue embeddings.
construct coarsened context sequence
E(b,) Σ EbEb: E(s,)/b,
ab..c = - Djje/b,
E cn..
(14)
Now apply RefineGNN to generate the CDR residues while iteratively refining the global graph (sb,,.(s)) by predicting the coordinates all blocks. The only change is that the structure prediction loss is defined over block coordinates Zb,,. Lastly, combine both the attention mech- anism and coarse-graine coarse-grained modeling to keep both fine-grained and coarse-grained information. The decoding process of this conditional RefineGNN is illustrated in Algorithm| 1
5

---

Published as a conference paper at ICLR 2022
Algorithm/ RefineGNN decoding
Algorithm 2 ITA- sequence optimization
Require:
Context sequence 8cl; 8yr
Require: A set of antibodies D to be optimized
1: Predict the CDR length
Require: A neutralization predictor f.
2: Coarsen the context sequence into (8)
Require: A set of neutralizing antibodies Q
3: Construct the initial graph g(o)
1: for each iteration do
2: Sample an antibody s from D, remove its
4: for = 0 to - do
5: Encode (()) using the sequence MPN
CDR and get a context sequence (s,
3: for = 1 to M do
6: Predict distribution of the next residue Pt+11
4: Sample s ~ Pe bi,
7: Sample St+1 categorical(pit))
5:
if f(()) > (f)x) 0.5) then
8: Encode g(t++0.) with the structure MPN
6:
Q - Q (s'))
9: Predict all residue coordinates xi,e (f+1)
7:
Sample batch of antibodies from Q
8:
Update model parameter θ by minimizing the sequence prediction loss Leeq.
10: Update ((++1) using the new coordinates
3.4
PROPERTY-GUIDED SEQUENCE OPTIMIZATION
Our ultimate goal is generate antibodies with desired properties such neutralizing par- ticular virus. This task can be formulated as an optimization problem. Let Y be a binary indicator variable for neutralization. Our goal is to learn conditional generative model (e (()))),))))) that maximizes the probability of neutralization for a training set of antibodies 2, i.e.
Σ.ED log P(Y (s), E.ED log (a
(15)
EED
where f(s') is a predictor for P(Y = 1ss'). Assuming f is given, this problem can be solved by iterative target augmentation (ITA) (Yang al., (020b). Before ITA optimization starts, first pretrain our model on a set of real antibody structures to learn a prior distribution over CDR sequences and structures. each ITA finetuning step, we first randomly sample sequence from D. a set of antibodies whose CDRs need to be redesigned. Next, we generate M new sequences given its context by.- (s). generated sequence s is added to our training set Q it is predicted as neutralizing. Initially, the training set Q contains antibodies that are known to be neutralizing (Y 1). Lastly, we sample a batch of neutralizing antibodies from Q and update the model parameter by minimizing their sequence prediction loss Leea (0). The structure prediction loss Estruct is excluded in ITA finetuning phase because the structure of a generated sequence unknown.
EXPERIMENTS
Setup. We construct three evaluation setups quantify the performance of our approach. Following standard practice in generative model evaluation, we first measure the perplexity of different models on new antibodies in set created based on sequence similarity split. We also measure structure prediction eиror by comparing generated and ground truth CDR structures recorded in the Structural Antibody Database (Dunbar et al., 2014). Results for this task are shown in section 4.1
Second, we evaluate our method on an existing antibody design benchmark of 60 antibody-antigen complexes from Bryff-Br et al. (2018). The goal is design the CDR-H3 of an antibody so that it binds to given antigen. Results for this task are shown in section 4.2
Lastly, we propose an antibody optimization task which aims to redesign CDR-H3 of antibodies in the Coronavirus Antibody Database (Raybould et al. 2021) to improve their neutralization against SARS-CoV-2. CDR-H3 design with a fixed framework is common practice in the antibody engi- neering community (Adolf-Bryfolie et al.) 2018 Liu et al. 2020). Following works in molecular design (Jin et al.) (020b), we use a predictor to evaluate the neutralization generated antibodics since cannot experimentally test them in wet labs. Results for this task reported in section 4.3
We consider three baselines for comparison (details in the appendix). The first baseline is requence-based LSTM model used in Saka et al. b): Akbar al. (2021). This model does not utilize any 3D structure information. It consists of an encoder that learns to encode a context sequence decoder that decodes a CDR sequence, and attention layer connecting the two.
Baselines.

---

Published as a conference paper at ICLR 2022
Table 1: Left: Language modeling results. We report perplexity (PPL) and root mean square de- viation (RMSD) for each CDR in the heavy chain. Right: Results on the antigen-binding antibody design task. We report the amino acid recovery (AAR) for all methods.
CDR-HI
CDR-H2
CDR-H3
Model
AAR
Model
PPL
RMSD
PPL
RMSD
PPL
RMSD
RAbD
28.53%
LSTM
6.79
7.21
9.70
LSTM
22.53%
AR-GNN
6.44
2.97
6.86
2.27
9.44
3.63
AR-GNN 23.86%
RefineGNN
6.09
1.18
6.58
0.87
8.54
2.50
RefineGNN 34.14%
The second baseline an autoregressive graph generation model (AR-GNN) whose architecture is similar to You et al. (2018); Jin et al.] (2020b) but tailored for antibodies. AR-GNN generates an antibody graph residue by residue. In each step i, first predicts the amino acid type of residue and then generates edges between t and previous residues. Importanly, AR-GNN cannot modify partially generated 3D structure residues $1 ... $t-1 because it trained by teacher forcing.
On the bodyDesign (RAbD) AAdolf-Bryfogle et al. 2018). We apply their de novo design protocol com- posed of graft design followed by 250 iterations of sequence design and energy minimization. We cannot afford to run more iterations because takes more than 10 hours per antibody. We also could not apply RAbD to the SARS-CoV-2 task because it requires 3D structures be given. This information is unavailable for antibodies in CoVAbDab.
task, we include an additional boysics-based baseline called RosettaAnti-
Hyperparameters. We performed hyperparameter tuning to find the best setting for each method. For RefineGNN, both its structure and sequence MPN have four message passing layers, with a hidden dimension of 256 and block size b 4. All models are trained by the Adam optimizer with a learning rate of 0.001. More details are provided in the appendix.
4.1 LANGUAGE MODELING AND 3D STRUCTURE PREDICTION
Data. The Structural Antibody Database (SAbDab) consists of 4994 antibody structures renumbered according to the IMGT numbering scheme (Lefrane al.] 2003). To measure a model's ability to generalize novel CDR sequences, we divide the heavy chains into training, validation, and test sets based on CDR cluster split. We illustrate our cluster split process using CDR-H3 as an example. First,
MMseqs2 (Steinegger & Söding 2017) to cluster all the CDR-H3 sequences. The sequence identity is calculated under the BLOSUM62 substitution matrix (Henikoff & Henikoff] 1992]. Two antibodies are put into the same cluster if their CDR-H3 sequence identity is above 40%. We then randomly split the clusters into training, validation, and test set with 8:1:1 ratio. We repeat the same procedure for creating CDR-HI and CDR-H2 splits. In total, there are 1266, 1564, and 2325 clusters for CDR-HI, H2, and H3. The size of training, validation, and test sets for each CDR is shown in the appendix.
For each method, we report the perplexity (PPL) of test sequences and the root mean square deviation (RMSD) between predicted structure and its ground truth structure reported in SAbDab. RMSD is calculated by the Kabsch algorithm (Kabsch 1976) based on C, coordinate of CDR residues. Since the mapping between sequences and structures deterministic in RefineGNN, we can calculate perplexity in the same way as standard sequence models.
Metrics.
Results.
Since the LSTM baseline does not involve structure prediction, we report RMSD for graph- RefineGNN significantly outperforms all baselines in
based methods only. As shown in Table [ both metrics. For CDR-H3, our model gives 13% PPL reduction (8.54 v.s. 9.70) over sequence only model and 10% PPL reduction over AR-GNN (8.54 v.s. 9.44). RefineGNN also predicts the struc- ture more accurately, with 30% relative RMSD reduction over AR-GNN. In Figure 3 examples of predicted 3D structures of CDR-H3 loops.
we provide
Ablation studies. We further conduct ablation experiments on the CDR-H3 generation task to study the importance of different modeling choices. First, when remove the attention mechanism and context coarsening step in section 3.3] the PPL increases from 8.54 to 8.86 (Figure 3C, row 2) and 9.01 (Figure 3C, row 3) respectively. We also tried remove both the attention and coarsening
7

---

Published as a conference paper at ICLR 2022
B
RefineGNN
6.54
No attention
8.66
CDR-H3
CDR-HS
coarsening
9.01
Unconditioaal
8.95
Structure- conditoned
7.39
Ground truth RefineGNN
Ground truth
AR-GNN
CDR-H3 Perplexity
Figure 3: (A) CDR-H3 structure predicted by RefineGNN (PDB: 4bkl, RMSD 0.57). The pre- dicted structure (cyan) is aligned to the true structure (green) using the Kabsch algorithm. (B) CDR- H3 structure predicted by AR-GNN (PDB: 4bkl, RMSD 2.16). (C) Ablation studies of different modeling choices in RefineGNN in the CDR-H3 perplexity evaluation task.
modules and trained the model without conditioning on the context sequence. The PPL of this unconditional variant much worse than our conditional model (Figure 3C. row 4). Lastly, we train drructurecoodditions model by feeding the ground truth structure to RefineGNN at every generation step (Figure BC, row 5). While this structure-conditionn expected (7.39 8.54), it is not too far away from the sequence only model (PPL 9.70). This
model gives a lower PPL as
suggests that RefineGNN is able to extract a decent amount information from the partial structure co-evolving with the sequence.
4.2 ANTIGEN-BINDING ANTIBODY DESIGN
Data. Adolf-Brytogle al. 2018 selected 60 antibody-antigen complexes an antibody design benchmark. Given the framework of an antibody. the goal is to design its CDR-H that binds to its corresponding antigen. For simplicity, none of the methods conditioned on the antigen structure during CDR-H3 generation. We leave antigen-conditioned CDR generation for future work.
Metric. Following Adolf-Bryfogle et al. (2018), we use amino acid recovery (AAR) as the evalu- ation metric. For any generated sequence, we define its AAR as the percentage of residues having the same amino acid as the corresponding residue in the original antibody.
Results. For LSTM, AR-GNN, and RefineGNN, the training set in this setup is the entire SAbDab except antibodies in the same cluster as any the test antibodies. At test time, we generate 10000 CDR-H3 sequences for each antibody and select the top 100 candidates with the lowest perplexity. For simplicity, all methods are configured to generate CDRs of the same length as the original CDR. As shown in Table I our model achieves the highest AAR score, with around 7% absolute improvement over the best baseline. H3 sequence and highlight residues that different from the original antibody. We also found that sequences with lower perplexity tend to have a lower AA recovery error (Pearson R = 0.427, Figure 4B). This suggests that we can use perplexity as the ranking criterion for antibody design.
Figure FA, we show an example of a generated CDR-
4.3 SARS-COV-2 NEUTRALIZATION OPTIMIZATION
Data. The Coronavirus Antibody Database (CoVAbDab) contains 2411 antibodies, each associ- ated with multiple binary labels indicating whether it neutralizes coronavirus (SARS-CoV-1 or SARS-CoV-2) at certain epitope. Similar to the previous experiment, we divide the antibodics into training, validation, and test sets based CDR-H3 cluster split with 8:1:1 ratio.
Neutralization predictor. The predictor takes as input the VH sequence of an antibody and out- puts neutralization probability for the SARS-CoV-I and SARS-CoV-2 viruses. Each residue is embedded into a 64 dimensional vector, which fed to a SRU encoder (Lei, (02T) followed by average-pooling and two-layer feed forward network. The final outputs the probabilities pr
8

---

Published as a conference paper at ICLR 2022
CDR-HS
heopperpeexay
Designed antibody
Peerson R 0.427
Original antibody
Original:
ARCDYYGBWSLDY
82 0.6
68
0.
14
Designed: ARADYYGSCAMDY
Antigen
recovery
Figure 4: (A) Visualization of generated CDR-H3 sequence and its structure complex with an antigen (PDB: 4cmh). The predicted structure is aligned and grafted onto the original antibody using Kabsch algorithm. Residues different from the original antibody are highlighted in red. (B) The correlation between the perplexity of generated sequence and AA recovery emror
Table 2: SARS-CoV-2 neutralization optimization results. For each method, report the PPL VobAbDab after pretraining on SAbDab and then report the average neutralization score after ITA finetuning. The average neutralization probability of original CoVAbDab antibodies is 69.3%.
Original
LSTM
AR-GNN
RefineGNN
CoVAbDab PPL (4)
9.40
8.67
7.86
Neutralization (f)
69.3%
72.0%
70.4%
75.2%
and p2 of neutralizing SARS-CoV-I and SARS-CoV-2 and our scoring function f(s) = p2. The predictor achieved 0.81 test AUROC for SARS-CoV-2 neutralization prediction.
CDR sequence constraints. Therapeutic antibodies must be free from developability issues such glycosylation and high charges (Raybould et al. 2019). Thus, include four constraints CDR-H3 sequence s: 1) Its net charge must be between -2.0 and 2.0 (Raybould et al. 2019). The definition of net charge is given in the appendix. 2) It must contain the N-X-S/T motif which is prone to glycosylation. 3) Any amino acid should not repeat more than five times (e.g. SSSSS) 4) Perplexity of a generated sequence given LSTM, AR-GNN, and RefineGNN should be all less than 10. The last two constraints force generated sequences to be realistic. We use all three models in the perplexity constraint to ensure a fair comparison for all methods.
Metric. For each antibody in the test set, we generate 100 new CDR-H3 sequences, concatenate them with its context sequence form 100 full VH sequences, and feed them into the neutralization predictor f. We report the average neutralization score of antibodies in the test set. Neutralization score of generated sequence s' equals f(s') if satisfies all the CDR sequence constraints. Oth- erwise the score is the same the original sequence. In addition, we pretrain each model on the SAbDab CDR-H3 sequences and evaluate its PPL on the CoVAbDab CDR-H3 sequences.
All methods are pretrained on SAbDab antibodies and finetuned on CoVAbDab using the ITA algorithm to generate neutralizing antibodies. Our model outperforms the best baseline by 3% increase in terms of average neutralization score (Table 2). Our pretrained RefineGNN also achieves a much lower perplexity on Vo antibodies (7.86 v.s. 8.67). Examples generated CDR-H3 sequences and their predicted neutralization scores shown in the appendix.
Results.
CONCLUSION
In this paper, we developed a RefineGNN model for antibody sequence and structure co-design. The advantage of our model over previous graph generation methods is its ability to revise a gen- erated subgraph to accommodate addition new residues. Our approach significantly outperforms sequence-based and graph-based approaches three antibody generation tasks.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
We would like to thank Rachel Wu, Xiang Fu, Jason Yim, and Peter Mikhael for their valuable feedback the manuscript. We also want to thank Nitan Shalon, Nicholas Webb, Jae Hyeon Lee, Qiu Yu, and Galit Alter for their suggestions on method development. We are grateful for the generous support of Mark and Lisa Schwartz, funding Defense Threat Reduction Agency (DTRA), C3.ai Digital Transformation Institute, Eric and Wendy Schmidt Center at the Broad Institute, Abdul Latif Jameel Clinic for Machine Learning in Health, DTRA Discovery of Medical Countermeasures Against New and Emerging (DOMANE) threats program, and DARPA Accelerated Molecular Discovery program.
form of research grant from Sanofi,
REFERENCES
Abul K Abbas, Andrew H Lichtman, and Shiv Pillai. Cellular and molecular immunology E-book. Elsevier Health Sciences, 2014.
Jared Adolf-Bryfogle, Oleks Kalyuzhniy, Michael Kubitz, Brian D Weitzner, Xiaozhen Hu, Yumiko Adachi, William R Schief, and Roland Dunbrack Jr. Rosettaantibodydesiiu (rabd): A general framework for computational antibody design. PLoS computational biology, 14(4):e1006112 2018.
Rahmad Akbar, Philippe Robert, Cédric R Weber, Michael Widrich, Robert Frank, Milena Pavlović, Lonneke Scheffer, Maria Cherigoovkkyy,, Igor Snapkov, Andrei Slabodkin, al. In sil- ico proof of principle of machine learning-based antibody design at unconstrained scale. BioRxiv, 2021.
Mohammed M Al Qaraghuli, Karina Lubiak-Ossowsker Valerie A Ferro, and Paul A Mulheran. Antibody-protein binding and conformational changes: identifying allosteric signalling pathways to engineer better effector response. Scientific reports, 10(1):1-10, 2020.
Ethan C Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M Church. Unified rational protein engineering with sequence-base -bssed deep representation learning. Nature methods, (111))1115-1222 2019.
Minkyung Back, Frank DiMaio, Ivan Anishchenko, Justas Dauparas, Sergey Ovchinnikov, Gyu Rie Lee, Jue Wang, Qian Cong, Lisa N Kinch, Dustin Schaeffer, et al. Accurate prediction of protein structures and interactions using three-track neural network. Science, 373(6557):871- 876, 2021.
Yue Cao, Payel Das, Vijil Chenthamarakshan, Pin- Yu Chen, Igor Melnyk, and Yang Shen. Fold2seq: A joint sequence (1d)-fold (3d) embedding-bassee generative model for protein design. In Inter- national Conference on Machine Learning, pp. 1261-1271. PMLR, 2021.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre. Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using encoder-decoder for statistical machine translation. arXiv preprint 1406. 2014.
James Dunbar, Konrad Krawezyk, Jinwoo Leem, Terry Baker, Angelika Fuchs, Guy Georges, Jiye Shi, and Charlotte M Deane. Sabdab: the structural antibody database. Nucleic acids research, (101)::D110-DDD 146, 2014.
Sharon Fischman and Yanay Ofran, Computational design of antibodies. Current opinion in struc- tural biology, 51:156-162, 2018.
Niklas WA Gebauer, Michael Gastegger, and Kristof T Schütt. Symmetry-adaptee generation of 3d point sets for the targeted discovery of molecules. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pp. 7566-7578, 2019.
Aditya Grover, Aaron Zweig, and Stefano Ermon. Graphite: Iterative generative modeling of graphs. In International conference on machine learning, pp. 2434-24444 22444. PMLR, 2019.
Steven Henikoff and Jorja G Henikoff. Amino acid substitution matrices from protein blocks. Pro- ceedings of the National Academy of Sciences, 89(22): 10915-10919, 1992.
10

---

Published as a conference paper at ICLR 2022
John Ingraham, Adam Riesselman, Chris Sander, and Debora Marks. Learning protein structure with a differentiable simulator. In International Conference on Learning Representations, 2018.
John Ingraham, Vikas K Garg, Regina Barzilay, and Tommi Jaakkola. Generative models for graph- based protein design. Neural Information Processing Systems, 2019.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Hierarchical generation of molecular graphs using structural motifs. In Proceedings of the 37th International Conference on Machine Learn- ing, volume 119, pp. 4839-4844 PMLR, 2020a.
Wengong Jin, Regina Barzilay, and Tommi Jaakkola. Multi-objective molecule generation using interpretable substructures. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 4849-4859 PMLR, 2020b.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-588) 2021.
Wolfgang Kabsch. A solution for the best rotation to relate two sets of vectors. Acta Crystallo- graphica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography, 32 (5):922-923, 1976.
Mostafa Karimi, Shaowen Zhu, Yue Cao, and Yang Shen. De novo protein design for novel folds using guided conditional wasserstein generative adversarial networks. Journal of Chemical Infor- mation and Modeling, 60(12):5677-5681 2020.
Gideon D Lapidoth, Dror Baran, Gabriele M Pszolla, Christoffer Norn, Assaf Alon, Michael D) Tyka, and Sarel J Fleishman. Abdesign: A algorithm for combinatorial backbone design guided by natural conformations and sequences. Proteins: Structure, Function, and Bioinformatiic, 83 (8):1385-1406, 2015.
Andrew Leaver-Fay, Michael Tyka, Steven M Lewis, Oliver F Lange, James T'hompson, Ron Jacak, Kristian W Kaufman, P Douglas Renfrew, Colin A Smith, Will Sheffler, et al. Rosetta3: an object- oriented software suite for the simulation and design of macromolecules. Methods in enzymology, 487:545-5744 574, 2011.
Marie-Paule Lefranc, Christelle Pommié, Manuel Ruiz, Véronique Giudicelli, Elodie Foulquier, Lisa Truong, Valérie Thouvenin-Comee and Gérard Lefranc. Imgt unique numbering for im- munoglobulin and cell receptor variable domains and ig superfamily v-like domains. Develop- mental Comparative Immunology, 27(1):55-77, 2003.
Tao Lei. When attention meets fast recurrence: Training language models with reduced compute.
arXiv preprint arXiv:2102.12459, 2021.
Tong Li, Robert J Pantazes, and Costas D Maranas. Optmaven-a new framework for the de novo design of antibody variable region models targeting specific antigen epitopes. PloS one, 9()): e105954, 2014.
Yujia Li, Oriol Vinyals, Chris Dyer, Razvan Pascanu, and Peter Battaglia. Learning deep generative models of graphs. arXiv preprint 1003.0/800.0332.. 2018.
Renjie Liao, Yujia Li, Yang Song. Shenlong Wang. Will Hamilton, David K Duvenaud, Raquel Urtasun, and Richard Zemel. Efficient graph generation with graph recurrent attention networks. Advances in Neural Information Processing Systems, 32:4255 2019.
Ge Liu, Haoyang Zeng, Jonas Mueller, Brandon Carter, Ziheng Wang. Jonas Schilz, Geraldine Horny, Michael E Birnbaum, Stefan Ewert, and David K Gifford. Antibody complementarity determining region design using high-capacity machine learning. Bioinformatics, 36(7):2126 2133, 2020.
Qi Liu, Miltiadis Allamanis, Mare Brockschmidt, and Alexander L Gaunt. Constrained graph vari- ational autoencoders for molecule design. Neural Information Processing Systems, 2018
11

---

Published as a conference paper at ICLR 2022
Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2837-2845, 2021.
James O'Connell, Zhixiu Li, Jack Hanson, Rhys Heffeman, James Lyons, Kuldip Paliwal, Abdollah Dehzangi, Yuedong Yang, and Yaoqi Zhou. Spin2: Predicting sequence profiles from protein structures using deep neural networks. Proteins: Structure, Function, and Bioinformaties, 86(6); 629-633, 2018.
RJ Pantazes and Costas D Maranas. Optedr: a general computational method for the design of antibody complementarity determining regions for targeted epitope binding. Protein Engineering, Design & Selection. (3(11):849-33 2010.
Dora Pinto, Young-Jun Park, Martina Beltramello, Alexandra C Walls, M Alejandra Tortorici, Siro Bianchi, Stefano Jaconi, Katja Culap, Fabrizia Zatta, Anna De Marco, et al. Cross-neutralization of sars-cov-2 by a human monoclonal sars-cov antibody. Nature, 583(7815):290-
2020.
Matthew IJ Raybould, Claire Marks, Konrad Krawezyk, Bruck Taddese, Jaroslaw Nowak, Alan Lewis, Alexander Bujotzek, Jiye Shi, and Charlotte M Deane. Five computational developability guidelines for therapeutic antibody profiling. Proceedings of the National Academy of Sciences, 116(10):4025-43330 2019.
Matthew IJ Raybould, Aleksandr Kovaltsuk, Claire Marks, and Charlotte M Deane. Cov-abdab: the coronavirus antibody database. Bioinformatics, 37(5):734-735, 2021.
Koichiro Saka, Taro Kakuzaki, Shoichi Metsugi, Daiki Kashiwagi, Kenji Yoshida, Manabu Wada, Hiroyuki Tsunoda, and Reiji Teramoto. Antibody design using Istm based deep generative model from phage display library for affinity maturation. Scientific reports, 11(1):1-13, 2021.
Chence Shi, Shitong I Luo, Minkai Xu, and Jian Tang. Learning gradient fields for molecular confor- mation generation. International Conference on Machine Learning, 2021.
Jung-Eun Shin, Adam Riesselman, Aaron W Kollasch, Conor McMahon, Elana Simon, Chris Sander, Aashish Manglik, Andrew C Kruse, and Debora Marks. Protein design and variant prediction using autoregressive generative models. Nature communications, 12(1):1-11, 2021.
Martin Steinegger and Johannes Söding. Mmseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature biotechnology. 35(1 (5(11):0066-008, 2017.
Alexey Strokach, David Becerra, Carles Verge, Albert Perez-Riba, and Philip M Kim. Fast and flexible design of novel proteins using graph neural networks. BioRxiv, pp. 868935, 2020.
Doug Tischer, Sidney Lisanza, Jue Wang, Runze Dong, Ivan Anishchenko, Lukas F Milles, Sergey Ovchinnikov, and David Baker. Design of proteins presenting discontinuous functional sites using deep learning. bioRxiv, 2020.
Jianyi Yang, Ivan Anishchenko, Hahnbeom Park. Zhenling Peng, Sergey Ovchinnikov, and David Baker. Improved protein structure prediction using predicted interresidue orientations. Proceed- ings of the National Academy of Sciences, (173)):496-1155 2020a.
Kevin Yang. Wengong Jin, Kyle Swanson, Regina Barzilay, and Tommi Jaakkola. Improving molec- ular design by stochastic iterative target augmentation. In International Conference Machine Learning, pp. 10726. PMLR, 2020b.
Jiaxuan You, Rex Ying, Xiang Ren, William I. Hamilton, and Jure Leskovec. Graphrnn: A deep generative model for graphs. International Conference on Machine Learning, 2018.
12

---

i W.h, Published as a conference paper at ICLR 2022
A MODEL ARCHITECTURE DETAILS
A.1 REFINEGNN
Node features. Each node feature et; encodes three dihedral angles as follows.
v; (cos ti,, cos 1/1, CO8 wi, sin di, sin sht, sin w))
(16)
Edge features. The orientation matrix O [b,, n4, b, x n.] defines local coordinate system for each residue i (Ingraham et al. 2019). which is calculated as
2; -:-1
u, u
U X u,+1
b,
n.
(17)
x;: x2-1
u; u
u; u++1
Attention mechanism. The attention layer used in Eq. 13 is standard bilinear attention:
exp(c] Wh;)
attention(CIm: h₂) = Σ as
04, = Σ, E, exp(c] Wh.)
(18)
A.2 AR-GNN
AR-GNN generates an antibody graph autoregressively. In each generation step AR-GNN learns to encode the current subgraph Giæ induced from residues (s,, } into list of vectors
(h1,- ... hr} MPNa)
(19)
For fair comparison, we use the same MPN architecture for both RefineGNN and AR-GNN, In terms of structure prediction, AR-GNN first predicts the node feature ut41 of the next residue t + 1, namely the dihedral angle between its three atoms Ca, C, N.
(20)
In addition, AR-GNN predicts the pairwise distance between 8,-1 and previous residues 8,,
8t-
di FFN(W,h, + Unh, + VaE 1-11)
(21)
where FFN a feed-forward network with one hidden layer and Epss is the positional encoding of t+ 1 i, the gap between residue 81+1 and in the sequence. Lastly, AR-GNN predicts the amino acid type of residue 81+1 by
Pc+1 softmax W Wo9++1), (g1.... "1.,..99411" MPN (G1.411
(22)
Note that AR-GNN also uses two separate MPNs for structure and sequence prediction. However, unlike RefineGNN, AR-GNN is trained under teacher forcing we need to feed it the ground truth structure and sequence in each generation step. In particular, we find data augmentation to be crucial for AR-GNN performance. Data augmentation is essential because of the discrepancy between training and testing. The model is trained under teacher forcing, but it needs to decode a graph without teacher forcing at test time. We find mistakes made in previous steps have great impact on subsequent predictions during decoding.
Specifically, for every antibody s, we create a corrupted graph by adding independent random Gaussian noise to every coordinate: 2; = wi 3e, (N,00) I). In each generation step, apply MPN over the corrupted graph instead.
( } (P.))
(ō), 9 MPN MPNo
(23)
The node and edge labels are still defined by the ground truth structure. Specifically, let o, and d be the ground truth dihedral angle and pairwise distance calculated from the original, uncorrupted graph G. AR-GNN loss function is defined as the following.
CAR Σ |ida, 1.
di Σ é vill + Lee (P).
(24)
Similar to RefineGNN, AR-GNN also uses attention mechanism for conditional generation. Specif- ically, we concatenate the residue representations h G. from MPN with context vectors learned from an attention layer.
h h, @ attention(eL.n: h.)) 9t 9: attentionn(C.m..
(25)
13

---

Published as a conference paper at ICLR 2022
Table 3: SARS-CoV-2 neutralization optimization results. Here we show examples of new CDR- H3 sequences generated by our model and their predicted neutralization improvement over original antibodies S1D7 and C694 in the CoVAbDab database.
Antibody:
S1D7
C694
Old CDR-H3
TRGHSDY
ARDRGYDSSGPDAFDI
New CDR-H3
ARWWMDV
ARERIIIVSISAWMDV
Improvement
63% - 73%
82% 91%
B EXPERIMENTAL DETAILS
Hyperparameters. For AR-GNN and RefineGNN, we tried hidden dimension d, E {128, 256} and number message passing layers L E {1, 4, 5). We found d, 256, worked the best for RefineGNN and d3 = 256, I, = 3 worked the best for AR-GNN. For LSTM, we tried d, E {128, 256, 512, 1024]. We found dn = 256 worked the best. All models are trained by an Adam optimizer with dropout of 0.1 and learning rate of 0.001.
SAbDab data. The dataset statistics of SAbDab is the following (after deduplication). For CDR- H1, the vrain/validaiion/testt size is 4050, 359, and 326. For CDR-H2, the train/validationntest size is 3876, 483, and 376. For CDR-H3, the train/validation/test size is 3896, 403, and 437.
Since SAbDab includes both bound and unbound structures, removed all antigens and used the bound antibody structure for training. Specifically, 65% of our training data came from bound state structures. We included all data in our training set because the mismatch between bound and unbound structures is relatively small. In fact, Al Qaraghuli et al.] (2020) studied eight antibodies and found that the RMSD between bound and unbound structures over VH domains less than 0.7 on average.
RAbD configuration. We provided details of the de novo design setup of RosettaAntibodyyesig (RAbD) here. For each antibody in the test set, RAbD starts by randomly selecting a CDR from RAbD's internal database of known CDR structures. The chosen CDR-H3 sequence is required to have same length as the original sequence, but it cannot be exactly the same as the original CDR-H3 sequence. After the initial CDR structure chosen, RAbD grafts it onto the antibody and performs energy minimization stabilize its structure. Next, RAbD runs 100 iterations of sequence design to modify the grafted CDR-H3 structure by randomly substituting amino acids. In each sequence design iteration, it performs energy minimization to adjust the structure according to the changed amino acid. Lastly, the model returns the generated CDR-H3 sequence with the lowest energy.
SARS-CoV-2 neutralization. Each generative model is pretrained the SAbDab data to learn prior distribution over CDR H3 structures. Given fixed predictor f. we use the ITA algorithm to finetune our pretrained models generate neutralizing antibodies. Each model is trained for 3000 ITA steps with M 100. Generated CDR-H3 sequences from our model are visualized in Table 3
Our neutralization predictor is trained on the Co database. For simplicity, we only sider two viruses, SARS-CoV-1 and SARS-CoV-2 since other coronavirus have very little training data. For the same reason, we only consider the spike protein receptor binding domain epitope. The predictor is trained in multi-task fashion to predict both SARS-CoV. and SARS- CoV-2 neutralization labels. The SRU encoder has a hidden dimension of 256. The model was trained with dropout of 0.2, learning rate of 0.0005, and batch size of 16.
target
The charge of a residue is defined as C(s)) I[s; E {R, K]] + 0.1 I[s; H I[s, (D, RRaybould et al. 2019). The net charge of a sequence ... Sp is defined as Σ, C(s)).
14


Reviews:
[{"review_id": "wb38rxdj6Ga", "reviewer": "Reviewer_NDVA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper provides an interesting study of antibody loop generation with both novel methodology and extensive empirical evaluation. This combination of strengths makes it an excellent paper that should be of wide interest.\n\nStrengths:\n- The design of each component follows naturally from previous work (such as Ingraham et al 2019). This helps isolate out key methodological contributions, for example no teacher forcing in structure, in an easy-to-interpret way.\n- The coarsening procedure is of wider applicability in protein modeling beyond sampling CDR loops and an ablation study demonstrates its value empirically for this work.\n- Baselines chosen highlight key interactions in RefineGNN components and show that the model appears to effectively leverage the added complexity.\n- Evaluations are thorough and span a number of datasets, one of the biggest strengths of this paper.\n\nWeaknesses:\n- The use of a predictor to evaluate neutralization is justified based on very recent work, and it is unclear that this practice is in line with broader norms in the antibody engineering community.\n", "summary_of_the_paper": "This paper introduces a method for jointly sampling structure and sequence of antibody loops in an iterative fashion that constrains structure generation less. ", "summary_of_the_review": "This paper provides an interesting study of antibody loop generation with both novel methodology and extensive empirical evaluation. This combination of strengths makes it an excellent paper that should be of wide interest.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "dvZyS6Bu-zk", "reviewer": "Reviewer_9GqB", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies the important problem of computationally designing antibody CDRs. The joint modeling approach of structures and sequences is novel and interesting. The generation method is flexible and the authors also adapt it for conditional generation (conditioned on the rest of the antibody and on given properties of interest with the predictive model already available).\n\nThe empirical results are convincing. Both perplexity and sequence recovery are standard metrics in protein design, and the proposed method showed improvements in both perplexity and sequence recovery on two separate data sets. The two baselines used for comparison are well described and meaningful.\n\nRequest for clarification: In section 4.2, it is not immediately clear whether the antigen is also included in the conditioning or only the antibody itself.\n\nIt would be appreciated if the authors can add more discussion around when this proposed method applies in practice. For example, the method requires already knowing the frame of the antibody. When is a fixed frame variable CDR-H3 design a reasonable assumption? Or, when can we realistically expect to have accurate predictors for properties of interest?\n\nWhile the model is trained on unbound antibody structures, the second evaluation task is conditioned on the antibody structures in bound state within a complex (if I’m not mistaken), will there be a mismatch between the bound and unbound structures? Clarification and discussion around this point would also be appreciated.\n", "summary_of_the_paper": "This paper proposes a joint generative model (co-designing the sequence and the structure at the same time) for the CDRs of antibodies, with the goal to enhance binding specificity or neutralization capabilities. \n\nThe proposed method and two existing baselines are evaluated on 1) perplexity on hold-out set; 2) perplexity and sequence recovery on known antigen-antibody complexes; and 3) redesign CDR-H3 of existing antibodies for coronavirus neutralization as measured by a given neutralization predictor. RefineGNN, the proposed method, shows improved performance in all three tasks.", "summary_of_the_review": "Interesting approach to an important problem. Convincing empirical results and baselines. The approach might be limited to specific use cases in practice depending on the availability of a predictive model for the properties of interest and the knowledge of the antibody frame.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "EKF57rZZivx", "reviewer": "Reviewer_meMq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The paper is well-written and easy to follow. \n- The proposed refinement method has high novelty and outperform state-of-the-art baseline methods. \n\nWeakness:\n- The experimental evaluation may be problematic, it is not convincing to use machine learning methods to predict the neutralization ability based on CDR H3. ", "summary_of_the_paper": "The paper proposed a deep generative model named iterative refinement graph neural network to generate antibody CDR for Y-shaped antibodies. Specifically, it sequentially generates the CDR residue sequence and refines the global structure iteratively. Empirical results show superior performance compared with baselines. ", "summary_of_the_review": "The proposed method is novel and the paper is well written, validated by thorough empirical studies. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes use of a novel generative modelling approach, over both sequences and structure of proteins, to co-design the CDR region of antibodies so achieve good binding/neutralization. The reviewers are in agreement that the problem is one of importance, and that the technical and empirical contributions are strong. There are concerns over the relevance of evaluating the method by using a predictive model as ground truth. Still, the overall contributions remain.
Decision: Accept (Spotlight)