AbDiffuser: full-atom generation of in-vitro functioning antibodies
Authors: Karolis Martinkus
Jan Ludwiczak
WEI-CHING LIANG
Julien Lafrance-Vanasse
Isidro Hotzel
Arvind Rajpal
Yan Wu
Kyunghyun Cho
Richard Bonneau
Vladimir Gligorijevic
Andreas Loukas
Venue: NeurIPS 2023 spotlight (2023)
Keywords: antibody generation
diffusion
equivariance
TL;DR: We propose a new and more efficient diffusion model specifically tailored for antibody generation. Generated samples are confirmed to bind in-Vitro.

Abstract: We introduce AbDiffuser, an equivariant and physics-informed diffusion model for the joint generation of antibody 3D structures and sequences. AbDiffuser is built on top of a new representation of protein structure, relies on a novel architecture for aligned proteins, and utilizes strong diffusion priors to improve the denoising process. Our approach improves protein diffusion by taking advantage of domain knowledge and physics-based constraints; handles sequence-length changes; and reduces memory complexity by an order of magnitude, enabling backbone and side chain generation. We validate AbDiffuser in silico and in vitro. Numerical experiments showcase the ability of AbDiffuser to generate antibodies that closely track the sequence and structural properties of a reference set. Laboratory experiments confirm that all 16 HER2 antibodies discovered were expressed at high levels and that 57.1% of the selected designs were tight binders.

AbDiffuser: Full-Atom Generation of in vitro Functioning Antibodies
Karolis Martinkus', Jan Ludwiczak', Kyunghyun Cho Wei-Ching Liang",
Julien anfrance-Vanaseee Isidro Hotzel?, Arvind Rajpal', Yan Wu², Richard Bonneau',
Vladimir Gligorijevic', Andreas Loukas' Prescient Design, Genentech, Roche Antibody Engineering, Genentech, Roche "NYU
Abstract
We introduce AbDiffuser, an equivariant and ohysics-informed diffusion model for the joint generation of antibody 3D structures and sequences. AbDiffuser is built on top of new representation of protein structure, relies on novel architecture for aligned proteins, and utilizes strong diffusion priors to improve the denoising process. Our approach improves protein diffusion by taking advantage domain knowledge and physics-based constraints; handles sequence-length changes; and reduces memory complexity by order of magnitude, enabling backbone and side chain generation. We validate AbDiffuser in silico and vitro. Numerical experiments showcase the ability of AbDiffuser to generate antibodies that closely track the sequence and structural properties of a reference set. Laboratory experiments confirm that all 16 HER2 antibodies discovered were expressed at high levels and that 57.1 % of the selected designs were tight binders.
1 Introduction
We focus on the generation of immunoglobulin proteins, also known as antibodies, that help the immune system recognize and neutralize pathogens. Due to their potency and versatility, antibodies constitute one of the most popular drug modalities, with 10 out of 37 newly FDA-approved drugs in 2022 being immunoglobulins (58). The ability generate new antibodies with pre-defined biochemical properties in silico carries the promise of speeding up the drug design process.
Several works have attempted to generate antibodies by learning to form new sequences that resemble those found in nature 21 19 82). An issue with sequence-based approaches is that it is hard to determine the properties that render a protein a functional molecule (and an antibody potent drug) without inspecting a 3D model of the functional state such as an interface or active site. So far, almost all of the first design methods that have enabled novel protein design used carefully curated structure- function information to score the designs 45] 48 30 The determinant role of structure on function
has motivated numerous works to co-design sequence and structure 4
24
89
34
43
[1 to
first design the protein backbone and then fill in the amino acid identities
90
32
15
8
50
92
An emerging paradigm for protein generation is that of equivariant diffusion 900
Protein diffusion models combine ideas from equivariant and physics-informed protein structure representa-
39
51
tions (37 with advances in denoising diffusion probabilistic models (28) to gradually transform noise to partial structure. The generated structure is then refined by adding side chains and optimizing atom positions to form full-atom 3D model of protein.
pertinent challenge when generating protein structure satisfying the appropriate equivariance and physics constraints while also balancing modeling complexity and fidelity. Most current protein models rely on equivariant transformers
73 or graph neural networks [1] 43 to satisfy SE(3)
Correspondence to Andreas Loukas Candreas www.kae@rocee..com.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

equivariance and typically represent parts of the protein geometry in angle space. The latter recipe can be set up to build a physics-informed model that respects (many) desired constraints but comes with increased complexity. As a consequence, these models are expensive to train and are often applied to partial protein, ignoring the side chain placement 43 51 or even deferring the determination of amino acid identity to a later stage [90] [2). Alternatively, some works focus on the regeneration of perplleentary-ddtrmmmiiii
regions (CDRs) of antibodies that are of particular importance to 53 78 44], which also helps to reduce the complexity of the problem.
binding (4 43 33
Our work is motivated by the observation that key large protein families, here we focus on the antibody protein family, typically have strong properties, such as an ability to be mapped to a reliable sequence ordinate via sequence alignment. Our main contribution is an equivariant diffusion model called AbDiffuser that is designed to exploit these properties. We show that incorporating family-spp -upecf-speccfic priors into the diffusion process significantly improves generation efficiency and quality.
AbDiffuser relies on a new universal SE(3) equivariant neural network that we call the Aligned Protein Mixer (APMixer). In contrast to existing equivariant architectures used for antibodies and proteins, APMixer models residue-to-residd
relations implicitly and is particularly effective in handling sequence length changes. Additionally, its significantly smaller memory footprint makes it possible to generate full antibody structures, including framework, CDRs, and side chains. Our approach to residue representation is made possible by a projection method guaranteeing that bond and angle constraints are respected while operating in coordinate and not angle space, a better match to diffusion with Gaussian noise. We also benefit from the effects of overparameterriztion by scaling the network size to hundreds of millions of parameters on single GPU; an order of magnitude improvement over the corresponding (E)GNN architectures. Having powerful model that generates the full antibody structure is shown to be beneficial for the quality of the designed proteins.
We evaluate AbDiffuser on the generation of antibodies from paired Observable Antibody Space (pOAS) [61], modeling of HER2 binders and antigen-conditioned CDR redesign. Numerical experi- ments demonstrate that the proposed representation, architecture, and priors enable AbDiffuser to better model the sequence and structural properties of natural antibodies. We also submitted a subset of our proposed HER2 binders for experimental validation ted, 100% were expressed as actual antibodics and 37. 5% bound to the target with an average pKD of 8.7. Of these, the subset containing raw/filtered samples achieved a binding rate of 22. 22.2% 1% and an average pKD) of 8.53/8.78 78, with our tightest binder slightly improving upon the affinity of the cancer drug Trastuzumab. These results provide the first experimental evidence that a generative model trained on mutagenesis data can reliably (with high probability) create new antibody binders of high affinity, even without post-selection by learned or physics- based binding predictors.
laboratory. Of the 16 samples submit-
Due to space limitations, we refer the reader to the appendix for a detailed discussion of related work, for method details including proofs, as well as for additional numerical results.
2 Denoising Diffusion for Protein Generation
This section descrihes how we utilize denoising diffusion to generate antibody sequence and structure.
The ideas presented are influenced by previous work on denoising diffusion [26 5. 28 The main contributions that distinguish our work are presented in Sections and
We adopt the denoising diffusion framework in which, given a data point Xo. the forward diffusion process gradually adds noise to form corrupted samples X.. These samples form a trajectory (Xo. Xi..., Xt, .... Xrr) of increasingly noisy data, interpolating from the data distribution X,- ~ P(X,) that of an masy-to-sample prior Xx ~ p(Xx) such as a Gaussian. The process is constructed to be Markovian, so that g(X., Xr (X) = 4(xx (o) 2XX
To generate new data, a neural network learns to approximate the true denoising process Xn o(x,,t) t), which can be achieved by minimizing the variational upper bound of the negative log- likelihood [84] [6. In our diffusion process, we factorize the posterior probability distribution over the atom positions and residue types:
q( (X, (x,) = ( xx q( Tes xx=)),
where Gaussian and a categorical distribution govern the noise applied to atom positions and residue types, respectively. We cover these in detail in Sections B. 2 and B.3 The reverse diffusion process is

---

joint: the model jointly considers atom positions and residue types (Xx0 .Xx"") = ((x, . .t). Throughout the paper, we use P E R=x3 as matrix of antibody atom positions and X R as matrix of one-hot encodings of residue types (20 amino acids and a gap).
3 Aligned Protein Mixer
We present Aligned Protein Mixer (APMixer), a novel neural network for processing proteins from a family of aligned proteins. In particular, we focus on antibodies, although the method can in principle be applied any sufficiently large protein family by using a family-specific global alignment [8T]57]. We first explain how ensure SE(3) equivariance and how account for variable sequence lengths in Sections 3.1 3.2 The model is detailed in Section 3.3 and our approach to incorporate physics-based constraints bond angles and lengths is described Section 3.4
3.1 SE(3) Equivariance by Frame Averaging
can be made equivariant or invariant to group transformations by averaging the model outputs over a carefully selected subset of group elements called frames F(X) [8]. For example, this has been used successfully to make equivariant self-attention models for the prediction of protein-ligand binding energy [35]
Any neural network o whose input X has dimension
We achieve the desired equivariance to rotations and translations (the SE(3) group) as follows:
X
Σ
6(X R X 1t,
[F(XP)))
(RX())F(X))))
where 1 X is the centroid of our points and the four canonical rotation matrices R forming the four frames F(XP=)) C SE(3) needed to achieve equivariance can be determined based on Principle Component Analysis. More specifically, we obtain three unit-length eigenvectors v1, U2, f3 corresponding to the eigenvalues \1. d2, As from the cigendecomposition of the covariance matrix C' (X 1t) (X 1t) E R3x3 and define the four frames as
F(XP** (lav1, Bog, 001 X Bu2), |a, 3 E (-1,
To respect the fixed chirality of proteins observed in humans, we desire equivariance w.r.t. SE(3) and not E(3) which also includes reflections. As such, when constructing frames the third axis sign is not varied but its direction is determined by the right-hand rule (cross product of the first two axes).
SE(3) invariance can be similarly achieved:
res
Σ
((X RO 1t, X(s)
[F(X)0)]
(XX)))F(X"")
We make use of equation 2 when denoising residue types X as they are invariant to the rotation of the antibody, and equation T for the prediction of atom positions X
3.2 Handling Length Changes by Multiple Sequence Alignment
The version of APMixer investigated in this work is built on top of the AHo antibody residue num- bering scheme proposed by Honegger and Plückthun 227 This numbering scheme was constructed in data-driven fashion using information from known antibody structures. For each residue in antibody chain, assigns an integer posítion in [1, 149) based on the structural role of the residue (c.g. being in a particular CDR loop or in the framework region between two particular CDR loops). Essentially all known antibodies fit into this representation [27].
As are modeling paired antibody sequences (the heavy and light chain), the full representation is 2 x 149 element sequence, where 149 heavy chain elements are followed by 149 light chain elements. We represent AHo gaps physically as 'ghost' residue placeholders; their position is determined in data pre- rre-processing by linearly interpolating between the corresponding atoms of the nearest existing residues (trivial, due the use of AHo numbering).
Our experiments confirm that the proposed representation consistently improves generation quality. The reason is two-fold: a) Each antibody chain is now represented as fixed-length sequence with
3

---

149 positions that are either empty gaps or are filled with an appropriate amino acid. This fixed-length representation encompasses antibodies with diverse loop lengths via alignment and thus allows our generative model to internally choose how many residues the generated protein will have. In contrast, many of the non-sequence-based protein, small molecule, and graph generative models [4] 28 90 [2] require the number of elements in the object to be specified beforehand. b) The sequence position directly implies the structural role that the amino acid needs to perform, which makes it easier for the model to pick up structure-specific rules.
3.3 The APMixer Architecture
Our use of fixed length representation for the immunoglobin variable domain fold family (Sec- tion 3.2) also allows us to forgo traditional architecture choices in favor of a more efficient architecture inspired by the MLP-Mixer [86]. We build the model architecture out of blocks, where each block consists of two MLPs, that are applied consecutively on the columns and the rows
,, x W2p LayyrrNorm )) all j E [c] X,, = X., + Wap(W Layerroomm( Xs,-))for all i e [r].
of the input matrix X IRTx, with p being an activation function, and using the notation [k] (1..... k). We define the model input as a fixed-size matrix X combining X and x with one sequence element per row (2 x 149 rows in total). In each row, we encode the residue type and all of the atom positions for that residue (e.g., the C, Ca N, Cg backbone atoms). These input matrix rows are embedded in higher-dimensional vectors using an MIP. Specifically, using this representation, our input matrix X has = 2 x 149 rows and 21 + 4 x 3 columns.
To achieve equivariance atom translations and rotations, we can either apply frame averaging (Section 3.1) on the whole model or on each AbDiffuser block individually. We chose the second option in our preliminary experiments this improved performance. Frame averaging can be applied on high-dimensional embeddings simply by splitting them into three-dimensional sub-vectors and using each to compute the SE(3) frames [68]. To account for that residue types are invariant to Euclidean transformations, while the atom positions are equivariant, we split each block's ' input and output vectors in half, with one half being treated equivariantly and the other invariantly.
Model complexity and -uiiesiiiyysslltyy. APMixer models pairwise interactions implicitly by operating on rows and columns of the input interchangeably. Thus, its memory complexity grows linearly with the number of residues. This contrasts with the usual quadratic complexity of traditional structure-basd models and allows us to do more with a fixed run-time, parameter, and/or memory budget. In Appendix G we prove that the model on top of which APMixer is built is SE(3) meaning that it can approximate any SE(3)-equivariant function.
We also remark that, in principle, other models, such as a ID CNN or a transformer, could be used in place of the MLPs in APMixer. With such vequence-lengthe Jength-independent models, we would no longer require a multiple sequence alignment of the given protein family, though this would possibly come at the expense of universality and efficiency.
3.4 Physies-informed Residue Representation by Projection
Atoms within a protein adhere to strong constraints. In principle, a neural network trained with enough data can learn to respect these constraints. However, over fixed data budget, it can be advantageous to construct model in a manner that guarantees that its outputs never violate the constraints. Previous work commonly represents proteins in terms of rotation and translation of rigid residue frames and uses idcalized residue representations to recover atom positions [37 73). Although there is a long tradition of modeling backbone and side chain degrees of freedom in angle space 66 14, operating in angle space adds modeling complexity and makes diffusion potentially inaccurate 194 g 90). We take a different route and devise a way to respect bond constraints while operating in a global coordinate frame, which works seamlessly with standard Gaussian diffusion for atom positions.
Specifically, inspired by interior- prrntt methods 72 that alternate between optimization and projection steps, we design a new non-parametric projection layer that is applied to both model inputs and outputs. The model and the noising process are allowed to move the atoms freely, and the projection layer then corrects their positions such that the constraints are respected.

---

Backbone residue projection. We use reference residue backbone (C, N. with idealized bond lengths and angles [7]. As the backbone atoms are rigid, we rely on the Kabsch algorithm [8] to identify the optimal voto-translation between the projection layer's s input and the rigid reference residue's 8 atom positions. We apply the transformation to the reference residue and output the resulting atom positions. We also ensure that the distance between the corresponding C and O atoms in the output is 1. 1.231Awhile staying close possible to the input O position. The reference residue is also used to represent the AHo ghost residue atoms. Idealizing a backbone in this way usually results
in negligible RMSE the original structure of 5 10 -A
Side chain projection. We employ a similar idea to constrain the placement of side chains. In contrast structure prediction 37: 46 we cannot use one ide- alized side chain per amino acid, since the sequence is unknown during generation. Our solution is to con- vert all side chains enough degrees of freedom to account for the exact placement of all atoms. The amino acid side chains have up to 4 bonds that they can rotate around by dihedral angles. These degrees of freedom can be captured by constructing side-chain template that consists of 4 pseudo-carbon atoms, for which the dihedral angles are rotated in the same way as for the original side chain. If the original side chain has fewer degrees of freedom, we simply set the corre- sponding dihedral angles to 180 such that the atoms forming the dihedral angle lie on plane. The pro- jection layer then only has to ensure that the bond lengths between the pseudo-atoms are respected. We set the bond length to 1.54A because carbon atoms are the most common atoms that form the dihedral angles of real amino acids. This representation can be seen in Figure [ To recover the full-atom structure, extract the dihedral angles from the side chain template; the angles are then applied to idealized amino acid-specifie templates.
Lysino
Tyrosine
Valino
Ful Ful-atom vepersentatoo
generic representation with
Gentric representaioon
Figure The proposed internal generic side chain representation. The dihheral-defining atoms (orange) from the full-atom represen- tation (top) are used to construct a generic four-atom representation (bottom). If the side chain has fewer than four angles, additional atoms (gray) are placed in the generic side chain to correspond to 180 angle. The full atom representation recovered by apply- ing matching rotations to appropriate side chain template.
Informative Diffusion Priors
has been hypothesized that diffusion processes for generation should be tailored to the particular task to increase performance [8 [5. In Appendix E.I]] we present a theoretical argument confirming that choosing good prior reduces the complexity of the learnable model needed to achieve good- quality denoising. Our bound stated in Theorem E. reveals that one cannot learn simple generative model that fits the data well unless the prior has a small Wasserstein distance to the data distribution.
Armed with this knowledge, we next introduce two types of priors that are incorporated into our diffusion model: a) position-specific residue frequeeciess that describe sequence conservation of the immunoglobulin fold, and b) conditional dependencies between atom positions.
4.1 AHo-specific Residue Frequencies
A separate benefit of having fixed-length sequence representation (see Section (.2) is that we can use the residue frequency each position a diffusion prior. To do this, we estimate marginal position-specife
categorical distributions Q',..., Q2- over residue types from the training data and use these to define the noise in the discrete residue-type diffusion (see Appendix [B3)). Since AHo aligns residues based on their structural role within the protein, the estimated prior distributions exhibit significant variability across positions and have low entropy preserved regions of the immunoglobulin fold. In this way, the noise x(Xp* X7()) = Cat( X (3, + (1 {)Q')) at every step of the forward process depends on the residue position in the fixed-length sequence representation. Gap frequencies are also captured in these noise matrices.
5

---

4.2 Encoding Conditional Atom Dependencies
It is a consequence of their chain-like structure that neighboring residue atom positions of proteins are strongly correlated. Theorem E. suggests that encoding these correlations within the diffusion process eases the denoising complexity and can free the model to focus on more challenging aspects.
To this end, we capture the conditional independence relations between atom positions by a Gaussian Markov Random Field (GMRF) 42 The latter corresponds to a Gaussian (0, Σ) over atom positions whose precision matrix Σ = I. + aT is equal to the shifted graph Laplacian I, = diag(A1) - A associated with the adjacency matrix A. The GMRF operates under the assumption that node features (e.g., 3D positions) for which there is no edge in the adjacency A are conditionally independent. Some concurrent works 32 36 also considered conditional atom dependencies by hand-crafting correlation matrices that capture chain relations between residues. We take step further by proposing to automatically learn the sparse conditional dependence relations from the training data. Our approach entails estimating a sparse adjacency matrix A that captures the data variance under the GMRF model. The details are described in Appendix F
Experiments
We evaluate AbDiffuser's ability to design antibodies. After describing the experimental setup and metrics, Section 5.T presents in silico tests illustrating the effect of our proposed changes on the generation quality. Section 55.2 in vitro results showcasing the ability of our method to design new expressing antibodics that bind to a known antigen. Further analyses can be found in the K and D. whereas experiments on CDR redesign in SAbDab are in Appendix N
Appendices
74 whose output is folded to recover the structure; b) an E(n) Equivariant Graph Neural Network (EGNN) 75 which a popular choice for tasks such as 3D molecule generation (28 antibody CDR loop inpainting 43 and antibody generation [89]; and c) FA-GNN [68], corresponding to a standard GNN with SE(3) equivariance attained by frame averaging. We also evaluate the proposed informative antibody-specifie priors using all of these architectures. To ensure the comparison is performed fairly and under similar settings, we always use the projection layer and our diffusion implementation only varying the denoising model architecture. In the OAS task, also compare our diffusion-biie based approach with d) the lgLM [80] antibody language model. To ensure that it generates paired OAS-like sequences, condition the generation on the subsequences of the first and last few amino acids taken from POAS sequences (something that our approach does not need). It is also important to note that the comparison is not fair, since the publicly available IgLM model was trained on 558M sequences that also include the whole paired OAS (105k sequences) and the test set we use. So, in many ways, IgLM's performance represents the best results we could ever hope achieve with sequence-only approach. We also compare with e) dyMEAN (44), which is the only other antibody generative model previously shown to be able jointly generate the full strucutre and sequence. In
Baselines. We compare APMixer with: a) a sequence transformer based on BERT 12
the binder generation task, we compare with f) RefineGNN [34), g) MEAN 43 and h) DiffAb [53
three state-of-the-arr geometric deep learning methods for CDR redesign.
Metrics. The quality of generated sequences is measured in terms of their naturalness (inverse perplexity of the AntiBERTy (74) model), closeness to the closest antibody in the training set in terms of edit distance, and stability estimated by IgFold. We also verify that the generated antibody sequences satisfy the appropriate biophysical properties using four additional structure- based metrics [69]: CDR region hydrophobicity (CDR PSH), patches of positive (CDR PPC), and negative charge (CDR PNC), and symmetry of electrostatic charges of heavy and light chains (SFV CSP). The metrics applied to generated structures focus primarily on the estimated free energy AG using Rosetta (2] and RMSD for backbone heavy atom positions as compared to IgFold [73 predictions. More details can be found in Appendix [
As we want to estimate how well the entire data distribution is captured, in all cases except RMSD, we report the Wasserstein distance between the scores of the sequences in the test split and the scores of the generated sequences. As reference, we also report the baseline metries achieved by the sequences and structures in the validation set. A generative model that approaches or matches these values is effectively good modeling the distribution of the specific metric in consideration as ii.d. sampling. The test set and the generated set always have 1000 examples.
Further details on training and implementation can be found in Appendix M

---

Model
W; (No.)))
[Cle)]
(S
(((PSH) J
[[(PPC)
W W((NC))
W((CSP)
WI(AC)
RMSD
Vulidotion Set
0,0150
0,0043
0,0022
0.8301
0.0441
0.0176
LO014
Transormer
0,5308
0.4400
1. 1.2284
25,8265
0.2324
0.2278
2.7925
Transformer (AHlo)
0,4456
0.3474
0.5351
6.4190
0.1641
0.0593
2.3472
1gLM [0
0.1103
0.0484
0.0677
11,0675
0.0413
1.9274
MyMEAN [4]
0.1319
0.1600
0.0423
3.9145
0,1566
0.2929
2.3711
601.1153
3.8157
EGNN
0,3988
0.2655
0.3547
2.1115
0.1486
0.1085
1.9881
1586.0160
9.8231
EGNN (AHo)
0,3329
0.2229
0,2904
1620
0.1263
0.1075
0,7978
1714,2734
10,0628
EGNN (AHo Cav.)
0.3482
0.2374
0.2443
2.5632
FA-GINN
B.4141
0.2822
0.4302
2,5330
0.1696
0.1164
1.7896
22.7988
0.8617
VA-GNN (AHi))
0,3407
0.2263
0,2344
2.3272
0,1411
0.1306
1,016
8,7506
0.8321
FA-GNN (AHo & Cow)
0,2785
0,1669
0,08115
5.4440
0,0493
0.0212
0,7768
15,3670
0.8814
AbDilTuser
priun)
0.2837
3.1364
1,3874
38
AbDiTliser (80 projection)
8.2378
0.1529
2.3530
0.0637
0,0793
0.7376
6313,2495
11,1431
ANDifliser (no Cov.)
0,2309
0.1107
0.1235
1.2392
0.0664
0,0511
0.6483
17,7322
0.6302
AbDiffuser
0,1979
0.0921
0.0662
2.3219
0,0314
0.0285
0.6662
13,3051
0.5230
AhDiffuser (side chains)
0,0916
0,0520
0,0186
6.3166
0,0209
0,0754
0.8676
16.6117
0.4962
Table 1: Antibody generation based on Paired OAS 61 AHo denotes models that AHo numbering and position-specific residue frequencies. Cov denotes models that use the learned covariance. IgLM is denoted by since it trained significantly more data (including the test set) and was given part of the sequence to bootstrap generation. The top three results in each column highlighted as First, Second, Third.
Model
Parameters 1
Memory (training) | Memory (gencration)
Generation time J
Transformer
84M
14GB
15GB
3.2 min
EGNN
39.3M
78GB
16GB
22.6 min
BA-GNN
9.4M
75GB
38GB
9.5 min
AhDiffuser
169M
12GB
3GB
2.3 min
Table 2: Number of parameters, model memory consumption during training with a batch size of 4 and memory consumption with the time taken to generate a batch of 10 examples for paired OAS.
5.1 Paired OAS Generation
We focus on matching the distribution of 105k paired sequences from the Observed Antibody Space database [6]] folded with IgFold and optimized with Rosetta
Table summarizes the results. Generated samples by AbDiffuser improve upon baselines on nearly all fronts, even compared to the IgLM language model which was trained magnitudes more data (especially when concerning structure-related metrics). dyMEAN GNN-based model struggles in this distribution modeling task and is the only model tested that does not achieve the perfect uniqueness of the generated samples (58. 2% unique). The experiment also corroborates our analysis (Theorem) [..] on the benefit of informative priors to diffusion: using a position-specifie residue type frequency (AHo) and encoding conditional atom dependencies through a learned covariance matrix (Cov.) helps to improve the ability of most models to capture the pOAS distribution. Interestingly, including the learned covariance can sometimes noticeably improve the quality of the generated sequences (FA-GNN), but its strong benefit to structure quality is only felt when the model is powerful enough to model the structure well (APMixer). Inclusion of AHo numbering and position-specife frequencies improves all models. We perform similar ablation for APMixer by setting the prior distribution to uniform (uniform prior) and observe a similar performance drop.
To interpret the fidelity of the generated structures, we recall that IgFold uses an ensemble of models followed by Rosetta optimization and that, on average, individual IgFold models (before ensembling) achieve an RMSD of 0..2399 on the test set. Therefore, in this regard, the structures created by AbDiffuser are nearly indistinguishable from the test set structures (RMSD of 0.4962). more detailed analysis of per-region RMSD can be found in Appendix K We further observe that when no projection layer is used and instead one uses standard diffusion to predict the noise added to the atom positions (28] the training becomes less stable and the model can fail learn to generate good structures. Encouragingly. forcing AbDiffuser to model side chain positions alongside the backbone tends to improve the similarity of the generated sequences (Naturalness, Closeness, Stability). This is likely due to the strong coupling between feasible side chain conformations and residue types. The generated side chains get an average Rosetta packing score of 0.624, whereas folded and Rosetta-optimized structures have mean packing score of 0.666. Recalling that a packing we deduce that AbDiffuser able to generate physically plausible side chain conformations. When available, also use the side chain positions
score above 0.6 is widely considered good [77] 66 R
7

---

Modell
3
WijClo]]
RMSD
Unoq.
Notifutton Ser Baseline
0,9071
3,0002
0.0052
1.2783
0.0296
0.074
0.3280
2,0500
0.8676
100%
MEAN
0,0022
0.00
0:0267
3,4144
0,0184
040231
04108
8,5951
0,7992
0.7767
58 9%%
DifAAb
N3
0.0244
0.0044
0,0496
0,5451
0,0097
0
3-6647
6.7419
0.151
0.8876
99.7%
ReineONN
0.0011
0J0004
0.0026
0,5482
0,0053
0,0846
0.1360
0,7132
100%
TranoGetaa (Afoe)
0.0344
2
0.0097
1.3681
0.017L
0.0140
0.2657
0.3627
1005
EGNN (AHo & Coul
0.0013
0U0030
0.0002
1.1241
0.013
0,0236
D.241
1967.2280
9.2180
0.3626
100S
FA-CINN (AHf Core)
0.0018
0.010
0.0061
0.418
0.0107
0.0056
0.7646
3.18K0
0.4766
100%
AbDiifracs
0.0213
3.0018
0.0035
0.0205
D.1588
6.4301
0.5761
1005
AbDiffieer (side chains!
0,0010
0J0005
0,0062
1.2909
0.0115
0.0029
32,0464
0.4046
0.5848
100%
AhDiffmeer
0U001
0.0148
100PS
Ahthffrmor (sc.. 0.75)
0,0126
0.3953
AbDiiliiass
0.0065
2.5964
0.0206
AbDilfiass
0.0015
2.0024
0.0159
1.5043
0.0210
0.5179
114.4843
0.6795
Table 3: Generating Trastuzumab mutants based on the dataset by Mason et al. (55] The top three results in each column are highlighted as First, Second, and Third. Multiple approaches can generate sequences similar to the test set, but generating predicted binders (large Poina) is considerably harder.
predicted by the model for the AG energy estimation. Even though this procedure is expected to generate results slightly different when compared to backbone-only models (in the latter case missing side chain atoms are first repacked based on the rotamer libraries before the energy minimization step), we still observe a high overlap between the resultant energies. This further highlights the quality of side-chain prediction.
It should be noted that AG energy computation and some additional structure-basee metrics [69 (i.e., CDR PSH, CDR PPC, CDR PNC, SFV CSP) are inherently susceptible to even minor changes in the geometry of the modeled structures. Thus, in line with the discussions by Raybould et al. [69], the overal trends of these metrics can be used to assess the generated samples as similar or dissimilar to the reference distributions, but one should not fall into the trap of overly focusing on the specific values attained. From this perspective, most structure-based models do sufficiently well on these metries, perhaps with the exception of EGNN AG.
In Table 2 we show that APMixer is able to use an order of magnitude more parameters with a smaller memory footprint during training and offers more efficient sample generation, compared to the baseline architectures, on Nvidia A100 80GB GPU.
5.2 Generating HER2 Binders
Antibody generative models can be used to explore a subset of the general antibody space, such as the binders of the target antigen. The purpose of modeling and expanding set of binders is twofold: a) it allows us to rigorously validate our generative models in a setup more tractable than denovo design; b) from practical standpoint, it sets the ground for the optimization of properties that render binding antibodies drugs, such as developability, immunogenicity. and expression, allowing efficient exploration of the binder space 121 63]. Note that sufficiently large libraries consisting of antibodies of variable binding affinity are usually discovered during the very early stages of the drug design process by means of high-throughput experiments (64 76 55 Thus, the data used here can be sufficiently easier to obtain than the crystal structures usually assumed in CDR redesign experiments.
We use the Trastuzumab CDR H3 mutant dataset by Mason et al. (55] which was constructed by mutating 10 positions in the CDR H3 of the cancer drug Trastuzumab. The mutants were then evaluated using high-throughput noisy assay determine if they bind the HER2 antigen. After discarding all duplicate sequences and sequences that appear in both the binder set and the non-binder set, are left with 9k binders and 25k bindess. The generative models were trained only on the binder set. Separately, we train a classifier based on the APMixer architecture to distinguish binders from non-binders. The classifier achieves 87.8% accuracy on a randomly selected test set of 3000 antibodies, implying that the predicted binding probability is a somewhat informative metric for how tightly the binder distribution is approximated by the models.
The computational results summarized in Table 3 evidence that AbDiffuser closely approximates the binder distribution. Although most models achieve results close to those achieved by the validation sef]] the sequence-based baseline struggles to capture the features that convey binding; evidenced by the low predicted binding probability. GNN-based baselines struggle to predict the correct structure
'The Wasserstein distance of the validation set is likely overestimated here due to the validation set being smaller than the test set (100 vs 1000 samples).
8

---

(RMSD above 3) producing structures with unreasonably high Amber energy. As a result, the binding probability of their designs is lower than that of the more powerful structural models.
Next, we look at the two baselines that only redesign CDR H3 instead of generating the whole antibody. RefineGNN 34 manages to closely capture the sequence and biophysical characteristics and generates sufficiently good binders. DiffAb [53]. which is another diffusion-bascd model, achieves the best binding probability out of the CDR redesign baselines. MEAN 43 generated few unique sequences (38. 9% uniqueness) and failed some of the distribution-base metrics (Naturalness, CDR PSH). dyMEAN 44 collapsed to always generating a single sample, thus we did not report its results. The overfitting behavior of MEAN and dyMEAN can be attributed to the reconstruction-basst training objective and the fact that they were designed for slightly different task of CDR redesign with different antigens, instead of the single antigen we have here.
Due to the use of iterative refinement and reconstruction- training objectives, RefineGNN and MEAN focus on the generation of the most likely examples. Focusing on high-likelihood modes is especially beneficial here, the experiment that was used to create the dataset is noisy. Diffusion models can also be adjusted to focus on prevalent modes by reducing the temperature T of the denoising process. In the Gaussian case, we specified the temperature as the scaling factor of the noise added during the reverse process (Appendix B.2), whereas in the discrete case we specified it as the temperature of the model output softmax function (Appendix B.3). We observe that slight reduction of the temperature helps to improve the general distribution fit across almost all metrics. Reducing the temperature further boosts the binding probabilitty, but, as expected, can result in a slight loss of diversity. Using a higher temperature slightly increases the Stability Wasserstein distance while improving Stability. The phenomenon occurs because the model no longer concerned with fitting low- wwwllikelhood modes of the real distribution that contain structures of poor stability.
In contrast to MEAN, RefineGNN and DiffAb, which only redesign the CDR H3 of the test-set structures, AbDiffuser generates full antibodies and still achieves better sequence similarity. MEAN 43 also produced structures of noticeably worse RMSD, which can be explained as follows: a) as we see in Appendix K MEAN does not predict CDR H3 positions as well: b) changing CDR H3 can impact the most-likely conformation of the overall antibody. something that the CDR inpainting models cannot account for. We do not report the RMSD and AG for RefineGNN as it does not place the generated CDR H3 loop the reference frame of the original antibody.
The AbDiffuser model that also generates side chains generally achieved better sequence similarity (Naturalness, Closeness) and better binding probability than the backbone-only model, but worse similarity of sequence stability. Furthermore, while the side chain model achieved a worse overall structure quality (AG, RMSD). as we see in Appendix K it predicted CDR H3 positions more precisely, which is the main desideratum in this experiment.
5.3 In Vitro Validation
We further validate our designs through an in vitro experiment. As shown in Figure 2 all submitted designs were expressed and purified successfully (average concentration of 1.25 mg/ml) and an average of 37.5% of the designs were confirmed binders with pKD E [8.32, 9.50] (higher is better) whose average was 8.70. The binding rate was improved (from 22. 2% to 57. 1%) when considering designs that were additionally filtered so that they were better than the bottom 25th quantile in every metric (naturalness, RMSDD etc.) and a classifier trained to distinguish binders from non-binders predicted that they bound with high confidence. This increase in binding for filtered samples suggests that our selected metrics indeed correlate with desirable in vitro characteristics. Our best binder belonged to the latter set and its pKD of 9.50 was slightly above Trastuzumab (not in the training set) while differing in 4 positions of the CDR H3 loop. Further examination of the top binders is performed in Figure 3 and Appendix
L
In contrast to the only other comparable study for ML-based Trastuzumab CDR H3 mutant design by Shanehsazzadeh al. (76), our best binder had pKD = 10 = 9.50, while the best binder found by Shanehsazzadeh et al. had pKD = 9.03. Two important differences between the two studies are that: we trained on binders and tested 16 samples in the wet lab, while Shanehsazzadeh et al. used simpler sequence infiling model trained on a large set of generic antibodies to generate 440k candidate sequences that were filtered in the wet-lab using high-throughput screening to identify 4k binders, of which 421 were selected to be tested using precise SPR measurements. The fact that our
9

---

60
57.1%
puriicctton concentration (mo/m))
1.4
94-
50
1.2
0.2-
(%)
40-
0.8
30
binding
22.2%
20-
8.5
0.2
84-
10
Fitereed
fitered
filtered
Figure 2: vitro validation of AbDiffuser designs terms of their ability to express (left), binding affinity (center), and binding rate (right). The 'raw' column corresponds randomly selected generated antibodies, whereas 'filtered' designs were additionally filtered by in silico screening.
Dieciigg Intarfacs
Deiggas
(4]
4:72440
417x40
SUHAARFAASSS
1000-00
12
6:1e-00
LIGHT
HIGH
Figure 3: HER2 structure and the 3D structures of generated binders highlighted in different colors. AbDiffuser has learned to redesign part of the binding interface while maintaining affinity. Our tightest binder achieved a pKD of 9.5 which, accounting for experimental variability, is akin to Trastuzumab whose average measured pKD was 9.21.
approach required 26x fewer precise SPR wet-lab experiments to find better binder hints toward major improvement in efficiency. This highlights the importance of new and more efficient generative models for antibodies and more efficient, and thus powerful architectures such APMixer.
The work of Shanehsazzadeh et [76] also shows that throughhou wet-lab experiments can be used to build sufficiently large datasets of target-specific binders without starting from a known binder. This provides straightforward way to train AbDiffuser on new targets.
After good binders are determined through high-throughput and SPR experiments, at the end of the drug discovery process, experimental 3D structure of the bound complex is often produced. In recent works, redesigning CDRs in such co-crystal structures has been a popular task [53] 43 34). To that end, Appendix NN investigaaess how AbDiffuser can be adopted for and shows that it offers much better sequence recovery rates than the current stteeooftthe-ar diffusion model for the CDR redesign (53).
6 Conclusions
We propose a denoising model that deviates from conventional practice in deep learning for protein design. APMixer enjoys the benefits of the best protein models (SE(3) equivariance, universality), while being significantly more memory efficient. We also show how to tailor the diffusion process to antibody generation by incorporating priors that enable the handling variable-length sequences and allow the model to navigate strong bond constraints while operating in an extrinsic coordinate space that supports Gaussian noise. In our future work, we want to apply our method to protein families beyond antibodies, as organized in CATH [81] and Pfam [57). Promising examples include the TIM-barrel fold and enzyme families, such as SAM-dependent methyltransferases, long-chain alcohol oxidases, and amine dehydrogenases
g
10

---

References
[1] R. Akbar, P. A. Robert, M. Pavlović, R. Jeliazkov, I. Snapkov, A. Slabodkin, C. R. Weher, L. Scheffeer, E. Miho, I. H. Haff, et al. A compact vocabulary of paratope-cpitope interactions enables predictability of antibody-antigen binding. Cell Reports, 34(11): 2021.
[2] R. F. Alford, A. Leaver-Fay, R. Jeliazkov, M. J. O'Maar, F. DiMaio, H. Park, M. Shapovalov, P. D. Renfrew, V. K. Mulligan, K. Kappel, et al. The rosetta all-atom energy function for macromolccular modeling and design. Journal of chemical theory and computation, 13(6):3031-3048, 2017.
[3] N. Anand, R. Eguchi, I. I. Mathews, C. P. Perez, A. Derry, R. B. Altman, and P.-.. Huang. Protein sequence design with learned potential. Nature communications, 13(1):1-11, 2022.
[4] Anishchenko, S. Pellock, T. M. Chidyausiku, T. A. Ramelot, S. Ovchinnikov, J. Hao, K. Bafna, C. Norn, A. Kang, A. K. Bera, et al. De novo protein design by deep network hallucination. Nature, 600(7889):547-552, 2021.
[5] Austin, D. D. Johnson, J. Ho, Tarlow, and R. van den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34: 17981-17993, 2021.
[6] S. Bachas, G. Rakocevic, D. Spencer. A. V. Sastry, Haile, J. M. Sutton, G. Kasun, A. Stachyra, J. M. Gutierrez, E. Yassine, et al. Antibody optimization enabled by artificial intelligence predictions of binding affinity and naturalness. bioRxiv, 2022.
[7] S. Taylor, P. Hessey, H. Sasaki, T. P. Breckon, and C. G. Willcocks. Unleashing trans- formers: Parallel token prediction with discrete absorbing diffusion for fast high-resolution image generation from qutto--qantized codes. In European Conference on Computer Vision (ECCV), 2022.
[8] V. Bortoli, E. Mathieu, M. J. Hutchinson, J. Thortoo,, Y. W. Teh, and A. Doucet. Riemannian score-based generative modelling. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022.
[9] B. Buchfink, K. Reuter, and H.-G. Drost, Sensitive protein alignments at tree-of-life scale using diamond. Nature methods, 186)):366-368 2021.
[10] A. V. Collis, A. P. Brouwer, and A. C. Martin. Analysis of the antigen combining site: correla- tions between length and sequence composition of the hypervariable loops and the nature of the antigen. Journal of molecular biology, 325(2):337-354, 2003.
[11] J. Dauparas, I. Anishchenko, N. Bennett, H. Bai, R. Ragotte, L. Milles, B. Wicky, A. Courbet, R. de Haas, N. Bethel, et al. Robust deep learning based protein sequence design using proteinmpnn. bioRxiv, 2022.
[12] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:
2018.
[13] J. Dunbar, K. Krawezyk, J. Leem, T. Baker, A. Fuchs, G. Georges, J. Shi, and C. M. Deane. Sabdab: the structural antibody database. Nucleic acids research, 42(D1):D1 1110)):414-D114 2014.
[14] R. L. Dunbrack Jr. Rotamer libraries in the 21st century. Current opinion in structural biology, 12(4):431-44) 2002.
[15] R. R. Eguchi, C. A. Choe, and P.-S. Huang. Ig-vae: Generative modeling of protein structure by direct 3d coordinate generation. Biorxiv, pages 2020-08, 2022.
[16] A. Elnaggar, M. Heinzinger, C. Dallago, G. Rihawi, Y. Wang, L. Jones, T. Gibbs, T. Fe- her, C. Angerer, M. Steinegger, et al. Prottrans: towards cracking the language of life's code through self-supervised deep learning and high performance computing, arXiv preprint arXiv:2007.000 06225, 2020.
[17) R. A. Engh and Huber. Structure quality and target parameters. International Tables for Crystallography, 2006.

---

[18] N. Ferruz, M. Heinzinger, M. Akdel, A. Goncearenco, L. Naef, and C. Dallago. From sequence function through structure: deep learning for protein design. Computational and Structural Biotechnology Journal, 2022.
[19] N. Ferruz, S. Schmidt, and B. Höcker. Protgpt2 is deep unsupervised language model for protein design. Nature communications, 13(1):1-10) 2022.
[20] K. Gao, L. Wu, J. Zhu, T. Peng. Y. Xia, L. He, S. Xie, T. Qin, H. Liu, K. He, et al. Incorporating pre-training paradigm for antibody sequence-structure co-design. bioRxiv, pages 2022-11, 2022.
[21] V. Gligorijević, D. Berenberg, Ra, A. Watkins, S. Kelow, K. Cho, and R. Bonneau. Function- guided protein design by deep manifold sampling. bioRxiv, 2021.
[22] K. K. Haefeli, K. Martinkus, N. Perraudin, and R. Wattenhofer. Diffusion models for graphs benefit from discrete state spaces. In The First Learning on Graphs Conference, 2022.
[23] B. Hanin. Universal function approximation by deep neural nets with bounded width and relu activations. Mathematics, 7(10):992, 2019.
[24] Z. Harteveld, J. Southern, M. Defferrard, A. Loukas, P. Vandergheynst, M. Bronstein, and
B. Correia. Deep sharpening of topological features for de novo protein design. In ICLR2022 Machine Learning for Drug Discovery, 2022.
[25] D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv preprint 1606. 2016.
[26] J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
[27] A. Honegger and A. Plückthun. Yet another numbering scheme for immunoglobulin variable domains: an automatic modeling and analysis tool. Journal of molecular biology, 309(3): 657-670, 2001.
[28] E. Hoogeboom, V. G. Satorras, C. Vignac, and M. Welling. Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning, pages 8867-8887. PMLR, 2022.
[29] Y.-C. Hsiao, Y.J.J. Chen, L. D. Goldstein, J. Wu, Z. Lin, K. Schneider, S. Chaudhuri, A. Antony, K. Bajaj Pahuja, Z. Modrusan, al. Restricted epitope specificity determined by variable region germline segment pairing in rodent antibody repertoires. In MAbs, volume 12, page 1722541. Taylor & Francis, 2020.
[30] P.-.. Huang, Love, and S. L. Mayo. A de novo designed protein-protein interface. Protein Science, 16(12)2770-2774, 2007.
[31] W. Huang, Han, Y. Rong. T. Xu, F. Sun, and Huang. Equivariant graph mechanics networks with constraints. In International Conference on Learning Representations, 2022.
[32] Ingraham, M. Baranov, Z. Costello, V. Frappier, A. Ismail, S. Tie, W. Wang. V. Xue, Ober- meyer, A. Beam, et al. Illuminating protein space with programmable generative model. bioRviv, 2022.
[33] W. Jin, R. Barzilay, and T. Jaakkola. Antibody-antigen docking and design via hierarchical structure refinement. In International Conference on Machine Learning, pages 10217-10227. PMLR, 2022.
[34] W. Jin J. Wohlwend, R. Barzilay, and T. S. Jaakkola. Iterative refinement graph neural network for antibody sequence-structure co-design. In International Conference on Learning Representations, 2022.
[35] W. Jin, S. Sarkizova, X. Chen, N. Hacohen, and C. Uhler. Unsupervised protein-ligand binding energy prediction via neural euler's 'ss rotation equation. arXiv preprint arXiv:2301.1082 2023.
12

---

[36] B. Jing, E. Erives, P. Pao-Huang, G. Corso. B. Berger, and T. Jaakkola. Eigenfold: Generative protein structure prediction with diffusion models. arXiv preprint arXiv:2304.021 02990,,, 2023.
[37] J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Zidek, A. Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-58) 2021.
[38] W. Kabsch. A discussion of the solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallog- raphy, 34(5):827-828, 1978.
[39] V. Kalofolias. How to learn graph from smooth signals. Artificial Intelligence and Statistics, pages 920-929. PMLR, 2016.
[40] M. Ketata, C. Laue, R. Mammadov, H. Stärk, M. Wu, G. Corso, C. Marquet, R. Barzilay, and T. S. Jaakkola. Diffdock- pp: Rigid protein-protein docking with diffusion models. arXiv preprint arXiv:2304.03888) 2023.
[41] D. Kingma, T. Salimans, B. Poole, and J. Ho. Variational diffusson models. Advances in neural information processing systems, 34:21696-21707, 2021.
[42] D. Koller and N. Friedman. Probabilistic graphical models: principles and techniques. MIT press, 2009.
[43] X. Kong, W. Huang, and Y. Liu. Conditional antibody design as 3d equivariant graph translation. In The Eleventh International Conference on Learning Representations, 2023.
[44] X. Kong, W. Huang, and Y. Liu. End-to-end full-atom antibody design. arXiv preprint arXiv:2302.00203, 2023.
[45] B. Kuhlman, G. Dantas, G. C. Ireton, G. Varani, B. L. Stoddard, and D. Baker. Design of novel globular protein fold with atomic-level accuracy. science, (02(5649):1344-1333 2003.
[46] J. H. Lee, P. Yadollahpour, A. Watkins, N. C. Frey, A. Leaver-Fay, S. Ra, K. Cho, V. Gligorijevic, A. Regev, and R. Bonneau. Equifold: Protein structure prediction with a novel coarse-grained structure representation. bioRxiv, 2022.
[47] S. Lee, J. Kim, and P. M. Kim. Score-based generative modeling for de novo protein design. Nature Computational Science, pages 1-11, 2023.
[48] J. K. Leman, B. D. Weitzner, S. M. Lewis, J. Adolf-Bryfogle, N. Alam, R. F. Alford, M. Apra- hamian, D. Baker, K. A. Barlow, P. Barth, et al. Macromolecular modeling and design rosetta: recent methods and frameworks. Nature methods, 17(7))665-680 2020.
[49] R. Liao, Y. Li, Y. Song, S. Wang, W. Hamilton, K. Duvenaud, R. Urtasun, and R. Zemel. Efficient graph generation with graph recurrent attention networks. In Advances in Neural Information Processing Systems, pages 4255 4265, 2019.
[50] Y. Lin and M. AlQuraishi. Generating novel, designable, and diverse protein structures by equivariantly diffusing oriented residue clouds. arXiv preprint w:Xi::2301.1248. 2023.
[51] S. L. Lisanza, J. M. Gershon, S. W. K. Tipps. L. Amoldt, S. Hendel, J. N. Sims, X. Li, and D. Baker. Joint generation of protein sequence and structure with rosettafold sequence space diffusion. bioRxiv, pages 2023-05, 2023.
[52] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[53] Luo, Y. Su, X. Peng. S. Wang, J. Peng, and J. Ma. Antigen-specific antibody design and optimization with diffusion-based generative models for protein structures. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems, 2022.
13

---

[54] K. Martinkus, A. Loukas, N. Perraudin, and R. Wattenhofer. Spectre: Spectral conditioning helps to overcome the expressivity limits of one-shot graph generators. In Proceedings of the International Conference on Machine Learning (ICMI., 2022.
[55] D. M. Mason, S. Friedensohn, C. R. Weber, Jordi, B. Wagner, S. M. Meng, R. A. Ehling, L. Bonati, J. Dahinden, P. Gainza, et al. Optimization of therapeutic antibodies by predicting antigen specificity from antibody sequence via deep learning. Nature Biomedical Engineering, 5(6):600-612 2021.
[56] I. Melnyk, V. Chenthamarakshan, P.-Y. Chen, P. Das, A. Dhurandhar, I. Padhi, and D. Das. Reprogramming large pretrained language models for antibody sequence infilling, arXiv e-prints, pages arXiv-2210, 2022.
[57] J. Mistrry S. Chuguransky, L. Williams, M. Qureshi, G. A. Salazar, E. L. Sonnhammer, S. C. Tosatto, L. Paladin, S. Raj. L. J. Richardson, et al. Pfam: The protein families database in 2021. Nucleic acids research, 49(D1):D422-D41) 2021.
[58] A. Mullard, 2022 fda approvals. Nature Reviews Drug Discovery, 2023.
[59] A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In Interna- tional Conference on Machine Learning, pages 8162-8171. PMLR, 2021.
[60] C. Norn, B. I. Wicky, D. Juergens, S. Liu, D. Kim, D. Tischer, B. Koepnick, I. Anishchenko, F. Players, D. Baker, et al. Protein sequence design by conformational landscape optimization. Proceedings of the National Academy of Sciences, 118(1)) (:(11222072281111111 18, 2021.
[61] T. H. Olsen, F. Boyles, and C. M. Deane. Observed antibody space: A diverse database of cleaned, annotated, and translated unpaired and paired antibody sequences. Protein Science, 31 (1):141-146, 2022.
[62] T. H. Olsen, I. H. Moal, and C. M. Deane. AbLang: an antibody language model for completing antibody sequences. Bioinformatics Advances, 2(1), 06 2022.
[63] W. Park, S. Stanton, S. Saremi, A. Watkins, H. Dwyer, V. Gligorijevic, R. Bonneau, S. Ra, and
K. Cho. Propertydag: Multi-objective bayesian optimization of partially ordered, mixed-variable properties for biological sequence design. arXiv preprint arXiv:2210.04096. 2022.
[64] J. Parkinson, R. Hard, and W. Wang. The resp ai model accelerates the identification of tight-binding antibodies. Nature Communications, 14(1):454, 2023.
[65] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf. E. Yang. Z. Vetito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024 0044. Curran Associates, Inc., 2019.
[66] N. F. Polizzi, WU, Lemmin, A. M. Maxwell, S.-Q. Zhang, J. Rawson, D. N. Beratan, M. J. Therien, and W. F. DeGrado. De novo design of hyperstable non-natural protein-ligand complex with sub-å accuracy. Nature chemistry, (12): 2017.
[67] W. Ponder and F. M. Richards. Tertiary templates for proteins: use of packing criteria in the enumeration of allowed sequences for different structural classes. Journal of molecular biology, 193(4):775-79,, 1987.
[68] O. Puny, M. Atzmon, E. J. Smith, I. Misra, A. Grover, H. Ben-Hamu, and Y. Lipman. Frame averaging for invariant and equivariant network design. In International Conference on Learning Representations, 2022.
[69] M. L. Raybould, C. Marks, K. Krawezyk, B. Taddese, J. Nowak, A. P. Lewis, A. Bujotzek, J. Shi, and C. M. Deane. Five computational developability guidelines for therapeutic antibody profiling. Proceedings of the National Academy of Sciences, 116(10))40254400)
2019.
[70] P. B. Reis, G. P. Barletta, L. Gagliardi, S. Fortuna, M. A. Soler, and W. Rocchia. Antibody- antigen binding interface analysis in the big data era. Frontiers Molecular Biosciences, 9, 2022.
14

---

[71] A. Rives, J. Meier, T. Sercu, S. Goyal, Z. Lin, J. Liu. D. Guo, M. Ott, C. L. Zitnick, J. Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 1 118(5):c2016233118 2021.
[72] C. Roos, Terlaky, and J.-P. Vial. Interior point methods for linear optimization. 2005.
[73] A. Ruffolo and J. J. Gray. Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodies. Biophysical Journal, 121(3):15&a-156, 2022.
[74] J. A. Ruffolo, J. J. Gray, and J. Sulam. Deciphering antibody affinity maturation with language models and weakly supervised learning. arXiv preprint 2:Xi::2//2007700 2021.
[75] V. G. Satorras, E. Hoogeboom, and M. Welling. E (n) equivariant graph neural networks. In International conference on machine learning, pages 9323-93333333 93332.932222 PMLR, 2021.
[76] A. Shanebsazzadch, S. Bachas, G. Kasun, M. Sutton, A. K. Steiger, R. Shuai, C. Kohnert, A. Morehead, A. Brown, C. Chung, et al. Unlocking de novo antibody design with generative artificial intelligence. bioRxiv, pages 2023-01, 2023. URL https: h/wps:///www.. biorxiv org/
content/10 1101/2023 01 08. 523187v2
[77] W. Sheffler and D. Baker. Rosettaholes: rapid assessment of protein core packing for structure prediction, refinement, design, and validation. Protein Science, 18(1):229-23) 2009.
[78] C. Shi, C. Wang, J. Lu, Zhong, and Tang. Protein sequence and structure co-design with equivariant translation. arXiv preprint arXiv:2210.08700 2022.
[79] E. L. Shrock, R. T. Timms, T. Kula, E. L. Mena, A. P. West Jr, R. Guo, I.-H. Lee, A. A. Cohen, L. G. McKay, C. Bi, al. Germline-encoded amino acid- bind-biniin motifs drive immunodominant public antibody responses. Science, (880(660))eadc94999)
2023.
[80] R. W. Shuai, J. A. Ruffolo, and J. J. Gray. Generative language modeling for antibody design. bioRxiv, pages 2021-12, 2021.
[81] Sillitoe, N. Bordin, N. Dawson, V. Waman, P. Ashford, H. M. Scholes, C. S. Pang. L. Woodridge, C. Rauer, N. Sen, et al. Cath: increased structural coverage of functional space. Nucleic acids research, 49(DI):D266-DD) D276,,D27322000 2021.
[82] M. Singer, S. Novotney, D. Strickland, H. K. Haddox, N. Leiby, G. J. Rocklin, C. M. Chow, A. Roy, A. K. Bera, F. C. Motta, et al. Large-scale design and refinement of stable proteins using sequence-only models. PloS one, 17(3):e0265020, 2022.
[83] F. I.. Sirota, S. Maurer-Stroh, Z. Li, F. Eisenhaber, and B. Fisenhaber. Functional classification of super-large families of enzymes based on substrate binding pocket residues for biocatalysis and enzyme engineering applications. Frontiers in bioengineering and biotechnology, page 674, 2021.
[84] J. Sohl-Dickstein, F. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, pages 2256-2265. PMLR, 2015.
[85] S. Stanton, W. Maddox, N. Gruver, Maffettone, E. Delaney, P. Greenside, and A. G. Wilson. Accelerating bayesian optimization for biological sequence design with denoising autoencoders. In International Conference on Machine Learning, pages 20459-20478. PMLR, 2022.
[86] L. O. Tolstikhin, N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner,) J. Yung, A. Steiner, D. Keysers, J. Uszkoreit, al. Mlp-mixer: An all-mlp architecture for vision. Advances in Neural Information Processing Systems, 34:24261 01124261-2422.. 2021.
[87] B. L. Trippe, J. Yim, D. Tischer, T. Broderick, D. Baker, R. Barzilay, and T. Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motif-scaffolding problem. arXiv preprint Xiv:22066.04199 2022.
[88] C. Vignac, I. Krawezuk, A. Siraudin, B. Wang, V. Cevher, and Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv:2209, 14734, 2022.
15

---

[89] R. Vinod, K. K. Yang, and L. Crawford. Joint protein sequence struucure co-design via equiv- ariant diffusion. NeurIPS 2022 Workshop on Learning Meaningful Representations of Life, 2022.
[90] J. L. Watson, D. Juergens, N. R. Bennett, B. L. Trippe, J. Yim, H. E. Eisenach, W. Ahern, A. J. Borst, R. J. Ragotte, L. Milles, et al. Broadly applicable and accurate protein design by integrating structure prediction networks and diffusion generative models. bioRxiv, 2022.
[91] N. Woolfson. A brief history of de novo protein design: minimal, rational, and computational. Journal of Molecular Biology, (33(20))66766) 2021.
[92] K. E. Wu, K. K. Yang, R. v. d. Berg, J. Y. Zou, A. X. Lu, and A. P. Amini. Protein structure generation via folding diffusion. arXiv preprint arXiv:2209.16611, 2022.
[93] K. Xu, W. Hu, J. Leskovee, and Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019.
[94] J. Yim, B. L. Trippe, V. De Bortoli, E. Mathieu, A. Doucet, R. Barzilay, and T. Jaakkola. Se (3) diffusion model with application to protein backbone generation. arXiv preprint 20Xiv:2302.02277,00 2023.
16

---

A Related Work
The literature on computational methods for protein design is vast. We focus here machine learning methods for protein sequence and structure generation and refer the interested reader the relevant reviews by Woolfson [91] Ferruz et al. [8 for broader context.
A.1 Protein Language Models
The first works protein generation focused language modeling [6 observation that the natural forces driving antibody sequence formation, namely V(D)J recombination and the selective pressure in the presence of an antigen referred as somatic hypermutation, are language models. Therein,
71 85 Motivated by the
distinct from other proteins, recent works have also built antibody-speci large-scale models trained on millions of sequences, such as AntiBERTy [74] and IgLM Т0, have been used to generate full-length antibody sequences across different species or restore missing regions in the antibody sequence 62 56
A.2 Protein Structure Generation
These approaches use equivariant architectures to generate proteins and antibodies 3D. We broadly distinguish two strategies: backbone-firs. and sequence-structure co-design.
The backbone-fire strategy simplifies the generation problem by splitting it into two steps: the generation of the protein backbone and of the protein sequence given the backbone structure. A backbone can be designed by refining a rationally constructed template [24] or by a structure-based 87
generative model [90] 32 15
50 92 47]. On the other hand, the determination of the residues that fit the backbone can be cast within the self-supervised learning paradigm I or by minimizing some score that is proxy for energy [60]. Despite the promise of backbone-fiig strategies, in the context of deep learning. it is generally preferable to solve problems in an end-to-end fashion. Decoupling the problem into separate steps also prohibit the modeling of complex dependencies between backbone and sequence, as early decisions are not updated in light new evidence. As failure mode of the backbone-first strategy is obtaining a secondary structure that is not physically realizable with natural amino acids. It is also worth mentioning that structure diffusion generative models can also be used for protein-proter docking, as shown by Ketata et al. [40).
discussed by Harteveld et al. [24],
Sequence and structure co-design entails learning to generate full protein in an end-to-end fash- ion [5T] [8). this paper we also adopt this strategy. As highlighted in the introduction, AbDiffuser differs from these previous works in the neural architecture and protein representation (APMixer), as well as by shaping the diffusion process to better suit the problem domain.
A relevant line of works has also focused on the re-generation of the cammllemmnnartty-deeemmmnn regions (CDRs) antibodies, a subset of the heavy and light chains containing typically between 48 and 82 residues [10 of particular relevance to binding [34, 20, 43
33
53
78 44). These constitute
meaningful first steps towards the goal of full-atom antibody design that we consider in this work. Having the ability to regenerate the full antibody sequence and structure significantly enlarges the design space while also enabling the probing of properties beyond affinity, such as stability and immunogenicity. Furthermore, even in the context of affinity optimization, it can be beneficial to also redesign the framework and vernier regions, as the latter sometimes interact with the antigen and affect the CDR conformation [1] [7 79].
B Antibody Diffusion
This section complements the discussion in Section by describing in greater detail how we frame denoising diffusion.
17

---

B.1 Additional Background on Denoising Diffusion
Recall that the forward process (..,,.... (o)) q(Xi (x, Markovian. This Markovianity allows for the recovery of the reverse process
(X g(xx| go),
Thus, generate data, neural network learns approximate the true denoising process Xo o(X,,,) t), which can be accomplished by minimizing the variational upper bound the negative loglikelihood 84 26 5
Lelbo = Ex(Xo)
OkL(g(Xr|Xo)lp(a) (r((o)
(3)
T +
(x)1)0)) DxI(1(X--11X) Lc(Xa)
2m 2
(x)) log(po (XolXi))
Li(Xa)
In practice, the model is trained optimize each of the (X(o)) loss terms and the Lo loss term individually, by sampling an arbitrary time step E {1, Note that (Xx) approaches 0 for any data distribution if sufficiently many time steps are used. The latent variable distribution q usually defined so that one can directly sample any step in the trajectory efficiently.
As discussed in Section 2 in diffusion process, we factorize the posterior probability distribution over the atom positions and residue types:
(Xx,(Xx-1) q(Xp" x (x
We discuss the constituent processes in the following two sections.
B.2 Equivariant Diffusion of Atom Positions
One of the important features in the training of diffusion model is the ability to efficiently sample any step in the forward process q( x ). This is easily achieved if we use Gaussian dis- tribution q( (XP X
(ap. L)., where a; and of are positive, scalar-valued functions of t, with 01 ~ 1 and = O. A common way to define the noising process is the variance- variance-preserving parameterization 884] 26 with 02 =I1 of A forward process step in this general diffusion for- mulation can be expressed as q(X!'0 X = (aapp-188 of - E), where O'jje-1 0s/01-1 and - of - Similarly, reverse process step (XPP [XP00 x8 can then be expressed as the following Gaussian:
N 1
X o-- 10-1007
X0
of
1/1-1
of
of
Similarly to Watson et al. 900, we can reduce the temperature т of the Brownian motion of the backward process by scaling the variance of this Gaussian distribution by т.
Kingma et al. [41] showed that the training objective for the Gaussian distribution can be simplified to
(x
(SNR(1 - 1) - SNR()) |8 X
where the signal-to-noise ratio SNR(!) (a./a.)² and || 11 Lr (()-E-))))
is quite common the literature to choose a slightly different optimization objective, In particular, pus
it has been found that predicting the noise é = 6(X .t) that has been added X! X
otE,
18

---

(0, Σ) and using an unweighted loss ( ğ - improves model training 26
A possible explanation is that this parameterization makes easier for the model to minimize the loss at time steps close to T. We found that a similar effect can be obtained by re-weighing the mean prediction loss (Appendix C
28
1 L( (NR(t) x
x0 -
(4)
which empirically renders the mean X p 1) and error é 6(x;", () prediction alter- natives perform comparably. We rely on equation 4 throughout our experiments, since predicting X = (Xp t) allows for casier incorporation of known constraints, such as atom bond lengths Ta (Xpe) can also be optimized using the same objective.
(see Section 3.4 As shown in Appendix D
B.3 Discrete Residue-type Diffusion
To define the discrete diffusion process for the residue types we adopt the formulation introduced by
Austin et al. [5]. If have discrete random variable with l categories, represented as one-hot vector X we define the forward process (X [xx*)) = Cat( (r)))r). where the transition matrix B.)Q is linear interpolation between identity and target distribution Q (e.g..
Qi B + uniform distribution). Similarly to atom diffusion, one forward process step is (X77 [x]])=
(a(XX 11) where Quj-11 Q Qt. From this, single reverse process step can be expressed
as:
X & X Q0-1
(X Xees X²x) = Cat
Z
for an appropriate normalizing constant 2. In the discrete case, we implement temperature 7 scaling the temperature of the softmax applied to the model output. This serves sharpen the target categorical distribution p!/T.
Similarly the atom diffusion case, use a simplified objective for the discrete diffusion 22 which has been found to lead to better results:
(x(x))) = -3, log
Loggo((An) Pa pX
Hyperparameter 33 ensures that the loss weight proportional to the fraction unchanged classes and we set (3,- og to keep the noise schedules for the residue-type diffusion and coom-position diffusion the same. We also note that now the L1 (Xo) loss term is simply the cross-entropy of the model prediction (Xo) log po (i) The derivation of our loss can be found in Appendix E
C Re-weighted Optimization Objective for Atom Positions
As shown by Kingma et al.
41 the Le term in the variational upper bound corresponds to
L. (XP M
(SNR(1 - 1) - SNR( 1 X 12
If instead parameterize X P
X8 + orE we can directly re-write this
Lı(e) SNR(t -1) |E -
SNR(I)
Meanwhile, the simplified objective of Ho et al. 26 is instead defined as
1
Le(e)
2 -
we reverse the parameterization as -a. and using the fact that SNR (1) = au/o? this becomes
pos 2
X
X pos
X
SNR())
1|
XP 2 1
2
σ
σ
x-1 2
19

---

D Reconstruction Loss for Atom Positions
For the Gaussian atom position diffusion (Section B.2 we need to account for a few considerations in the reconstruction term L1 (Xo) of the variational upper bound. Assuming that the data probability is
constant, (Xx ( x N (
X 2). Similarly to Hoogeboom et al. [28] we parameterize the
mean using the model prediction instead:
(xx = log 2
1
( 1
2
As highlighted by Hoogeboom et al. (28] this increases the quality of the results.
Isotropic distributions, such as Gaussians with appropriate covariance matrices, assign the same prob- ability to every rotation of a vector in 3D, implying that the resulting atom diffusion process is equiv- ariant w.r.t. orthogonal group O3. To additionally impose that the diffusion cunslation-equivaan we follow Hoogeboom et al. (28 and use a re-normalized multivariate Gaussian distribution, where the sample center of mass fixed zero: Σ X 0. The latter has the same law as an ordinary
Gaussian with the main difference that the normalizing constant is now Z = (V2TT) atoms in 3D space.
3(n-1) for n
is clear to see that the 03 property is satisfied if we have an identity covariance. However, it also holds for non-isotropic covariance matrices. Let us define Z ~ (0, of dimension and X KZ, where E-1 KKT. The latter can be re-written as To = vec(X.)
(1 & (e)
/ We then note that for any 3x3 unitary matrix U, we have x (U & 1V & Nwo =7 (UT @ (I
LU T) = =i (U U L)m = (I @ L)rg. In other words, since the covariance acts on the node dimension and the unitary matrix acts on the spatial dimension, they are orthogonal and do not affect each other.
E Simplified Discrete Diffusion Optimization Objective
We obtain the simplified training objective by taking slightly different bound on the negative log-likelihood to the one presented in Equation [2
(X1:T( )
po (Xo:r)
llog (pe (Xo)) < E Ex Ex (x,r
- logg
Po (Xor))
log
1 (X Xu)
po (X
po (Xo
X1) (x) (XxlXXo)
E
log (po (Xrr)) - Σ
log
log
(X,(xx-11)
t=2
(XolXi)
Eg
log (po (Xr)) log
(X1]Xo)
(X Xx, Xo) po (XolX,)
q (X+-1|X0)
Σ log
(Xi-1l Xo)
q (o)
pe (XT)
E,
-log
log (po (Xolx,))
llog (po (X))
9 (X)
Lx(Xo)
Lz(Xa)
Lr (Xo)
In this formulation, compared to equation 3. the term Le = DKL (g (Xe-l Xo) ) Ilpo (X)- is replaced by
L, - log (pe (XolX.))
To form the discrete loss (XP'), we additionally weight each loss term L, by Br to match the loss weighting used for the atom positions (Appendix Note that when this objective minimized, the original DRL. term will also be at a minimum.
20

---

E.1 Non-Informative Priors Demand Higher Complexity Denoising Models
Denote by p the data distribution and by pa the learned distribution, the latter of which is obtained by the push-forward po(So(Z)) = q( 2) of some prior measure by function fa- In the context of denoising diffusion, Z XT, while fo = ( T) is the learned reverse process.
We prove the following general result which relates the informativeness of the prior with the quality of the generative model and the complexity of the function computed:
Theorem E.1. For any fo that invertible equivariant function w.r.t. subgroup G of the general-linear group GLAd,R) R) the following must hold:
(G) 2 We(p, ②) - We(p,po),
where (g( := (minge: Ez [I 7) 210))* quantifies the expected complexity of the learned model under q, Wp(p. po) the Wasserstein t-distance of our generative model to the data distribution, and W (P, 9) is the distance between prior and posterior.
The bound asserts that one cannot learn a simple (in terms of cg(f)) generative model that fits the data well unless the prior has small Wasserstein distance the data distribution.
The complexity measure (f) is particularly intuitive in the context denoising diffusion. For the diffusion of atom positions with SE(3) equivariant model d, we have
Co ()) = min Ex, olo( Xr. 7) 1T SESE(3
that corresponds to the expected amount of denoising that the model performs throughout the reverse process, discounting for rotations. By selecting informative prior, we decrease (, q) and therefore reduce the amount of denoising our model needs to do.
Proof. We recall that the Wasserstein /-distance between two probability measures p.p' with finite t-moments
1/t
(.(pp') = inf (ET(p'))
E(x,x*)~<1IX [X'])) X'I1a)
where the coupling y a joint measure on Rd x Rd whose marginals satisfy Sge 7XX, X' )d'' p(X) and Sgx 7Xx X')dz = p' (X'), whereas [pp p') is the set of all couplings of p. p.p/.
We will first lower bound Wi (po. 4). To achieve this, let X f(2) and X' fo( and select y"XX, X') = when Z + Z' and (X, X') = (Z) otherwise, which coupling because satisfies fx "X,XX')
(Z') (X()). Then, exploiting that the Wasserstein distance is metric (and thus abides by the triangle inequality). we get
(Z) p(X) and I'x "XXxx"))
We(po. 4) 2 W.(p.g) - W.(P.Po)
(by the triangle inequality)
X.X' / XX,X')IX -x dæ dx'
1/a
W.(p..) g) - (veinf
inf
2 W.(p.a) U (XXXx x' de de!
1/e
W.(p. UJ ( ( XXXXx")xXXX X'I1: do' dx
1/2
w.(p.g) ( - fo(f
(by the definition of 7" and invertibility of f)
W.(p.g) g) - (Ex~p So(s Jo(J-1(X))1 1/2
W.(p. q) - (Ezng [llf(Z) -f (F(Z))I
-W((p.) - (Ezng [lf -fo(2) 1/*
(5)
21

---

Next, we upper bound Wi (pep q). Let G be any subgroup of the general linear group GL(d, R) for which fo is equivariant and further denote by g E G an element of the said group.
We fix (X, Z) a(Z) when X = fo(gZ) and 7* (X, 2) O, otherwise. To see that the latter is a valid coupling notice that fog is bijective as the composition of two bijective functions, implying:
/
/ 7x1(X, Z)dr (g(),
7*th (x, 2)dz (2xx) and
with 2n =foo o og)= (X).. Further,
((2x) po ((x
(since px(X)= (Z))
ffllo (X)))
(by the equivariance of fo)
(o
(X)),,
(by the exchangeability of pe)
as required for 7 to be coupling. We continue by bounding the Wasserstein metric light of "
/ /7(Xx /x(X,2)IIX Z)IIX - Z1li dr dz
9)²= =
inf 7E((p).2)
(by definition)
$ / 7*h (x, 2)11 - -21: do dz
(due to the inifimum)
-J 7* (x, (,, -71/e dæ dz
/ (()) (lf(l))) -211: dz
(by definition of y
B2zglllS0(00 2lla]
(6)
Combining equation Jn equation [6] we get
min gEG
- |i 2 We(p, 4) (12)
(7)
as claimed.
In the context of our paper, we have Z = Xy N = - f()) Xo. G' is the standard Euclidean group SE(3), and we select t = Then, the aforementioned inequality becomes
min Ex, [Ilo(XX, T) g 1 W2(p. v Ex o (X 111 DESF(3)
(8)
where we have changed the position of g by exploiting that the l2 norm is unitary invariant and that ø is SE(3) equivariant. Alternatively, we can obtain the following tighter bound by stopping earlier at the derivation of equation 5
Ex, Exollo(77 T) XT 2 Wa(p, gT) W2(Po.P),
where or is the law of Xx.
F Learning the Conditional Dependence Relations Between Atoms
We consider the zero-mean multivariate Gaussian whose inverse covariance matrix Đ - L + al corresponds to a sparse weighted Laplacian matrix L. Intuitively, Lij should be zero the position of the point i is independent of that of j given the positions of the neighbors of being fixed. The term a is a small constant whose role is to shift the spectrum L and thus make the Laplacian invertible.
We follow the framework of 33 and optimize L. by minimizing the following objective:
min (xx PX + J(L),
LEC
22

---

where the m X matrix X contains the 3D positions of the points modeled and fL) are some optional constraints we can enforce on the graph Laplacian (and thus also the adjacency matrix A).
The log-likelihood of vectors X on the graph defined by I. can be directly expressed as a function of the graph adjacency matrix and the pairwise distance matrix Z:
(f LXP=) A A
Σ A 1
X
Kalofolias (39 propose the following optimization objective, recover the optimal adjacency, which with logarithmic term also ensures that every node has at least one neighbor, which is something we also desire:
- 11112 2
min A 3 2||1 1 log( (A1)
An efficient primal-dual algorithm exists for solving this optimization problem 39 The adjacency matrix that recover using the mean distance matrix over all of the proteins in paired OAS can be seen in Figure 4 Our use of AHo numbering allows us to directly estimate such mean distances, as with our representation every antibody effectively possesses the
number of residues.
To increase the stability of the Cholesky decomposition L KK*, which is needed for sampling
co-variate noise X Ct2 0x + o, Ke, (0, 1), we instead use the generalized Laplacian diag(A1) - for the diffusion process.
Figure 4: We define Gaussian prior the atom positions by learning adjacency (conditional dependence) matrix for antibody backbone atom positions from all of the folded paired heavy and light chains in the Observed Antibody Space (OAS) 61]. Dependencies between framework residue atom positions and even correlations between heavy (top-left) and light (bottom-right) chain atom positions are distinctly captured.
G Proof of SE(3) universality
In the following, we prove the SE(3) universality of an MLP-mixer backbone model combined with frame-averaging.
Concretely, let be an n x m matrix with bounded entries and consider the backbone model: 6(X) = Cy. ort-1
C3 e, (X), where the subscript denotes layer, is an MLP operating independently on each column of its input matrix, and similarly, rı is an MLP operating independently on each of its input.
a consequence of Theorem backbone model that is universal over any compact set. then combining with caaeeaveeragigg frame- finite domain (i.c., domain within which the frame cardinality is finite) yields a G-equivariant universal model (Ø) J. Since the SE(3) frame is bounded and thus the domain is frame-finite (see also Proposition (68) ). to show that (Ø) is a universal approximator of any continuous SE(3) function, it suffices to show that the MLP-mixer is universal approximator over continuous functions
(68) that if F is a bounded G-cquivariant frame and if ø is a
any compact set (or simply universal).
To proceed, we will show that there exists parameterization so that ((X) = C], v(e;v(eee(X)) vec(X))) for some vector z. We can then rely on the universal approximation property 23 of ML.Ps conclude that ò is also universal.
23

---

4. A row MLP zeros out all elements except those indexed by the one-hot encoding in the first entries. The latter can be done by ReLU MLP with one hidden layer: if all entries of our input are smaller than b and the input is (onehot(i), x)), the first linear layer adds b to only the entries [n + in n + in + n)) and shifts every element by -b using its bias. The latter lets the correct elements pass the ReLU with an unchanged value while We first provide an exact constructive argument on how this can be achieved:
Let V be a (n + 1) x (n + 1) unitary matrix whose first column is the constant vector 1 and set U V. to be the + 1 x n submatrix. The first MLP embeds each column vector x isometrically into n as follows: 6j (1(x) = Uv. By construction, every column of the output X' is now orthogonal to 1.
Xn-1
2.
The network appends U to the left of X' and (UX; ;UX) to the right. Since appending to the right involves replicating each row multiple times, can be done by a linear row operation. Appending to the left is slightly more involved and can be achieved by iteratively building U column by column, with each column added by using a row MLP to add one dimension with constant value and column MLP to project the introduced to the appropriate column of U. The key here is that, since we have guaranteed that the columns are always orthogonal to the constant vector, the column MLP can always distinguish the newly introduced column from the ones already at play. Let X" be the result of this process.
constant vector
3. A column MLP projects each column of the (n + 1) x (n + rm) matrix dimensions by multiplying by UT The output of this layer is X" = :I: :.].
setting all other elements to zero, as required. Let the output of this layer be X!" with
X = -Ji,;; 0: X..; 0;
5. A column MLP sums up all entries. The output now a row vector (1; (I: " :TT) of length + nm, required.
Steps and 2 of the above constructive argument are inefficient because they rely on very simple layers. This can be avoided if we utilize more powerful MI.Ps (i.e., functions q and rı) and an existence argument. Specifically, as there exists a continuous function that achieves step 2, by the universality of MLPs there also exists an MIPP that approximates the step arbitrarily well. Similarly, rather than increasing the dimension to n + at step I, one may argue that the ML. can assign some vector outside of the bounded domain where the input data resides, and thus the dimension can remain n. Therefore, in this setting, only a constant number of MLPMixer blocks are needed to obtain the desired output.
This concludes the proof.
H Noise Schedule
In our denoising model, we use the cosine-like schedule proposed by Hoogeboom al. 28 with 1000 denoising steps:
1 T
0; = (1- 2s)
+
where a precision s 1e is used to avoid numerical instability. As o, /I- of and 3 a². is sufficient to define just a, Following Nichol and Dhariwal [59] we clip Orjt-1 to be at least 0.001, to avoid instability during sampling and then recompute ar a cumulative product. To further ensure numerical stability, as recommended by Kingma et al. 441 we compute the negative log SNR curve 7(t) = - log of + log oz. This parameterization allows for numerically stable computation of all important values [41]. for example, of = sigmoid( (?))), of = =sigmoid(<(h)) ai h expml (sofpplusss) 1)) softplus(?(1))) -expm((7(1-1) {{)))a} and SNR exp( (t)) Where expml (x) = exp(x)) (a) and softplus(x)) = log(1 + exp(x)).
24

---

I Full Training and Sampling Algorithms
For convenience, in Algorithms
| and we describe the full training and sampling procedures.
Algorithm Training
1: repeat
2: Get a data point:
3: X -x0 (Xx0 Xx
4: Sample some time step:
5: Uniform(1 (1, T)
6: X (Na, of Σ)
7: X Categorical (Xo Qz), here Q = B.T (1 3.)Q
8: Predict:
9: X X = (Project (x xx,11
X Project
(x5
11: Compute losses:
12: ( SNR( [xpp
X0 12-1
13: (xxx))= CrossEnnyyypyy
(x X7s
Take gradient step on: V
(L((8) + ))
15: until converged
Algorithm 2 Sampling
1: Initialize:
2: X ~ ~N(0, Σ)
3: X7 ~ CCategorical(()
4: Loop over the time steps:
5: for t-= =TT do
6: Predict:
7: X-pos 2000 So (Project
(1)
X = Project pos
8:
9: Sample next state:
10:
if 1 1 then
X N 181--192-18 X +c0 +00-100 10: 8000 01 E
11:
0 R.- XP
12:
X
Categorical
13:
else
14:
P
N(
-
15:
X-80
Categorical (8pey
return X seq
J Metrics Used in the Numerical Experiments
The metrics applied to generated sequences include methods that measure naturalness, similarity to the closest extant antibody, and stability. To examine Naturalness we use the inverse perplexity of the AntiBERTy 774 model trained on large corpus antibody chain sequences. shown by Bachas et al. 6 this score correlates with antibody developability and immunogenicity. To explore Closeness we use fast sequence alignment [9] to determine the closest sequence the training set. The mean sequence identity (fractional edit distance) to the closest training sequence is then reported as a score. Here Stability is operationalized by estimating the error of the folded structure using IgFold [73]. This error can be used to rank the sequences by how stable we expect their folds to
25

---

be. We take the 90th percentile of the residue error as an estimate of the sequence fold stability; the latter typically corresponds to the CDR H3 and L3 loops, which have the most influence over the antibody's s functional properties. To further verify that the generated antibody sequences satisfy appropriate biophysical properties, we rely on four additional structure-based metries Raybould et al. [9:] CDR region hydrophobicity (CDR PSH), patches of positive (CDR PPC), and negative charge (CDR PNC), and symmetry of electrostatic charges of heavy and light chains (SFV CSP). These metrics operate on folded sequences (using IgFold) and take into account distance-aggregateed structural properties of CDR regions (and their spatial vicinity), and their significant deviation from reference values is typically associated with bad developability properties of antibodies such poor expression, aggregation, or non-specific binding [6]]
To evaluate the overall similarity of the generated and the test-set distributions, for all of these sequence metrics, we report the Wasserstein distance between the scores of the sequences in the test split and the scores of the generated sequences.
The metries applied the generated structures focus primarily on scores known to correlate with free energy and RMSD. a) We use free energy AG estimated using Rosetta [2] to evaluate the stability of the generated structure, Although lower energy is normally associated with higher stability of the protein structure, one needs to be careful not to reward disproportionally small energies achieved when a miss-formed protein collapses into a morph-less aggregate. Thus, we again report the Wasserstein distance between the generated and test energy distributions. b) RMSD. We also re-fold the generated sequence with IgFold [73] and report the RMSD for the backbone N, C. C, Cg and O atom positions. RMSD is reported as mean over the generated structures as it captures how well each generated structure matches its sequence.
K Detailed Generated Structure RMSD Evaluation
Model
Full 1
Fr
H
H1I
H2.,
L
CDR LII
L2]
CDR L31
EGNN
9,8231
9,3710
9,2929
13.1720
13.0032
10.3360
9.3918
14.6768
10.1584
10.4860
EGNN (AHo)
10,0628
9.1717
9.3552
13.1730
13.4611
12.2434
9.5314
15.3884
10.6975
11.0732
EGNN (AHo Cov.)
9,4814
8.7581
12,9454
13.2237
12.0939
8.8174
15.2841
10.0504
11.1167
FA-GNN
0.6677
0,5748
0,5093
0.6671
0.7438
2,2530
0.6157
0.8199
0.5946
L1556
FA- GNN (AHo)
0.8321
0.4777
0.4618
0,6881
0.7867
2,1884
0,4860
0.9398
0.5053
L.1665
FA-GNN (AHo Cov.)
0,8814
0.5934
0,5236
0.5966
0.7907
0.4521
1,3536
AbDiffuser (uniform peior)
0,8398
0,5937
0,5742
0.7623
0.6705
188365
0.6095
0.8825
0J4795
1. 1,6988
AbDifTuser (80 projection)
11,1431
11,0062
10.8779
13,8692
14.4139
10:4367
11,1709
15,7536
11 525220
11,2404
AbDiffuser Cox.)
0.6302
0.4011
0,8310
AbDiffuser
0.5230
0.3109
0,2862
0.3568
0.3917
1.5073
0.3322
0.4033
0.3257
0.7399
AbDiffuser (side chains)
0.4962
0.3371
0.3072
0.3415
0,3768
1.3370
0.3637
0.3689
0,3476
0.8173
Table 4: Detailed RMSD for generated antibodies based on Paired OAS dataset 661. The top three results in each column are highlighted as First, Second, Third.
Model
Pull
Fr
Fr. H.J
CDR HI|
CDR H2|
CDR H3]
Fr. L
CDR LI,
CDR 1.2,
CDR L.3
MRAN
0.7792
0.3360
0.3045
0.4569
0.3359
2.9053
0.3645
0.4425
0.2490
0.6862
EGNN (AHlo& & Cov.)
9,2180
8.7818
K.5527
12.0018
12.5770
10.0308
8.9396
14,2269
9,5391
10,4077
FA-GNN (AHo Cov)
3,3761
1.8529
0.6446
0.5225
2.0202
4,2721
0.5633
0.5376
3.4047
AbDilluser
0.3822
0.2186
0.1669
0.3611
0.2737
L.1699
0.2610
0.1937
0.2006
0.6648
AbDiffuser (side chains)
0.4046
0.2686
0.2246
0.3861
0.3115
1.1191
0.3073
0.2242
0.2379
0.7122
AbDiffuxer (r 0.75)
0.3707
0.2138
0.2709
1.1210
0.2563
0.1946
0.6615
AbDilfuser (s.c.. 7 = 0.75)
0,3982
0,2729
0.2277
0.3914
0.2917
1.624
0.3127
0.2492
0.2548
0.7131
AbDilluser B.01)
0.2000
0.1463
0.3389
0.2723
0.9556
0.2430
0.1530
0.1792
0.6582
AhDiffuser (s.c. 0.01)
0.6795
0.6168
0.5938
0.8161
0.7113
1.5500
0.6396
0.7938
0.7048
0.8395
Table 5: Detailed RMSD for generated antibodies based on Trastuzumab mutant dataset 55 The top three results in each column are highlighted as First, Second, Third.
Here, in Tables and we provide per-region distances between folded and optimized structures of the generated sequences and the generated structures. As you can see these distances correlate well with the overall RMSD reported in the main text. As the folded structures are optimized, we also investigate how the error changes if similarly to folding models we also use Rosetta (2) to optimize the generated structures in Tables and 7 As can be seen, the optimization slightly reduces the RMSDs, but the relatively modest change for AbDifftuse-geenerate structures hints at the fact that
26

---

the generated structures were relatively physically plausible. In Table we we see that while the model that generates the side chains achieves worse overall RMSD, in most cases models CDR H3 more precisely, which is the most important part function- -iise... This seeming focus on CDR H3 might explain why this model achieved a better predicted binding probability, even while modeling the overall structure slightly worse.
Model
Full
Pr
Fr. H
CDR HiL
CDR H2]
CDR H3]
Fr. L
CDR I.1,
CDR 1.2]
CDR 1.3]
EGNN
9.8129
9.3187
9.2617
13.2306
9.3722
10.1526
10.6565
EGNN (AHo)
10.1364
9,5233
9,3961
13,3611
13,7014
12
9,5918
15,6919
10.8115
11.3710
EGNN (AHo & Cov.)
9,5411
8.8202
8.6761
13.0186
13.3938
12.1843
8.8949
15.4368
10.1352
11,2356
FA-GNN
0.7817
0-4844
0,4228
0.5558
0.6293
2:1533
0,5205
0.7222
0.4617
1 10457
FA-GNN (AHo)
0.7767
0,4116
0.3918
0,6002
0.7031
2,2311
0.4228
0.8372
0.4054
LD333
FA-GNN (AHo Cov.)
0.7798
0.5061
0.4541
0.5485
1.9846
0.5195
0.3914
AbDiffuser (uniform poior)
0.8122
0.5528
0.5300
0.7053
0.6184
1,8129
0.5704
0.3326
0.3914
LD416
AbDiffuser (ao projection layer)
10,9194
10.7255
10.5253
13.6499
15 15.0046
10.9846
10.9083
15,9310
11,6059
11,7446
AbDiffuser (ao Cov.)
0.5867
0.3425
0.3206
0.4272
0.4848
16626
0.3606
0.4921
0.3296
0,7801
AbDilTuser
0.50668
0.2899
0.3282
0.2611
0.7334
AbDiTTuser (side chains)
0.4463
0.2751
0.2426
0.2764
0.3266
1.2869
0.3025
0.3187
0.2390
0.7533
Table 6: Detailed RMSD for generated antibodics based on Paired OAS dataset [6]] after optimization The top three results in each column are highlighted as First, Second, Third.
with Rosetta (2).
Model
Fr
H.
CDR HIl
CDR
CDR H3)
Fr.
CDR LI,
CDR
CDR
MEAN 43
0,7412
0,3004
0,2718
0,5395
0.2909
2,7830
0.3261
0,3758
0.2593
0.6849
BGNN (AHo & Cov.)
9.2535
8.8120
8.5701
12:0330
12.7993
10,1256
8.9911
14.4588
9,7059
10.6565
FA-GNN (AHo & Cov.
2.1631
2,2522
1.1541
0.6734
0.5783
2.0892
2.9101
1.4517
0.5797
2.1591
AbDiffuser
0,3692
0,2017
0.1415
0,3349
0.2474
1,1464
0.2479
0,1743
0.1589
0.6625
AbDiffuser (side chains)
0,4087
0.2755
0.2304
0.3632
0.3044
1.1065
0.3141
0.2957
0.1920
0.7217
AbDiffuser (r - 0.75)
0.3584
0.1969
0.1358
0.3283
0.2459
1.003
0.2434
0.1642
0.1513
0.6599
AbDiffuser
0.75)
0.3981
0.2747
0.2267
0.3615
0.2795
1,0497
0.3155
0.2939
0,2050
0.7151
AbDiffuser (r 0.01)
0.3210
0.1837
0.1202
0.3023
0.2464
0.9288
0.2306
0.1257
0.1285
0.6583
AbDiffuser (5.c. 0.01)
0.6298
0.5661
0.5450
0.7135
0.6025
1,0870
0.3868
0.6878
0.6012
0.8720
Table 7: Detailed RMSD for generated antibodies based on Trastuzumab mutant dataset 155 after optimization with Rosetta (2]. The top three results in each column are highlighted as First, Second, Third.
L Wet-Lab Validation of Designs
In our wet-lab experiments, we follow the procedure of Hsiao et al. [29]. As we focus Trastuzumab mutants, the experiments need to account for the fact that the part of the HER2 antigen to which Trastuzumab binds is prone to misfolding in vitro. In the usual experimental procedure for the SPR measurements of Kd values, the antibody is immobilized and the antigen (in this case HER2) is used ligand. However, for the domain to which Trastuzumab is binding in HER2, taking into account the problem of misfolding, the standard procedure described by Hsiao et al. 22] is to immobilize the antigen and treat the antibody as a ligand, making the 'misfolded' HER2 essentially invisible in the binding kinetics.
As pointed out by Mason et al. [55] mutating Trastuzumab such that the resulting antibody still binds to HER2 is quite challenging, as some binders in the dataset are just one edit away from non- non-binders. We also check the similarity of our binders verified in the wet-lab experiments to both the binder and non-binded sets from the dataset. As can be seen in Table some of the best binders we discovered are equidistant to both sets.
To further highlight the similarity of the non- non-binder and the binder sets from Mason al. [55 we include the residue frequency (logo) plots for the CDR H3 positions that were varied in Figure
M Implementation and Training Details
As mentioned the main text, all of the models are evaluated using the same denoising procedure. They also share the input embedding procedure: we utilize learned dictionary for each amino acid type and another learned dictionary for the chain type (light or heavy). These initial embeddings summed together to form the input to the actual model. For all models except the APMixer, an
27

---

Generated binder
Binder dist.
Num. closest binders
Non-binder dist.
Num. closest non-binders
KDJ
SRYGSSGFYOFTY
2
2
2
5.35c-08
SRWLASGFYTEAY
2
4.72c-09
SRWSGDGFYQFDY
4.12c-09
SRWRGSGFYFFDY
2.89c-09
SRWR ASWFAAYYYYY
2
19
2.00c-09
SRYGGFOIYOFDY
2
3
2
1.86e-09
SRYGGSGFYTFDY
2
8
3.17e-10
Table 8: Edit distances of our generated binders to closest binders and non-binders in the dataset by Mason et al. 555 together with the number of such closest (non-)binders.
GFY F
W
Y
GE
Figure 5: Logo plots for relative amino acid frequencies in the mutated CDR H3 positions for binders (top) and non-binders (bottom) the dataset by Mason al. 55
additional sinusoidal sequence position embedding is included in the sum. The model also receives the time step t as an additional input, concatenated with the other inputs. Similar to Hoogeboom et al. 28 we found this simple time-step conditioning to work well. The input and output of the model are always processed with our constraint layer from Section 3.4] The amino acid type predictions are made by computing vector product with the input amino acid dictionary.
We will cover the model-specific architectural details in the following.
M.1 APMixer Details
We build ab mixer out of blocks as described in Section 3.3 where each block consists of two MLPs, that are applied consecutively on the columns
X.j = x +
all j E [c]
and on the rows
x, X1, + Wap Wap(WaLayenNom((X, forr all i [r]]
The whole APMixer used 8 such blocks plus embedding MLP and two readout MLPs, for predicting residue atom positions and another for the residue type predictions. Their outputs are respectively processed using eguivariant and invariant frame averaging formulations.
As we noted earlier, each block is wrapped frame averaging. We choose to process half of the output equivalently and half invariantly. It is possible just use the equivariant formulation throughout the inner blocks, but we found this to slightly improve the performance of the model. Possibly because this aligns slightly better with having position-invarian sequence features. In this case, extend the APMixer block to have a linear layer before the column MLP, to merge the two representations.
We use 2 layer MLPs throughout, with the embedding dimension of h = 1920. The column MLP is applied on three columns a time ensure easier modeling of 3D interactions. Also, to facilitate easier modeling of multiplicative interactions (e.g. forces) these MLPs use gated activation in the hidden layer x Wo.h a (Ih:2)), where x is the output the linear layer, the embedding dimension (the MI.Ps hidden dimension is effectively 2 It) and is the activation function. We chose to use the SiLU 255 activation function appears to perform the best on geometric tasks, both APMixer and the bascline GNN architectures.
28

---

M.2
Baselines
M.2.1
Transformer
For the sequence-only generation, we use a standard transformer encoder 12 as implemented by PyTorch [65] with 3 blocks, embedding dimension of 2048, and pre-normalization. The transformer
uses the GELU 25 activation function as standard.
M.2.2 EGNN
We use the EGNN 75| as implemented for the equivariant molecule diffusion by Hoogeboom et al. 28 We extend work with residues that have many atoms by using the formulation proposed by Huang et al. [1] and also used by Kong et al. [43), which for updating each node (residue) make use of the distances z = X; x, between atoms in each residue:
ma - (1l R,, (z))))
m, Σ Mig.
1+11 onn (14 mi):
Z 1Z +1 (M.. Zi (2))²)
X/=1 =x! Σ
where X; a matrix of x 3 we have atoms per residue.
To improve the model stability. apply a layer normalization on hL at the beginning of each such block. We wrap every block in EGNN in our constraint layer Section 3.4 The EGNN has 6 such blocks and uses an embedding dimension of 576. As standard, we use SILU activation and, similarly to AgentNet, we found it slightly beneficial to use hidden dimension in the MI.P of twice the embedding size. The model uses readout MLP to make the final embedding which is used to predict the amino acid type. This layer is also preceded by layer normalization. embedding MLP used for the inputs.
M.2.3 FA GNN
We use the same GNN architecture as used by Puny et
mi.j =bb h2)
m; Σ mi.j
6621 =bh m)).
which is essentially the same architecture as EGNN, just without the distance-based edge features and explicit position updates. To process the positions, wrap each of these GNN blocks using frame averaging, and for the very first embedding MLP we supply residue atom positions as inputs together with the other features.
The FA-GNN has these GNN blocks and uses an embedding dimension of 384. SiLU activation and the hidden MLP dimension of twice the embedding dimension are used. The FA-GNN uses the same setup as APMixer with two readout heads for amino acid type prediction and atom position prediction.
M.3 Training
We use AdamW for training [52] with a weight decay 0.01 and with a learning rate of 10-4 for all structure models, while the transformer used a learning rate of 1- 10 We experimented with weight decay of le - 12 and learning rates in the range of 10-3 determine the chosen values. We also normalize the gradient norm to unit length during training. All models use a batch size of This is the maximum batch size that allows training the baseline models on our 80GB GPU. The APMixer due to the better memory complexity allows for batch sizes up to
10 for all models, to
29

---

32, but to keep the setup similar, we also used a batch size of 4 for it. For paired OAS we train the models for 50 epochs, while for HER2 binder generation we train for 600 epochs, for around 1. 1.25M training steps in both cases. The Paired OAS dataset [61] was split into 1000 test samples and 1000 validation samples. Similarly, the HER2 binder dataset [55 was split to have 1000 test samples and 100 validation samples.
We keep track of the exponential moving average of the model weights, for all models, with coefficient of 0.995, and use these weights at test time. No model selection is performed.
idealized atom positions returned by projection layer (Section 3.4 as the target. This ensures that the exact target configuration is reachable by the model.
During training,
The CDR redesign baselines of RefineGNN [4 and MEAN
43 were trained for the same amount
of steps using the default parameters and early stopping.
All experiments were performed on an 80GB A100 GPU.
M.4 APMixer Classifier
We adapt APMixer for HER2 binder classification (Section 5.2 by including global readout. This readout is constructed by applying mean pooling over the frames and residues after each APMixer block and then applying layer normalization and linear layer. All of the linear layer outputs are summed to obtain the final prediction. This follows a popular graph classification readout used in GNNs [93]
The classifier is trained for 100 epochs using a batch size of 16. No exponential moving average of weights is used, and the model with the best validation accuracy is selected. The full HER2 dataset was split to have 3000 test samples and 1000 validation samples.
N Conditional CDR Redesign
While co-crystal structures usually come quite late in the drug discovery process, performing CDR structure and sequence redesign for a given, unseen co-crystal structure is a popular benchmark for
deep generative models (53 43 44 34] Here, we the data split from Luo al. [53] derived from
the SAbDab database of antibody-antigen structures [3). This split ensures that all antibodies similar to those of the test set (> 50% CDR H3 identity) are removed from the training set. In contrast the
experimental setup of Luo et al. [53], we report the results for the more difficult task of redesigning all CDRs at oned? rather than redesigning a single CDR at a time. Following Luo et al. [53], the results are averaged over 100 samples for each of the 19 test structures. We observed that in this setup the minimization protocol used by Luo al. 53 increased the RMSD values (Table therefore, we report the results without it.
To adapt AbDiffuser to this conditional case, we build a bipartite graph between the antibody CDR residues and the nearest antigen residues that are at most 10Å away, in terms of Cg- Cp distance. Then we simply replace the initial MLP embedding layer with a one-layer FA-GNN. The rest of the APMixer architecture remains unchanged. We trained AbDiffuser with batch size of 8 for 30k steps, with the rest of hyperparameters staying the same as before.
Table 2 shows that in this conditional setup AbDiffuser outperforms the saaeeoof-the-a diffusion baseline (DiffAb) in amino acid recovery (AA) by a wide margin. This good performance could be attributed to the usefulness of AHo numbering, and APMixer being better at modeling sequences than a GNN. Our ablation experiments on paired OAS generation (Table corroborate this hy hypothesis. Although the CDR RMSD results are generally quite comparable between DiffAb and AbDiffuser, importantly, AbDiffuser performs better in CDR H3, which has the most variability in its structure and can influence the binding the most. Both methods strongly outperform RosettaAntibody antibodyessign (RAbD) (2) the state-of-the-art computational biology antibody design program, in terms of amino acid recovery.
We use the checkpoint for this task provided by the original authors.
30

---

CDR H11
CDR
CDR
CDR
CDR
CDR L3
AAT
RMSD1
RMSD]
AA1
RMSD
RMSD
RMSD
AAT
22.85%
1,204
1,624
(Mimmized)
1.188
26.7816
(Miumzzed)
1.371
DIIAA
0.802
28.2756
3,500
62.9119
62.59%
49.38%
85.08%
1.138
73,68%
AbDiftueer Iside chains)
76.301
1.584
65.72%
1,449
34.10%
3.346
$1.44%
1,462
83.22%
1.397
73.15%
1.391
AbDifluse ( 0.75)
79.76%
1,083
72.96%
1.9900
36.53%
2,805
85.00%
1,008
95.71%
1.237
76.29%
0.955
ArDiftuser (SC., 7 0.75)
0618
1.715
66.38%
1,512
34,71%
3.214
81,938
1,373
83.54%
1.351
73.324
1.509
ArDattuser (F 0.013
81.11%
1.075
74.27%
0.9446
37.2716 96
2.795
86.20%
1.115
86.85%
1.238
77.06%
0.966
AbDifluser (s..) 0.01)
33.369
2.663
66.89%
2,010
55.56%
3.124
83.10S
1.525
82.0598
1.623
74.19 1946
1.502
Table 9: Generating CDR structure and sequence for existing co-crystal structures from SAbDab [3] All CDRs are generated once. The amino acid recovery rate (AA) and RMSD are reported for each individual region. Baseline models from Luo et al. 53 marked by generated CDRs one at time.
31


Reviews:
[{"review_id": "0CA3YnDjkw", "reviewer": "Reviewer_AFyF", "summary": "Antibody design has an extreme importance for both fundamental and application biologic science. The submission intriduces an equivariant and physics-informed diffusion model called AbDiffuser", "questions": "It can be useful to discuss AbDiffuser compared to other diffusion tools focused on proteins and antibodies such as RFdiffusion, DiffDock-PP, and DiffAb ((https://github.com/RosettaCommons/RFdiffusion, https://github.com/ketatam/DiffDock-PP, https://github.com/luost26/diffab)", "limitations": "Authors need to include a section about limitations and weaknesses of AbDiffuser as well as potential improvement of the method. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Originality: There are few tools that have already addressed the same problem (for example, DiffAb, https://github.com/luost26/diffab). However, due to the lack of systematic evaluation and comparison between the approaches, AbDiffuser remains useful and important new tool. \nQuality: The submission provides both in silico and in vitro validation that is crucial for antibody design. Authors provide a detailed supplementary with clear explanation of used methods and approaches. However, the paper lack a proper discussion about potential weaknesses and limitations of AbDiffuser.\nClarity: The test is clear and all the sections are presented in structures manner. \nSignificance: The results are important and useful for biology and drug discovery in particular. AbDiffuser can be used by researchers for a fast and efficient design of antibodies to particular antigen. ", "weaknesses": "Even though the paper provides comparison of AbDiffuser with other models, it lacks comparison to specific to antibody-antigen diffusion models (for example, https://github.com/luost26/diffab). It can be useful to discuss AbDiffuser compared to these tools. Also, there are several new protein-focused diffusion models such as RFdiffusion (https://github.com/RosettaCommons/RFdiffusion) or DiffDock-PP (https://github.com/ketatam/DiffDock-PP), the discussion of or comparison to can be valuable. I suggest extending the \"Related Work\" section to highlight the mentioned above issues. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and recognising the potential of our work. We answer your comments below.\n\n> There are few tools that have already addressed the same problem (for example, DiffAb, https://github.com/luost26/diffab). (..) Even though the paper provides comparison of AbDiffuser with other models, it lacks comparison to specific to antibody-antigen diffusion models (for example, https://github.com/luost26/diffab). It can be useful to discuss AbDiffuser compared to these tools.\n\nFrom a method standpoint, DiffAb and similar tools (e.g., MEAN) aim to re-design the CDR loops in a known co-crystal structure, and not the full antibody as we do. Often, as in the case of DiffAb, they are only tested at re-generating one CDR loop at a time. We discuss the works using such approaches in more detail in the related work (Appendix A, line 650 and briefly in lines 41-42 of the main text).\n\nTo test how well these methods work in comparison to AbDiffuser, we retrained DiffAb and dyMEAN for the HER2 binder design task (see our general response above). For completeness, we also added experiments on conditioned CDR redesign using the SAbDab split from DiffAb. The results showcase that in the conditional CDR redesign AbDiffuser performs much better in terms of CDR sequence recovery than DiffAb with an impressive 1.58x improvement in CDR H3 amino acid recovery.\n\n> However, the paper lacks a proper discussion about potential weaknesses and limitations of AbDiffuser.\n\nThe two main limitations that we see are that 1) our method as is can only be applied to proteins that belong to a conserved fold (such as the Abs, TCRs, TIM-barrels, etc), and 2) that the original experiments focused on unconditional generation, so each new target will require a new dataset (obtained e.g., by a high throughput display experiment).  We mention these limitations in the paper, but we will make sure to highlight them further in the final version.\n\nWe also note that the second limitation was mitigated by the new experiment we performed on SAbDab CDR inpainting task – the same experiment performed by DiffAb (see general response).\n\n> Also, there are several new protein-focused diffusion models such as RFdiffusion (https://github.com/RosettaCommons/RFdiffusion) or DiffDock-PP (https://github.com/ketatam/DiffDock-PP), the discussion of or comparison to can be valuable. I suggest extending the \"Related Work\" section to highlight the mentioned above issues.\n\nThe paper discusses RFDiffusion and explains that the latter’s utility for Abs is limited as the method is not built to generate the sequence but only the protein backbone (which is less challenging for Abs). We will extend the discussion accordingly and include DiffDock-PP in it, explaining that there are interesting advances in protein design that address different problems from generation (that we focus on). \n\n"}, {"review_id": "6El9nmXixT", "reviewer": "Reviewer_is6j", "summary": "The authors propose a new approach, called AbDiffuser, that improves protein diffusion by leveraging domain knowledge and physics-based constraints. They follow MLPMixer's architecture to reduce memory complexity by an order of magnitude, enabling backbone and side chain generation. They validate AbDiffuser in silico and in-vitro.", "questions": "- During forward diffusion, is the residue type continuous or discrete? If it is continuous, how do you perform \"backbone/side-chain projection\" (section 3.4) since the type of residue is undecided? If it is discrete, during the forward process (by interpolating between a prior distribution and the final one-hot distribution), will the residue type remain unchanged? For example, assuming there are 5 amino acid types, the true state is (0, 0, 0, 0, 1) and the prior distribution is (0.2, 0.2, 0.2, 0.2, 0.2), the residue type will always be the 5th state.\n- Are the priors computed over the entire dataset, instead of some antibody family? What would be the performance if directly sampling from the sequence and structure prior?", "limitations": "NA", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- APMixer architecture significantly reduces GPU memory, which is an interesting improvement since many related works rely on a triangular update mechanism that has O(N3) complexity. It is interesting whether such an approach can be adopted for more general tasks, such as protein folding and design.\n- The projection layer looks interesting, as it helps maintain the intermediate diffusion state on the \"manifold\" of valid structures. Similar to the above, it remains unknown whether this may benefit general protein design.\n- The authors validate the method in-vitro, increasing the solidness and usefulness.", "weaknesses": "- AbDiffuser does not directly use epitope information, which may harm generalization. For example, given a newly seen antigen, how can this method be applied to design an antibody that has a high binding affinity?\n- Since the sequence dataset is folded with IgFold (line 271), it may indicate that AbDiffuser's \"folding\" performance is bounded by IgFold. The model is ranked with another classifier trained with binding probability (line 316). It appears that the IgFold and classifier act as the model's \"teacher.\" How does the model outperform its \"teacher\"? If we have such a \"teacher\", why not try to use it directly, with MCMC, Bayesian optimization, etc?\n- In line 314, a random mutation approach may generate 9k binders and 25k non-binders, indicating that random mutation causes a success rate of 9/(9+25)=26%. In line 66, AbDiffuser has a success rate of 22.2%. Although the number may increase to 57.1% by post processing, does this mean that \"AbDiffuser performs worse than a random mutation baseline\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments, we answer them below.\n\n> AbDiffuser does not directly use epitope information, which may harm generalization. For example, given a newly seen antigen, how can this method be applied to design an antibody that has a high binding affinity?\n\nThis is a pertinent question. As explained in lines 359-361, to create diversified and improved binders for a new antigen, we would need a starting dataset of potential binders. Thankfully this can be relatively easily acquired via high-throughput experiments (yeast & phage display, immunogenic campaigns). Due to their speed and low-cost, these experiments are the 1st step in drug discovery taken to roughly scope the Ab landscape when looking for a new drug. Starting with this collection of potential binders, the aim of the HER2 experiment is to select diversified high-binding molecules that, down the line, can be tested for good developability properties. On the other hand, co-crystal structure determination (needed by the current CDR-redesign methods such as DiffAb or MEAN) is usually done only for valuable identified leads.\n\nNevertheless, to prove that AbDiffuser can also bring benefits to the conditional generation task, we performed a new experiment on SAbDab CDR-redesign, where we compare against DiffAb and Rosetta (see general reply). The method extension was achieved by adding a GNN that performed information exchange between Ag and Ab as was done previously. \n\n> ... AbDiffuser's \"folding\" performance is bounded by IgFold. The model is ranked with another classifier trained with binding probability (line 316). It appears that the IgFold and classifier act as the model's \"teacher.\" How does the model outperform its \"teacher\"? Why not try to use such a \"teacher\" directly, with MCMC, Bayesian optimization, etc?\n\nIndeed, the model’s folding capacity is upper bounded by IgFold folding performance when trained only on folded structures, though as we also show in the new experiments it’s also possible to train on crystal structures. Also, the classifier is not used as a teacher but strictly for evaluating how well the generative models model the binder distribution (note that the generative models are trained only on binders, while the classifier sees both binders and non-binders). \n\nNevertheless, it is interesting to consider the spirit of your question. In general, there are two main arguments why employing a generative model to model the distribution is a better choice than using search:  \n\nFirst, there is an intimate interaction between sequence and structure which motivates building a model that generates sequence and structure jointly: structural features are known to be quite informative of what amino acid types are acceptable in a given structural position (e.g., due to charge complementarity of nearby amino acid side chains). So the hypothesis is that generating structure as well, should make the model generate better sequences. This is confirmed by our experiments, since the structure+sequence models outperformed sequence-only models, even when they were trained on much more data.\n\nSecond, a key pitfall of directly searching (MCMC/Bayesian opt) for Abs that abide to some properties according to some classifiers is that the classifier predictions p(y|x) are only trustworthy close to their training data distribution and do not reveal the all-important data likelihood p(x). Unfortunately, without controlling p(x) design is doomed to fail! That is because we can only trust what the classifier predicts if we have the ability to select likely samples according to the distribution it was trained on. The latter is what a good generative model (like a denoising diffusion) excels at, but is not achieved if we search over the space of Abs that the folding model and the classifier predict as positives.  \n\n> During forward diffusion, is the residue type continuous or discrete? (...).\n\nThe residue type is always discrete (both in forward and reverse processes) and is selected by sampling. In your example, if at t=500 the categorical probability distribution was (0.1, 0.1, 0.1, 0.1, 0.6), we would sample from it to produce the noised discrete sample which could result in a residue in any state (it would end up at state 1 with probability 0.1 and so on).\n\n> Are the priors computed over the dataset? What would be the performance if directly sampling from the sequence and structure prior?\n\nWe compute the priors over the training set. But it could make sense to say compute the priors over all known antibody structures and sequences.\n\nSampling the structure prior still gives you a random (Gaussian) point cloud. Atom positions in it are of course correlated, but as a whole it looks nothing like a true antibody. So the performance of any structural metrics is very poor.\n\nIn sequences, the positions in the antibody framework regions can be highly conserved (e.g., we can have just a couple of different amino acid types observed in a particular AHo position). So randomly sampling the prior would give you a highly statistically similar sequence in the framework region (though it might not achieve high expression in vitro). CDR regions are much more diverse, especially CDR H3 and CDR L3. So there the sampling is more random. When we use the priors to sample sequence and structure for paired OAS, we get WD(Naturalness): 0.4004, WD(Closeness): 0.2969, WD(Stability): 0.3871, RMSD:  17.3944. These results are an order of magnitude worse than what we can achieve with AbDiffuser: WD(Naturalness): 0.0916 , WD(Closeness): 0.0520, WD(Stability): 0.0186, RMSD:  0.4962.\n\n> Random mutation approach success rate\n\nIn short, this is not true as the dataset considered does not contain random mutations but combinations of carefully selected mutations determined by experts using a deep mutagenesis experiment (see Mason et al, DOI: 10.1038/s41551-021-00699-9). Due to the space limit of the rebuttal we will expand on this in the discussion period.\n\n"}, {"review_id": "HgfoZzPMQ6", "reviewer": "Reviewer_9R9d", "summary": "The paper proposes AbDiffuser, a new diffusion model for antibody structure and sequence generation. AbDiffuser proposes several ideas to improve generation: (i) Euclidean diffusion with frame averaging to achieve SE(3) equivariant diffusion, (ii) antibody sequence alignment to standardize sequence length across antibodies, (iii) atom to rigid body projections to maintain inter-residue physical constraints, and (iv) informative priors when sampling sequence and structure. The method is evaluated on the Observable Antibody Space and HER2 binder dataset against a selection of prior works but not state-of-the-art baselines. Performance is measured with how close the generated sequences match statistics of the test set as well as some physics based metrics. The authors ablate most components of the method to find it improves performance. Lastly, they demonstrate 16 of their designs fold into antibodies and have high binding affinity to HER2.", "questions": "My questions have been written above.", "limitations": "Limitations are not discussed though I believe there are many issues with the paper.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- AbDiffuser explores many ideas that are different to current state-of-the-art protein generation diffusion models. (1) Frame averaging for conferring SE(3) equivariance to MLPMixer over positions. (2) Aligning antibodies to a reference sequence for handling variable lengths. (3) Projecting coordinates to rigid body constraints on input and output of the denoising model. (4) Informative priors over the coordinates and sequence based on training set statistics. Each component of the method seems to help based on the task and metrics defined in the paper.\n\n- I appreciate the wet-lab validation of their designs which is almost never present in protein papers at machine learning conferences.\n\n\n\n", "weaknesses": "- **Biggest issue: poor presentation**. There are simply too many new ideas that comprise AbDiffuser and all the details are left to the appendix. The reader should not have to go through 28 pages (main text and appendix) to understand the method. Even after looking through the appendix, I am unsure what the full algorithm is. In fact, there is no algorithm or figure of how AbDiffuser works from training to sampling.\n\n- **Missing motivation for frame averaging**. Frame averaging (FA) is introduced without motivation. On line 37-38, the authors state previous methods that confer SE(3) equivariance and physical constraints come with \"increased complexity\". These complexities are never spelled out. How is AbDiffuser simpler? The paper proposes frame averaging with informative priors *and* the rigid body projection step. This seems quite complicated to me. Additional points on FA:\n\n- **Missing related work**.\n  - FA for proteins. the authors fail to cite [1] which already proposed FA for proteins.\n  - The projection step seems related to the projection step, \"Equivariant Consensus Structure from Weighted Inter-residue Geometries\", in Appendix L of Chroma but this connection is not mentioned.\n\n- **Gaussian diffusion plus projection step vs. SE(3) diffusion**. The projection step seems to me as a way to circumvent working with rigid body frames by always projecting the gaussian diffusion over coordinates onto the rigid body constraints. But this seems close to SE(3) diffusion [2] over frames where the C-a coordinates diffuse along with the backbone atoms which always have the rigid body constraints. Why is the gaussian diffusion and projection step necessary to develop if we have SE(3) diffusion? Furthermore, SE(3) diffusion is not compared with so we do not know which is better.\n\n- **Unjustified claims**. I have concerns with several claims in the paper.\n  - Line 3 claims AbDiffuser uses a new representation of protein structure which as I understand is the FA component. But as stated this has already been done [1].\n  - Line 5-6 \"Our approach improves protein diffusion ... handles sequence-length changes\". This is not true. The authors achieve this by aligning all antibodies to a reference alignment (the AHo numbering) that has been specifically developed for antibodies. This does not exist for nearly all proteins so their method cannot be applied. Even if it can be applied, the authors cannot claim it works for other proteins until it is demonstrated.\n  - Line 67 - 69 \"These results provide the first experimental evidence that a generative model trained on mutagenesis data can create new antibody binders...\". This is not true. [4] is just one paper that showed new antibody binders can be created with generative models. In addition, RFdiffusion [5] which can generate de novo binders is not mentioned.\n  - Title \"AbDiffuser: full-atom generation of in-vitro functioning antibodies\". This a bold claim that is **not justified** by the results of generating 16 functioning antibodies for a single target HER2. Furthermore, it is unclear how novel these antibodies are.\n  - Line 87. The authors claim APMixer is a novel neural network for processing proteins. The claimed novelty seems very incremental. MLPMixer is not new, FA is not new. The only component which I believe is novel is the sequence alignment of each antibody such that they all have the same sequence length as input. But this does not warrant technical novelty. Furthermore, I don't understand why SE(3) universality isn't already provided by FA.\n\n- **Unclear experimental set-up**. The experimental set-up is poorly written. The metrics are not clearly defined in the main text. The training dataset is unclear: OAS does not have structures so how are the structures obtained for training? Do the authors train RefineGNN and MEAN baselines only on the CDR which they were designed for or also give them the full sequence? How are the splits performed? The Wasserstein distance metric has no equation so I am unclear what it is exactly measuring. Why is there no metric on diversity and novelty of the sequences? Perhaps this information is buried in the appendix but it should be up front in the main text.\n\n- Line 250-252. Why would we expect IgLM to have the best accuracy just because it was trained on more data? IgLM is not trained on the structure like AbDiffuser or the other baselines.\n\n- **Experimental results**. The authors claim they have designed binders which have been tested in the wet-lab but this result must be accompanied by experimental results in the supplementary or plots in the supplement. This is standard in bio journals. In addition it is unclear how novel each design is. If only a few mutations were necessary to obtain a tight binder then this task is not difficult.\n\n- **Lack of baselines**. The authors miss out on state-of-the-art. On top of the ones already discussed there have been diffusion models for antibody generation [6, 7].\n\n[1] https://arxiv.org/pdf/2301.10814.pdf\n\n[2] https://arxiv.org/abs/2302.02277\n\n[3] https://www.biorxiv.org/content/10.1101/2022.12.01.518682v1.full.pdf\n\n[4] https://www.biorxiv.org/content/biorxiv/early/2023/03/29/2023.01.08.523187.full.pdf\n\n[5] https://www.biorxiv.org/content/10.1101/2022.12.09.519842v1\n\n[6] https://www.biorxiv.org/content/10.1101/2022.07.10.499510v2\n\n[7] https://arxiv.org/abs/2302.00203", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your in depth questions. We urge you to carefully consider our reply as it addresses the issues you raised and used to motivate your low score.\n\n> Biggest issue: poor presentation. There are simply too many new ideas that comprise AbDiffuser and all the details are left to the appendix. The reader should not have to go through 28 pages (main text and appendix) to understand the method. Even after looking through the appendix, I am unsure what the full algorithm is. In fact, there is no algorithm or figure of how AbDiffuser works from training to sampling.\n\nThank you for appreciating the number of contributions the paper makes. Due to the limited space we naturally focus on our main contributions in the main paper and delegate less innovative parts, such as the definition of the metrics and the analysis of the results to the appendix. To improve the at-a-glance understanding of the method in the PDF attached to the main response we included sampling and training pseudo algorithms. We will also include them in the final version.\n\n> Frame averaging (FA) is introduced without motivation.\n\nThe motivation for frame averaging is twofold 1) the original paper has shown it to compare favorably to alternatives 2) it allows us to conveniently deal with SE(3) equivariance and frees us to utilize a non-relational equivariant model using well understood an easy-to-train blocks (MLPs). \n\n> On line 37-38, the authors state previous methods that confer SE(3) equivariance and physical constraints come with \"increased complexity\". These complexities are never spelled out. How is AbDiffuser simpler? The paper proposes frame averaging with informative priors and the rigid body projection step. This seems quite complicated to me.\n\nWe refer to the fact that building neural networks that work on rotation manifolds is complicated by the non-euclidean structure of the manifold. There are many options for how the rotations can be represented and what losses can be used. These choices have big implications on the model performance, and usually rely on approximations. We elaborate on these approximations for the SO(3) diffusion in our answer to one of your follow up comments.\n\nAnother point is that SE3-equivariant GNNs are limited in their ability to capture global geometry (https://arxiv.org/pdf/2301.09308.pdf). Further, these GNN and transformer-based models are well known to require a lot of memory which hinders them from scaling to the full-atom generation we do here. We also witnessed the expressivity issues of GNNs in our numerical experiments, where the GNN models consistently achieved worse RMSD than AbDiffuser variants.\n\nAdmittedly, “complexity” was too general a term here. In the final version we will write that “ .. they can require various approximations and architectural considerations that tend to lead to an increased memory complexity.”.\n\nOn the other hand, informative priors do not have anything to do with the model complexity or the complexity of the overall approach. They are add-ons that could potentially be added to any diffusion model. We just show (theoretically and practically) that it makes sense to add this and how to build such data-driven priors. As we show in Table 1, models with such priors removed still function well, albeit with reduced performance.\n\n> FA for proteins. the authors fail to cite [1] which already proposed FA for proteins.\n\nThank you for the reference, we will include it. Note that the authors of this work apply FA to the usual relational self-attention model. We use FA to build a non-relational model, which, as shown in Table 2, has significantly smaller memory/time consumption.\n\n> The projection step seems related to the projection step, \"Equivariant Consensus Structure from Weighted Inter-residue Geometries\", in Appendix L of Chroma but this connection is not mentioned.\n\nAs the title of the referenced chapter suggests, the paper you referenced concerns interactions between residues. This is different from our projection step, which handles the constraints within each residue. Specifically, their approach changes the model to output a specification for an optimization problem that is then solved to build the protein structure (similar to how AlphaFold1 outputs a distance matrix that is later used to find atom positions by gradient descent). In our case the model directly outputs the large-scale protein structure, similarly to for example AlphaFold2. So we don’t really see any similarity between our projection layer and the chapter you reference: the two approaches solve different problems in different ways.\n\n> Line 3 claims AbDiffuser uses a new representation of protein structure which as I understand is the FA component. But as stated this has already been done [1].\n\nThe representation we propose goes beyond frame averaging (which can also be employed with GNN & transformers as we and others have demonstrated). \n\nWe are proposing to represent members of an aligned protein family by a fixed size “table”, which holds amino acid types and all atom positions and can accommodate any antibody and any side-chains. To achieve this, we rely on a global family alignment scheme and the 4-atom sidechain representation we introduce in Figure 1 and lines 176-200. A key distinction of our representation is that it goes against the common practice of thinking of residues as a set (attention / GNNs) or a variable-length sequence (when adding positional encodings). As we explain in the paper, our representation simplifies learning by allowing the model to associate specific structural roles to particular table rows as well as to easily change the length of a protein mid-generation. The practical advantages of our representation are consistently illustrated in our experiments.  \n\nAs you have raised a lot of questions there is not enough space in the rebuttal to answer them all. We will answer the remaining ones in the comments bellow, as soon as they are enabled."}, {"review_id": "aCrX7v2ObW", "reviewer": "Reviewer_NpkL", "summary": "This paper introduces AbDiffuser, a new equivariant diffusion model, designed to generate full antibody structures efficiently and with high quality. By incorporating family-specific priors and utilizing a novel neural network architecture called APMixer, AbDiffuser demonstrates improved performance in modeling the sequence and structural properties of natural antibodies. Experimental validation shows that the generated antibody binders have high affinity, improving the affinity of a cancer drug. These findings highlight the potential of generative models in creating high-affinity antibody binders without the need for post-selection by binding predictors.", "questions": "Table 2 demonstrates that having more parameters is beneficial, contrary to the general notion that fewer parameters are desirable. It raises the question of why an increased number of parameters is advantageous in this specific context and how it leads to improved performance. Could the authors elaborate on the reasons behind the observed benefit of having a larger parameter count in their model?", "limitations": "The choice of HER2 binder design presented in the paper could potentially be limited to being just one successful case study. It is important to acknowledge that the success achieved in this particular case study does not guarantee similar outcomes in other design scenarios. Providing a clear remark about this limitation would be valuable for the scientific community, considering that the drug discovery process is known to be expensive and challenging. It would help manage expectations and promote a realistic understanding of the potential generalizability and applicability of the proposed methodology beyond the specific case study presented.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The proposed approach outperforms other baseline methods across various metrics related to antibody designing. This could indicate potential effectiveness in generating high-quality antibody structures.\n\n* The novel architecture introduced in this paper exhibits a lower memory footprint during training, which is a significant advantage compared to the baseline architectures. This not only reduces resource requirements but also enhances the efficiency of sample generation, making it a more practical and scalable solution.\n\n* The design choices made in this work are backed by theoretical results and are further supported by in silico experiments. The alignment between theoretical motivations and experimental outcomes adds credibility to the overall approach.", "weaknesses": "* The core components of the proposed model, including the diffusion model and the mixer model, bear a strong resemblance to corresponding reference models. While there are some minor additions, such as the physics-informed residue representation, AHo-specific residue frequencies, and encoding conditional atom dependencies, these modifications may be viewed as relatively insignificant. This could raise questions about the extent of novelty and originality introduced by the proposed model.\n\n* In lines 138-139, it would be beneficial for the authors to provide further insights or explanations regarding why the second choice is expected to improve the performance. This would enhance the clarity and understanding of the motivations behind the model design decisions.\n\n* A minor typographical error is present at line 173, where a missing space between \"\\AA\" and \"while\" is observed. While this is a minor issue, attention to detail and thorough proofreading are important aspects of a scientific publication.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We address your comments in turn.\n\n> The core components of the proposed model, including the diffusion model and the mixer model, bear a strong resemblance to corresponding reference models. While there are some minor additions, such as the physics-informed residue representation, AHo-specific residue frequencies, and encoding conditional atom dependencies, these modifications may be viewed as relatively insignificant. This could raise questions about the extent of novelty and originality introduced by the proposed model.\n\nFrom a neural network architecture perspective, our main contribution lies in the “table” representation we propose and not in the specific combinations of layers we employ. Our approach stands in contrast to current relational approaches (GNNs and transformers), is proven to be universal (whereas for usual relational models such as equivariant GNNs universality is not guaranteed (https://arxiv.org/pdf/2301.09308.pdf), and yields demonstrable practical benefits (1-order of magnitude less memory, improved generative model quality, seamless handling of sequence length changes). \n\nAt the same time, it is crucial to stress that our work goes beyond the proposal of a neural network and innovates both in the generation aspect (with the introduction of the informative priors and the proof of the benefit they bring in Theorem E.1) and the physics informed residue representation (with the projection layer). The former is shown to carry strong empirical benefits, and the latter makes it for the first time possible to respect the known geometry constraints of protein residues while operating in an external coordinate frame (i.e., coordinates and not angles), thus allowing us to use Gaussian diffusion while respecting the constraints and avoiding the more inaccurate IGSO3 diffusion (https://openreview.net/pdf?id=BY88eBbkpe5).\n\nIn our view, the current antibody generative models are more direct adaptations of graph or sequence generative models, while in this work we aimed to design an approach where each component was specifically made with antibodies in mind.\n\n> In lines 138-139, it would be beneficial for the authors to provide further insights or explanations regarding why the second choice is expected to improve the performance. This would enhance the clarity and understanding of the motivations behind the model design decisions.\n\nThe main reason for applying the frame averaging on every layer separately was that it performed better in our experiments. We don’t have a mathematical reason to explain this. \n\n> A minor typographical error is present at line 173, where a missing space between \"\\AA\" and \"while\" is observed. While this is a minor issue, attention to detail and thorough proofreading are important aspects of a scientific publication.\n\nThank you for pointing this out. We will make sure to proofread the paper a few more times for the final version.\n\n> Table 2 demonstrates that having more parameters is beneficial, contrary to the general notion that fewer parameters are desirable. It raises the question of why an increased number of parameters is advantageous in this specific context and how it leads to improved performance. Could the authors elaborate on the reasons behind the observed benefit of having a larger parameter count in their model?\n\nAs far as we are aware, the good performance of deep neural networks is attributed to overparameterization, as it is presumed to smooth the loss landscape (e.g. see http://proceedings.mlr.press/v119/buhai20a/buhai20a.pdf). \nIn addition to this, the biggest risk of having too many parameters is higher potential of overfitting but this is less of an issue for generative/diffusion models.\nIt is also the case that, for example, large language generative model performance increases with size (e.g. Table 3 in https://arxiv.org/pdf/2307.09288.pdf).\n\n> The choice of HER2 binder design presented in the paper could potentially be limited to being just one successful case study. It is important to acknowledge that the success achieved in this particular case study does not guarantee similar outcomes in other design scenarios. Providing a clear remark about this limitation would be valuable for the scientific community, considering that the drug discovery process is known to be expensive and challenging. It would help manage expectations and promote a realistic understanding of the potential generalizability and applicability of the proposed methodology beyond the specific case study presented.\n\nIt is true that drug design is a complicated endeavor, that there are many variables that can impact the outcome, and that success is not guaranteed. Our model, like any other, is not exempt from this: naturally, the achieved binding rates and the required dataset sizes could differ quite a bit depending on the problem at hand. We believed that this was already clear, but we will be happy to make it explicit.\n\nThe fact that our model does well on general antibody generation (paired OAS) gives some hope that it is able to properly capture many different aspects of antibodies and that the results would transfer to some extent. \n\nTo also evaluate our model in different scenarios, we tested our model on antigen conditioned CDR inpainting on the SAbDab split used in DiffAb. You can find the results in the general response. It can be seen that our model does quite well on this task, which further increases our hope of it generalizing to other antigens and problems.\n"}]

Author rebuttal:
We thank all reviewers for their thoughtful comments and suggestions. In response, we performed additional experiments, adding two state-of-the-art baselines to our current comparison and showcasing how AbDiffuser significantly outperforms previous methods in SAbDab CDR inpainting.

### HER2 binder generation task – DiffAb and dyMEAN (Table 3). 

We found that DiffAb trained on HER2 binders focused on high-likelihood modes of the distribution and did not generate perfectly unique samples, but achieved good binding probabilities. To produce a head-to-head comparison, we also generated samples from AbDiffuser by removing the additional noise added during every step of the reverse process, similarly to what was proposed in RFDiffusion. As shown in the table below, the binding probability was greatly increased (surpassing DiffAb) at the cost of focusing more on high-likelihood modes.

| Model | \\(W_1(\text{Nat.})\downarrow\\) | \\(W_1(\text{Clo.})\downarrow\\) | \\(W_1(\text{Sta.})\downarrow\\) | \\(W_1(\text{PSH})\downarrow\\) | \\(W_1(\text{PPC})\downarrow\\) | \\(W_1(\text{PNC})\downarrow\\) |  \\(W_1(\text{CSP})\downarrow\\) | \\(W_1(\Delta G)\downarrow\\) | \\(\text{RMSD}\downarrow\\) | \\(p_{\text{bind}}\uparrow\\) | \\(\text{Uniq.}\uparrow \\) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| DiffAb | 0.0074 | 0.0014 | 0.0498 | 0.5481 | 0.0097 | 0.0067 | 3.4647 | 6.7419  | 0.4151 | 0.8876 | 99.7% | 
| AbDiffuser | 0.0013 | 0.0018 | 0.0028 | 0.4968 | 0.0205 | 0.0113 | 0.1588 | 6.4301 | 0.3822 | 0.5761 | 100% |
| AbDiffuser (side chains) | 0.0010 | 0.0005 | 0.0062 | 1.2909 | 0.0115 | 0.0029 | 0.0948 | 32.0464 | 0.4046 | 0.6848 | 100% |
| AbDiffuser (no noise) | 0.0008 | 0.0014 | 0.0265 | 2.5944 | 0.0206 | 0.0053 | 0.2378 | 15.2200 | 0.3345 | 0.9115 | 99.7% |
| AbDiffuser (side chains, no noise) | 0.0015 | 0.0024 | 0.0159 | 1.5043 | 0.0210 | 0.0126 | 0.5173 | 114.4841 | 0.6795 | 0.9436 | 91.4% |


We also trained dyMEAN on the HER2 dataset (code became available after the NeurIPS submission deadline). Unfortunately, we found that dyMEAN was very prone to collapse. Over 4 distinct runs, the model with the best validation loss always generated only a single example. 

### Full-Ab generation on paired OAS - dyMEAN (Table 1). 

In contrast to MEAN and DiffAb, the latest version of dyMEAN has been shown to be able to generate full antibodies. We thus also trained dyMEAN on the paired OAS generation (Table 1) and got the following results:

| Model | \\(W_1(\text{Nat.})\downarrow\\) | \\(W_1(\text{Clo.})\,\downarrow\\) | \\(W_1(\text{Sta.})\downarrow\\) | \\(W_1(\text{PSH})\downarrow\\) | \\(W_1(\text{PPC})\downarrow\\) | \\(W_1(\text{PNC})\downarrow\\) |  \\(W_1(\text{CSP})\downarrow\\) | \\(W_1(\Delta G)\downarrow\\)  | \\(\text{RMSD}\downarrow\\) | \\(\text{Uniq.}\uparrow \\) |
|---|---|---|---|---|---|---|---|---|---|---|
dyMEAN | 0.1319 | 0.1600 | 0.0423 | 3.9145 | 0.1566 | 0.2929 | 2.3711 | 601.1153 | 3.8157 | 58.2% |
AbDiffuser | 0.1979 | 0.0921 | 0.0662 | 2.3219 | 0.0314 | 0.0285 | 0.6662 | 13.3051 | 0.5230 | 100% |
AbDiffuser (side chains) | 0.0916 | 0.0520 | 0.0186 | 6.3166 | 0.0209 | 0.0754 | 0.8676 | 16.6117 | 0.4962 | 100% | 

Only 58.2% of the 1k generated sequences were unique, which is problematic considering that paired OAS has 100k antibodies in the training set. Our results indicate that further advances would be needed to avoid mode collapse and to accurately model paired OAS with dyMEAN. 

### SABDab CDR inpainting 

We evaluated AbDiffuser on the SABDab co-crystal structure CDR inpainting task from DiffAb using their splits and experimental setup (Section 4.1 in https://www.biorxiv.org/content/10.1101/2022.07.10.499510v5.full.pdf). We conditioned AbDiffuser on the antigen by replacing the initial embedding MLP in APMixer by the output of a GNN layer that passes messages from the closest 10 antigen residues to the CDR residues. We will add further details in the supplement.

The table below shows that AbDiffuser outperformed Rosetta and DiffAb on amino-acid recovery with a wide margin. The CDR RMSD results are quite comparable between DiffAb and AbDiffuser, but importantly AbDiffuser does better than DiffAb on CDR H3 and CDR L3 which have the most variability in their structure and can influence binding the most. The good performance of AbDiffuser in amino-acid recovery could be attributed to the usefulness of AHo numbering and APMixer likely being better at modeling sequences than a GNN. Our ablation experiments on paired OAS generation (Table 1) corroborate this hypothesis. 

| Model | AA CDR H1 | AA CDR H2  | AA CDR H3 | RMSD CDR H1 | RMSD CDR H2 |  RMSD CDR H3 | AA CDR L1 | AA CDR L2  | AA CDR L3 | RMSD CDR L1 | RMSD CDR L2 |  RMSD CDR L3 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Rosetta (RAbD) | 22.85% | 25.5% | 22.14% | 2.261 | 1.641 | 2.9 | 34.27% | 26.3% |  20.73% | 1.204 | 1.767 | 1.624 | 
| DiffAb | 65.75% | 49.31% | 26.78% | 1.188 | 1.076 | 3.597 | 55.67% | 59.32% | 46.47% | 1.388 | 1.373 | 1.627 |
| AbDiffuser | 78.48% | 63.68% | 42.19%  | 1.507 | 1.296 | 3.431 | 83.84% | 90.35% | 70.98% | 1.268  | 1.433  | 1.312 |

AA stands for Amino Acid recovery and RMSD measures the CA position difference to the CDR CA positions in the original co-crystal structure.

DiffAb also included the estimated improvement in binding score (Rosetta ddG) in their metrics. Since Rosetta ddG has been proven to be an unreliable metric for determining binding affinity (see e.g., Figure 2b in https://pubs.acs.org/doi/10.1021/acs.jpcb.7b11367) we avoid using it for model comparison.

Note that we used the DiffAb split as it properly ensures that antibodies similar to the ones in the test set are removed from the training set (sequences are clustered for up to 50% CDR H3 identity). dyMEAN in their split does not ensure this non-overlap so their reported results are not comparable. 

Additionally we attach a PDF with training and sampling pseudocode.


Meta-review: This paper presents an antibody-specific diffusion model that can be used for antibody design, an important application in biological sciences. Reviewers appreciated the design of the method, which leverages antibody domain knowledge and physics-based constraints, and the extensive evaluation of the method, including experimental results validating the antibody designs.
Decision: Accept (spotlight)