Title: AbDiffuser: full-atom generation of in-vitro functioning antibodies
Venue: NeurIPS 2023
Accepted: yes
Keywords: antibody generation

diffusion

equivariance
Abstract: We introduce AbDiffuser, an equivariant and physics-informed diffusion model for the joint generation of antibody 3D structures and sequences. AbDiffuser is built on top of a new representation of protein structure, relies on a novel architecture for aligned proteins, and utilizes strong diffusion priors to improve the denoising process. Our approach improves protein diffusion by taking advantage of domain knowledge and physics-based constraints; handles sequence-length changes; and reduces memory complexity by an order of magnitude, enabling backbone and side chain generation. We validate AbDiffuser in silico and in vitro. Numerical experiments showcase the ability of AbDiffuser to generate antibodies that closely track the sequence and structural properties of a reference set. Laboratory experiments confirm that all 16 HER2 antibodies discovered were expressed at high levels and that 57.1% of the selected designs were tight binders.

AbDiffuser: Full-Atom Generation of in vitro Functioning Antibodies

Abstract

We introduce AbDiffuser , an equivariant and physics-informed diffusion model for the joint generation of antibody 3D structures and sequences. AbDiffuser is built on top of a new representation of protein structure, relies on a novel architecture for aligned proteins, and utilizes strong diffusion priors to improve the denoising process. Our approach improves protein diffusion by taking advantage of domain knowledge and physics-based constraints; handles sequence-length changes; and reduces memory complexity by an order of magnitude, enabling backbone and side chain generation. We validate AbDiffuser in silico and in vitro . Numerical experiments showcase the ability of AbDiffuser to generate antibodies that closely track the sequence and structural properties of a reference set. Laboratory experiments confirm that all 16 HER2 antibodies discovered were expressed at high levels and that 57.1% of the selected designs were tight binders.

1 Introduction

We focus on the generation of immunoglobulin proteins, also known as antibodies, that help the immune system recognize and neutralize pathogens. Due to their potency and versatility, antibodies constitute one of the most popular drug modalities, with 10 out of 37 newly FDA-approved drugs in 2022 being immunoglobulins [58]. The ability to generate new antibodies with pre-defined biochemical properties in silico carries the promise of speeding up the drug design process.

Several works have attempted to generate antibodies by learning to form new sequences that resemble those found in nature [21; 19; 82]. An issue with sequence-based approaches is that it is hard to determine the properties that render a protein a functional molecule (and an antibody a potent drug) without inspecting a 3D model of the functional state such as an interface or active site. So far, almost all of the first design methods that have enabled novel protein design used carefully curated structurefunction information to score the designs [45; 48; 30]. The determinant role of structure on function has motivated numerous works to co-design sequence and structure [4; 24; 89; 34; 43; 3; 51] or to first design the protein backbone and then fill in the amino acid identities [90; 32; 15; 87; 50; 92; 47].

An emerging paradigm for protein generation is that of equivariant diffusion [90; 89; 51]. Protein diffusion models combine ideas from equivariant and physics-informed protein structure representations [37] with advances in denoising diffusion probabilistic models [28] to gradually transform noise to a partial structure. The generated structure is then refined by adding side chains and optimizing atom positions to form a full-atom 3D model of a protein.

A pertinent challenge when generating protein structure is satisfying the appropriate equivariance and physics constraints while also balancing modeling complexity and fidelity. Most current protein models rely on equivariant transformers [37; 73] or graph neural networks [11; 43] to satisfy SE(3)

equivariance and typically represent parts of the protein geometry in angle space. The latter recipe can be set up to build a physics-informed model that respects (many) desired constraints but comes with increased complexity. As a consequence, these models are expensive to train and are often applied to a partial protein, ignoring the side chain placement [43; 51] or even deferring the determination of amino acid identity to a later stage [90; 32]. Alternatively, some works focus on the regeneration of complementarity-determining regions (CDRs) of antibodies that are of particular importance to binding [34; 43; 33; 53; 78; 44], which also helps to reduce the complexity of the problem.

Our work is motivated by the observation that key large protein families, here we focus on the antibody protein family, typically have strong properties, such as an ability to be mapped to a reliable sequence ordinate via sequence alignment. Our main contribution is an equivariant diffusion model called AbDiffuser that is designed to exploit these properties. We show that incorporating family-specific priors into the diffusion process significantly improves generation efficiency and quality.

AbDiffuser relies on a new universal SE(3) equivariant neural network that we call the Aligned Protein Mixer (APMixer). In contrast to existing equivariant architectures used for antibodies and proteins, APMixer models residue-to-residue relations implicitly and is particularly effective in handling sequence length changes. Additionally, its significantly smaller memory footprint makes it possible to generate full antibody structures, including framework, CDRs, and side chains. Our approach to residue representation is made possible by a projection method guaranteeing that bond and angle constraints are respected while operating in coordinate and not angle space, a better match to diffusion with Gaussian noise. We also benefit from the effects of overparameterization by scaling the network size to hundreds of millions of parameters on a single GPU; an order of magnitude improvement over the corresponding (E)GNN architectures. Having a powerful model that generates the full antibody structure is shown to be beneficial for the quality of the designed proteins.

We evaluate AbDiffuser on the generation of antibodies from paired Observable Antibody Space (pOAS) [61], modeling of HER2 binders and antigen-conditioned CDR redesign. Numerical experiments demonstrate that the proposed representation, architecture, and priors enable AbDiffuser to better model the sequence and structural properties of natural antibodies. We also submitted a subset of our proposed HER2 binders for experimental validation in a laboratory. Of the 16 samples submitted, 100% were expressed as actual antibodies and 37. 5% bound to the target with an average pKD of 8.7. Of these, the subset containing raw/filtered samples achieved a binding rate of 22.2%/57.1% and an average pKD of 8.53/8.78, with our tightest binder slightly improving upon the affinity of the cancer drug Trastuzumab. These results provide the first experimental evidence that a generative model trained on mutagenesis data can reliably (with high probability) create new antibody binders of high affinity, even without post-selection by learned or physics-based binding predictors.

Due to space limitations, we refer the reader to the appendix for a detailed discussion of related work, for method details including proofs, as well as for additional numerical results.

2 Denoising Diffusion for Protein Generation

This section describes how we utilize denoising diffusion to generate antibody sequence and structure. The ideas presented are influenced by previous work on denoising diffusion [26; 5; 28]. The main contributions that distinguish our work are presented in Sections 3 and 4.

We adopt the denoising diffusion framework in which, given a data point X 0 , the forward diffusion process gradually adds noise to form corrupted samples X t . These samples form a trajectory ( X 0 , X 1 , ..., X t , ..., X T ) of increasingly noisy data, interpolating from the data distribution X 0 ∼ p ( X 0 ) to that of an easy-to-sample prior X T ∼ p ( X T ) , such as a Gaussian. The process is constructed to be Markovian, so that q ( X 2 , ..., X T | X 0 ) = q ( X 1 | X 0 ) Π T t =2 q ( X t | X t -1 ) .

To generate new data, a neural network learns to approximate the true denoising process ˆ X 0 = ϕ ( X t , t ) , which can be achieved by minimizing the variational upper bound of the negative loglikelihood [84; 26; 5]. In our diffusion process, we factorize the posterior probability distribution over the atom positions and residue types:

where a Gaussian and a categorical distribution govern the noise applied to atom positions and residue types, respectively. We cover these in detail in Sections B.2 and B.3. The reverse diffusion process is

joint: the model jointly considers atom positions and residue types ( ˆ X pos 0 , ˆ X res 0 ) = ϕ ( X pos t , X res t , t ) . Throughout the paper, we use X pos ∈ R n × 3 as a matrix of antibody atom positions and X res ∈ R r × 21 as a matrix of one-hot encodings of residue types (20 amino acids and a gap).

3 Aligned Protein Mixer

We present Aligned Protein Mixer (APMixer), a novel neural network for processing proteins from a family of aligned proteins. In particular, we focus on antibodies, although the method can in principle be applied to any sufficiently large protein family by using a family-specific global alignment [81; 57]. We first explain how we ensure SE(3) equivariance and how we account for variable sequence lengths in Sections 3.1 and 3.2. The model is detailed in Section 3.3 and our approach to incorporate physics-based constraints on bond angles and lengths is described in Section 3.4.

3.1 SE(3) Equivariance by Frame Averaging

Any neural network ϕ whose input X has dimension n × 3 can be made equivariant or invariant to group transformations by averaging the model outputs over a carefully selected subset of group elements called frames F ( X ) [68]. For example, this has been used successfully to make equivariant self-attention models for the prediction of protein-ligand binding energy [35].

We achieve the desired equivariance to rotations and translations (the SE(3) group) as follows:

where t = 1 n 1 T X pos is the centroid of our points and the four canonical rotation matrices R forming the four frames F ( X pos ) ⊂ SE (3) needed to achieve equivariance can be determined based on Principle Component Analysis. More specifically, we obtain three unit-length eigenvectors v 1 , v 2 , v 3 corresponding to the eigenvalues λ 1 , λ 2 , λ 3 from the eigendecomposition of the covariance matrix C = ( X pos -1 t ) T ( X pos -1 t ) ∈ R 3 × 3 and define the four frames as

To respect the fixed chirality of proteins observed in humans, we desire equivariance w.r.t. SE(3) and not E(3) which also includes reflections. As such, when constructing frames the third axis sign is not varied but its direction is determined by the right-hand rule (cross product of the first two axes).

SE(3) invariance can be similarly achieved:

We make use of equation 2 when denoising residue types X res as they are invariant to the rotation of the antibody, and equation 1 for the prediction of atom positions X pos .

3.2 Handling Length Changes by Multiple Sequence Alignment

The version of APMixer investigated in this work is built on top of the AHo antibody residue numbering scheme proposed by Honegger and Plückthun [27]. This numbering scheme was constructed in a data-driven fashion using information from known antibody structures. For each residue in an antibody chain, it assigns an integer position in [1 , 149] based on the structural role of the residue (e.g. being in a particular CDR loop or in the framework region between two particular CDR loops). Essentially all known antibodies fit into this representation [27].

As we are modeling paired antibody sequences (the heavy and light chain), the full representation is a 2 × 149 element sequence, where 149 heavy chain elements are followed by 149 light chain elements. We represent AHo gaps physically as 'ghost' residue placeholders; their position is determined in data pre-processing by linearly interpolating between the corresponding atoms of the nearest existing residues (trivial, due to the use of AHo numbering).

Our experiments confirm that the proposed representation consistently improves generation quality. The reason is two-fold: a) Each antibody chain is now represented as a fixed-length sequence with

149 positions that are either empty gaps or are filled with an appropriate amino acid. This fixed-length representation encompasses antibodies with diverse loop lengths via alignment and thus allows our generative model to internally choose how many residues the generated protein will have. In contrast, many of the non-sequence-based protein, small molecule, and graph generative models [49; 54; 28; 90; 32] require the number of elements in the object to be specified beforehand. b) The sequence position directly implies the structural role that the amino acid needs to perform, which makes it easier for the model to pick up structure-specific rules.

3.3 The APMixer Architecture

Our use of a fixed length representation for the immunoglobin variable domain fold family (Section 3.2) also allows us to forgo traditional architecture choices in favor of a more efficient architecture inspired by the MLP-Mixer [86]. We build the model architecture out of blocks, where each block consists of two MLPs, that are applied consecutively on the columns and the rows

of the input matrix X ∈ R r × c , with ρ being an activation function, and using the notation [ k ] = (1 , . . . , k ) . We define the model input as a fixed-size matrix X combining X pos and X res with one sequence element per row ( 2 × 149 rows in total). In each row, we encode the residue type and all of the atom positions for that residue (e.g., the C, C α , N, C β backbone atoms). These input matrix rows are embedded in higher-dimensional vectors using an MLP. Specifically, using this representation, our input matrix X has r = 2 × 149 rows and c = 21 + 4 × 3 columns.

To achieve equivariance to atom translations and rotations, we can either apply frame averaging (Section 3.1) on the whole model or on each AbDiffuser block individually. We chose the second option as in our preliminary experiments this improved performance. Frame averaging can be applied on high-dimensional embeddings simply by splitting them into three-dimensional sub-vectors and using each to compute the SE(3) frames [68]. To account for that residue types are invariant to Euclidean transformations, while the atom positions are equivariant, we split each block's input and output vectors in half, with one half being treated equivariantly and the other invariantly.

Model complexity and SE(3)-universality. APMixer models pairwise interactions implicitly by operating on rows and columns of the input interchangeably. Thus, its memory complexity grows linearly with the number of residues. This contrasts with the usual quadratic complexity of traditional structure-based models and allows us to do more with a fixed run-time, parameter, and/or memory budget. In Appendix G we prove that the model on top of which APMixer is built is SE(3)-universal, meaning that it can approximate any SE(3)-equivariant function.

We also remark that, in principle, other models, such as a 1D CNN or a transformer, could be used in place of the MLPs in APMixer. With such sequence-length-independent models, we would no longer require a multiple sequence alignment of the given protein family, though this would possibly come at the expense of universality and efficiency.

3.4 Physics-informed Residue Representation by Projection

Atoms within a protein adhere to strong constraints. In principle, a neural network trained with enough data can learn to respect these constraints. However, over a fixed data budget, it can be advantageous to construct a model in a manner that guarantees that its outputs never violate the constraints. Previous work commonly represents proteins in terms of rotation and translation of rigid residue frames and uses idealized residue representations to recover atom positions [37; 73]. Although there is a long tradition of modeling backbone and side chain degrees of freedom in angle space [67; 14], operating in angle space adds modeling complexity and makes diffusion potentially inaccurate [94; 8; 90]. We take a different route and devise a way to respect bond constraints while operating in a global coordinate frame, which works seamlessly with standard Gaussian diffusion for atom positions.

Specifically, inspired by interior-point methods [72] that alternate between optimization and projection steps, we design a new non-parametric projection layer that is applied to both model inputs and outputs. The model and the noising process are allowed to move the atoms freely, and the projection layer then corrects their positions such that the constraints are respected.

Backbone residue projection. We use a reference residue backbone ( C, C α , N, C β ) with idealized bond lengths and angles [17]. As the backbone atoms are rigid, we rely on the Kabsch algorithm [38] to identify the optimal roto-translation between the projection layer's input and the rigid reference residue's atom positions. We apply the transformation to the reference residue and output the resulting atom positions. We also ensure that the distance between the corresponding C and O atoms in the output is 1 . 231 Åwhile staying as close as possible to the input O position. The reference residue is also used to represent the AHo ghost residue atoms. Idealizing a backbone in this way usually results in a negligible RMSE to the original structure of ∼ 5 · 10 -3 Å.

Side chain projection. We employ a similar idea to constrain the placement of side chains. In contrast to structure prediction [37; 46], we cannot use one idealized side chain per amino acid, since the sequence is unknown during generation. Our solution is to convert all side chains to a generic representation with enough degrees of freedom to account for the exact placement of all atoms. The amino acid side chains have up to 4 bonds that they can rotate around by dihedral angles. These degrees of freedom can be captured by constructing a side-chain template that consists of 4 pseudo-carbon atoms, for which the dihedral angles are rotated in the same way as for the original side chain. If the original side chain has fewer degrees of freedom, we simply set the corresponding dihedral angles to 180 ◦ such that the atoms forming the dihedral angle lie on a plane. The projection layer then only has to ensure that the bond lengths between the pseudo-atoms are respected. We set the bond length to 1.54Å because carbon atoms are the most common atoms that form the dihedral angles of real amino acids. This representation can be seen in Figure 1. To recover the full-atom structure, we extract the dihedral angles from the side chain template; the angles are then applied to idealized amino acid-specific templates.

4 Informative Diffusion Priors

It has been hypothesized that diffusion processes for generation should be tailored to the particular task to increase performance [88; 5]. In Appendix E.1, we present a theoretical argument confirming that choosing a good prior reduces the complexity of the learnable model needed to achieve goodquality denoising. Our bound stated in Theorem E.1 reveals that one cannot learn a simple generative model that fits the data well unless the prior has a small Wasserstein distance to the data distribution.

Armed with this knowledge, we next introduce two types of priors that are incorporated into our diffusion model: a) position-specific residue frequencies that describe sequence conservation of the immunoglobulin fold, and b) conditional dependencies between atom positions.

4.1 AHo-specific Residue Frequencies

A separate benefit of having a fixed-length sequence representation (see Section 3.2) is that we can use the residue frequency at each position as a diffusion prior. To do this, we estimate marginal position-specific categorical distributions Q 1 , ..., Q 2 ∗ 149 over residue types from the training data and use these to define the noise in the discrete residue-type diffusion (see Appendix B.3). Since AHo aligns residues based on their structural role within the protein, the estimated prior distributions exhibit significant variability across positions and have low entropy in preserved regions of the immunoglobulin fold. In this way, the noise q ( X res t | X res 0 ) = Cat ( X res 0 ( β t I + (1 -β t ) Q i )) at every step of the forward process depends on the residue position i in the fixed-length sequence representation. Gap frequencies are also captured in these noise matrices.

Lysine

Tyrosine

Full-atom representation

Generic representation

4.2 Encoding Conditional Atom Dependencies

It is a consequence of their chain-like structure that neighboring residue atom positions of proteins are strongly correlated. Theorem E.1 suggests that encoding these correlations within the diffusion process eases the denoising complexity and can free the model to focus on more challenging aspects.

To this end, we capture the conditional independence relations between atom positions by a Gaussian Markov Random Field (GMRF) [42]. The latter corresponds to a Gaussian N (0 , Σ) over atom positions whose precision matrix Σ -1 = L + aI is equal to the shifted graph Laplacian L = diag ( A 1 ) -A associated with the adjacency matrix A . The GMRF operates under the assumption that node features (e.g., 3D positions) for which there is no edge in the adjacency A are conditionally independent. Some concurrent works [32; 36] also considered conditional atom dependencies by hand-crafting correlation matrices that capture chain relations between residues. We take a step further by proposing to automatically learn the sparse conditional dependence relations from the training data. Our approach entails estimating a sparse adjacency matrix A that captures the data variance under the GMRF model. The details are described in Appendix F.

5 Experiments

We evaluate AbDiffuser's ability to design antibodies. After describing the experimental setup and metrics, Section 5.1 presents in silico tests illustrating the effect of our proposed changes on the generation quality. Section 5.2 details in vitro results showcasing the ability of our method to design new expressing antibodies that bind to a known antigen. Further analyses can be found in the Appendices K and L, whereas experiments on CDR redesign in SAbDab are in Appendix N.

Baselines. We compare APMixer with: a) a sequence transformer based on BERT [12; 74] whose output is folded to recover the structure; b) an E(n) Equivariant Graph Neural Network (EGNN) [75] which is a popular choice for tasks such as 3D molecule generation [28], antibody CDR loop inpainting [43], and antibody generation [89]; and c) a FA-GNN [68], corresponding to a standard GNNwith SE(3) equivariance attained by frame averaging. We also evaluate the proposed informative antibody-specific priors using all of these architectures. To ensure the comparison is performed fairly and under similar settings, we always use the projection layer and our diffusion implementation only varying the denoising model architecture. In the OAS task, we also compare our diffusion-based approach with d) the IgLM [80] antibody language model. To ensure that it generates paired OAS-like sequences, we condition the generation on the subsequences of the first and last few amino acids taken from pOAS sequences (something that our approach does not need). It is also important to note that the comparison is not fair, since the publicly available IgLM model was trained on 558M sequences that also include the whole paired OAS (105k sequences) and the test set we use. So, in many ways, IgLM's performance represents the best results we could ever hope to achieve with a sequence-only approach. We also compare with e) dyMEAN [44], which is the only other antibody generative model previously shown to be able to jointly generate the full strucutre and sequence. In the binder generation task, we compare with f) RefineGNN [34], g) MEAN [43] and h) DiffAb [53], three state-of-the-art geometric deep learning methods for CDR redesign.

Metrics. The quality of generated sequences is measured in terms of their naturalness (inverse perplexity of the AntiBERTy [74] model), closeness to the closest antibody in the training set in terms of edit distance, and stability estimated by IgFold. We also verify that the generated antibody sequences satisfy the appropriate biophysical properties using four additional structurebased metrics [69]: CDR region hydrophobicity ( CDR PSH ), patches of positive ( CDR PPC ), and negative charge ( CDR PNC ), and symmetry of electrostatic charges of heavy and light chains ( SFV CSP ). The metrics applied to generated structures focus primarily on the estimated free energy ∆ G using Rosetta [2] and RMSD for backbone heavy atom positions as compared to IgFold [73] predictions. More details can be found in Appendix J.

As we want to estimate how well the entire data distribution is captured, in all cases except RMSD, we report the Wasserstein distance between the scores of the sequences in the test split and the scores of the generated sequences. As a reference, we also report the baseline metrics achieved by the sequences and structures in the validation set. A generative model that approaches or matches these values is effectively as good at modeling the distribution of the specific metric in consideration as i.i.d. sampling. The test set and the generated set always have 1000 examples.

Further details on training and implementation can be found in Appendix M.

5.1 Paired OAS Generation

We focus on matching the distribution of 105k paired sequences from the Observed Antibody Space database [61] folded with IgFold and optimized with Rosetta [2].

Table 1 summarizes the results. Generated samples by AbDiffuser improve upon baselines on nearly all fronts, even compared to the IgLM language model which was trained on magnitudes more data (especially when concerning structure-related metrics). dyMEAN GNN-based model struggles in this distribution modeling task and is the only model tested that does not achieve the perfect uniqueness of the generated samples (58. 2% unique). The experiment also corroborates our analysis (Theorem E.1) on the benefit of informative priors to diffusion: using a position-specific residue type frequency (AHo) and encoding conditional atom dependencies through a learned covariance matrix (Cov.) helps to improve the ability of most models to capture the pOAS distribution. Interestingly, including the learned covariance can sometimes noticeably improve the quality of the generated sequences (FA-GNN), but its strong benefit to structure quality is only felt when the model is powerful enough to model the structure well (APMixer). Inclusion of AHo numbering and position-specific frequencies improves all models. We perform a similar ablation for APMixer by setting the prior distribution to uniform (uniform prior) and observe a similar performance drop.

To interpret the fidelity of the generated structures, we recall that IgFold uses an ensemble of 4 models followed by Rosetta optimization and that, on average, individual IgFold models (before ensembling) achieve an RMSD of 0 . 4239 on the test set. Therefore, in this regard, the structures created by AbDiffuser are nearly indistinguishable from the test set structures (RMSD of 0.4962). A more detailed analysis of per-region RMSD can be found in Appendix K. We further observe that when no projection layer is used and instead one uses standard diffusion to predict the noise added to the atom positions [28], the training becomes less stable and the model can fail to learn to generate good structures. Encouragingly, forcing AbDiffuser to model side chain positions alongside the backbone tends to improve the similarity of the generated sequences (Naturalness, Closeness, Stability). This is likely due to the strong coupling between feasible side chain conformations and residue types. The generated side chains get an average Rosetta packing score of 0.624, whereas folded and Rosetta-optimized structures have a mean packing score of 0.666. Recalling that a packing score above 0.6 is widely considered good [77; 66; 3], we deduce that AbDiffuser is able to generate physically plausible side chain conformations. When available, we also use the side chain positions

Model

W

1

(

Nat.

)

↓

W

1

(

Clo.

)

↓

W

1

(

Sta.

)

↓

W

1

(

PSH

)

↓

W

1

(

PPC

)

↓

W

1

(

PNC

)

↓

W

1

(

CSP

)

↓

W

1

(∆

G

)

↓

RMSD

↓

p

bind

↑

Uniq.

↑

Validation Set Baseline

0.0011

0.0003

0.0061

1.3183

0.0196

0.0114

0.3280

2.0350

-

0.8676

100%

MEAN [43]

0.0072

0.0009

0.0267

8.4184

0.0184

0.0231

0.4108

8.5981

0.7792

0.7767

38.9%

DiffAb [53]

0.0074

0.0014

0.0498

0.5481

0.0097

0.0067

3.4647

6.7419

0.4151

0.8876

99.7%

RefineGNN [34]

0.0011

0.0004

0.0026

0.5482

0.0053

0.0046

0.1260

-

-

0.7132

100%

Transformer (AHo)

0.0014

0.0031

0.0097

1.3681

0.0171

0.0140

0.2657

-

-

0.3627

100%

EGNN (AHo & Cov.)

0.0013

0.0030

0.0102

1.1241

0.0123

0.0236

0.2441

1967.5280

9.2180

0.3626

100%

FA-GNN (AHo & Cov.)

0.0018

0.0030

0.0063

0.4158

0.0107

0.0056

0.2644

76.7852

3.1800

0.4576

100%

0.0013

0.0018

0.0028

0.4968

0.0205

0.0113

0.1588

6.4301

0.3822

0.5761

100%

0.0010

0.0005

0.0062

1.2909

0.0115

0.0029

0.0948

32.0464

0.4046

0.6848

100%

τ

= 0

.

75

)

0.0005

0.0011

0.0054

0.3934

0.0148

0.0129

0.1785

6.2468

0.3707

0.6382

100%

τ

= 0

.

75

)

0.0005

0.0004

0.0126

1.8510

0.0126

0.0017

0.0917

12.8923

0.3982

0.7796

100%

τ

= 0

.

01

)

0.0008

0.0014

0.0265

2.5944

0.0206

0.0053

0.2378

15.2200

0.3345

0.9115

99.7%

τ

= 0

.

01

)

0.0015

0.0024

0.0159

1.5043

0.0210

0.0126

0.5173

114.4841

0.6795

0.9436

91.4%

predicted by the model for the ∆ G energy estimation. Even though this procedure is expected to generate results slightly different when compared to backbone-only models (in the latter case missing side chain atoms are first repacked based on the rotamer libraries before the energy minimization step), we still observe a high overlap between the resultant energies. This further highlights the quality of side-chain prediction.

It should be noted that ∆ G energy computation and some additional structure-based metrics [69] (i.e., CDR PSH , CDR PPC , CDR PNC , SFV CSP ) are inherently susceptible to even minor changes in the geometry of the modeled structures. Thus, in line with the discussions by Raybould et al. [69], the overal trends of these metrics can be used to assess the generated samples as similar or dissimilar to the reference distributions, but one should not fall into the trap of overly focusing on the specific values attained. From this perspective, most structure-based models do sufficiently well on these metrics, perhaps with the exception of EGNN ∆ G .

In Table 2 we show that APMixer is able to use an order of magnitude more parameters with a smaller memory footprint during training and offers more efficient sample generation, compared to the baseline architectures, on Nvidia A100 80GB GPU.

5.2 Generating HER2 Binders

Antibody generative models can be used to explore a subset of the general antibody space, such as the binders of the target antigen. The purpose of modeling and expanding a set of binders is twofold: a) it allows us to rigorously validate our generative models in a setup more tractable than denovo design; b) from a practical standpoint, it sets the ground for the optimization of properties that render binding antibodies drugs, such as developability, immunogenicity, and expression, allowing efficient exploration of the binder space [21; 63]. Note that sufficiently large libraries consisting of antibodies of variable binding affinity are usually discovered during the very early stages of the drug design process by means of high-throughput experiments [64; 76; 55]. Thus, the data used here can be sufficiently easier to obtain than the crystal structures usually assumed in CDR redesign experiments.

We use the Trastuzumab CDR H3 mutant dataset by Mason et al. [55] which was constructed by mutating 10 positions in the CDR H3 of the cancer drug Trastuzumab. The mutants were then evaluated using a high-throughput noisy assay to determine if they bind the HER2 antigen. After discarding all duplicate sequences and sequences that appear in both the binder set and the non-binder set, we are left with 9k binders and 25k non-binders. The generative models were trained only on the binder set. Separately, we train a classifier based on the APMixer architecture to distinguish binders from non-binders. The classifier achieves 87 . 8% accuracy on a randomly selected test set of 3000 antibodies, implying that the predicted binding probability is a somewhat informative metric for how tightly the binder distribution is approximated by the models.

The computational results summarized in Table 3 evidence that AbDiffuser closely approximates the binder distribution. Although most models achieve results close to those achieved by the validation set 1 , the sequence-based baseline struggles to capture the features that convey binding; evidenced by the low predicted binding probability. GNN-based baselines struggle to predict the correct structure

(RMSD above 3) producing structures with unreasonably high Amber energy. As a result, the binding probability of their designs is lower than that of the more powerful structural models.

Next, we look at the two baselines that only redesign CDR H3 instead of generating the whole antibody. RefineGNN [34] manages to closely capture the sequence and biophysical characteristics and generates sufficiently good binders. DiffAb [53], which is another diffusion-based model, achieves the best binding probability out of the CDR redesign baselines. MEAN [43] generated few unique sequences (38.9% uniqueness) and failed some of the distribution-based metrics (Naturalness, CDR PSH). dyMEAN [44] collapsed to always generating a single sample, thus we did not report its results. The overfitting behavior of MEAN and dyMEAN can be attributed to the reconstruction-based training objective and the fact that they were designed for a slightly different task of CDR redesign with different antigens, instead of the single antigen we have here.

Due to the use of iterative refinement and reconstruction-based training objectives, RefineGNN and MEAN focus on the generation of the most likely examples. Focusing on high-likelihood modes is especially beneficial here, as the experiment that was used to create the dataset is noisy. Diffusion models can also be adjusted to focus on prevalent modes by reducing the temperature τ of the denoising process. In the Gaussian case, we specified the temperature as the scaling factor of the noise added during the reverse process (Appendix B.2), whereas in the discrete case we specified it as the temperature of the model output softmax function (Appendix B.3). We observe that a slight reduction of the temperature helps to improve the general distribution fit across almost all metrics. Reducing the temperature further boosts the binding probability, but, as expected, can result in a slight loss of diversity. Using a higher temperature slightly increases the Stability Wasserstein distance while improving Stability. The phenomenon occurs because the model is no longer concerned with fitting low-likelihood modes of the real distribution that contain structures of poor stability.

In contrast to MEAN, RefineGNN and DiffAb, which only redesign the CDR H3 of the test-set structures, AbDiffuser generates full antibodies and still achieves better sequence similarity. MEAN [43] also produced structures of noticeably worse RMSD, which can be explained as follows: a) as we see in Appendix K, MEAN does not predict CDR H3 positions as well; b) changing CDR H3 can impact the most-likely conformation of the overall antibody, something that the CDR inpainting models cannot account for. We do not report the RMSD and ∆ G for RefineGNN as it does not place the generated CDR H3 loop in the reference frame of the original antibody.

The AbDiffuser model that also generates side chains generally achieved better sequence similarity (Naturalness, Closeness) and better binding probability than the backbone-only model, but a worse similarity of sequence stability. Furthermore, while the side chain model achieved a worse overall structure quality ( ∆ G , RMSD), as we see in Appendix K it predicted CDR H3 positions more precisely, which is the main desideratum in this experiment.

5.3 In Vitro Validation

We further validate our designs through an in vitro experiment. As shown in Figure 2, all submitted designs were expressed and purified successfully (average concentration of 1.25 mg/ml) and an average of 37.5% of the designs were confirmed binders with pKD ∈ [8 . 32 , 9 . 50] (higher is better) whose average was 8.70. The binding rate was improved (from 22. 2% to 57. 1%) when considering designs that were additionally filtered so that they were better than the bottom 25th quantile in every metric (naturalness, RMSD, etc.) and a classifier trained to distinguish binders from non-binders predicted that they bound with high confidence. This increase in binding for filtered samples suggests that our selected metrics indeed correlate with desirable in vitro characteristics. Our best binder belonged to the latter set and its pKD of 9.50 was slightly above Trastuzumab (not in the training set) while differing in 4 positions of the CDR H3 loop. Further examination of the top binders is performed in Figure 3 and Appendix L.

In contrast to the only other comparable study for ML-based Trastuzumab CDR H3 mutant design by Shanehsazzadeh et al. [76], our best binder had pKD = -log 10 (3 . 17 -10 ) = 9 . 50 , while the best binder found by Shanehsazzadeh et al. had pKD = 9 . 03 . Two important differences between the two studies are that: we trained on binders and tested 16 samples in the wet lab, while Shanehsazzadeh et al. used a simpler sequence infiling model trained on a large set of generic antibodies to generate 440k candidate sequences that were filtered in the wet-lab using high-throughput screening to identify 4k binders, of which 421 were selected to be tested using precise SPR measurements. The fact that our

raw

filtered

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

purification concentration (mg/ml)

raw

filtered

8.4

8.6

8.8

9.0

9.2

9.4

binding affinity (pKD)

raw

filtered

0

10

20

30

40

50

60

HER2 binders (%)

22.2%

57.1%

Design H3

SRWLASGFYTFAY

4.72e-09

SRWSGDGFYQFDY

4.12e-09

SRWRGSGFYEFDY

2.89e-09

SRWRASGFYAYDY

2.00e-09

SRYGGFGFYQFDY

1.86e-09

SRYGGSGFYTFDY

3.17e-10

SRWGGDGFYAMDY

6.12e-10

HEAVY

LIGHT

)

Binding interfaces

D

ANTIGEN

: erbB-2 (HER2)

approach required 26x fewer precise SPR wet-lab experiments to find a better binder hints toward a major improvement in efficiency. This highlights the importance of new and more efficient generative models for antibodies and more efficient, and thus powerful architectures such as APMixer.

The work of Shanehsazzadeh et al. [76] also shows that high-throughput wet-lab experiments can be used to build sufficiently large datasets of target-specific binders without starting from a known binder. This provides a straightforward way to train AbDiffuser on new targets.

After good binders are determined through high-throughput and SPR experiments, at the end of the drug discovery process, an experimental 3D structure of the bound complex is often produced. In recent works, redesigning CDRs in such co-crystal structures has been a popular task [53; 43; 34]. To that end, Appendix N investigates how AbDiffuser can be adopted for it and shows that it offers much better sequence recovery rates than the current state-of-the-art diffusion model for the CDR redesign [53].

6 Conclusions

We propose a denoising model that deviates from conventional practice in deep learning for protein design. APMixer enjoys the benefits of the best protein models (SE(3) equivariance, universality), while being significantly more memory efficient. We also show how to tailor the diffusion process to antibody generation by incorporating priors that enable the handling of variable-length sequences and allow the model to navigate strong bond constraints while operating in an extrinsic coordinate space that supports Gaussian noise. In our future work, we want to apply our method to protein families beyond antibodies, as organized in CATH [81] and Pfam [57]. Promising examples include the TIM-barrel fold and enzyme families, such as SAM-dependent methyltransferases, long-chain alcohol oxidases, and amine dehydrogenases [83].

References

R. Akbar, P. A. Robert, M. Pavlovi´ c, J. R. Jeliazkov, I. Snapkov, A. Slabodkin, C. R. Weber, L. Scheffer, E. Miho, I. H. Haff, et al. A compact vocabulary of paratope-epitope interactions enables predictability of antibody-antigen binding. Cell Reports , 34(11):108856, 2021.

R. F. Alford, A. Leaver-Fay, J. R. Jeliazkov, M. J. O'Meara, F. P. DiMaio, H. Park, M. V. Shapovalov, P. D. Renfrew, V. K. Mulligan, K. Kappel, et al. The rosetta all-atom energy function for macromolecular modeling and design. Journal of chemical theory and computation , 13(6):3031-3048, 2017.

N. Anand, R. Eguchi, I. I. Mathews, C. P. Perez, A. Derry, R. B. Altman, and P.-S. Huang. Protein sequence design with a learned potential. Nature communications , 13(1):1-11, 2022.

I. Anishchenko, S. J. Pellock, T. M. Chidyausiku, T. A. Ramelot, S. Ovchinnikov, J. Hao, K. Bafna, C. Norn, A. Kang, A. K. Bera, et al. De novo protein design by deep network hallucination. Nature , 600(7889):547-552, 2021.

J. Austin, D. D. Johnson, J. Ho, D. Tarlow, and R. van den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems , 34: 17981-17993, 2021.

S. Bachas, G. Rakocevic, D. Spencer, A. V. Sastry, R. Haile, J. M. Sutton, G. Kasun, A. Stachyra, J. M. Gutierrez, E. Yassine, et al. Antibody optimization enabled by artificial intelligence predictions of binding affinity and naturalness. bioRxiv , 2022.

S. Bond-Taylor, P. Hessey, H. Sasaki, T. P. Breckon, and C. G. Willcocks. Unleashing transformers: Parallel token prediction with discrete absorbing diffusion for fast high-resolution image generation from vector-quantized codes. In European Conference on Computer Vision (ECCV) , 2022.

V. D. Bortoli, E. Mathieu, M. J. Hutchinson, J. Thornton, Y. W. Teh, and A. Doucet. Riemannian score-based generative modelling. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022.

B. Buchfink, K. Reuter, and H.-G. Drost. Sensitive protein alignments at tree-of-life scale using diamond. Nature methods , 18(4):366-368, 2021.

A. V. Collis, A. P. Brouwer, and A. C. Martin. Analysis of the antigen combining site: correlations between length and sequence composition of the hypervariable loops and the nature of the antigen. Journal of molecular biology , 325(2):337-354, 2003.

J. Dauparas, I. Anishchenko, N. Bennett, H. Bai, R. Ragotte, L. Milles, B. Wicky, A. Courbet, R. de Haas, N. Bethel, et al. Robust deep learning based protein sequence design using proteinmpnn. bioRxiv , 2022.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

J. Dunbar, K. Krawczyk, J. Leem, T. Baker, A. Fuchs, G. Georges, J. Shi, and C. M. Deane. Sabdab: the structural antibody database. Nucleic acids research , 42(D1):D1140-D1146, 2014.

R. L. Dunbrack Jr. Rotamer libraries in the 21st century. Current opinion in structural biology , 12(4):431-440, 2002.

R. R. Eguchi, C. A. Choe, and P.-S. Huang. Ig-vae: Generative modeling of protein structure by direct 3d coordinate generation. Biorxiv , pages 2020-08, 2022.

A. Elnaggar, M. Heinzinger, C. Dallago, G. Rihawi, Y. Wang, L. Jones, T. Gibbs, T. Feher, C. Angerer, M. Steinegger, et al. Prottrans: towards cracking the language of life's code through self-supervised deep learning and high performance computing. arXiv preprint arXiv:2007.06225 , 2020.

R. A. Engh and R. Huber. Structure quality and target parameters. International Tables for Crystallography , 2006.

N. Ferruz, M. Heinzinger, M. Akdel, A. Goncearenco, L. Naef, and C. Dallago. From sequence to function through structure: deep learning for protein design. Computational and Structural Biotechnology Journal , 2022.

N. Ferruz, S. Schmidt, and B. Höcker. Protgpt2 is a deep unsupervised language model for protein design. Nature communications , 13(1):1-10, 2022.

K. Gao, L. Wu, J. Zhu, T. Peng, Y. Xia, L. He, S. Xie, T. Qin, H. Liu, K. He, et al. Incorporating pre-training paradigm for antibody sequence-structure co-design. bioRxiv , pages 2022-11, 2022.

V. Gligorijevi´ c, D. Berenberg, S. Ra, A. Watkins, S. Kelow, K. Cho, and R. Bonneau. Functionguided protein design by deep manifold sampling. bioRxiv , 2021.

K. K. Haefeli, K. Martinkus, N. Perraudin, and R. Wattenhofer. Diffusion models for graphs benefit from discrete state spaces. In The First Learning on Graphs Conference , 2022.

B. Hanin. Universal function approximation by deep neural nets with bounded width and relu activations. Mathematics , 7(10):992, 2019.

Z. Harteveld, J. Southern, M. Defferrard, A. Loukas, P. Vandergheynst, M. Bronstein, and B. Correia. Deep sharpening of topological features for de novo protein design. In ICLR2022 Machine Learning for Drug Discovery , 2022.

D. Hendrycks and K. Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415 , 2016.

J. Ho, A. Jain, and P. Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

A. Honegger and A. Plückthun. Yet another numbering scheme for immunoglobulin variable domains: an automatic modeling and analysis tool. Journal of molecular biology , 309(3): 657-670, 2001.

E. Hoogeboom, V. G. Satorras, C. Vignac, and M. Welling. Equivariant diffusion for molecule generation in 3d. In International Conference on Machine Learning , pages 8867-8887. PMLR, 2022.

Y.-C. Hsiao, Y.-J. J. Chen, L. D. Goldstein, J. Wu, Z. Lin, K. Schneider, S. Chaudhuri, A. Antony, K. Bajaj Pahuja, Z. Modrusan, et al. Restricted epitope specificity determined by variable region germline segment pairing in rodent antibody repertoires. In MAbs , volume 12, page 1722541. Taylor & Francis, 2020.

P.-S. Huang, J. J. Love, and S. L. Mayo. A de novo designed protein-protein interface. Protein Science , 16(12):2770-2774, 2007.

W. Huang, J. Han, Y. Rong, T. Xu, F. Sun, and J. Huang. Equivariant graph mechanics networks with constraints. In International Conference on Learning Representations , 2022.

J. Ingraham, M. Baranov, Z. Costello, V. Frappier, A. Ismail, S. Tie, W. Wang, V. Xue, F. Obermeyer, A. Beam, et al. Illuminating protein space with a programmable generative model. bioRxiv , 2022.

W. Jin, R. Barzilay, and T. Jaakkola. Antibody-antigen docking and design via hierarchical structure refinement. In International Conference on Machine Learning , pages 10217-10227. PMLR, 2022.

W. Jin, J. Wohlwend, R. Barzilay, and T. S. Jaakkola. Iterative refinement graph neural network for antibody sequence-structure co-design. In International Conference on Learning Representations , 2022.

W. Jin, S. Sarkizova, X. Chen, N. Hacohen, and C. Uhler. Unsupervised protein-ligand binding energy prediction via neural euler's rotation equation. arXiv preprint arXiv:2301.10814 , 2023.

B. Jing, E. Erives, P. Pao-Huang, G. Corso, B. Berger, and T. Jaakkola. Eigenfold: Generative protein structure prediction with diffusion models. arXiv preprint arXiv:2304.02198 , 2023.

J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko, et al. Highly accurate protein structure prediction with alphafold. Nature , 596(7873):583-589, 2021.

W. Kabsch. A discussion of the solution for the best rotation to relate two sets of vectors. Acta Crystallographica Section A: Crystal Physics, Diffraction, Theoretical and General Crystallography , 34(5):827-828, 1978.

V. Kalofolias. How to learn a graph from smooth signals. In Artificial Intelligence and Statistics , pages 920-929. PMLR, 2016.

M. A. Ketata, C. Laue, R. Mammadov, H. Stärk, M. Wu, G. Corso, C. Marquet, R. Barzilay, and T. S. Jaakkola. Diffdock-pp: Rigid protein-protein docking with diffusion models. arXiv preprint arXiv:2304.03889 , 2023.

D. Kingma, T. Salimans, B. Poole, and J. Ho. Variational diffusion models. Advances in neural information processing systems , 34:21696-21707, 2021.

D. Koller and N. Friedman. Probabilistic graphical models: principles and techniques . MIT press, 2009.

X. Kong, W. Huang, and Y. Liu. Conditional antibody design as 3d equivariant graph translation. In The Eleventh International Conference on Learning Representations , 2023.

X. Kong, W. Huang, and Y. Liu. End-to-end full-atom antibody design. arXiv preprint arXiv:2302.00203 , 2023.

B. Kuhlman, G. Dantas, G. C. Ireton, G. Varani, B. L. Stoddard, and D. Baker. Design of a novel globular protein fold with atomic-level accuracy. science , 302(5649):1364-1368, 2003.

J. H. Lee, P. Yadollahpour, A. Watkins, N. C. Frey, A. Leaver-Fay, S. Ra, K. Cho, V. Gligorijevic, A. Regev, and R. Bonneau. Equifold: Protein structure prediction with a novel coarse-grained structure representation. bioRxiv , 2022.

J. S. Lee, J. Kim, and P. M. Kim. Score-based generative modeling for de novo protein design. Nature Computational Science , pages 1-11, 2023.

J. K. Leman, B. D. Weitzner, S. M. Lewis, J. Adolf-Bryfogle, N. Alam, R. F. Alford, M. Aprahamian, D. Baker, K. A. Barlow, P. Barth, et al. Macromolecular modeling and design in rosetta: recent methods and frameworks. Nature methods , 17(7):665-680, 2020.

R. Liao, Y. Li, Y. Song, S. Wang, W. Hamilton, D. K. Duvenaud, R. Urtasun, and R. Zemel. Efficient graph generation with graph recurrent attention networks. In Advances in Neural Information Processing Systems , pages 4255-4265, 2019.

Y. Lin and M. AlQuraishi. Generating novel, designable, and diverse protein structures by equivariantly diffusing oriented residue clouds. arXiv preprint arXiv:2301.12485 , 2023.

S. L. Lisanza, J. M. Gershon, S. W. K. Tipps, L. Arnoldt, S. Hendel, J. N. Sims, X. Li, and D. Baker. Joint generation of protein sequence and structure with rosettafold sequence space diffusion. bioRxiv , pages 2023-05, 2023.

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations , 2019.

S. Luo, Y. Su, X. Peng, S. Wang, J. Peng, and J. Ma. Antigen-specific antibody design and optimization with diffusion-based generative models for protein structures. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, Advances in Neural Information Processing Systems , 2022.

K. Martinkus, A. Loukas, N. Perraudin, and R. Wattenhofer. Spectre: Spectral conditioning helps to overcome the expressivity limits of one-shot graph generators. In Proceedings of the International Conference on Machine Learning (ICML) , 2022.

D. M. Mason, S. Friedensohn, C. R. Weber, C. Jordi, B. Wagner, S. M. Meng, R. A. Ehling, L. Bonati, J. Dahinden, P. Gainza, et al. Optimization of therapeutic antibodies by predicting antigen specificity from antibody sequence via deep learning. Nature Biomedical Engineering , 5(6):600-612, 2021.

I. Melnyk, V. Chenthamarakshan, P.-Y. Chen, P. Das, A. Dhurandhar, I. Padhi, and D. Das. Reprogramming large pretrained language models for antibody sequence infilling. arXiv e-prints , pages arXiv-2210, 2022.

J. Mistry, S. Chuguransky, L. Williams, M. Qureshi, G. A. Salazar, E. L. Sonnhammer, S. C. Tosatto, L. Paladin, S. Raj, L. J. Richardson, et al. Pfam: The protein families database in 2021. Nucleic acids research , 49(D1):D412-D419, 2021.

A. Mullard. 2022 fda approvals. Nature Reviews Drug Discovery , 2023.

A. Q. Nichol and P. Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning , pages 8162-8171. PMLR, 2021.

C. Norn, B. I. Wicky, D. Juergens, S. Liu, D. Kim, D. Tischer, B. Koepnick, I. Anishchenko, F. Players, D. Baker, et al. Protein sequence design by conformational landscape optimization. Proceedings of the National Academy of Sciences , 118(11):e2017228118, 2021.

T. H. Olsen, F. Boyles, and C. M. Deane. Observed antibody space: A diverse database of cleaned, annotated, and translated unpaired and paired antibody sequences. Protein Science , 31 (1):141-146, 2022.

T. H. Olsen, I. H. Moal, and C. M. Deane. AbLang: an antibody language model for completing antibody sequences. Bioinformatics Advances , 2(1), 06 2022.

J. W. Park, S. Stanton, S. Saremi, A. Watkins, H. Dwyer, V. Gligorijevic, R. Bonneau, S. Ra, and K. Cho. Propertydag: Multi-objective bayesian optimization of partially ordered, mixed-variable properties for biological sequence design. arXiv preprint arXiv:2210.04096 , 2022.

J. Parkinson, R. Hard, and W. Wang. The resp ai model accelerates the identification of tight-binding antibodies. Nature Communications , 14(1):454, 2023.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019.

N. F. Polizzi, Y. Wu, T. Lemmin, A. M. Maxwell, S.-Q. Zhang, J. Rawson, D. N. Beratan, M. J. Therien, and W. F. DeGrado. De novo design of a hyperstable non-natural protein-ligand complex with sub-å accuracy. Nature chemistry , 9(12):1157-1164, 2017.

J. W. Ponder and F. M. Richards. Tertiary templates for proteins: use of packing criteria in the enumeration of allowed sequences for different structural classes. Journal of molecular biology , 193(4):775-791, 1987.

O. Puny, M. Atzmon, E. J. Smith, I. Misra, A. Grover, H. Ben-Hamu, and Y. Lipman. Frame averaging for invariant and equivariant network design. In International Conference on Learning Representations , 2022.

M. I. Raybould, C. Marks, K. Krawczyk, B. Taddese, J. Nowak, A. P. Lewis, A. Bujotzek, J. Shi, and C. M. Deane. Five computational developability guidelines for therapeutic antibody profiling. Proceedings of the National Academy of Sciences , 116(10):4025-4030, 2019.

P. B. Reis, G. P. Barletta, L. Gagliardi, S. Fortuna, M. A. Soler, and W. Rocchia. Antibodyantigen binding interface analysis in the big data era. Frontiers in Molecular Biosciences , 9, 2022.

A. Rives, J. Meier, T. Sercu, S. Goyal, Z. Lin, J. Liu, D. Guo, M. Ott, C. L. Zitnick, J. Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences , 118(15):e2016239118, 2021.

C. Roos, T. Terlaky, and J.-P. Vial. Interior point methods for linear optimization. 2005.

J. A. Ruffolo and J. J. Gray. Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodies. Biophysical Journal , 121(3):155a-156a, 2022.

J. A. Ruffolo, J. J. Gray, and J. Sulam. Deciphering antibody affinity maturation with language models and weakly supervised learning. arXiv preprint arXiv:2112.07782 , 2021.

V. G. Satorras, E. Hoogeboom, and M. Welling. E (n) equivariant graph neural networks. In International conference on machine learning , pages 9323-9332. PMLR, 2021.

A. Shanehsazzadeh, S. Bachas, G. Kasun, J. M. Sutton, A. K. Steiger, R. Shuai, C. Kohnert, A. Morehead, A. Brown, C. Chung, et al. Unlocking de novo antibody design with generative artificial intelligence. bioRxiv , pages 2023-01, 2023. URL https://www.biorxiv.org/ content/10.1101/2023.01.08.523187v2 .

W. Sheffler and D. Baker. Rosettaholes: rapid assessment of protein core packing for structure prediction, refinement, design, and validation. Protein Science , 18(1):229-239, 2009.

C. Shi, C. Wang, J. Lu, B. Zhong, and J. Tang. Protein sequence and structure co-design with equivariant translation. arXiv preprint arXiv:2210.08761 , 2022.

E. L. Shrock, R. T. Timms, T. Kula, E. L. Mena, A. P. West Jr, R. Guo, I.-H. Lee, A. A. Cohen, L. G. McKay, C. Bi, et al. Germline-encoded amino acid-binding motifs drive immunodominant public antibody responses. Science , 380(6640):eadc9498, 2023.

R. W. Shuai, J. A. Ruffolo, and J. J. Gray. Generative language modeling for antibody design. bioRxiv , pages 2021-12, 2021.

I. Sillitoe, N. Bordin, N. Dawson, V. P. Waman, P. Ashford, H. M. Scholes, C. S. Pang, L. Woodridge, C. Rauer, N. Sen, et al. Cath: increased structural coverage of functional space. Nucleic acids research , 49(D1):D266-D273, 2021.

J. M. Singer, S. Novotney, D. Strickland, H. K. Haddox, N. Leiby, G. J. Rocklin, C. M. Chow, A. Roy, A. K. Bera, F. C. Motta, et al. Large-scale design and refinement of stable proteins using sequence-only models. PloS one , 17(3):e0265020, 2022.

F. L. Sirota, S. Maurer-Stroh, Z. Li, F. Eisenhaber, and B. Eisenhaber. Functional classification of super-large families of enzymes based on substrate binding pocket residues for biocatalysis and enzyme engineering applications. Frontiers in bioengineering and biotechnology , page 674, 2021.

J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015.

S. Stanton, W. Maddox, N. Gruver, P. Maffettone, E. Delaney, P. Greenside, and A. G. Wilson. Accelerating bayesian optimization for biological sequence design with denoising autoencoders. In International Conference on Machine Learning , pages 20459-20478. PMLR, 2022.

I. O. Tolstikhin, N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Yung, A. Steiner, D. Keysers, J. Uszkoreit, et al. Mlp-mixer: An all-mlp architecture for vision. Advances in Neural Information Processing Systems , 34:24261-24272, 2021.

B. L. Trippe, J. Yim, D. Tischer, T. Broderick, D. Baker, R. Barzilay, and T. Jaakkola. Diffusion probabilistic modeling of protein backbones in 3d for the motif-scaffolding problem. arXiv preprint arXiv:2206.04119 , 2022.

C. Vignac, I. Krawczuk, A. Siraudin, B. Wang, V. Cevher, and P. Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv:2209.14734 , 2022.

R. Vinod, K. K. Yang, and L. Crawford. Joint protein sequence-structure co-design via equivariant diffusion. In NeurIPS 2022 Workshop on Learning Meaningful Representations of Life , 2022.

J. L. Watson, D. Juergens, N. R. Bennett, B. L. Trippe, J. Yim, H. E. Eisenach, W. Ahern, A. J. Borst, R. J. Ragotte, L. F. Milles, et al. Broadly applicable and accurate protein design by integrating structure prediction networks and diffusion generative models. bioRxiv , 2022.

D. N. Woolfson. A brief history of de novo protein design: minimal, rational, and computational. Journal of Molecular Biology , 433(20):167160, 2021.

K. E. Wu, K. K. Yang, R. v. d. Berg, J. Y. Zou, A. X. Lu, and A. P. Amini. Protein structure generation via folding diffusion. arXiv preprint arXiv:2209.15611 , 2022.

K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations , 2019.

J. Yim, B. L. Trippe, V. De Bortoli, E. Mathieu, A. Doucet, R. Barzilay, and T. Jaakkola. Se (3) diffusion model with application to protein backbone generation. arXiv preprint arXiv:2302.02277 , 2023.

A Related Work

The literature on computational methods for protein design is vast. We focus here on machine learning methods for protein sequence and structure generation and refer the interested reader to the relevant reviews by Woolfson [91]; Ferruz et al. [18] for broader context.

A.1 Protein Language Models

The first works on protein generation focused on language modeling [16; 21; 71; 85]. Motivated by the observation that the natural forces driving antibody sequence formation, namely V(D)J recombination and the selective pressure in the presence of an antigen referred to as somatic hypermutation, are distinct from other proteins, recent works have also built antibody-specific language models. Therein, large-scale models trained on millions of sequences, such as AntiBERTy [74] and IgLM [80], have been used to generate full-length antibody sequences across different species or restore missing regions in the antibody sequence [62; 56].

A.2 Protein Structure Generation

These approaches use equivariant architectures to generate proteins and antibodies in 3D. We broadly distinguish two strategies: backbone-first and sequence-structure co-design .

The backbone-first strategy simplifies the generation problem by splitting it into two steps: the generation of the protein backbone and of the protein sequence given the backbone structure. A backbone can be designed by refining a rationally constructed template [24] or by a structure-based generative model [90; 32; 15; 87; 50; 92; 47]. On the other hand, the determination of the residues that fit the backbone can be cast within the self-supervised learning paradigm [3; 11] or by minimizing some score that is a proxy for energy [60]. Despite the promise of backbone-first strategies, in the context of deep learning, it is generally preferable to solve problems in an end-to-end fashion. Decoupling the problem into separate steps can also prohibit the modeling of complex dependencies between backbone and sequence, as early decisions are not updated in light of new evidence. As discussed by Harteveld et al. [24], one failure mode of the backbone-first strategy is obtaining a secondary structure that is not physically realizable with natural amino acids. It is also worth mentioning that structure diffusion generative models can also be used for protein-protein docking, as shown by Ketata et al. [40].

Sequence and structure co-design entails learning to generate a full protein in an end-to-end fashion [51; 89]. In this paper we also adopt this strategy. As highlighted in the introduction, AbDiffuser differs from these previous works in the neural architecture and protein representation (APMixer), as well as by shaping the diffusion process to better suit the problem domain.

A relevant line of works has also focused on the re-generation of the complementarity-determining regions (CDRs) of antibodies, a subset of the heavy and light chains containing typically between 48 and 82 residues [10] of particular relevance to binding [34; 20; 43; 33; 53; 78; 44]. These constitute meaningful first steps towards the goal of full-atom antibody design that we consider in this work. Having the ability to regenerate the full antibody sequence and structure significantly enlarges the design space while also enabling the probing of properties beyond affinity, such as stability and immunogenicity. Furthermore, even in the context of affinity optimization, it can be beneficial to also redesign the framework and vernier regions, as the latter sometimes interact with the antigen and affect the CDR conformation [1; 70; 79].

B Antibody Diffusion

This section complements the discussion in Section 2 by describing in greater detail how we frame denoising diffusion.

B.1 Additional Background on Denoising Diffusion

Recall that the forward process q ( X 1 , ..., X T | X 0 ) = q ( X 1 | X 0 ) Π T t =2 q ( X t | X t -1 ) , is Markovian. This Markovianity allows for the recovery of the reverse process

Thus, to generate new data, a neural network learns to approximate the true denoising process ˆ X 0 = ϕ ( X t , t ) , which can be accomplished by minimizing the variational upper bound on the negative loglikelihood [84; 26; 5]:

In practice, the model is trained to optimize each of the L t ( X 0 ) loss terms and the L 0 ( X 0 ) loss term individually, by sampling an arbitrary time step t ∈ { 1 , T } . Note that L T ( X 0 ) approaches 0 for any data distribution if sufficiently many time steps are used. The latent variable distribution q is usually defined so that one can directly sample any step in the trajectory efficiently.

As discussed in Section 2, in our diffusion process, we factorize the posterior probability distribution over the atom positions and residue types:

We discuss the constituent processes in the following two sections.

B.2 Equivariant Diffusion of Atom Positions

One of the most important features in the training of a diffusion model is the ability to efficiently sample any step in the forward process q ( X pos t | X pos 0 ) . This is easily achieved if we use a Gaussian distribution q ( X pos t | X pos 0 ) = N ( α t X pos 0 , σ 2 t Σ) , where α t and σ 2 t are positive, scalar-valued functions of t , with α 1 ≈ 1 and α T ≈ 0 . A common way to define the noising process is the variance-preserving parameterization [84; 26] with α t = √ 1 -σ 2 t . A forward process step in this general diffusion formulation can be expressed as q ( X pos t | X pos t -1 ) = N ( α t | t -1 X pos 0 , σ 2 t | t -1 Σ) , where α t | t -1 = α t /α t -1 and σ 2 t | t -1 = σ 2 t -α 2 t | t -1 σ 2 t -1 . Similarly, a reverse process step q ( X pos t -1 | X pos t , X pos 0 ) can then be expressed as the following Gaussian:

Similarly to Watson et al. [90], we can reduce the temperature τ of the Brownian motion of the backward process by scaling the variance of this Gaussian distribution by τ .

Kingma et al. [41] showed that the training objective for the Gaussian distribution can be simplified to

where the signal-to-noise ratio is SNR ( t ) = ( α t /σ t ) 2 and ∥ · ∥ 2 Σ -1 = tr ( ( · ) T Σ -1 ( · ) ) .

It is quite common in the literature to choose a slightly different optimization objective. In particular, it has been found that predicting the noise ˆ ϵ = ϕ ( X pos t , t ) that has been added X pos t = α t X pos 0 + σ t ϵ ,

ϵ ∼ N (0 , Σ) and using an unweighted loss L t ( X pos 0 ) = 1 2 ∥ ˆ ϵ -ϵ ∥ 2 Σ -1 improves model training [26; 41; 28]. A possible explanation is that this parameterization makes it easier for the model to minimize the loss at time steps close to T . We found that a similar effect can be obtained by re-weighing the mean prediction loss (Appendix C):

which empirically renders the mean ˆ X pos 0 = ϕ ( X pos t , t ) and error ˆ ϵ = ϕ ( X pos t , t ) prediction alternatives perform comparably. We rely on equation 4 throughout our experiments, since predicting ˆ X pos 0 = ϕ ( X pos t , t ) allows for easier incorporation of known constraints, such as atom bond lengths (see Section 3.4). As shown in Appendix D, L 1 ( X pos 0 ) can also be optimized using the same objective.

B.3 Discrete Residue-type Diffusion

To define the discrete diffusion process for the residue types we adopt the formulation introduced by Austin et al. [5]. If we have a discrete random variable with k categories, represented as a one-hot vector X res , we define the forward process q ( X res t | X res 0 ) = Cat ( X res 0 Q t ) , where the transition matrix Q t = β t I + (1 -β t ) Q is a linear interpolation between identity and target distribution Q (e.g., uniform distribution). Similarly to atom diffusion, one forward process step is q ( X res t | X res t -1 ) = Cat ( X res t Q t | t -1 ) , where Q t | t -1 = Q -1 t -1 Q t . From this, a single reverse process step can be expressed as:

for an appropriate normalizing constant Z . In the discrete case, we implement temperature τ scaling as the temperature of the softmax applied to the model output. This serves to sharpen the target categorical distribution ∼ p 1 /τ .

Similarly to the atom diffusion case, we use a simplified objective for the discrete diffusion [22; 7] which has been found to lead to better results:

Hyperparameter β t ensures that the loss weight is proportional to the fraction of unchanged classes and we set β t = α 2 t to keep the noise schedules for the residue-type diffusion and atom-position diffusion the same. We also note that now the L 1 ( X 0 ) loss term is simply the cross-entropy of the model prediction L 1 ( X 0 ) = log p θ ( X 0 | X 1 ) . The derivation of our loss can be found in Appendix E.

C Re-weighted Optimization Objective for Atom Positions

As shown by Kingma et al. [41], the L t term in the variational upper bound corresponds to

If we instead parameterize X pos t = α t X pos 0 + σ t ϵ we can directly re-write this as

Meanwhile, the simplified objective of Ho et al. [26] is instead defined as

If we reverse the parameterization as ϵ = X pos t -α t X pos 0 σ t and using the fact that SNR ( t ) = α 2 t /σ 2 t this becomes

D Reconstruction Loss for Atom Positions

For the Gaussian atom position diffusion (Section B.2 we need to account for a few considerations in the reconstruction term L 1 ( X 0 ) of the variational upper bound. Assuming that the data probability is constant, q ( X pos 0 | X pos 1 ) ≈ N ( X pos 1 α 1 , σ 2 1 α 2 1 Σ) . Similarly to Hoogeboom et al. [28], we parameterize the mean using the model prediction instead:

As highlighted by Hoogeboom et al. [28], this increases the quality of the results.

Isotropic distributions, such as Gaussians with appropriate covariance matrices, assign the same probability to every rotation of a vector in 3D, implying that the resulting atom diffusion process is equivariant w.r.t. orthogonal group O3. To additionally impose that the diffusion is translation-equivariant, we follow Hoogeboom et al. [28] and use a re-normalized multivariate Gaussian distribution, where the sample center of mass is fixed to zero: ∑ n i =1 X pos i = 0 . The latter has the same law as an ordinary

Gaussian with the main difference that the normalizing constant is now Z = ( √ 2 πσ ) 3( n -1) for n atoms in 3D space.

It is clear to see that the O3 property is satisfied if we have an identity covariance. However, it also holds for non-isotropic covariance matrices. Let us define Z ∼ N (0 , I ) of dimension n × 3 and X pos = KZ , where Σ -1 = KK T . The latter can be re-written as x 0 = vec ( X 0 ) = ( I ⊗ K ) vec ( Z ) = Σ 1 / 2 z . We then note that for any 3x3 unitary matrix U , we have x ⊤ 0 ( U ⊤ ⊗ I )Σ -1 ( U ⊗ I ) x 0 = x ⊤ 0 ( U ⊤ ⊗ I )( I ⊗ L )( U ⊗ I ) x 0 = x ⊤ 0 ( U ⊤ U ⊗ L ) x 0 = x ⊤ 0 ( I ⊗ L ) x 0 . In other words, since the covariance acts on the node dimension and the unitary matrix acts on the spatial dimension, they are orthogonal and do not affect each other.

E Simplified Discrete Diffusion Optimization Objective

We can obtain the simplified training objective by taking a slightly different bound on the negative log-likelihood to the one presented in Equation 3 [22]:

In this formulation, compared to equation 3, the term L t = D KL ( q ( X t -1 | X t , X 0 ) ∥ p θ ( X t -1 | X t )) is replaced by

To form the discrete loss L t ( X res 0 ) , we additionally weight each loss term L t by β t to match the loss weighting used for the atom positions (Appendix C). Note that when this objective is minimized, the original D KL term will also be at a minimum.

E.1 Non-Informative Priors Demand Higher Complexity Denoising Models

Denote by p the data distribution and by p θ the learned distribution, the latter of which is obtained by the push-forward p θ ( f θ ( Z )) = q ( Z ) of some prior measure q by function f θ . In the context of denoising diffusion, Z = X T , while f θ = ϕ ( X T , T ) is the learned reverse process.

We prove the following general result which relates the informativeness of the prior with the quality of the generative model and the complexity of the function computed:

Theorem E.1. For any f θ that is an invertible equivariant function w.r.t. a subgroup G of the general-linear group GL(d, R ) the following must hold:

where c q ( f ) := (min g ∈ G E Z ∼ q [ ∥ f ( g Z ) -Z ∥ t t ]) 1 /t quantifies the expected complexity of the learned model under q , W t ( p, p θ ) is the Wasserstein t -distance of our generative model to the data distribution, and W t ( p, q ) is the distance between prior and posterior.

The bound asserts that one cannot learn a simple (in terms of c q ( f ) ) generative model that fits the data well unless the prior has a small Wasserstein distance to the data distribution.

The complexity measure c q ( f ) is particularly intuitive in the context of denoising diffusion. For the diffusion of atom positions with a SE(3) equivariant model ϕ , we have

that corresponds to the expected amount of denoising that the model performs throughout the reverse process, discounting for rotations. By selecting an informative prior, we decrease W t ( p, q ) and therefore reduce the amount of denoising our model needs to do.

Proof. We recall that the Wasserstein t -distance between two probability measures p, p ′ with finite t -moments is

where the coupling γ is a joint measure on R d × R d whose marginals satisfy ∫ R d γ ( X,X ′ ) dx ′ = p ( X ) and ∫ R d γ ( X,X ′ ) dx = p ′ ( X ′ ) , whereas Γ( p, p ′ ) is the set of all couplings of p, p ′ .

̸

We will first lower bound W 1 ( p θ , q ) . To achieve this, let X = f ( Z ) and X ′ = f θ ( Z ′ ) and select γ ∗ ( X,X ′ ) = 0 when Z = Z ′ and γ ∗ ( X,X ′ ) = q ( Z ) otherwise, which is a coupling because it satisfies ∫ X ′ γ ∗ ( X,X ′ ) dx ′ = q ( Z ) = p ( X ) and ∫ X γ ∗ ( X,X ′ ) dx = q ( Z ′ ) = p θ ( X ′ ) . Then, exploiting that the Wasserstein distance is a metric (and thus abides by the triangle inequality), we get

Next, we upper bound W t ( p θ , q ) . Let G be any subgroup of the general linear group GL ( d, R ) for which f θ ( Z ) is equivariant and further denote by g ∈ G an element of the said group.

We fix γ # ( X,Z ) = q ( Z ) when X = f θ ( gZ ) and γ # ( X,Z ) = 0 , otherwise. To see that the latter is a valid coupling notice that f θ g is bijective as the composition of two bijective functions, implying:

with z x = ( f θ ◦ g ) -1 ( X ) . Further,

as required for γ # to be a coupling. We continue by bounding the Wasserstein metric in light of γ # :

Combining equation 5 and equation 6, we get

as claimed.

In the context of our paper, we have Z = X T ∼ N (0 , Σ) , f θ ( Z ) = ˆ X 0 = ϕ ( X T , T ) , f ( Z ) = X 0 , G is the standard Euclidean group SE(3), and we select t = 2 . Then, the aforementioned inequality becomes

where we have changed the position of g by exploiting that the ℓ 2 norm is unitary invariant and that ϕ is SE(3) equivariant. Alternatively, we can obtain the following tighter bound by stopping earlier at the derivation of equation 5:

where q T is the law of X T .

F Learning the Conditional Dependence Relations Between Atoms

We consider the zero-mean multivariate Gaussian whose inverse covariance matrix Σ -1 = L + aI corresponds to a sparse weighted Laplacian matrix L . Intuitively, L ij should be zero if the position of the point i is independent of that of j given the positions of the neighbors of i being fixed. The term a is a small constant whose role is to shift the spectrum of L and thus make the Laplacian invertible.

We follow the framework of [39] and optimize L by minimizing the following objective:

where the m × 3 matrix X contains the 3D positions of the m points modeled and f ( L ) are some optional constraints we can enforce on the graph Laplacian (and thus also to the adjacency matrix A ).

The log-likelihood of vectors X on the graph defined by L can be directly expressed as a function of the graph adjacency matrix and the pairwise distance matrix Z :

Kalofolias [39] propose the following optimization objective, to recover the optimal adjacency, which with a logarithmic term also ensures that every node has at least one neighbor, which is something we also desire:

An efficient primal-dual algorithm exists for solving this optimization problem [39]. The adjacency matrix that we recover using the mean distance matrix over all of the proteins in paired OAS can be seen in Figure 4. Our use of AHo numbering allows us to directly estimate such mean distances, as with our representation every antibody effectively possesses the same number of residues.

To increase the stability of the Cholesky decomposition L = KK T , which is needed for sampling co-variate noise X pos t = α t X pos 0 + σ t Kϵ , ϵ ∼ N (0 , 1) , we instead use the generalized Laplacian L = diag ( A 1 ) -A + I for the diffusion process.

G Proof of SE(3) universality

In the following, we prove the SE(3) universality of an MLP-mixer backbone model combined with frame-averaging.

Concretely, let X be an n × m matrix with bounded entries and consider the backbone model: ϕ ( X ) = c L ◦ r L -1 ◦ · · · ◦ c 3 ◦ r 2 ◦ c 1 ( X ) , where the subscript denotes layer, c l is an MLP operating independently on each column of its input matrix, and similarly, r l is an MLP operating independently on each row of its input.

It is a consequence of Theorem 4 in [68] that if F is a bounded G -equivariant frame and if ϕ is a backbone model that is universal over any compact set, then combining ϕ with frame-averaging over a frame-finite domain (i.e., a domain within which the frame cardinality is finite) yields a G -equivariant universal model ⟨ ϕ ⟩ F . Since the SE(3) frame is bounded and thus the domain is frame-finite (see also Proposition 1 [68]), to show that ⟨ ϕ ⟩ F is a universal approximator of any continuous SE(3) function, it suffices to show that the MLP-mixer is a universal approximator over continuous functions over any compact set (or simply universal).

To proceed, we will show that there exists a parameterization so that ϕ ( X ) = c L ([ z ; vec ( X )]) for some vector z . We can then rely on the universal approximation property [23] of MLPs to conclude that ϕ is also universal.

We first provide an exact constructive argument on how this can be achieved:

Let V be a ( n +1) × ( n +1) unitary matrix whose first column is the constant vector 1 and set U = V : , 1: to be the n +1 × n submatrix. The first MLP embeds each column vector x isometrically into n +1 as follows: c 1 ( x ) = Ux . By construction, every column of the output X ′ is now orthogonal to 1 .

The network appends U to the left of X ′ and ( × n -1 ︷ ︸︸ ︷ UX ; · · · ; UX ) to the right. Since appending to the right involves replicating each row multiple times, it can be done by a linear row operation. Appending to the left is slightly more involved and can be achieved by iteratively building U column by column, with each column added by using a row MLP to add one dimension with a constant value and a column MLP to project the new constant vector introduced to the appropriate column of U . The key here is that, since we have guaranteed that the columns are always orthogonal to the constant vector, the column MLP can always distinguish the newly introduced column from the ones already at play. Let X ′′ be the result of this process.

A column MLP projects each column of the ( n +1) × ( n + nm ) matrix to n dimensions by multiplying by U ⊤ . The output of this layer is X ′′′ = [ I ; X ; · · · ; X ] .

A row MLP zeros out all elements except those indexed by the one-hot encoding in the first n entries. The latter can be done by a ReLU MLP with one hidden layer: if all entries of our input are smaller than b and the input is ( onehot ( i ) , x ) , the first linear layer adds b to only the entries [ n + in : n + in + n ) and shifts every element by -b using its bias. The latter lets the correct elements pass the ReLU with an unchanged value while setting all other elements to zero, as required. Let the output of this layer be X ′′′′ with

A column MLP sums up all entries. The output is now a row vector ( 1 ; X ⊤ 1 , : ; · · · ; X ⊤ n, : ) ⊤ of length n + nm , as required.

Steps 1 and 2 of the above constructive argument are inefficient because they rely on very simple layers. This can be avoided if we utilize more powerful MLPs (i.e., functions c l and r l ) and an existence argument. Specifically, as there exists a continuous function that achieves step 2, by the universality of MLPs there also exists an MLP that approximates the step arbitrarily well. Similarly, rather than increasing the dimension to n + 1 at step 1, one may argue that the MLP can assign some vector outside of the bounded domain where the input data resides, and thus the dimension can remain n . Therefore, in this setting, only a constant number of MLPMixer blocks are needed to obtain the desired output.

This concludes the proof.

H Noise Schedule

In our denoising model, we use the cosine-like schedule proposed by Hoogeboom et al. [28], with a 1000 denoising steps:

where a precision s = 1 e -4 is used to avoid numerical instability. As σ t = √ 1 -α 2 t and β t = α 2 t , it is sufficient to define just α t . Following Nichol and Dhariwal [59] we clip α t | t -1 to be at least 0 . 001 , to avoid instability during sampling and then recompute α t as a cumulative product. To further ensure numerical stability, as recommended by Kingma et al. [41] we compute the negative log SNR curve γ ( t ) = -log α 2 t + log σ 2 t . This parameterization allows for numerically stable computation of all important values [41], for example, α 2 t = sigmoid ( -γ ( t )) , σ 2 t = sigmoid ( γ ( t )) , σ 2 t | t -1 = -expm1 ( softplus ( γ ( t -1)) -softplus ( γ ( t ))) = -expm1 ( γ ( t -1) -γ ( t )) σ 2 t , and SNR = exp( -γ ( t )) . Where expm1 ( x ) = exp( x ) -1 and softplus ( x ) = log(1 + exp( x )) .

I Full Training and Sampling Algorithms

For convenience, in Algorithms 1 and 2 we describe the full training and sampling procedures.

Algorithm 1 Training

14: Take a gradient step on : ∇ θ ( L t ( ˆ X pos 0 ) + L t ( ˆ X seq 0 )

15: until converged

Algorithm 2 Sampling

1: Initialize : 2: X pos T ∼ N (0 , Σ) 3: X seq T ∼ Categorical ( Q ) 4: Loop over the time steps : 5: for t = T, ..., 1 do 6: Predict : 7: ˆ X pos 0 , ˆ X seq 0 = f θ ( Project ( X pos t ) , X seq t , t ) 8: ˆ X pos 0 = Project ( ˆ X pos 0 ) 9: Sample next state : 10: if t > 1 then 11: X pos t -1 ∼ N ( α t | t -1 σ 2 t -1 X pos t + α t -1 σ 2 t | t -1 ˆ X pos 0 σ 2 t , σ t | t -1 σ t -1 σ t Σ ) 12: X seq t -1 ∼ Categorical ( X res t Q T t | t -1 ⊙ ˆ X res 0 Q t -1 Z ) 13: else 14: X pos 0 ∼ N ( ˆ X pos 0 , σ 2 1 α 2 1 Σ) 15: X seq 0 ∼ Categorical ( ˆ X seq 0 ) 16: return X pos 0 , X seq 0 .

J Metrics Used in the Numerical Experiments

The metrics applied to generated sequences include methods that measure naturalness, similarity to the closest extant antibody, and stability. To examine Naturalness we use the inverse perplexity of the AntiBERTy [74] model trained on a large corpus of antibody chain sequences. As shown by Bachas et al. [6] this score correlates with antibody developability and immunogenicity. To explore Closeness we use fast sequence alignment [9] to determine the closest sequence in the training set. The mean sequence identity (fractional edit distance) to the closest training sequence is then reported as a score. Here Stability is operationalized by estimating the error of the folded structure using IgFold [73]. This error can be used to rank the sequences by how stable we expect their folds to

be. We take the 90th percentile of the residue error as an estimate of the sequence fold stability; the latter typically corresponds to the CDR H3 and L3 loops, which have the most influence over the antibody's functional properties. To further verify that the generated antibody sequences satisfy appropriate biophysical properties, we rely on four additional structure-based metrics Raybould et al. [69]: CDR region hydrophobicity ( CDR PSH ), patches of positive (CDR PPC), and negative charge ( CDR PNC ), and symmetry of electrostatic charges of heavy and light chains ( SFV CSP ). These metrics operate on folded sequences (using IgFold) and take into account distance-aggregated structural properties of CDR regions (and their spatial vicinity), and their significant deviation from reference values is typically associated with bad developability properties of antibodies such as poor expression, aggregation, or non-specific binding [69].

To evaluate the overall similarity of the generated and the test-set distributions, for all of these sequence metrics, we report the Wasserstein distance between the scores of the sequences in the test split and the scores of the generated sequences.

The metrics applied to the generated structures focus primarily on scores known to correlate with free energy and RMSD. a) We use free energy ∆ G estimated using Rosetta [2] to evaluate the stability of the generated structure. Although lower energy is normally associated with higher stability of the protein structure, one needs to be careful not to reward disproportionally small energies achieved when a miss-formed protein collapses into a morph-less aggregate. Thus, we again report the Wasserstein distance between the generated and test energy distributions. b) RMSD. We also re-fold the generated sequence with IgFold [73] and report the RMSD for the backbone N,C α , C, C β and O atom positions. RMSD is reported as a mean over the generated structures as it captures how well each generated structure matches its sequence.

K Detailed Generated Structure RMSD Evaluation

Here, in Tables 4 and 5 we provide per-region distances between folded and optimized structures of the generated sequences and the generated structures. As you can see these distances correlate well with the overall RMSD reported in the main text. As the folded structures are optimized, we also investigate how the error changes if similarly to folding models we also use Rosetta [2] to optimize the generated structures in Tables 6 and 7. As can be seen, the optimization slightly reduces the RMSDs, but the relatively modest change for AbDiffuser-generated structures hints at the fact that

the generated structures were relatively physically plausible. In Table 5 we see that while the model that generates the side chains achieves a worse overall RMSD, in most cases it models CDR H3 more precisely, which is the most important part function-wise. This seeming focus on CDR H3 might explain why this model achieved a better predicted binding probability, even while modeling the overall structure slightly worse.

L Wet-Lab Validation of Designs

In our wet-lab experiments, we follow the procedure of Hsiao et al. [29]. As we focus on Trastuzumab mutants, the experiments need to account for the fact that the part of the HER2 antigen to which Trastuzumab binds is prone to misfolding in vitro. In the usual experimental procedure for the SPR measurements of Kd values, the antibody is immobilized and the antigen (in this case HER2) is used as a ligand. However, for the domain to which Trastuzumab is binding in HER2, taking into account the problem of misfolding, the standard procedure described by Hsiao et al. [29] is to immobilize the antigen and treat the antibody as a ligand, making the 'misfolded' HER2 essentially invisible in the binding kinetics.

As pointed out by Mason et al. [55], mutating Trastuzumab such that the resulting antibody still binds to HER2 is quite challenging, as some binders in the dataset are just one edit away from non-binders. We also check the similarity of our new binders verified in the wet-lab experiments to both the binder and non-binder sets from the dataset. As can be seen in Table 8 some of the best binders we discovered are equidistant to both sets.

To further highlight the similarity of the non-binder and the binder sets from Mason et al. [55] we include the residue frequency (logo) plots for the CDR H3 positions that were varied in Figure 5.

M Implementation and Training Details

As mentioned in the main text, all of the models are evaluated using the same denoising procedure. They also share the input embedding procedure: we utilize a learned dictionary for each amino acid type and another learned dictionary for the chain type (light or heavy). These initial embeddings are summed together to form the input to the actual model. For all models except the APMixer, an

GEY&E.

W

additional sinusoidal sequence position embedding is included in the sum. The model also receives the time step t as an additional input, concatenated with the other inputs. Similar to Hoogeboom et al. [28], we found this simple time-step conditioning to work well. The input and output of the model are always processed with our constraint layer from Section 3.4. The amino acid type predictions are made by computing a vector product with the input amino acid dictionary.

We will cover the model-specific architectural details in the following.

M.1 APMixer Details

We build ab mixer out of blocks as described in Section 3.3, where each block consists of two MLPs, that are applied consecutively on the columns

and on the rows

The whole APMixer used 8 such blocks plus an embedding MLP and two readout MLPs, one for predicting residue atom positions and another for the residue type predictions. Their outputs are respectively processed using equivariant and invariant frame averaging formulations.

As we noted earlier, each block is wrapped in frame averaging. We choose to process half of the output equivalently and half invariantly. It is possible to just use the equivariant formulation throughout the inner blocks, but we found this to slightly improve the performance of the model. Possibly because this aligns slightly better with having position-invariant sequence features. In this case, we extend the APMixer block to have a linear layer before the column MLP, to merge the two representations.

We use 2 layer MLPs throughout, with the embedding dimension of h = 1920 . The column MLP is applied on three columns at a time to ensure easier modeling of 3D interactions. Also, to facilitate easier modeling of multiplicative interactions (e.g. forces) these MLPs use gated activation in the hidden layer x = x 0: h ∗ σ ( x h :2 h ) , where x is the output of the linear layer, h is the embedding dimension (the MLPs hidden dimension is effectively 2 × h ) and σ is the activation function. We chose to use the SiLU [25] activation function as it appears to perform the best on geometric tasks, both on APMixer and the baseline GNN architectures.

M.2 Baselines

M.2.1 Transformer

For the sequence-only generation, we use a standard transformer encoder[12] as implemented by PyTorch [65], with 3 blocks, embedding dimension of 2048, and pre-normalization. The transformer uses the GELU [25] activation function as standard.

M.2.2 EGNN

We use the EGNN [75] as implemented for the equivariant molecule diffusion by Hoogeboom et al. [28]. We extend it to work with residues that have many atoms by using the formulation proposed by Huang et al. [31] and also used by Kong et al. [43], which for updating each node (residue) make use of the distances Z l ij = X i -X j between atoms in each residue:

where X i is a matrix of a × 3 if we have a atoms per residue.

To improve the model stability, we apply a layer normalization on h t at the beginning of each such block. We wrap every block in EGNN in our constraint layer Section 3.4. The EGNN has 6 such blocks and uses an embedding dimension of 576 . As standard, we use SiLU activation and, similarly to AgentNet, we found it slightly beneficial to use a hidden dimension in the MLP of twice the embedding size. The model uses a readout MLP to make the final embedding which is used to predict the amino acid type. This layer is also preceded by layer normalization. An embedding MLP is used for the inputs.

M.2.3 FA GNN

We use the same GNN architecture as used by Puny et al. [68]:

which is essentially the same architecture as EGNN, just without the distance-based edge features and explicit position updates. To process the positions, we wrap each of these GNN blocks using frame averaging, and for the very first embedding MLP we supply residue atom positions as inputs together with the other features.

The FA-GNN has 4 of these GNN blocks and uses an embedding dimension of 384 . SiLU activation and the hidden MLP dimension of twice the embedding dimension are used. The FA-GNN uses the same setup as APMixer with two readout heads for amino acid type prediction and atom position prediction.

M.3 Training

We use AdamW for training [52] with a weight decay of 0 . 01 and with a learning rate of 2 · 10 -4 for all structure models, while the transformer used a learning rate of 1 · 10 -4 . We experimented with weight decay of 1 e -12 and learning rates in the range of 1 · 10 -3 to 1 · 10 -4 for all models, to determine the chosen values. We also normalize the gradient norm to unit length during training. All models use a batch size of 4. This is the maximum batch size that allows training the baseline models on our 80GB GPU. The APMixer due to the better memory complexity allows for batch sizes up to

32, but to keep the setup similar, we also used a batch size of 4 for it. For paired OAS we train the models for 50 epochs, while for HER2 binder generation we train for 600 epochs, for around 1.25M training steps in both cases. The Paired OAS dataset [61] was split into 1000 test samples and 1000 validation samples. Similarly, the HER2 binder dataset [55] was split to have 1000 test samples and 100 validation samples.

We keep track of the exponential moving average of the model weights, for all models, with a coefficient of 0 . 995 , and use these weights at test time. No model selection is performed.

During training, we use idealized atom positions as returned by our projection layer (Section 3.4) as the target. This ensures that the exact target configuration is reachable by the model.

The CDR redesign baselines of RefineGNN [34] and MEAN [43] were trained for the same amount of steps using the default parameters and early stopping.

All experiments were performed on an 80GB A100 GPU.

M.4 APMixer Classifier

We adapt APMixer for HER2 binder classification (Section 5.2) by including a global readout. This readout is constructed by applying mean pooling over the frames and residues after each APMixer block and then applying a layer normalization and a linear layer. All of the linear layer outputs are summed to obtain the final prediction. This follows a popular graph classification readout used in GNNs [93].

The classifier is trained for 100 epochs using a batch size of 16. No exponential moving average of weights is used, and the model with the best validation accuracy is selected. The full HER2 dataset was split to have 3000 test samples and 1000 validation samples.

N Conditional CDR Redesign

While co-crystal structures usually come quite late in the drug discovery process, performing CDR structure and sequence redesign for a given, unseen co-crystal structure is a popular benchmark for deep generative models [53; 43; 44; 34]. Here, we use the data split from Luo et al. [53], derived from the SAbDab database of antibody-antigen structures [13]. This split ensures that all antibodies similar to those of the test set ( > 50% CDR H3 identity) are removed from the training set. In contrast to the experimental setup of Luo et al. [53], we report the results for the more difficult task of redesigning all CDRs at once 2 , rather than redesigning a single CDR at a time. Following Luo et al. [53], the results are averaged over 100 samples for each of the 19 test structures. We observed that in this setup the minimization protocol used by Luo et al. [53] increased the RMSD values (Table 9), therefore, we report the results without it.

To adapt AbDiffuser to this conditional case, we build a bipartite graph between the antibody CDR residues and the nearest antigen residues that are at most 10Å away, in terms of C β -C β distance. Then we simply replace the initial MLP embedding layer with a one-layer FA-GNN. The rest of the APMixer architecture remains unchanged. We trained AbDiffuser with a batch size of 8 for 30k steps, with the rest of hyperparameters staying the same as before.

Table 9 shows that in this conditional setup AbDiffuser outperforms the state-of-the-art diffusion baseline (DiffAb) in amino acid recovery (AA) by a wide margin. This good performance could be attributed to the usefulness of AHo numbering, and APMixer being better at modeling sequences than a GNN. Our ablation experiments on paired OAS generation (Table 1) corroborate this hypothesis. Although the CDR RMSD results are generally quite comparable between DiffAb and AbDiffuser, importantly, AbDiffuser performs better in CDR H3, which has the most variability in its structure and can influence the binding the most. Both methods strongly outperform RosettaAntibodyDesign (RAbD) [2], the state-of-the-art computational biology antibody design program, in terms of amino acid recovery.

CDR H1

CDR H2

CDR H3

CDR L1

CDR L2

CDR L3

Model

AA

↑

RMSD

↓

AA

↑

RMSD

↓

AA

↑

RMSD

↓

AA

↑

RMSD

↓

AA

↑

RMSD

↓

AA

↑

RMSD

↓

Rosetta (RAbD)* [2]

22.85%

2.261

25.5%

1.641

22.14%

2.9

34.27%

1.204

26.3%

1.767

20.73%

1.624

DiffAb (Minimized)* [53]

65.75%

1.188

49.31%

1.076

26.78%

3.597

55.67%

1.388

59.32%

1.373

46.47%

1.627

DiffAb (Minimized) [53]

66.37%

1.371

42.82%

1.337

28.27%

3.798

62.91%

1.520

62.59%

1.653

49.38%

1.616

DiffAb [53]

66.37%

0.802

42.82%

0.722

28.27%

3.550

62.91%

1.120

62.59%

1.025

49.38%

1.066

79.09%

1.120

72.33%

0.995

36.14%

2.921

85.08%

1.138

85.02%

1.273

75.68%

0.990

76.30%

1.584

65.72%

1.449

34.10%

3.346

81.44%

1.462

83.22%

1.397

73.15%

1.591

τ

= 0

.

75

)

79.76%

1.083

72.96%

0.950

36.53%

2.805

85.60%

1.098

85.71%

1.237

76.29%

0.955

τ

= 0

.

75

)

76.06%

1.713

66.38%

1.512

34.71%

3.214

81.93%

1.373

83.54%

1.351

73.32%

1.509

τ

= 0

.

01

)

81.11%

1.075

74.27%

0.946

37.27%

2.795

86.26%

1.115

86.85%

1.238

77.06%

0.966

τ

= 0

.

01

)

75.36%

2.463

66.89%

2.010

35.56%

3.124

83.10%

1.525

82.95%

1.623

74.19%

1.502

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces AbDiffuser, a new equivariant diffusion model, designed to generate full antibody structures efficiently and with high quality. By incorporating family-specific priors and utilizing a novel neural network architecture called APMixer, AbDiffuser demonstrates improved performance in modeling the sequence and structural properties of natural antibodies. Experimental validation shows that the generated antibody binders have high affinity, improving the affinity of a cancer drug. These findings highlight the potential of generative models in creating high-affinity antibody binders without the need for post-selection by binding predictors.\n\nSTRENGTHS:\n* The proposed approach outperforms other baseline methods across various metrics related to antibody designing. This could indicate potential effectiveness in generating high-quality antibody structures.\n\n* The novel architecture introduced in this paper exhibits a lower memory footprint during training, which is a significant advantage compared to the baseline architectures. This not only reduces resource requirements but also enhances the efficiency of sample generation, making it a more practical and scalable solution.\n\n* The design choices made in this work are backed by theoretical results and are further supported by in silico experiments. The alignment between theoretical motivations and experimental outcomes adds credibility to the overall approach.\n\nWEAKNESSES:\n* The core components of the proposed model, including the diffusion model and the mixer model, bear a strong resemblance to corresponding reference models. While there are some minor additions, such as the physics-informed residue representation, AHo-specific residue frequencies, and encoding conditional atom dependencies, these modifications may be viewed as relatively insignificant. This could raise questions about the extent of novelty and originality introduced by the proposed model.\n\n* In lines 138-139, it would be beneficial for the authors to provide further insights or explanations regarding why the second choice is expected to improve the performance. This would enhance the clarity and understanding of the motivations behind the model design decisions.\n\n* A minor typographical error is present at line 173, where a missing space between \"\\AA\" and \"while\" is observed. While this is a minor issue, attention to detail and thorough proofreading are important aspects of a scientific publication.\n\nQUESTIONS:\nTable 2 demonstrates that having more parameters is beneficial, contrary to the general notion that fewer parameters are desirable. It raises the question of why an increased number of parameters is advantageous in this specific context and how it leads to improved performance. Could the authors elaborate on the reasons behind the observed benefit of having a larger parameter count in their model?\n\nLIMITATIONS:\nThe choice of HER2 binder design presented in the paper could potentially be limited to being just one successful case study. It is important to acknowledge that the success achieved in this particular case study does not guarantee similar outcomes in other design scenarios. Providing a clear remark about this limitation would be valuable for the scientific community, considering that the drug discovery process is known to be expensive and challenging. It would help manage expectations and promote a realistic understanding of the potential generalizability and applicability of the proposed methodology beyond the specific case study presented.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes AbDiffuser, a new diffusion model for antibody structure and sequence generation. AbDiffuser proposes several ideas to improve generation: (i) Euclidean diffusion with frame averaging to achieve SE(3) equivariant diffusion, (ii) antibody sequence alignment to standardize sequence length across antibodies, (iii) atom to rigid body projections to maintain inter-residue physical constraints, and (iv) informative priors when sampling sequence and structure. The method is evaluated on the Observable Antibody Space and HER2 binder dataset against a selection of prior works but not state-of-the-art baselines. Performance is measured with how close the generated sequences match statistics of the test set as well as some physics based metrics. The authors ablate most components of the method to find it improves performance. Lastly, they demonstrate 16 of their designs fold into antibodies and have high binding affinity to HER2.\n\nSTRENGTHS:\n- AbDiffuser explores many ideas that are different to current state-of-the-art protein generation diffusion models. (1) Frame averaging for conferring SE(3) equivariance to MLPMixer over positions. (2) Aligning antibodies to a reference sequence for handling variable lengths. (3) Projecting coordinates to rigid body constraints on input and output of the denoising model. (4) Informative priors over the coordinates and sequence based on training set statistics. Each component of the method seems to help based on the task and metrics defined in the paper.\n\n- I appreciate the wet-lab validation of their designs which is almost never present in protein papers at machine learning conferences.\n\nWEAKNESSES:\n- **Biggest issue: poor presentation**. There are simply too many new ideas that comprise AbDiffuser and all the details are left to the appendix. The reader should not have to go through 28 pages (main text and appendix) to understand the method. Even after looking through the appendix, I am unsure what the full algorithm is. In fact, there is no algorithm or figure of how AbDiffuser works from training to sampling.\n\n- **Missing motivation for frame averaging**. Frame averaging (FA) is introduced without motivation. On line 37-38, the authors state previous methods that confer SE(3) equivariance and physical constraints come with \"increased complexity\". These complexities are never spelled out. How is AbDiffuser simpler? The paper proposes frame averaging with informative priors *and* the rigid body projection step. This seems quite complicated to me. Additional points on FA:\n\n- **Missing related work**.\n  - FA for proteins. the authors fail to cite [1] which already proposed FA for proteins.\n  - The projection step seems related to the projection step, \"Equivariant Consensus Structure from Weighted Inter-residue Geometries\", in Appendix L of Chroma but this connection is not mentioned.\n\n- **Gaussian diffusion plus projection step vs. SE(3) diffusion**. The projection step seems to me as a way to circumvent working with rigid body frames by always projecting the gaussian diffusion over coordinates onto the rigid body constraints. But this seems close to SE(3) diffusion [2] over frames where the C-a coordinates diffuse along with the backbone atoms which always have the rigid body constraints. Why is the gaussian diffusion and projection step necessary to develop if we have SE(3) diffusion? Furthermore, SE(3) diffusion is not compared with so we do not know which is better.\n\n- **Unjustified claims**. I have concerns with several claims in the paper.\n  - Line 3 claims AbDiffuser uses a new representation of protein structure which as I understand is the FA component. But as stated this has already been done [1].\n  - Line 5-6 \"Our approach improves protein diffusion ... handles sequence-length changes\". This is not true. The authors achieve this by aligning all antibodies to a reference alignment (the AHo numbering) that has been specifically developed for antibodies. This does not exist for nearly all proteins so their method cannot be applied. Even if it can be applied, the authors cannot claim it works for other proteins until it is demonstrated.\n  - Line 67 - 69 \"These results provide the first experimental evidence that a generative model trained on mutagenesis data can create new antibody binders...\". This is not true. [4] is just one paper that showed new antibody binders can be created with generative models. In addition, RFdiffusion [5] which can generate de novo binders is not mentioned.\n  - Title \"AbDiffuser: full-atom generation of in-vitro functioning antibodies\". This a bold claim that is **not justified** by the results of generating 16 functioning antibodies for a single target HER2. Furthermore, it is unclear how novel these antibodies are.\n  - Line 87. The authors claim APMixer is a novel neural network for processing proteins. The claimed novelty seems very incremental. MLPMixer is not new, FA is not new. The only component which I believe is novel is the sequence alignment of each antibody such that they all have the same sequence length as input. But this does not warrant technical novelty. Furthermore, I don't understand why SE(3) universality isn't already provided by FA.\n\n- **Unclear experimental set-up**. The experimental set-up is poorly written. The metrics are not clearly defined in the main text. The training dataset is unclear: OAS does not have structures so how are the structures obtained for training? Do the authors train RefineGNN and MEAN baselines only on the CDR which they were designed for or also give them the full sequence? How are the splits performed? The Wasserstein distance metric has no equation so I am unclear what it is exactly measuring. Why is there no metric on diversity and novelty of the sequences? Perhaps this information is buried in the appendix but it should be up front in the main text.\n\n- Line 250-252. Why would we expect IgLM to have the best accuracy just because it was trained on more data? IgLM is not trained on the structure like AbDiffuser or the other baselines.\n\n- **Experimental results**. The authors claim they have designed binders which have been tested in the wet-lab but this result must be accompanied by experimental results in the supplementary or plots in the supplement. This is standard in bio journals. In addition it is unclear how novel each design is. If only a few mutations were necessary to obtain a tight binder then this task is not difficult.\n\n- **Lack of baselines**. The authors miss out on state-of-the-art. On top of the ones already discussed there have been diffusion models for antibody generation [6, 7].\n\n[1] https://arxiv.org/pdf/2301.10814.pdf\n\n[2] https://arxiv.org/abs/2302.02277\n\n[3] https://www.biorxiv.org/content/10.1101/2022.12.01.518682v1.full.pdf\n\n[4] https://www.biorxiv.org/content/biorxiv/early/2023/03/29/2023.01.08.523187.full.pdf\n\n[5] https://www.biorxiv.org/content/10.1101/2022.12.09.519842v1\n\n[6] https://www.biorxiv.org/content/10.1101/2022.07.10.499510v2\n\n[7] https://arxiv.org/abs/2302.00203\n\nQUESTIONS:\nMy questions have been written above.\n\nLIMITATIONS:\nLimitations are not discussed though I believe there are many issues with the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new approach, called AbDiffuser, that improves protein diffusion by leveraging domain knowledge and physics-based constraints. They follow MLPMixer's architecture to reduce memory complexity by an order of magnitude, enabling backbone and side chain generation. They validate AbDiffuser in silico and in-vitro.\n\nSTRENGTHS:\n- APMixer architecture significantly reduces GPU memory, which is an interesting improvement since many related works rely on a triangular update mechanism that has O(N3) complexity. It is interesting whether such an approach can be adopted for more general tasks, such as protein folding and design.\n- The projection layer looks interesting, as it helps maintain the intermediate diffusion state on the \"manifold\" of valid structures. Similar to the above, it remains unknown whether this may benefit general protein design.\n- The authors validate the method in-vitro, increasing the solidness and usefulness.\n\nWEAKNESSES:\n- AbDiffuser does not directly use epitope information, which may harm generalization. For example, given a newly seen antigen, how can this method be applied to design an antibody that has a high binding affinity?\n- Since the sequence dataset is folded with IgFold (line 271), it may indicate that AbDiffuser's \"folding\" performance is bounded by IgFold. The model is ranked with another classifier trained with binding probability (line 316). It appears that the IgFold and classifier act as the model's \"teacher.\" How does the model outperform its \"teacher\"? If we have such a \"teacher\", why not try to use it directly, with MCMC, Bayesian optimization, etc?\n- In line 314, a random mutation approach may generate 9k binders and 25k non-binders, indicating that random mutation causes a success rate of 9/(9+25)=26%. In line 66, AbDiffuser has a success rate of 22.2%. Although the number may increase to 57.1% by post processing, does this mean that \"AbDiffuser performs worse than a random mutation baseline\"?\n\nQUESTIONS:\n- During forward diffusion, is the residue type continuous or discrete? If it is continuous, how do you perform \"backbone/side-chain projection\" (section 3.4) since the type of residue is undecided? If it is discrete, during the forward process (by interpolating between a prior distribution and the final one-hot distribution), will the residue type remain unchanged? For example, assuming there are 5 amino acid types, the true state is (0, 0, 0, 0, 1) and the prior distribution is (0.2, 0.2, 0.2, 0.2, 0.2), the residue type will always be the 5th state.\n- Are the priors computed over the entire dataset, instead of some antibody family? What would be the performance if directly sampling from the sequence and structure prior?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAntibody design has an extreme importance for both fundamental and application biologic science. The submission intriduces an equivariant and physics-informed diffusion model called AbDiffuser\n\nSTRENGTHS:\nOriginality: There are few tools that have already addressed the same problem (for example, DiffAb, https://github.com/luost26/diffab). However, due to the lack of systematic evaluation and comparison between the approaches, AbDiffuser remains useful and important new tool. \nQuality: The submission provides both in silico and in vitro validation that is crucial for antibody design. Authors provide a detailed supplementary with clear explanation of used methods and approaches. However, the paper lack a proper discussion about potential weaknesses and limitations of AbDiffuser.\nClarity: The test is clear and all the sections are presented in structures manner. \nSignificance: The results are important and useful for biology and drug discovery in particular. AbDiffuser can be used by researchers for a fast and efficient design of antibodies to particular antigen.\n\nWEAKNESSES:\nEven though the paper provides comparison of AbDiffuser with other models, it lacks comparison to specific to antibody-antigen diffusion models (for example, https://github.com/luost26/diffab). It can be useful to discuss AbDiffuser compared to these tools. Also, there are several new protein-focused diffusion models such as RFdiffusion (https://github.com/RosettaCommons/RFdiffusion) or DiffDock-PP (https://github.com/ketatam/DiffDock-PP), the discussion of or comparison to can be valuable. I suggest extending the \"Related Work\" section to highlight the mentioned above issues.\n\nQUESTIONS:\nIt can be useful to discuss AbDiffuser compared to other diffusion tools focused on proteins and antibodies such as RFdiffusion, DiffDock-PP, and DiffAb ((https://github.com/RosettaCommons/RFdiffusion, https://github.com/ketatam/DiffDock-PP, https://github.com/luost26/diffab)\n\nLIMITATIONS:\nAuthors need to include a section about limitations and weaknesses of AbDiffuser as well as potential improvement of the method.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}